Method, apparatus, and computer-readable storage medium with instructions for controlling a display of an augmented reality head-up display device for a motor vehicle

Description

[0001] The present invention relates to a method, a device, and a computer-readable storage medium with instructions for controlling a display of an augmented reality head-up display device for a motor vehicle. The invention further relates to a motor vehicle in which a method or device according to the invention is used.

[0002] With the continuous development of virtual and augmented reality technologies and applications, these are also finding their way into the automobile. Augmented reality (AR) is the enhancement of the real world with virtual elements that are accurately registered in three-dimensional space and enable real-time interaction. Since the term "augmented reality" has prevailed over "extended reality" in the German-speaking world, the former will be used below.

[0003] One possible technical implementation for enhancing the driver's workplace with perspective-correct virtual extensions is the head-up display (HUD). The light rays from a display installed in the dashboard are folded across several mirrors and lenses and reflected onto a projection surface into the driver's eye, allowing them to perceive a virtual image outside the vehicle. In the automotive sector, the windshield is often used as a projection surface, and its curved shape must be taken into account when displaying the image. As an alternative, an additional pane of glass or plastic is sometimes used, which is positioned on the dashboard between the driver and the windshield. The optical overlay of the display and the driving scene means fewer head and eye movements are required to read the information.In addition, the adaptation effort for the eyes is reduced because, depending on the virtual distance of the display, less or no accommodation is required. Current series HUDs, however, are limited to displays directly below the driver's primary field of vision and contain redundant information, such as the speedometer, which can also be found elsewhere in the cockpit. Although this display technology reduces the number of eyes being diverted from the road, it still has the disadvantage that the presented information must be interpreted and transferred to the real situation because it is not registered in the real scene. This can represent a mentally demanding cognitive process in complex situations. By highlighting objects and displaying information at their real reference location, i.e. using a contact-analog display, environmental information can be presented directly in the driver's field of vision.This direct graphical enrichment of the environment in the form of augmented reality can significantly reduce cognitive transfer requirements.

[0004] In this context, DE 10 2010 013 232 A1 discloses a head-up display for a motor vehicle. The head-up display comprises a display with light-emitting particles or microstructures over a predefined area of ​​the windshield, allowing a luminescent display. Eye detection and head detection devices enable an estimation of the eye location and registration of images in the head-up display so that the images correspond to the operator's view. Map data can be used to determine lane geometries.

[0005] DE 101 38 719 A1 describes a method for displaying driving instructions, particularly in car navigation systems. When calculating a perspective view from the driver's perspective, the method takes into account the inclination of the vehicle around its longitudinal and transverse axes relative to the road surface, as well as the inclination of the road or the three-dimensionality of the terrain, when generating a displayed driving instruction. This allows for a better match between the image of the surroundings in the driver's field of vision captured by the vehicle camera and the displayed driving instruction.

[0006] Augmented reality offers a wide range of application possibilities for supporting drivers through contact-analog marking of lanes and objects. Relatively obvious examples mostly relate to the field of navigation. While classic navigation displays in conventional HUDs generally show schematic representations, e.g., a right-angled arrow indicating that a right turn should be made at the next opportunity, AR displays offer significantly more effective options. Because the displays can be presented as "part of the environment," extremely quick and intuitive interpretations by the user are possible. In modern motor vehicles, a wide variety of data is recorded by environmental sensors. Current developments aim to meaningfully integrate this data into augmented reality displays. An important component of the display is data that is placed directly on the road surface.

[0007] US 9,625,270 B2 describes a control method for an electronic device. In the method, the position of a vehicle being operated is determined. Furthermore, using map data, information about a guidance point located a predetermined distance ahead of the determined position of the vehicle is acquired. Using the information about the guidance point, an object is generated that indicates the guidance point. The thus generated object is finally displayed using an augmented reality head-up display.

[0008] US 2013 / 0194110 A1 describes an augmented reality head-up display for a motor vehicle. The augmented reality head-up display includes a viewing angle calculation unit for estimating a direction of a driver's line of sight using a facial orientation detected based on images of the driver's face and the center positions of the pupils, and calculating a viewing angle. A matching unit matches a position of real-world information located in front of a driver's seat with a position of corresponding information about virtual objects located in front of the driver's seat based on the direction of the line of sight and the viewing angle. A display unit displays results of the adjustment by the display unit, and the display unit enables rotation and relocation of the display.

[0009] DE 10 2012 210 375 A1 describes a method for operating a head-up display of a vehicle. In the method, predefined route data are provided for a route lying in front of the vehicle in the direction of travel. Furthermore, at least one predefined predictive operating variable is provided for the vehicle. Depending on the route data, a three-dimensional model for a predefined environment of the vehicle is determined. Furthermore, a graphic element is determined depending on the three-dimensional model and the at least one predictive operating variable. The head-up display has a predefined display field. The graphic element is displayed on the predefined display field, wherein the graphic element is determined and displayed in such a way that it is perceived by a viewer of the display field from a predefined viewing position with a direct reference to the real environment of the vehicle.

[0010] Registering virtual information in the driver's real field of vision in augmented reality places very high demands on the technical implementation. To be able to display virtual content in reality with precise location and perspective, very detailed knowledge of the vehicle's surroundings and its own movement is necessary.

[0011] There are a number of challenges with regard to registration. For example, the specific plane in or on which a virtual object is to be displayed depends on the orientation of the vehicle. For example, the vehicle's payload significantly influences the vehicle's angle of inclination. In addition, the course of the road is usually not planar. Depending on the evenness of the surroundings, the road surface must therefore be continuously measured. However, due to possible traffic in front of the vehicle, the road plane may be partially obscured, meaning that it cannot be detected by the vehicle's sensors, or can only be detected with difficulty.

[0012] It is an object of the invention to provide improved solutions for controlling a display of an augmented reality head-up display device.

[0013] This object is achieved by a method having the features of claim 1, by a computer-readable storage medium with instructions according to claim 6, and by a device having the features of claim 7. Preferred embodiments of the invention are the subject of the dependent claims.

[0014] According to a first aspect of the invention, a method for controlling a display of an augmented reality head-up display device for a motor vehicle comprises the steps: Determining a pose of the motor vehicle relative to a map; extracting reference points from the map, wherein the reference points originate from an area within a field of vision of a driver of the motor vehicle and are calculated from a pre-calculated geometric abstraction of a topography stored in the map, and wherein the reference points are extracted from a map area in the direction of travel of the motor vehicle, which map area comprises a basic area and an extended area dependent on a speed of the motor vehicle; transforming the reference points into vehicle coordinates, wherein the transformed reference points represent the course of an area in or on which a virtual object is to be displayed;Generating the virtual object for contact-analog display by the augmented reality head-up display device, taking into account the transformed reference points, wherein graphic elements of the virtual object are adapted to a course of the transformed reference points in such a way that the optical impression is created that the graphic elements lie in the area represented by the transformed reference points or stand on this area; and outputting the virtual object for display by the augmented reality head-up display device.

[0015] According to another aspect of the invention, a computer-readable storage medium contains instructions that, when executed by a computer, cause the computer to perform the following steps for controlling a display of an augmented reality head-up display device for a motor vehicle: Determining a pose of the motor vehicle relative to a map; extracting reference points from the map, wherein the reference points originate from an area within a field of vision of a driver of the motor vehicle and are calculated from a previously calculated geometric abstraction of a topography stored in the map, and wherein the reference points are extracted from a map area in the direction of travel of the motor vehicle, which map area comprises a basic area and an extended area dependent on a speed of the motor vehicle; transforming the at least one reference point into vehicle coordinates, wherein the transformed reference points represent the course of an area in or on which a virtual object is to be displayed;Generating the virtual object for contact-analog display by the augmented reality head-up display device, taking into account the transformed reference points, wherein graphic elements of the virtual object are adapted to a course of the transformed reference points in such a way that the optical impression is created that the graphic elements lie in the area represented by the transformed reference points or stand on this area; and outputting the virtual object for display by the augmented reality head-up display device.

[0016] The term "computer" should be understood broadly. In particular, it also includes control units and other processor-based data processing devices.

[0017] According to a further aspect of the invention, a device for controlling a display of an augmented reality head-up display device for a motor vehicle comprises: a pose determination unit for determining a pose of the motor vehicle relative to a map; an extraction unit for extracting reference points from the map, wherein the reference points originate from an area within a field of vision of a driver of the motor vehicle and are calculated from a pre-calculated geometric abstraction of a topography stored in the map, and wherein the reference points are extracted from a map area in the direction of travel of the motor vehicle, which map area comprises a basic area and an extended area dependent on a speed of the motor vehicle; a transformation unit for transforming the at least one reference point into vehicle coordinates, wherein the transformed reference points represent the course of an area in or on which a virtual object is to be displayed;a graphics unit for generating the virtual object for contact-analog display by the augmented reality head-up display device, taking into account the at least one transformed reference point, wherein graphic elements of the virtual object are adapted to a course of the transformed reference points in such a way that the optical impression is created that the graphic elements lie in the area represented by the transformed reference points or stand on this area; and an output for outputting the virtual object for display by the augmented reality head-up display device.

[0018] The solution according to the invention uses a highly accurate map with information about at least one street level, which was surveyed in advance. This information is used to generate a correct augmented reality display. The map also forms the basis for highly accurate localization of the vehicle. The vehicle first locates itself in terms of position and orientation relative to the map. Reference points for the desired display level, for example ground points, are then determined from the map data. These reference points are transformed into vehicle coordinates and represent the area in or on which a virtual object is to be displayed. To generate the augmented reality display of the head-up display, the graphic elements of the virtual object are adapted to the course of the transformed reference points, creating the visual impression that the graphic elements are located in the desired area.stand on the desired surface.

[0019] According to the invention, the reference points are calculated from a previously calculated geometric abstraction of a topography stored in the map. For example, the geometric abstraction of the topography can comprise one-dimensional or two-dimensional polynomials or a grid. To keep the map's data volume within reasonable limits, it is advisable to store only an approximate description of the topography in the map. Polynomials or a grid are suitable for such an abstraction. The reference points can then be easily calculated or interpolated from the map data as needed.

[0020] According to the invention, the reference points are extracted from a map area in the direction of travel of the motor vehicle. The map area comprises a basic area and an extended area dependent on the speed of the motor vehicle. By extracting reference points from a map area in the direction of travel of the motor vehicle, it is ensured that the reference points are reliably within the driver's field of vision. The extended area dependent on the speed of the motor vehicle can ensure, particularly at higher speeds, that a suitable reference point is already available even for a more distant area that can be reached in a relatively short time due to the higher speed.

[0021] According to one aspect of the invention, the area within the driver's field of vision from which the reference points originate is a ground surface. Typically, the virtual elements are intended to be perceived as lying or standing on the ground. For this purpose, knowledge of the ground surface is required. Furthermore, the ground surface, i.e., typically the road surface, can be relatively easily captured in advance for the purpose of providing the map data.

[0022] According to one aspect of the invention, during the generation of the virtual object, graphic elements of the virtual object are transformed and distorted onto the ground surface. This creates the visual impression that the virtual object, for example, a navigation instruction or a warning notice, is lying directly on the ground.

[0023] According to one aspect of the invention, to determine the pose of the motor vehicle relative to the map, features detected by a sensor system of the motor vehicle are correlated with features of the map. A vehicle can locate itself with high precision when driving in an already surveyed environment or map. The features detected by the vehicle are correlated with the previously surveyed features to determine the position and orientation of the vehicle. Examples of features that can be used for this purpose include lane markings, other ground markings, road edges, traffic lights, traffic signs, or landmarks near the roadway.

[0024] A method according to the invention or a device according to the invention is particularly advantageously used in a vehicle, in particular a motor vehicle.

[0025] Further features of the present invention will become apparent from the following description and the appended claims taken in conjunction with the figures. Fig. 1 schematically shows a method for controlling a display of an augmented reality head-up display device for a motor vehicle; Fig. 2 shows a first embodiment of a device for controlling a display of an augmented reality head-up display device for a motor vehicle; Fig. 3 shows a second embodiment of a device for controlling a display of an augmented reality head-up display device for a motor vehicle; Fig. 4 schematically shows a motor vehicle in which a solution according to the invention is implemented; Fig. 5 schematically shows the general structure of a head-up display device for a motor vehicle; Fig. 6 shows an example of an augmented reality representation of a head-up display device; and Fig. 7 shows a system design of the solution according to the invention for controlling a display of an augmented reality head-up display device for a motor vehicle.

[0026] To better understand the principles of the present invention, embodiments of the invention are explained in more detail below with reference to the figures. It is understood that the invention is not limited to these embodiments and that the described features may also be combined or modified without departing from the scope of the invention as defined in the appended claims.

[0027] Fig. 1schematically shows a method for controlling a display of an augmented reality head-up display device for a motor vehicle. In a first step, a pose of the motor vehicle relative to a map is determined 10. For this purpose, features detected by a sensor system of the motor vehicle can be correlated with features of the map. Subsequently, reference points are extracted from the map 11, wherein the reference points originate from an area within the field of vision of a driver of the motor vehicle. The reference points are extracted from a map area in the direction of travel of the motor vehicle. The map area comprises a basic area and an extended area dependent on a speed of the motor vehicle. The reference points are calculated from a previously calculated geometric abstraction of a topography that is stored in the map, e.g.in the form of one-dimensional or two-dimensional polynomials or as a grid. The area in the driver's field of vision can be, for example, a floor surface. After a transformation 12 of the reference points into vehicle coordinates, a virtual object is generated 13 for display by the augmented reality head-up display device, taking the transformed reference points into account. Graphic elements of the virtual object are transformed onto the area, in particular a floor surface, and distorted. The virtual object is finally output 14 for display by the augmented reality head-up display device.

[0028] Fig. 2shows a simplified schematic representation of a first embodiment of a device 20 for controlling a display of an augmented reality head-up display device for a motor vehicle. The device 20 has an input 21 via which data from a sensor system can be received, e.g. from a front camera of the motor vehicle or from lidar, radar, or ultrasonic sensors. The device 20 also has a pose determination unit 22 for determining a pose of the motor vehicle relative to a map. For this purpose, the pose determination unit 22 can, for example, correlate features detected by a sensor system of the motor vehicle with features of the map. An extraction unit 23 extracts reference points from the map, wherein the reference points originate from an area in a field of vision of a driver of the motor vehicle. The extraction unit 23 extracts the reference points from a map area in the direction of travel of the motor vehicle.The map area comprises a basic area and an extended area dependent on the speed of the motor vehicle. The reference points are calculated from a previously calculated geometric abstraction of a topography that is stored in the map, e.g. in the form of one-dimensional or two-dimensional polynomials or as a grid. The area in the driver's field of vision can be, for example, a ground surface. A transformation unit 24 transforms the reference points into vehicle coordinates. A graphics unit 25 then generates a virtual object for display by the augmented reality head-up display device, taking the transformed reference points into account. The graphics unit 25 transforms and distorts graphic elements of the virtual object onto the area, in particular a ground surface.The generated virtual object is output to a control unit 42 of the augmented reality display device via an output 28 of the device 20. Alternatively, only an instruction to generate a corresponding virtual object can be output to the control unit 42. The control unit 42 can then insert the generated virtual object into a display of the augmented reality head-up display device.

[0029] The pose determination unit 22, the extraction unit 23, the transformation unit 24, and the graphics unit 25 can be controlled by a control unit 26. Settings of the pose determination unit 22, the extraction unit 23, the transformation unit 24, the graphics unit 25, or the control unit 26 can be changed via a user interface 29 if necessary. The data generated in the device 20 can be stored in a memory 27 of the device 20, for example, for later evaluation or for use by the components of the device 20. The pose determination unit 22, the extraction unit 23, the transformation unit 24, the graphics unit 25, and the control unit 26 can be implemented as dedicated hardware, for example, as integrated circuits.Of course, they can also be partially or completely combined or implemented as software running on a suitable processor, for example, a GPU. The input 21 and the output 28 can be implemented as separate interfaces or as a combined bidirectional interface. Preferably, the device 20 is integrated into the control unit 42 of the augmented reality head-up display device.

[0030] Fig. 3shows a simplified schematic representation of a second embodiment of a device 30 for controlling a display of an augmented reality head-up display device for a motor vehicle. The device 30 has a processor 32 and a memory 31. For example, the device 30 is a computer or a control unit. Instructions are stored in the memory 31 which, when executed by the processor 32, cause the device 30 to carry out the steps according to one of the described methods. The instructions stored in the memory 31 thus embody a program executable by the processor 32 which implements the method according to the invention. The device 30 has an input 33 for receiving data from a sensor system, e.g., from a front camera of the motor vehicle or from lidar, radar, or ultrasonic sensors. Data generated by the processor 32 is provided via an output 34.In addition, they can be stored in memory 31. The input 33 and the output 34 can be combined to form a bidirectional interface.

[0031] The processor 32 may include one or more processor units, such as microprocessors, digital signal processors, or combinations thereof.

[0032] The memories 27, 31 of the described embodiments can have both volatile and non-volatile memory areas and can comprise a wide variety of storage devices and storage media, for example hard disks, optical storage media or semiconductor memories.

[0033] Fig. 4schematically illustrates a motor vehicle 40 in which a solution according to the invention is implemented. The motor vehicle 40 has an augmented reality head-up display device 41 with an associated control unit 42. Furthermore, the motor vehicle 40 has a device 20 for controlling a display of the augmented reality head-up display device 41. The device 20 can also be integrated into the control unit 42 for the augmented reality head-up display device 41. Further components of the motor vehicle 40 are a sensor system 43, a navigation system 44, a data transmission unit 45, and a series of assistance systems 46, one of which is shown as an example. By means of the data transmission unit 45, for example, a connection to service providers can be established.Using the sensor system 43, which may include, for example, a camera as well as lidar, radar, or ultrasonic sensors, information about the roadway and the vehicle's surroundings can be collected, among other things. A memory 47 is provided for storing data. Data exchange between the various components of the motor vehicle 40 takes place via a network 48.

[0034] Fig. 5shows a schematic diagram of a head-up display device 41 for a motor vehicle 40, with the aid of which content can be displayed on a projection surface 52 of the motor vehicle 40, for example on the windshield or on an additional pane made of glass or plastic, which is arranged between the driver and the windshield on the dashboard. The displayed content is generated by an imaging unit 50 and projected onto the projection surface 52 with the aid of an optical module 51. Typically, the projection takes place in an area of ​​the windshield above the steering wheel. The imaging unit 50 can be an LCD-TFT display, for example. The augmented reality head-up display device 41 is generally installed in a dashboard of the motor vehicle 40.

[0035] Fig. 6shows an example of an augmented reality representation of an augmented reality head-up display device. A number of elements are superimposed on the actual environment. In the example in Fig. 6 These include the current speed, the distance to a vehicle ahead, and navigation information. In addition, several virtual objects 60 are shown in a contact-analog representation, specifically an indicator arrow for a possible or impending overtaking maneuver, a visual representation of the distance to a vehicle ahead, and a "virtual wall" on the right sideline as a safety warning. The elements displayed are predominantly based on information acquired by vehicle sensors. The specialist has a wide range of design options available for the specific design of the display.

[0036] Fig. 7shows a simplified system design of the inventive solution for controlling a display of an augmented reality head-up display device 41 for a motor vehicle. A map 61 is stored in the motor vehicle, which contains, among other things, information on previously measured static features. Examples of such static features are lane markings, other road markings, road edges, traffic lights, traffic signs, or landmarks near the roadway. The map 61 can be provided by a service provider and stored, for example, in a memory of the navigation system. The map-based features for the area immediately in front of the motor vehicle are transferred to the device 20 for controlling the display of the augmented reality head-up display device 41. The PSD protocol (PSD: predictive route data) can be used for this purpose, for example.By correlating static features detected from sensor data of a sensor system 43 with the static features known from the map 61, the pose of the motor vehicle relative to the map 61 is determined 10. Taking into account the pose thus determined, at least one reference point is extracted from the map 61 11. For this purpose, the map contains information on the topography, preferably in the form of a geometric abstraction of the topography. The exact position and orientation of the sensor system 43 is subject to manufacturing tolerances and can vary between different motor vehicles. The sensor system 43 is therefore calibrated during production or its position and orientation are estimated during operation. The resulting parameters are stored in the vehicle. The reference point is therefore converted to a vehicle reference coordinate system or target coordinate system 62 according to the associated transformation rule 12, e.g.the center of the front axle. This eliminates all inaccuracies affecting the sensor system 43. Finally, taking the transformed reference point into account, a virtual object is generated 13 and output to a control unit 42 of the augmented reality head-up display device 41. Alternatively, an instruction can simply be output to the control unit 42 to generate a corresponding virtual object. The control unit 42 can then insert the generated virtual object into a display of the augmented reality head-up display device 41. List of reference symbols

[0037] 10Determining a pose of the vehicle relative to a map 11Extracting reference points from the map 12Transforming the reference points into vehicle coordinates 13Generating a virtual object for display 14Outputting the virtual object for display 20Device 21Input 22Pose determination unit 23Extraction unit 24Transformation unit 25Graphics unit 26Control unit 27Memory 28Output 29User interface 30Device 31Memory 32Processor 33Input 34Output 40Motor vehicle 41Augmented reality head-up display device 42Control unit of the augmented reality head-up display device 43Sensor system 44Navigation system 45Data transmission unit 46Assistance system 47Memory 48Network 50Imaging unit 51Optical module 52Projection surface 60Virtual object 61Map data 62Target coordinate system

Claims

1. Method for controlling a display of an augmented reality head-up display device (41) for a motor vehicle (40), comprising the following steps: - determining (10) a pose of the motor vehicle (40) relative to a map; - extracting (11) reference points from the map, wherein the reference points originate from a surface in a field of view of a driver of the motor vehicle (40) and are calculated from a previously calculated geometric abstraction of a topography stored in the map, and wherein the reference points are extracted from a map region in the direction of travel of the motor vehicle (40), which map region comprises a main region and an extended region dependent on a speed of the motor vehicle (40); - transforming (12) the reference points into vehicle coordinates, wherein the transformed reference points represent the course of a surface in or on which a virtual object (60) is to be displayed; - generating (13) the virtual object (60) for contact-analogous display by the augmented reality head-up display device (41), taking into account the transformed reference points, wherein graphic elements of the virtual object (60) are adapted to a course of the transformed reference points in such a way that the visual impression is created whereby the graphic elements lie in the surface represented by the transformed reference points or are positioned on this surface; and - outputting (14) the virtual object (60) for display by the augmented reality head-up display device (41).

2. Method according to claim 1, wherein the geometric abstraction of the topography comprises one-dimensional or two-dimensional polynomials or a grid.

3. Method according to claim 1 or 2, wherein the surface in the field of vision of the driver of the motor vehicle (40) from which the reference points originate is a ground surface.

4. Method according to claim 3, wherein, during the generation (12) of the virtual object (60), graphic elements of the virtual object (60) are transformed and distorted onto the ground surface.

5. Method according to any of the preceding claims, wherein features which were detected by a sensor system (43) of the motor vehicle (40) are correlated with features of the map in order to determine (10) the pose of the motor vehicle (40) relative to the map.

6. Computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to execute the steps of a method according to any of claims 1 to 5 for controlling a display of an augmented reality head-up display device (41) for a motor vehicle (40).

7. Device (20) for controlling a display of an augmented reality head-up display device (41) for a motor vehicle (40), comprising: - a pose determination unit (22) for determining (10) a pose of the motor vehicle (40) relative to a map; - an extraction unit (23) for extracting (11) reference points from the map, wherein the reference points originate from a surface in a field of view of a driver of the motor vehicle (40) and are calculated from a previously calculated geometric abstraction of a topography stored in the map, and wherein the reference points are extracted from a map region in the direction of travel of the motor vehicle (40), which map region comprises a main region and an extended region dependent on a speed of the motor vehicle (40); - a transformation unit (24) for transforming (12) the reference points into vehicle coordinates, wherein the transformed reference points represent the course of a surface in or on which a virtual object (60) is to be displayed; - a graphics unit (25) for generating (13) the virtual object (60) for contact-analogous display by the augmented reality head-up display device (41), taking into account the transformed reference points, wherein graphic elements of the virtual object (60) are adapted to a course of the transformed reference points in such a way that the visual impression is created whereby the graphic elements lie in the surface represented by the transformed reference points or are positioned on this surface; and - an output (28) for outputting (14) the virtual object (60) for display by the augmented reality head-up display device (41).

8. Motor vehicle (40) having an augmented reality head-up display device (41), characterized in that the motor vehicle (40) comprises a device (20) according to claim 7 or is designed to execute a method according to any of claims 1 to 5 for controlling a display of the augmented reality head-up display device (41).