USER INTERFACE, MEANS OF MOTION AND METHOD FOR DISPLAYING A VIRTUAL THREE-DIMENSIONAL NAVIGATION MAP

Description

[0001] The present invention relates to a means of transportation, a user interface, and a method for displaying a virtual three-dimensional navigation map in a means of transportation. In particular, the present invention relates to a particularly intuitively understandable representation of a navigation project and intuitively understandable guidance of a user along a route.

[0002] Navigation systems in vehicles typically use display elements located in the instrument cluster, a central information display, and / or a head-up display. Sometimes, the display presents (inexperienced) users with difficulties in interpreting the information, as details are relatively small and rotated relative to reality. This is particularly due to the fact that the displays installed in vehicles are typically oriented vertically rather than horizontally, whereas a road map would naturally be best understood horizontally.

[0003] Augmented reality displays are also known, allowing users to view artificial / virtual content in a predefined relationship to the translucent reality. This requires the augmented reality glasses / display to become familiar with the environment, or rather, to establish a relationship between the display content and reality. Only then can the display elements be displayed in a quasi-contact-analog manner with the environment and used to improve the user's information.

[0004] From WO 2016 / 014878 A1 the representation of a 3D object on the surface of an environment using augmented reality is known.

[0005] DE 10 2013 210746 A1 discloses the display of a virtual navigation map on a surface of an interior of a means of transport by means of an augmented reality display device.

[0006] Based on the aforementioned prior art, it is an object of the present invention to provide a user with the best possible support in a navigation project. Disclosure of the invention

[0007] The above-mentioned object is achieved according to the invention by a method for displaying a virtual three-dimensional navigation map in a means of transport. The three-dimensional navigation map can be present as a data model in a data storage device of the means of transport or can be gradually streamed into the means of transport. Alternatively or additionally, the three-dimensional navigation map can also be stored in a local data storage device of an augmented reality (hereinafter also "AR") display device. The augmented reality display device is located in the means of transport and can be capable of optically scanning a surface of the interior of the means of transport, in particular a dashboard of the means of transport, or of storing a data set representing the surface.This process can also be understood as calibrating the augmented reality display device to the interior surface of the vehicle. In this process, a spatial relationship between the surfaces of the interior of the vehicle and the position of the augmented reality display device is made known to the augmented reality display device, or this relationship is established in such a way that the augmented reality display device learns its own position as well as the position of the respective surface areas of the interior. In this process, the augmented reality display device is taught, so to speak, to assign certain environmental areas to the surface of the interior of the vehicle, while other areas (e.g., window areas) are assigned to an inertial system located outside the vehicle. In addition, a three-dimensional navigation map is loaded from a data storage device.This data storage can be provided within the means of transport, in particular within the augmented reality display device, and / or outside the means of transport. In particular, for data storage located outside the means of transport, an over-the-air IT connection (via WiFi, WLAN, LTE, LTE-A, etc.) can be provided.

[0008] Due to the optical scanning or calibration, the shape / form of the surface of the vehicle's interior is known. The three-dimensional navigation map or 3D map model can now be adapted to the vehicle's surface. In other words, the navigation map can be "virtually" adhered to the entire surface of a curved interior of the vehicle, while the three-dimensional navigation map is usually defined and displayed in a plane (e.g., the screen plane or similar). The adapted three-dimensional navigation map, along with a representation of the vehicle's ego position, is then displayed as a virtual, raised model on the surface using the augmented reality display device.As is common with conventional navigation displays, either the ego position of the vehicle is guided through the streets or the ego position of the vehicle is fixed, while the adapted three-dimensional navigation map is moved according to the movement of the actual vehicle. This form of display has several advantages: On the one hand, a particularly large display of the navigation map can be achieved on the dashboard, as there is comparatively more space there and the size of the screens in the vehicle is no longer a limiting factor. On the other hand, the essentially horizontal orientation of the majority of the dashboard areas is suitable for creating a particularly intuitively understandable display of the road map, as the viewer does not need to adjust / interpret the perspective.In addition, the augmented reality display device can create a particularly intuitive 3D representation of the road map, so that the viewer receives a realistic and perspective-realistic display of the surroundings of the ego vehicle.

[0009] The subclaims show preferred developments of the invention.

[0010] As already mentioned above, the surface in the interior of the means of transport can be optically scanned in a preceding step. This step can also be understood as scanning or measuring / localizing the surface of the interior. It can take place (automatically) in the respective vehicle, for example, after an initial pairing of an augmented reality display device with the means of transport. Alternatively or additionally, this step can be carried out at the factory to provide an initial model for the surface. This model can, for example, be stored in the augmented reality display device, transmitted to it by the means of transport, or made available for download from a server. The scanning process can also be carried out, for example, using a smartphone carried by the user.The scanning step can be performed each time the means of transport and / or the augmented reality display device is put into operation in order to be able to accommodate any changes in the function, geometry and color of the surface that may have been made in the meantime.

[0011] The optical scanning of the surface of the interior of the vehicle can be carried out, in particular, by means of an optical sensor in the display device. In other words, the augmented reality display device comprises an optical sensor arranged in a housing of the AR display device. In this way, a fixed relationship is always maintained between the orientation of the optical sensor and the display surface.

[0012] For example, when the AR display device is used for the first time in the vehicle, a handshake or calibration process can be initiated to familiarize the vehicle's surfaces or its interior and create a kind of "canvas" for the subsequent display of the adapted three-dimensional navigation map. The optical sensor can, for example, be a camera oriented in a direction similar to the user's line of sight in AR glasses. Thus, the optical sensor can always optically capture the area the user sees during use. This also allows objects in the user's field of vision to be recognized and / or body parts of the user to be identified and / or optically highlighted as input devices.

[0013] The AR display device can, in particular, comprise a transparent screen, which can be configured, for example, as a head-mounted monitor in the form of a pixel display. Alternatively or additionally, the display device can also comprise a smartphone, tablet, or similar device, whose camera provides data for displaying the real background on the screen, while the processor augments the camera image with additional display content or the adapted three-dimensional navigation map.

[0014] The three-dimensional navigation map can, for example, be loaded from a data storage device of the means of transport. In particular, the means of transport can be configured to support the navigation task using sensors of the means of transport (in particular, wheel speed sensors, GPS antenna(s), or similar). The required sections of the three-dimensional navigation map can be made available to the augmented reality display device by the means of transport. Alternatively or additionally, a navigation system located in the augmented reality display device can be used, which obtains map data from the means of transport or another data storage device and uses its own sensors to determine position and progress.This has the advantage that the navigation process can be continued even after exiting the vehicle without the need for wheel speed sensors or a projection onto a dashboard according to the invention.

[0015] In a first step, the AR display device can be linked to the means of transport via information technology. This can be done using a cable (especially a USB or monitor cable) and / or an antenna for transmitting wireless information signals (especially via Bluetooth, Wi-Fi, WLAN, or similar). After the information technology connection has been established, the means of transport can communicate with the AR display device and vice versa.

[0016] In particular, sensor information, map data and information about the respective operating state can be exchanged in order to be able to react to them if necessary. For example, a change in the optical appearance of the means of transport in the area of ​​the surface used to display the adapted three-dimensional navigation map can be determined through other circumstances. For example, a display element of the means of transport, in particular a windscreen root display or a display device located in the dashboard, can be put into operation or operated with a different intensity and / or with a different content. In order to prevent the modified optical appearance from compromising the optical display of the adapted three-dimensional navigation map orinterferes with this, the adapted three-dimensional navigation map and / or its representation can be adapted using the AR display device. For example, content can be moved to a different position on the surface of the vehicle interior. Alternatively or additionally, the intensity and / or color of a displayed element of the adapted three-dimensional navigation map can be changed in order to assert itself against the modified optical appearance of the vehicle or not to interfere with it. To determine the modified optical appearance, the vehicle, which naturally becomes aware of any changes to its signaling devices first, can send a wireless message to the AR display device.Alternatively or additionally, the AR display device can use its own optical sensor to determine the modified optical appearance of the vehicle in the area of ​​the surface and, independent of communication with the vehicle, make a suitable adjustment to the display of the adapted three-dimensional navigation map. This enables a particularly unproblematic coexistence of the vehicle's display device and the adapted three-dimensional navigation map.

[0017] A navigation task can be supported particularly intuitively by automatically determining an upcoming maneuver of the vehicle. Such events are typically supported in the prior art by detailed displays and / or the output of acoustic instructions to the driver. However, in accordance with the invention, in response to this (particularly additionally), the representation of the ego position of the vehicle is extracted from the adapted three-dimensional navigation map. For example, a cursor or indication that initially coincides with the representation of the ego position is optically lifted upwards and transferred from the adapted three-dimensional navigation map located in the vehicle into the vehicle's surroundings.In other words, the ego position or a pointer that initially correlates with it rises from the surface and locates itself, in a contact-analogous manner, with the position in the real vehicle environment at which the maneuver is to be carried out. This can mean, for example, that a representation of the ego position, initially rising vertically (Z-direction), from the dashboard of the vehicle is displayed using the AR display device as a cursor that overlaps the windshield and initially rises in a position that visually corresponds to an intersection at which the vehicle must turn. The representation or the element that rises from the adapted three-dimensional navigation map can change shape, color, and / or intensity to make it clear to the user that, strictly speaking, this is no longer the ego position, but rather a future target ego position.a directional arrow or similar is represented. The transition between the location of the representation of the ego position in the adapted three-dimensional navigation map and a virtual contact-analog movement element guides the user's gaze and attention particularly intuitively, making the navigation process more understandable than with the state of the art.

[0018] As soon as the vehicle changes direction, especially at an intersection, the display of the three-dimensional navigation map can be automatically adjusted accordingly, so that the viewer sees the navigation map rotating on its surface, particularly parallel to the external environment. This can depend in particular on the preselected "Map Shows Direction of Travel" setting. Optionally, this rotation can also be performed with a time offset so that the driver is not distracted by the rotation during a maneuver.

[0019] According to an additional aspect, the user can interact with the elements of the customized three-dimensional navigation map. For example, the user can tap on a building or its representation to display information related to the building using the augmented reality display device. For example, the information can be displayed on the surface of the 3D representation of the respective building and / or illustrated via a text field floating above the building or its representation.

[0020] If the building selected by the user is already in the user's field of vision, the information can also be alternatively or additionally assigned to the real building in the surrounding area using the AR display device. The building of interest to the user is thus selected within the adapted three-dimensional navigation map and is determined in particular by the optical sensor of the AR display device, while the information can be displayed optionally in the 3D representation of the building and / or in a contact-analog manner in the (real) surroundings of the vehicle.

[0021] According to a second aspect of the present invention, a user interface for displaying a virtual three-dimensional navigation map in a means of transport is proposed. The user interface comprises a data input, an evaluation unit, and a data output. The aforementioned elements can be configured, for example, as components of an electronic control unit provided exclusively for implementing the present invention in the means of transport or as hardware shared with other functions. The user interface is configured to optically scan a surface of an interior of the means of transport using the data input (and an optical sensor) or to obtain a data representation of the same.In other words, the interior surface is optically scanned / located / localized for position and shape, or the user interface is calibrated accordingly to load a three-dimensional navigation map and adapt it to the surface using data technology. Via the data output, the adapted three-dimensional navigation map, along with a representation of the ego position of the vehicle, can be displayed as a virtual, raised model on the surface using an AR display device. The AR display device can have a transparent screen, which is either part of the user interface or is linked to it via the data output.In other words, the user interface is designed to implement the features, combinations of features and the advantages resulting from them of the method according to the invention in such a way that, in order to avoid repetition, reference is made to the first-mentioned aspect of the invention.

[0022] According to a third aspect of the present invention, a means of transportation is proposed, which can be configured, for example, as a car, van, truck, motorcycle, aircraft, and / or watercraft. The means of transportation has a user interface according to the second aspect of the invention and is therefore capable of supporting a user's navigation intention with the same features, feature combinations, and advantages.

[0023] The invention is explained in more detail below using exemplary embodiments. They show: Figure 1 shows a perspective view of a passenger workstation of an exemplary embodiment of a means of transport designed according to the invention, in which a user carries out an exemplary embodiment of a method according to the invention using an exemplary embodiment of a user interface; Figure 2 shows a perspective view of a passenger workstation from the perspective of a passenger as an occupant in a first operating state; Figure 3 shows a perspective view of a passenger workstation from the perspective of a passenger as an occupant in a second operating state; Figure 4 shows a perspective view of a passenger workstation from the perspective of a passenger as an occupant in a third operating state; and Figure 5 shows a flowchart illustrating steps of an exemplary embodiment of a method according to the invention for displaying a virtual three-dimensional navigation map in a means of transport. Embodiments of the invention

[0024] Figure 1shows a driver's workstation of a means of transport 10 designed according to the invention in the form of a car, in whose passenger seat an occupant 4 is seated. The occupant 4 wears an AR display device 2, which has an optical sensor 3 in the form of a 2D camera, essentially aligned in the direction of view. The optical sensor 3 is connected for information technology purposes to the data input 8 of an electronic control unit 9. A data memory 5 located in the AR display device 2 is also connected for information technology purposes via the data input 8. Image material acquired by the electronic control unit 9 can be displayed on the AR display device 2 via a data output 11.By means of the optical sensor 3, three-dimensional navigation map material transmitted by a control unit 9 of the means of transport via an antenna 13 of the means of transport is sent to the AR display device 2 and virtually positioned on the dashboard 1 of the means of transport 10 in such a way that a representation of a 3D map model adapted to the curved surface of the dashboard 1 results in the form of an adapted three-dimensional navigation map 6. Within the adapted three-dimensional navigation map 6, a representation 7 of an ego position of the means of transport 10 is shown. In particular, a limb 12 of the occupant 4 in the form of a hand has been detected in the detection range of the optical sensor 3. The occupant 4 points to a representation of a building within the adapted three-dimensional navigation map 6.In response to a confirmation or the expiration of a predefined period of time, information relating to the building is displayed by means of the augmented reality display device 2 in a contact-analogous manner to the representation of the building (not shown).

[0025] Figure 2 shows the view of the occupant (reference number 4 in Figure 1 ) while approaching a T-junction. The ego position of the vehicle 10 is marked in the adapted three-dimensional navigation map 6 by a representation 7.

[0026] Figure 3 shows the Figure 2The situation shown follows a further approach of the means of transport 10 to the T-junction. As illustrated by an arrow P, a modified representation 7' ​​of the means of transport 10, as the future ego position of the means of transport 10, has merged into the (real) environment of the ego-means of transport 10. The modified representation 7' ​​now virtually "flies" ahead of the means of transport 10, changing its size and shape.

[0027] Figure 4 shows the Figure 3The situation shown is after the modified representation 7' ​​has "reached" a predefined turning position. At this position, a further modified representation 7" is displayed in the form of an arrowhead, which announces the left-turn maneuver to the driver of the vehicle 10 at the predefined position in the environment. In a manual driving mode or a manually guided vehicle 10, the user must now reduce speed and turn the steering wheel to the left. In an autonomously driving vehicle 10, the user is merely informed that the vehicle 10 will now decelerate and turn left. In the autonomously driving vehicle 10, this effectively prevents kinetosis, as the user can adjust to the expected forces.

[0028] Figure 5shows steps of an exemplary embodiment of a method according to the invention for displaying a virtual three-dimensional navigation map in a means of transport. In step 100, an augmented reality (AR) display device is connected to the means of transport via information technology. A wireless communication connection is established between the AR display device and the means of transport. In step 200, a surface of an interior of the means of transport is optically scanned using an optical sensor of the AR display device. The AR display device creates a surface model of the interior of the means of transport. In step 300, a three-dimensional navigation map is loaded from a data storage device. In step 400, the loaded three-dimensional navigation map is adapted to the surface of the means of transport according to the result of the scan.Here, a curvature of the three-dimensional navigation map is generated in order to achieve a full-surface virtual correspondence of the three-dimensional navigation map with the surface of the interior. In step 500, the adapted three-dimensional navigation map, along with a representation of an ego position of the means of transport, is displayed as a virtual, raised model on the surface of the means of transport using an augmented reality (AR) display device. This creates a large and perspectively realistic representation of the vehicle's surroundings and is used to illustrate the intended navigation. In step 600, a modification of a visual appearance of the means of transport in the area of ​​the surface is subsequently determined. For example, the means of transport can report that a display in a windscreen root display could require an overlay with content from the adapted three-dimensional navigation map.In step 700, the adapted three-dimensional navigation map is therefore shifted in order to be able to display the content shown in the disc root display without being overlaid by content from the AR display device. In step 800, an upcoming maneuver for the means of transport is automatically determined. To make it as easy as possible for the user to relate to the environmental position of the maneuver, in step 900 a representation of the ego position is first copied from the adapted three-dimensional navigation map and then extracted. In step 1000, the extracted representation of the ego position is superimposed on a detail of the surroundings of the means of transport in a contact-analog manner, so that the user can immediately recognize the extracted representation as the environmental position at which the upcoming maneuver is to be performed. List of reference symbols

[0029] 1Dashboard 2Augmented Reality (AR) display device 3Optical sensor 4Occupant 5Data storage 6Adapted three-dimensional navigation map 7Representation of the ego position 7`Modified representation of the ego position 7"Further modified representation of the ego position 8Data input 9Electronic control unit 10Means of transport 11Data output 12User's limb 13Antenna 100 to 1000Procedure steps

Claims

1. Method for displaying a virtual three-dimensional navigation map in a means of transport (10) comprising the steps of: • loading (300) the three-dimensional navigation map, • automatically adjusting (400) the three-dimensional navigation map to a surface (1) of an interior of the means of transport (10) and • displaying (500) the adjusted three-dimensional navigation map (6) along with a representation of an ego position of the means of transport (10) as a virtual, raised model on the surface (1) by means of an augmented reality display device (2).

2. Method according to claim 1, wherein the surface (1) comprises an instrument panel and / or a dashboard.

3. Method according to either claim 1 or claim 2, wherein the augmented reality display device (2) comprises a screen attached to the head of an occupant (4).

4. Method according to any of the preceding claims, wherein the three-dimensional navigation map is loaded from a data memory (5) of the means of transport (10).

5. Method according to any of the preceding claims, further comprising - connecting (100), in terms of information technology, the augmented reality display device (2) to the means of transport (10) by means of a cable and / or wirelessly.

6. Method according to any of the preceding claims, further comprising - determining (600) a modification of an optical appearance of the means of transport (10) in the region of the surface (1) and - adjusting (700) the adjusted three-dimensional navigation map (6) and / or the display thereof by means of the augmented reality display device (2).

7. Method according to any of the preceding claims, further comprising the following steps: - automatically determining (800) an impending maneuver of the means of transport (10) and in response thereto - automatically extracting (900) the representation (7') of the ego position from the adjusted three-dimensional navigation map (6) and - superimposing (1000) the representation (7) of the ego position on a detail of an environment of the means of transport (10) by means of the augmented reality display device (2).

8. Method according to any of the preceding claims, further comprising the following steps: - optically scanning (200), using a sensor, the surface (1) of an interior of the means of transport (10), in particular by means of an optical sensor (3) of the display device (2), and / or - loading a data-technical representation of the surface (1).

9. User interface for displaying a virtual three-dimensional navigation map in a means of transport (10), comprising: • a data input (8), • an evaluation unit (9) and • a data output (11), wherein the user interface is configured, • by means of the data input (8), ∘ to determine a representation of a surface (1) of an interior of the means of transport (10) and ∘ to load the three-dimensional navigation map, and is further configured, • to adjust the three-dimensional navigation map to the surface (1) and • by means of the data output (11), to display the adjusted three-dimensional navigation map (6) along with a representation (7) of an ego position of the means of transport (10) as a virtual, raised model on the surface (1) by means of an augmented reality display device (2).

10. Means of transport comprising a user interface according to claim 9.