Object association for contact-analog display on an HMD

Abstract

Method for contact-analogous display of a hint (11, 12) for an object to be marked in the vicinity of a vehicle (1), comprising: Providing information by the vehicle (1) about the object to be marked in the vicinity of the vehicle (1) in a contact-analogous manner, in particular the position of the object to be marked; Recognition of objects in the recordings of a camera (5) of a mobile device (2), in particular smart glasses; Identifying an object in the recordings of the camera (5) of the mobile device (2) as the object to be marked; Contact-analogous display of the indicator (11, 12) for the identified object on a display (4) of the mobile device (2).

Description

[0001] The invention relates to a method for the contact-analog display of an indication for an object to be marked in the vicinity of a vehicle and to a system for the same purpose.

[0002] Today, smart glasses, sometimes also called head-mounted displays (HMDs), are well known; they can be used to display information to the wearer. The smart glasses are worn on the head like regular glasses used as a visual aid. Today, smart glasses are in the research stage, where the image to be displayed is projected directly into the eye (onto the retina), for example using weak lasers. Typically, however, HMDs include a display that is positioned close to the user's eye(s) when the smart glasses are worn. The display can comprise two sub-displays, one for each eye. The display can show the user information and instructions in the form of text, graphics, or a mixture of these.The display can, in particular, be partially transparent, i.e., designed so that the wearer can also perceive the surroundings behind the display. Particularly preferably, the information is presented to the wearer in a contact-analog manner, which is sometimes also referred to as augmented reality. The wearer of the data glasses is presented with the information at a location oriented to the location of an object in the environment or an area within the environment. The information can consist of marking an object or an area of ​​the environment using a geometric shape, which, for example, is displayed adjacent to the object or superimposed on it. The geometric shape can be adapted (with correct perspective) to the contour of the object to be marked.Furthermore, the geometric shape can be a so-called bounding box, which is arranged as a rectangle (or other simple geometric shape) in such a way that it encloses an object or an area in the surrounding area (from the perspective of the wearer of the data glasses). Furthermore, the marking information can be in the form of text or pictograms that are displayed nearby or overlaid on the object. Typically, the display location of contact-analog information is tracked to compensate for movements of the data glasses and the object. Data glasses can also be used in vehicles to display contact-analog information for objects in the surrounding area of ​​the vehicle. Objects here include items such as traffic signs or guard rails, but also other road users such as vehicles and pedestrians. The term “own vehicle” refers to the vehicle in which the wearer is located.

[0003] To implement the contact analogy, the position of the object in the environment (of the wearer and thus often of the vehicle) and the pose of the smart glasses in relation to the object are typically taken into account, i.e. the position of the smart glasses and the orientation of the smart glasses. For use in vehicles, the pose of the smart glasses is typically determined using a smart glasses tracking system. A smart glasses tracking system can, for example, comprise a camera installed in the vehicle that records the driver's head together with the smart glasses. These recordings are then used to determine the pose of the smart glasses in the vehicle. The position and often also the dimension of the object to be marked using contact analogy is typically determined using sensors installed in the vehicle, such as radar, lidar and / or a camera system.

[0004] Implementing a contact-analog display requires high accuracy in both the smart glasses tracking system and the determination of the position of the object to be marked. For example, a total error of just 1° in determining the pose of the smart glasses and the position of the object to be marked results in an error of 1.75 meters at a distance of 100 meters. However, for the contact-analog marking of a vehicle ahead on the highway (e.g., a target vehicle of an automatic cruise control system), this already represents an offset of just one car width.

[0005] The main disadvantage of current systems lies in the underlying processing chain, i.e., the need to convert between the various reference systems (environmental sensors of the vehicle's own system, smart glasses tracking, smart glasses), resulting in the accumulation of individual errors while simultaneously placing high demands on overall quality. This is independent of whether the final calculation is performed in the reference system of the vehicle, the smart glasses tracking system, or the smart glasses. For this reason, current smart glasses tracking systems and currently deployed vehicle sensors rarely allow for an overall quality of less than 1°. Even in the medium term, this level of accuracy is not expected to be achievable with current systems.

[0006] Data glasses have become known which also each include a camera that takes pictures in the direction the wearer of the data glasses is looking, see for example WO 2013 / 012 914 A2.

[0007] The documents US 2012 / 0 194 418 A1, DE 10 2008 042 397 A1 and EP 1 510 849 A1 disclose relevant prior art.

[0008] The object underlying the invention is to provide an alternative method that enables the most accurate contact-analogous display of references to objects.

[0009] The problem is solved by the subject matter of the independent claims. Advantageous further developments are defined in the dependent claims.

[0010] In one aspect, a method for the contact-analog display of an indication for an object to be marked in the surroundings of a vehicle comprises: providing information by the vehicle about the object to be marked in the surroundings of the vehicle, in particular the position of the object to be marked; detecting objects in the images of a camera of a mobile terminal, in particular data glasses; identifying an object in the images of the camera of the mobile terminal as the object to be marked; and contact-analog display of the indication for the identified object on a display of the mobile terminal. The mobile terminal comprises a camera and a display, wherein the relative orientation and relative position of the camera and the display to one another are known and / or unchangeable.Typically, the method further comprises: providing information about other objects in the surroundings of the vehicle; assigning objects detected in the images of the camera of the mobile device to the other objects provided.

[0011] According to the method proposed here, the contact-analog display is thus realized based on object detection in the images captured by the mobile device's camera. A comparison, which can also be understood as an association, takes place between the objects detected by the vehicle's sensors and the objects detected in the images captured by the mobile device's camera. While determining the pose of the data glasses, as provided for in prior art methods, is additionally possible, it is not necessary to realize the contact analogy. In some implementations, a relatively inaccurate pose determination is performed.

[0012] The method is not only suitable for smart glasses, but also applicable to smartphones and tablets. Smartphones and tablets include a touch-sensitive display and a camera. A contact-analog representation is realized on the display by showing a video stream from the camera, in which the object to be marked is provided with the information in a contact-analog manner. Mobile devices are defined here as movable and portable devices with a display and camera, including smart glasses, smartphones, and tablets.

[0013] The method offers significant advantages in the accuracy of the contact-analog display: In state-of-the-art methods, errors of the subsystems accumulate. For example, augmented reality displays place high demands on the accuracy of the vehicle sensors and the smart glasses tracking systems for determining the pose of the smart glasses. The proposed method circumvents this chain of errors by using the actual target information (the object to be marked in the real driving scene) to determine the display location on the mobile device's display. This exploits the fact that the relative position and orientation of the mobile device's camera to the mobile device's display can be determined with high accuracy (for example, in a configuration process after the mobile device has been manufactured).

[0014] In the example processing chain, the vehicle and its sensors and evaluation units detect objects in the surrounding area and determine the object to be marked. The mobile device also detects objects in the images captured by its camera and sends a list of detected objects to the vehicle. The vehicle detects the object to be marked in this list (based on an object association) and sends information about it back to the mobile device. The object association can generally also be carried out by the mobile device, although in this case the vehicle sends a list of objects to the device. Taking into account the relative orientation and position of the camera and the display of the mobile device, the mobile device calculates the display location for the contact-analog information and displays the information on the display.The object to be marked can then be tracked by the mobile device and the contact-analog display can be adjusted accordingly (tracking, especially camera-based tracking).

[0015] Especially when tracking an object, this results in advantages in terms of a delay in tracking, as well as less shaking during high-frequency movements of the mobile device.

[0016] This association allows for the optimal combination of the advantages of both sensor technologies. The on-board sensor technology is used to determine dynamic variables (speed, ACC target, etc.) of the relevant objects. The camera's optics, with their static and precisely known reference to the display on the mobile device, are used for visual marking (contact-analog marking).

[0017] In a further development, the method further comprises: determining the orientation of the mobile device relative to the vehicle based on the image of the interior of the vehicle in the camera images; wherein the determined orientation of the mobile device is taken into account when identifying the object. The identification of the object does not take place solely on the basis of the determined orientation. The determined orientation serves to provide an improved starting point when identifying the object to be marked. Instead of having to assign objects in an area of ​​180° in front of the vehicle to objects detected in the images, for example, only objects in the range of 20° to 110°, for example, can be taken into account, provided that it is determined that the mobile device, due to its orientation, only provides images from this area and / or can only display information in this area.This results in shorter latencies and reduced computing requirements. To determine orientation, the vehicle's interior can be detected in the images and compared with pre-stored data about the interior. From this, the rough orientation of the smart glasses can be determined – with a limited amount of computing power.

[0018] In a preferred implementation, displaying the indicator comprises determining the shape and position of the indicator based on the image of the identified object in the camera recordings. The proposed method offers a further advantage: In prior art systems, the size and shape of a contact-analog marker are determined based on the perspective of the vehicle's sensors. However, this differs from the driver's perspective. For example, when cornering sharply, the vehicle only sees the rear of the vehicle in front thanks to a radar sensor in the bumper, while the driver can also see the side of the vehicle in front. Calculating the contact analogy and the size and shape of the marker based on essentially the same driver perspective (camera and display are included in the mobile device) thus enables improved contact-analog markers.

[0019] In an advantageous implementation, the different geometric arrangement of the camera in relation to the display of the mobile device is taken into account when displaying the shape and position of the notice, whereby the geometric arrangement of the camera and the display differ in particular in position and orientation.

[0020] The information about the object to be marked in a contact-analog manner can be provided using sensors installed in the vehicle, based on vehicle-to-vehicle communication and / or from a remote computer.

[0021] In another aspect, a system for the contact-analog display of an indication for an object to be marked in the surroundings of a vehicle comprises: a mobile terminal, in particular data glasses, comprising a display and a camera; an electronic control device; wherein the system is configured to carry out one of the methods described above. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 schematically shows an apparatus for carrying out an exemplary method according to the invention. Fig. 2 schematically shows the fields of view of the camera of data glasses and the surroundings detection of a vehicle according to an embodiment. Fig. 3 shows schematically the arrangement of camera and display of data glasses according to an embodiment. Fig. 4 and Fig. 5 schematically show contact-analog markings of objects according to embodiments.

[0022] Like reference numerals refer to corresponding elements throughout the figures. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] Fig. 1 schematically shows an apparatus for carrying out an exemplary method according to the invention. Individual blocks of the Fig. 1 represent functions that can be implemented using hardware / software or a combination thereof. Vehicle 1 includes radar, lidar, and camera sensors, as well as other sensors if necessary. The measurement data from these sensors is merged to create an environment model. The environment model specifies the position (relative to the vehicle) outside vehicle 1 of objects, as well as their classification (e.g., pedestrian, cyclist, vehicle, etc.). In this example, the environment model is provided as an object list that includes the aforementioned data. In other words: Based on the available vehicle-local sensors, vehicle-to-vehicle communication, and / or a central backend, the vehicle merges this data into a vehicle-local environment model, including other road users such as vehicles ahead, so that this object data is available in the vehicle-local coordinate system.

[0024] The data glasses 2 also include a camera, which is statically mounted on the data glasses, as well as a semi-transparent display with two partial displays for each eye. A model of the cockpit, i.e., the interior and, in particular, the passenger compartment around the driver, is stored in the camera. The model can also reflect only certain features of the interior, such as the window frame or details of the dashboard. Using the images from the data glasses' camera, which depict parts of the vehicle's interior, and the cockpit model, the rough orientation of the camera can be determined. For this purpose, so-called SLAM algorithms (Simultaneous Localization and Mapping) can be used, for example.Methods for determining the pose of the smart glasses based on images of the interior are described in the pending application DE (applicant's reference EM27047) filed by the same applicant; the pose of the smart glasses includes the pose and orientation (also referred to herein as orientation). Determining the rough orientation of the smart glasses requires comparatively little computational effort compared to determining a precise orientation of the smart glasses.

[0025] The cockpit model also serves to distinguish between the interior and exterior of the camera images. The objects to be marked here are located outside the vehicle. Accordingly, the object recognition of the smart glasses can be limited to the areas of the camera images that depict the exterior. The object recognition of the smart glasses also generates an object list in which the position and classification of the detected objects are noted. Object recognition of the smart glasses typically occurs in the coordinate system of the smart glasses.

[0026] The object list from the smart glasses is transmitted to Vehicle 1, for example, using the Wi-Fi or Bluetooth standard. Object association then takes place in Vehicle 1. The coarse orientation of the smart glasses in the vehicle coordinate system, along with the detected objects in the vehicle coordinate system, is used to compare the coarse field of view of the smart glasses' camera with the vehicle coordinate system and to limit the number of objects to be associated. Fig. Figure 2 schematically shows the principle of pre-selection of the objects to be associated. The driver 7 of a vehicle 6 wears data glasses 2, whose camera captures images of objects in area 8. The vehicle's sensors enable environment detection in area 9. Thus, the object list of the vehicle environment model also includes vehicle 10. However, this cannot be recognized by the data glasses 2 because the field of view 8 of the camera of the data glasses 2 does not image the vehicle 10. Consequently, the vehicle 10 does not need to be considered in the object association.

[0027] Based on the pre-selection of objects detected by the vehicle, the objects detected by the data glasses are associated with the objects detected by the vehicle. An attempt is made to assign each object detected by the data glasses to an object detected by the vehicle in its surroundings. Algorithms known from sensor data fusion for vehicle environment detection are used here. These include approaches that optimize the global assignment of objects from different sources to one another with regard to a quality measure (so-called global nearest neighbor approaches, such as auction algorithms, one of which is described, for example, in "A distributed asynchronous relaxation algorithm for the assignment problem", Bertsekas, DP, Proceedings of the 24th IEEE Conference on Decision and Control Vol. 24, 1985, 1703-1704).

[0028] Vehicle 1 selects one or more of the objects detected in the environment model as the target object for which the contact-analog indication is to be displayed (multiple objects to be marked together are understood here as a single target object). The target object can result from the function of a driver assistance system. For example, the vehicle that was detected as the vehicle in front for an ACC function can be selected as the target object. An attempt is made to identify the target object among the objects detected by the data glasses. In doing so, the previously performed object association is used. The identified object is transmitted back to the data glasses 1, for example, using the Wi-Fi or Bluetooth standard.

[0029] Based on the identified object, the smart glasses 1 calculate the shape of the contact-analog indicator and its display location on the smart glasses display. In some implementations, a camera recording or camera frame is used for this purpose. For example, if the contours of a vehicle are to be marked, the contours of the vehicle must be recognized and marked with the indicator. Known state-of-the-art contouring techniques can be used for this purpose. In this way, the side of a vehicle to be marked that is traveling around a curve and is visible to the driver is correctly outlined.

[0030] The displacement and rotation of the camera and the display of the data glasses are also taken into account when calculating (or determining) the contact-analog information. Fig. Figure 3 shows this rotation and shift schematically and exaggerated for clarity. The data glasses 2 comprise the camera 5 and the display 4. The perspective of the user's eye 3 (represented by a viewing triangle) is shifted and rotated relative to the field of view of the camera 5. The translation and rotation must be taken into account when calculating the shape of the contact-analog cue. For this purpose, the translation and rotation can be determined once in advance. A cue shape determined for the camera recordings can then be converted for the display, e.g., using a homogeneous transformation.

[0031] The contact-analog cue thus determined is finally shown on the display of the data glasses.

[0032] Fig. 4 and Fig. 5 schematically show contact-analog markings of objects using the method according to the invention according to embodiments. Fig. Figure 4 shows the contact-analog marking of a navigation instruction 11 on the roadway. The instruction 11 is displayed statically, meaning it does not move relative to the vehicle's surroundings. Fig. Figure 5 shows the contact-analog marking 12 of an ACC target vehicle. The marking consists of two parts as shown in Fig. 5 shown.

Claims

[1] Method for the contact-analog display of an indication (11, 12) for an object to be marked in the environment of a vehicle (1), comprising: Providing information by the vehicle (1) about the object to be marked in a contact-analog manner in the surroundings of the vehicle (1), in particular the position of the object to be marked; Detecting objects in the images of a camera (5) of a mobile terminal (2), in particular data glasses; Identifying an object in the images of the camera (5) of the mobile terminal (2) as the object to be marked; Contact-analogous display of the indication (11, 12) for the identified object on a display (4) of the mobile terminal (2). [2] The method of claim 1, further comprising: Providing information about other objects in the vicinity of the vehicle (1); Assigning objects recognized in the images of the camera (5) of the mobile terminal (2) to the other objects provided. [3] Method according to one of the preceding claims, further comprising: Determining the orientation of the mobile terminal (2) relative to the vehicle (1) based on the image of the interior of the vehicle (1) in the images of the camera (5); When identifying the object, the specific orientation of the mobile device (2) is taken into account. [4] Method according to one of the preceding claims, wherein displaying the indication (11, 12) comprises: Determining the shape and position of the indication (11, 12) based on the image of the identified object in the camera images (5). [5] Method according to claim 4, wherein when displaying the shape and position of the indication (11, 12), the different geometric arrangement of the camera (5) in relation to the display of the mobile terminal (2) is taken into account, wherein the geometric arrangement of the camera (5) and the display differ in particular in position and orientation. [6] Method according to one of the preceding claims, wherein the information on the object to be marked in a contact-analog manner is provided by means of sensors installed in the vehicle (1), based on vehicle-to-vehicle communication and / or by a remote computer. [7] System for the contact-analog display of an indication (11, 12) for an object to be marked in the environment of a vehicle (1), comprising: A mobile terminal (2), in particular data glasses, comprising a display (4) and a camera (5); An electronic control device; The system is configured to carry out a method according to one of the preceding claims.

Images