Method for operating a field of vision display device and vehicle

Abstract

The invention relates to a method for operating a field of view display device in a vehicle, wherein an in-vehicle computing unit for generating an augmented reality representation (AR) on a projection surface (1) of the field of view display device for a driver (2) effects a contact-analog representation of an object marking (3) with an object (4) detected by the vehicle based on an evaluation of environmental sensor data, and wherein a positional difference (ΔL) perceptible by the driver (2) between the object marking (3) and the object (4) within the projection surface (1) is determined. The method according to the invention is characterized in that the computing unit, for determining the positional difference (ΔL), takes into account the reflection (3*) of the object marking (3) from the projection surface (1) perceptible in a corneal reflection image (CRI) of the driver (2).

Description

[0001] The invention relates to a method for operating a field of view display device in a vehicle according to the type defined in more detail in the preamble of claim 1 and to a vehicle for carrying out the method.

[0002] The integration of head-up displays (HUDs) into vehicles has been common practice for some time. While initially, information such as navigation instructions or the current speed were simply projected into specific areas of a projection surface, for example, integrated into the windshield of a car, today it is increasingly the case that an augmented reality representation of the surroundings is displayed. This type of representation is also referred to as augmented reality (AR). The vehicle can thus be capable of detecting and, if necessary, classifying surrounding objects with the help of environmental sensors such as cameras, ultrasonic sensors, radar systems, and / or LiDARE. This can be used to identify a vehicle ahead or a lane marking.In particular, when using various driver assistance systems, such as an adaptive cruise control or a lane keeping assistant, the leading vehicle and / or said lane marking can be prominently displayed in the projection area of the head-up display device. For this purpose, the corresponding environmental object is marked analogously to a contact with a virtual object marking superimposed on the projection area.

[0003] In order to mark a corresponding environmental object for the driver of the vehicle or the user of the field of view display device to provide an augmented reality representation, it is necessary to display the object marking in a resulting target area of ​​the projection surface, taking into account the position of the driver's eyes within the vehicle interior and their line of sight. This position can be estimated based on the known installation position of the environmental sensors used to detect environmental objects and the projection surface of the field of view display device. The driver's line of sight can be captured using an interior camera, for example, integrated into the vehicle's instrument cluster, which makes it possible to correctly position and track the object marking on the projection surface.

[0004] Nevertheless, despite appropriate tracking of the display position of the object marker in the projection surface, positional deviations may occur between the surrounding object and the object marker from the driver's perspective. This is particularly the case when the vehicle is subjected to significant acceleration or deceleration. In these cases, the vehicle performs comparatively strong pitching movements around the vehicle's transverse axis, causing the object marker to assume a vertical offset from the surrounding object to be marked in the projection surface.

[0005] Various approaches to solving this problem are already known. With the help of vehicle-integrated acceleration sensors or so-called "vehicle level sensors", pitching movements of the vehicle can be detected and this information can be used to correct the display position of the object marking. However, such systems are expensive and represent an additional source of error. Another approach is to detect the pitching movements of the vehicle using computer vision and compensate for them accordingly. However, this approach requires the presence of structural features in the environment that can be detected with the help of camera systems. If such structures are missing, for example because the road is covered in snow over a large area, corresponding pitching movements cannot be detected using machine vision. Such a system is therefore unreliable.

[0006] A device and a method for correcting the position of markings in the projection surface of a field-of-view display device are also known from DE 10 2016 223 179 A1. This document discloses the use of vibration sensors, acceleration sensors, and / or gyro sensors to detect changes in the vehicle's posture, which accordingly contribute to a positional deviation between the object marker and the object to be marked. A disadvantage of this is that the positional deviation is indirectly derived from the change in the vehicle's posture, which can lead to errors. Thus, the actual positional deviation can be greater or smaller than the positional deviation indirectly resulting from the corresponding change in the vehicle's posture.

[0007] The present invention is based on the object of providing an improved method for operating a field of view display device in a vehicle, with the aid of which it is possible to more reliably determine the height of corresponding positional deviations between an object marking and an object to be marked in the projection surface of the field of view display device.

[0008] According to the invention, this object is achieved by a method for operating a field of view display device having the features of claim 1. Advantageous embodiments and further developments as well as a vehicle for carrying out the method emerge from the dependent claims.

[0009] A generic method for operating a field of view display device in a vehicle, wherein an in-vehicle computing unit for generating an augmented reality display on a projection surface of the field of view display device for a driver causes a contact-analogous display of an object marking with an object detected by the vehicle based on an evaluation of environmental sensor data, and wherein a positional difference perceptible by the driver between the object marking and the object within the projection surface is determined, is further developed according to the invention in that the computing unit for determining the positional difference takes into account the reflection of the object marking from the projection surface perceptible in a corneal reflection image of the driver.With the aid of the method according to the invention, it is possible to detect the positional difference between the position of the object marker and the object to be marked in the augmented reality representation generated in or on the projection surface in a particularly reliable manner, since this positional difference is measured directly and not indirectly derived from other variables. For this purpose, the applicant has recognized that the driver's eyes act like a mirror, which enables the computing unit to visually detect the eyes or the object.

[0010] Recording the corresponding corneal reflection image allows the augmented reality representation generated on the projection surface of the field of view display device to be captured.

[0011] The computing unit is capable of detecting at least the reflection of the object marking in this reflected representation of the display content of the projection surface and, taking into account known geometric relationships or installation positions of the corresponding device components, deriving the positional deviation from the object to be marked. If the object to be marked can itself be detected in the corneal reflection image, the computing unit can even directly measure the positional difference between the object marking and the object to be marked in the corneal reflection image. However, since, based on currently available technology, detecting the object in the corneal reflection image may only be possible under certain conditions or may not even be possible at all, the position of the object in the corneal reflection image can also be estimated or validated by the computing unit, which will be discussed in more detail below.However, since at least the reflection of the object marking is visible in the corneal reflection image, a more accurate and reliable determination of the position difference is possible compared to the state of the art.

[0012] As already described, the driver's eyes serve as a mirror for recognizing the display content of the augmented reality display. Since the driver's eyes are directly observed, sources of error that would result from an indirect estimation cannot affect the desired result. This improves the reliability of determining the position difference.

[0013] According to an advantageous embodiment of the method according to the invention, it comprises the following method steps: - Recording the corneal reflection image with an interior camera that captures the driver's eye area; - Determination of the driver's line of sight in the corneal reflection image by the computing unit; - Capturing the surroundings of the vehicle with an environment camera, with the object in the field of view of the environment camera; - Detection of the object in a camera image recorded by the surrounding camera by the computing unit, whereby the computing unit determines a position and dimensions of the object in the camera image; - generating a reference graphic shape element by the computing unit based on the position and dimensions of the object, so that the reference graphic shape element assumes a specified pose relative to the object in the camera image of the environment camera; - mapping the position of the reference graphic form element from the camera image of the surrounding camera into the corneal reflection image, taking into account the viewing direction, by the computing unit; - Determining the position of the reflection of the object marker in the corneal reflection image; and - Determining a difference between the position of the reference graphic feature and the position of the reflection of the object marker in the corneal reflection image and setting the difference as a position difference.

[0014] By implementing the method steps according to the invention, it is possible to determine the position difference by the computing unit even if the object is not or only insufficiently recognizable in the reflection in the corneal reflection image. This contributes to a further increase in the reliability of the method according to the invention.

[0015] Modern vehicles usually already have an interior camera for recording the driver. This makes it possible, for example, to provide fatigue detection or to record the driver's attention to the driving situation. By using the interior camera to capture the image on the projection surface of the field of view display device, it is possible to dispense with the provision of additional cameras directly aimed at the projection surface. Such a solution would also require estimating the spatial relationship of the driver's eyes to the image on the projection surface, which represents another source of error.

[0016] In general, the object to be marked in the augmented reality display, such as a vehicle ahead, a lane marking, or the like, can be detected using any environmental sensors, in particular a LiDAR. However, with the help of cameras, it is possible to capture camera images, which, since the corneal reflection image serves as a reference, allows for an even simpler transformation of the corresponding position information. Since the representation of the display content of the projection surface in the corneal reflection image is mirrored compared to the camera image captured by the environmental camera, the corneal reflection image or the camera image can be mirrored accordingly by the computing unit to ensure consistent orientation.

[0017] The object can be recognized in the respective camera image using proven machine vision algorithms. Taking into account the object's position in the camera image, its position can be determined. Depending on how many pixels of the camera image the object occupies, the processing unit can determine the corresponding dimensions of the object.

[0018] The driver's gaze direction can also be determined in a proven way. For example, the eye translation and orientation can be determined in the corneal reflection image or in multiple corneal reflection images taken in succession, for example, using so-called Purkinje reflections.

[0019] The reference graphic form element generated by the computing unit does not necessarily have to be displayed on a display device, but rather serves the computing unit merely to virtually or mathematically determine the position difference. Based on the known installation position of the surround camera on the vehicle, the interior camera, and the viewing direction, the position of the reference graphic form element can then be transformed from the camera image of the surround camera into the corneal reflection image. This then allows the computing unit to determine the difference between the position of the reflection of the object marking and the position of the reference graphic form element, and thus ultimately the position difference.

[0020] According to a further advantageous embodiment of the method according to the invention, the computing unit adapts the projection of the object marking onto or into the projection surface taking into account the positional difference, such that the positional difference is minimized. Thus, determining the positional difference can be used to shift over time the position at which the object marking is projected onto the projection surface in order to compensate for or correct the positional difference. Thus, according to the invention, positional differences resulting from pitching movements performed by the vehicle can be compensated for. With the aid of the method according to the invention, it is also possible to compensate for other positional differences due to other causes of origin, for example horizontal positional differences and / or oblique positional differences. Such a positional difference can include, in addition to a distance difference between the object to be marked and the object marking, angular errors.This means that the object marking is rotated relative to the object to be marked in the plane of the projection surface. Accordingly, appropriate shapes are used for the reference graphic shape element to compensate for such angular errors.

[0021] The method according to the invention thus improves the generation of augmented reality representations. This increases driving comfort. Hardware components already installed in the vehicle can be used to implement the method. Additional pattern projection into the environment, which can improve the detection of pitching movements using machine vision methods, can be dispensed with. In particular, the method according to the invention allows for correction of the position difference independent of the existing scenery. The compensation of the position difference can be repeated cyclically in the context of a control loop so that the position difference is ultimately compensated to zero, if possible.The target position determined by the computing unit to compensate for the position difference for the projection of the object marker can also serve, for a certain period of time, as a primary source for determining the way in which the object marker is projected onto the projection surface.

[0022] A further advantageous embodiment of the method according to the invention further provides that the computing unit transforms the position of content in the corneal reflection image and the camera image into a common reference coordinate system. This facilitates the determination of the difference between the position of the reference graphic shape element and the position of the reflection of the object marker in the corneal reflection image, allowing for faster calculation or execution of the corresponding method steps.

[0023] According to a further advantageous embodiment of the method according to the invention, the corneal reflection image is rectified after recording and before further processing to compensate for a spherical projection. Human eyes are approximately spherical. This results in the corneal reflection image containing a correspondingly curved representation of the surroundings. Therefore, the corneal reflection image is advantageously rectified to compensate for said curvature. This can further facilitate the recognition of content in the corneal reflection image. Proven image processing algorithms can be used for this purpose. The corneal reflection image can also be mirrored, particularly horizontally.

[0024] A further advantageous embodiment of the method according to the invention further provides that the reference graphic shape element has an L-shape. In this context, the reference graphic shape element can also be referred to as a so-called "L-shape". To determine the size and position of the reference graphic shape element in the camera image of the environmental camera, a so-called "bounding box" can be placed around the object to be recognized. The center point of the vertical leg of the L-shape can then run through the center point of the bounding box. The horizontal leg of the L-shape can then run through the lower right corner point of the bounding box. This enables reliable placement of the reference graphic shape element in the camera image.

[0025] After transforming or mapping the reference graphic shape element into the corneal reflection image, the L-shape can be additionally mirrored horizontally to create a so-called T-shape. The underlying "T" is upside down. The horizontal leg of the T serves as a reference marker for the processing unit to determine the distance to the reflection of the object marker detected in the corneal reflection image. This ensures that the processing unit can reliably determine the distance between the reflection of the object marker and the reference graphic shape element.

[0026] According to a further advantageous embodiment of the method according to the invention, it is further provided that the computing unit determines the pose of the reference graphic form element relative to the object in the camera image depending on an object marking type. In general, various object marking types can be projected by means of the field of view display device to generate the augmented reality representation. For example, vehicles driving ahead can be highlighted by a horizontal bar displayed below the vehicle from the driver's perspective, lane markings by a luminous line running along the lane marking, traffic signs by a box framing the respective traffic sign, and the like. Depending on the object to be marked and the corresponding object marking orDepending on the object marking type, the object marking will also assume a different position, i.e., distance and orientation relative to the object to be marked. This may make it necessary to select different poses for the reference graphic shape element depending on the object to be marked or the object marking type. This is possible according to the invention, which allows for reliable determination and, in particular, correction of position differences in all conceivable application scenarios in the context of generating augmented reality representations.

[0027] A further advantageous embodiment of the method according to the invention further provides that the computing unit stores values ​​for correcting the positional difference in the form of a calibration for the field of view display device. This allows for proactive and timely correction of corresponding positional differences. Furthermore, corresponding calibrations can be exchanged across vehicles, or different calibration values ​​can be used for different vehicle users. This allows for adaptive adaptation of the augmented reality display to individual users.

[0028] Preferably, the computing unit stores the values ​​used for calibration specifically depending on an object marking type, in particular taking into account a target shape and / or target position on the projection surface intended for the respective object marking type. This ensures that, using said calibration, corresponding position differences can be reliably compensated for in all conceivable application scenarios of the augmented reality display.

[0029] In a vehicle comprising a field of view display device configured to output an augmented reality representation, an interior camera directed toward the driver's eye area, an environment camera capturing at least a portion of the surroundings, and a computing unit, the field of view display device, the interior camera, the environment camera, and the computing unit are configured to execute a method described above. The vehicle can be any road vehicle, such as a car, truck, van, bus, or the like.

[0030] The computing unit has at least read access to a computer-readable storage medium, storing machine-interpretable instructions which, when executed by a processor of the computing unit, cause the processor to carry out the method according to the invention described above.

[0031] Since, as already mentioned, modern vehicles usually have said hardware components anyway, the method according to the invention can be easily and cost-effectively integrated into such vehicles, even retrospectively.

[0032] Further advantageous embodiments of the method according to the invention for operating field of view display devices and of the vehicle according to the invention also emerge from the exemplary embodiments which are described in more detail below with reference to the figures.

[0033] Showing: Fig. 1 is a schematic representation of the display content of a front-view display device in a vehicle, showing an augmented reality display; Fig. 2 the schematic representation from Fig. 1, where there is a positional difference perceptible to a driver between an object to be marked and a corresponding object marking; Fig. 3 a schematic representation of the principle underlying an operating method according to the invention for field of view display devices; Fig. 4 a flowchart of the method according to the invention; and Fig. 5 a schematic detailed representation of the individual process steps of the process according to the invention.

[0034] Fig. 1 shows a view of a person driving a vehicle through the windshield of a vehicle onto the roadway. The vehicle includes a field of vision display device, with corresponding display content being projected onto or into a projection surface 1 by a suitable projection device. The road course as well as surrounding objects such as delineators, traffic signs, a lane marking, and a vehicle 9 traveling ahead can be seen. Various display contents 10 are shown by means of the field of vision display device, such as references to said traffic signs, the current speed of the vehicle, and the like, as well as an object marker 3 for the vehicle 9 traveling ahead, which in this context will be referred to hereinafter simply as object 4.

[0035] The object marker 3 serves to visually highlight the object 4 for the person driving the vehicle, so that an augmented reality representation AR can be generated by means of the field of view display device.

[0036] In order to ensure a contact-analogous representation of the object marking 3 to the object 4, it is necessary that a computing unit (not shown in detail) used to control the field of view display device takes into account the corresponding installation position and orientation of the environmental sensors used to detect surrounding objects as well as the projection surface 1 with respect to the person driving the vehicle. Fig. 5c) the direction of gaze 6 of a driver 2 and the location of the eyes 11 must be taken into account.

[0037] Due to various causes of errors, it can lead to Fig. 2 between the position at which the driver 2 perceives the object marking 3 on the projection surface 1 and the object 4. For example, the vehicle may perform pitching movements around the transverse axis due to strong braking or acceleration processes, so that the object marking 3 appears to move vertically upwards or downwards within the projection surface 1 for the driver 2.

[0038] With the aid of a method according to the invention for operating the field of view display device, these position differences ΔL can at least be detected or even preferably corrected. The underlying functional principle is explained using Fig. 3 illustrates.

[0039] The driving situation is shown in a schematic side view. The eye(s) 11 of the driver 2 can be seen, with the driver 2's view being directed through a windshield 12 into the surroundings, with the projection surface 1 being arranged inside or on the windshield 12. Also shown is the display content in the projection surface 1, i.e. the object marking 3, as well as the object 4 perceivable by the driver 2 when looking through the windshield 12. Due to the augmented reality representation AR, the object marking 3 appears to be located in the longitudinal direction of the vehicle between the windshield 12 and the object 4. The driver 2 is recorded using an interior camera 5, which records at least the eye area of ​​the driver 2. This makes it possible to capture a mirror image of reality on the eye(s) 11. This mirror image is also referred to as a corneal reflection image CRI, or.referred to as “corneal reflective image”.

[0040] With the help of the interior camera 5 it is possible to generate the said corneal reflection image CRI and to create a Fig. 5h) of the object marking 3. This in turn enables the computing unit to determine the position difference ΔL directly, instead of deriving it indirectly from sensor-detected variables. In particular, it may also be possible to detect a reflection of the object 4 in the corneal reflection image CRI. However, it is also possible to map the position of the object 4 into the corneal reflection image CRI, which will be discussed in more detail below, should a direct measurement of the position difference ΔL by the computing unit not be possible when considering the corneal reflection image CRI alone.

[0041] The process sequence according to the invention is Fig. 4. The individual process steps are described in detail in Fig. 5 shown.

[0042] In a step 401, see also Fig. 5a), an image is taken using the interior camera 5 to capture the driver 2 or the eye area.

[0043] In a step 402, see also Fig. 5b), the corneal reflection image CRI is derived from this image. Fig. In the embodiment shown in Figure 5b), a rectification is also performed to compensate for the spherical projection. The resulting corneal reflection image CRI shows a multi-story building as an example.

[0044] In a step 403, see Fig. 5c), the eye position is detected in the corneal reflection image CRI and the orientation of the eyes 11 or the gaze direction 6 is determined. By implementing proven methods for detecting the gaze direction 6, the head position can also be determined, which is indicated by a coordinate system drawn inside the head.

[0045] Subsequently, in step 404, see Fig. 5d), with a Fig. 5g) of the vehicle's surroundings. This creates a camera image 7. In Fig. 5d), camera image 7 shows an example of a driving situation on a highway. Vehicles ahead were detected and classified using machine vision methods, which is indicated by a corresponding bounding box.

[0046] In this case, an environment camera is used which has at least a detection area that matches the field of view of the driver 2, so that the object 4 can be found in the camera image 7 generated by the environment camera. Subsequently, in a step 405, see Fig. 5e), performs object detection for the camera image 7 in order to recognize corresponding objects 4. The computing unit determines the position of the object 4 in the camera image 7 as well as its dimensions.

[0047] The computing unit then generates in a step 406, see Fig. 5f), a so-called reference graphic form element 8, preferably in the form of an L. The reference graphic form element 8 is therefore also referred to as a so-called L-shape.

[0048] In a step 407, see Fig. 5g), the computing unit transforms or maps the position of the reference graphic form element 8 from the camera image 7 into the corneal reflection image CRI. This is done taking into account the corresponding installation positions and orientation of the interior camera 5, the projection surface 1, and the environmental sensor on the vehicle. Furthermore, the computing unit takes into account the viewing direction 6 determined in step 403.

[0049] In a step 408, see also Fig. 5h), the computing unit compares the respective position of the depicted reference graphic form element 8 and the reflection 3* of the object marking 3 in the corneal reflection image CRI. This enables the computing unit in a step 409, see also Fig. 5i), to determine a corresponding difference, which is then defined as the position difference ΔL.

[0050] Particularly preferably, the computing unit can in a further method step 410, see also Fig. 5j), compensate for this position difference ΔL, i.e. correct it.

[0051] As in Fig.4 indicated by an arrow 411, if the process steps are executed iteratively multiple times in the subsequent process sequence, the execution of process steps 404 to 407 can also be omitted. The recurring execution of all process steps is indicated by an arrow 412. For this purpose, a cycle time to be observed can be defined.

Claims

[1] Method for operating a field of view display device in a vehicle, wherein an in-vehicle computing unit for generating an augmented reality display (AR) on a projection surface (1) of the field of view display device for a driver (2) effects a contact-analogous display of an object marking (3) with an object (4) detected by the vehicle based on an evaluation of environmental sensor data, and wherein a position difference (ΔL) perceivable by the driver (2) between the object marking (3) and the object (4) within the projection surface (1) is determined, characterized by that the computing unit for determining the position difference (ΔL) takes into account the reflection (3*) of the object marking (3) from the projection surface (1) that can be seen in a corneal reflection image (CRI) of the driver (2). [2] Method according to claim 1, characterized by the following procedural steps: - Recording the corneal reflection image (CRI) with an interior camera (5) capturing the driver's eye area (2); - Determination of the viewing direction (6) of the driver (2) in the corneal reflection image (CRI) by the computing unit; - capturing the surroundings of the vehicle (2) with an environment camera, wherein the object (4) is in the field of view of the environment camera; - detecting the object (4) in a camera image (7) recorded by the surrounding camera by the computing unit, wherein the computing unit determines a position and dimensions of the object (4) in the camera image (7); - generation of a reference graphic form element (8) by the computing unit based on the position and dimensions of the object (4), so that the reference graphic form element (8) assumes a fixed pose relative to the object (4) in the camera image (7) of the surrounding camera; - mapping the position of the reference graphic form element (8) from the camera image (7) of the surrounding camera into the corneal reflection image (CRI) taking into account the viewing direction (6), by the computing unit; - Determining the position of the reflection (3*) of the object marker (3) in the corneal reflection image (CRI); and - Determining a difference between the position of the reference graphic shape element (8) and the position of the reflection (3*) of the object marker (3) in the corneal reflection image (CRI) and setting the difference as position difference (ΔL). [3] Method according to claim 1 or 2, characterized by that the computing unit adapts the projection of the object marking (3) onto the projection surface (1) taking into account the position difference (ΔL) so that the position difference (ΔL) is minimized. [4] Method according to claim 2 or 3, characterized bythat the computing unit transforms the position of contents in the corneal reflection image (CRI) and in the camera image (7) into a common reference coordinate system. [5] Method according to one of claims 2 to 4, characterized by that the corneal reflection image (CRI) is rectified after acquisition and before further processing to compensate for a spherical projection. [6] Method according to one of claims 2 to 5, characterized by that the reference graphic form element (8) has an L-shape. [7] Method according to one of claims 2 to 6, characterized by that the computing unit determines the pose of the reference graphic form element (8) relative to the object (4) in the camera image (7) depending on an object marking type. [8] Method according to one of claims 3 to 7, characterized by that the computing unit stores values for correcting the position difference (ΔL) in the form of a calibration for the field of view display device. [9] Method according to claim 8, characterized by that the computing unit stores the values used for the calibration specifically as a function of an object marking type, in particular taking into account a target shape and / or target position on the projection surface (1) provided for the respective object marking type. [10] Vehicle, comprising a field of view display device configured to output an augmented reality (AR) representation, an interior camera (5) directed towards the eye area of a driver (2), an environmental camera capturing at least a section of the surroundings, and a computing unit, characterized by that the field of view display device, the interior camera (5), the surroundings camera and the computing unit are set up to carry out a method according to one of claims 1 to 9.

Images