Method for visualizing an assistance graphic

Abstract

The invention relates to a method for visualizing an assistance graphic on a display device in a motor vehicle, wherein the assistance graphic is represented as a sequence of individual images which are arranged one after the other at a frame rate.The method according to the invention is characterized in that the frame rate is dynamically adaptively adjusted, for which purpose a person using the assistance graphic is detected by a driver observation camera and their gaze direction is determined, after which the resulting field of view (14) of the person is assigned to an object (12) shown in the assistance graphic, after which it is detected whether the field of view (14) follows a trajectory (11) of the shown object (12) to which the field of view (14) has been assigned, wherein in the case that the field of view (14) follows the trajectory (11), the fixation of the object (12) along the trajectory (11) is detected, after which an evaluation of the scatter of the fixation is carried out, and after which the frame rate is adjusted depending on whether the scatter exceeds or falls below a predetermined threshold value.

Description

[0001] The invention relates to a method for visualizing an assistance graphic according to the type defined in more detail in the preamble of claim 1.

[0002] Display devices in motor vehicles are known from the prior art. It is also known to monitor a person driving a motor vehicle using one or more cameras, for example, to determine their gaze direction. In this context, DE 10 2015 015 136 A1 describes a head-up display as a display device that adapts the information displayed to the actual use of the vehicle by the person driving. Essentially, the system addresses the fact that some people prefer more information, while others prefer less. By recording the person's gaze direction, it is possible to analyze which information the person regularly looks at and uses, and which information they do not. The display device can then be controlled according to the individual's preferences.

[0003] DE 10 2011 114 059 A1 describes a device for displaying moving images in a motor vehicle comprising a screen and a control unit which switches between two modes depending on the vehicle speed and the distance to objects ahead, whereby the frame rate is reduced in the second mode and the thresholds for the modes are adjusted by an isotonic function of the distance.

[0004] Furthermore, JP 2015 - 101 189 A describes a vehicle display that detects when the driver's gaze is not directed at the display and then reduces the light flicker component to improve the visibility of the vehicle display.

[0005] It is well known that displaying constantly changing content on a display device is relatively resource-intensive. This can be particularly problematic in a vehicle, which inherently has certain limitations in its data processing resources. This is especially true for so-called driver assistance graphics, which, by definition, depict moving objects, such as other road users, and stationary objects, such as roads and buildings, in the vehicle's surroundings. The individual objects are detected, in whole or in part, by the vehicle's environmental sensors and displayed to the driver, for example, on an instrument cluster or a head-up display. The display is presented as several consecutive frames, similar to a video display.The initial refresh rate of the individual frames is typically specified at 60 Hz. However, in practice, particularly in highly dynamic scenarios with many moving objects, such as in city traffic, it becomes apparent that the display device can no longer render some of the objects due to resource limitations. In such cases, either the number of visualized objects must be reduced to display the remaining objects, or the frame rate must be lowered.

[0006] Reducing the refresh rate can, however, lead to a jerky display for the person viewing or using the assistive graphics. This is due to the so-called flicker fusion frequency of the affected person, above which they perceive the sequence of images not as a continuous film but as successive individual frames. In practice, this flicker fusion frequency varies considerably, as it is both an individual characteristic of each person and also depends on the time of day, fatigue, and other factors, even for the same person.

[0007] The object of the present invention is to provide an improved method for visualizing an assistance graphic in which the refresh rate can be throttled if necessary, but always remains above the current critical flicker fusion frequency of the person viewing or using the assistance graphic.

[0008] According to the invention, this problem is solved by a method with the features in claim 1, and in particular in the characterizing part of claim 1. Advantageous embodiments and further developments are described in the dependent claims.

[0009] The inventive method provides for the dynamic adaptive adjustment of the frame rate to visualize an assistance graphic as defined above. To prevent a jerky image impression for the person viewing the display, the person using the assistance graphic is detected by a driver observation camera, and their gaze direction is determined. Based on this gaze direction, a viewing area of ​​the person can then be assigned to an object depicted in the assistance graphic. This can also be referred to as a static correspondence. Subsequently, it is detected whether the viewing area follows a trajectory of the depicted object to which the viewing area has been assigned. This would then be the dynamic correspondence. If this is the case, the fixation, i.e., the targeted viewing of the object along this trajectory, is detected. The variance of the fixation is then evaluated.The variation in fixation allows conclusions to be drawn about whether the person using the assistive graphic perceives the current image sequence as a smooth flow, or, with a sufficiently high variation, as a series of individual images. This leads to a jerky appearance in the perceived image sequence. Since the person cannot always directly follow the object's movements, but rather their gaze is sometimes still in the previous position when the object has already moved further along its trajectory, this results in a greater variation in fixation.

[0010] According to the invention, the frame rate can now be adjusted depending on whether the variation exceeds or falls below a predetermined threshold. This ensures that the user always sees the assistance graphic in a smooth sequence of images, while simultaneously reducing the frame rate as much as possible, at least for resource-intensive displays, to save resources and to be able to depict as many captured objects as possible with sufficient dynamic range.

[0011] The advantage of such adaptive adjustment of the refresh rate to the needs and current situation of the individual user enables exceptionally efficient resource utilization. In particular, resources can be freed up when a low flicker fusion frequency is detected. No human intervention is required; the adaptive dynamic adjustment occurs without any active participation from the person using the assistive graphics.

[0012] This method allows for highly realistic assistance graphics, as typically no objects need to be deleted due to resource constraints. The method is independent of the existing scene and automatically takes lighting and weather conditions into account if these affect the current flicker fusion frequency of the person.

[0013] The process can be repeated cyclically within a control loop and can be applied to virtually all dynamic display devices, such as screens, head-up displays, and the like.

[0014] A particularly advantageous embodiment of the method according to the invention provides that, prior to the method, the driver observation camera is calibrated in a higher-level coordinate system, so that the coordinate systems of the driver observation camera and the vehicle are calibrated relative to each other. If the location of the display device in this higher-level coordinate system is known, e.g., that of the vehicle, it is then possible to determine simply and precisely which of the objects, whose position on the display device and thus in the space of the vehicle is known, the person's gaze is directed at.

[0015] Another very advantageous design option involves displaying the assistance graphics at the start of the procedure in an instrument cluster or head-up display using a predefined refresh rate. This predefined refresh rate could, for example, be set at a typical refresh rate of 60 Hz, at which most people perceive the image sequence as smooth.

[0016] A further highly advantageous embodiment of the method according to the invention provides that the person's gaze direction is projected onto the display device in order to assign the person's field of vision to one of the displayed objects. Particularly when, due to the calibrated driver observation camera and the matching coordinate systems of the vehicle and the display device on the one hand, and the driver observation camera on the other, the spatial position of the objects on the surface of the display device is known, such a projection of the gaze direction onto the display device, which can be visualized but does not necessarily have to be, allows for the very simple and efficient identification of the object to which the person's gaze is directed.

[0017] Another highly advantageous approach involves determining the standard deviation between the object's position and the person's field of vision as a measure of dispersion. According to a particularly beneficial further development, a threshold value on the order of 1 can be used for the standard deviation, allowing for appropriate action depending on whether the determined dispersion value is above or below this threshold.

[0018] According to a particularly advantageous embodiment of the inventive method, it is now provided that, in the event that the variation lies below the predetermined threshold, the frame rate is maintained or reduced. With regard to the particularly advantageous embodiment using the standard deviation with a threshold of 1, this would mean that the described case, in which the frame rate is maintained or reduced, always occurs when the standard deviation is less than 1.

[0019] Maintaining the frame rate is correspondingly simple. However, to free up resources, possibly for subsequent operations, or in case of a current resource shortage, the frame rate can also be reduced. In particular, such a reduction can be performed iteratively, according to a very advantageous advanced technique, in order to gradually approach the threshold and determine the currently minimum possible frame rate in several iterative steps.

[0020] Conversely, a highly advantageous embodiment provides that if the dispersion exceeds a predetermined threshold, or, in the described embodiment with the standard deviation, is greater than 1, the frame rate is increased. In principle, it would of course be conceivable to increase it directly to the initially predetermined value or a maximum value to be on the safe side in any case. However, it is more resource-efficient if, according to a highly advantageous embodiment of the inventive method, the increase is also carried out iteratively in order to approach the minimum necessary frame rate for displaying a smooth motion sequence, as already explained above – but now from the other direction.

[0021] In particular, but not exclusively, in the case of iterative adjustment, a highly advantageous further development may entail repeating the process with the newly adjusted frame rate as the new initial frame rate. The process is thus repeated to ensure continuous adaptation to the current conditions. What does not need to be repeated is the upstream step of calibrating the driver monitoring camera.

[0022] Further advantageous embodiments and developments of the method according to the invention also become apparent from the exemplary embodiment, which is described in more detail below with reference to the figures.

[0023] This shows: Fig. 1 a schematic flowchart of a possible embodiment of the method according to the invention; Fig. 2 a representation according to procedure step VS 7 from the representation of the Fig. 1; and Fig. 3 a representation of the detection according to procedure step VS 8 from the representation according to Fig. 1.

[0024] The core of this description is to adapt the visualization of an assistance graphic in the aforementioned sense to the resource load in such a way that a comfortable visualization is still guaranteed, taking into account the individual critical flicker fusion frequency.

[0025] As shown in the depiction of the Fig. As illustrated in an exemplary flow diagram, in a first upstream process step VS 1 a driver observation camera is calibrated in a higher-level coordinate system, in particular the coordinate system of the vehicle, whereby the intrinsic and extrinsic position in relation to a coordinate origin within the vehicle is known.

[0026] In a further process step, VS 2, the assistance graphic is generated based on data from the environmental sensors and map data and displayed on a display device. This makes it known which objects, with which position and movement trajectory, are visualized on the display device, for example, an instrument cluster, a head-up display, or the like.

[0027] In the subsequent third process step, VS 3, the gaze direction of a person using the assistance graphic is recorded and analyzed. This person is typically the driver of the vehicle. The recording and analysis of gaze directions is a well-established technique, so the specific eye-tracking algorithms used (Purkinje Reflection, Iris Silhouette Tracking, etc.) need not be discussed further. This allows for the detection of saccades (gaze movements) and fixations (gaze fixations). Based on the three-dimensional relationship between the driver observation camera and the vehicle's coordinate system, established during calibration, the person's field of view can be projected onto the display device in a fifth process step, VS 5. The position and trajectory of the objects in this display were previously stored in a fourth process step, VS 4.

[0028] This allows the saccade, and especially the person's fixation, to be assigned to a specific visualized object, which, as mentioned above, is also referred to as static correspondence. This is done in the representation of the Fig. In step VS 6, the object's movement (its trajectory) is compared to the eye movements of a person for a predetermined number of frames. If the object's movement vectors and the person's eye movements are similar, a dynamic correspondence is assumed. The person then observes the identified visualized object and follows its trajectory with their gaze. If no similar movement vectors are detected, the correlation from step VS 6 is rechecked using a different object. Steps VS 6 and VS 7 are repeated until a sufficient number of similar movement vectors between the object's trajectory and the person's eye movements are identified.

[0029] In the subsequent crucial step of the procedure, VS 8, the scatter of the object's fixation is evaluated along its trajectory in relation to the object's trajectory. A high scatter suggests that the person—as described above—does not perceive the visualization as a continuous sequence of images, but rather at least partially resolves individual frames, resulting in a jerky display.

[0030] If this variation exceeds a defined threshold, a control unit for the display device is instructed to increase the refresh rate in iterative steps, as the current refresh rate setting is potentially below the flicker fusion frequency currently present in the individual. This allows for adaptive adjustment of the refresh rate to the individual and highly variable flicker fusion frequency of each person at any given time. This enables resource-efficient visualization that is perceived as good quality by the person. This approach also allows for the visualization of highly realistic assistive graphics, as increased resource usage does not necessarily lead to a reduction in the number of objects displayed.As explained above, resources can be freed up by reducing the refresh rate until the minimum refresh rate at which the person still perceives the process as smooth is reached.

[0031] The aforementioned threshold comparison is visualized in the representation of the Fig. 1 where process step VS 9, process step VS 10 shows the reduction or increase of the frame rate.

[0032] The entire process between steps VS 2 and VS 10 is then repeated cyclically, ensuring continuous adjustment. For this purpose, step VS 2 is started with the previously determined new frame rate as the initial frame rate for the next process run.

[0033] In the presentation of the Fig. Section 2 will now demonstrate, using two exemplary examples, how the dispersion of fixation can behave. In the representation of the Fig. 2a) A trajectory 11 of an object labeled 12 is shown. The object 12 is depicted multiple times along its trajectory 11. In the vertical direction, various so-called frames of the representation are schematically indicated and labeled with the corresponding times T0 to T4. Within each of the multiple depictions of object 12, a filled point labeled 13 at the bottom of the object 12 is visible, representing the center of the object 12 on its trajectory 11. Additionally, an unfilled circle marks the gaze area 14 of the person, i.e., the area on which the person is currently looking in the current frame or subframe. The distance shown by the thin line between the center 13 of object 12 and the gaze area 14 symbolizes the dispersion of the fixation of the gaze area 14 relative to the actual position of object 12. Fig. 2b) This is shown again on the right.

[0034] This is in the Fig. 2a) to recognize that the gaze areas 14 largely follow the objects 12. The dispersion of fixation is low. In the representation of the Fig. In contrast, 2b) shows that several viewing areas 14 are assigned to one object position, while no viewing area 14 is assigned to other positions of the object 12. One of the two viewing areas 14 at the second object position from below actually belongs to the third object position above it. The dispersion between the viewing area 14 on the one hand and the center 13 of the object 12 on the other is therefore significantly higher here.

[0035] The presentation of Fig. Section 3 addresses this accordingly. The unfilled bars, which represent the time axis on the x-axis, indicate that... Fig. 2a) shows the case in which the dispersion, represented by the distance between the center 13 of object 12 and the viewpoint 14 on the y-axis, is comparatively small. In each of the individual frames, the case of the hatched bars is also shown. Fig. 2b) shown, where it can be seen that the deviation here is analogous to the representation in Fig. 2b), between the viewing area 14 and the center 13 of the object 12 in the third and fifth representations is very large.

[0036] This graphic can therefore be used to illustrate the first in Fig. 2a) In the case shown, a rather low dispersion can be perceived very easily; for the second case in the Fig. 2b) a very high dispersion. In the embodiment shown here, the standard deviation for the case according to Fig. 2a) at approximately 0.4, while the standard deviation for the case according to Fig. 2b) is 1.9.

[0037] In the case according to Fig. 2a) the refresh rate can be reduced, whereas in the case according to Fig. 2b) must be increased in order to visualize a smooth-looking representation of the assistance graphic for the person.

Claims

[1] Method for visualizing an assistance graphic on a display device in a motor vehicle, wherein the assistance graphic is represented as a sequence of individual images which follow one another at a frame rate, characterized by, that the frame rate is dynamically adaptively adjusted, for which purpose a person using the assistance graphic is detected by a driver observation camera and their gaze direction is determined, after which the resulting field of view (14) of the person is assigned to an object (12) shown in the assistance graphic, after which it is detected whether the field of view (14) follows a trajectory (11) of the shown object (12) to which the field of view (14) has been assigned, wherein in the case that the field of view (14) follows the trajectory (11), the fixation of the object (12) along the trajectory (11) is detected, after which an evaluation of the scatter of the fixation is carried out, and after which the frame rate is adjusted depending on whether the scatter exceeds or falls below a predefined threshold value. [2] Method according to claim 1, characterized bythat the procedure is preceded by a calibration of the driver observation camera in a higher-level coordinate system of the vehicle. [3] Method according to claim 1 or 2, characterized by , that a visualization of the assistance graphics at the beginning of the procedure takes place in an instrument cluster or a head display as a display device using an initially specified refresh rate. [4] Method according to claim 3, characterized by , that the initial refresh rate is set to approximately 60 Hz. [5] Method according to any one of claims 1 to 4, characterized by , that a projection of the person's gaze direction into the visualization of the assistance graphic on the display device takes place in order to assign the person's gaze area (14) to one of the depicted objects (12). [6] Method according to any one of claims 1 to 5, characterized by, that the standard deviation between the position of the object (12) and the field of view (14) of the person is determined as a measure of dispersion. [7] Method according to claim 6, characterized by , that the threshold for the standard deviation is set at approximately 1. [8] Method according to any one of claims 1 to 7, characterized by , that if the variation is below the specified threshold, the refresh rate will be maintained or reduced. [9] Method according to claim 8, characterized by that the reduction only occurs if the available resources are utilized beyond a predetermined limit. [10] Method according to claim 8 or 9, characterized by that the reduction takes place in iterative steps. [11] Method according to any one of claims 1 to 10, characterized by , that if the variation is above the specified threshold, the refresh rate will be increased. [12] Method according to claim 11, characterized by that the increase occurs in iterative steps. [13] Method according to any one of claims 1 to 12, characterized by , that the procedure is repeated with the adjusted refresh rate as the new initial refresh rate.

Images