Adaptive filter chain for displaying environmental model in vehicle

Abstract

The invention relates to a system for displaying information on objects in the surroundings of a vehicle, in which a computing unit (5) determines the direction of sight of the driver with reference to the surroundings by viewing a camera (3) by the driver and checks whether the objects in the surroundings are thus tracked; if the tracking situation is confirmed, a deviation between a line-of-sight direction trajectory pointing to the object to be tracked and a position trajectory based on data of the environment sensor unit (1) is calculated in the projection image, and filter parameters of a filter for displaying the object to be tracked and applied to the sensor data are adjusted in order to minimize the deviation. And a display unit (7) is controlled to display the position of the tracked object with time in a filtered manner.

Description

Technical Field

[0001] The present invention relates to a system for displaying object information in a vehicle environment, a vehicle equipped with the system, and a method for displaying object information in a vehicle environment. Background Technology

[0002] It is known in the prior art to generate and render an environmental model based on sensor data acquisition, to trace sensor data about objects in the environment, and to display it on a display device.

[0003] In this context, DE 10 2021 201 065 A1 relates to a method for displaying an environmental model of the environment surrounding a motor vehicle using an environmental detection system for a motor vehicle; wherein the environment surrounding the motor vehicle is detected by at least one detection device of the environmental detection system, and the detected environment is graphically displayed on the display device of the environmental detection system in the form of an environmental model; wherein the multi-lane situation of the road on which the motor vehicle is located is displayed in the environmental model, the road containing at least two lanes for travel in the same direction; wherein group data is received from an electronic computing device outside the motor vehicle by a group data receiving device of the environmental detection system, and the multi-lane situation of the road is identified and displayed based on the group data.

[0004] Furthermore, DE 10 2018 112 345 A1 relates to a method for generating high dynamic range images using an HDR camera system; wherein the method includes taking multiple exposure shots with a camera of the HDR camera system, generating a first HDR image from a first subset of the multiple exposures (each exposure in the first subset has a different exposure value), and generating a second HDR image from a second subset of the multiple exposures (the second subset includes at least one exposure from the first subset, and at least one additional exposure taken closer to the exposure time of the first subset, and each exposure in the second subset has a different exposure value).

[0005] Meanwhile, the sensor units in a vehicle are typically designed to acquire data for driver assistance systems. This falls under the category of environmental perception, whose primary task is to generate data for driver assistance systems such as driving or parking functions. Therefore, sensor data is primarily suitable for such driver assistance systems, but may not be optimally suited for visualizing environmental models containing dynamic and / or static objects in the vehicle's surroundings, models that are determined using data from these sensor units. These environmental models can be generated, for example, directly from the sensor unit data, or using the results of fusion of static or dynamic sensor data. This design is also known as "navigation perception." In this process, ADAS (Advanced Driver Assistance System) sensor data and navigation data are typically displayed to the driver on a central screen.

[0006] The primary purpose of environmental model visualization is to enable drivers to understand the behavior of such driver assistance systems; however, the visualization itself relies heavily on data from the vehicle's perception channels, which can utilize multiple sensors on the vehicle individually or in combination. However, due to the aforementioned limitations in the applicability of environmental model data visualization, the visualized data may exhibit significant temporal jitter or spatial shifts, resulting in spatial instability of displayed objects relative to their surroundings—a phenomenon that manifests as shaking of objects in the image to the observing driver, thus creating a sense of disturbance. The quality of object recognition depends heavily on the object category (especially in data-driven approaches) and the specific environmental conditions. Therefore, to correct for position-dependent behavior of objects in the displayed image, filter chains (e.g., box filters or Kalman filters spanning multiple frames) are suitable.

[0007] Eliminating interference, noise, and artifacts in visualizations through filtering loops is often complex. Furthermore, this approach results in the filter applying the same filtering treatment to all objects regardless of sensor performance and environmental conditions—which can lead to over- or under-filtering of object positions. Such filter chains with suboptimal parameters can even cause lag in the display of objects. This effect is particularly pronounced when using time filters with fixed window widths. While the data exhibits temporal stability, this also results in visualization lag. Therefore, a conflict arises between the goals of artifact elimination and accurate display on the display unit. Summary of the Invention

[0008] The purpose of this invention is to improve the filtering effect of the position information of objects to be displayed in an environment model.

[0009] This invention derives from the features of the independent claims. Advantageous improvements and designs are the subject of the dependent claims.

[0010] A first aspect of the invention relates to a system for displaying object information in a vehicle's surrounding environment, comprising a sensor unit for acquiring sensor data about objects, a driver observation camera, a computing unit, and a display unit; wherein the computing unit is configured to determine the driver's current gaze direction using data from the driver observation camera, associate each gaze direction with an observed target in the environment, and then analyze the trajectory of the observed target to verify: i) whether the gaze direction is tracking an object in the environment within a preset time period; and ii) whether the tracked object is consistent with an object detected by the sensor unit and to be displayed on the display unit within the preset time period; if both conditions i) and ii) are satisfied, then in a shared projection image suitable for determining the deviation, the deviation between the gaze direction trajectory pointing to the tracked object and the corresponding object position trajectory determined by the sensor unit based on sensor data is determined, and at least one filter parameter of a filter applied to the sensor data for displaying the tracked object is adjusted such that the position trajectory of the corresponding object displayed on the display unit by applying the filter changes, thereby reducing the deviation, and then the display unit is controlled such that the position of the tracked object displayed on the display unit over time is determined by the adjusted filter parameter.

[0011] The objective of this invention is primarily achieved by observing and utilizing the driver's eye movements to combine and coordinate the acquired information with information from the vehicle's environmental perception channels. In other words, by obtaining relevant information from the driver's eye movements and their deducible gaze directions, the data collected by the sensor unit for displaying dynamic and / or static objects in the vehicle's surrounding environment is enhanced.

[0012] Specifically, this enhanced support is achieved by using the parameters of one or more filters (particularly those determining the dynamic bandwidth of the filters) as optimization variables to optimize the positional trajectory displayed by the eye-tracked object. This is achieved in such a way that the positional trajectory of the eye-tracked object in the vehicle's surrounding environment follows the eye trajectory, thus ideally following the object itself. This following behavior is achieved thanks to at least one appropriately optimized filter parameter value.

[0013] Here, whether the filter parameters of one or more filters are adjusted, whether a single filter or multiple filters are used, and whether multiple filter parameters of a single filter are adjusted or multiple filter parameters are adjusted, is in principle not important. The only crucial point is that the behavior of the applied filter chain is adapted as described above and below. In particular, a Kalman filter can be used as the filter. Examples of different filter parameters include filter windows and filter functions.

[0014] The driver observes the camera directly capturing eye movements. Based on these eye movements, the direction of gaze for each eye can be determined, thus identifying the focal point. To correlate the driver's gaze with the environment, a data format compatible with the sensor unit's information is required. This is achieved through image projection; specifically, the environmental information acquired by the sensor unit is mapped onto a preferably two-dimensional projected image, and the gaze direction is mapped to corresponding coordinates within that projected image. Thus, the projected image establishes a unified benchmark for comparing the environmental information acquired by the sensor with the gaze direction of the observed environment.

[0015] It is preferable to use a so-called "distance image" as the projected image (which is essentially the result of projecting the 3D environment onto a 2D plane). This allows for direct geometric comparison, analysis of the driver's line of sight sweeping across the environment, or which object the line of sight is tracking relative to the surrounding environment, and what information the sensor units provide about these trajectories or objects.

[0016] Specifically, the computing unit determines, based on the driver's gaze direction, whether the driver is tracking an object with their eyes within a preset time period—that is, focusing on the object and following it with their gaze. This ensures the driver continuously observes and focuses on a moving object over a longer period. The verification is performed by defining a region around a fixed window on the projected image (e.g., a "distance image"). If this region remains stable, it can be inferred that a location related to the same object is being focused on. This allows the spline curves of the moving object to be derived on the two-dimensional plane of the projected image (e.g., the "distance image"). These spline curves are then used as reference splines to adjust at least one filter parameter.

[0017] When these preconditions are met, relevant information is generated based on the acquired driver eye movement data, which can be used in subsequent processing. Therefore, it is also necessary to verify whether the object being tracked by the driver's gaze has also been captured with sufficient information from the sensor unit, and whether the object needs to be displayed on the vehicle's display unit (especially the central screen inside the vehicle) so that the driver can form a spatial cognitive image of the surrounding environment based on the environmental model presented on the display unit.

[0018] If these conditions are met, the process continues as described above: In the projected image, the trajectory of the tracked object, obtained based on data from the driver's observation camera, is compared with the position trajectory obtained based on the sensor unit to determine geometric deviation. The sensor unit records the trajectory of an object moving relative to its environment in three-dimensional space and adapts it using filters. The updated filter chain is then preferentially projected into the projected image and compared with a reference spline curve. For example, the resulting deviation can be defined as the orthogonal L2 distance.

[0019] The goal of subsequent optimization is to minimize this deviation. This involves adjusting at least one filter parameter so that the position trajectory based on the sensor unit is adapted to the tracking trajectory based on the driver's line-of-sight behavior. This results in improved filter parameter values, and the object's position trajectory is presented more reliably and realistically on the display unit. The optimization process ends when the corresponding minimum value or termination criterion is reached, and the current filter bank is used to filter the ADAS perception data for the corresponding time period for use by the visualization system.

[0020] The object whose trajectory is displayed on the screen by the filter whose parameters are to be adjusted, specifically a dynamic object, that is, an object moving relative to its surrounding environment, such as another road user. It is these other road users who fundamentally determine the behavior of the driver assistance system, especially when the system responds actively, i.e., reacting to the surrounding traffic conditions. Understanding these responses is particularly important for the driver, which is why an environmental model including the vehicle's surroundings, especially dynamic objects, is presented on the display unit.

[0021] The optimization cycle can be initiated at preset time intervals or correlated with the availability of a reference spline curve; the latter is particularly advantageous in rural environments. Similarly, correlation with perceived quality metrics is also feasible. Optimization typically requires continuous repetition because the quality of sensor data can fluctuate significantly and may be affected by sensor contamination levels, specific lighting conditions, or weather conditions.

[0022] The beneficial effects of this invention are that, through adaptive adjustments to the auxiliary visualization, objects in the environment are presented more naturally on the display unit, unaffected by lighting and weather conditions. Furthermore, when displaying moving objects around a vehicle, they are presented better independently of static scenes. Additionally, the system and method can adjust various filter parameters during optimization, regardless of the specific sensor type used in the sensor unit or whether they are fused together. Individual sensors in the sensor unit may include, for example, cameras, radar, and lidar.

[0023] According to an advantageous embodiment, the computing unit is configured to project data from the sensor unit onto a two-dimensional projected image in order to determine the deviation, and to map the line of sight direction to the coordinates of the projected image, such that the trajectory of the observed target in the line of sight direction in the projected image and the position trajectory of the corresponding object in the projected image are both referenced to the same standardized position in the environment, thereby determining the deviation by calculating the corresponding direct geometric differences.

[0024] Geometric differences are preferably calculated as L2 distance, and more preferably as the effective integral of the distance between the spline curve of the gaze point on the tracked object and the object trajectory relative to the environment based on the same reference projection image according to sensor data at their respective time points.

[0025] According to another advantageous embodiment, the filter includes at least one of the following: a Kalman filter, a box filter, a low-pass filter, and a moving average filter.

[0026] According to another advantageous embodiment, the computing unit is configured to classify the corresponding objects detected by the sensor unit, and determine and apply specific filter parameters for the corresponding objects presented on the display unit according to their categories based on the classification results.

[0027] In particular, objects that move relative to their environment can be categorized into different types, such as cyclists, passenger cars, trucks, and pedestrians.

[0028] According to another advantageous embodiment, the calculation unit is configured to determine a preset time period based on the classification of the corresponding object and / or based on the distance between the corresponding object and the vehicle.

[0029] According to another advantageous embodiment, the system also has a communication module, wherein the computing unit is configured to send at least one adjusted filtering parameter to a central computer or another traffic participant, particularly another traffic participant of the same class as the vehicle, via the communication module.

[0030] The classification of this vehicle can be based on the classification of other traffic participants. Therefore, specifically, when this vehicle is a passenger car, the optimized filter parameters can be passed only to other passenger cars.

[0031] According to another advantageous embodiment, the computing unit is configured to verify whether the line of sight is in the tracked environment within a preset time period, and to evaluate whether the target point of the line of sight falls within the planar area projected onto the vehicle environment surrounding each object for at least a preset time period.

[0032] According to another advantageous embodiment, the computing unit is configured to determine the size of the planar region based on the distance between the corresponding object and the vehicle.

[0033] According to another advantageous embodiment, the computing unit is configured to verify whether the tracked object matches the object collected by the sensor unit and to be displayed on the display unit within a preset time period; and to perform image similarity comparison for this purpose, particularly calculating the structural similarity index.

[0034] Another aspect of the invention relates to a vehicle having the systems described above and below. The advantages and preferred improvements of the proposed vehicle are derived through similar and analogous transfers of the above-described embodiments related to the proposed system.

[0035] Another aspect of the present invention relates to a method for displaying object information in the environment surrounding a vehicle, comprising the following steps:

[0036] - Sensor data about an object is acquired through a vehicle sensor unit, and the following operations are performed by the computing unit:

[0037] -Use data from the vehicle driver's observation camera to determine the driver's real-time gaze direction;

[0038] - Associate each line of sight with the observed target in the environment and check the trajectory of the observed target to determine: i) whether the line of sight is tracking an object in the environment within a preset time period, and ii) whether the tracked object matches the object detected by the sensor unit and displayed on the display unit within the preset time period; if both conditions i) and ii) are satisfied:

[0039] - In a shared projection image suitable for determining the deviation, determine the deviation between the trajectory of the line of sight pointing to the tracked object and the corresponding object position trajectory determined by the sensor unit based on sensor data;

[0040] - Adjust at least one filter parameter of the filter applied to the sensor data for displaying the tracked object, such that the position trajectory of the corresponding object displayed on the display unit changes by applying the filter, thereby reducing the deviation; and subsequently

[0041] - Control the display unit so that the position of the tracked object on the display unit over time is determined by the adjusted filter parameters.

[0042] The advantages of the proposed method and the preferred improvements are derived through similar and analogous transfers of the above-described implementation schemes related to the proposed system.

[0043] Further advantages, features, and details are derived from the following description, in which at least one embodiment is described in detail—referring, if applicable, to the accompanying drawings. Identical, similar, and / or functionally identical parts are labeled with the same reference numerals. Attached Figure Description

[0044] In the attached image:

[0045] Figure 1 An in-vehicle system for displaying information about objects in the environment surrounding a vehicle, according to an embodiment of the present invention.

[0046] Figure 2 A method for displaying object information in the environment surrounding a vehicle, according to an embodiment of the present invention. Detailed Implementation

[0047] The contents shown in the attached diagram are illustrative and not drawn to scale.

[0048] Figure 1A partial view of a vehicle equipped with a system for displaying information about objects in the vehicle's surrounding environment is shown. The vehicle has a sensor unit 1, which is part of the system, for acquiring sensor data about the vehicle's surrounding environment. Therefore, this sensor data also includes information about moving objects (e.g., other road users) in the vehicle's detectable environment. The sensor data primarily serves the vehicle's driver assistance system or autonomous driving control system, providing it with real-time information about the surrounding environment and the road users present therein, enabling the driver assistance system or autonomous driving control system to execute appropriate responses. To enable the vehicle's driver to better understand these responses, data about road users is presented on the vehicle's display unit 7, which maps the actual movement of the road users to the movement of symbols representing these road users on the display unit 7. Since the sensor data is specifically designed for use in the vehicle's driver assistance system or autonomous driving control system and not necessarily for display on the display unit 7, one or more filters are used to adapt this sensor data, which contains information about other road users and their positional trajectories over time, thereby achieving more realistic and smoother positional trajectories of other road user movements on the display unit 7. The at least one filter has at least one filter parameter, and the application of the filter and the control of the display unit 7 are performed by the computing unit 5, which is particularly (but not necessarily) located within the vehicle. Since predetermined filter parameters are unlikely to be optimal for all situations, it is advantageous to reference the driver's behavior (where the behavior includes relevant information) to improve the display of the positional trajectories of other traffic participants in the vehicle's surrounding environment. For this purpose, the driver's gaze trajectory in the environment over time is acquired by the driver's observation camera 3, and these gaze directions are mapped onto a projected image of the current environment, which also maps the information acquired by the sensor unit 1. When the computing unit 5 determines that the focused gaze point continuously falls on another traffic participant for a preset time period (…), Figure 1(Seen as a solid black circle within a rectangular defined area surrounding the other traffic participant). The calculation unit 5 then compares the position trajectory determined by the calculation unit 5 through scanning the environment via the direction of the driver's gaze with the position trajectory of the other traffic participant detected by the sensor unit 1, which is also mapped onto the projected image. The rectangular defined area is defined by the sensor data of the sensor unit 1 as an area that moves synchronously with the other traffic participant and where the driver's gaze focus must continuously fall within a preset time period, so that the calculation unit 5 can identify that the driver is indeed tracking the object during that preset time period. If this is the case, usable information is obtained, and the filter can be optimized using the driver's eye tracking of the other traffic participant. The filter parameters are optimized such that the deviation of the motion data of the other traffic participant calculated by the sensor data of the sensor unit 1 is as consistent as possible with the deviation of the motion data of the other traffic participant tracked by the driver's gaze direction. Accordingly, the filter parameters are adjusted, and the resulting filter parameters are used to display the motion trajectory of the other traffic participant on the display unit 7. More details are available in... Figure 2 The method is shown in the illustration.

[0049] Figure 2 A corresponding method for displaying object information in the environment surrounding a vehicle is shown, which can be used in, for example... Figure 1The system operates as described. In the first step of the method, sensor data regarding objects is determined by sensor unit 1 used for environmental observation of the vehicle (S1). Cameras, radar, lidar, ultrasonic distance sensors, and other typical sensors can be used in this process. Calculation unit 5 determines the driver's real-time gaze direction (S2) by observing data from camera 3, and associates each gaze direction (S3) with the observed target in the environment by providing a projected image in the form of a so-called "distance image" and detecting the driver's gaze direction on the previously derived "distance image." The trajectory of the observed target is then examined to determine: i) whether the gaze direction is tracking an object in the environment within a preset time period, and ii) whether the tracked object matches the object detected by sensor unit 1 and displayed on display unit 7 within the preset time period. Step ii) is determined by using a "structural similarity index measurement" (SSIM) compared to a preset threshold to determine the correspondence between the object focused by the driver and the three-dimensional spatial object derived from the sensor data. The degree of overlap is checked according to a defined error metric. To this end, the driver's focus is repeatedly determined, thereby deriving a reference spline curve of the object across multiple frames based on the data from the driver's observation camera 3. The reference spline curve is derived from the focused "distance image". Next, the deviation between the reference spline curve based on the trajectory of the gaze direction pointing to the tracked object and the position trajectory of each object determined by the sensor unit 1 based on sensor data (both referencing the projected image) is determined (S4); then, at least one filter parameter of the filter applied to the sensor data for displaying traffic participants tracked by the driver's gaze in front of them is adjusted (S5) so that the corresponding object displayed on the display unit 7 by applying the filter (such as...) Figure 1 The position trajectory of the traffic participants (in the process) is altered in this way to reduce deviation. Subsequently, control of display unit 7 is performed S6 so that the position of the tracked object on display unit 7 over time is actually determined and displayed by the adjusted filter parameters.

[0050] While the invention has been described and explained in more detail through preferred embodiments, it is not limited to the disclosed examples, and those skilled in the art can derive other variations therefrom without departing from the scope of protection of the invention. Therefore, it is apparent that numerous variations are possible. It is also clear that the above embodiments are merely illustrative examples and should not be construed in any way as limiting, for example, the scope of protection, applicability, or configuration of the invention. Rather, the foregoing description and the description of the figures enable those skilled in the art to specifically implement the exemplary embodiments, wherein, knowing the disclosed spirit of the invention, various changes can be made, for example, with respect to the function or arrangement of individual elements described in the exemplary embodiments, without departing from the scope of protection defined by the claims and their legal counterparts.

Claims

1. System for displaying information about objects in the surroundings of a vehicle, with a sensor unit (1) for acquiring sensor data about the objects, a driver observation camera (3), a computing unit (5), and a display unit (7); wherein, The computing unit (5) is configured to determine the current gaze direction of the driver using the data of the driver observation camera (3), to correlate each gaze direction to an observed target in the environment, and to then analyze the trajectory of the observed target in order to verify that: i) the gaze direction is tracking an object in the environment for a preset time period; and ii) the tracked object is consistent with an object detected by the sensor unit (1) and to be displayed on the display unit (7) for a preset time period; if both conditions i) and ii) are met, to determine, in a common projection image suitable for determining the deviation, a deviation between the trajectory of the gaze direction pointing to the tracked object and the trajectory of the corresponding object position determined by the sensor unit (1) based on the sensor data, and to adjust at least one filter parameter of a filter to be applied to the sensor data for displaying the tracked object such that the trajectory of the position of the corresponding object displayed on the display unit (7) by applying the filter is changed to reduce the deviation, and to subsequently control the display unit (7) such that the position of the displayed tracked object over time on the display unit (7) is determined by the adjusted filter parameter.

2. The system according to claim 1, wherein The computing unit (5) is configured to, in order to determine the deviation, project the sensor data of the sensor unit (1) into a two-dimensional projection image and to correspond the gaze direction to coordinates of the projection image such that the trajectory of the observed target of the gaze direction in the projection image and the trajectory of the position of the corresponding object in the projection image both refer to the same standardized position in the environment, so that the deviation can be determined by calculating the corresponding direct geometric difference.

3. The system according to any one of the preceding claims, wherein The filter comprises at least one of the following: Kalman filter, box filter, low-pass filter, moving average filter.

4. The system according to any one of the preceding claims, wherein The computing unit (5) is configured to classify the corresponding objects detected by the sensor unit (1) and to determine and apply specific filter parameters for the corresponding objects presented on the display unit (7) according to the class, respectively, depending on the classification result.

5. The system according to claim 4, wherein The computing unit (5) is configured to determine the preset time period depending on the classification of the corresponding object and / or depending on the distance of the corresponding object from the vehicle.

6. The system according to any one of the preceding claims, Also having a communication module, wherein, The computing unit (5) is configured to transmit at least one adjusted filter parameter to a central computer or one other road user, in particular one other road user of the same type as the vehicle, having one of the systems, via the communication module.

7. The system according to any one of the preceding claims, wherein The computing unit (5) is configured to verify whether the gaze direction is tracking an object in the environment for a preset time period and for this to assess whether the target point of the gaze direction is continuously falling within a planar area projected into the environment of the vehicle around each object for a preset time period.

8. The system according to claim 7, wherein The computing unit (5) is configured to determine the size of the planar region depending on the distance of the respective object from the vehicle.

9. System according to any of the preceding claims, wherein The computing unit (5) is configured to check whether the tracked object is consistent with the object captured by the sensor unit (1) and to be displayed on the display unit (7) within a predetermined time period; and for this performs an image similarity comparison, in particular a calculation of a structural similarity index.

10. Vehicle having a system according to any of the preceding claims.

11. Method for displaying object information in the environment of a vehicle, having the following steps: - acquiring (S1) sensor data about objects by means of a sensor unit (1) of the vehicle; and performing the following by means of a computing unit (5) respectively: - determining (S2) the respective real-time line of sight of the driver of the vehicle using data of an observation camera (3) of the driver; - associating (S3) the respective line of sight to an observed target in the environment and checking the observed target trajectory to determine whether i) the line of sight is tracking an object in the environment within a predetermined time period and ii) the tracked object is consistent with an object detected by means of the sensor unit (1) and to be displayed on the display unit (7) within a predetermined time period; if both conditions i) and ii) are fulfilled: - determining (S4) a deviation between the line of sight trajectory pointing to the tracked object and a respective object position trajectory determined by means of the sensor unit (1) based on the sensor data in a common projection image suitable for determining the deviation; - adjusting (S5) at least one filter parameter of a filter applied to the sensor data for displaying the tracked object such that the position trajectory of the respective object displayed on the display unit (7) by applying the filter is changed in such a way that the deviation is reduced; and subsequently - controlling (S6) the display unit (7) in such a way that the position of the displayed tracked object over time on the display unit (7) is determined by the adjusted filter parameter.

Images