Adaptive filter chain for displaying an environment model in a vehicle

Abstract

The invention relates to a system for displaying information about objects in the surroundings of a vehicle, wherein a computing unit (5), by means of a driver observation camera (3), determines driver viewing directions with respect to the surroundings and checks whether an object in the surroundings is thus being tracked, and in the positive case determines a deviation between the course of the viewing directions directed at the tracked object and a position course according to the data of an environment sensor unit (1) in a projection image and, in order to minimize the deviation, adapts a filter parameter of a filter to be applied to the sensor data in order to display the tracked object, and also controls the display unit (7) for filtered display of the time-dependent position of the tracked object.

Description

[0001] Adaptive filter chain for displaying an environment model in a vehicle

[0002] The invention relates to a system for displaying information about objects in the surroundings of a vehicle, a vehicle with such a system, and a method for displaying information about objects in the surroundings of a vehicle.

[0003] In the prior art, it is known to generate and render an environment model obtained from sensor data as an image in order to reproduce sensor data about objects in the environment and to display it on a display device.

[0004] In this context, DE 102021 201 065 A1 relates to a method for displaying an environment model of the environment of a motor vehicle by means of an environment detection system of the motor vehicle, in which the environment of the motor vehicle is detected by means of at least one detection device of the environment detection system and the detected environment is displayed graphically on a display device of the environment detection system as the environment model, wherein the environment model displays a multi-lane roadway on which the motor vehicle is located and which has at least two lanes that can be driven in the same direction, wherein swarm data are received from an electronic computing device external to the motor vehicle by means of a swarm data receiving device of the environment detection system, wherein the multi-lane roadway is detected and displayed on the basis of the swarm data.

[0005] DE 102018 112 345 A1 further relates to a method for generating high dynamic range images with an HDR camera system, the method comprising recording a plurality of camera exposures with a camera of the HDR camera system, generating a first HDR image from a first subset of the plurality of camera exposures, each camera exposure of the first subset having a different exposure value, and generating a second HDR image from a second subset of the plurality of camera exposures, the second subset comprising at least one exposure from the first subset and at least one additional exposure recorded more recently than the exposures of the first subset, each exposure of the second subset having a different exposure value.

[0006] A sensor unit in a vehicle typically serves the purpose of obtaining data for a driver assistance system. This occurs within the framework of environment perception, which has the primary task of generating data for driver assistance systems such as the vehicle's driving or parking functions. Accordingly, the sensor data is primarily suitable for such driver assistance systems, but not necessarily optimal for visualizing an environment model with moving and / or static objects in the vicinity of the vehicle, which is determined using this data from the sensor unit. Such an environment model can, for example, be generated directly from the sensor unit data or using the result of a static or dynamic sensor data fusion. This process is also called "navistency."Typically, ADAS (short for "advanced driver assistance system") sensor data and navigation data are displayed to the driver on a central screen.

[0007] The visualization of the environment model serves the particular purpose of making the behavior of such a driver assistance system understandable for the driver, while the visualization itself is primarily based on the data in the vehicle's perception channel, which can use a variety of the vehicle's sensors individually or in combination with one another through data fusion. However, due to the aforementioned partial lack of suitability of the data from the environment model for visualization, the visualized data can exhibit a high temporal jitter or a spatial offset, each resulting in spatial instability of a displayed object relative to the environment. For the viewing driver, such behavior can be apparent as wobbling objects in the display and can be distracting.The quality of object detection can depend heavily on the object class (especially with data-driven approaches) and the respective environmental conditions. Therefore, it is advisable to use a filter chain (e.g., a box filter over multiple frames or a Kalman filter) to clean up the behavior of objects with regard to their position in the display. Removing interference, noise, and artifacts in the visualization using filter loops tends to be complex. This approach also results in the filter filtering all objects equally, regardless of sensor quality and environmental conditions – and this, in turn, can lead to over- or under-filtering of the objects to be displayed with regard to their position. Such a filter chain with suboptimal parameters can even cause objects to appear sluggish in the visualization.This effect is particularly evident when temporal filters with fixed window widths are applied to the data. This can sometimes result in temporal stability, but also a sluggish visualization. This creates a trade-off between artifact removal and realistic display on a display unit.

[0008] The object of the invention is to improve the filtering of information to be displayed about objects in their position in an environment model.

[0009] The invention is based on the features of the independent claims. Advantageous developments and refinements are the subject of the dependent claims.

[0010] A first aspect of the invention relates to a system for displaying information about objects in the surroundings of a vehicle, comprising a sensor unit for determining sensor data about the objects, a driver observation camera, a computing unit and a display unit, wherein the computing unit is designed to determine a current viewing direction of the driver using the data from the driver observation camera, to relate the respective viewing direction to a viewed target in the surroundings, and to check a course of the viewed targets to determine whether i) the viewing directions track an object in the surroundings during a predetermined period of time, and whether ii) the tracked object corresponds to an object detected by the sensor unit and to be displayed on the display unit during the predetermined period of time, and if i) and ii) are present,to determine a deviation between the course of the viewing directions directed at the tracked object and a positional course of the respective object determined by the sensor unit on the basis of the sensor data in a common projection image suitable for determining the deviation, and to adapt at least one filter parameter of a filter to be applied to the sensor data for displaying the tracked object in such a way that the positional course of the respective object to be displayed by the application of the filter on the display unit is changed in such a way that the deviation is reduced, and to subsequently control the display unit in such a way that the time-dependent position of the displayed tracked object on the display unit is determined by the adapted filter parameter.

[0011] The object of the invention is achieved in particular by observing the driver's eye movements and using them to combine and harmonize the information thus available with that from the vehicle's environment perception channel. In other words, the data obtained from the sensor unit for displaying dynamic and / or static objects in the vehicle's surroundings are supported by relevant information from the driver's eye movements and the resulting gaze directions.

[0012] In particular, this support is achieved by using one or more parameters of one or more filters, which in particular determine a dynamic bandwidth of the one or more filters, as an optimization variable to optimize the position of a gaze-tracked object for display. This is done, in particular, such that this position of a gaze-tracked object follows the gaze trajectory with respect to the surroundings of the vehicle and thus ideally the object itself. The fact that this tracking behavior results is attributable to the value of at least one filter parameter, which is optimized accordingly.

[0013] In principle, it is irrelevant whether a filter parameter of one or more filters is adjusted, whether one or more filters are used, or whether many filter parameters of one filter or several filters are adjusted. The only decisive factor is that the behavior of an applied filter chain is adjusted, as explained above and below. A Kalman filter, in particular, can be used as a filter. Examples of different filter parameters include a filter window and the filter function.

[0014] The driver observation camera directly records eye movements. From these eye movements, the gaze direction of each eye can be determined, which also determines the focus. In order to assign the driver's gaze direction to the surroundings, a data format compatible with the information from the sensor unit is required. This is achieved using the projection image, in which, in particular, the sensor unit's information about the surroundings is mapped onto this preferably two-dimensional projection image, and the gaze directions are assigned to corresponding coordinates in this projection image. The projection image thus creates a common basis for comparing the sensor unit's information about the surroundings and the gaze directions into the surroundings.

[0015] A so-called "range image" is preferably used as the projection image (this is essentially the result of projecting a 3D environment onto a 2D plane). This allows for direct geometric comparisons to be made regarding which paths the driver's line of sight is following in the environment, or which object moving relative to the rest of the environment they are following, and what information the sensor unit has regarding these paths or objects.

[0016] In particular, the computing unit uses the driver's gaze direction to determine whether the driver is following an object with their gaze over a specified period of time, i.e., whether they are focusing on it over the specified period of time and whether their gaze is following the object. The computing unit thus ensures that the driver is constantly observing and focusing on a moving object over a longer period of time. The check is carried out in particular by determining an area around a fixed window on the projection image, such as a "range image." If the area is repeatedly constant, it can be assumed that the same object-related position is in focus. From this, a spline of a moving object on the 2D plane of the projection image, such as a "range image," can be derived. A respective spline is then used as a reference spline to adjust at least one filter parameter.

[0017] Once these requirements are met, relevant information based on the driver's recorded eye movements is available, which can be used for further processing. Therefore, it is still being examined whether the object tracked by the driver's gaze was also detected with sufficient information from the sensor unit, and whether this object should be displayed on a display unit of the vehicle, in particular a central screen in the vehicle's interior, in such a way that the driver can create a spatial image of the surroundings based on the display unit using the environment model displayed on the display unit.

[0018] If these conditions are met, the process described above continues, comparing the geometric deviation of the tracked object's trajectory in the projection image based on the data from the driver observation camera with the position history from the sensor unit. The trajectory of an object moving relative to its surroundings in 3D space can be recorded by the sensor unit and adjusted using the filter. This updated filter chain is then preferentially projected into the projection image and compared with the reference spline. The resulting deviation can be defined, for example, as the orthogonal L2 distance.

[0019] The goal of the subsequent optimization is, in particular, to minimize this deviation. At least one filter parameter is adjusted so that the position curve according to the sensor unit adapts to the tracked trajectory based on the driver's gaze behavior. This results in an improved filter parameter value, and the object's position curve can be displayed more reliably and realistically on the display unit. Once the corresponding minimum or termination criterion is reached, the optimization is terminated, and the underlying filter set is used to filter the ADAS perception data for a corresponding period of time and make it available for visualization.

[0020] The object whose movement is subjected to the filter with the filter parameter to be changed for display on the screen is, in particular, a dynamic object, i.e., an object that is moving relative to the surrounding environment, such as another road user. Other road users in particular significantly determine the behavior of a driver assistance system, especially when it operates reactively, i.e., reacts to the surrounding traffic. It is precisely such reactions that are of interest to the driver of the vehicle, which is why the environment model, particularly the moving objects in the vehicle's surroundings, is displayed on the display unit.

[0021] An optimization loop can either be initialized at specified intervals or linked to the availability of the reference spline; the latter is particularly advantageous in rural environments. Linking it to a perception quality measure is also conceivable. Continuous optimization repetition is typically necessary because the quality of the sensor data can fluctuate significantly and may depend on the degree of contamination of the sensors, the prevailing lighting conditions, or weather conditions.

[0022] Advantageous effects of the invention are that, through the adaptive adjustment of the assistance visualization, objects in the surrounding area are displayed more naturally on the display unit, regardless of light and weather conditions. If moving objects are displayed in the vehicle's surroundings, they can also be displayed more effectively, independent of the static scenery. The described system and method can also adapt a variety of filter parameters during optimization and is independent of the specific sensor types used in the sensor unit, as well as whether they are fused with one another. Individual sensors of the sensor unit can be, for example, cameras, radar, or lidar.

[0023] According to an advantageous embodiment, the computing unit is designed to project data from the sensor unit into a two-dimensional projection image in order to determine the deviation, and to assign the viewing directions to the coordinates of the projection image, so that the course of the observed targets of the viewing directions in the projection image and the position course of a respective object in the projection image are related to the same standardized positions of the environment and thus a respective direct geometric difference can be determined to determine the deviation.

[0024] The geometric difference is preferably calculated as an L2 distance, particularly preferably as an integral valid for a respective point in time of the distances between the spline of the focused viewpoints on the tracked object and the trajectory of the object relative to the environment based on the sensor data, referred to the same reference projection image.

[0025] According to a further advantageous embodiment, the filter comprises at least one of the following: Kalman filter, box filter, low-pass filter, moving average filter.

[0026] According to a further advantageous embodiment, the computing unit is designed to classify a respective object which is detected by the sensor unit and, depending on the result of the classification, to determine and apply special filter parameters depending on the class for displaying the respective object on the display unit.

[0027] Examples of different classes into which objects moving in relation to their surroundings can be divided are: cyclists, passenger cars, trucks, pedestrians.

[0028] According to a further advantageous embodiment, the computing unit is designed to determine the predetermined period of time depending on the classification of the respective object and / or depending on the distance of the respective object from the vehicle.

[0029] According to a further advantageous embodiment, the system further comprises a communication module, wherein the computing unit is designed to transmit the at least one adapted filter parameter to a central computer or to another road user, in particular another road user in the same class as the vehicle, by means of the communication module.

[0030] The classification of your own vehicle can be carried out in the same way as the classification of other road users. In particular, the filter parameters determined through optimization can only be passed on to other passenger vehicles if your own vehicle is a passenger car.

[0031] According to a further advantageous embodiment, the computing unit is designed to check whether the viewing directions track an object in the environment during the predetermined period of time and to evaluate whether the targets of the viewing directions lie within a planar area around a respective object projected into the environment of the vehicle, at least during the predetermined period of time.

[0032] According to a further advantageous embodiment, the computing unit is designed to determine the size of the planar area depending on the distance of the respective object from the vehicle.

[0033] According to a further advantageous embodiment, the computing unit is designed to check whether the tracked object corresponds to an object detected by the sensor unit and to be displayed on the display unit during the predetermined period of time, and to carry out an image-analytical similarity comparison, in particular a determination of a "structural similarity index measure".

[0034] A further aspect of the invention relates to a vehicle with a system as described above and below. Advantages and preferred developments of the proposed vehicle arise from an analogous and analogous application of the statements made above in connection with the proposed system.

[0035] A further aspect of the invention relates to a method for displaying information about objects in the surroundings of a vehicle, comprising the steps of: - Determining sensor data about the objects by a sensor unit of the vehicle, and in each case by a computing unit:

[0036] - Determining the driver's current viewing direction using data from a driver observation camera on the vehicle;

[0037] - relating the respective viewing direction to a viewed target in the environment, and checking a course of the viewed targets to determine whether i) the viewing directions track an object in the environment during a specified period of time, and whether ii) the tracked object corresponds to an object detected by the sensor unit and to be displayed on the display unit during the specified period of time, and if i) and ii) are present:

[0038] - Determining a deviation between the course of the viewing directions directed at the tracked object and a position course of the respective object determined by the sensor unit on the basis of the sensor data in a common projection image suitable for determining the deviation;

[0039] - adjusting at least one filter parameter of a filter to be applied to the sensor data to display the tracked object in such a way that the position history of the respective object to be displayed on the display unit by applying the filter is changed in such a way that the deviation is reduced; and then

[0040] - Controlling the display unit such that the time-dependent position of the displayed tracked object on the display unit is determined by the adjusted filter parameter.

[0041] Advantages and preferred developments of the proposed method result from an analogous and analogous transfer of the statements made above in connection with the proposed system.

[0042] Further advantages, features, and details will become apparent from the following description, which – where appropriate with reference to the drawings – describes at least one embodiment in detail. Identical, similar, and / or functionally equivalent parts are provided with the same reference numerals.

[0043] They show:

[0044] Fig. 1 : A system in a vehicle for displaying information about objects in the surroundings of the vehicle according to an embodiment of the invention.

[0045] Fig. 2: A method for displaying information about objects in the surroundings of a vehicle according to an embodiment of the invention.

[0046] The representations in the figures are schematic and not to scale.

[0047] Fig. 1 shows a section of a vehicle with a system for displaying information about objects in the vehicle's surroundings. As part of the system, the vehicle has a sensor unit 1 for acquiring sensor data about the vehicle's surroundings. This sensor data therefore also includes corresponding information about moving objects, such as other road users, located in the detectable surroundings of the vehicle. The sensor data primarily serves to provide a driver assistance system or an automatic driving control system of the vehicle with current information about the surroundings and the road users present therein, so that it can execute appropriate reactions.To make the reactions more comprehensible for the vehicle driver, the data on the road users is displayed on a display unit 7 of the vehicle in such a way that real movements of the actual road users are mapped by movements of the representations of these road users on the display unit 7. Since the sensor data is designed with particular suitability for use in the driver assistance system or the automatic driving control system of the vehicle, but not necessarily for suitability for display on a display unit 7, one or more filters are used to adapt this sensor data with the information about the other road users and their temporal position history, so that a realistic and smooth position history of these movements of the other road users is achieved on the display unit 7.This at least one filter has at least one filter parameter, and the application of the filter as well as the control of the display unit 7 is handled by a computing unit 5 of the system, in particular but not necessarily located in the vehicle. Since a predetermined filter parameter cannot be optimal for all situations, the driver's behavior is advantageously used where this contains relevant information in order to be able to improve the display of a position history of another road user in the vicinity of the vehicle. For this purpose, the temporal course of the driver's gaze directions into the surroundings is recorded by means of a driver observation camera 3, and the gaze directions are mapped onto a projection image of the current surroundings, into which the information recorded by the sensor unit 1 is also mapped.The position progression, which is determined by the computing unit 5 by sweeping the viewing directions over the surroundings, and the position progression of another road user determined by the sensor unit 1, also depicted in the projection image, are compared by the computing unit 5 when the computing unit 5 has already determined that a focused viewpoint is held on another road user over a predetermined period of time, which is sketched in Fig. 1 as a black filled circle within a rectangular defined area around the other road user.The rectangular defined area is defined using the sensor data from sensor unit 1 as the area that moves physically with the other road user, and within which the focus of the driver's gaze must lie over a specified period of time so that the computing unit 5 can recognize that the driver is actually following the object for the specified period of time. If this is the case, usable information is available to exploit the driver's gaze tracking of the other road user in order to optimize the filter. The filter parameter is optimized such that the deviation of the movement data of the other road user calculated using the sensor data from sensor unit 1 corresponds as closely as possible to that calculated by tracking the other road user using the driver's gaze.The filter parameter is adjusted accordingly, and the resulting filter parameter is used to display the trajectory of the other road user on the display unit 7. Further details are given in the method of Fig. 2.

[0048] Fig. 2 shows a corresponding method for displaying information about objects in the surroundings of a vehicle, which can be implemented in a system as described in Fig. 1. In a first step of the method, sensor data about the objects is determined S1 by a sensor unit 1 of the vehicle for environmental monitoring. Cameras, radar, lidar, ultrasonic distance sensors, and other typical sensors can be used here. The computing unit 5 determines S2 the driver's current viewing direction using data from a driver observation camera 3 of the vehicle and relates S3 the respective viewing direction to a viewed target in the surroundings by providing a projection image in the form of a so-called "range image" and detecting the driver's viewing direction on the previously derived "range image."Furthermore, a check is performed on the course of the observed targets to determine whether i) the viewing directions track an object in the environment during a specified period of time, and whether ii) the tracked object corresponds to an object detected by the sensor unit 1 and displayed on the display unit 7 during the specified period of time. Check ii) is performed by determining a correspondence between the focused object by the driver and the object derived from the sensor data in 3D space using a "structural similarity index measurement," abbreviated "SSIM," which is compared with a specified threshold value. Congruence is checked using defined error metrics. For this purpose, the driver's focus is repeatedly determined in order to derive a reference spline of the object over several frames based on the data from the driver observation camera 3. The reference spline is derived from the focused "range images."Finally, a deviation between the reference spline is determined S4 based on the course of the viewing directions directed at the tracked object, with a position course of the respective object determined by the sensor unit 1 on the basis of the sensor data, with respective references to the projection image. Finally, the at least one filter parameter for displaying the road user in front of the driver viewed with his gaze is adjusted S5 with the filter to be applied to the sensor data such that the position course of the respective object, such as the road user from Fig. 1, to be displayed on the display unit 7 by applying the filter is changed such that the deviation is reduced. The display unit 7 is then controlled S6 such that the time-dependent position of the displayed tracked object is actually determined and displayed on the display unit 7 by the adjusted filter parameter.

[0049] Although the invention has been illustrated and explained in detail by preferred embodiments, the invention is not limited by the disclosed examples, and other variations may be derived therefrom by those skilled in the art without departing from the scope of the invention. It is therefore clear that a multitude of variations exist. It is also clear that exemplary embodiments are truly only examples and should not be construed as limiting the scope, possible applications, or configuration of the invention in any way.Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without departing from the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description.

Claims

Patent claims 1. A system for displaying information about objects in the surroundings of a vehicle, comprising a sensor unit (1) for determining sensor data about the objects, a driver observation camera (3), a computing unit (5), and a display unit (7), wherein the computing unit (5) is designed to determine a current viewing direction of the driver using the data from the driver observation camera (3), to relate the respective viewing direction to a viewed target in the surroundings, and to check a course of the viewed targets to determine whether i) the viewing directions track an object in the surroundings during a predetermined period of time, and whether ii) the tracked object corresponds to an object detected by the sensor unit (1) and to be displayed on the display unit (7) during the predetermined period of time, and if i) and ii) are present,to determine a deviation between the course of the viewing directions directed at the tracked object and a position course of the respective object determined by the sensor unit (1) on the basis of the sensor data in a common projection image suitable for determining the deviation, and to adapt at least one filter parameter of a filter to be applied to the sensor data for displaying the tracked object such that the position course of the respective object to be displayed on the display unit (7) by applying the filter is changed such that the deviation is reduced, and to subsequently control the display unit (7) such that the time-dependent position of the displayed tracked object on the display unit (7) is determined by the adapted filter parameter.

2. System according to claim 1, wherein the computing unit (5) is designed to project sensor data of the sensor unit (1) into a two-dimensional projection image in order to determine the deviation, and to assign the viewing directions to the coordinates of the projection image, so that the course of the observed targets of the viewing directions in the Projection image and the position progression of a respective object in the projection image are related to the same standardized positions in the environment and thus a respective direct geometric difference can be determined to determine the deviation.

3. System according to one of the preceding claims, wherein the filter comprises at least one of the following: Kalman filter, box filter, low-pass filter, moving average filter.

4. System according to one of the preceding claims, wherein the computing unit (5) is designed to classify a respective object which is detected by the sensor unit (1) and, depending on the result of the classification, to determine and apply special filter parameters depending on the class for displaying the respective object on the display unit (7).

5. System according to claim 4, wherein the computing unit (5) is designed to determine the predetermined period of time depending on the classification of the respective object and / or depending on the distance of the respective object from the vehicle.

6. System according to one of the preceding claims, further comprising a communication module, wherein the computing unit (5) is designed to transmit the at least one adapted filter parameter to a central computer or to another road user, in particular another road user in the same class as the vehicle with the system, by means of the communication module.

7. System according to one of the preceding claims, wherein the computing unit (5) is designed to check whether the viewing directions track an object in the environment during the predetermined period of time, and to evaluate whether the targets of the viewing directions lie within a planar area around a respective object projected into the environment of the vehicle, at least during the predetermined period of time.

8. System according to claim 7, wherein the computing unit (5) is designed to calculate the size of the areal area depending on the distance of the respective object from the vehicle.

9. System according to one of the preceding claims, wherein the computing unit (5) is designed to check whether the tracked object corresponds to an object detected by the sensor unit (1) and to be displayed on the display unit (7) during the predetermined period of time, and to carry out an image-analytical similarity comparison, in particular a determination of a "structural similarity index measure".

10. Vehicle with a system according to one of the preceding claims.

11. A method for displaying information about objects in the surroundings of a vehicle, comprising the steps: - Determining (S1) sensor data about the objects by a sensor unit (1) of the vehicle; and by a respective computing unit (5): - Determining (S2) a current viewing direction of the driver using the data of a driver observation camera (3) of the vehicle; - relating (S3) the respective viewing direction to a viewed target in the environment, and checking a course of the viewed targets to determine whether i) the viewing directions track an object in the environment during a predetermined period of time, and whether ii) the tracked object corresponds to an object detected by the sensor unit (1) and to be displayed on the display unit (7) during the predetermined period of time, and if i) and ii) are present: - Determining (S4) a deviation between the course of the viewing directions directed at the tracked object and a position course of the respective object determined by means of the sensor unit (1) on the basis of the sensor data in a common projection image suitable for determining the deviation; - adjusting (S5) at least one filter parameter of a filter to be applied to the sensor data to display the tracked object such that the position history of the respective object to be displayed on the display unit (7) by applying the filter is changed such that the deviation is reduced; and subsequently - controlling (S6) the display unit (7) such that the time-dependent position of the displayed tracked object on the display unit (7) is determined by the adapted filter parameter.

Images