Method for displaying an environment image from a virtual viewing angle, and a navigation assistance system therefor

Abstract

The invention relates to a method (6) for displaying an environment image from a virtual viewing angle in a vehicle by means of a navigation assistance system (1). In this case, the virtual viewing angle is adjusted situationally according to the driver's accommodation. The invention also relates to a navigation assistance system (1) for carrying out the method (6).

Description

[0001] Method for displaying an image of the surroundings from a virtual perspective and a navigation assistance system therefor

[0002] The invention relates to a method for displaying an image of the surroundings from a virtual viewing angle in a vehicle by means of a navigation assistance system according to the preamble of claim 1. The invention also relates to the navigation assistance system for carrying out the method.

[0003] A vehicle's navigation assistance system can, among other things, be designed to provide a driver with an image of the surroundings on a monitor. A virtual viewing angle of the surroundings image is usually constantly positioned behind the vehicle. In modern navigation assistance systems, for example, a zoom level of the viewing angle can be tied to the vehicle's speed. If the vehicle's speed increases, the image of the surroundings on the monitor is zoomed out accordingly. Similarly, the image of the surroundings on the monitor is zoomed in when the vehicle's speed decreases. However, this approach does not link the viewing angle to warning levels and, furthermore, does not address the driver's necessity or relevance on the system side. Alternatively, modern navigation assistance systems can, for example, implement a warning-based zoom level and a viewing angle correction.In this approach, the viewing angle can be adjusted depending on the warning. For example, if a BSM (Blind Spot Monitoring) warning is generated on the left, the viewing angle is concentrated or focused on the rear left area of ​​the vehicle. Unfortunately, this approach does not necessarily meet the driver's needs. If, for example, a BSM and / or AEB (Autonomous Emergency Braking) warning is present, the driver may already have recognized the situation and classified it as non-critical. Therefore, adjusting the viewing angle would not be effective in this case.

[0004] DE 10 2018201 631 A1 discloses a method for generating a virtual representation to expand the field of vision in a vehicle. An image of the surroundings is generated on a monitor depending on the driver's viewing direction. EP 1974998 A1 discloses a method for detecting an area that is hidden as a blind spot due to the presence of a vehicle pillar, using a blind spot camera mounted in the vehicle.

[0005] US 2013096820 A1 discloses a display system for a vehicle with monitors that display an image from external cameras according to the driver's line of sight.

[0006] DE 102009 054231 A1 discloses a head-up display for stereoscopic information display in a motor vehicle.

[0007] DE 10 2013 021 150 A1 discloses a method and an arrangement for displaying optical information in vehicles in the central field of vision of the driver in such a way that this display (virtual image) is superimposed on the real traffic environment.

[0008] WO 2014 / 130 049 A1 discloses a system for extending a rear view display in a vehicle.

[0009] US 2018 / 0 312 111 A1 discloses a collision avoidance system or a vision system or an imaging system for a vehicle that uses at least one camera to capture image data outside the vehicle and provide it to the driver in the interior of the vehicle.

[0010] The object of the invention is therefore to provide an improved or at least alternative embodiment for a method of the generic type, in which the described disadvantages are overcome. The object of the invention is also to provide a corresponding navigation assistance system.

[0011] This object is achieved according to the invention by the subject matter of the independent claims. Advantageous embodiments are the subject matter of the dependent claims.

[0012] The present invention is based on the general idea of ​​adjusting the virtual viewing angle situationally according to the driver's accommodation and accordingly taking the driver's needs into account in the adjustment.

[0013] The method according to the invention is designed or provided for displaying an image of the surroundings from a virtual perspective in a vehicle using a navigation assistance system. The navigation assistance system has a driver observation camera, a depth-sensing sensor, and a monitor for displaying the image of the surroundings. A calibration phase of the method is carried out first, followed by an online phase. In the calibration phase, images of the driver are first recorded using the driver observation camera. Accommodation states of a driver of the vehicle are then determined from the images of the driver for each direction of gaze of the driver. An image of the surroundings is then recorded using the depth-sensing sensor, and sensor depth values ​​are generated from this.A calibration function is then derived between the generated sensor depth values ​​and the determined accommodation states of the driver for each of the driver's gaze directions. In other words, a relationship is determined between the driver's accommodation states and the sensor depth values ​​for each of the driver's gaze directions. In the online phase of the process, current images of the driver are first recorded using the driver observation camera. The current accommodation states of the driver are then determined from the current images for each of the driver's gaze directions. Current accommodation-specific sensor depth values ​​or accommodation depth values ​​are then determined from the calibration function. A current image of the surroundings is then recorded using the depth-sensing sensors, and current sensor depth values ​​are generated from this.The deviation between the current sensor depth values ​​and the current accommodation-specific sensor depth values ​​or the accommodation depth values ​​is then determined for each direction of the driver's gaze. The virtual viewing angle is then focused on an area with the greatest deviation when displaying the surrounding image.

[0014] In the method according to the invention, the virtual viewing angle is adaptively or situationally adjusted to the driver's needs when displaying the surroundings image. In other words, the virtual viewing angle of a virtual camera showing the surroundings image is adjusted situationally. This check is carried out to determine whether the driver's perception matches the perception of the sensors. If a deviation or delta occurs, the viewing angle is virtually focused or adjusted to the position of this deviation. This makes it possible, in particular, to take into account deviations in perception that can occur, for example, when driving in heavy spray, in fog, when driving at night, when the windows are very dirty, etc. To carry out the method, no changes to the hardware of the existing navigation assistance system are necessary. The method can also use FOSS DPI SW.The method according to the invention can, in particular, increase driver comfort through adapted visualization of the surroundings. Furthermore, the driver's confidence in the navigation assistance system can be increased.

[0015] The calibration phase serves to derive the calibration function between the sensor depths or sensor depth values ​​and the driver's accommodation states. Accommodation describes the eye's ability to adapt the refractive power of the lens by changing its shape or curvature, and thus to focus on objects at different distances. This change can be detected in the images recorded by the driver observation camera, and the corresponding accommodation state of the driver or the driver's eye can be determined. The calibration function can then establish the relationship between the driver's accommodation states and the sensor depth values. From this, the corresponding accommodation-specific sensor depth values ​​or accommodation depth values ​​can be read from the calibration function for each accommodation state of the driver, depending on the driver's gaze direction.The online phase is used to determine the divergence between the sensor depth values ​​and the accommodation depth values ​​or the accommodation-specific sensor depth values, which are calculated based on the calibration function determined in the calibration phase. The online phase is also used to adapt or correct the virtual viewing angle when displaying the surrounding image.

[0016] When executing the procedure, it is assumed that the individual components of the vehicle and / or the individual components of the navigation assistance system are extrinsically and intrinsically calibrated to each other.

[0017] When capturing the environment image, the environment image can be captured three-dimensionally and / or as a 3D environment model. When capturing the environment image, the environment image can be captured in a reference system or coordinate system of the depth-perceiving sensor. Before deriving the calibration function, the environment image can then be converted into a reference system or coordinate system.

[0018] The coordinate system of the driver observation camera can be transformed. This allows the sensor depth values ​​and / or the accommodation states and / or the accommodation-specific sensor depth values ​​and / or the viewing direction to be calculated in a common reference system or coordinate system. When deriving the calibration function, a driver's well-fill factor and / or an environmental factor can be taken into account. In other words, the calibration function can be extended for any parameters. The calibration function can therefore be two-dimensional or multi-dimensional. The driver's well-fill factor can be, for example, their fatigue state, and the environmental factor can be, for example, the brightness of the surrounding image. By taking additional parameters into account, the accuracy of the calibration function can be increased.

[0019] When determining the deviation, the deviation can be calculated as the difference values ​​between the individual current sensor depth values ​​and the individual accommodation-specific sensor depth values. The respective difference value can, for example, be calculated between an absolute value of the current sensor depth value and an absolute value of the accommodation-specific sensor depth value. The difference values ​​can then be projected onto a projection plane according to their spatial assignment in the environment image. The projection plane can be a two-dimensional plane or a voxel mesh plane. This creates a projection plane in which the difference values, or deltas, between the current sensor depth values ​​and the accommodation-specific sensor depth values ​​are mapped.

[0020] Assuming that the corresponding difference values ​​or deltas are correspondingly large when driving in heavy spray, fog, night driving, or when the windows are very dirty, etc., the position of the greatest deviation in the driver's perception and the sensor perception can be determined based on the difference values ​​on the projection plane. From the projected difference values, an area in the projection plane in which a maximum number of positive and / or negative difference values ​​are located can be determined. This area can then be defined as the area with the maximum deviation for adjusting the virtual viewing angle mentioned above. The virtual viewing angle can then be focused on this area when displaying the surrounding image.In other words, a virtual camera is positioned so that a maximum of an area with the greatest difference is displayed on the monitor. If, for example, the driver only sees the close-up area in front of the vehicle with a reduced visibility when driving through spray, it can be assumed that the close-up area shows a correspondingly large difference. The virtual camera is therefore focused on this area with the virtual viewing angle. This allows the driver to be shown exactly the areas in which their visibility is impaired and the assistance can be adapted to the driver's needs on a situation-by-situation basis. The calibration phase and / or the online phase can be carried out periodically at a predefined interval. This allows the assistance to be adapted to the driver's needs in real time and on a situation-by-situation basis.

[0021] The invention also relates to a navigation assistance system for a vehicle for carrying out the method described above. The navigation assistance system is, in particular, a so-called Navistance. The navigation assistance system offers a combined visualization of assistance and navigation content in an instrument cluster or IC (IC: Integrated Circuit). The navigation assistance system can have a driver observation camera for recording images of the driver. The driver observation camera can be mounted or arranged inside the vehicle and directly in front of the driver. The navigation assistance system can also have depth-sensing sensors for recording an image of the vehicle's surroundings. The depth-sensing sensors can be a component of an ADAS (ADAS: Advanced Driver Assistance System).The depth-sensing sensor system can, for example, comprise a vision sensor and / or radar sensor and / or a fusion sensor. The navigation assistance system can also comprise a monitor for displaying the recorded image of the vehicle's surroundings. As already described above, the viewing angle of the surroundings image can be adjusted on the monitor. The navigation assistance system can also comprise a control unit configured to execute the method described above.

[0022] Further important features and advantages of the invention emerge from the subclaims, from the drawings and from the associated description of the figures based on the drawings.

[0023] It is understood that the features mentioned above and those to be explained below can be used not only in the combination specified in each case, but also in other combinations or on their own, without departing from the scope of the present invention.

[0024] Preferred embodiments of the invention are illustrated in the drawings and are explained in more detail in the following description, wherein the same reference numerals refer to the same or similar or functionally identical components.

[0025] The following show, each schematically: Fig. 1 shows a view of a navigation assistance system according to the invention;

[0026] Fig. 2 is a flow chart of a method according to the invention;

[0027] Fig. 3 is a view of a 3D environment image recorded by the depth-sensing sensor in the method according to the invention;

[0028] Fig. 4 shows a visualization of sensor depth values ​​in the method according to the invention;

[0029] Fig. 5 shows a visualization of accommodation-specific sensor depth values ​​in the method according to the invention;

[0030] Fig. 6 shows a visualization of difference values ​​between the sensor depth values ​​and the accommodation-specific sensor depth values ​​in the method according to the invention;

[0031] Fig. 7 is a view of a projection plane with the difference values ​​in the method according to the invention;

[0032] Fig. 8 shows a visualization of an adjustment of a virtual viewing angle in the method according to the invention;

[0033] Fig. 9 shows a visualization of the rendering and display of the 3D environment image with the adjusted viewing angle in the method according to the invention;

[0034] Fig. 10 is a view of a calibration function in the method according to the invention.

[0035] Fig. 1 shows a view of a navigation assistance system 1 according to the invention for a vehicle. The navigation assistance system 1 comprises a driver observation camera 2, a depth-sensing sensor 3, a monitor 4 for displaying the surroundings image, and a control unit 5. The navigation assistance system 1 is designed to execute a method 6 according to the invention.

[0036] Fig. 2 shows a flow chart of a method 6 according to the invention for displaying the surroundings image from a virtual viewing angle in a vehicle by means of the navigation assistance system 1. In the embodiment of the method 6 shown here, steps VS1 to VS11 are carried out successively.

[0037] In method 6, a calibration phase KPH is first performed. The calibration phase KPH serves to derive a calibration function or a transfer function between sensor-generated depth values ​​or sensor depth values ​​and the driver's accommodation states. The calibration phase KPH comprises steps VS1 to VS6. In step VS1, an image is acquired using the driver observation camera 2 of the navigation assistance system 1. In step VS2, the driver's accommodation state is determined based on the images recorded in step VS2. In step VS3, the driver's line of sight is determined using the driver observation camera 2 of the navigation assistance system 1 in a reference system or coordinate system of the driver observation camera 2. In step VS4, a 3D environment image or a 3D environment model is captured using the depth-sensing sensor system 3 or an ADAS object sensor system. Fig.3 shows a view of the 3D environment image recorded by the depth-sensing sensor 3. In step VS5, the 3D environment image or 3D environment model is transformed into the reference system or coordinate system of the driver observation camera 2. In step VS6, a calibration function or a transfer function between sensor depth values ​​and accommodation states is established. An accommodation-specific sensor depth value or accommodation depth value assigned to a specific accommodation state can then be read from the calibration function. Fig. 10 shows the calibration function, which describes a relationship between the driver's accommodation state and the driver depth information or accommodation depth values ​​and the sensor depth information or sensor depth values. With step VS6, the calibration phase KPH is completed.

[0038] The online phase (OPH) is then performed. The online phase (OPH) determines the divergence, or difference, between the sensor depth values ​​derived from the data from sensor 3 and the accommodation depth values ​​generated from the calibration function. The online phase (OPH) comprises steps VS7 to VS10.

[0039] In step VS7, the driver's line of sight is determined online. In step VS8, a depth calibration is performed. Difference values ​​or deltas between sensor depth values ​​and the accommodation depth values ​​are calculated using the calibration function. For this purpose, the current sensor depth values ​​of the current surroundings image and the driver's current accommodation states can first be determined. Fig. 4 shows a visualization of the sensor depth values, and Fig. 5 shows a visualization of the accommodation-specific sensor depth values ​​or accommodation depth values. Fig. 6 shows a visualization of difference values ​​between the sensor depth values ​​and the accommodation-specific sensor depth values ​​or accommodation depth values. In step VS9, a 2D projection of the difference values ​​or deltas from step VS8 is projected onto a projection plane PE according to their local position.A view of the projection plane PE with the difference values ​​from step VS8 is shown in Fig. 7. Here, the difference values ​​deviating from zero are highlighted in gray. In step VS10, a virtual camera adjustment is performed, thus adjusting the virtual viewing angle of the surrounding image. Fig. 8 shows a visualization of an adjustment of the virtual viewing angle. Step VS10 concludes the online phase OPH.

[0040] In step VS11, a rendering of the navigation assistance system 1, or a so-called navi-stance, is performed. Subsequently, the surroundings image is displayed on monitor 4 according to the updated virtual camera position or the adjusted virtual viewing angle. Fig. 9 shows a visualization of the rendering and the display of the adjusted 3D surroundings image.

Claims

Patent claims 1. Method (6) for displaying an image of the surroundings from a virtual perspective in a vehicle by means of a navigation assistance system (1), wherein the navigation assistance system (1) has a driver observation camera (2), a depth-sensing sensor (3), a monitor (4) for displaying the image of the surroundings, wherein in a calibration phase (KPH) of the method (6): - accommodation states of a driver of the vehicle are determined from images taken by the driver observation camera (2) for each direction of view of the driver, - an image of the surroundings is recorded using the depth-sensing sensor (3) and sensor depth values ​​are generated from this, - a calibration function is derived between the generated sensor depth values ​​and the determined accommodation states of the driver for each gaze direction of the driver, whereby in an online phase (OPH) of the method (6) following the calibration phase (KPH): - current accommodation states of the driver are determined from current images taken by the driver observation camera (2) for each direction of view of the driver and from this, current accommodation-specific sensor depth values ​​are determined based on the calibration function, - a current image of the surroundings is recorded using the depth-sensing sensor (3) and current sensor depth values ​​are generated from this, - a deviation between the current sensor depth values ​​and the current accommodation-specific sensor depth values ​​is determined for each direction of the driver's gaze, - the virtual viewing angle is focused on an area with the greatest deviation when displaying the surrounding image.

2. Method (6) according to claim 1, characterized in that When taking the surrounding image, the surrounding image is taken in three dimensions.

3. Method (6) according to claim 1 or 2, characterized in that when recording the environment image, the environment image is recorded as a 3D environment model.

4. Method (6) according to one of the preceding claims, characterized in that - when recording the image of the surroundings, the image of the surroundings is recorded in a reference system of the depth-perceiving sensor system (3), and - that before deriving the calibration function, the surroundings image is transformed into a reference system of the driver observation camera (2).

5. Method (6) according to one of the preceding claims, characterized in that a driver comfort factor and / or an environmental factor are taken into account when deriving the calibration function.

6. Method (6) according to one of the preceding claims, characterized in that - when determining the deviation, the deviation is calculated as the difference values ​​between the individual current sensor depth values ​​and the individual accommodation-specific sensor depth values, and - that the difference values ​​are projected onto a projection plane (PE), preferably a 2D plane or a voxel mesh plane, according to their spatial assignment in the environment image.

7. Method (6) according to claim 6, characterized in that the difference value between an amount of the current sensor depth value and an amount of the accommodation-specific sensor depth value is calculated.

8. Method (6) according to claim 6 or 7, characterized in that - from the projected difference values, an area in the projection plane (PE) in which a maximum number of positive and / or negative difference values ​​is arranged is determined, and - that this area is set as the area with the maximum deviation for adjusting the virtual viewing angle.

9. Method (6) according to one of the preceding claims, characterized in that the calibration phase (KPH) and / or the online phase (OPH) are carried out periodically at a predefined time interval.

10. Navigation assistance system (1) for a vehicle for carrying out the method (6) according to one of the preceding claims, - wherein the navigation assistance system (1) has a driver observation camera (2) for recording images of the driver, - wherein the navigation assistance system (1) has a depth-sensing sensor system (3) for recording an image of the surroundings of the vehicle, - wherein the navigation assistance system (1) has a monitor (4) for displaying the recorded image of the surroundings of the vehicle, and - wherein the navigation assistance system (1) has a control unit (5) which is designed such that it carries out the method (6) according to one of the preceding claims.

Images