Method for displaying an image of the surrounding environment from a virtual perspective and navigation assistance system therefor

Abstract

This invention relates to a method (6) for displaying images of the surrounding environment from a virtual perspective in a vehicle using a navigation assistance system (1). The virtual perspective is contextualized based on the driver's visual focus. The invention also relates to a navigation assistance system (1) for implementing the method (6).

Description

[0001] This invention relates to a method for displaying images of the surrounding environment from a virtual perspective in a vehicle using a navigation assistance system, as described in the preamble of claim 1. The invention also relates to a navigation assistance system for implementing this method.

[0002] Additionally, a vehicle's navigation assistance system can be designed to provide the driver with an image of the surrounding environment on a monitor. Here, the virtual view of the surrounding environment is typically positioned constantly behind the vehicle. In modern navigation assistance systems, the zoom level of the view can be, for example, tied to the vehicle's speed. As the vehicle's speed increases, the image of the surrounding environment on the monitor subsequently zooms out accordingly. Similarly, as the vehicle's speed decreases, the image of the surrounding environment on the monitor zooms in. However, this approach does not establish a link between the view and the warning level, and furthermore, it does not cover driver needs or the relevance to the driver at the system level. Alternatively, in modern navigation assistance systems, for example, warning-based zoom level adjustments and view corrections can be performed. In this approach, the view can be adjusted based on the warning. If, for example, a left-side BSM (Blind Spot Detection) warning is generated, the view is focused or concentrated on the left rear area of ​​the vehicle. Disadvantageously, this approach does not necessarily cover the driver's needs. For example, if a BSM and / or AEB (Autonomous Emergency Braking) warning is present, the situation may have already been identified by the driver and assessed as non-serious. Therefore, adjusting the perspective here is not targeted.

[0003] DE 10 2018 201 631 A1 discloses a method for generating a virtual display to expand the field of view in a vehicle. Here, an image of the surrounding environment is generated on a monitor based on the driver's line of sight.

[0004] EP 1974998 A1 discloses a method for detecting areas that are obscured as blind spots due to the presence of vehicle pillars using a blind spot camera installed in a vehicle.

[0005] US 2013096820 A1 discloses a display system for a vehicle that has a monitor that displays images from an external camera according to the driver's line of sight.

[0006] DE 10 2009 054 231 A1 discloses a head-up display for displaying stereoscopic information in motor vehicles.

[0007] DE 10 2013 021 150 A1 discloses a method and components for displaying optical information in the driver's central field of vision in a vehicle, such that the display (virtual image) is superimposed on the real traffic environment.

[0008] WO 2014 / 130 049 A1 discloses a system for extending the reversing display in a vehicle.

[0009] US 2018 / 0 312 111 A1 discloses a collision avoidance system, vision system, or imaging system for a vehicle, which uses at least one camera to acquire image data of the exterior of the vehicle and provide it to the driver inside the vehicle.

[0010] Therefore, the object of the present invention is to provide an improved or at least alternative implementation of the same type of method, which overcomes the aforementioned disadvantages. Another object of the present invention is to provide a corresponding navigation assistance system.

[0011] This objective is achieved according to the invention through the subject matter of the independent claim. Advantageous embodiments are the subject matter of the dependent claims.

[0012] The overall concept of this invention is to contextualize the virtual perspective based on the driver's visual focus and to take the driver's needs into account in the setting.

[0013] The method according to the invention is designed or configured to display images of the surrounding environment from a virtual perspective in a vehicle using a navigation assistance system. Here, the navigation assistance system includes a driver observation camera, a depth sensing device, and a monitor for displaying images of the surrounding environment. The method first performs a calibration phase, followed by an online phase. In the calibration phase, an image of the driver is first captured using the driver observation camera. Then, the driver's visual focus state is determined from the driver's image for each gaze direction. Then, an image of the surrounding environment is captured using the depth sensing device, thereby generating a sensor depth value. Subsequently, for each gaze direction of the driver, a calibration function is derived between the generated sensor depth value and the driver's determined visual focus state. In other words, for each gaze direction of the driver, the correlation between the driver's visual focus state and the sensor depth value is determined. In the online phase of the method, a current image of the driver is first captured using the driver observation camera. Then, the driver's current visual focus state is determined from the current image for each gaze direction. Then, the sensor depth value or visual focus depth value currently associated with visual focus is determined by the calibration function. Now, an image of the current surrounding environment is captured using a depth-sensing device, generating a current sensor depth value. Then, for each direction of the driver's gaze, the deviation between the current sensor depth value and the current sensor depth value or visual focus depth value is determined. Subsequently, when displaying the image of the surrounding environment, the virtual viewpoint is focused on the area with the largest deviation.

[0014] In the method according to the invention, when displaying an image of the surrounding environment, the virtual viewing angle is adaptively and / or contextually adjusted according to the driver's needs. In other words, the virtual viewing angle of the virtual camera displaying the image of the surrounding environment is contextually adjusted. Here, it is checked whether the driver's perception is consistent with the perception of the sensor device. If a deviation or increment occurs in this case, the viewing angle is virtually focused or set to the position of the deviation. Thus, perceptual deviations that may occur in environments with splashing water, driving in fog, driving at night, or with severely dirty windows can be taken into account. To implement this method, no changes to the hardware of existing navigation assistance systems are required. Furthermore, the method can also invoke FOSS DPI SW. By means of the method according to the invention, driver comfort can be improved, in particular, through the adjusted visualization of the surrounding environment. In addition, the driver's trust in the navigation assistance system can be increased.

[0015] The calibration phase is used to derive a calibration function relating sensor depth or sensor depth values ​​to the driver's visual focus state. Visual focus describes the eye's ability to adjust the refractive power of the lens by changing its shape or curvature, thereby focusing on objects at different distances. This change can be identified in images captured by a camera viewed by the driver, and the corresponding visual focus state of the driver or the driver's eyes can be determined. The calibration function establishes a correlation between the driver's visual focus state and sensor depth values. Thus, for each visual focus state of the driver, depending on the driver's gaze direction, the corresponding sensor depth value or visual focus depth value related to visual focus can be read from the calibration function. The online phase is used to determine the divergence between sensor depth values ​​and visual focus depth values ​​or related sensor depth values, calculated based on the calibration function determined in the calibration phase. Furthermore, the online phase is also used to contextualize or correct the virtual viewpoint when displaying images of the surrounding environment.

[0016] When implementing this method, the premise is that the various components of the vehicle and / or the various components of the navigation assistance system are correspondingly calibrated with each other's extrinsic and intrinsic parameters.

[0017] When capturing images of the surrounding environment, these images can be captured in three dimensions and / or captured as 3D models of the surrounding environment. Here, the images can be captured within the reference frame or coordinate system of the depth sensing device. Before deriving the calibration function, the surrounding environment images can be transformed into the reference frame or coordinate system of the driver's viewing camera. Therefore, sensor depth values ​​and / or visual focus states and / or sensor depth values ​​and / or gaze direction related to visual focus can be calculated in a common reference frame or coordinate system.

[0018] When deriving the calibration function, driver comfort factors and / or environmental factors can be considered. In other words, the calibration function can be extended for any parameter. Therefore, the calibration function can be two-dimensional or multi-dimensional. Driver comfort factors can be, for example, their fatigue state, and environmental factors can be, for example, the brightness of the surrounding environment. By considering other parameters, the accuracy of the calibration function can be improved.

[0019] When determining the deviation, it can be calculated as the difference between each current sensor depth value and each sensor depth value related to visual focus. For example, the absolute value of the current sensor depth value and the absolute value of the sensor depth value related to visual focus can be calculated. This difference can then be projected onto a projection plane based on its positional correspondence in the surrounding environment image. Here, the projection plane can be a two-dimensional plane or a voxel grid plane. This results in a projection plane in which the difference or increment between the current sensor depth value and the sensor depth value related to visual focus is mapped.

[0020] Assuming driving conditions such as splashing water, foggy weather, nighttime driving, or severely dirty windows, where the corresponding differences or increments are relatively large, the location with the strongest deviation between the driver's perception and the sensor's perception can be determined based on the differences on the projection plane. To this end, a region can be defined on the projection plane based on the projected differences, with the maximum number of positive and / or negative differences distributed within that region. This region can then be identified as the area with the largest deviation for setting the virtual viewing angle, as mentioned above. Correspondingly, when displaying an image of the surrounding environment, the virtual viewing angle can be focused on this region. In other words, the virtual camera is positioned so that the maximum value of the region with the highest difference is displayed on the monitor. For example, when the vehicle is driving in a splashing water environment, if the driver's vision is only impaired in the immediate vicinity in front of the vehicle, it can be assumed that this immediate vicinity has a correspondingly high difference. Correspondingly, the virtual camera focuses on this region using the virtual viewing angle. Thus, the areas where the driver's vision is impaired can be accurately displayed to the driver, and the assistance functions can be contextually adjusted according to the driver's needs.

[0021] Here, the calibration phase and / or online phase can be performed periodically at predefined time intervals. This allows for real-time contextual adjustments to the driver assistance functions based on the driver's needs.

[0022] The present invention also relates to a navigation assistance system for a vehicle for implementing the methods described above. This navigation assistance system is particularly a so-called navigation assistance device. In this case, the navigation assistance system simultaneously displays assistance functions and navigation content on the instrument panel or IC (Instrument Cluster). The navigation assistance system may have a driver observation camera for capturing images of the driver. The driver observation camera may be mounted or positioned inside the vehicle and directly in front of the driver. The navigation assistance system may also have a depth sensing device for capturing images of the vehicle's surrounding environment. The depth sensing device may be a component of ADAS (Advanced Driver Assistance Systems). The depth sensing device may, for example, have a visual sensor and / or a radar sensor and / or a fusion sensor. The navigation assistance system may also have a monitor for displaying the captured images of the vehicle's surrounding environment. As described above, the viewing angle of the surrounding environment images can be adjusted at the monitor. The navigation assistance system may also have a control unit designed to perform the methods described above.

[0023] Other important features and advantages of the invention are described in the dependent claims, the drawings, and related figures based on the drawings.

[0024] It should be understood that the above features and the features to be explained below can be used not only in their respective specified combinations, but also in other combinations or individually, without departing from the scope of the invention.

[0025] Preferred embodiments of the present invention are shown in the accompanying drawings and described in more detail in the following description, wherein the same reference numerals refer to the same or similar or functionally identical components.

[0026] Here: Figure 1 A view of the navigation assistance system according to the present invention is shown; Figure 2 A flowchart of the method according to the present invention is shown; Figure 3 A view showing a 3D image of the surrounding environment captured by a depth-sensing device in accordance with the method according to the invention; Figure 4 This illustrates a visualization of sensor depth values ​​in the method according to the invention; Figure 5 A visualization of sensor depth values ​​related to visual focusing is shown in the method according to the invention. Figure 6A visualization of the difference between sensor depth values ​​and sensor depth values ​​related to visual focusing is shown in the method according to the invention. Figure 7 A view showing the projection plane having a difference in the method according to the invention; Figure 8 A visualization of virtual perspective adjustment in the method according to the invention is shown; Figure 9 A visualization is shown of rendering and displaying a 3D surrounding environment image using an adjusted viewpoint in the method according to the invention; Figure 10 A view showing the calibration function in the method according to the invention.

[0027] Figure 1 A view of a vehicle navigation assistance system 1 according to the present invention is shown. The navigation assistance system 1 includes a driver observation camera 2, a depth sensing device 3, a monitor 4 for displaying images of the surrounding environment, and a control unit 5. The navigation assistance system 1 is designed to implement the method 6 according to the present invention.

[0028] Figure 2 A flowchart of a method 6 according to the present invention for displaying images of the surrounding environment from a virtual perspective in a vehicle using a navigation assistance system 1 is shown. In the embodiment of method 6 shown here, steps VS1 to VS11 are performed sequentially.

[0029] In method 6, the calibration phase KPH is performed first. The calibration phase KPH is used to derive a calibration function or transformation function between the depth value generated by the sensor or the sensor depth value and the driver's visual focus state. The calibration phase KPH here includes steps VS1 to VS6.

[0030] In step VS1, image acquisition is performed using the driver observation camera 2 of the navigation assistance system 1. In step VS2, the driver's visual focus state is determined based on the image captured in step VS2. In step VS3, the driver's line of sight is determined in the reference frame or coordinate system of the driver observation camera 2 of the navigation assistance system 1. In step VS4, a 3D image or model of the surrounding environment is acquired using the depth sensing device 3 or the ADAS object sensing device. Figure 3This image shows a 3D surrounding environment image captured by the depth sensing device 3. In step VS5, the 3D surrounding environment image or 3D surrounding environment model is transformed into the reference frame or coordinate system of the driver's observation camera 2. In step VS6, a calibration function or transformation function is established between the sensor depth value and the visual focus state. From the calibration function, the sensor depth value or visual focus depth value related to a specific visual focus state can be read. Figure 10 The calibration function is shown, which describes the relationship between the driver's visual focus state and driver depth information or visual focus depth value, as well as sensor depth information or sensor depth value. The calibration phase KPH ends with step VS6.

[0031] Then, the online phase OPH is executed. The online phase OPH is used to determine the divergence or difference between the sensor depth value derived from the data of sensing device 3 and the visual focusing depth value generated by the calibration function. The online phase OPH here includes steps VS7 to VS10.

[0032] In step VS7, the driver's gaze direction is determined online. In step VS8, depth comparison is performed. Here, the difference or increment between the sensor depth value and the visual focus depth value is calculated according to a calibration function. For this purpose, the current sensor depth value of the current surrounding environment image and the driver's current visual focus state can first be determined. Figure 4 Visualization of sensor depth values, Figure 5 This displays a visualization of sensor depth values ​​or visual focus depth values ​​related to visual focusing. Figure 6 The diagram illustrates the visualization of the difference between the sensor depth value and the sensor depth value or visual focus depth value associated with visual focusing. In step VS9, the difference or increment from step VS8 is projected onto the projection plane PE according to its respective spatial location, forming a 2D projection. Figure 7 The image shows a view of the projection plane PE, which has the difference from step VS8. Here, differences that are not equal to zero are represented by a gray background. In step VS10, the virtual camera is adjusted, and thereby the virtual viewpoint of the surrounding environment image is adjusted. Figure 8 A visualization of the virtual viewpoint adjustment is shown. The online phase OPH ends with step VS10.

[0033] In step VS11, the navigation assistance system 1, or so-called navigation assistance device, is now rendered. Subsequently, an image of the surrounding environment is displayed on the monitor 4 according to the updated virtual camera position or adjusted virtual perspective. Figure 9 This demonstrates a visualization of the rendering and display of the adjusted 3D surrounding environment image.

Claims

1. A method (6) for displaying images of the surrounding environment from a virtual perspective in a vehicle using a navigation assistance system (1), wherein the navigation assistance system (1) has a driver observation camera (2), a depth sensing device (3), and a monitor (4) for displaying the images of the surrounding environment. In the calibration phase (KPH) of method (6): - Determine the driver's visual focus state from the images captured by the driver's observation camera (2) for each direction of the driver's gaze in the vehicle. - The surrounding environment is captured using the depth sensing device (3), and a sensor depth value is generated therefrom. - For each gaze direction of the driver, a calibration function is derived between the generated sensor depth value and the driver's determined visual focus state. In the online phase (OPH) following the calibration phase (KPH) of method (6): - Determine the driver's current visual focus state for each direction of the driver's gaze from the current image captured by the driver's observation camera (2), and thereby determine the current sensor depth value related to visual focus based on the calibration function. - The current surrounding environment image is captured using the depth sensing device (3), and the current sensor depth value is generated therefrom. - For each direction of the driver's gaze, determine the deviation between the current sensor depth value and the current sensor depth value related to visual focus. - When displaying the image of the surrounding environment, focus the virtual viewpoint on the area with the greatest deviation.

2. The method (6) according to claim 1. Its features are, When capturing images of the surrounding environment, the images are captured in three dimensions.

3. The method (6) according to claim 1 or 2. Its features are, When capturing the surrounding environment image, the surrounding environment image is captured as a 3D surrounding environment model.

4. The method (6) according to any one of the preceding claims. Its features are, - When capturing the image of the surrounding environment, the image of the surrounding environment is captured in the reference frame of the depth sensing device (3), and - Before deriving the calibration function, the surrounding environment image is transformed into the reference frame of the driver observation camera (2).

5. The method (6) according to any one of the preceding claims. Its features are, When deriving the calibration function, the driver's comfort factors and / or environmental factors are taken into account.

6. The method (6) according to any one of the preceding claims. Its features are, - In determining the deviation, the deviation is calculated as the difference between each current sensor depth value and each sensor depth value related to visual focusing, and - The difference is projected onto a projection plane (PE) according to its positional correspondence in the surrounding environment image, preferably a 2D plane or a voxel mesh plane.

7. The method (6) according to claim 6. Its features are, Calculate the difference between the absolute value of the current sensor depth value and the absolute value of the sensor depth value related to visual focusing.

8. The method (6) according to claim 6 or 7. Its features are, - A region is defined in the projection plane (PE) by the projected differences, in which the maximum number of positive and / or negative difference values ​​are arranged, and - The region is determined as the region with the maximum deviation for setting the virtual viewpoint.

9. The method (6) according to any one of the preceding claims. Its features are, The calibration phase (KPH) and / or the online phase (OPH) are performed periodically at predefined time intervals.

10. A navigation assistance system (1) for a vehicle for implementing the method (6) according to any one of the preceding claims. - The navigation assistance system (1) therein has a driver observation camera (2) for taking images of the driver. - The navigation assistance system (1) wherein the navigation assistance system (1) has a depth sensing device (3) for capturing images of the vehicle’s surrounding environment. - The navigation assistance system (1) wherein the navigation assistance system (1) has a monitor (4) for displaying images of the vehicle's surrounding environment taken by camera, and - The navigation assistance system (1) therein has a control unit (5) which is designed to perform the method (6) according to any one of the preceding claims.

Images