Methods for visualizing a vehicle environment and vehicle

Abstract

The invention relates to a method for visualizing a vehicle environment using a surround-view system, wherein parts of the vehicle (1) are displayed as semi-transparent in order to avoid obscuring objects in the vehicle environment from a driver's perspective. The invention is characterized in that a corneal reflection image (CRI) is detected by means of a driver observation camera (7) based on two-dimensional images, wherein in an image of the corneal reflection image (CRI) a vehicle interior is segmented as the foreground and the part of the vehicle environment recognizable from the driver's perspective is segmented as the background, wherein a surround-view visualization of the surround-view system is generated depending on a predetermined driver viewing direction, wherein the foreground segmented from the image of the corneal reflection image (CRI) is projected semi-transparently into the surround-view visualization and displayed together with it.

Description

[0001] The invention relates to a method for visualizing a vehicle's surroundings using a surround-view system of the type defined in more detail in the preamble of claim 1. The invention also relates to a vehicle equipped for carrying out the method.

[0002] Fundamentally, it is a known technology, based on the state of the art, to use so-called surround-view systems to determine a complete vehicle environment based on images from various cameras and to visualize this together with a graphic representation of the vehicle. This primarily serves to improve vehicle control when maneuvering, e.g., when parking or when driving into narrow driveways, around tight curves, etc.

[0003] However, correspondence problems are increasingly arising. This describes the difficulty the driver has in determining the exact position of objects, etc., from the 360° image of the vehicle. For example, it is sometimes difficult for the driver to determine the position of pylons, etc., along the side silhouette when reversing into a parking space, leading to an increased reliance on visual observation of the object itself. To counteract this, a 3D view is generated, allowing the driver to view the scene virtually from the outside using a virtual camera. Typically, the scene is displayed in the foreground, with the vehicle in the background. Although this view already provides significant support, a visualization not from the outside of the vehicle, but from the inside, especially from the driver's seat, would be desirable.

[0004] In this context, DE 11 2018 000 858 T5 describes such an interior perspective of a vehicle displayed on a screen. By overlaying images in the display, the impression is created that the vehicle is partially transparent, allowing one to see through it. However, the images are captured at different times, which, unlike a live image, can lead to safety risks.

[0005] German Patent DE 10 2015 201 642 A1 describes something similar. The driver wears video glasses. These glasses generate images that make the vehicle's interior partially transparent, allowing the driver to see hidden objects in the vehicle's surroundings from inside the vehicle.

[0006] From DE 10 2018 105 441 A1, it is also known to superimpose objects onto a semi-transparent image of the vehicle that cannot be recognized by the driver because they are located in obscured areas or in the blind spot, but which have been detected by environmental sensors. The driver's line of sight can be taken into account when superimposing these objects.

[0007] For further information on the state of the art, reference can also be made to US 2021 / 0197665A1. This allows for the overlaying of translations of texts captured on signs onto captured images of the surroundings.

[0008] The object of the present invention is to create an improved visualization of a vehicle environment which avoids or at least significantly reduces the correspondence problems.

[0009] According to the invention, this problem is solved by a method with the features of claim 1, and in particular in the characterizing part of claim 1. Advantageous embodiments of the method according to the invention are set forth in the dependent claims. Furthermore, a vehicle according to claim 9 solves the problem.

[0010] The method is used to visualize a vehicle's surroundings using a surround-view system. Similar to the state of the art, parts of the vehicle are displayed semi-transparently to avoid obscuring objects in the vehicle's environment from the driver's perspective.

[0011] According to the invention, a corneal reflection image is detected using a driver observation camera based on two-dimensional images. This allows for a reliable recording of what the driver of the vehicle can currently see. The vehicle interior is then segmented as the foreground and the portion of the vehicle's surroundings visible from the driver's perspective as the background within this corneal reflection image. Thus, the foreground, i.e., the vehicle interior from the driver's point of view, is available as the essential information. Depending on a predefined direction of the driver's gaze, a surround-view visualization of the surround-view system is then generated. This creates the image of the vehicle's surroundings that the driver would currently see if there were no other vehicles around.

[0012] In the inventive method, the foreground, segmented from the corneal reflective image, is projected semi-transparently into the surround-view visualization, and both are then displayed together. This creates an ideal view of the vehicle's surroundings, with the passenger compartment partially superimposed. By capturing the vehicle interior as seen by the driver from the corneal reflective image (CRI), the semi-transparent interior can be ideally positioned from the driver's perspective, thus avoiding or at least significantly reducing the correspondence problems described above.

[0013] The advantage of this invention is an adaptive surround-view visualization based on the driver's line of sight and the visualization of the vehicle's semi-transparent interior, enabling the correspondence between the viewing direction and the object. The method according to the invention requires no hardware modification and utilizes hardware already present in many production vehicles. The method is independent of lighting and scenario and significantly simplifies maneuverability at low speeds.

[0014] According to the invention, the segmentation of the foreground and background is further provided for by projecting a computer-generated three-dimensional image of the interior onto the corneal reflection image. Typically, a CAD 3D model of the vehicle's interior is available. This can then be projected onto the CRI. Subsequently, pixel registration is applied, which clearly delineates the vehicle interior within the CRI. This can be implemented very efficiently with minimal computational effort. This is particularly true if—as provided for in an advantageous embodiment—a driver-related calibration of at least the driver observation camera to a reference coordinate system is performed before the process begins. This allows the coordinates of the CAD 3D model and the driver's coordinates to be ideally aligned.

[0015] A particularly advantageous embodiment of the method according to the invention can further provide that the current driver's gaze direction is evaluated with regard to temporal constancy, whereby a predetermined driver gaze direction is generated from the current driver gaze direction only if temporal constancy is given. This ensures that the display, e.g., on a head unit, is possible without constantly changing as soon as the driver even briefly looks away.

[0016] Temporal constancy is particularly favorably established when the gaze direction remains within a defined range around the gaze direction recorded at the beginning of the time period for a given period. In other words, the gaze remains within the same visual corridor for a sufficiently long period.

[0017] A further highly advantageous embodiment of the method according to the invention can provide that the display of the surround-view visualization combined with the semi-transparent foreground is activated below a predetermined vehicle speed. This makes it very easy to ensure that the visualization is available for potential maneuvering of the vehicle, but remains active at higher speeds to avoid unnecessary driver distraction. Preferably, the entire process can be repeated cyclically only until the predetermined vehicle speed is exceeded. In addition to the display itself, this also saves on computational effort.

[0018] Even though a display would in principle be possible on any type of display in the vehicle, an advantageous further development of the inventive method provides that the surround-view visualization combined with the semi-transparent foreground is displayed on a head-up display or on a display of the vehicle's instrument cluster. The display is thus optimally positioned within the driver's field of vision; depending on the hardware installed in the respective vehicle and personal preference, the driver can choose between the display in the head-up display and the display on the central screen behind the steering wheel.

[0019] According to an advantageous embodiment of the inventive method, when the viewer's gaze shifts away from the display, the surround-view visualization combined with the semi-transparent foreground continues to be displayed in the viewing direction with a time delay when the viewer's gaze is directed back towards the display. Such a time delay or latency, preferably a few seconds, and particularly preferably about 3-4 seconds, allows obstacles located to the side and / or behind the vehicle to be detected and taken into account during maneuvering, largely without correspondence problems.

[0020] Preferably, it should be indicated during the delay that the image is not live. Various methods can be used for this, including warning tones or voice prompts, as well as text overlays such as "Caution; not a live image," or other visual / graphic measures.

[0021] The method according to the invention, in one or more of the described variants, can preferably be used in a vehicle. This vehicle comprises a surround-view system, a driver observation camera, and at least one computing unit configured to execute the method according to the invention.

[0022] Further advantageous embodiments will also become clear with reference to the following preferred embodiment, which is described with reference to the figures.

[0023] This shows: Fig. 1 schematic representation of a vehicle according to the invention; Fig. 2 a flowchart for a preferred embodiment of the method according to the invention; and Fig. 3 a schematic representation of a possible display according to the procedure.

[0024] In Fig. Figure 1 shows a vehicle 1 from a top-down perspective. A person 2, also referred to as driver 2 in the rest of the description, is shown driving the vehicle 1, indicated by a schematic representation of part of their head. The vehicle 1 has a surround-view system with several cameras 3, 4, 5, 6. One camera is located at the front, one at the rear, and one in each of the exterior mirrors. In addition, the vehicle 1 includes at least one driver observation camera 7 and a display unit 8. These components are connected to a processing unit 9.

[0025] The flowchart of Fig. The method 2 shown is intended to enable an adaptive and transparent view of the surround-view system based on the position of the driver 2 in the vehicle 1.

[0026] In the first step (VS 1), the driver observation camera 7 is calibrated in a vehicle reference system. This can be, for example, a reference coordinate system that also forms the basis for the CAD models used or created in vehicle design. In principle, this first step (VS 1) could also be omitted. The reasons for this and why the reference coordinate system used in vehicle design is particularly advantageous will be explained in detail in one of the subsequent steps.

[0027] In the next process step VS 2, an image of the driver's eye(s) is captured using the driver observation camera 7. Based on this image, the so-called corneal reflection image CRI, which essentially represents the environmental reflection on the driver's cornea, is derived in process step VS 3.

[0028] In the next process step, VS 4, the temporal stability of the CRI is evaluated. If this stability exceeds a defined time, the process continues with the next steps. Alternatively, process steps VS 2, VS 3, and VS 4 are repeated, as indicated by the arrow labeled 10. This process step, VS 4, is necessary to enable visualization on the display unit 8. Otherwise, the visualization would change repeatedly, even if the driver 2 only briefly shifts their gaze.

[0029] If temporal constancy is given, the process follows arrow 11, and in the next step, VS 5, the differentiation between foreground and background is calculated. This can be achieved through several approaches—for example, by searching for connected horizontal and vertical edge elements in the CRI or by using a region growing algorithm starting from the driver's viewpoint 2. However, it is particularly advantageous if a precise projection of the CAD 3D model of the interior is made into the CRI. A pixel registration is then applied to this projection, clearly defining and identifying the portion of the driver's interior (foreground) within the CRI. This makes it very easy to separate the interior as the foreground from the rest of the image content in the CRI, or to easily segment the foreground and background. For this approach, it is necessary to store a CAD model of the interior, which, for example,This can take place in a memory 12 of the processing unit 9, and it is necessary that the driver observation camera 7 has completed the first procedure step VS 1 of the calibration. In principle, any coordinate system can be converted into one another. However, it saves resources and effort if the calibration is performed directly in the coordinate system on which the CAD model is based, as mentioned above.

[0030] In the subsequent process step VS 6, the image from the surround-view system is generated. This is done taking into account the driver's (2) viewing direction, such that the camera image from the surround-view system is first generated for the CRI (corner-independent view) area. Then, in the next process step VS 6, the previously detected interior area is overlaid onto the generated camera image from the surround-view system. This creates an image that, from the driver's (2) perspective, visualizes the surroundings and also depicts the interior with a certain predefined level of transparency, allowing the driver (2) to see a correspondence to specific areas of the interior, such as the A-pillar.

[0031] In process step VS 8, this combination of the image from the surround-view system and the transparent interior is then displayed. Fig. Figure 3 illustrates this with a schematic representation from the driver's perspective (2), for example, of a head-up display (8). The lines and objects in the surround-view system's image are represented by solid lines. The semi-transparent interior of the vehicle (1) projected onto it is shown with dotted lines. To better distinguish the transparent area in the Fig. 3 Its upper edge is marked with a slightly thicker, dotted line. Particularly easy to see in the partially transparent area of ​​the interior are a ventilation nozzle 14 and a rectangular central display 15.

[0032] Starting from process step VS 9, a repetition of the process is then initiated without the recalibration in process step VS 1. Fig.This is indicated by arrow 13. Various termination criteria for the procedure can be considered, e.g., a vehicle speed at which maneuvering is no longer possible, a distance to objects that exceeds a predetermined distance, or similar criteria.

Claims

[1] Method for visualizing a vehicle environment using a surround-view system, wherein parts of the vehicle (1) are displayed semi-transparently in order to avoid occlusion of objects in the vehicle environment from a driver's perspective, wherein a surround-view visualization of the surround-view system is generated depending on a given driver's viewing direction, characterized by, that a corneal reflection image (CRI) is detected by means of a driver observation camera (7) based on two-dimensional images, wherein in an image of the corneal reflection image (CRI) a vehicle interior is segmented as the foreground and the part of the vehicle environment recognizable from the driver's perspective is segmented as the background, wherein the foreground segmented from the image of the corneal reflection image (CRI) is projected semi-transparently into the surround-view visualization and displayed together with it, wherein the segmentation of the foreground and the background is carried out by means of a projection of a computer-generated three-dimensional image of the vehicle interior into the image of the corneal reflection image (CRI). [2] Method according to claim 1, characterized by, that a current driver's gaze direction is evaluated with regard to temporal constancy, whereby the specified driver's gaze direction is generated from the current driver's gaze direction only if temporal constancy is given. [3] Method according to claim 2, characterized by , that temporal constancy is established if the current driver's gaze direction remains within a defined range around the driver's gaze direction recorded at the beginning of the time period for a given period of time. [4] Method according to any one of claims 1 to 3, characterized by , that the display of the surround-view visualization combined with the semi-transparent foreground occurs below a specified vehicle speed limit (1). [5] Method according to claim 4, characterized by , that the procedure is repeated cyclically until the specified limit speed of the vehicle (1) is exceeded. [6] Method according to any one of claims 1 to 5, characterized by , that before the start of the procedure a driver-related calibration of at least the driver observation camera (7) to a reference coordinate system is carried out. [7] Method according to any one of claims 1 to 6, characterized by , that the surround-view visualization combined with the semi-transparent foreground is displayed on a head-up display or on a display of the instrument display or head unit of the vehicle (1). [8] Method according to claim 7, characterized by , that if the driver's gaze is directed away from the display, the display of the surround-view visualization combined with the semi-transparent foreground in the driver's line of sight will continue to be displayed with a time delay when the gaze is directed back towards the display. [9] Vehicle (1) with a surround-view system, a driver observation camera (7) and at least one computing unit (9) which are configured to perform the method according to any one of claims 1 to 8.

Images