METHOD FOR REPRESENTING A VEHICLE ENVIRONMENT OF A VEHICLE

Description

State of the art

[0001] The present invention relates to a method for displaying a vehicle environment of a vehicle.

[0002] Driver assistance systems aim to support a driver in driving a vehicle. Such driver assistance can be provided, for example, through the following functions: Displaying an environment close to the vehicle to avoid collisions with obstacles that are not in the driver's immediate field of vision, taking over some of the driver's tasks to increase driving comfort for the driver, monitoring driver activities and intervening in dangerous situations, and / or automated driving without the necessary presence of the driver.

[0003] In particular, the present invention relates to a method for displaying a vehicle's surroundings using a composite view, as is done particularly in so-called surround-view systems. In such surround-view systems, a plurality of cameras are typically arranged on a vehicle in such a way that a 360° view of the vehicle's surroundings is enabled. Typically, camera images from cameras arranged on the vehicle, typically four cameras, are projected onto a geometric projection surface, which is typically bowl-shaped. This is done by means of image mapping, which is based on intrinsic and extrinsic camera calibration.

[0004] DE102011121473A1 discloses a method for displaying images on a display device of a motor vehicle by capturing a camera image of an area surrounding the motor vehicle by means of a camera of the motor vehicle.

[0005] US2014 / 0139676A1 discloses a vision system for a vehicle comprising at least one image sensor, a controller and a display.

[0006] EP1179958A1 discloses an image processing device in which areas with a predetermined distance to a vehicle are displayed in a blurred manner.

[0007] WO2008150153A1 discloses a method for generating a panoramic view in which a vehicle environment is represented by an arrangement of 3D points. Disclosure of the invention

[0008] The method according to the invention for displaying a vehicle environment of a vehicle comprises capturing the vehicle environment in camera images by means of a plurality of cameras, projecting the camera images onto a geometric projection surface in a virtual space, creating a depth map for a field of view of a virtual camera which describes a distance between a plurality of points of the geometric projection surface and the virtual camera in the virtual space, calculating an image of the virtual camera which depicts the geometric projection surface in the virtual space, determining a specific region of the image of the virtual camera based on the depth map in which the geometric projection surface lies at a specific distance range from the virtual camera, and blurring the image of the virtual camera in the region in which the specific region is depicted.

[0009] In this way, the driver's gaze is guided when viewing the calculated image. In particular, unintended differences between the image from the virtual camera and the vehicle surroundings depicted by it, which lie in specific areas of the geometric projection surface, are blurred and thus concealed. Such unintended differences sometimes arise because the geometric projection surface in the virtual space does not accurately reflect the vehicle surroundings.

[0010] The subclaims show preferred developments of the invention.

[0011] It is advantageous to model the geometric projection surface based on the vehicle's surroundings. This minimizes unintended differences between the image from the virtual camera and the vehicle's surroundings.

[0012] It is also advantageous if the specific distance range extends beyond a minimum distance from the virtual camera. This creates a depth effect and makes the virtual camera image appear particularly vivid.

[0013] Furthermore, it is advantageous if a virtual model of the vehicle is arranged in the virtual space, and the depth map is created in such a way that a virtual model of the vehicle is arranged in the virtual space, and the depth map is created in such a way that it describes a distance of the points of the virtual model 14 of the vehicle 1 depicted in those areas of the image of the virtual camera in which the virtual model 14 of the vehicle 1 is captured from the virtual camera. The virtual model of the vehicle gives a user a reference point in the image of the virtual camera. By taking the virtual model of the vehicle into account when creating the depth map, the vehicle is integrated into the image of the virtual camera and is therefore not perceived as a foreign body in the vehicle's surroundings.

[0014] It is also advantageous if the method includes aligning the virtual camera to a specific point of interest in the virtual space. This allows the user's attention to be directed to the specific point.

[0015] It is also advantageous if the specific point of interest is a point where the vehicle is approaching an object. This alerts the user to this approach and can prevent collisions.

[0016] A device which is designed to carry out the method according to the invention is also advantageous and has all the advantages of the method according to the invention. Short description of the drawings

[0017] Embodiments of the invention are described in detail below with reference to the accompanying drawings. In the drawing: Figure 1 shows a flowchart of the method according to the invention in a first embodiment, Figure 2 shows a vehicle with a device for carrying out the method in the first embodiment, Figure 3 shows an image of a virtual camera, wherein no blurring of predetermined areas of camera images has taken place, Figure 4 shows an image of the virtual camera which was generated by means of the method according to the invention in the first embodiment, Figure 5 shows an image of the virtual camera which was generated by means of the method according to the invention in a second embodiment. Embodiments of the invention

[0018] Figure 1 shows a flowchart of the method according to the invention for displaying a vehicle environment of a vehicle 1 in a first embodiment. The method is carried out by a device 2 for displaying a vehicle environment of the vehicle 1. The vehicle 1 is in Figure 2 shown as an example.

[0019] A first camera 3a, a second camera 3b, a third camera 3c, and a fourth camera 3d are arranged on the vehicle 1. Each of the cameras 3a, 3b, 3c, 3d is equipped with a fisheye lens in order to capture the largest possible section of the vehicle surroundings of the vehicle 1. The first camera 3a is arranged on a right-hand exterior mirror of the vehicle 1. An optical axis of the first camera 3a is directed away from the vehicle 1 along a transverse axis of the vehicle 1. The second camera 3b is arranged on a vehicle front of the vehicle 1. An optical axis of the second camera 3b is directed away from the vehicle along a longitudinal axis of the vehicle 1. The third camera 3c is arranged on a left-hand exterior mirror of the vehicle 1. An optical axis of the third camera 3c is directed away from the vehicle along the transverse axis of the vehicle 1.The fourth camera 3d is arranged at the rear of the vehicle 1. An optical axis of the fourth camera 3d is directed away from the vehicle 1 along the longitudinal axis of the vehicle 1. Each of the cameras 3a, 3b, 3c, 3d is coupled via a respective signal line to the device 2 for displaying the vehicle surroundings of the vehicle 1. The device 2 for displaying the vehicle surroundings of the vehicle 1 is a digital processing unit.

[0020] The method according to the invention is started as soon as the device 2 for displaying the vehicle surroundings of the vehicle 1 is put into operation. After the method is started, a first step S1 is executed.

[0021] In the first step S1, the vehicle surroundings of the vehicle 1 are captured in camera images using the cameras 3a, 3b, 3c, 3d. An image is captured by each of the cameras 3a, 3b, 3c, 3d and transmitted to the device 2. The camera images from the cameras 3a, 3b, 3c, 3d, which are arranged adjacent to one another on the vehicle 1, overlap one another and thus depict a similar area of the vehicle surroundings in a section.

[0022] After the first step S1, a second step S2 is executed.

[0023] In the second step S2, the camera images are projected onto a geometric projection surface in a virtual space. The virtual space is a mathematical space that is computationally generated by the device 2. The virtual space is a mathematical representation of the actual space around the vehicle 1. A virtual model 14 of the vehicle 1 is arranged in the virtual space. The geometric projection surface encloses the virtual model 14 of the vehicle 1 in the virtual space.

[0024] The geometric projection surface is modeled based on the vehicle's surroundings. For this purpose, the vehicle's surroundings are scanned using vehicle sensors. The vehicle sensors are a system for detecting the surroundings of the vehicle 1, which is based, for example, on ultrasonic sensors, stereo cameras, or a LIDAR system. Distance values are recorded that describe the distance of objects located in the vehicle's surroundings from the vehicle 1. These distance values are transferred to the virtual space, and the distance between the geometric projection surface and the virtual model 14 of the vehicle 1 is adjusted according to the distance values.

[0025] When projecting the camera images onto the geometric projection surface, the camera images from cameras 3a, 3b, 3c, and 3d are superimposed onto the geometric projection surface as textures. The camera images are rectified such that the actual objects in the vehicle's surroundings depicted in the camera images appear at a position on the geometric projection surface whose position relative to the virtual model 14 of the vehicle 1 corresponds to a position of the actual object relative to the vehicle 1.

[0026] After the second step S2, a third step S3 is executed.

[0027] In the third step S3, the image 10 of the virtual camera is calculated, which depicts the geometric projection surface in the virtual space.

[0028] To do this, a position and orientation of the virtual camera in the virtual space are first determined. For example, the position of the virtual camera is to the right behind the model 14 of vehicle 1. The camera is thus positioned in the area of a rear of the model 14 of vehicle 1 in the virtual space. The orientation of the virtual camera is selected, for example, along a right side of the model 14 of vehicle 1 in the direction of a front of the model 14 of vehicle 1.

[0029] The image of the virtual camera 10 consists of a matrix of pixels. Based on the position and orientation of the virtual camera, it is calculated for each pixel which part or point of the geometric projection surface is represented by a pixel. A color value and / or brightness value of a pixel of the virtual camera image is selected according to a color value and / or brightness value of the texture arranged on the corresponding part or point of the projection surface.

[0030] Figure 3shows an example image from the virtual camera where no blurring has yet occurred. The virtual model 14 of the vehicle 1 is arranged in a left-hand area of the image 10 from the virtual camera. In the area around the virtual model 14 of the vehicle 1, the geometric projection surface with the camera images stored thereon as textures can be seen. In an area 11 of the image 10 of the virtual camera, which lies to the right in front of the virtual model 14 of the vehicle 1, there is an unintentional difference between the image from the virtual camera and the vehicle surroundings depicted by it. It can be seen that a single vehicle 15 is displayed twice.

[0031] The individual vehicle 15 is depicted twice, as it is depicted in the camera image of the first camera 3a and in the camera image of the second camera 3b. When projecting the camera images onto the geometric projection surface, the two representations of the individual vehicle 15 were not arranged at the same location on the geometric projection surface, as the shape of the geometric projection surface does not fully correspond to the shapes in the vehicle's surroundings.

[0032] After the third step S3, a fourth step S4 is executed.

[0033] In the fourth step S4, a depth map is created for a field of view of the virtual camera, which describes the distance between several points of the geometric projection surface and the virtual camera in the virtual space. The field of view of the virtual camera corresponds to the area represented in the virtual camera image.

[0034] For each pixel of the virtual camera's image 10, the distance of the point of the geometric projection surface depicted in this pixel to the position of the virtual camera is calculated. This distance is stored in the depth map as a distance value. In this embodiment, the depth map therefore has as many distance values as the virtual camera's image 10 has pixels. Each distance value of the depth map is assigned to a pixel of the virtual camera's image 10.

[0035] The depth map is created in such a way that, for those areas of the virtual camera image in which the virtual model 14 of the vehicle 1 is captured, it describes a distance of the points of the virtual model 14 of the vehicle 1 depicted in these areas to the virtual camera in the virtual space. Thus, a distance value describing a distance between the virtual model 14 of the vehicle 1 is entered into the depth map when the geometric projection surface for the corresponding image point is obscured by the virtual model 14 of the vehicle.

[0036] After the fourth step S4, a fifth step S5 is executed.

[0037] In the fifth step S5, a specific area 12 of the image of the virtual camera is determined based on the depth map, in which the geometric projection surface lies within a specific distance range relative to the virtual camera.

[0038] In this first embodiment, the specific distance range is a range that extends outside a minimum distance from the virtual camera. The minimum distance is a distance in the virtual space. The minimum distance is defined by a threshold value. Each of the distance values of the depth map is compared with the threshold value. If a distance value is above the threshold value, the pixel of the virtual camera image 10 corresponding to this distance value belongs to the specific range 12. If a distance value is below the threshold value or is equal to the threshold value, the pixel of the virtual camera image corresponding to this distance value does not belong to the specific range 12.

[0039] After the fifth step S5, a sixth step S6 is executed.

[0040] In the sixth step S6, the image of the virtual camera is blurred in the area in which the specific area 12 is imaged. For this purpose, for example, a Gaussian blur or any other graphic filter that enables blurring is applied to the specific area 12 of the image 10 of the virtual camera.

[0041] Figure 4shows an image 10 of the virtual camera, which was generated according to the first embodiment of the invention. In a left-hand area of the image of the virtual camera, the virtual model 14 of the vehicle 1 is arranged. In the area around the virtual model 14 of the vehicle 1, the geometric projection surface with the camera images stored thereon as textures can be seen. In the upper area of the image 10 of the virtual camera, the specific area 12 is depicted, which is blurred and depicts an area of the real world that lies in front of the vehicle 1. In this specific area, the geometric projection surface has a distance from the virtual camera that, according to the depth map, is greater than the threshold value. It can be seen that the area 11, in which the unintentional difference lies, has also been blurred.

[0042] After executing the sixth step S6, the method branches back to the first step S1. The method is thus executed in a loop. The method is terminated when the device 2 is shut down.

[0043] According to the invention, the depth map is thus generated depending on the position of the virtual camera. In addition, a partially blurred image of the vehicle's surroundings is generated. The image depth can be adjusted according to the focus range of an actual camera. This effect can be dynamically adjusted to simulate the behavior of the human eye. This can be achieved, in particular, by changing the threshold value used in the fifth step S5.

[0044] A second embodiment of the invention corresponds to the first embodiment of the invention. However, the method according to the invention additionally comprises a further step in which the virtual camera is aligned to a specific point of interest in the virtual space. The specific point of interest is a point at which the vehicle is approaching an object.

[0045] For this purpose, the distance sensors of vehicle 1 are used to determine whether vehicle 1 is approaching an object. The point of interest here is a center of the area of the vehicle's surroundings in which the object is approaching vehicle 1. The position of the virtual camera is selected such that it lies directly above point of interest 16 in the virtual space, and the orientation of the virtual camera is selected such that it looks down onto point of interest 16 from a bird's eye view.

[0046] Figure 5 shows an image 10 of the virtual camera, which was calculated according to the second embodiment of the invention. In the Figure 5 In the image shown by the virtual camera, the rear of the virtual model 14 of the vehicle 1 is visible. In a focus area 13, which lies to the right behind the virtual model 14 of the vehicle 1, the vehicle 1 approaches an object. The point of interest is set in the center of this focus area 13.

[0047] With the selected position and orientation of the virtual camera, the distance of the geometric projection surface, which is flat in this area and arranged corresponding to a road surface, from the virtual camera increases with increasing distance to the point of interest 16. The threshold value of the first embodiment is selected in this second embodiment such that those points of the geometric projection surface that lie in the focus area 13 each have a distance from the virtual camera that is below the threshold value. The focus area 13 thus corresponds to an area that lies outside the specific area 12 according to the invention.

[0048] Thus, the area of the virtual camera image 10 that does not lie within this focus area 13 is blurred. Thus, only the focus area 13 is displayed in focus in the virtual camera image. The driver's attention is thus directed to this area.

[0049] It is pointed out that predefined areas of the geometric projection surface can be excluded from blurring in the method according to the invention.

[0050] In addition to the above written revelation, explicit reference is made to the revelation of Figures 1 to 5 referred to.

Claims

1. Method for representing a vehicle environment of a vehicle (1), comprising: - capturing the vehicle environment in camera images by means of a plurality of cameras (3a, 3b, 3c, 3d), - projecting the camera images onto a geometric projection surface in a virtual space, - creating a depth map for a field of view of a virtual camera, which describes a distance of a plurality of points of the geometric projection surface from the virtual camera in the virtual space, - calculating an image (10) of the virtual camera which images the geometric projection surface in the virtual space, - ascertaining a specific region of the image of the virtual camera based on the depth map in which the geometric projection surface is located in a specific distance region relative to the virtual camera, wherein the specified distance region is a region extending outside a minimum distance relative to the virtual camera, and - blurring the image (10) of the virtual camera in the region in which the specific region (12) is imaged.

2. Method according to Claim 1, characterized in that the geometric projection surface is modelled based on the vehicle environment.

3. Method according to either of the preceding claims, characterized in that - a virtual model (14) of the vehicle (1) is arranged in the virtual space, and - the depth map is created in such a way that for those regions of the image of the virtual camera in which the virtual model (14) of the vehicle (1) is captured, it describes a distance of the points of the virtual model (14) of the vehicle (1) which are imaged in these regions from the virtual camera.

4. Method according to any of the preceding claims, further comprising: - aligning the virtual camera to a specific point of interest in the virtual space.

5. Method according to Claim 4, characterized in that the specific point of interest is a point at which the vehicle (1) approaches an object.

6. Apparatus (2) configured to carry out the method according to any of the preceding claims.