Method and device for generating and using a map of the surroundings for a vehicle

Abstract

The present invention relates to a method and a device for generating and using a map of the surroundings (10) for a vehicle, the method comprising: a first step for detecting surroundings of the vehicle by means of a plurality of cameras of a panoramic-view system of the vehicle, a second step for combining respective individual images of the cameras to form a map of the surroundings (10) at least partially enveloping the vehicle (10), wherein there is at least one gap (50) in the map of the surroundings (10) which represents a region which cannot be detected by the plurality of cameras, a third step for supplementing the at least one gap (50) in the map of the surroundings (10) on the basis of image information of at least one of the individual images and / or on the basis of information deviating from the currently detected individual images in order to obtain a supplemented map of the surroundings with a contiguous gap-free area, and at least partially outputting the supplemented map of the surroundings (15) together with a depiction of a predefined 3D model (60) of the vehicle in a display of the vehicle.

Description

[0001] Description

[0002] title

[0003] Method and device for generating and using an environment map for a vehicle

[0004] State of the art

[0005] The present invention relates to a method and a device for generating and using an environment map for a vehicle.

[0006] All-round vision systems for vehicles are known from the state of the art, which are often based on a large number of cameras (in particular cameras with fisheye lenses) arranged at different positions and with different orientations on the vehicles in order to capture the surroundings of the vehicles.

[0007] By combining individual images from the respective cameras, more or less complete all-round visibility information for the vehicle can be determined, which can be used, for example, by the vehicle's driver assistance systems to support driving maneuvers by appropriately displaying an all-round view on a vehicle display.

[0008] Furthermore, methods are known from the prior art for generating realistic reflections on the surfaces of computer-generated objects in a virtual 3D landscape. Examples of such methods include so-called "environment mapping," which utilize, among other things, cubic and / or spherical mapping variants.

[0009] Disclosure of the invention According to a first aspect of the present invention, a method for generating and using an environment map for a vehicle is proposed.

[0010] Such a vehicle is, for example, a road vehicle (e.g. motorcycle, car, van, truck) or a rail vehicle or an aircraft / plane and / or a watercraft.

[0011] In a first step of the method according to the invention, the surroundings of the vehicle are captured using a plurality of cameras of a surround view system of the vehicle. For this purpose, preferably at least two and more preferably at least three or more cameras are arranged at different positions on the vehicle (e.g., on respective side areas of the vehicle). Furthermore, the cameras are advantageously aligned with the surroundings of the vehicle in such a way that the largest and most contiguous part of the surroundings of the vehicle can be captured. For this purpose, for example, at least some of the cameras are equipped with a so-called fisheye lens in order to achieve the widest possible field of view for the respective cameras.

[0012] The aforementioned method step and / or method steps described below are carried out, for example, by means of an evaluation unit of the vehicle, which can also be configured to preprocess individual images generated by the cameras (e.g., to correct distortion in images generated using fisheye objects, etc.). Alternatively or additionally, such preprocessing can be performed at least partially in the cameras themselves and / or in another vehicle component. Advantageously, the cameras are arranged and aligned on the vehicle such that the respective fields of view of the individual cameras are at least adjacent to one another and / or partially overlap.

[0013] In a second step of the method according to the invention, respective individual images from the cameras are combined to form a map of the surroundings that at least partially envelops the vehicle, wherein at least one gap exists in the map of the surroundings, which represents an area that cannot be captured by the plurality of cameras. The map of the surroundings is to be understood as an area of ​​any desired shape (e.g. a partial area of ​​a spherical surface, a polygonal surface, a surface of a cuboid, etc.) that at least partially surrounds the vehicle. Advantageously, the edge of this area touches a road surface on which the vehicle is located, while the area envelops the vehicle starting from this road surface, but is not limited to this. It is also conceivable, for example, for the edge of the area to lie above the road surface if the field of view of the cameras does not capture the road surface or only captures it partially.In addition, different characteristics of the area of ​​the surrounding map can be used.

[0014] The environmental map is designed, for example, in the form of a cubic map (cube map) known from computer graphics and / or a spherical map (sphere map) and / or in a different form.

[0015] Since vehicle surround view systems are generally designed to capture as much of the vehicle's surroundings as possible that are relevant for maneuvering, the cameras of such a system are usually arranged at an angle on the vehicle at which an area of ​​the sky is not captured or only partially captured, as this area is normally not relevant for a driving maneuver.

[0016] Accordingly, with such surround-view systems, a gap in the sky area can usually be assumed, but this is not always the case. Alternatively or additionally, it is also possible that gaps may exist in other areas of the surrounding map, i.e., in the area that at least partially encloses the vehicle, due to non-adjacent and / or non-overlapping fields of view of the cameras.

[0017] In a third step of the method according to the invention, the at least one gap in the environment map is supplemented based on image information from at least one of the individual images and / or based on information that differs from the currently acquired individual images in order to create a supplemented environment map with a coherent, gap-free area. The gap is supplemented, for example, by using color and / or brightness and / or texture information about the environment in at least one of the individual images in the region of the gap in order to interpolate the gap based on this information. Further options for supplementing the gap are explained in more detail below in the course of the description of advantageous embodiments of the present invention.In a fourth step of the method according to the invention, the supplemented map of the surroundings is output at least partially together with a representation of a predefined 3D model of the vehicle in a display of the vehicle, wherein the display is, for example, a central display and / or a display of an instrument cluster and / or a head-up display and / or a display of the vehicle that differs therefrom.

[0018] By using the supplementary map of the surroundings, a particularly realistic representation of the vehicle's surroundings can be achieved on the display, which, among other things, makes orientation easier for a user of the vehicle.

[0019] Furthermore, such augmented environmental map offers the advantage of increased flexibility in selecting one or more perspectives from which the display is presented, since there are no longer any gaps that would otherwise have to be avoided in the display. This increased flexibility can be advantageously used, for example, when using state-of-the-art virtual camera movements around the 3D model of the vehicle (e.g., to display a perspective of the vehicle's surroundings that is particularly suitable for a driving maneuver).

[0020] A further advantage of the method according to the invention is that, with respect to currently deployed all-round vision systems, it is not necessary to expand the all-round vision system with additional cameras to additionally capture previously undetectable areas in the vehicle's surroundings (e.g., using an upwardly oriented camera mounted on the roof of the vehicle). Instead, the method according to the invention can be implemented particularly cost-effectively by expanding (e.g., in software) an existing processing chain for processing the camera images of the all-round vision system.

[0021] The subclaims show preferred developments of the invention.

[0022] In an advantageous embodiment of the present invention, the supplemented environment map is output in order to generate reflections on a surface of the 3D model of the vehicle that correspond to the current environment of the vehicle. This is done, for example, on the basis of an application of an “environment mapping” method described above and known from the prior art, in which the environment map is mapped onto the 3D model of the vehicle in a suitable manner to generate environment-related reflections. The at least partial output of the environment map in the fourth method step can in this case relate both to a restriction to that part of the environment map that results from a currently selected perspective on the 3D model and the environment, and to a restriction to those parts of the environment map that lead to a reflection on the vehicle surface (which, for example,have a minimum brightness).

[0023] In a further advantageous embodiment of the present invention, the supplemented environment map is output in order to surround the 3D model of the vehicle with a 3D scene corresponding to the current environment of the vehicle and / or in order to display the 3D model and the surrounding 3D scene from a perspective from which at least part of a supplemented area in the supplemented environment map is visible.

[0024] Particularly advantageously, the first step of the method according to the invention comprises detecting the surroundings of the vehicle by means of at least one first camera arranged in a front region of the vehicle and oriented substantially in the forward direction of travel of the vehicle, a second camera arranged on a right-hand side of the vehicle and oriented substantially to the right, a third camera arranged on a left-hand side of the vehicle and oriented substantially to the left, and a fourth camera arranged in a rear region of the vehicle and oriented substantially opposite to the forward direction of travel. The term “substantially” should be understood here to mean orientation angles which, for example, relate to a plane of the vehicle in which the axes of the vehicle lie (i.e. a horizontal plane, ora plane spanned by an X-axis and a Y-axis of a vehicle coordinate system) between 0° and ± 60° and preferably between 0° and ± 45°, without being restricted to such values. Advantageously, the at least one gap in the environment map lies in an area above the vehicle (i.e. in the positive direction of the Z-axis or vertical axis of the vehicle), so that to fill the gap, a region of the sky must generally be interpolated which can be assumed to be easier and / or more reliable to interpolate than other, generally more complex, regions of the environment. This results, for example, from an arrangement of the respective cameras as described immediately above.

[0025] Particularly advantageously, the information determined outside of the all-round vision system represents information about current lighting conditions (e.g., brightness, position of the sun, a respective proportion of direct and / or diffuse light, etc.) and / or a current time and / or a current date (wherein the time and / or date can be used, for example, to determine brightness and / or position of the sun, etc.) and / or a geographical position and / or orientation of the vehicle and / or a current weather situation (e.g., a type and / or number of clouds, a type and / or amount of precipitation, etc.) and / or objects in the vicinity of the vehicle (e.g., tall buildings that can cause shading and / or reflections, etc.) and / or a topography surrounding the vehicle (e.g., to take into account mountains in the vicinity that can affect light irradiation and / or generate reflections).By taking into account at least one of the above pieces of information, correspondingly accurate and / or realistic additions can be determined in the area of ​​the gap in the surrounding map.

[0026] Alternatively or additionally, the information obtained outside the all-round vision system corresponds to predefined image information (e.g., pre-rendered areas of a sky, etc.) and / or image information acquired in the past, which is stored, for example, in a storage unit connected to the evaluation unit via information technology and / or is provided with respective classification information in order to be able to select it based on current environmental conditions. Alternatively or additionally, the information obtained outside the all-round vision system corresponds to image information obtained outside the vehicle. Such image information obtained outside the vehicle is determined, for example, based on other vehicles and / or a cloud service, etc., and transmitted to the vehicle.Further alternatively or additionally, when filling the at least one gap in the environment map, additional image information from the captured individual images is taken into account, which relates, for example, to colors and / or brightnesses and / or structure and / or textures, etc. in areas preferably directly surrounding the gap, in order to fill the gap in line with this information.

[0027] The method according to the invention is particularly advantageously carried out repeatedly at predefined times. In particular, in a case in which time intervals when repeating the method are selected to be so short that changing ambient conditions are taken into account virtually instantaneously when the surroundings map is displayed on the vehicle's display while the vehicle is traveling, a particularly realistic impression of the representation of the vehicle and its surroundings on the display is created, which, among other things, enables particularly good orientation for a user, in particular a driver of the vehicle. Respective repetition intervals can be predefined uniform intervals (e.g. once per second, multiple times per second) and / or non-uniform intervals, in particular intervals adapted to current boundary conditions (e.g. adapted to a vehicle's speed).

[0028] According to a second aspect of the present invention, a device for generating and using an environmental map for a vehicle is proposed, which device has an evaluation unit configured to execute a method according to the first aspect of the invention. The evaluation unit is configured, for example, as an ASIC, FPGA, processor, digital signal processor, microcontroller, or the like. The features, feature combinations, and the advantages resulting therefrom correspond to those explained in connection with the first aspect of the invention, so that reference is made to the above explanations to avoid repetition.

[0029] Short description of the drawings

[0030] In the following, embodiments of the invention are described in detail with reference to the accompanying drawings, in which: Figure 1 shows an exemplary representation of a gap-prone

[0031] Environment map in conjunction with a 3D model of a vehicle;

[0032] Figure 2 shows an example of a supplemented

[0033] surrounding area map;

[0034] Figure 3 shows an example representation of a 3D model of a

[0035] vehicle taking into account a supplemented map of the surrounding area; and

[0036] Figure 4 shows a device according to the invention in connection with a

[0037] Vehicle.

[0038] Embodiments of the invention

[0039] Figure 1 shows an exemplary representation of a gapped environment map 10 in conjunction with a 3D model 60 of a vehicle (not shown).

[0040] The surroundings map 10 is created by capturing a plurality of individual images in a first step of a method according to the invention for capturing the surroundings of the vehicle by means of a plurality of cameras (not shown) of an all-round vision system of the vehicle.

[0041] In a second step of the method according to the invention, the respective individual images of the cameras are combined to form an environment map 10 enveloping the vehicle, wherein a gap 50 is present in the environment map 10, which represents an area above the vehicle that cannot be captured by the plurality of cameras.

[0042] In addition, Figure 1 shows, by way of example, an object 80 contained in the environment map 10, for which a reflection 85 on a surface of the 3D model 60 of the vehicle is calculated in order to be able to represent the 3D model 60 with a particularly high degree of realism. Figure 2 shows an exemplary representation of a supplemented environment map 15, which is determined on the basis of a third step of the method according to the invention, in which a gap 50 (see Figure 1) in a previously determined environment map 10 (see Figure 1) is supplemented on the basis of color and brightness information from individual images and on the basis of information about a current position of the sun, a current weather situation and on the basis of image information relating to the gap 50 acquired in the past.

[0043] The gap 50 is thus closed by a supplemented area 17 in order to achieve the supplemented environment map 15 with a coherent, gap-free area.

[0044] Figure 2 also shows several objects 80, which here represent clouds in an upper area of ​​the supplemented environmental map 15. Since the gap 50 in the supplemented environmental map 15 shown here is closed, the objects 80 are accordingly shown continuously in the supplemented area 17 without interruption.

[0045] Figure 3 shows an exemplary representation of a 3D model 60 of a vehicle in a display 70 of the vehicle taking into account a supplemented environment map 15 (see Figure 2), wherein the supplemented environment map 15 is used to display a reflection 85 of an object 80 (see Figure 2) on a surface of the 3D model 60 of the vehicle.

[0046] Figure 4 shows a device according to the invention in connection with a vehicle 20, wherein the device has an evaluation unit 90, which is designed here as an ASIC.

[0047] The evaluation unit 90 is connected for information technology purposes to a first camera 40, a second camera 42, a third camera 44, and a fourth camera 46 and is thus configured to receive individual images from the cameras 40, 42, 44, 46 at predefined time intervals (here 10 times per second) and to determine a supplemented environmental map 15 (see Figure 2) for the vehicle 20 based on the individual images. Furthermore, the evaluation unit 90 is configured to display the determined supplemented environmental map 15 in conjunction with a 3D model 60 (see Figure 3) of the vehicle 20 in a central display 70 of the vehicle 20 by using the environmental information contained in the supplemented environmental map 15 to surround the vehicle in the central display 70 with an image of the current surroundings of the vehicle 20.

Claims

Claims 1. A method for generating and using an environment map (10) for a vehicle (20), comprising: • a first step for detecting an environment (30) of the vehicle by means of a plurality of cameras (40, 42, 44, 46) of an all-round vision system of the vehicle (20), • a second step for merging respective individual images of the cameras (30) to form an environment map (10) at least partially enveloping the vehicle (20), wherein at least one gap (50) in the environment map (10) which represents an area not detectable by the plurality of cameras (40), • a third step for supplementing the at least one gap (50) in the environment map (10) on the basis of image information from at least one of the individual images and / or on the basis of information that differs from the currently acquired individual images in order to obtain a supplemented environment map (15) with a continuous, gap-free area, and • a fourth step for at least partially outputting the supplemented environmental map (15) together with a representation of a predefined 3D model (60) of the vehicle (20) in a display of the vehicle (20).

2. The method according to claim 1, wherein the supplemented environment map (15) is output to generate reflections (85) corresponding to the current environment (30) of the vehicle (20) on a surface of the 3D model (60) of the vehicle (20).

3. Method according to one of the preceding claims, wherein • the supplemented environment map (15) is output to surround the 3D model (60) of the vehicle (20) with a 3D scene corresponding to the current environment (30) of the vehicle (20), and / or • the 3D model (60) and the surrounding 3D scene are displayed from a perspective from which at least part of an augmented area (17) in the augmented environment map (15) is visible.

4. Method according to one of the preceding claims, wherein the first step comprises detecting the surroundings (30) of the vehicle (20) at least by means of • a first camera (40) arranged in a front area of ​​the vehicle (20) and directed substantially in the forward direction of travel of the vehicle (20), • a second camera (42) arranged on a right side of the vehicle (20) and directed substantially to the right, • a third camera (44) arranged on a left side of the vehicle (20) and directed substantially to the left, and • a fourth camera (46) arranged in a rear region of the vehicle (20) and oriented substantially opposite to the forward direction of travel.

5. Method according to one of the preceding claims, wherein the at least one gap (50) in the environment map (10) is present in an area above the vehicle (20).

6. Method according to one of the preceding claims, wherein the information determined outside the all-round vision system is information about • current lighting conditions, and / or • a current time, and / or • a current date, and / or • a geographical position and / or orientation of the vehicle (20), and / or • current weather conditions, and / or • Objects in the surroundings (30) of the vehicle (20), and / or • represents a topography surrounding the vehicle (20).

7. Method according to one of the preceding claims, wherein the information determined outside the all-round vision system • predefined image information, and / or • image information captured in the past, and / or • corresponds to image information obtained away from the vehicle (20).

8. The method according to one of the preceding claims, wherein, when filling the at least one gap (50) in the environment map (10), additional image information from the captured individual images is taken into account.

9. The method according to one of the preceding claims, wherein the method is repeatedly executed at predefined times.

10. Device for generating and using an environment map for a vehicle, comprising an evaluation unit 90, wherein the evaluation unit (90) is configured to carry out a method according to one of the preceding to execute claims.

Images