Method for the creation of a representation of surroundings representing vehicle surroundings, control device, driver assistance system, and vehicle

Abstract

A method for creating a representation (15) of surroundings representing vehicle surroundings, involving the steps of: - providing camera images (12) captured by one or more vehicle (1) cameras (3), each of said camera images (12) representing at least a portion of the vehicle surroundings; - producing three-dimensional surroundings data from at least one of the camera images (12), the at least one camera image (12) having been captured at a first capturing time; - identifying a concealed sector (14) of the surroundings for at least one of the cameras from the surroundings data; - acquiring image data for filling in the concealed sector (14) of the surroundings, the image data being acquired from at least one additional camera image captured by another one of the cameras (2) at the first capturing time and / or from at least one stored camera image captured by the one or more cameras (3) at a second capturing time that precedes the first capturing time, creating the representation of the surroundings from the at least one camera image (12) and from the image data, the image data being used to represent, in the representation (15) of the surroundings, at least a portion of the sector (14) of the surroundings that is concealed in the at least one camera image (12).

Description

[0001] Description

[0002] Method for generating an environmental representation representing a vehicle environment, control device, driver assistance system and vehicle

[0003] The invention relates to a method for generating an environmental representation representing a vehicle's surroundings. Furthermore, the invention relates to a control device, a driver assistance system, and a vehicle.

[0004] Modern vehicles are increasingly equipped with driver assistance systems that support the driver in performing driving maneuvers by displaying a view of the surroundings. Such views can, for example, be so-called surround-view displays, which depict at least part of the vehicle's surroundings. Such a view of the surroundings is usually generated based on camera images captured using one or more of the vehicle's cameras.

[0005] The surrounding area can be displayed from a virtual camera position that differs from the actual camera arrangement on the vehicle. For this purpose, the camera images can be projected onto a virtual projection surface, for example, using the projection surface to determine a view of the surrounding area from a different vantage point. However, it may happen that individual areas of the vehicle's surroundings are obscured by objects in the vehicle's surroundings.

[0006] DE 10 2015 223 176 A1 discloses a driver assistance system for a vehicle with environmental sensors that detect the vehicle's surroundings. The sensor data from the environmental sensors is evaluated to detect obstacles in the vehicle's surroundings, and masking areas in the vehicle's surroundings are determined based on the detected obstacles. The masking areas are concealed by the obstacles and limit the field of view of the driver assistance system's optical environmental sensors. The determined masking areas can be covered with textured surfaces when displaying the camera images.

[0007] DE 10 2018 203 590 A1 describes a surround view system for a vehicle. Using an evaluation unit of the surround view system, an object is detected in recorded environmental data and the object's 3D shape is determined. A projection surface used to generate a surround view is then supplemented with the specific 3D shape of the object. Image areas obscured by the object can be further filled with a predefined color, with an interpolation of surrounding areas, and / or with a reflection of environmental data from the front of the object.

[0008] In order to provide the driver with the most complete overview possible of the vehicle's surroundings, it is desirable that, when displaying the surroundings from a virtual camera position that does not correspond to the actual camera position, there are as few areas as possible in which the actual vehicle's surroundings are not displayed.

[0009] The invention is based on the object of specifying an improved method for generating an environmental representation representing a vehicle environment, which in particular avoids or at least reduces the occurrence of areas without a representation of the actual vehicle environment.

[0010] This object is achieved according to the invention by a method for generating an environmental representation representing a vehicle environment, the method comprising the following steps:

[0011] Providing camera images recorded with one or more cameras of the vehicle, wherein at least a part of the vehicle's surroundings is shown in each of the camera images, - generating three-dimensional environmental data from at least one of the camera images, wherein the at least one camera image was recorded at a first recording time,

[0012] - Determining a hidden environmental area for at least one of the cameras from the environmental data,

[0013] - Determining image data for at least partially filling the concealed surrounding area, wherein the image data are determined from at least one further camera image of another of the cameras recorded at the first recording time and / or from at least one stored camera image of the one or more cameras recorded at a second recording time prior to the first recording time,

[0014] - generating the environmental representation from the at least one camera image and the image data, wherein at least a part of the environmental area hidden in the at least one camera image is represented in the environmental representation using the image data.

[0015] The camera images used to generate the environmental representation can, for example, be generated by one or more cameras of the vehicle and transmitted to a control device configured to carry out the method via a communication connection to provide the camera images. The camera images can, in particular, be generated and transmitted continuously. At least some of the camera images can also be stored, at least temporarily, in a storage device of the vehicle. In this way, camera images recorded at previous recording times are also available for generating the environmental representation.

[0016] The portion of the vehicle's surroundings depicted in each camera image depends on the position of the camera generating the respective camera images on the vehicle. It is possible for multiple cameras arranged at different positions on the vehicle to be used to generate the camera images. For example, at least one front camera arranged at the front of the vehicle, one or more side cameras arranged on the side of the vehicle, and / or at least one rearview camera arranged at the rear of the vehicle can be used. According to the invention, multiple cameras of a camera arrangement designed as a surround-view system of the vehicle can be used.

[0017] A surround view can be generated, in particular, as the environmental representation. The environmental representation can be generated, in particular, from a virtual camera position that differs from the actual arrangement positions of the one or more cameras on the vehicle. The change in perspective can be achieved by projecting the camera images onto a projection surface, for example, a bowl-shaped projection surface that virtually surrounds the vehicle. The environmental representation can then be derived from the projected camera images, starting from the virtual camera position.

[0018] By changing the perspective of the display from the actual camera position from which the camera images are generated to the position of the virtual camera, it may occur that image areas in the camera images that are obscured by an environmental object are no longer obscured from the perspective of the virtual camera. However, for such an obscured environmental area, the camera image does not contain any information describing the vehicle's surroundings within the obscured environmental area.

[0019] In order to determine these hidden environmental areas, three-dimensional environmental data is first generated from at least one of the camera images, wherein the at least one camera image was recorded at a first recording time. The three-dimensional environmental data describes the vehicle's surroundings, in particular in a three-dimensional coordinate system. In particular, the three-dimensional environmental data can describe the geometry of environmental objects as well as the respective position of the environmental objects relative to the vehicle. Using the environmental data, a hidden environmental area can then be determined. The hidden environmental area refers to one of the cameras and describes the part of the vehicle's surroundings that is hidden by an object from the camera and thus cannot be viewed.In other words, the hidden environmental area assigned to one of the cameras is a description of that part of the vehicle's surroundings for which no image information is contained in the camera images taken with the respective camera.

[0020] Image data is then determined to fill the concealed surrounding area. The image data used to fill the concealed surrounding area in particular shows at least a portion of the vehicle's surroundings within the concealed surrounding area. In other words, the image data used to fill the concealed surrounding area contain a description of at least a portion of the actual vehicle's surroundings in the concealed image area, so that they can be used in the surroundings view to display the concealed surrounding area. The image data can describe the entire concealed surrounding area, or image data can be determined that describes only one or more sub-areas of the concealed surrounding area.

[0021] The image data for filling the concealed environmental area can be taken from at least one camera image taken at the first recording time from another of the cameras. This can be the case, for example, if the vehicle has multiple cameras, each of which takes camera images, whereby the detection areas of the cameras and thus also the environmental sub-areas depicted in the camera images partially overlap. The image data can, for example, be taken from one of the overlapping areas of a camera image from the further camera. Additionally or alternatively, image data from a stored camera image taken at a second recording time can be used according to the invention. The second recording time lies before the first recording time, so that the image data is taken from an earlier recorded camera image, which can also be referred to as a historical camera image.The stored camera image can, for example, be stored in a storage device in the vehicle.

[0022] The previously recorded camera image can be used if an image area is obscured by a static surrounding object, especially if the vehicle has moved between the first recording time and the second recording time. If multiple camera images are stored, a stored camera image can be selected that was recorded from a different position of the vehicle, and thus also from a different camera position.

[0023] The stored camera image whose image data is used to replace the obscured surrounding area determined for one of the cameras can be recorded by the same camera whose obscured surrounding area is to be replaced. Additionally or alternatively, a stored camera image recorded by another camera in the vehicle can also be used.

[0024] The environment representation is then generated from the at least one camera image and the image data. In this case, at least a portion of the surrounding area hidden in the at least one camera image is represented in the environment representation using the image data. In other words, in the environment representation, the hidden surrounding area is at least partially filled with the image data of another of the cameras and / or with image data recorded at an earlier point in time. This means that the environment representation contains additional information about the actual vehicle environment, which is available when generating the

[0025] A representation of the surroundings would not be possible if it were composed solely of a single camera image or of several camera images combined at their overlapping areas. The use of image data from a previously recorded camera image represents at least an approximation of the current vehicle surroundings, especially if the first recording time and the second recording time are only a short period of time apart.

[0026] The surroundings display can be displayed visibly for a driver, particularly on a display device of the vehicle, in particular on a display arranged in the vehicle interior. This provides assistance to the driver, for example, when performing maneuvers in confined spaces.

[0027] The method according to the invention advantageously enables the driver to improve their situational awareness. By incorporating the image data into the surroundings display, additional information or context is provided, which can make driving easier for the driver. In particular, gaining an understanding of the situation is simplified, which advantageously allows the driver to react more quickly and / or better to the current traffic situation in the vehicle's surroundings.

[0028] By filling in obscured areas of the vehicle's surroundings, the occurrence of blind spots or blind spots can be prevented or at least reduced, which advantageously increases road safety. A driver's better understanding of the situation can reduce the risk of collisions or accidents caused by obscured surroundings.

[0029] In particular, the insertion of image data from a stored camera image makes it possible to implement additional functions such as object tracking, object recognition, and / or augmentation of the environment representation with additional information when evaluating the three-dimensional vehicle environment, which can be done, for example, based on the generated environment representation. This also simplifies the implementation of additional functions that rely on the most complete understanding of the current traffic situation possible.

[0030] Particularly in the case of static environmental objects, valuable information regarding the presence and position of objects, landmarks and / or obstacles can also be obtained from previously recorded camera images if these are combined with a current camera image of the vehicle's surroundings in the environmental display.

[0031] By reducing hidden image areas in which no image content or only schematic image content such as a static color or the like is displayed, it is further achieved that the vehicle's surroundings can be continuously reproduced in a consistent manner. This particularly simplifies the perception of the surroundings when dynamic conditions, such as changing lighting conditions and / or dynamic objects, are present in the vehicle's surroundings. In such cases, the method according to the invention can advantageously be used to generate a consistent representation of the surroundings that essentially corresponds to human perception and can be viewed by a driver of the vehicle.

[0032] According to the invention, it can be provided that it is first checked whether the image data for filling the concealed surrounding area are present in at least one further camera image of a further camera recorded at the first recording time, wherein if the image data from the at least one further camera image are present, these image data are used to generate the representation of the surroundings and wherein if the image data from the at least one further camera image are not present, the image data from the at least one stored camera image are used to generate the representation of the surroundings.

[0033] This two-step approach advantageously allows for the initial check to determine whether additional current camera images (i.e., images also taken at the time of the first recording) are available from which the image data for filling the obscured surrounding area can be extracted. This can be done, for example, using the three-dimensional surrounding data or the obscured surrounding areas derived from them, if these were determined for multiple cameras in the vehicle. If such current image data is available, it can be used preferentially to fill the obscured surrounding area.

[0034] If no further camera images from another camera are available at the first recording time from which the image data for the concealed surrounding area could be extracted, then it can be checked whether there are any stored camera images recorded by the camera or another camera of the vehicle at an earlier, second recording time that contain the image data for filling the concealed surrounding area. Accordingly, the image data from the at least one stored camera image can then be used to generate the surrounding view.

[0035] In the case of an environmental representation which is formed from several camera images taken at the first recording time, it can be checked for each of the camera images whether the image data of another camera taken at the first recording time or historical image data taken from a camera image taken at a second recording time are used for a possibly existing hidden environmental area.

[0036] In a preferred embodiment of the invention, it can be provided that the image data for filling the concealed environmental area are determined based on the environmental data, wherein the concealed environmental area of ​​the at least one camera at the first recording time is compared with the concealed environmental area of ​​the at least one further camera and / or with concealment information assigned to the stored camera image, which describes the concealed environmental area of ​​the camera generating the stored camera image at the second recording time. By comparing the concealed environmental areas for two of the cameras, it can be easily determined whether image data for the concealed environmental area in the camera images of one of the cameras is contained in the camera images of the other camera. This is the case if the environmental area concealed for one camera is not concealed for the further camera.This applies equally to the comparison of two camera images taken at the first recording time and to the comparison between a camera image taken at the first recording time and a stored camera image taken at a second recording time.

[0037] Additionally or alternatively, it is possible for the image data to be determined based on occlusion information associated with a stored camera image. The occlusion information can describe which part of the vehicle's surroundings is occluded in the stored camera image, thus simplifying the process of determining whether the camera image contains image data to fill a occluded area of ​​the surroundings. The occlusion information can, for example, be determined at or shortly after the time the camera image is captured and stored with the camera image in a storage device.

[0038] According to the invention, the environmental data can be determined as a function of depth information associated with the camera image, as a function of map information describing the position of the vehicle and the vehicle's surroundings, and / or as a function of the measurement data of at least one distance sensor of the vehicle. The three-dimensional environmental data can be generated from the camera image by determining depth information from the image content reproduced in the camera image.

[0039] Additionally or alternatively, it is possible for the three-dimensional environmental data to be determined based on map information describing the vehicle's surroundings. The map information can be determined, for example, using a method such as simultaneous localization and mapping (SLAM) and / or comparable methods.

[0040] It is also possible for the three-dimensional environmental data to be determined based on the measurement data from at least one distance sensor of the vehicle. Using the distance sensor, a distance between the sensor or the vehicle and an object in the vehicle's surroundings can be determined. The distance sensor can be implemented, for example, as a radar sensor, a lidar sensor, and / or an ultrasonic sensor.

[0041] According to the invention, the environmental representation can be generated starting from a virtual camera position, wherein the image data for displaying at least part of the concealed environmental area is transformed depending on the virtual camera position. Because the image data can be taken from a camera image whose perspective differs from the perspective starting from the virtual camera, an improved reproduction of the actual vehicle surroundings can be achieved by transforming the image data to fill the concealed environmental area in the environmental representation. By transforming the image data, the difference in perspective can be at least partially compensated for, resulting in a coherent overall image.

[0042] In a preferred embodiment of the invention, it can be provided that the image data and the at least one camera image are combined to generate the surrounding view depending on at least one image harmonization function. With the aid of the image harmonization function, for example, differences in exposure, color, and / or resolution between the camera image and the inserted image data can be at least partially compensated. By using at least one image harmonization fusion, a more coherent overall image can thus be achieved in the representation of the surroundings. According to the invention, it can be provided that several cameras of a camera arrangement designed as a surround view system of the vehicle are used.Accordingly, the environment representation may preferably be a surround view representation which, starting from a virtual camera position, displays, for example, a virtual model of the vehicle as well as the part of the vehicle surroundings visible from the position of the virtual camera.

[0043] Furthermore, the invention relates to a control device configured to carry out a method according to the invention. The step of providing the camera images can be performed, for example, by the control device being configured to receive the camera images via a communication connection and / or to retrieve the camera images from a storage device of the vehicle.

[0044] For a driver assistance system according to the invention, it is provided that it comprises at least one camera which is designed to provide camera images representing at least part of a vehicle environment, and a control device designed to carry out a method according to the invention.

[0045] For an invention according to the vehicle, it is provided that it has a driver assistance system according to the invention.

[0046] All advantages and embodiments described above with respect to the method according to the invention also apply correspondingly to the control unit according to the invention, the driver assistance system according to the invention, and the vehicle according to the invention, and vice versa. The advantages and embodiments described with respect to the control device according to the invention, the driver assistance system according to the invention, and the vehicle according to the invention can also be applied analogously to the other objects. Further advantages and details of the invention will become apparent from the exemplary embodiments described below and from the drawings. These are schematic representations and show:

[0047] Fig. 1 shows an embodiment of a vehicle according to the invention comprising an embodiment of a driver assistance system according to the invention,

[0048] Fig. 2 is a block diagram of an embodiment of a method according to the invention,

[0049] Fig. 3 shows an example of a camera image taken with a camera of the vehicle and

[0050] Fig. 4 shows an embodiment of an environment representation generated using the method according to the invention.

[0051] Fig. 1 shows an embodiment of a vehicle 1. The vehicle 1 can be, for example, a motor vehicle, in particular a

[0052] A passenger car, a truck, or another type of commercial vehicle. It is also possible for vehicle 1 to be a non-motorized vehicle such as a trailer, or a combination consisting of a towing vehicle and a trailer. Vehicle 1 can also be a rail-bound vehicle, such as a tram or the like, or a robot.

[0053] The vehicle 1 comprises an exemplary embodiment of a driver assistance system 2, which comprises one or more cameras 3 and a control device 4. In the present exemplary embodiment, the vehicle 1 comprises four cameras 3, which form a camera arrangement designed as a surround-view system. A first camera 5 is arranged as a front camera of the vehicle 1, a second camera 6 as a rear-view camera of the vehicle 1, and a third camera 7 and a fourth camera 8 are each arranged as a side camera of the vehicle 1. The control device 4 can be designed, for example, as a microcontroller, as a processor, or as another type of computing device.

[0054] The driver assistance system 2 further comprises a display device 9, via which graphic information can be presented to a driver or a user of the vehicle 1. The display device 9 can, for example, be one or more screens or displays arranged in an interior of the vehicle 1.

[0055] The vehicle 1 further comprises at least one distance sensor 10. The at least one distance sensor 10 can be used to determine the distances to objects, in particular to objects located in front of, next to, and behind the vehicle. The distance sensor 10 can be designed, for example, as a radar sensor, a lidar sensor, and / or an ultrasonic sensor. The vehicle 1 can, in particular, have a plurality of distance sensors 10, which can be designed as any combination of different sensor types and arranged at different positions on the vehicle 1.

[0056] Using the cameras 3, image data can be generated from a sub-area of ​​the surroundings of the vehicle 1. The cameras 3 and the at least one distance sensor 10 are connected to the control unit 4 via a communication connection 11. The communication connection 11 can, for example, comprise a plurality of point-to-point connections or be a bus connection such as a CAN bus or the like. Image data or camera images are transmitted, in particular continuously, from the cameras 3 to the control unit 4 via the communication connection 11.

[0057] The control device 4 is designed to carry out a method for generating an environmental representation representing the vehicle's surroundings. For this purpose, the control device 4 uses a plurality of camera images recorded by the cameras 3 of the vehicle 1. In order to gain access to camera images recorded at an earlier point in time or an earlier vehicle position, at least some of the recorded camera images can be stored in the vehicle 1, for example in a storage device of the control device 4 and / or in a storage device of the respective camera. The storage device can be, for example, a temporary data storage device such as a main memory or a permanent data storage device such as a hard drive.

[0058] Fig. 2 shows a block diagram of an embodiment of a method for generating an environmental representation representing the vehicle environment.

[0059] In a first step S1 of the method, a plurality of camera images are provided, which were recorded using the cameras 3 of the vehicle 1, wherein the camera images each show a partial area of ​​the vehicle's surroundings. The camera images can be current camera images, which are generated at a first recording time and transmitted to the control device. Additionally or alternatively, historical camera images can also be provided, which were recorded at a second recording time prior to the first recording time and which are stored in the vehicle 1 so that they can be retrieved by the control device 4.

[0060] Subsequently, in a step S2, three-dimensional environmental data is generated from at least one of the camera images recorded at a first recording time. The three-dimensional environmental data describe objects and / or features of the environmental structure with regard to their geometric shape and their arrangement, in particular in relation to the position of the vehicle 1.

[0061] The environmental data can be derived from the image content of at least one camera image. The environmental data can be determined by the control device 4, which can evaluate the image content of one or more of the camera images, for example, using one or more algorithms. The image content can be evaluated, for example, using methods such as semantic segmentation, structure-from-motion, SLAM, or similar methods.

[0062] The environmental data can be determined based on depth information associated with the camera image. The depth information can indicate which image areas have which depth, i.e., which distance from the vehicle 1. The depth information can, for example, be determined by the control device 4 using one or more of the algorithms, as described above. For stored camera images, it is possible for corresponding depth information for the stored camera images to also be stored in a memory device so that it can be retrieved by the control device 4.

[0063] It is also possible for the environmental data to be determined based on map information describing the position of the vehicle and the vehicle surroundings. The map information may contain information about the vehicle surroundings as well as the position of the vehicle within the vehicle surroundings. A map used in the motor vehicle

[0064] Map information can in particular be continuously updated depending on the camera images from the cameras 3 and / or the measurement data from the distance sensor 10.

[0065] It is also possible for the measurement data from the distance sensor 10, which describes the distances of objects in the surroundings of the vehicle 1 to the vehicle 1, to be used to determine the environmental data. For this purpose, the measurement data from the distance sensor 10 can be transmitted from the distance sensor 10 to the control device 4 via a communication connection.

[0066] Subsequently, in step S3 of the method, obscured environmental areas are determined for each of the individual cameras 3 of the vehicle 1 from the previously determined three-dimensional environmental data. In particular, the areas in which a part of the vehicle's surroundings cannot be viewed due to being obscured by an object in the surroundings are identified.

[0067] Fig. 3 shows, by way of example, a camera image 12 in which a part of the vehicle's surroundings is obscured by an object 13. The camera image 12 can, for example, have been recorded by one of the side cameras 7, 8 of the vehicle 1. The obscured image area 14 comprises those sub-areas which lie behind the objects 13 designed as posts. No image information is available in the camera image 12 for the areas of the vehicle's surroundings which lie behind the objects 13. The obscured image areas 14 each relate to a camera image 12 or to one of the cameras 3, so that the obscured image areas 14 can differ for different cameras 3.

[0068] In step S4, image data for filling the concealed surrounding area 14 is determined. In a first sub-step S4A, the image data can be taken from at least one further camera image recorded at the first recording time from one of the further cameras 3 of the vehicle 1. For example, a camera image from the front camera 5 or the rear-view camera 6, also recorded at the first recording time, can be used if they capture the concealed surrounding area 14 of the side cameras 7, 8 unobscured due to an overlap of the detection areas with the side cameras 7, 8.

[0069] In a sub-step S4B, the image data can be determined from at least one stored camera image of the one or more cameras 3 of the vehicle 1, recorded at a second recording time prior to the first recording time. In this sub-step, the image data are thus taken from a historical camera image recorded at an earlier time. This stored camera image can be loaded by the control device 4 from a storage device of the vehicle 1. The stored camera image can be recorded by the same camera as the camera image showing the obscured surrounding area, or it can be a stored camera image recorded by a different one of the cameras 3.

[0070] Substeps S4A and S4B can be performed alternatively or sequentially. In particular, it can first be checked whether the image data for filling the concealed surrounding area 14 is present in at least one additional camera image recorded at the first recording time by another of the cameras 3. If the image data from the at least one additional camera image is present, this image data is subsequently used to generate the surrounding representation.

[0071] The image data for filling the concealed surrounding area can be determined, for example, based on the three-dimensional surrounding data. The concealed surrounding area 14 of the at least one camera 3 at the first recording time can be compared with the concealed surrounding area of ​​the at least one further camera. In this way, it can be checked whether image data for the concealed surrounding areas 14 are present in the further camera image.

[0072] If another camera image from the first recording time is not available, or if the image data cannot be obtained from the other available camera images, a stored camera image can subsequently be determined and used to generate the environmental representation. The stored camera image can, in particular, be determined in such a way that it depicts the vehicle's surroundings from a different camera position. This can be the case, for example, if the vehicle has moved between the second recording time and the first recording time. This means that the stored camera image contains image data for the surrounding areas 14 concealed by the camera image recorded at the first recording time.

[0073] When using a saved camera image, the image data can be

[0074] Filling the concealed surrounding area 14 can also be determined, for example, using occlusion information associated with the stored camera image. The occlusion information can describe the concealed surrounding area of ​​the camera 3 generating the stored camera image at the second recording time. Using the occlusion information, the concealed surrounding area of ​​the stored camera image and the further camera image can be compared. The occlusion information can, for example, be associated with the stored camera image and also stored in the storage device. It is possible that the occlusion information was already generated when the stored camera image was recorded and stored in the storage device for later use.

[0075] Finally, in a step S5, the surroundings representation is generated based on the at least one camera image 12 and the image data used to fill the hidden surroundings. The image data previously determined in step S4 are inserted into the hidden surroundings region 14, so that, particularly when displaying the surroundings representation based on the position of a virtual camera, this region can be at least partially represented with image data that at least approximately describe the actual surroundings of the vehicle 1.

[0076] Fig. 4 shows an environmental representation 15. The environmental representation 15 was generated based on a virtual camera position, so that the viewing angle of the vehicle's surroundings does not depend on the actual position of one of the cameras 3. This allows the vehicle 1, or a virtual model of the vehicle 1, to also be shown in the environmental representation. The image data for representing at least part of the concealed environmental area 14 can be transformed, in particular depending on the virtual camera position, in order to compensate for the difference in perspective.

[0077] In the environmental representation 15 shown in Fig. 4, the image data was used to fill the partial areas of the hidden environmental region 14 hidden by the objects 13 in the form of posts with data about the actual vehicle surroundings. Using the image data enables the driver to view a complete image of the vehicle surroundings that has no or at least fewer gaps and / or black areas or areas not filled with image data. Preferably, image data for all hidden areas 14 are determined for the environmental representation 15, so that a complete, gap-free reproduction of the environment is possible. If image data is not available for all hidden environmental regions 14, the remaining hidden partial areas for which no image data is available can, for example, be displayed in black or filled by extrapolating the respective surrounding image areas.

[0078] Furthermore, it is possible for the image data and the at least one camera image 12 to be combined to generate the environmental representation 15 depending on at least one image harmonization function. Using image harmonization, for example, different brightnesses, different color saturations, and the like can be compensated to generate a harmonious environmental representation 15.

[0079] The environmental representation 15 shown in Fig. 4 is purely exemplary; in particular, other types of environmental representations, for example, environmental representations 15 recorded at a different angle or from a different virtual camera position, can also be generated. For example, it is possible for a top view, which shows a bird's-eye view of the vehicle 1, to be generated as the environmental representation 15.

[0080] The environmental representation 15 can be displayed to a driver of the vehicle 1 on the display device 9, allowing the driver 1 to view it for performing vehicle maneuvers. This significantly simplifies the execution of precise driving maneuvers, particularly maneuvers performed in confined spaces such as parking maneuvers. In addition, the environmental representation 15 can also be used for other driver assistance functions of the vehicle 1, which include, for example, tracking objects, landmarks, or obstacles in the vicinity of the vehicle 1.

Claims

Patent claims 1 . Method for generating an environmental representation (15) representing a vehicle environment, comprising the steps: - Providing camera images (12) recorded with one or more cameras (3) of the vehicle (1), wherein at least part of the vehicle surroundings is shown in each of the camera images (12), - generating three-dimensional environmental data from at least one of the camera images (12), wherein the at least one camera image (12) was recorded at a first recording time, - determining a hidden environmental area (14) for at least one of the cameras from the environmental data, - Determining image data for at least partially filling the concealed surrounding area (14), wherein the image data are determined from at least one further camera image of another of the cameras (3) recorded at the first recording time and / or from at least one stored camera image of the one or more cameras (3) recorded at a second recording time prior to the first recording time, - generating the environmental representation from the at least one camera image (12) and the image data, wherein at least a part of the environmental region (14) concealed in the at least one camera image (12) is represented in the environmental representation (15) using the image data.

2. Method according to claim 1, characterized in that it is first checked whether the image data for filling the concealed surrounding area (14) are present in at least one further camera image taken at the first recording time from another of the cameras (3), wherein if the image data from the at least one further camera image, these image data are used to generate the environmental representation (15), and wherein, if the image data from the at least one further camera image are not present, the image data from the at least one stored camera image are used to generate the environmental representation (15).

3. Method according to claim 1 or 2, characterized in that the image data for filling the concealed surrounding area (14) are determined on the basis of the surrounding data, wherein the concealed surrounding area (14) of the at least one camera (3) at the first recording time is compared with the concealed surrounding area (14) of the at least one further camera (3) and / or with concealment information assigned to the stored camera image, which describes the concealed surrounding area (14) of the camera (3) generating the stored camera image at the second recording time.

4. Method according to one of the preceding claims, characterized in that the environmental data are determined as a function of depth information associated with the camera image (12), as a function of map information describing the position of the vehicle (1) and the vehicle environment and / or as a function of the measurement data of at least one distance sensor (10) of the vehicle (1).

5. Method according to one of the preceding claims, characterized in that the environmental representation (15) is generated starting from a virtual camera position, wherein the image data for representing the at least part of the hidden environmental area (14) are transformed depending on the virtual camera position.

6. Method according to one of the preceding claims, characterized in that the image data and the at least one camera image (12) are combined to generate the environmental view (15) depending on at least one image harmonization function.

7. Method according to one of the preceding claims, characterized in that several cameras (3) of a camera arrangement designed as a surround view system of the vehicle (1) are used.

8. Control device configured to carry out a method according to one of the preceding claims.

9. Driver assistance system comprising at least one camera (3) which is designed to provide camera images representing at least part of a vehicle environment, and a control device (4) according to claim 8.

10. Vehicle comprising a driver assistance system (2) according to claim 9.

Images