Method for generating an environment view for a vehicle, control device, driver assistance system and vehicle

Abstract

The invention relates to a method for generating an environment view (14) for a vehicle (1), wherein the environment view (14) shows at least one environment area of the vehicle (1), wherein the environmental view is generated from at least one current camera image of the environment area and from image data stored in a storage device (12), wherein the at least one current camera image is generated from at least one environment camera (3) of the vehicle (1), wherein the stored image data comprises camera images of the environment area generated by the at least one environment camera (3) of the vehicle (1) during an earlier stay of the vehicle (1) in the environment area, and / or camera images of the environment area generated by at least one further environment camera of a further vehicle.

Description

[0001] Description

[0002] Method for generating an environmental view for a vehicle, control device, driver assistance system and vehicle

[0003] The invention relates to a method for generating an environmental view for a vehicle, wherein the environmental view represents at least one environmental area of ​​the vehicle. The environmental view is generated from at least one current camera image of the environmental area and from image data stored in a storage device. The at least one current camera image is generated by at least one environmental camera of the vehicle. Furthermore, the invention relates to a control device, a driver assistance system, and a vehicle.

[0004] To facilitate navigation and / or maneuvering for vehicle drivers, modern vehicles can display a view of the surroundings on a display device. Such a view of the surroundings can be generated using one or more of the vehicle's cameras and, for example, enable an overhead view of the vehicle and its immediate surroundings. Furthermore, views of the surroundings can also be generated from satellite images, which provide an overhead view of the area in which the vehicle is currently located. Various approaches to this are known from the state of the art.

[0005] DE 10 2010 042 063 A1 discloses a method for determining processed image data about the surroundings of a vehicle. A satellite image is combined with a position and orientation of the vehicle and with a top view of the surroundings generated from camera images of the vehicle. The satellite image can be stored in a database and transmitted to the vehicle via an internet link. To display the surroundings, the top view generated from the camera images can be semi-transparently superimposed on the satellite image. WO 2012 / 019901 A1 describes a method for displaying images on a display device of a driver assistance system in a motor vehicle. Navigation data is received by the driver assistance system, and sensor data about the surroundings of the motor vehicle is acquired by at least one sensor of the driver assistance system.The acquired sensor data is processed into an image of the surroundings, which shows at least one area surrounding the motor vehicle. A map image is displayed on the display device, which is generated from stored map data or satellite images, taking the navigation data into account. The map image is enlarged or zoomed in on the display device, and after reaching a predetermined image scale or zoom factor of the map image, the display switches from the map image to the image of the surroundings.

[0006] However, using satellite images can have the disadvantage that they are no longer up-to-date and therefore do not accurately depict the vehicle's surroundings. Furthermore, the satellite images of the current vehicle surroundings may be available at a significantly lower resolution than the images captured by the vehicle's cameras, which can limit the image quality of the surrounding view.

[0007] The invention is based on the object of specifying an improved method for generating an environmental view, which in particular enables a more up-to-date environmental representation and / or an environmental representation with improved image quality.

[0008] To achieve this object, the invention provides for a method of the type mentioned at the outset that the stored image data, when the vehicle was previously in the surrounding area, comprise camera images of the surrounding area generated by at least one surrounding camera of the vehicle and / or camera images of the surrounding area generated by at least one further surrounding camera of another vehicle. The surrounding view to be generated can, for example, be a plan view comprising a vehicle model or a vehicle graphic as well as a representation of the immediate vehicle surroundings, in particular 360° around the vehicle. The vehicle surroundings are recorded by at least one surrounding camera of the vehicle and depicted in at least one camera image of the surrounding camera.

[0009] The surround view camera is in particular a camera mounted on the vehicle which records at least a partial area of ​​the vehicle's surroundings. The vehicle can comprise one or more surround view cameras, which can each be designed as a front camera, a side camera, a rear camera or a reversing camera, for example. The at least one surround view camera in particular continuously generates current camera images in which the surrounding area to be displayed in the surrounding view is displayed in full or at least in part. In particular, the vehicle can have a plurality of surround view cameras with which the vehicle's surroundings around the vehicle can be recorded completely, i.e. in 360° around the vehicle. The plurality of surround view cameras can be designed, for example, as a surround view system.

[0010] The camera images recorded by the at least one surroundings camera of the vehicle are used to generate the surroundings view. In addition, image data stored in a storage device are used to generate the surroundings view. The stored image data can be camera images recorded by the at least one surroundings camera during an earlier stay of the vehicle in the surroundings area, i.e. at an earlier point in time. At the earlier point in time, the images can, for example, have been transmitted from the vehicle to the storage device or stored therein, so that these camera images can be used at a later point in time. The earlier stay of the vehicle in the surroundings area or the earlier point in time can, in particular, relate to a previous drive through the surroundings area and can, for example, have taken place minutes, hours, days, weeks or months earlier.

[0011] Additionally or alternatively, the stored image data includes camera images of the surrounding area generated by at least one additional surrounding camera of another vehicle. These additional camera images may also have been stored in the storage device by the additional vehicle at an earlier time during a stay in the surrounding area.

[0012] The use of stored image data generated by the vehicle or another vehicle using a surroundings camera has the advantage that this data can be created more frequently and, thanks to storage in the storage device, can also be updated more frequently than satellite images or similar. Particularly for vehicles that regularly travel along the same or at least a similar route, images of the same surrounding area can thus be regularly generated, which can then be used by the vehicle itself and / or other vehicles to create a view of the surroundings. Advantageously, this allows image data to be stored in the storage device that is generally more up-to-date than images such as satellite images, which are difficult to generate.

[0013] Furthermore, using the stored image information has the advantage that the surrounding view can be generated with higher image quality, since image data captured by a vehicle using surround-view cameras reproduces at least the immediate surroundings of the vehicle with a higher resolution or better image quality than satellite images. This makes it possible to reproduce the surrounding view, at least in certain areas, with improved image quality or a higher image resolution, and thus with more detail.

[0014] By using the stored image data, it is advantageously possible for a first sub-area of ​​the surrounding area, which is visible to the at least one surrounding camera, to be displayed using the at least one camera image, and a second sub-area, which is not visible to the at least one surrounding camera, to be displayed using the stored image data. In this context, a non-visible area can be understood, for example, as an area lying outside the detection range of the at least one surrounding camera and / or an area lying within the detection range, but which is obscured by an object in the surrounding camera.In other words, the use of the stored image data makes it possible to display a larger surrounding area, which can also include areas that cannot currently be seen, or at least not completely, by the vehicle's surrounding cameras.

[0015] According to the invention, it can be provided that a vehicle-external storage device is used as the storage device, wherein the vehicle is communicatively connected to the storage device via a communication link. The storage device can be, for example, a server or similar device that is connected to the vehicle via an internet connection. The use of a vehicle-external storage device has the advantage that it can be easily accessed by a large number of other vehicles. Thus, updated image data of a specific environmental area from many vehicles can be stored in the storage device, and this image data can be retrieved by a large number of other vehicles.

[0016] In order to reduce the amount of data in the storage device, the invention can provide that, depending on the fulfillment of at least one storage criterion, the image data of the surrounding area stored in the storage device are updated and / or supplemented with the at least one current camera image of the surrounding area. In other words, the storage criterion determines whether camera images generated by the vehicle or another vehicle are stored in the storage device.In a preferred embodiment of the invention, it can be provided that a storage criterion is used which is met when the current camera image of the surrounding area has a better image quality, in particular a higher resolution, than the stored image data of the surrounding area, that a storage criterion is used which is met when the current camera image represents a surrounding area which is not or only partially contained in the stored image data, and / or that a storage criterion is used which is met when at least a predetermined time period has passed between the recording time of the current camera image of the surrounding area and a recording time of the stored image data of the surrounding area.

[0017] A current camera image can therefore be transmitted to the control device or stored in the storage device if it reproduces the surrounding area with better image quality or if the surrounding area is not included or only partially included in the stored image data. Furthermore, by specifying a minimum time period between recordings, it can be avoided that a large number of camera images of the same surrounding area are stored in the storage device within a very short period of time or are used to update the data already stored in the storage device. The specified time period can, for example, be at least one or more minutes.

[0018] According to the invention, it can be provided that the surroundings view is a top view, wherein a first partial area of ​​the surroundings lying within a predetermined distance around the vehicle is displayed using the at least one current camera image and a second partial area of ​​the surroundings lying outside the predetermined distance around the vehicle is displayed using the stored image data in the surroundings view. For the first partial area of ​​the surroundings lying within the predetermined distance from the vehicle, the camera images of the vehicle can advantageously be used, since these depict the first partial area up-to-date and with high image quality. For second partial areas lying further away, the stored image data which were determined at an earlier time by the vehicle and / or another vehicle can be used.In this way, the top view can be expanded beyond the area displayed by the camera images, thus encompassing additional sub-areas of the surrounding area. This allows the top view to have a smaller zoom level or, for example, to zoom out from a view with a higher zoom level, while the additional sub-areas to be displayed, for which stored image data is available, can still be reproduced with high quality in the surrounding view.

[0019] In a preferred embodiment of the invention, the stored image data can contain transformed camera images and / or representations generated from one or more camera images using projection. In particular, if the environmental view to be generated is a top view of the vehicle and the vehicle's surroundings, the creation of the environmental view can be simplified by storing camera images already transformed into a top view. In addition to individual transformed camera images, representations derived from multiple camera images using a projection of the camera images, in particular environmental views executed as a top view, can also be stored.This makes it easy to create the environmental view by appending the stored image data to a top view determined from the current camera images without requiring any transformation of the other camera images and / or a representation derived from them. This also applies analogously to other environmental views not executed as top views.

[0020] According to the invention, it can be provided that the stored image data additionally include satellite images of the surrounding area. The additional provision of satellite images as part of the stored image data has the advantage that even areas not visible to vehicles can be displayed in the surrounding view. Thus, for example, a top view or the like can also be extended to include parts of the surrounding area that cannot be seen by the vehicle or other vehicles, for example, parts of the surrounding area that are not visible from a road.

[0021] According to the invention, the vehicle can comprise a position-determining device, in particular a satellite navigation device, wherein the stored image data is selected depending on current position information generated by the position-determining device. The position information represents, for example, an absolute position or a coordinate position of the vehicle and / or an orientation of the vehicle, so that image data stored for the surrounding area can be retrieved from the storage device to match the current vehicle position. The stored image data in the storage device can also be provided with a position indication, in particular with an indication of the position of the vehicle or of the other vehicle when the corresponding camera images are generated, whereby an assignment of the stored image data to the current vehicle position is easily possible.

[0022] Using a satellite navigation device for position determination has the advantage of enabling simple position determination. Alternatively, however, other methods for position determination can also be used, for example, methods based on a map and odometry, determining the position by evaluating the image content of at least one camera image, or similar methods.

[0023] It is provided for a control device according to the invention that it is set up to carry out a method according to the invention.

[0024] A driver assistance system according to the invention is provided to comprise at least one surroundings camera and a control device according to the invention. A vehicle according to the invention is provided to have a driver assistance system according to the invention.

[0025] All advantages and embodiments described above with respect to the method according to the invention apply analogously to the control device according to the invention, the driver assistance system according to the invention, and the vehicle according to the invention, and vice versa. The advantages and embodiments described for the control device according to the invention, the driver assistance system according to the invention, and the vehicle according to the invention can also be applied analogously to the other subject matters of the invention.

[0026] Further advantages and details of the invention will become apparent from the exemplary embodiments described below and from the drawings. These are schematic representations and show:

[0027] Fig. 1 shows an embodiment of a vehicle according to the invention,

[0028] Fig. 2 is a block diagram for explaining an embodiment of a method according to the invention and

[0029] Fig. 3 is a schematic representation of an environmental view to explain the embodiment of the method according to the invention.

[0030] Fig. 1 shows an embodiment of a vehicle 1. The vehicle 1 can be, for example, a motor vehicle, in particular a passenger car, a truck or another type of commercial vehicle. It is also possible for the vehicle 1 to be a non-motorized vehicle such as a trailer or for it to be a combination consisting of a towing vehicle and a trailer. The vehicle 1 can also be a rail-bound vehicle, such as a tram or the like, or a robot. The vehicle 1 comprises an embodiment of a driver assistance system 2, which comprises one or more surroundings cameras 3 and a control device 4. In the present embodiment, the vehicle 1 comprises four surroundings cameras 3, which form a camera arrangement designed as a surround view system.A first surroundings camera 5 is arranged as a front camera of the vehicle 1, a second surroundings camera 6 as a rearview camera of the vehicle 1, and a third surroundings camera 7 and a fourth surroundings camera 8 are each arranged as a side camera of the vehicle 1. The control device 4 can be designed, for example, as a microcontroller, as a processor, or as another type of computing device.

[0031] The vehicle 1 further comprises a display device 9, via which graphic information can be presented to a driver or a user of the vehicle 1. The display device 9 can, for example, be one or more screens or displays arranged in an interior of the vehicle 1.

[0032] By means of the surrounding cameras 3, image data can be generated from a sub-area of ​​the surroundings of the vehicle 1. The surrounding cameras 3 are connected to the control device 4 via a communication connection 10. The communication connection 10 can, for example, comprise a plurality of point-to-point connections or be a bus connection such as a CAN bus or the like. Camera images or image data are transmitted, in particular continuously, from the surrounding cameras 3 to the control device 4 via the communication connection 10. At least some of the camera images and / or a surrounding view generated from several camera images can then be displayed to a driver of the vehicle 1 on the display device 9.

[0033] The vehicle 1 further comprises at least one communication unit 11, via which the vehicle 1 can communicate with an external storage device 12 via a further, particularly wireless, communication connection, for example, an internet connection. The external storage device 12 can be implemented, for example, as a server on which image data can be stored and retrieved by the vehicle 1 and other vehicles.

[0034] The vehicle 1 further comprises a position-determining device 13, which is embodied, for example, as a satellite navigation device, in particular as a GPS unit. The position-determining device 13 is designed to generate position information that uniquely describes the current position of the vehicle 1.

[0035] Fig. 2 shows a block diagram of an embodiment of a method for generating an environmental view. The control device 4 of the driver assistance system 2 is configured to carry out such a method, wherein the environmental view represents at least one area of ​​the vehicle's surroundings. The start of the method can be triggered, for example, by a user selection by a driver of the vehicle 1, for example, via a user interface on the display device 9 or the like.

[0036] In step S1, at least one current camera image of the surrounding area of ​​the vehicle 1 to be displayed in the surrounding view is recorded. The camera image is recorded with one or more of the surrounding cameras 3 of the vehicle 1. In particular, camera images can be recorded by several or all of the surrounding cameras 3, for example, to generate a surrounding view designed as a top view of the vehicle 1.

[0037] In step S2, stored image data are retrieved from the storage device 12, which are also used to generate the surrounding view. The stored image data comprise camera images recorded by one or more of the surrounding cameras 3 during a previous stay of the vehicle 1 in the surrounding area and / or camera images of the surrounding area generated by at least one further surrounding camera of another vehicle. The stored image data can be present in the storage device 12 as transformed camera images. Additionally or alternatively, the stored image data can also contain representations generated from one or more camera images using projection, such as top views of the surrounding area or the like. It is also possible for the stored image data to contain satellite images.

[0038] In order to retrieve appropriate stored image data for the current surrounding area of ​​the vehicle 1 from the storage device, positioning information describing the current position and / or orientation of the vehicle can be generated using the position determination device 13. This can be used to retrieve stored image data, which are assigned an identical or at least a similar or adjacent position and / or orientation, from the storage device 12 and to transmit them to the control device 4.

[0039] In step S3, the surrounding view is generated from the at least one current camera image and the stored image data, as described in more detail below. Subsequently, in step S4, the surrounding view is displayed to a driver of the vehicle 1, for example, by displaying the surrounding view on the display device 9 of the vehicle 1.

[0040] The environmental view determined by the control device 4 is based on the current camera images and the stored image data extracted from the storage device 12. In order to combine the image data extracted from the storage device 12 with the camera images of the vehicle 1, a top view of the vehicle 1 can be determined, for example, from the camera images of the vehicle 1. A top view of the environmental view can be created, for example, by projecting the camera images onto a bowl-shaped projection surface, for example, wherein the top view can then be generated from a virtual camera that looks at the projection surface. The position of the vehicle 1 and its orientation can be determined using the position determination device.The stored image data, which in particular also contain position information and / or orientation information, can then be appended to the top view generated from the current camera images. A scaling factor for the top view can be determined from the camera images and / or the top view generated from them, which, for example, indicates the actual length of one of the pixels or image points in the camera image or top view. The scaling factor can, for example, be derived directly from the calculation rules used to determine the top view from the current camera images.

[0041] To enable stitching, the scaling factor of the stored image data for a sub-area of ​​the surrounding area to be imaged can be evaluated. The scaling factor assigned to the respective image data can also be stored in the storage device 12 and retrieved by the control device 4 of the vehicle 1. By scaling the current camera images and / or the stored image data, they can be brought to the same magnification scale so that they can be stitched together.

[0042] Based on a vehicle orientation determined by means of the position determination device 13 and / or by an evaluation of the image content of the current camera images and the stored image data, it can be recognized which orientation the current camera images, or a top view generated therefrom, have in relation to the image data taken from the storage device 12.

[0043] If necessary, an orientation adjustment can be achieved by rotating the camera images or the top view derived therefrom and / or the image data taken from the storage device 12. The stored image data can then be supplemented with the top view determined from the current camera images. This makes it possible, for example, to display a larger surrounding area around the vehicle than would be possible using only the current camera images of the vehicle 1. Since the stored image data from the vehicle 1 include camera images taken during one or more previous visits to the surrounding area and / or camera images of the surrounding area generated by other vehicles, image data is available that is generally more current and / or of better image quality than satellite images.Areas where no such image data is available can be supplemented by satellite images in order to enable a representation of parts of the surrounding area that were not recorded during previous passes and / or by other vehicles.

[0044] Fig. 3 shows an exemplary embodiment of an environmental view 14 generated using the method. This contains, for example, a vehicle graphic 15 depicting the vehicle 1 from above. The environmental view 14 comprises a first partial area 16 of the environmental area to be imaged, which lies within a predetermined distance around the vehicle 1, and a second partial area 17, which lies outside the predetermined distance around the vehicle 1. The current camera images acquired with the environmental cameras 3 of the vehicle 1 can be used to display the first partial area 16. These images show the current state of the environmental area to be displayed, since they are generated by the environmental cameras 3, in particular continuously as a video stream. The second partial area 17 can be filled using the stored image data recorded by the vehicle 1 and / or another vehicle.In this way, the top view of vehicle 1 can be displayed at a lower zoom level than would be possible based solely on the current camera images.

[0045] To achieve an appealing appearance of the surroundings view, the boundary area between the first partial area 16 and the second partial area 17 can be processed using a blending process, for example, using alpha blending or the like, and / or using blurring or blurring. This can particularly improve the image quality of the surroundings view if part of the stored image data has a different resolution and / or brightness and / or if it is, for example, satellite data that is used to fill in a portion of the surroundings to be displayed that has not yet been captured by vehicle cameras.

[0046] The image data stored in the storage device 12 may have been generated by the vehicle 1 during a previous pass through the surrounding area to be displayed. The previous pass may have occurred at an earlier point in time, for example, several minutes, hours, days, weeks, or months ago. The image data recorded during this pass may have been transmitted to the storage device 12 via the communication connection.

[0047] Other vehicles that are also configured to perform such a method can also transmit camera images to the storage device 12, so that they are available to all vehicles that have a correspondingly configured control device. The camera images generated by vehicle 1 or the other vehicles can be used, in particular, to update and / or supplement satellite images stored in the storage device 12.

[0048] In order to determine whether current camera images currently acquired by the vehicle 1 or the surrounding area cameras 3 of the vehicle 1 should be used to update and / or supplement image data already stored in the storage device 12, it can be evaluated, in particular by the control device 4, whether at least one storage criterion is met. In this case, a storage criterion can be used which is met if the current camera image of the surrounding area has better image quality than the image data of the surrounding area stored in the storage device 12. Better image quality in this context can be, for example, a higher resolution. Additionally or alternatively, a storage criterion can be used which is met if the current camera image represents a surrounding area that is not or only partially contained in the stored image data.It is also possible to use a storage criterion which is met if at least a predetermined time period has passed between the recording time of the current camera image of the surrounding area and a recording time of the stored image data of the surrounding area.

[0049] In addition to using a vehicle-external storage device 12, it is also possible to use a vehicle-internal storage device 12. This can, in particular, contain stored image data generated during previous passages through the surrounding area.

[0050] In addition to an environmental view formed as a top view, other types of environmental views, ie environmental views from a different angle or the like, can also be generated as part of the process for generating the environmental view.

Claims

Patent claims 1. A method for generating an environmental view (14) for a vehicle (1), wherein the environmental view (14) represents at least one environmental region of the vehicle (1), wherein the environmental view (14) is generated from at least one current camera image of the environmental region and from image data stored in a storage device (12), wherein the at least one current camera image is generated by at least one environmental camera (3) of the vehicle (1), characterized in that the stored image data comprise, during a previous stay of the vehicle (1) in the environmental region, camera images of the environmental region generated by the at least one environmental camera (3) of the vehicle (1) and / or camera images of the environmental region generated by at least one further environmental camera of a further vehicle.

2. Method according to claim 1, characterized in that a vehicle-external storage device is used as the storage device (12), wherein the vehicle (1) is communicatively connected to the storage device (12) via a communication connection.

3. Method according to claim 1 or 2, characterized in that, depending on the fulfillment of at least one storage criterion, image data of the surrounding area stored in the storage device (12) are updated and / or supplemented with the at least one current camera image of the surrounding area.

4. Method according to claim 3, characterized in that a storage criterion is used which is met when the current camera image of the surrounding area has a better image quality, in particular a higher resolution, than the stored image data of the surrounding area, that a storage criterion is used which is met when the current camera image represents a surrounding area which is not or only partially contained in the stored image data, and / or that a storage criterion is used which is met when at least a predetermined time period has passed between the recording time of the current camera image of the surrounding area and a recording time of the stored image data of the surrounding area.

5. Method according to one of the preceding claims, characterized in that the surrounding view (14) is a plan view, wherein a first partial area (16) of the surrounding area lying within a predetermined distance around the vehicle (1) is displayed using the at least one current camera image and a second partial area (17) of the surrounding area lying outside the predetermined distance around the vehicle (1) is displayed using the stored image data in the surrounding view (14).

6. Method according to one of the preceding claims, characterized in that the stored image data contain transformed camera images and / or representations generated from one or more camera images using projection.

7. Method according to one of the preceding claims, characterized in that the vehicle (1) comprises a position determination device (13), in particular a satellite navigation device, wherein the stored image data are selected depending on current position information generated by the position-determining device (13).

8. Control device configured to carry out a method according to one of the preceding claims.

9. Driver assistance system for a vehicle (1), comprising at least one surroundings camera (3) and a control device (4) according to claim 8.

10. Vehicle comprising a driver assistance system (2) according to claim 9.

Images