Method for generating an environment view for a vehicle, control unit, driver assistance system and vehicle

Abstract

Method for generating an environment view (14) for a vehicle (1), wherein the environment view (14) represents at least one environment area of ​​the vehicle (1), wherein the environment view is generated from at least one current camera image of the environment area and from image data stored in a storage device (12), wherein the at least one current camera image is generated by at least one environment camera (3) of the vehicle (1), wherein the stored image data comprise camera images of the environment area generated by the at least one environment camera (3) of the vehicle (1) during a previous stay of the vehicle (1) in the environment area and / or camera images of the environment area generated by at least one further environment camera of another vehicle.

Description

[0001] The invention relates to a method for generating a view of the surroundings of a vehicle, wherein the view represents at least one area of ​​the vehicle's surroundings, and wherein the view is generated from at least one current camera image of the surroundings and from image data stored in a storage device, the at least one current camera image being generated by at least one camera of the vehicle's surroundings. The invention further relates to a control unit, a driver assistance system, and a vehicle.

[0002] To facilitate navigation and / or maneuvering for drivers, modern vehicles can display a surround view on a screen. Such a view can be generated using one or more of the vehicle's cameras and, for example, provide a top-down view of the vehicle and its immediate surroundings. Alternatively, surround views can be generated from satellite imagery, offering a top-down view of the area where the vehicle is currently located. Several approaches to achieving this are known in the prior art.

[0003] German patent DE 10 2010 042 063 A1 discloses a method for determining processed image data about a vehicle's surroundings. This involves combining a satellite image with the vehicle's position and orientation, as well as with a top-down view of the surroundings generated from the vehicle's camera images. The satellite image can be stored in a database and transmitted to the vehicle via an internet link. To display the surroundings, the satellite image can be overlaid semi-transparently with the top-down view generated from the camera images.

[0004] WO 2012 / 019901 A1 describes a method for displaying images on a display unit of a driver assistance system in a motor vehicle. Navigation data is received by the driver assistance system, and sensor data about the vehicle's surroundings are acquired by at least one sensor of the driver assistance system. The acquired sensor data is processed into an image of the surroundings, which shows at least one area adjacent to the vehicle. A map image is displayed on the display unit, which is generated from stored map data or satellite imagery, taking the navigation data into account. The map image is enlarged or zoomed in on the display unit, and once a predetermined scale or zoom factor of the map image is reached, the display switches from the map image to the image of the surroundings.

[0005] However, using satellite imagery can have the disadvantage that it may be outdated and therefore not accurately depict the vehicle's surroundings. Furthermore, satellite images of the current vehicle environment may only be available with a significantly lower resolution than images captured by the vehicle's cameras, which can limit the image quality of the view.

[0006] The invention is based on the objective of providing an improved method for generating an environment view, which in particular enables a more up-to-date environment display and / or an environment display with improved image quality.

[0007] To solve this problem, in a method of the type mentioned at the outset, it is provided according to the invention that the stored image data comprise camera images of the surrounding area generated by at least one surrounding camera of the vehicle and / or camera images of the surrounding area generated by at least one further surrounding camera of another vehicle during a previous stay of the vehicle in the surrounding area.

[0008] The environmental view to be generated can, for example, be a top-down view comprising a vehicle model or graphic and a representation of the immediate surroundings, in particular a 360° view around the vehicle. The vehicle's surroundings are captured by at least one surround-view camera and displayed in at least one image from that camera.

[0009] A surround-view camera is, in particular, a camera mounted on the vehicle that captures at least a portion of the vehicle's surroundings. The vehicle can include one or more surround-view cameras, which can be configured, for example, as a front camera, a side camera, or a rear camera / reversing camera. The at least one surround-view camera continuously generates updated camera images, in which the area to be displayed in the surround-view view is shown in whole or in part. In particular, the vehicle can have multiple surround-view cameras that can capture the entire area around the vehicle, i.e., in 360°. The multiple surround-view cameras can, for example, be configured as a surround-view system.

[0010] The camera images captured by the vehicle's at least one surround-view camera are used to generate the surround-view view. Additionally, image data stored in a storage device is used to generate this view. This stored image data can include camera images captured by the at least one surround-view camera during a previous period when the vehicle was in the surrounding area, i.e., at an earlier time. At that earlier time, the images may have been transmitted by the vehicle to the storage device or stored therein, allowing these camera images to be used at a later time. The previous period when the vehicle was in the surrounding area, or the earlier time, can refer in particular to a previous passage through the surrounding area and may have occurred, for example, minutes, hours, days, weeks, or months earlier.

[0011] Additionally or alternatively, the stored image data includes camera images of the surrounding area generated by at least one further surround-view camera of another vehicle. These additional camera images may also have been stored in the storage device by the other vehicle during a previous period when it was in the surrounding area.

[0012] Using stored image data generated by the vehicle or another vehicle using a surround-view camera has the advantage that this data can be created more frequently and updated more often by storing it in the storage device than satellite images or similar sources. Particularly for vehicles that regularly travel along the same or at least a similar route, images of the same surrounding area can be generated regularly. These images can then be used by the vehicle itself and / or by other vehicles to create a visual representation of the surroundings. This allows for the advantageous storage of image data in the storage device that is generally more up-to-date than images like satellite images, which are only generated with considerable effort.

[0013] Furthermore, using stored image information has the advantage of allowing the environmental view to be generated with higher image quality, since image data captured by a vehicle using surround-view cameras depicts at least the immediate surroundings of the vehicle with higher resolution and better image quality than satellite images. This makes it possible to reproduce the environmental view, at least in certain areas, with improved image quality and higher resolution, and thus with more detail.

[0014] By using the stored image data, it is advantageously possible to display, in the environment view, a first sub-area of ​​the environment, visible to the at least one environment camera, using the at least one camera image, and a second sub-area, not visible to the at least one environment camera, using the stored image data. In this context, an area not visible can be understood, for example, as an area lying outside the detection range of the at least one environment camera and / or an area lying within the detection range but obscured by an environment object as seen from the perspective of the environment camera.In other words, using the stored image data makes it possible to display a larger area of ​​the environment, which can also include areas that are currently not visible, or at least not completely visible, to the vehicle's surround-view cameras.

[0015] According to the invention, an external storage device can be used, with the vehicle being communicatively connected to the storage device via a communication link. The storage device can, for example, be a server or similar device connected to the vehicle via an internet connection. Using an external storage device has the advantage that it can be easily accessed by a large number of other vehicles. Thus, updated image data of a specific area can be stored in the storage device by many vehicles, and this image data can be retrieved by a large number of other vehicles.

[0016] To reduce the amount of data in the storage device, the invention provides that, depending on the fulfillment of at least one storage criterion, image data of the surrounding area stored in the storage device are updated and / or supplemented with at least one current camera image of the surrounding area. In other words, the storage criterion determines whether camera images generated by the vehicle or another vehicle are stored in the storage device.

[0017] In a preferred embodiment of the invention, it may be provided that a storage criterion is used which is fulfilled if the current camera image of the surrounding area has a better image quality, in particular a higher resolution, than the stored image data of the surrounding area; that a storage criterion is used which is fulfilled if the current camera image represents a surrounding area which is not included or only partially included in the stored image data; and / or that a storage criterion is used which is fulfilled if at least a predetermined time interval has elapsed between the recording time of the current camera image of the surrounding area and a recording time of the stored image data of the surrounding area.

[0018] A current camera image can therefore be transmitted to the control unit or stored in the storage device if it depicts the surrounding area with better image quality, or if the surrounding area is not included, or only partially included, in the stored image data. Furthermore, by specifying a minimum time interval between recordings, it can be prevented that a large number of camera images of the same surrounding area are stored in the storage device within a very short time, or used to update data already stored in the storage device. This specified time interval can, for example, be at least one minute or several minutes.

[0019] According to the invention, the environmental view can be a top view, wherein a first sub-area of ​​the environmental area, located within a predetermined distance around the vehicle, is displayed using at least one current camera image, and a second sub-area of ​​the environmental area, located outside the predetermined distance around the vehicle, is displayed using the stored image data. For the first sub-area of ​​the environmental area located within the predetermined distance of the vehicle, the camera images from the vehicle can advantageously be used, as these depict the first sub-area in real time and with high image quality. For the more distant, second sub-areas, the stored image data, which was acquired at an earlier time by the vehicle and / or another vehicle, can be used.In this way, the view can be extended beyond the area depicted by the camera images, thus encompassing additional parts of the surrounding environment. This allows the view to be displayed at a lower zoom level, or, for example, to zoom out from a view with a higher zoom level, while the additional areas to be displayed, for which stored image data exists, can still be reproduced in high quality in the environmental view.

[0020] In a preferred embodiment of the invention, the stored image data may contain transformed camera images and / or representations generated from one or more camera images using projection. In particular, if the environmental view to be generated is a top-down view of the vehicle and its surroundings, the creation of the environmental view can be simplified by storing camera images that have already been transformed into a top-down view. In addition to individual transformed camera images, representations derived from multiple camera images using projection, especially top-down environmental views, may also be stored.This makes it easy to create an environment view by simply appending the stored image data to a top-down view derived from the current camera images, without requiring any transformation of the other camera images and / or a derived representation. This also applies analogously to other environment views not executed as top-down views.

[0021] According to the invention, the stored image data can additionally include satellite images of the surrounding area. Providing satellite images as part of the stored image data has the advantage that areas not visible to vehicles can also be displayed in the environmental view. Thus, for example, a bird's-eye view or similar can be extended to include parts of the surrounding area that cannot be seen by the vehicle or other vehicles, such as areas not visible from a road.

[0022] According to the invention, the vehicle may include a positioning device, in particular a satellite navigation device, wherein the stored image data are selected based on current position information generated by the positioning device. This position information represents, for example, the absolute position or coordinate position of the vehicle and / or its orientation, so that image data stored in the storage device can be retrieved from the storage device that corresponds to the current vehicle position in the surrounding area. The stored image data in the storage device may also include position information, in particular the position of the vehicle or of other vehicles when the corresponding camera images were generated, thus enabling a simple assignment of the stored image data to the current vehicle position.

[0023] Using a satellite navigation system for position determination has the advantage of enabling simple position fixing. Alternatively, other methods for position determination can also be used, such as map-based and odometry-based methods, determining the position by analyzing the image content of at least one camera image, or similar techniques.

[0024] A control device according to the invention is provided that it is configured to carry out a method according to the invention.

[0025] For a driver assistance system according to the invention, it is provided that it comprises at least one surround view camera and a control unit according to the invention.

[0026] A vehicle according to the invention is provided that it has a driver assistance system according to the invention.

[0027] All advantages and embodiments described above in relation to the method according to the invention apply analogously to the control device, the driver assistance system, and the vehicle according to the invention, and vice versa. The advantages and embodiments described for the control device, the driver assistance system, and the vehicle according to the invention are also transferable analogously to the other inventions.

[0028] Further advantages and details of the invention will become apparent from the exemplary embodiments described below and from the drawings. These are schematic representations and show: Fig. 1 an embodiment of a vehicle according to the invention, Fig. 2 a block diagram to illustrate an embodiment of a method according to the invention and Fig. 3 A schematic representation of an environmental view to illustrate the embodiment of the method according to the invention.

[0029] In Fig. Figure 1 shows an embodiment of a vehicle 1. The vehicle 1 can be, for example, a motor vehicle, in particular a passenger car, a truck, or another type of commercial vehicle. It is also possible that the vehicle 1 is an unmotorized vehicle such as a trailer or that it is a combination of a towing vehicle and a trailer. The vehicle 1 can also be a rail-bound vehicle, such as a tram or the like, or a robot.

[0030] The vehicle 1 comprises an embodiment of a driver assistance system 2, which includes one or more surround-view cameras 3 and a control unit 4. In the present embodiment, the vehicle 1 comprises four surround-view cameras 3, which form a camera arrangement configured as a surround-view system. A first surround-view camera 5 is arranged as a front camera of the vehicle 1, a second surround-view camera 6 as a rear-view camera of the vehicle 1, and a third surround-view camera 7 and a fourth surround-view camera 8 are each arranged as a side camera of the vehicle 1. The control unit 4 can be configured, for example, as a microcontroller, a processor, or another type of computing device.

[0031] Vehicle 1 further includes a display device 9, which can present graphical information to a driver or user of vehicle 1. The display device 9 can, for example, be one or more screens or displays arranged in the interior of vehicle 1.

[0032] Using the surround-view cameras 3, image data can be generated from a specific area of ​​the vehicle 1's surroundings. The surround-view cameras 3 are connected to the control unit 4 via a communication link 10. The communication link 10 can, for example, comprise multiple point-to-point connections or be a bus connection such as a CAN bus or the like. Camera images or image data are transmitted from the surround-view cameras 3 to the control unit 4 via the communication link 10, particularly continuously. At least some of the camera images and / or a view of the surroundings generated from multiple camera images can then be displayed on the display unit 9 to the driver of the vehicle 1.

[0033] The vehicle 1 further comprises at least one communication unit 11, via which the vehicle 1 can communicate with an external storage device 12 via a further communication connection, in particular a wireless one, for example an internet connection. The external storage device 12 can, for example, be implemented as a server on which image data is stored and can be retrieved by the vehicle 1 as well as by other vehicles.

[0034] The vehicle 1 further comprises a positioning device 13, which is implemented, for example, as a satellite navigation device, in particular as a GPS unit. The positioning device 13 is designed to generate position information that uniquely describes the current position of the vehicle 1.

[0035] In Fig. Figure 2 shows a block diagram of an embodiment of a method for generating a view of the surroundings. The control unit 4 of the driver assistance system 2 is configured to carry out such a method, wherein the view of the surroundings represents at least one area surrounding the vehicle 1. The method can be started, for example, by a user selection made by the driver of the vehicle 1, for example, via a user interface on the display unit 9 or the like.

[0036] In step S1, at least one current camera image of the area surrounding vehicle 1 to be displayed in the surround view is captured. The camera image is captured with one or more of the vehicle 1's surround cameras 3. In particular, camera images can be captured from several or all of the surround cameras 3, for example, to create a surround view designed as a top-down view of vehicle 1.

[0037] In step S2, image data stored in storage device 12 is retrieved and used to generate the environmental view. This stored image data comprises camera images taken by one or more of the surround-view cameras 3 during a previous presence of vehicle 1 in the surrounding area, and / or camera images of the surrounding area generated by at least one additional surround-view camera of another vehicle. The stored image data may be present in storage device 12 as transformed camera images. Additionally or alternatively, the stored image data may also contain projections created from one or more camera images, such as top-down views of the surrounding area or similar representations. It is also possible that the stored image data includes satellite images.

[0038] In order to retrieve suitable stored image data for the current surroundings of the vehicle 1 from the storage device, positioning information describing the current position and / or orientation of the vehicle can be generated using the position determination device 13. This information can then be used to retrieve stored image data from the storage device 12, or to transmit it to the control unit 4, data that has the same or at least a similar or adjacent position and / or orientation.

[0039] In step S3, the surroundings view is generated from at least one current camera image and the stored image data, as described in more detail below. Subsequently, in step S4, the surroundings view is displayed to a driver of vehicle 1, for example, by showing the surroundings view on the display unit 9 of vehicle 1.

[0040] The environmental view determined by the control unit 4 is based on the current camera images and the stored image data taken from the storage unit 12. To combine the image data taken from the storage unit 12 with the camera images of vehicle 1, a top-down view of vehicle 1 can be generated from the camera images of vehicle 1. Such a top-down environmental view can be created, for example, by projecting the camera images onto a projection surface, such as a bowl-shaped one. The top-down view can then be generated from a virtual camera looking at the projection surface.

[0041] The positioning device can determine the position and orientation of vehicle 1. The stored image data, which also includes position and / or orientation information, can then be added to the overhead view generated from the current camera images. A scaling factor for the overhead view can be determined from the camera images and / or the generated overhead view. This factor indicates, for example, the actual length of one of the pixels or image points in the camera image or the overhead view. The scaling factor can, for instance, be derived directly from the calculation rules used to determine the overhead view from the current camera images.

[0042] To enable image stitching, the scaling factor of the stored image data for a sub-area of ​​the environment to be imaged can be evaluated. The scaling factor assigned to the respective image data can also be stored in the storage device 12 and retrieved by the control unit 4 of the vehicle 1. By scaling the current camera images and / or the stored image data, they can be brought to the same magnification scale so that they can be stitched together.

[0043] Based on the vehicle orientation determined by the positioning device 13 and / or by evaluating the image content of the current camera images and the stored image data, it is possible to determine the orientation of the current camera images, or a top-down view derived from them, relative to the image data taken from the storage device 12. If necessary, the orientation can be adjusted by rotating the camera images or the top-down view derived from them and / or the image data taken from the storage device 12. Subsequently, the stored image data can be supplemented with the top-down view determined from the current camera images. This allows, for example, a larger area around the vehicle to be displayed than would be possible using only the current camera images of vehicle 1.

[0044] Since the stored image data includes camera images taken by vehicle 1 during one or more previous passes in the surrounding area and / or camera images of the surrounding area generated by other vehicles, image data is available that is generally more up-to-date and / or of better quality than satellite images. Areas for which no such image data is available can be supplemented with satellite images to enable representation of sub-areas of the surrounding area not captured during previous passes and / or by other vehicles.

[0045] In Fig.Figure 3 shows an embodiment of an environment view 14 generated using the method. This view includes, for example, a vehicle graphic 15 depicting the vehicle 1 from above. The environment view 14 comprises a first sub-area 16 of the environment to be depicted, located within a predefined distance around the vehicle 1, and a second sub-area 17, located outside the predefined distance around the vehicle 1. The current camera images acquired by the vehicle 1's surround-view cameras 3 can be used to display the first sub-area 16. These images show the current state of the environment to be displayed, as they are generated by the surround-view cameras 3, particularly continuously as a video stream. The second sub-area 17 can be filled using the stored image data recorded by the vehicle 1 and / or another vehicle.In this way, the view of vehicle 1 can be displayed at a lower zoom level than would be possible based solely on the current camera images.

[0046] To achieve an appealing appearance of the surroundings view, the boundary between the first sub-area 16 and the second sub-area 17 can be processed using blending, for example, alpha blending or similar techniques, and / or by means of a wash or blur effect. This can particularly improve the image quality of the surroundings view if part of the stored image data has a different resolution and / or brightness, and / or if it consists of satellite data used to fill in a portion of the surroundings to be displayed that has not yet been captured by the vehicle cameras.

[0047] The image data stored in storage device 12 may have been generated by vehicle 1 during a previous pass through the area to be displayed. This previous pass may have occurred at an earlier time, for example, several minutes, hours, days, weeks, or months ago. The image data recorded during this pass may have been transmitted to storage device 12 via the communication link.

[0048] Other vehicles, also equipped to perform such a procedure, can transmit camera images to the storage device 12, making them available to all vehicles equipped with a correspondingly equipped control unit. The camera images generated by vehicle 1 and the other vehicles can be used, in particular, to update and / or supplement satellite images stored in the storage device 12.

[0049] To determine whether current camera images acquired by vehicle 1 or the vehicle's surround-view cameras 3 should be used to update and / or supplement image data already stored in storage device 12, the control unit 4 can evaluate whether at least one storage criterion is met. A storage criterion can be used that is met if the current camera image of the surrounding area has a better image quality than the image data of the surrounding area stored in storage device 12. Better image quality in this context could, for example, mean a higher resolution. Additionally or alternatively, a storage criterion can be used that is met if the current camera image depicts an area of ​​the surroundings that is not included, or only partially included, in the stored image data.It is also possible to use a storage criterion which is met if at least a predetermined time period has elapsed between the recording time of the current camera image of the surrounding area and a recording time of the stored image data of the surrounding area.

[0050] In addition to using an external vehicle storage device 12, it is also possible to use an internal vehicle storage device 12. This can contain, in particular, stored image data that was generated during previous passes through the surrounding area.

[0051] In addition to an environment view designed as a top-down view, other types of environment views, i.e., environment views from a different perspective or similar, can also be generated as part of the process for generating the environment view. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] DE 10 2010 042 063 A1

[0003] WO 2012 / 019901 A1

[0004]

Claims

[1] Method for generating an environmental view (14) for a vehicle (1), wherein the environmental view (14) represents at least one environmental area of ​​the vehicle (1), wherein the environmental view (14) is generated from at least one current camera image of the environmental area and from image data stored in a storage device (12), wherein the at least one current camera image is generated by at least one environmental camera (3) of the vehicle (1), characterized by that the stored image data comprise camera images of the surrounding area generated by the at least one surrounding camera (3) of the vehicle (1) during a previous stay of the vehicle (1) in the surrounding area and / or camera images of the surrounding area generated by at least one further surrounding camera of a further vehicle. [2] Method according to claim 1, characterized bythat a vehicle-external storage device is used as the storage device (12), wherein the vehicle (1) is communicatively connected to the storage device (12) via a communication connection. [3] Method according to claim 1 or 2, characterized in that, depending on the fulfillment of at least one storage criterion, image data of the surrounding area stored in the storage device (12) are updated and / or supplemented with the at least one current camera image of the surrounding area. [4] Method according to claim 3, characterized bythat a storage criterion is used which is met when the current camera image of the surrounding area has a better image quality, in particular a higher resolution, than the stored image data of the surrounding area, that a storage criterion is used which is met when the current camera image represents a surrounding area which is not or only partially contained in the stored image data, and / or that a storage criterion is used which is met when at least a predetermined time period has passed between the recording time of the current camera image of the surrounding area and a recording time of the stored image data of the surrounding area. [5] Method according to one of the preceding claims, characterized bythat the surrounding view (14) is a top view, wherein a first partial area (16) of the surrounding area lying within a predetermined distance around the vehicle (1) is displayed with the aid of the at least one current camera image and a second partial area (17) of the surrounding area lying outside the predetermined distance around the vehicle (1) is displayed with the aid of the stored image data in the surrounding view (14). [6] Method according to one of the preceding claims, characterized by that the stored image data contain transformed camera images and / or representations generated from one or more camera images using projection. [7] Method according to one of the preceding claims, characterized bythat the vehicle (1) comprises a position-determining device (13), in particular a satellite navigation device, wherein the stored image data are selected as a function of current position information generated by the position-determining device (13). [8] Control device arranged to carry out a method according to one of the preceding claims. [9] Driver assistance system for a vehicle (1), comprising at least one surroundings camera (3) and a control device (4) according to claim 8. [10] Vehicle comprising a driver assistance system (2) according to claim 9.

Images