Method for generating a view of the surroundings of a vehicle

Abstract

The invention relates to a method for generating a view (A) of the surroundings (U) of a vehicle (2), wherein the vehicle (2) comprises a sensor system (1) for capturing the surroundings having at least two sensors (1a-1d) which each capture a portion of the surroundings (U) and are positioned in different subregions of the vehicle (2), the method comprising the following method steps: - receiving sensor data (D(t)) from the sensor system (1) as a function of time (t), - determining subregions (T) of surroundings (Uo) above the vehicle (2) in the sensor data (D(t)) received at different points in time (t), and - generating a view (A) of the surroundings (Uo) above the vehicle (2) using the ascertained subregions (1). The invention also relates to: an image-recording device for a vehicle (2) comprising a sensor system (1), a display device (D) and a control device, wherein the control device is designed to carry out the method according to the invention; a vehicle (2) comprising an image-recording device according to the invention; a computer program for carrying out the method according to the invention; and a computer program product.

Description

[0001] Description

[0002] Method for generating an environmental view of a vehicle

[0003] The present invention relates to a method, in particular a computer-implemented method, for generating a view of a vehicle environment of a vehicle, wherein the vehicle has a sensor system for environmental detection with at least two sensors each detecting a partial area of ​​the vehicle environment and positioned in different partial areas of the vehicle, an image recording device for a vehicle comprising a sensor system, a display device and a control device which is designed to carry out the method according to the invention, a vehicle with an image recording device according to the invention, a computer program and a computer program product.

[0004] To facilitate maneuvering a vehicle, various environmental views can be generated using sensor systems for environmental detection, such as cameras, LIDAR sensors, radar sensors, and / or ultrasonic sensors. A sensor system in the form of a surround-view camera system, which comprises a plurality of cameras positioned differently relative to the vehicle, is particularly suitable for this purpose. An artificial top-view view of the vehicle can be generated from the image data from the various cameras, for example, by projecting the image data onto a projection surface.

[0005] DE102019207415A1, for example, describes a method for generating an image of a vehicle's surroundings based on projections of images received from a plurality of cameras onto a virtual projection surface in a virtual 3D space. Further methods for generating different views of vehicle surroundings based on camera images are known, for example, from documents WO2018072793A1, WO2019170202A1, DE102015221340B4, DE102015221356B4, DE102016223391 A1, or DE 10 2021 207 558 A1. However, not only a top view of the respective vehicle is helpful for maneuvering the vehicle. Rather, a wide variety of views of the vehicle from a wide variety of perspectives, as well as projections of the vehicle's surroundings outside the driver's field of vision, are very helpful.

[0006] Based on this, the invention is based on the object of further improving the possibilities for displaying suitable views of vehicles and their surroundings.

[0007] This object is achieved by the method according to claim 1, the image recording device according to claim 11, the vehicle according to claim 12, the computer program according to claim 14 and the computer program product according to claim 15.

[0008] With regard to the method, the object underlying the invention is achieved by a method, in particular a computer-implemented method, for generating a view of a vehicle's surroundings, wherein the vehicle comprises a sensor system for environmental detection with at least two sensors, each of which detects a section of the vehicle's surroundings and is positioned in different sub-areas of the vehicle. The method comprises the following method steps:

[0009] - Receiving sensor data from the sensor system as a function of time,

[0010] - Determining sub-areas of a vehicle environment above the vehicle in the sensor data received at different times, and

[0011] - Creating a view of the vehicle environment above the vehicle based on the identified sub-areas.

[0012] The sensor system for environmental detection can comprise several similar or different sensors. In particular, a wide variety of ADAS sensors can be used. The at least two partial areas of the vehicle's surroundings or detection areas detected by the at least two sensors can partially overlap or differ completely from one another. It is advantageous if the largest possible area of ​​the vehicle's surroundings can be detected overall. It is therefore advantageous if the sensors are or will be positioned in such a way that the detection area of ​​the entire sensor system is maximized.

[0013] The sensor data is received or recorded as a function of time. In this context, repeated data recording at specific, predefined times, particularly periodically, or continuous recording is conceivable. At different times, the respective area of ​​the environment recorded varies as a function of the vehicle's movement.

[0014] According to the invention, those subregions of the vehicle's surroundings located above the vehicle are then determined or identified. For this purpose, the sensor data from individual, multiple, or all sensors of the sensor system can be used at specific, selected, or all recorded points in time. The determination of these subregions of the vehicle's surroundings can be carried out in many different ways, in particular automatically. Corresponding detection methods are known per se from the prior art. Based on the identified subregions relating to the vehicle's surroundings above the vehicle, a suitable view of the vehicle's surroundings above the vehicle can then be generated.

[0015] Knowledge of the surroundings above the vehicle is extremely helpful in numerous situations. The driver of a vehicle typically cannot directly perceive the surroundings above the vehicle from within the vehicle. Nevertheless, important information, such as traffic signs or similar, may be located above the vehicle. Overhead lane markings, such as those found in underpasses, bridges, or tunnels, must also be observed when driving a vehicle to avoid collisions.

[0016] In one embodiment of the method, the view of the vehicle's surroundings above the vehicle is displayed to the driver of the vehicle. The display can be static, or it can be a moving image, in particular a video, that continuously depicts this area of ​​the vehicle's surroundings.

[0017] Generating the view of the vehicle's surroundings above the vehicle can be accomplished at least partially with the aid of an object detector. Object detectors for detecting objects in images are known in numerous designs from the prior art. Typically, object detectors comprise a unit for locating objects and another unit for classifying the located objects. Within this framework, object localization, object classification, or both localization and classification can be performed.

[0018] In one embodiment, generating the view of the vehicle's surroundings above the vehicle comprises creating an environmental model or surroundings model for the vehicle based on the sensor data received at different times. The environmental model can be created, for example, from point clouds. The environmental model or surroundings model is preferably a two- or three-dimensional description of the vehicle's surroundings, which is derived from the recorded sensor data relating to the vehicle's surroundings. For example, information on other road users, infrastructure elements, the statistical environment, traffic control, lane boundaries, and the position of the vehicle can be taken into account, all or part of which can be extracted from the sensor data.

[0019] According to a further embodiment of the method according to the invention, generating the view of the vehicle surroundings above the vehicle comprises object localization of at least one object, in particular based on the environment model for the vehicle. The object localization can be carried out in particular based on the environment model, in particular based on 2D or 3D information from the environment model. Localization of objects in the vehicle surroundings is particularly useful for determining size relationships, in particular height relationships. It is therefore possible, for example, to relate the dimensions of one's own vehicle to the dimensions of different objects in the vehicle surroundings. For object classification, for example, different methods from the field of machine learning, in particular with the aid of neural networks, can be used. However, numerous other methods, such asa structure-from-motion method for obtaining 3D information from time-shifted 2D images can be used in the context of the present invention.

[0020] In a further embodiment, generating the view of the vehicle's surroundings above the vehicle comprises an object classification of at least one object, in particular a localized object. In this case, those objects are classified that are also being or have been localized. For object classification, various methods from the field of machine learning, such as semantic segmentation, can also be used.

[0021] It is advantageous if generating the view of the vehicle environment above the vehicle comprises a reconstruction and / or adaptive composition of the sub-areas of the sensor data received at different times that relate to the vehicle environment above the vehicle. The respective sub-areas of the vehicle environment detected differ from sensor to sensor and for each of the sensors from one detection time to the next. By taking into account the sensor data detected at different times by multiple sensors, comprehensive and complete information about the vehicle environment above the vehicle can be obtained. In this way, the vehicle environment above the vehicle can be fully reproduced and used to derive various measures.

[0022] It is further advantageous if the view of the vehicle environment above the vehicle is a 3D view, a perspective view, a top view, a side view, in particular a sectional view, or a bowl view. The type of view of the vehicle environment above the vehicle used can also be varied and selected appropriately, in particular depending on the respective application.

[0023] It is also advantageous if the sensor system comprises a camera system, in particular a surround view camera system, a radar sensor system, a lidar sensor system, and / or an ultrasonic sensor system.

[0024] A particularly preferred embodiment of the method according to the invention includes determining, based on the view of the vehicle surroundings above the vehicle, an obstacle for the vehicle above the vehicle, a static object or a static sign above the vehicle, height information, in particular relating to an enclosed spatial volume above the vehicle, or correct positioning of the vehicle relative to an object which is at least partially located above the vehicle. The obstacle can in particular be a bridge, a tunnel or an underpass, and the static object can be, for example, another permanently installed infrastructure element, a traffic light, a lighting device, a cable routed above the roadway or the like.Enclosed spatial volumes particularly refer to buildings or building-like elements, such as the heights of underground garages, toll stations, or gas stations. In spaces enclosed at the top, for example, it can be determined whether the height of the space is sufficient to open a vehicle's trunk. Finally, correct positioning of the vehicle relative to another object above it is relevant, for example, for loading or unloading operations that are carried out from above the vehicle.

[0025] In this context, it is advantageous if information and / or a warning is issued to the vehicle driver, especially in the event of danger. Such information or warning can be issued via a visual, acoustic, or optical signal. However, measures related to driving behavior can also be initiated, particularly automatically, such as initiating a braking process.

[0026] The object underlying the invention is further achieved by an image recording device for a vehicle comprising a sensor system, a display device, and a control device, wherein the control device is configured to carry out the method according to one of the preceding claims. The sensor system is also preferably a sensor system suitable for carrying out the method according to the invention, which may in particular comprise one or more identical or different sensors, in particular ADAS sensors.

[0027] The object underlying the invention is also achieved by a vehicle comprising an image recording device according to the invention.

[0028] The method according to the invention is advantageously used for vehicles, in particular in a driver assistance system of a vehicle.

[0029] The object underlying the invention is further achieved by a computer program with instructions which, when the computer program is executed by a computer, cause the computer to carry out the method according to the invention according to one of the described embodiments, and by a computer program product on which the computer program according to the invention is stored.

[0030] The invention and its advantageous embodiments are explained in more detail with reference to the following figures. They show:

[0031] Fig. 1 is a schematic flow diagram illustrating the method according to the invention,

[0032] Fig. 2 shows various applications for the method according to the invention, and Fig. 3 shows possible views of the vehicle environment above the vehicle.

[0033] In the figures, identical elements are always provided with the same reference symbol.

[0034] To generate a view of a vehicle environment Uo above a vehicle, a sensor system 1 comprising several sensors 1a-1d arranged in different sub-areas of the vehicle [here, by way of example and without limitation of generality, four similar sensors, for example, cameras] first receives sensor data D as a function of time t. The sensor data can, for example, be stored or buffered for a predeterminable time interval to enable the calculations subsequent to the data acquisition.

[0035] In the sensor data D(t) received and recorded at different times t, those sub-areas T are first determined which relate to the vehicle environment above the vehicle Uo, and on the basis of these sub-areas T, the view A of the vehicle environment above the vehicle Uo is then generated. The view A of the vehicle environment above the vehicle Uo can advantageously be based on a reconstruction and / or adaptive composition of the sub-areas T relating to the vehicle environment above the vehicle Uo of the sensor data D received at different times t. In this context, it is advantageous if the various sub-areas T are combined to form a view of the vehicle environment above the vehicle Uo. At different times t, different sections or sub-areas T of the vehicle environment above the vehicle Uo are visible in the respective sensor detection areas.It is therefore possible to combine the partial areas T, which have been detected by different sensors 1a-1d and / or at different times t, in a suitable manner in order to obtain the most complete view A of the vehicle surroundings above the vehicle U. o This view A(Uo) can be displayed, for example, to a driver of the vehicle, for example on a display D [not visible here] inside the vehicle.

[0036] Fig. 2 now relates to some exemplary application cases for the method according to the invention. Fig. 2a illustrates two vehicles 2a and 2b which are located in an underground car park. In the vehicle environment above the second vehicle 2b there is an obstacle 3 in the form of a hanging beam as well as various traffic signs to be observed. Based on the view A of the vehicle environment above the vehicle Uo, the traffic signs 4 can be displayed to the driver of the vehicle 2b and it can be estimated whether the vehicle 2b can pass the obstacle without collision by determining height information h, here for example relating to a gap between the vehicle 2b and the obstacle 3.

[0037] Fig. 2b concerns the case where a vehicle 2, here in the form of a truck, is to pass through an underpass. The underpass represents an obstacle 3 for the truck in the vehicle environment above the vehicle Uo. In such a case, it is useful to determine height information h, a height of the truck relative to the underpass, and, if necessary, to issue a warning if the underpass is impassable.

[0038] Fig. 2c concerns the case of correct positioning of a vehicle 2, again in the form of a truck, relative to an object 5 above the vehicle 2, here in the form of a bulk material device, by means of which the truck is to be loaded. For loss-free loading, correct positioning of the truck 2 relative to the bulk material device 5 is of central importance. The view A of the vehicle surroundings above the vehicle Uo generated according to the invention can enable correct positioning, which would otherwise not be readily possible without the assistance of another person outside the vehicle.

[0039] Fig. 3 illustrates two possibilities for generating view A of the

[0040] Vehicle environment above the vehicle Uo. Fig. 3a shows a side view of the vehicle 2 as well as a section of the vehicle environment above the vehicle Uo in the form of a sectional view. The vehicle 2 is a car with a trailer, which is located in an underground car park with a spatial volume V that is closed at the top. Based on the respectively generated view A of the vehicle environment above the vehicle Uo, a collision of the vehicle 2 and the trailer with the upper spatial boundary of the underground car park can be effectively prevented. For example, the method can additionally include the step of generating and outputting information or a warning W if a predefinable minimum distance to the ceiling of the underground car park is undershot.

[0041] Fig. 3b relates to the same example, but without the presence of a trailer. In contrast to Fig. 3a, in the case of Fig. 3b, a plan view as view A of the vehicle surroundings above the vehicle Uo is also generated, which is shown here as an example for three different points in time, t=n, t=n+1 and t=n+2. It is also conceivable within the scope of the present invention to provide a multi-stage warning system which, for example for the example shown here, issues different warnings at the different points in time t depending on the degree of risk of a collision, e.g. a first warning at time t=n, a more urgent warning at time t=n+1 due to the reduction of a predeterminable distance to the obstacle 3, which can be increased even further up to time t=n+2. The automatic initiation of a driving maneuver, here for example a braking maneuver, is also conceivable.

[0042] It goes without saying that in other embodiments, other views A of the vehicle surroundings above the vehicle Uo can also be generated, which also fall under the present invention.

Claims

Patent claims 1. Method, in particular a computer-implemented method, for generating a view (A) of a vehicle environment (U) of a vehicle (2), wherein the vehicle (2) comprises a sensor system (1) for environmental detection with at least two sensors (1a-1d) each detecting a section of the vehicle environment (U) and positioned in different sub-areas of the vehicle (2), comprising the following method steps: - receiving sensor data (D(t)) from the sensor system (1) as a function of time (t), - determining sub-areas (T) of a vehicle environment (Uo) above the vehicle (2) in the sensor data (D(t)) received at different times (t), and - Generating a view (A) of the vehicle surroundings (Uo) above the vehicle (2) based on the determined partial areas (1).

2. The method according to claim 1, wherein the view (A) of the vehicle surroundings (Uo) above the vehicle is displayed to a driver of the vehicle (2).

3. The method according to claim 1 or 2, wherein generating the view (A) of the vehicle environment (Uo) above the vehicle (2) comprises creating an environment model for the vehicle (2) based on the sensor data (D(t)) received at different times (t).

4. Method according to one of the preceding claims, wherein generating the view (A) of the vehicle environment (Uo) above the vehicle (2) comprises object localization of at least one object, in particular based on the environment model for the vehicle (2).

5. Method according to one of the preceding claims, wherein generating the view (A) of the vehicle surroundings (Uo) above the vehicle comprises an object classification of at least one object, in particular a localized object.

6. Method according to one of the preceding claims, wherein generating the view (A) of the vehicle surroundings (Uo) above the vehicle comprises a reconstruction and / or adaptive composition of the partial regions (T) relating to the vehicle surroundings above the vehicle of the sensor data (D(t)) received at different times (t).

7. Method according to one of the preceding claims, wherein the view (A) of the vehicle surroundings (Uo) above the vehicle (2) is a, in particular configurable, 3D view, a perspective view, a top view, a side view, in particular a sectional view, or a bowl view.

8. Method according to one of the preceding claims, wherein the sensor system (1) comprises a camera system, in particular a surround view camera system, a radar sensor system, a lidar sensor system, and / or an ultrasonic sensor system.

9. Method according to one of the preceding claims, wherein, based on the view (A) of the vehicle surroundings (Uo) above the vehicle (2), an obstacle (3) for the vehicle (2) above the vehicle (2), a static object or a static sign (4) above the vehicle (2), height information (h), in particular relating to a closed spatial volume (V) above the vehicle (2), or a correct positioning of the vehicle (2) relative to an object (5) which is at least partially located above the vehicle (2) is determined.

10. Method according to one of the preceding claims, wherein information and / or a warning (W), in particular in the event of danger, is issued to the driver of the vehicle (2).

11. Image recording device for a vehicle (2) comprising a sensor system (1), a display device (D) and a control device, wherein the control device is configured to carry out the method according to one of the preceding claims.

12. Vehicle (2) comprising an image recording device according to claim 11.

13. Use of the method according to one of claims 1-10 for a vehicle (2), in particular in a driver assistance system of the vehicle (2).

14. A computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out the method according to any one of claims 1-10.

15. A computer program product on which the computer program according to claim 14 is stored.

Images