REPRESENTING A VEHICLE'S ENVIRONMENT TO MOVE THE VEHICLE TO A DESTINATION POSITION

Description

[0001] The present invention relates to a method for displaying the environment of a vehicle, wherein the vehicle has a camera-based environment detection system for detecting the environment of the vehicle and for moving the vehicle to a target position in the environment.

[0002] The present invention also relates to a driving assistance system for displaying the environment of a vehicle, comprising a camera-based environment detection system for detecting the environment of the vehicle, and a processing unit which receives images of the environment of the vehicle, wherein the driving assistance system is configured to perform the above method.

[0003] Various driver assistance functions, or more generally, driver assistance systems, are known for moving a vehicle to a target position. This includes, for example, parking vehicles in a nearby parking lot, where the parking lot represents the target position. Such assistance systems can, for instance, scan the vehicle's surroundings to help identify parking spaces or other target positions. Furthermore, these systems can help determine an optimal trajectory for the driver to follow in order to park the vehicle or reach any other target position.

[0004] Furthermore, features such as autonomous or semi-autonomous parking are important functions for current vehicles, already used in various driver assistance systems to simplify parking. In these systems, the vehicle is maneuvered autonomously or semi-autonomously into a detected parking space. The driver may even be able to exit the vehicle before the parking maneuver begins. Such functions are known, for example, to autonomously park the vehicle in a home garage or any other parking space after exiting the vehicle.

[0005] Especially in metropolitan areas, parking spaces are often scarce, and parking and unparking can be time-consuming. Therefore, further improvements to vehicle parking are desirable.

[0006] To guide a vehicle to a target position, it is often helpful to visualize details of the vehicle's movement to the driver. This can increase confidence in the autonomous or semi-autonomous vehicle's operation and thus significantly improve the acceptance of these functions. To display these details, particularly regarding a parking space in the vehicle's vicinity, a user interface typically located within the vehicle, including a screen, is used. It is crucial that the driver can easily compare the displayed details with their own perception of the vehicle's surroundings.

[0007] Several concepts exist for depicting the details of a vehicle's movement to its target position. For example, an artificially generated vehicle environment can be represented, including details of the vehicle's movement to the target position. This typically involves a schematic representation that bears little resemblance to the real environment as perceived by the driver.

[0008] In principle, 360° bird's-eye view renderings using real camera images are also known. These renderings are based on a camera-based environmental sensing system that performs a 360° scan of the surroundings. Such camera-based environmental sensing systems include, for example, a surround-view camera system with four cameras mounted on the vehicle. However, this results in a distortion of the vehicle's real surroundings, making it difficult to compare the renderings with the actual environment. Similar renderings, such as bowl views or adaptive bowl views, are also known and partially mitigate these problems. Overall, there is still room for improvement in the rendering of relevant information about the vehicle's movement toward its target position.

[0009] From DE 103 17 044 A1, it is known that during difficult maneuvers, drivers of motor vehicles often find it challenging to estimate the path their vehicle will take and the necessary clearance to avoid a collision. This is particularly true when the driver is unfamiliar with the vehicle's dimensions or handling characteristics. A method for monitoring the clearance in the direction of travel of a vehicle uses a camera system to capture image data of the vehicle's surroundings. Additionally, based on the vehicle's operating parameters and dimensions, a signal processing unit calculates the clearance required for unimpeded travel. The driver is then shown at least parts of the image data captured by the camera system, displaying the required clearance on a screen.The image data associated with the required clearance is further processed, and as a result of this processing, the driver is informed whether or not sufficient clearance is available for unimpeded driving. This makes it possible, through continuous evaluation of the image data, to automatically react to dynamic changes in the vehicle's surroundings and inform the driver whether or not sufficient clearance is available for unimpeded driving.

[0010] DE 10 2011 082 483 A1 relates to a method for assisting a driver of a motor vehicle during a driving maneuver, comprising the following steps: a) recording data of the motor vehicle's surroundings, evaluating the recorded data to detect objects and visually displaying the detected objects, b) selection of at least one of the detected objects by the driver of the motor vehicle, c) determination of the minimum distance between the motor vehicle and the at least one selected object, d) outputting information to the driver of the motor vehicle about the minimum distance between the at least one selected object and the motor vehicle.

[0011] US 2015 / 0098623 A1 relates to an image processing device which, based on an image captured by a camera installed in a car and the distance to a measuring point on a peripheral object calculated by a distance sensor installed in the car, draws a virtual three-dimensional space in which an environmental area around the car is reconstructed.The image processing device comprises: an outline calculation unit configured to calculate an outline of an intersection plane between a plurality of grid planes defined in a predetermined coordinate system and the peripheral object; and an image processing unit configured to draw the outline calculated by the outline calculation unit onto a corresponding peripheral object located in the virtual three-dimensional space; and the multiple grid planes are configured with planes that are each perpendicular to an X-axis, a Y-axis and a Z-axis in the predetermined coordinate system.

[0012] Document DE102010051204 discloses a method for displaying an obstacle to a vehicle, comprising the steps of: - capturing an area of ​​the vehicle's surroundings with a camera system, wherein the obstacle lies outside the captured area, - displaying the image of the captured area on a screen, and - detecting the obstacle with a sensor device independent of the camera system, as well as - generating a virtual object representing the detected obstacle, - displaying on the screen a further area extending beyond the area captured by the camera, and - overlaying the virtual object within this further area on the screen at a distance from the vehicle, precisely to scale. Advantageously, this makes it possible to supplement a camera image with information obtained from other sensors.Obstacles can be superimposed onto the camera image, or the image can be supplemented with one or more obstacles.

[0013] Based on the aforementioned prior art, the invention is therefore based on the objective of providing a method for displaying the environment of a vehicle, wherein the vehicle has a camera-based environment detection system for detecting the environment of the vehicle, for moving the vehicle to a target position in the environment, and a corresponding driving assistance system that enables easy movement of the vehicle to the target position in the environment of the vehicle.

[0014] The problem is solved according to the invention by the features of the independent claims. Advantageous embodiments of the invention are specified in the dependent claims.

[0015] According to the invention, a method for representing the environment of a vehicle is thus specified, according to claim 1.

[0016] According to the invention, a driving assistance system for displaying the environment of a vehicle is also provided, comprising a camera-based environment detection system for detecting the environment of the vehicle, and a processing unit which receives images of the environment of the vehicle, wherein the driving assistance system is configured to perform the above method.

[0017] The basic idea of ​​the present invention is therefore to improve the movement of vehicles to a target position by providing the most intuitive possible representation of target information, such as a target position in the vehicle's surroundings, so that a driver is able to reliably process the target information with minimal time expenditure. To this end, the present method enables a representation that is, on the one hand, realistic and exhibits a high degree of optical correspondence with the vehicle's surroundings as perceived by the driver, and, on the other hand, provides additional information based on the processing of sensor information concerning the vehicle's surroundings in the form of a first superimposed layer.This allows the additional information to be displayed simply by overlaying the surrounding image in bird's-eye view with the information from the first overlay layer.

[0018] The vehicle can be any type of vehicle. It is preferably designed for autonomous or semi-autonomous maneuvering to move to the target position, particularly for parking. The driver may be able to exit the vehicle before it moves to the target position.

[0019] The driver assistance system provides a driver assistance function, or more generally, a driving support function. This function allows, for example, the system to detect the vehicle's surroundings in order to determine at least one target position and, if necessary, to determine an optimal trajectory that the driver can follow to move the vehicle to that target position.

[0020] Depending on the type of driver assistance system, the target position could be, for example, a parking space for parking the vehicle. With valet parking, the target position could be a handover point for transferring the vehicle to the valet. With other driver assistance systems, such as automatic garage parking, the target position could be defined by the garage itself. With learned parking, the target position could be any learned parking position, independent of a specific parking space. This is an example of possible target positions and is not exhaustive.

[0021] The environment refers to the area around the vehicle. This typically means an area within the detection range of the vehicle's environmental sensors, i.e., an area with a radius of, for example, 5-50 meters around the vehicle, preferably with a radius of no more than approximately 20 meters. Furthermore, the area can, in principle, be extended beyond this radius by means of sensor information previously received and stored by the environmental sensors.

[0022] Displaying the vehicle's surroundings includes outputting information via a graphical user interface in the vehicle, for example with a screen, preferably a touchscreen, to also receive input from the driver.

[0023] The camera-based environmental sensing system enables 360° coverage of the surroundings. Such systems typically include a surround-view camera system with four cameras mounted on the vehicle, one on each side. The cameras are preferably wide-angle cameras with a field of view of approximately 170-180°. Each of these four cameras can provide four images that together completely cover the vehicle's surroundings, thus providing a 360° view. Accordingly, images of the vehicle's surroundings are initially provided as multiple individual images by the camera-based environmental sensing system.

[0024] Generating a bird's-eye view of the environment typically involves processing multiple individual images provided jointly by the camera-based environmental perception system. The images are then processed and / or combined to create the 360° view.

[0025] Determining at least one target position in the vehicle's vicinity can be accomplished in various ways. Sensor information can be processed directly to determine this target position. Alternatively or additionally, an environmental map can first be generated based on the sensor information, and the target position can then be determined based on this map. Determining the target position could involve, for example, recognizing a parking space based on lines, signs, or other markers, identifying a valet handover point, or recognizing a garage.

[0026] The representation of at least one target position in the first overlay layer concerns a representation of the target position for moving the vehicle. The target position is preferably represented by a boundary line that completely or partially surrounds the target position. In principle, the target position can also be represented by a colored area or by a contour line with the outline of the vehicle to be moved to the target position. The target position can be an exact location or, for example, define a window into which the vehicle is moved. Accordingly, the representation of the respective target position in the first overlay layer can be carried out in fundamentally different ways.

[0027] By overlaying the surroundings image with the first overlay layer, a combined representation of the surroundings image, as perceived by the driver, is created, along with the target information, which in this case refers to a target position in the vehicle's vicinity. Overlaying means that existing parts of the surroundings image are replaced or supplemented by the overlay layer, for example, by a partially transparent overlay. It is not necessary for the surroundings image to completely fill the image area, which is also impossible, for example, if parts of the surroundings are obscured by obstacles. The surroundings image can also simply be supplemented with image information from the first overlay layer.

[0028] According to the invention, the method comprises the following additional steps: determining a non-drivable area in the vicinity of the vehicle, representing the non-drivable area in a second overlay plane that covers the vehicle's surroundings, and superimposing the surroundings image with the second overlay plane. The non-drivable area can either be determined directly, or a drivable area can first be determined, and the non-drivable area is then determined by inverting the drivable area. The second overlay plane represents an additional overlay plane to the first overlay plane, wherein the overlay planes can be used in any order, in principle, to superimpose first the surroundings image and, if necessary, the other overlay plane. The above statements regarding the superimposition of the surroundings image by the first overlay plane apply accordingly.

[0029] According to the invention, displaying the non-drivable area in a second overlay plane, which covers the vehicle's surroundings, comprises generating a side view representation of the non-drivable area based on images of the vehicle's surroundings provided by the camera-based environment sensing system. Such a side view corresponds to a representation such as that used, for example, in an adaptive bowl view. The side view enables a high degree of recognizability of the surroundings within the environmental image. The side view is preferably generated without distortion or at least with reduced distortion, for which appropriate image processing of the images from the camera-based environment sensing system is performed.

[0030] According to the invention, the method comprises the following additional steps: detecting at least one obstacle in the vicinity of the vehicle, representing the at least one obstacle in a third overlay plane that covers the vehicle's vicinity, and overlaying the environment image with the third overlay plane. The detection of the at least one obstacle can be performed directly based on sensor information. Alternatively or additionally, for example, an environment map can be generated based on the sensor information, which serves as the basis for detecting the at least one obstacle. The third overlay plane represents an additional overlay plane to the first and, optionally, the second overlay plane, whereby the overlay planes can be used in any order, in principle, to overlay first the environment image and, optionally, the other overlay plane(s).The above statements regarding the superimposition of the surrounding image by the first superimposition layer apply accordingly.

[0031] In an advantageous embodiment of the invention, representing the at least one obstacle in a third superimposed plane includes representing the boundaries of the at least one obstacle. The at least one obstacle is preferably represented by a boundary line that completely or partially surrounds the at least one obstacle. In principle, the at least one obstacle can also be represented by a colored area or in another way.

[0032] In an advantageous embodiment of the invention, the method comprises a step for identifying the at least one obstacle, and the representation of the at least one obstacle in a third overlay plane, which covers the vehicle's surroundings, comprises representing the at least one obstacle based on its identification. The identification of the at least one obstacle involves a classification to identify, for example, other vehicles, trees, people, buildings, garbage cans, or other obstacles. Based on this, a representation of the obstacle can be selected in accordance with the respective class. Thus, a kind of placeholder for the at least one obstacle is selected based on the identification and represented in the third overlay plane.The obstacle is preferably depicted in a top view that corresponds to the depiction of the surroundings. Alternatively, the obstacle can be depicted in a side view.

[0033] In an advantageous embodiment of the invention, displaying the at least one obstacle in a third superimposed plane, which covers the vehicle's surroundings, includes providing a camera image of the at least one obstacle. The camera image enables a particularly realistic representation of the at least one obstacle, thereby facilitating a simple correlation of the representation with the surroundings as perceived by the driver. The camera image is preferably generated in accordance with the representation of the surroundings in a top view, or the camera image is projected onto the top view.

[0034] In an advantageous embodiment of the invention, displaying the at least one obstacle in a third superimposed plane, which covers the vehicle's surroundings, includes a distance-dependent representation of the at least one obstacle. In particular, the distance-dependent representation includes depicting the obstacle or parts of the obstacle in different colors depending on the distance. For example, nearby obstacles can be displayed in red, while distant obstacles can be displayed in green, black, or gray. Such a representation is particularly useful for areas that are no longer actively detected by the environmental sensors. This allows the user to see that this area was previously detected by one of the environmental sensors, but is no longer being actively detected.For example, nearby areas of an obstacle can be represented in a different color than distant areas. Instead of a uniform color, a color gradient or a colored pattern can also be used.

[0035] According to the invention, representing the at least one obstacle in a third superimposed plane, which covers the vehicle's surroundings, comprises generating a side view representation of the at least one obstacle based on images of the vehicle's surroundings provided by the camera-based environment sensing system. Such a side view corresponds to a representation such as that used, for example, in an adaptive bowl view. The side view enables a high recognition value of the obstacle in the environment image. The side view is preferably generated distortion-free or with reduced distortion, for which appropriate image processing of the images from the camera-based environment sensing system is required.

[0036] In an advantageous embodiment of the invention, determining at least one target position in the vehicle's vicinity and / or determining a non-drivable area in the vehicle's vicinity and / or identifying at least one obstacle in the vehicle's vicinity is achieved by taking into account the images of the vehicle's surroundings provided by the camera-based environmental sensing system. The camera-based environmental sensing system thus serves as an environmental sensor for monitoring the vehicle's surroundings. In principle, no further environmental sensors are required, but they can be used to improve monitoring.

[0037] In an advantageous embodiment of the invention, the method comprises a step for receiving sensor information from at least one further environmental sensor, in particular a LiDAR-based environmental sensor, a radar sensor and / or a plurality of ultrasonic sensors, which detects at least a partial area of ​​the vehicle's environment, and the determination of at least one target position in the vehicle's environment and / or the determination of a non-drivable area in the vehicle's environment and / or the detection of at least one obstacle in the vehicle's environment is carried out taking into account the sensor information from the at least one further environmental sensor.By selecting suitable environmental sensors, which can be mounted on the vehicle in any combination and number, a particularly reliable detection of the vehicle's surroundings is enabled in order to determine or detect at least one target position, the non-drivable area, and / or at least one obstacle. The camera-based environmental sensing system can provide additional sensor information that is processed together with the sensor information from at least one other environmental sensor to determine or detect at least one target position, the non-drivable area, and / or at least one obstacle. Alternatively, only the sensor information from the at least one other environmental sensor is used. When using several identical and / or different types of environmental sensors, the sensor information from these sensors can be fused.

[0038] According to the invention, generating a bird's-eye view of the surroundings based on images of the vehicle's environment provided by the camera-based environmental sensing system includes generating the environmental image in the manner of a bowl view. This is a special bowl-shaped view in which the edges are pulled upwards. In contrast to a bird's-eye view, the edges can therefore be partially displayed in a side view. Compared to a purely top-down or bird's-eye view, this improves the correspondence with the environment as perceived by the driver, particularly in distant areas.

[0039] In an advantageous embodiment of the invention, representing the at least one target position in a first superimposed plane covering the vehicle's surroundings includes representing a trajectory for moving the vehicle to reach the target position. The trajectory allows for a good assessment of the movement required to reach the target position. In particular, it can be verified in advance whether the vehicle can even be moved to the target position. The trajectory can comprise several movements with reversals of direction.

[0040] In an advantageous embodiment of the invention, displaying a trajectory for moving the vehicle to reach the target position includes displaying an area swept by the vehicle while traveling the trajectory. This allows for easy determination, depending on the drivable area, of whether there is a risk of the vehicle leaving the drivable area. The area swept by the vehicle while traveling the trajectory is preferably displayed as a function of the dimensions of the respective vehicle to which the driving assistance system belongs. Alternatively, an average or maximum value for typical vehicles can be used as the corresponding vehicle dimension.

[0041] Alternatively or additionally, further target information can be displayed, such as stopping points along the trajectory, a speed profile when driving along the trajectory, preferably with a current speed being coded using different colors, activation of an access restriction (garage door, garden gate, bollard (retractable), barrier) when driving along the trajectory, or others.

[0042] In an advantageous embodiment of the invention, the method comprises a step for storing the images of the vehicle's surroundings provided by the camera-based environment sensing system, and the generation of an environment image in a bird's-eye view comprises generating at least a first area of ​​the environment image based on current images of the vehicle's surroundings provided by the camera-based environment sensing system and at least a second area with stored images of the vehicle's surroundings. This allows for a larger area around the vehicle to be covered compared to using only current images, whereby, of course, the timeliness of the stored images must be taken into account.Determining at least one target position in the vehicle's vicinity can also be performed based on the current images of the vehicle's surroundings provided by the camera-based environment sensing system, in combination with stored images. The same applies to identifying obstacles in the environment. The environment image can show a different representation of the at least one first area and the at least one second area to indicate a potential hazard from changes in the second area based on the stored images. For example, the first and second areas can have different colors (red, gray).

[0043] The invention is explained in more detail below with reference to the accompanying drawing and preferred embodiments. The features shown can represent an aspect of the invention, either individually or in combination. Features of different embodiments are transferable from one embodiment to another.

[0044] It shows Fig. 1 a schematic representation of a vehicle with a driver assistance system according to a first, preferred embodiment in a side view, Fig. 2 a first exemplary representation of the vehicle with its surroundings in accordance with the first embodiment, Fig. 3 a second exemplary representation of the vehicle with its surroundings in accordance with a second embodiment, Fig. 4 a third exemplary representation of the vehicle with its surroundings in accordance with a third embodiment, Fig. 5 a fourth exemplary representation of the vehicle with its surroundings in the manner of a bowl view in accordance with a fourth embodiment, Fig. 6 a fifth exemplary representation of the vehicle with its surroundings in the manner of an adaptive bowl view in accordance with a fifth embodiment, and Fig.7a flowchart of a procedure for representing a vehicle's environment. Fig. 1 in accordance with the vehicle and the driving assistance system of the first embodiment.

[0045] The Figure 1 Figure 1 shows a vehicle 10 with a driving assistance system 12 according to a first, preferred embodiment. The vehicle 10 is, in principle, any vehicle 10, preferably designed for autonomous or semi-autonomous maneuvering, for example, for parking the vehicle 10. During autonomous driving, a driver is enabled to leave the vehicle 10 before the corresponding parking maneuver is carried out.

[0046] The driver assistance system 12 comprises a camera-based environment sensing system 14 that performs a 360° scan of the vehicle 10's surroundings 16. For simplicity, the camera-based environment sensing system 14 is shown here as a single device. In this embodiment, it comprises four individual surround-view cameras (not shown individually in the figures) mounted on the vehicle 10. Specifically, one of the four cameras is mounted on each side of the vehicle 10. The four cameras are preferably wide-angle cameras with a field of view of approximately 170°–180°. Each of the four cameras provides four images that together completely cover the vehicle 10's surroundings 16, thus providing a 360° view.

[0047] The driving assistance system 12 also includes a processing unit 18, which receives the images from the camera-based environment detection system 14 via a data bus 20.

[0048] The driver assistance system 12 also includes an environmental sensor 22, which in this embodiment is designed as a radar sensor or as a LiDAR-based sensor. The environmental sensor 22 transmits sensor information concerning the environment 16 of the vehicle 10 to the processing unit 18 via the data bus 20. In an alternative embodiment, the environmental sensor 22 is designed as an ultrasonic sensor unit with a plurality of individual ultrasonic sensors.

[0049] Driving assistance system 12 provides a driver assistance function or, more generally, a driving assistance function, whereby the environment 16 of the vehicle 10 is detected to help determine target positions 24 and, if necessary, to determine an optimal trajectory which a driver can follow to move the vehicle 10 to the target position 24.

[0050] Accordingly, in this embodiment, the driving assistance system 12 is configured to perform a method for displaying the environment 16 of the vehicle 10 in order to move the vehicle 10 to a target position 24 in the environment 16. The method is described in Figure 7 presented as a flowchart and further explained below with additional reference to the Figures 2 to 6 described. The Figures 2 to 6 These are different representations of the environment 16 of the vehicle 10, all of which can be generated with the same driving assistance system 12. A change in the representation only requires a modified configuration or programming of the driving assistance system 12. The driving assistance system 12 of the first embodiment is designed for autonomous parking of the vehicle 10.

[0051] Accordingly, in this embodiment, the target position 24 is a parking space 24 for parking the vehicle 10.

[0052] The procedure begins with step S100, which involves providing images of the vehicle's surroundings 16 using the camera-based environment sensing system 14. The four individual images are then transmitted together via the data bus 20 to the processing unit 18 of the driver assistance system 12.

[0053] Step S110 involves generating a bird's-eye view environmental image 26 based on the images of the vehicle's surroundings 16 provided by the camera-based environmental sensing system 14. Accordingly, the environmental image 26 is generated by processing the individual images jointly provided by the camera-based environmental sensing system 14. The individual images are processed and / or combined to generate the 360° view. A corresponding representation of the bird's-eye view environmental image 26 is shown in the Figures 2 to 4 shown.

[0054] According to the claimed invention, which is in Figure 5 As shown, the environment image 26 is generated in the manner of a bowl view. The bowl view is a special type of view resembling a bowl, in which the edges are pulled upwards, so that, unlike a bird's-eye view, edges are at least partially shown in a side view. The environment image 26 is thus displayed in the manner of a bowl view. In an alternative, fifth embodiment, which is shown in Figure 6 As shown, the environment image 26 is generated in the manner of an adaptive bowl view.

[0055] Step S120 involves receiving sensor information from the environmental sensor 22, which detects at least a portion of the vehicle 10's environment 16. The sensor information from the environmental sensor 22 is transmitted to the processing unit 18 via the data bus 20.

[0056] Step S130 relates to determining at least one target position 24 in the vicinity 16 of the vehicle 10. In this embodiment, determining the at least one target position 24 involves identifying a parking space as the target position 24. This determination is performed by considering the sensor information from the environmental sensor 22 together with the sensor information from the camera-based environmental sensing system 14, i.e., the images provided by the camera-based environmental sensing system 14. The sensor information from the environmental sensor 22 and the camera-based environmental sensing system 14 is processed together to identify the at least one parking space 24. This involves an optional fusion of the sensor information from the environmental sensor 22 and the camera-based environmental sensing system 14.

[0057] Determining the location of at least one parking space 24 in the vicinity 16 of the vehicle 10 can be done in different ways. The sensor information from the environmental sensor 22 can be processed directly together with the sensor information from the camera-based environmental detection system 14 to determine the location of the at least one parking space 24. Alternatively or additionally, an environmental map can be generated based on the sensor information, which serves as the basis for determining the location of parking space 24.

[0058] Step S140 relates to representing the at least one target position 24, i.e., the at least one parking space 24, in a first overlay plane that covers the area 16 surrounding the vehicle 10. Representing the at least one parking space 24 in the first overlay plane relates to a representation of the parking space 24 for parking the vehicle 10. In this embodiment, the parking space 24 is represented by a boundary line that completely surrounds it. Alternatively or additionally, the parking space 24 can be represented by a colored area.

[0059] Step S150 involves determining a non-travelable area 30 in the vicinity 16 of the vehicle 10. The non-travelable area 30 can either be determined directly, or a travelable area 28 can first be determined, and the non-travelable area 30 is determined by inverting the travelable area 28.

[0060] The determination of the non-drivable area 30 in the vicinity 16 of the vehicle 10 is also carried out taking into account the sensor information from the environmental sensor 22 together with the sensor information from the camera-based environmental sensing system 14, i.e., the images provided by the camera-based environmental sensing system 14. The sensor information from the environmental sensor 22 and the camera-based environmental sensing system 14 is processed together to determine the non-drivable area 30. This involves an optional fusion of the sensor information from the environmental sensor 22 and the camera-based environmental sensing system 14.

[0061] Determining the non-drivable area 30 in the vicinity 16 of the vehicle 10 can be done in different ways. The sensor information from the environmental sensor 22 can be processed directly together with the sensor information from the camera-based environmental perception system 14 to determine the non-drivable area 30. Alternatively or additionally, an environmental map can be generated based on the sensor information, which serves as the basis for determining the non-drivable area 30.

[0062] Step S160 involves representing the non-travelable area 30 in a second overlay layer, which covers the area 16 surrounding the vehicle 10. The second overlay layer represents an additional overlay layer to the first overlay layer. In the representations of the Figures 2 to 4The non-drivable area 30 is marked by a uniform surface with a predetermined color, so that the surrounding image 26 in the non-drivable area 30 is overlaid by this surface and cannot be perceived.

[0063] According to the claimed invention, which is contained in the Figures 5 and 6 As shown, a side view is generated to depict the non-drivable area 30, based on the images of the vehicle's surroundings 16 provided by the camera-based environment detection system 14. The side view is preferably generated dynamically and continuously adjusted. Such a side view corresponds to a representation like that of a bowl view, which is shown, for example, in Figure 5 is shown, or an adaptive bowl view that is in Figure 6The adaptive bowl view is used to generate the side view with reduced distortion, for which corresponding image processing of the images from the camera-based environment detection system 14 is performed.

[0064] Step S170 involves identifying at least one obstacle 32 in the vicinity 16 of the vehicle 10.

[0065] The detection of at least one obstacle 32 in the vicinity 16 of the vehicle 10 is also carried out taking into account the sensor information of the environmental sensor 22 together with the sensor information of the camera-based environmental sensing system 14, i.e., the images provided by the camera-based environmental sensing system 14. The sensor information of the environmental sensor 22 and the camera-based environmental sensing system 14 is processed together to detect the at least one obstacle 32. This involves an optional fusion of the sensor information from the environmental sensor 22 and the camera-based environmental sensing system 14.

[0066] The detection of at least one obstacle 32 can be carried out directly based on the sensor information from the environmental sensor 22 together with the camera-based environmental detection system 14. Alternatively or additionally, an environmental map can be generated based on the sensor information, which serves as the basis for detecting at least one obstacle 32.

[0067] Step S180 involves representing at least one obstacle 32 in a third superposition plane, which covers the environment 16 of the vehicle 10.

[0068] In the first embodiment, which is described in Figure 2As shown, the representation of the at least one obstacle 32 in the third superimposition plane includes the representation of boundary lines of the at least one obstacle 32. In this embodiment, the boundary lines mark only the sides of the obstacles 32 facing the vehicle 10. Alternatively or additionally, the obstacles 32 can be represented with a color-contrasting area. This is done on the Figure 2Not shown, a distance-dependent representation of the obstacles 32 is used, with different colors depending on the distance. In this embodiment, for example, nearby areas of an obstacle 32 are represented in red, while distant areas of the obstacle 32 are represented in green, black, or gray. Such a representation is particularly useful for areas that are no longer actively detected by the environmental sensors. This allows the user to see that this area was previously detected by one of the environmental sensors, but is no longer being actively detected.

[0069] In the second embodiment, which is described in Figure 3As shown, a first step is performed to identify the at least one obstacle 32. This includes classifying the obstacles 32 to identify, for example, third-party vehicles, trees, people, buildings, garbage cans, or other obstacles 32. In the embodiments shown here, the obstacles 32 are third-party vehicles. Based on the identification, a representation for the obstacle 32 is chosen in accordance with the respective class used in the third overlay plane. In this example, which is not covered by the claims, the representation of the obstacle 32 is in accordance with the representation of the surroundings 26 in a top view. According to the invention, the obstacle 32 is represented in a side view.

[0070] Accordingly, at least one obstacle 32 is represented based on the identification.

[0071] In the third embodiment, which is described in Figure 4As illustrated, displaying the at least one obstacle 32 in the third superimposition plane comprises providing a camera image of the at least one obstacle 32 as captured by the camera-based environment sensing system 14. This generates a realistic representation of the at least one obstacle 32. In an example not covered by the claims, the camera image is generated in accordance with the representation of the environment image 26 in a top view, or the camera image is projected onto the top view. According to the invention, the camera image of the at least one obstacle 32 is generated in a side view based on the images of the environment 16 of the vehicle 10 provided by the camera-based environment sensing system 14. The at least one obstacle 32 is thus visualized in the manner of a representation such as that used, for example, in an adaptive bowl view.The side view is preferably generated without distortion or with reduced distortion, for which appropriate image processing of the images from the camera-based environmental detection system 14 is carried out.

[0072] Step S190 involves overlaying the environment image 26 with the first, second, and third overlay layers. During overlaying, existing parts of the environment image 26 are replaced or supplemented by the information from the overlay layers, for example, by a partially transparent overlay. It is not necessary for the environment image 26 to completely fill the image area. In this case, the environment image 26 is merely supplemented with image information from the overlay layers. The overlay layers can be arranged in any order and may overlap each other. The overlaid environment image 26 can be output via a user interface of the vehicle 10 and displayed to the driver.

[0073] By overlaying the environment image 26 with the overlay layers, a combined representation of the environment 16, as perceived by the driver, is created, together with the parking information, which here concerns a position of the parking space 24 in the environment 16 of the vehicle 10. Reference symbol list

[0074] 10 Vehicle 12 Driver assistance system 14 Camera-based environmental perception system 16 Environment 18 Processing unit 20 Data bus 22 Environmental sensor 24 Target position, parking space 26 Environmental image 28 Drivable area 30 Non-drivable area 32 Obstacle

Claims

1. Method for displaying an environment (16) of a vehicle (10), wherein the vehicle (10) has a camera-based environment detection system (14) for detecting the environment (16) of the vehicle (10), for moving the vehicle (10) to a target position (24) in the environment (16), comprising the steps Providing images of the environment (16) of the vehicle (10) with the camera-based environment detection system (14), Generating an environment image (26) in a bird's eye view based on the images of the environment (16) of the vehicle (10) provided by the camera-based environment detection system (14), Determining at least one target position (24) in the environment (16) of the vehicle (10), Displaying the at least one target position (24) in a first overlay layer, which covers the environment (16) of the vehicle (10), and Overlaying the environment image (26) with the first overlay layer, Determining a non-drivable area in the environment of the vehicle (10), Displaying the non-drivable area (30) in a second overlay layer, which covers the environment (16) of the vehicle (10), and Overlaying the environment image (26) with the second overlay layer, wherein the displaying of the non-drivable area (30) in a second overlay layer, which covers the environment (16) of the vehicle (10), includes generating a representation of the non-drivable area (30) in a side view based on the images of the environment (16) of the vehicle (10) provided by the camera-based environment detection system (14), Determining at least one obstacle (32) in the environment (16) of the vehicle (10), Displaying the at least one obstacle (32) in a third overlay layer, which covers the environment (16) of the vehicle (10), and Overlaying the environment image (26) with the third overlay layer, wherein the displaying of the at least one obstacle (32) in a third overlay layer, which covers the environment (16) of the vehicle (10), includes generating a representation of the at least one obstacle (32) in a side view based on the images of the environment (16) of the vehicle (10) provided by the camera-based environment detection system (14) and providing the side view representation of the at least one obstacle (32); wherein the non-drivable area (30) is either determined directly, or wherein first a drivable area (28) is determined and the non-drivable area (30) is determined by inverting the drivable area (28); wherein when overlaying the environment image (26) with the first, second and third overlay layers, existing parts of the environment image (26) are replaced or supplemented by the information of the overlay layers through partially transparent overlay.

2. Method according to one of the preceding claims, wherein - the displaying of the at least one obstacle (32) in a third overlay layer includes displaying boundaries of the at least one obstacle (32).

3. Method according to one of the preceding claims, wherein - the method includes a step for identifying the at least one obstacle (32), and the displaying of the at least one obstacle (32) in a third overlay layer, which covers the environment (16) of the vehicle (10), includes displaying the at least one obstacle (32) based on the identification of the at least one obstacle (32).

4. Method according to one of the preceding claims, wherein - the displaying of the at least one obstacle (32) in a third overlay layer, which covers the environment (16) of the vehicle (10), includes a distance-dependent representation of the at least one obstacle (32).

5. Method according to one of the preceding claims, wherein the determining of at least one target position (24) in the environment (16) of the vehicle (10) and / or the determining of a non-drivable area (30) in the environment (16) of the vehicle (10) and / or the determining of the at least one obstacle (32) in the environment (16) of the vehicle (10) is performed taking into account the images of the environment (16) of the vehicle (10) provided by the camera-based environment detection system (14).

6. Method according to one of the preceding claims, wherein the method includes a step for receiving sensor information from at least one additional environment sensor (22), in particular a LiDAR-based environment sensor, a radar sensor and / or a plurality of ultrasonic sensors, which detects at least a partial area of the environment (16) of the vehicle (10), and the determining of at least one target position (24) in the environment (16) of the vehicle (10) and / or the determining of a non-drivable area (30) in the environment (16) of the vehicle (10) and / or the determining of the at least one obstacle (32) in the environment (16) of the vehicle (10) is performed taking into account the sensor information of the at least one additional environment sensor (22).

7. Method according to one of the preceding claims, wherein the generating of an environment image (26) in a bird's eye view based on the images of the environment (16) of the vehicle (10) provided by the camera-based environment detection system (14) includes generating the environment image (26) in the manner of a bowl view.

8. Method according to one of the preceding claims, wherein the displaying of the at least one target position (24) in a first overlay layer, which covers the environment (16) of the vehicle (10), includes displaying a trajectory for moving the vehicle (10) to reach the target position (24).

9. Method according to claim 8, wherein the displaying of a trajectory for moving the vehicle (10) to reach the target position (24) includes displaying an area swept by the vehicle (10) when driving the trajectory.

10. Method according to one of the preceding claims, wherein the method includes a step for storing the images of the environment (16) of the vehicle (10) provided by the camera-based environment detection system (14), and the generating of an environment image (26) in a bird's eye view includes generating at least a first area of the environment image (26) based on images of the environment (16) of the vehicle (10) currently provided by the camera-based environment detection system (14) and at least a second area with stored images of the environment (16) of the vehicle (10).

11. Driver assistance system (10) for displaying an environment (16) of a vehicle (10), with a camera-based environment detection system (14) for detecting the environment (16) of the vehicle (10), and a processing unit (18), which receives images of the environment (16) of the vehicle (10), wherein the driver assistance system (12) is designed to perform the method according to one of the preceding claims 1 to 10.