Computer-implemented method for displaying the surroundings of a vehicle, computer program, computing device and vehicle

Abstract

The invention relates to a computer-implemented method for displaying the surroundings of a vehicle, comprising the following steps: acquiring (110) sensor data which represent at least a partial region of the surroundings of the vehicle; determining (120) a virtual model of the surroundings (200) on the basis of the sensor data; acquiring (130) odometry data of the vehicle, which represent the motion of the vehicle; identifying (140) a driving situation of the vehicle on the basis of the acquired sensor data, the acquired odometry data and / or a detected input by the driver; and displaying (150) the determined virtual model of the surroundings (200) from a virtual perspective, wherein the vehicle is represented in the model of the surroundings (200) by a synthetic object (210), wherein the displayed model of the surroundings (200) is switched over (160) from a first state to a second state depending on the identified driving situation, wherein, in the second state, the orientation of the virtual perspective in relation to the displayed and depicted surroundings does not change.

Description

[0001] Description

[0002] Computer-implemented method for displaying an environment of a vehicle, computer program, computing device and vehicle

[0003] The present invention relates to a computer-implemented method for displaying the surroundings of a vehicle, wherein a determined virtual environment model is displayed from a virtual perspective, wherein the vehicle is represented by a synthetic object in the environment model. The invention also relates to a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the method according to the invention. The invention also relates to a computing device having a computing unit configured to carry out the steps of the method according to the invention. Furthermore, the invention relates to a vehicle having the computing device.

[0004] State of the art

[0005] The document DE 10 2017 218 074 A1 discloses a method for displaying the surroundings of a vehicle, wherein a section represents the area of ​​the surroundings and the vehicle at which an approach of the vehicle to an obstacle was detected.

[0006] The document EP 3 571 091 B1 discloses a method for displaying an environment of a vehicle, comprising spanning a virtual projection surface in a virtual space which represents the environment of the vehicle, wherein the virtual projection surface is imaged from the viewpoint of a first virtual camera which is arranged in the virtual space.

[0007] Document DE 10 2015 212 370 B4 discloses a method for generating a representation of the vehicle's surroundings. Document DE 10 2021 208 473 A1 discloses a method for performing a movement maneuver with a driver assistance system.

[0008] The object of the present invention is to provide an improved method for displaying the surroundings of a vehicle.

[0009] Disclosure of the invention

[0010] The above object is achieved according to the invention according to independent claims 1 and 5 to 7.

[0011] The present invention relates to a computer-implemented method for displaying the surroundings of a vehicle. The method comprises acquiring sensor data representing at least a partial region of the surroundings of the vehicle. In particular, camera images are acquired by means of at least one camera and / or distance data are acquired by means of at least one distance sensor. The distance sensor has, in particular, at least one ultrasonic sensor, a radar sensor, and / or a lidar sensor. Optionally, map data from the surroundings of the vehicle can also be acquired. Subsequently, a virtual environment model is determined based on the acquired sensor data. Optionally, the virtual environment model is additionally determined based on the acquired map data. The virtual environment model is, in particular, a surround view or a view vertically from above or a top-down view.In a further step of the method, odometry data of the vehicle are recorded, which represent the movement of the vehicle. The recorded odometry data comprise, in particular, a rotational speed of at least one axle or a rotational speed of at least one drive axle or a rotational speed of at least one wheel of the vehicle. The recorded odometry data can alternatively or additionally comprise a speed and / or an acceleration of the vehicle, in particular their profiles over time. The recorded odometry data can alternatively or additionally also comprise an optical flow determined based on camera data. The recorded odometry data can alternatively or additionally comprise a position of the vehicle over time, wherein the position of the vehicle is recorded or determined in particular by means of a global navigation satellite system, for example by means of NAVSTAR GPS, Galileo, Beidou or GLONASS.The odometry data is advantageously acquired using an odometry sensor of the vehicle. For example, the odometry sensor comprises a rotational speed sensor, a speed sensor, an acceleration sensor, and / or a camera and / or an antenna for the global navigation satellite system. In another step of the method, a driving situation of the vehicle is recognized based on the acquired sensor data, the optionally acquired map data, the acquired odometry data, and / or a detected driver input. For example, an impending parking maneuver is automatically detected based on a current camera image, in which, in particular, a parking space was detected, and / or the detected current speed of the vehicle as the driving situation.Alternatively or additionally, the parking process is recognized as a driving situation based on the recorded driver input and / or based on map data and odometry data of the vehicle, in particular based on the current position of the vehicle as odometry data. For example, a recorded driver input to activate the parking process recognizes the parking process as a driving situation. It can be provided that a parking process is activated automatically and the parking process is recognized as a driving situation when a parking space or parking gap or garage is detected based on the sensor data and / or at a speed below a speed threshold and / or a current position of the vehicle is close to a destination of the navigation system.In a further step of the method, the determined virtual environment model is displayed from a virtual perspective, with the vehicle being represented by a synthetic object in the environment model. According to the invention, the display of the environment model is switched from a first (dynamic) state to a second (static) state depending on the detected driving situation.The first (dynamic) state represents in particular a display of the environment model with a fixed relationship between the synthetic (vehicle) object and the virtual camera perspective, advantageously a fixed viewing direction of the virtual camera perspective and / or a fixed distance between the virtual camera perspective and the synthetic object, whereby during the movement of the vehicle in particular the camera images projected onto a projection surface of the environment model and / or the position of superimposed objects in the view are advantageously continuously or dynamically adapted or updated in accordance with the movement of the vehicle.In other words, in the first (dynamic) state, in particular the displayed environment model is centered around the vehicle and the camera images or textures projected onto the projection plane are continuously adapted or updated in accordance with the movement or change in the captured camera images. Preferably, in the first state of the display of the environment model, there is at least a fixed directional reference between the direction of travel of the vehicle and the synthetic object or the displayed vehicle and optionally a fixed positional reference between the synthetic object representing the vehicle and the virtual perspective and / or the synthetic object representing the vehicle is displayed centrally in the environment model.The second (static) state represents, in particular, a display of the environment model, whereby the relationship between the vehicle and the virtual camera perspective is no longer fixed but is adjusted. For example, if an impending parking maneuver is detected as a driving situation, particularly after the parking maneuver has been activated by the detected driver input, the display of the environment model is switched to the second (static) state. In the second (static) state, the orientation and / or position of the virtual perspective relative to the displayed depicted environment is not changed.In other words, by switching to or into the second (static) state of the display, the viewer of the displayed environment model suddenly perceives a static display in which, apart from other moving objects, only the (ego) vehicle moves, since the orientation of the virtual camera relative to the depicted environment no longer changes. In other words, the images in the environment model and / or depicted static objects in the environment model are not shifted in the second (static) state according to a movement of the vehicle.Instead, the synthetic object representing the vehicle is moved in the environment model based on the acquired odometry data or the position of the virtual perspective is adjusted relative to the synthetic object representing the vehicle based on the odometry data, wherein an adjustment of the dimensions and / or the edge of the environment model can be provided, in particular an extension of the edges of the environment model, which can be associated with a zooming out of the camera in the second state, wherein the viewing direction or orientation of the virtual camera is advantageously maintained in the second state during the zooming out.The method has the advantage that in many driving situations the environment and / or driver assistance behavior is more easily understood by the driver and the driving situation is presented much more clearly, in particular with regard to other traffic and advantageously when the vehicle is automatically controlled for parking or leaving a parking space or during driving maneuvers at low speed.

[0012] In a particularly preferred embodiment of the invention, the switching from the first (dynamic) state to the second (static) state of the display of the environment model occurs at a time when the virtual perspective on the environment model is oriented toward the course of the road. This results in the course of the road as a reference for the orientation of the displayed environment model. This generally provides the driver with an intuitively understandable view or display of the environment model and a comprehensive overview, for example, of the current traffic situation in the surrounding area.

[0013] Preferably, in the second state, the synthetic object representing the vehicle is moved in the environment model based on the acquired odometry data, or in the second state, the position of the virtual perspective relative to the synthetic object representing the vehicle is adjusted based on the odometry data. These two variants particularly advantageously ensure that in the second (static) state, the orientation and / or position of the virtual perspective relative to the displayed imaged environment is not changed.

[0014] In one embodiment of the invention, at least one additional image is superimposed or displayed in the second state depending on the acquired sensor data and / or the determined virtual environment model, wherein the superimposed additional image represents, in particular, the shortest distance between the vehicle and an object in the environment. This embodiment ensures that the visualization of distances between the vehicle and nearby objects in the environment, which is reduced in the second state compared to the first state, can be emphasized, thus increasing the user's confidence in the method, particularly for automatic driving functions.

[0015] The invention also relates to a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the method according to the invention.

[0016] The invention further relates to a computing device, in particular a central computing device or a control unit. The computing device comprises a first signal input for providing a first input signal. The first input signal represents sensor data acquired by at least one environmental sensor of the vehicle, wherein the environmental sensor advantageously comprises at least one camera and / or a distance sensor. The sensor data represent or map at least a partial area of ​​the vehicle's surroundings. The computing device also has a second signal input for providing a second input signal, which represents odometry data acquired by at least one odometry sensor. The odometry data represents or describes the movement of the vehicle.The computing device further comprises a signal output for outputting an output signal to a display device, wherein the output signal represents the virtual environment model from the virtual perspective. The environment model advantageously comprises the synthetic object representing the vehicle. The computing device further comprises a computing unit, in particular a processor, configured to execute the steps of the method according to the invention.

[0017] The invention further relates to a vehicle comprising the computing device according to the invention and a display device.

[0018] Further advantages will become apparent from the following description of embodiments with reference to the figures.

[0019] Figure 1 : Flowchart of the procedure Figure 2a: View of the environment model in the first state

[0020] Figure 2b: further view of the environment model in the first state

[0021] Figure 2c: View of the environment model in the second state

[0022] Examples of implementation

[0023] Figure 1 shows a flowchart of the method schematically as a block diagram. In a step 110 of the method, sensor data is acquired which represents at least a partial area of ​​the vehicle's surroundings. The sensor data comprise, in particular, camera images and / or distance data. The camera images are advantageously acquired using a vehicle camera. The distance data is advantageously acquired or determined using an ultrasonic sensor, a radar sensor and / or a lidar sensor and / or acquired or determined using a vehicle camera using structure from motion and / or stereo vision methods. The vehicle preferably comprises at least four cameras, in particular wide-angle cameras, and at least four distance sensors, in particular ultrasonic sensors and / or radar sensors.Based on the sensor data, a virtual environment model is determined in step 120, in particular a surround view or a view vertically from above or a top-down view. For the surround view or a view vertically from above, camera images are advantageously transformed into the corresponding virtual perspectives and projected onto a projection surface of the environment model, in particular as a texture, wherein the projection surface is advantageously adjusted depending on the distance data and / or depending on detected static and / or dynamic objects in order to reduce image distortions or artifacts and / or fitting errors at joining lines. In a further step 130 of the method, odometry data of the vehicle are recorded, which represent or describe the movement of the vehicle.The odometry data includes, for example, the acceleration of the vehicle and / or the speed of the vehicle and / or the position of the vehicle and / or a rotational speed of a drive axle of the vehicle. Additionally or alternatively, the odometry data can, in particular, include a plurality of optical flow vectors determined based on a sequence of camera images, which represent at least the movement of the vehicle relative to the static environment. In another step 140 of the method, a driving situation of the vehicle is recognized based on the acquired sensor data, the acquired odometry data, and / or a detected driver input. The detected driving situation activates, in particular, an automatic driving maneuver, for example, an automatic parking function and / or an automatic maneuver at a narrow point on the road, etc.For example, an impending parking maneuver of the vehicle is detected as a driving situation based on a vehicle speed of less than 30 km / h and / or based on a position of the vehicle in a large parking lot and / or based on a position near roadside parking spaces and / or based on a position near the destination detected in a navigation system and / or depending on a camera-based parking space. The determined virtual environment model is then displayed 150 from a virtual perspective, with the vehicle being represented by a synthetic object in the environment model.The display 150 of the determined virtual environment model from the virtual perspective is initially carried out in a first state, in which the synthetic object, in particular, has a fixed distance and / or angle relationship to the virtual perspective, whereby the synthetic object is advantageously arranged centrally in the virtual environment model in the first state while the vehicle is traveling. According to the invention, in step 160, the environment model is switched from a first state to a second state depending on the recognized driving situation, wherein in the second state the position of the virtual perspective does not change relative to the displayed depicted environment.In particular, the synthetic object representing the vehicle is moved in the environment model based on the acquired odometry data, or the position of the virtual perspective is adjusted relative to the synthetic object representing the vehicle based on the odometry data. In the second (static) state, the position of the virtual perspective relative to the environment depicted in the environment model is therefore advantageously not changed. In other words, by switching 160 to or into the second (static) state of the display, the viewer of the displayed environment model suddenly perceives a static display in which, apart from other moving objects, only the synthetic object representing the (ego) vehicle moves, since the orientation of the virtual camera relative to the depicted environment no longer changes.In other words, depicted static objects in the environment model in the second (static) state are not shifted relative to the virtual perspective corresponding to a movement of the vehicle. Instead, after switching 160 to the second state, the synthetic object representing the vehicle is moved in the environment model based on the acquired odometry data, or the position of the virtual perspective is adjusted relative to the synthetic object representing the vehicle based on the odometry data. An adjustment of the dimensions of the environment model may be provided, in particular an extension of the edges of the environment model, which may be associated with zooming out the camera in the second state, wherein the viewing direction of the virtual camera is advantageously maintained in the second state during the zooming out.The switching 160 from the first state of the display of the environmental model to the second state of the display of the environmental model preferably occurs at a time when the virtual perspective on the environmental model is oriented in the direction of a roadway. As a result, the display of the environmental model in the second state is aligned with the roadway, resulting in a high level of clarity. Furthermore, in a subsequent optional step 170, at least one additional image is displayed in the second state depending on the acquired sensor data and / or the determined virtual environmental model, wherein the additional image represents, in particular, the environmental model in the first state or the shortest distance between the vehicle and an object in the environment.The additional image is in particular smaller than the displayed environment model in the second state, for example with an image area smaller than a quarter of the image area of ​​the displayed environment model. Preferably, the additional image represents a section of the environment model zoomed in or out. The additional image is advantageously superimposed on the displayed environment model. Figure 2a shows a view of the displayed environment model 200 in the first state, wherein the environment model 200 is shown here, for example, from a virtual perspective vertically from above. The display 150 of the environment model 200 in the first state comprises the (ego) vehicle as a synthetic object 210, wherein the synthetic object in the first state is typically shown in the center of the displayed environment model 200.The environment model 200 also includes, for example, other parked vehicles 220, a displayed overlay of a detected parking space P, a sidewalk 230, and / or static objects 230, 240, 250, 260, and 270, which are depicted and / or detected or determined in the projected camera images and are displayed in an abstract manner. The environment model 200 is determined, in particular, based on the acquired sensor data. In particular, acquired camera images are transformed into a corresponding virtual perspective and projected onto a projection plane. The static objects can be, for example, trees 240, a sidewalk 250 (e.g., in a park), and / or posts 260 of the sidewalk to prevent vehicles from entering.Optionally, dynamic objects, such as pedestrians and / or cyclists and / or other moving vehicles, can be displayed and, if necessary, marked (not shown here), thereby supporting the driver's attention to dynamic objects. Areas not visible by the vehicle sensors can be supplemented, determined, and / or estimated, for example, based on map data and / or sensor data stored for the current position of the vehicle, such as the course of the sidewalk 230 behind the other parked vehicles 220.

[0024] In Figure 2b, the display of the environment model 200 from Figure 2a is shown at a later point in time during a parking maneuver into the parking space P as a (recognized) driving situation, not according to the invention, in the first state, i.e., without switching 160 of the display of the environment model 200 from the first to the second state. Due to the lack of switching, the orientation of the display of the environment model 200 relative to the orientation and position of the vehicle 210 does not change, resulting in a transverse position of the displayed environment model relative to the course of the road. This creates poor visibility for the driver, since, for example, the parking space P is no longer displayed in its entirety, and the street and the parked other vehicles 220 are displayed at an angle.In contrast, Figure 2c shows the inventive view of the environment model 200 after switching from the first state according to Figure 2a to the second state, wherein the switching of the display of the environment model 200 to the second state occurs in particular based on the detected impending parking maneuver as the driving situation. The virtual perspective in the second state is aligned with the roadway course 280 in Figure 2c. The parking maneuver is carried out in particular automatically. In the second state of the displayed environment model 200, the virtual perspective is therefore no longer changed compared to the displayed environment. In the second state, the virtual perspective therefore only changes in relation to the (ego) vehicle represented as a synthetic object 210.This means, for example, that the positions and orientations of the static objects 230, 240, 250, 260, and 270 and the parked other vehicles 220, as well as the parking space P displayed as an overlay, no longer change in the displayed environment model 200. Instead, the (ego) vehicle 210 is moved in the displayed environment model 200 according to acquired odometry data, or the virtual perspective is changed in relation to the vehicle depending on the acquired odometry data. It can be provided that the environment model 200 is expanded according to a detection range of the vehicle's sensors that changes over time, whereby the edges of the environment model are changed and / or the display of the environment model is zoomed out.In order to be able to better display distances between the vehicle 210 and nearby objects, for example to the parked vehicle 220 and / or the static objects 230, 240, 250, 260 and 270, during the parking process in the second state, at least one additional image is preferably provided, which is displayed next to the displayed environment model and / or, for example, as an overlay in an area of ​​the environment model 200 that is not relevant to the detected driving situation. In Figure 2c, an additional image could be displayed as an overlay, for example, in the lower left corner of the display of the environment model, wherein the additional image represents, for example, the shortest distance between the vehicle displayed as synthetic object 210 and the upper parked other vehicle 220.

Claims

Claims 1 . Computer-implemented method for displaying an environment of a vehicle, comprising the following steps • Acquisition (110) of sensor data representing at least a partial area of ​​the vehicle’s surroundings, in particular camera images and / or distance data are acquired as sensor data, • Determination (120) of a virtual environment model (200) based on the sensor data, in particular a surround view or a view vertically from above, and • Acquisition (130) of odometry data of the vehicle, which represents the movement of the vehicle, • Detection (140) of a driving situation of the vehicle based on the acquired sensor data, the acquired odometry data and / or an acquired driver input, and • Display (150) of the determined virtual environment model (200) from a virtual perspective, wherein the vehicle is represented by a synthetic object (210) in the environment model (200), characterized in that • Switching (160) of the displayed environment model (200) from a first state to a second state depending on the detected driving situation, wherein in the second state the orientation of the virtual perspective relative to the displayed depicted environment does not change.

2. The method according to claim 1, wherein the switching (160) from the first to the second state takes place at a time in which the virtual perspective on the environmental model (200) is oriented in the direction of a roadway course.

3. The method according to any one of the preceding claims, wherein in the second state the synthetic object representing the vehicle is moved in the environment model based on the acquired odometry data or the position of the virtual perspective relative to the synthetic object representing the vehicle is adjusted based on the odometry data.

4. Method according to one of the preceding claims, wherein the following step is performed in the second state of the displayed environment model • Displaying (170) at least one additional image depending on the acquired sensor data and / or the determined virtual environment model (200), wherein the additional image represents in particular the shortest distance between the vehicle and an object in the environment.

5. A computer program comprising instructions which, when executed by a computer, cause the computer to carry out the steps of the method according to any one of the preceding claims.

6. Computing device, in particular central computing device or control device, comprising at least the following components • a first signal input for providing a first input signal which represents sensor data acquired by at least one environmental sensor of the vehicle, • a second signal input for providing a second input signal representing odometry data acquired by at least one odometry sensor, • a signal output for outputting an output signal for a display device, wherein the output signal represents a virtual environment model from a virtual perspective, and • a computing unit, in particular a processor, which is configured to carry out the steps of the method according to one of claims 1 to 5.

7. A vehicle comprising a computing device according to claim 6 and a display device.

Images