Computer-implemented method for displaying an environment of a vehicle, computer program, computing device and vehicle

Abstract

A computer-implemented method for displaying the surroundings of a vehicle, comprising the following steps: acquiring (110) sensor data representing at least a partial area of ​​the surroundings of the vehicle; determining (120) a virtual environment model (200) based on the sensor data; acquiring (130) odometry data of the vehicle representing the movement of the vehicle; recognizing (140) a driving situation of the vehicle based on the acquired sensor data, the acquired odometry data, and / or a detected driver input;and displaying (150) the determined virtual environment model (200) from a virtual perspective, wherein the vehicle is represented by a synthetic object (210) in the environment model (200), wherein a switching (160) of the displayed environment model (200) takes place from a first state to a second state depending on the recognized driving situation, wherein in the second state the orientation of the virtual perspective relative to the displayed depicted environment does not change.;

Description

The present invention relates to a computer-implemented method for displaying an environment of a vehicle, wherein a determined virtual environment model is displayed from a virtual perspective, wherein the vehicle is represented by a synthetic object in the environment model. The invention also relates to a computer program comprising instructions which, when the program is executed by a computer, cause the computer to execute the steps of the method according to the invention. The invention also relates to a computing device having a computing unit which is configured to carry out the steps of the method according to the invention. The invention further relates to a vehicle having the computing device.Prior ArtThe document DE 10 2017 218 074 A1 discloses a method for representing the surroundings of a vehicle, wherein a section represents that region of the surroundings and of the vehicle at which an approach of the vehicle to an obstacle has been detected.Document EP 3 571 091 B1 discloses a method for representing an environment of a vehicle, comprising a spanning of a virtual projection surface in a virtual space representing the environment of the vehicle, wherein the virtual projection surface is imaged from the view of a first virtual camera arranged in the virtual space.The document DE 10 2015 212 370 B4 discloses a method for generating a representation of a vehicle environment of a vehicle.The document DE 10 2021 208 473 A1 discloses a method for carrying out a movement maneuver using a driver assistance system.The object of the present invention is to provide an improved method for displaying an environment of a vehicle.Disclosure of the InventionThe above object is achieved according to the invention according to independent claims 1 and 5 to 7.The present invention relates to a computer-implemented method for displaying an environment of a vehicle. The method comprises a detection of sensor data which represent at least a partial area of the environment of the vehicle; in particular, camera images are detected by means of at least one camera and / or distance data are detected by means of at least one distance sensor as sensor data. The distance sensor has, in particular, at least one ultrasonic sensor, a radar sensor and / or a lidar sensor. Optionally, map data from the surroundings of the vehicle can additionally be acquired. A virtual environment model is then determined based on the acquired sensor data. Optionally, the virtual environment model is additionally determined based on the acquired map data. The virtual environment model is in particular a surround view view or a view perpendicular from above or a top down view. In a further step of the method, odometry data of the vehicle are acquired which represent the movement of the vehicle. The recorded odometry data comprise in particular a rotational speed of at least one axle or a rotational speed of at least one drive axle or a rotational speed of at least one wheel of the vehicle. The recorded odometry data can alternatively or additionally comprise a speed and / or an acceleration of the vehicle, in particular their curves over time. The captured odometry data can alternatively or additionally also comprise an optical flow determined on the basis of camera data. The recorded odometry data can alternatively or additionally comprise a position of the vehicle over time, wherein the position of the vehicle is recorded or determined in particular by means of a global navigation satellite system, for example by means of NAVSTAR GPS, Galileo, Beidou or GLONASS. The odometry data are advantageously acquired by means of an odometry sensor of the vehicle, for example the odometry sensor comprises a rotational speed sensor, a speed sensor, an acceleration sensor and / or a camera and / or an antenna for the global navigation satellite system. In another step of the method, a driving situation of the vehicle is detected based on the captured sensor data, the optionally captured map data, the captured odometry data and / or a captured input of the driver. For example, based on a current camera image in which in particular a parking space has been detected and / or the detected current speed of the vehicle, an imminent parking process is automatically detected as a driving situation. Alternatively or additionally, the parking process is recognized as a driving situation on the basis of the detected input of the driver and / or on the basis of map data and odometry data of the vehicle, in particular on the basis of the current position of the vehicle as odometry data. For example, by a detected input of the driver for activating the parking method, the parking process is recognized as a driving situation. It can be provided that a parking method is automatically activated by a parking space or a parking space or a garage recognized as a function of the sensor data and / or at a speed less than a speed threshold value and / or a current position of the vehicle in the vicinity of a destination of the navigation system and the parking process is recognized as a driving situation. In a further step of the method, the determined virtual environment model is displayed from a virtual perspective, wherein the vehicle is represented by a synthetic object in the environment model. According to the invention, the display of the environment model is switched from a first (dynamic) state to a second (static) state depending on the detected driving situation. The first (dynamic) state represents, in particular, a display of the environment model with a fixed relationship between the synthetic (vehicle) object and the virtual camera perspective, advantageously a fixed viewing direction of the virtual camera perspective and / or a fixed distance between the virtual camera perspective and the synthetic object, as a result of which, during the travel of the vehicle, in particular the camera images projected onto a projection surface of the environment model and / or the position of overlaid objects in the view are advantageously adapted or updated continuously or dynamically in accordance with the movement of the vehicle. In other words, in the first (dynamic) state, in particular the displayed environment model is centered around the vehicle and the camera images or textures projected onto the projection plane are continuously adapted or updated according to the movement or the change of the captured camera images. Preferably, in the first state of the display of the environment model, at least one fixed direction reference is present between the direction of travel of the vehicle and the synthetic object or the illustrated vehicle and optionally a fixed position reference is present between the synthetic object representing the vehicle and the virtual perspective, and / or the synthetic object representing the vehicle is displayed centrally in the environment model. The second (static) state represents, in particular, a display of the environment model, wherein the relationship between the vehicle and the virtual camera perspective is no longer fixed but rather is adapted. For example, in the case of an imminent parking process as a detected driving situation, in particular after activation of the parking process by the detected input of the driver, the display of the environment model is switched over to or to the second (static) state. In the second (static) state, the orientation and / or the position of the virtual perspective is not changed with respect to the displayed imaged environment. In other words, by switching to the second (static) state of the display, the viewer of the displayed environment model suddenly perceives a static display in which, apart from other moving objects, only the (ego) vehicle is moving, since the orientation of the virtual camera with respect to the imaged environment no longer changes. In other words, the maps in the environment model and / or mapped static objects in the environment model in the second (static) state are not shifted according to a movement of the vehicle. Instead, the synthetic object representing the vehicle is moved in the environment model based on the captured odometry data or the position of the virtual perspective is adjusted with respect to the synthetic object representing the vehicle based on the odometry data, wherein an adjustment of the dimensions and / or the edge of the environment model can be provided, in particular an extension of the edges of the environment model, which can be connected to zoom-out of the camera in the second state, wherein the viewing direction or orientation of the virtual camera is advantageously maintained in the second state during zoom-out. The method results in the advantage that in many driving situations the environment and / or a driving assistance behavior for the driver is better understood and the driving situation is shown much clearer, in particular with respect to the other traffic and advantageously in automatic control of the vehicle for parking or parking out or in driving maneuvers at low speed.In a particularly preferred embodiment of the invention, the switchover from the first (dynamic) state to the second (static) state of the display of the environment model takes place at a point in time at which the virtual perspective is aligned with the environment model in the direction of a roadway course. This results in the course of the roadway as a reference to the orientation of the displayed environmental model. This generally provides the driver with an intuitively comprehensible view or display of the environment model and a large overview, for example of the current traffic situation in the environment.Preferably, in the second state, the synthetic object representing the vehicle is moved in the environment model based on the acquired odometry data or, in the second state, the position of the virtual perspective with respect to the synthetic object representing the vehicle is adjusted based on the odometry data. These two variants particularly advantageously ensure that, in the second (static) state, the alignment and / or the position of the virtual perspective is not changed with respect to the displayed imaged environment.In one embodiment of the invention, at least one additional mapping is overlaid or displayed in the second state as a function of the captured sensor data and / or the determined virtual environment model, wherein the overlaid additional mapping represents in particular the shortest distance between the vehicle and an object in the environment. This embodiment achieves the effect that the visualization of distances between the vehicle and nearby objects in the environment, which is reduced in the second state compared to the first state, can be emphasized, so that the trust of the user in the method increases, in particular for automatic driving functions.The invention also relates to a computer program comprising instructions which, when the program is executed by a computer, cause the computer to execute the steps of the method according to the invention.The invention furthermore relates to a computing device, in particular a central computing device or a control device. The computing device comprises a first signal input for providing a first input signal. The first input signal represents sensor data recorded by means of at least one environment sensor of the vehicle, wherein the environment sensor advantageously comprises at least one camera and / or a distance sensor. The sensor data represent or form at least a partial area of the environment of the vehicle. The computing device also has a second signal input for providing a second input signal, which represents odometry data detected by means of at least one odometry sensor. The odometry data represent or describe the movement of the vehicle. The computing device also includes a signal output for outputting an output signal to a display device, the output signal representing the virtual environment model from the virtual perspective. The environment model advantageously comprises the synthetic object which represents the vehicle. The computing device furthermore has a computing unit, in particular a processor, which is configured such that it executes the steps of the method according to the invention.The invention further relates to a vehicle comprising the computing device according to the invention and a display device.Further advantages result from the following description of exemplary embodiments with reference to the figures. FIG. 1 : Flow diagram of the method FIG. 2 a : View of the environment model in the first state FIG. 2 b : further view of the environment model in the first state FIG. 2 c : View of the environment model in the second stateExemplary EmbodimentsFIG. 1 shows a flow diagram of the method schematically as a block diagram. In a step 110 of the method, sensor data are acquired which represent at least a partial area of the environment of the vehicle. The sensor data include, in particular, camera images and / or distance data. The camera images are advantageously captured by means of a vehicle camera. The distance data are advantageously captured or determined by means of an ultrasonic sensor, by means of a radar sensor and / or by means of a lidar sensor and / or captured or determined by means of a vehicle camera by means of structure from motion and / or stereo vision methods. The vehicle preferably comprises at least four cameras, in particular wide-angle cameras, and at least four distance sensors, in particular ultrasonic sensors and / or radar sensors. Based on the sensor data, in step 120, a virtual environment model is determined, in particular a surround view view or a view perpendicular from above or a top down view. For the surround view view or a view perpendicular from above, camera images are advantageously transformed into the corresponding virtual perspectives and projected onto a projection surface of the environment model, in particular as a texture, wherein the projection surface is advantageously matched as a function of the distance data and / or as a function of detected static and / or dynamic objects, in order to reduce image distortions or artifacts and / or registration errors at joint lines. In a further step 130 of the method, odometry data of the vehicle are acquired, which represent or describe the movement of the vehicle. The odometry data comprise, for example, the acceleration of the vehicle and / or the speed of the vehicle and / or the position of the vehicle and / or a rotational speed of a drive axle of the vehicle.Additionally or alternatively, the odometry data can in particular comprise a plurality of optical flow vectors determined based on a sequence of camera images, which at least represent the movement of the vehicle to the static environment. In another step 140 of the method, a driving situation of the vehicle is detected based on the captured sensor data, the captured odometry data and / or a captured input of the driver. The detected driving situation activates, in particular, an automatic driving maneuver, for example an automatic parking function and / or an automatic maneuver at a narrow point of the roadway, etc. For example, an imminent parking process of the vehicle is detected as a driving situation on the basis of a speed of the vehicle of less than 30 km / h and / or on the basis of a position of the vehicle in a large parking space and / or on the basis of a position in the vicinity of parking spaces on the road edge and / or on the basis of a position in the vicinity of the destination detected in a navigation system and / or on the basis of a parking space detected on a camera basis. A display 150 of the determined virtual environment model is then carried out from a virtual perspective, wherein the vehicle is represented by a synthetic object in the environment model. The display 150 of the determined virtual environment model from the virtual perspective takes place in particular first in a first state, in which the synthetic object in particular has a fixed distance and / or angle reference to the virtual perspective, whereby the synthetic object is advantageously arranged centrally in the virtual environment model during the travel of the vehicle in the first state. According to the invention, the environment model is switched from a first state to a second state in step 160 as a function of the detected driving situation, wherein the position of the virtual perspective does not change with respect to the displayed imaged environment in the second state. In particular, the synthetic object representing the vehicle is moved in the environment model based on the acquired odometry data or the position of the virtual perspective is adjusted with respect to the synthetic object representing the vehicle based on the odometry data. In the second (static) state, the position of the virtual perspective is therefore advantageously not changed to the environment depicted in the environment model. In other words, the viewer of the displayed environment model suddenly perceives a static display by switching 160 into the second (static) state of the display, in which, apart from other moving objects, only the synthetic object representing the (ego) vehicle moves, since the orientation of the virtual camera with respect to the imaged environment no longer changes. In other words, mapped static objects in the environment model in the second (static) state are not displaced from the virtual perspective according to a movement of the vehicle. Instead, after the switch 160 to the second state, the synthetic object representing the vehicle is moved in the environment model based on the captured odometry data or the position of the virtual perspective is adjusted with respect to the synthetic object representing the vehicle based on the odometry data, wherein an adjustment of the dimensions of the environment model can be provided, in particular an extension of the edges of the environment model, which can be connected to zoom-out of the camera in the second state, wherein the viewing direction of the virtual camera is advantageously maintained in the second state during zoom-out. The switchover 160 from the first state of the display of the environment model to the second state of the display of the environment model preferably takes place at a point in time at which the virtual perspective is oriented toward the environment model in the direction of a roadway course. As a result, the display of the environment model in the second state is aligned with the roadway course, resulting in great clarity. In addition, in a subsequent optional step 170, at least one additional mapping is displayed in the second state as a function of the captured sensor data and / or the determined virtual environment model, wherein the additional mapping represents in particular the environment model in the first state or the shortest distance between the vehicle and an object in the environment. The additional mapping is in this case illustrated in particular smaller than the displayed environment model in the second state, for example with an imaging area smaller than a quarter of the imaging area of the displayed environment model. Preferably, the additional map represents a zoomed out or zoomed out portion of the environment model. The additional mapping is advantageously superimposed on the displayed environment model.FIG. 2 ashows a view of the displayed environment model 200 in the first state, wherein the environment model 200 is depicted here, by way of example, from a virtual perspective perpendicularly from above. The display 150 of the environment model 200 in the first state comprises the (ego) vehicle as a synthetic object 210, wherein the synthetic object in the first state is typically represented in the middle of the displayed environment model 200. The environment model 200 also comprises, for example, other parked other vehicle 220, overlaid overlay on a recognized parking space P, a sidewalk 230 and / or static objects 230, 240, 250, 260 and 270 which are imaged and / or recognized or determined in the projected camera images and are overlaid in a abstract manner. The environment model 200 is determined in particular on the basis of the captured sensor data. In particular, captured camera images are transformed into a corresponding virtual perspective and projected onto a projection plane. The static objects may be, for example, trees 240, a sidewalk 250 (e.g., into a park), and / or posts 260 of the sidewalk to prevent entry of vehicles. Optionally, dynamic objects such as pedestrians and / or cyclists and / or other moving vehicles may be presented and optionally marked (not shown here), thereby assisting the driver's attention to dynamic objects. Areas that cannot be viewed by means of the vehicle sensors can be supplemented or determined and / or estimated, for example, on the basis of map data and / or on the basis of sensor data stored for the current position of the vehicle, for example the course of the sidewalk 230 behind the other parked other vehicles 220.FIG. 2 b shows the display of the environment model 200 from FIG. 2 a at a later point in time during a parking process to the parking space P as a (detected) driving situation not according to the invention in the first state, that is to say without switching 160 the display of the environment model 200 from the first state to the second state. Due to the lack of a changeover, the alignment of the display of the environment model 200 does not change with respect to the alignment and position of the vehicle 210, as a result of which a transverse position of the displayed environment model with respect to the roadway course results. This produces a poor clarity for the driver, since, for example, the parking space P is no longer displayed completely and the road and the parked other vehicles 220 are displayed obliquely.In contrast, FIG. 2 cshows the view according to the invention of the environment model 200 after the switchover from the first state according to FIG. 2 ato the second state, wherein the switchover of the display of the environment model 200 to the second state takes place in particular on the basis of the detected imminent parking process as driving situation. The virtual perspective in the second state is aligned with the roadway course 280 in FIG. 2 c. The parking process is carried out automatically, in particular. In the second state of the displayed environment model 200, the virtual perspective is therefore no longer changed with respect to the displayed environment. In the second state, the virtual perspective therefore changes only with respect to the (ego) vehicle represented as a synthetic object 210. This means that, for example, the positions and alignments of the static objects 230, 240, 250, 260 and 270 and of the parked foreign vehicles 220 and of the parking space P overlaid on them no longer change in the displayed environment model 200. Instead, the (ego) vehicle 210 is moved in the displayed environment model 200 according to captured odometry data or the virtual perspective is changed depending on the captured odometry data in the relationship to the vehicle. It can be provided that the environment model 200 is expanded in this case in accordance with a detection range of the sensors of the vehicle that changes over time, wherein the edges of the environment model are changed and / or the display of the environment model is zoomed out. In order to be able to better represent distances between the vehicle 210 and nearby objects, for example to the parked vehicle 220 and / or the static objects 230, 240, 250, 260 and 270, during the parking process in the second state, at least one additional mapping is preferably provided, which, in addition to the displayed environment model and / or for example as an overlay, is overlaid in a region of the environment model 200 that is not relevant for the detected driving situation. In FIG. 2 c, an additional mapping could be overlaid, for example, in the lower left corner of the display of the environment model, wherein the additional mapping represents, for example, the shortest distance between the vehicle represented as a synthetic object 210 and the upper parked other vehicle 220.References included in the specificationThis list of documents cited by the applicant has been produced in an automated manner and is only included for the better information of the reader. The list is not part of the German patent application or utility model application. The DPMA does not take any adhesion for any faults or omissions.Patent Literature citedDE 10 2017 218 074 A1

[0002] EP 3 571 091 B1

[0003] DE 10 2015 212 370 B4

[0004] DE 10 2021 208 473 A1

[0005]

Claims

Computer-implemented method for displaying an environment of a vehicle, comprising the following steps • acquisition (110) of sensor data which represent at least a partial area of the environment of the vehicle, in particular camera images and / or distance data are acquired as sensor data, • determination (120) of a virtual environment model (200) based on the sensor data, in particular a surround view view or a view perpendicular from above, and • acquisition (130) of odometry data of the vehicle which represent the movement of the vehicle, • detection (140) of a driving situation of the vehicle based on the acquired sensor data, the acquired odometry data and / or an acquired input of the driver, and • display (150) of the determined virtual environment model (200) from a virtual perspective, wherein the vehicle is represented by a synthetic object (210) in the environment model (200), characterized in that • switching (160) the displayed environment model (200) from a first state to a second state depending on the detected driving situation, wherein the orientation of the virtual perspective with respect to the displayed imaged environment does not change in the second state.Method according to Claim 1, wherein the changeover (160) from the first state to the second state takes place at a point in time at which the virtual perspective is aligned with the environment model (200) in the direction of a roadway course.Method according to one of the preceding claims, wherein in the second state the synthetic object representing the vehicle is moved in the environment model based on the acquired odometry data or the position of the virtual perspective with respect to the synthetic object representing the vehicle is adjusted based on the odometry data.Method according to one of the preceding claims, wherein the following step is carried out in the second state of the displayed environment model • overlay (170) of at least one additional mapping depending on the captured sensor data and / or the determined virtual environment model (200), wherein the additional mapping in particular represents the shortest distance between the vehicle and an object in the environment.A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the method according to any one of the preceding claims.Computing device, in particular central computing device or control device, comprising at least the following components • a first signal input for providing a first input signal, which represents sensor data detected by means of at least one environment sensor of the vehicle, • a second signal input for providing a second input signal, which represents odometry data detected by means of at least one odometry sensor, • a signal output for outputting an output signal for a display device, wherein the output signal represents a virtual environment model from a virtual perspective, and • a computing unit, in particular a processor, which is configured such that it carries out the steps of the method according to one of Claims 1 to 5.A vehicle comprising a computing device according to claim 6 and a display device.

Images