Method and system for operating a motor vehicle equipped with at least one driver assistance function and an external vehicle load

Abstract

The invention relates to a method for operating a motor vehicle (4) equipped with at least one driver assistance function and with an external vehicle load (9), wherein images or videos are recorded from outside the motor vehicle (4) using a mobile device (2), from which image data is obtained, which is processed and evaluated with regard to the external vehicle load (9), and wherein a three-dimensional model of the motor vehicle (4) with the external vehicle load (9) is generated. To simplify the operation of a motor vehicle (4) equipped with at least one driver assistance function and an external vehicle load, a realistic and true-to-scale three-dimensional geometry model (26) of the motor vehicle (4) with the external vehicle load (9) is generated using a special image processing method (10), whereby not only the geometry of a load carrier device (7) mounted or attached to the motor vehicle (4), but also an actual external load (9) of the motor vehicle (4) is captured, wherein the realistic three-dimensional and true-to-scale geometry model (26) of the motor vehicle (4) with the external vehicle load (9), in particular with a weight distribution determined therefrom, is used in the driver assistance function (30,31,32).

Description

[0001] The invention relates to a method and a system for operating a motor vehicle equipped with at least one driver assistance function and having an external vehicle load, wherein images or videos are recorded from outside the motor vehicle using a mobile terminal, from which image data are obtained, which are processed and evaluated with regard to the external vehicle load, wherein a three-dimensional model of the motor vehicle with the external vehicle load is generated.

[0002] German patent application DE 10 2020 112 549 A1 discloses a method for measuring an attachment mounted on a vehicle, wherein an exterior area of ​​the motor vehicle is recognized from at least one image taken by a mobile device, and based on this, a three-dimensional model of the recognized exterior area of ​​the motor vehicle is determined. In addition, an attachment is recognized from at least one image taken by the mobile device, and the dimensions of the attachment are determined using the three-dimensional model. German patent application DE 10 2022 101 823 A1 discloses a method for determining the dimensions of a trailer, wherein images from a camera of a motor vehicle towing the trailer or from a mobile device are evaluated, and the dimensions of the trailer are determined therefrom.German patent application DE 10 2018 008 996 A1 discloses a method for recognizing a trailer attached to a motor vehicle based on a comparison of currently recorded data and stored data of the trailer. The currently recorded data is determined by evaluating images from a camera of the motor vehicle and a mobile device. German patent application DE 10 2015 117 903 A1 discloses a motor vehicle with a coupled trailer, wherein the trailer's dimensions are specified via a mobile device.The German patent application DE 10 2016 212 181 A1 discloses a method for determining length and / or size ratios of a trailer for use with a trailer maneuvering assistant, wherein a digital image sequence of the trailer is recorded from at least two different positions, which is transmitted to an evaluation unit that determines the length and / or size ratios of the trailer by means of a 3D reconstruction.German patent application DE 10 2013 006 330 A1 discloses a method for determining dimensions of a vehicle and / or at least one object transported by the vehicle in an area to be measured, the method comprising the following: optical detection of an area of ​​the vehicle to be measured and / or of the at least one object transported by the vehicle from at least a first viewing angle using an electronic camera; optical detection of at least one reference dimension on the vehicle using the electronic camera; determination of dimensions of the vehicle and / or of the at least one object transported by the vehicle by comparing the reference dimension with the area to be measured.

[0003] The object of the invention is to simplify the operation of a motor vehicle equipped with at least one driver assistance function with an external vehicle load.

[0004] The object is achieved in a method for operating a motor vehicle equipped with at least one driver assistance function and having an external vehicle load, wherein images or videos are taken from outside the motor vehicle using a mobile terminal, from which image data are obtained, which are processed and evaluated with regard to the external vehicle load, wherein a three-dimensional model of the motor vehicle with the external vehicle load is generated, in that a realistic three-dimensional and true-to-scale three-dimensional geometric model of the motor vehicle with the external vehicle load is generated using a NeRF image processing method, wherein not only the geometry of a load-carrying device mounted or attached to the motor vehicle, but also an actual external load of the motor vehicle is recorded,The realistic three-dimensional and true-to-scale geometric model of the motor vehicle with the external vehicle load, in particular with a weight distribution determined from it, is used in the driver assistance function. The NeRF image processing method converts two-dimensional images or videos into a realistic three-dimensional model of the motor vehicle with the external vehicle load. Due to the known vehicle contour, external attachments can be recognized and, in particular, their geometric properties and shape can be captured. The abbreviation NeRF stands for Neural Rendering Field. This requires increased effort both in capturing the images or videos with the mobile camera and in the special image processing to create the realistic and true-to-scale three-dimensional geometric model of the motor vehicle with the external vehicle load.which can then be used extremely effectively in the driver assistance function. For example, the realistic three-dimensional geometric model can capture and account for bulky objects that protrude upwards, to the sides, forwards, and / or backwards beyond the normal vehicle dimensions, and / or generate airflow or air resistance in interaction with the vehicle geometry. The increased effort involved in capturing images or videos also means that a person, such as the driver of the vehicle, must walk around the vehicle with the camera at least once or several times before the journey in order to fully capture the geometrical details of the vehicle's external load. This special image processing method accepts the fact that a relatively high level of computing power and / or processor performance is required even before the journey begins.to generate a realistic three-dimensional geometric model of the vehicle. The function can be linked to conditions for starting a journey, for example, when the trailer hitch is connected or when a roof load is detected.

[0005] A preferred embodiment of the method is characterized in that the special image processing method is used to generate an aerodynamic consumption model of the motor vehicle, taking into account the additional, external geometric characteristics of the vehicle's external load. This model is used to calculate the range. This is particularly advantageous when the range, for example, of a purely electrically powered motor vehicle, is significantly affected by the external vehicle load or the use of a trailer.

[0006] A further preferred embodiment of the method is characterized in that the realistic three-dimensional geometric model of the motor vehicle with the external vehicle load is used to calculate a clearance height. This provides advantages, particularly when the external vehicle load significantly exceeds a normal vehicle height.

[0007] A further preferred embodiment of the method is characterized in that the realistic three-dimensional geometric model of the motor vehicle with the external vehicle load is used to calculate a passage width. This can prevent unwanted collisions caused by an excessively wide external vehicle load.

[0008] A further preferred embodiment of the method is characterized in that the realistic three-dimensional geometric model of the motor vehicle with the external vehicle load is used for an automated parking function. This allows automated parking to be performed with sufficient safety and without unwanted collisions, even with the external vehicle load.

[0009] A further preferred embodiment of the method is characterized in that the realistic three-dimensional geometric model of the motor vehicle with the external vehicle load is used to determine a center of gravity of the motor vehicle with the external vehicle load that has been altered by the vehicle load. This advantageously allows for automated driving to take into account, for example, the fact that the cornering behavior of the motor vehicle with the external vehicle load is impaired, or safety-critical driving conditions can be automatically avoided and prevented, such as tipping over, lurching, or excessive acceleration and deceleration, which can push load securing to its limits in high-performance drive systems.

[0010] A further preferred embodiment of the method is characterized in that the realistic three-dimensional geometric model of the motor vehicle with the external vehicle load is used in route planning for the motor vehicle with the external vehicle load. For example, passages that are too low or too narrow with regard to the external vehicle load can be avoided through appropriate route planning.

[0011] A further preferred embodiment of the method is characterized in that a driver of the motor vehicle with the external vehicle load records a video of the motor vehicle with the external vehicle load with the mobile terminal using a special mobile application before a journey, wherein the driver walks around the motor vehicle with the external vehicle load with the recording mobile terminal until the mobile application informs the driver that the acquired image data are sufficient to generate the realistic three-dimensional geometric model of the motor vehicle with the external vehicle load. The mobile terminal is, for example, a smartphone with a suitable camera. The special mobile application is, for example, an app that is installed on the mobile terminal.Of course, a passenger or any other person can also record the video of the vehicle with the external vehicle load using the mobile device.

[0012] The invention further relates to a computer program, in particular a mobile application, comprising instructions that, when the computer program is executed by a computing device, cause the device to execute a previously described method. Parts or modules of the computer program are stored, for example, in the mobile device. Further parts or modules of the computer program are stored, for example, in a driver assistance system of the motor vehicle.

[0013] In a system for operating a motor vehicle equipped with at least one driver assistance function with an external vehicle load according to a previously described method, the above-mentioned object is achieved alternatively or additionally by linking the mobile application to the driver assistance function via the special image processing method. This allows automated driving of the motor vehicle to be made safer and more comfortable, even with an external vehicle load.

[0014] Further advantages, features, and details of the invention will become apparent from the following description, in which various embodiments are described in detail with reference to the drawings. They show: Fig. 1 a schematic representation of a side section through a motor vehicle with an external vehicle load and with a person standing in front of the motor vehicle taking a picture with a mobile device; Fig. 2 a schematic representation of how a flow simulation can be carried out from a realistic and true-to-scale three-dimensional geometric model of the motor vehicle with the external vehicle load generated using image data; Fig. 3 a block diagram of an application layer; and Fig. 4 a block diagram illustrating a method for detecting an actual external vehicle load using NeRF image processing, wherein a realistic and true-to-scale three-dimensional geometric model of the motor vehicle with the external vehicle load is used in a driver assistance system.

[0015] In Fig. 1 shows how a person 8 is using a mobile device 2 to take pictures and / or a video of a motor vehicle 4 with an external load 9. The external load 9 is arranged on a load-carrying device 7 of the motor vehicle 4, designed as a roof rack 6. In addition, an optional trailer 5 is attached to the rear of the motor vehicle 4. A driver 1 is sitting in the motor vehicle 4 in Fig. 1.

[0016] Unlike in Fig. 1, the person 8 or the driver 1 will walk with the mobile terminal 2 at least once by three hundred and sixty degrees around the motor vehicle 4, if necessary with the trailer 5, so that the external load 9 of the motor vehicle 4 is completely recorded in all three spatial directions.

[0017] In Fig. 2 indicates that the image data acquired with the mobile terminal 2 are processed in a special image processing method 10. The special image processing method 10 is a NeRF image processing method.

[0018] In the NeRF image processing method 10, the Fig. 1 obtained two-dimensional images or videos, with the help of which a realistic three-dimensional model of the motor vehicle 4 with the external load 9 is generated.

[0019] In block 11, for example, a parameter for a range calculation is generated from a projection of the frontal area of ​​the motor vehicle 4 with the external load 9. In block 12, an aerodynamic consumption model is calculated from the three-dimensional geometric model using a flow simulation.

[0020] In Fig. Figure 3 shows an application level 15 with three blocks 16, 17, and 18. In block 16, the realistic three-dimensional geometric model of the motor vehicle with the external load generated by the special image processing method 10 is used to avoid collisions during parking. For example, an automated parking process is aborted if the clearance height or width in a garage, especially an underground garage, is too low.

[0021] In block 17, the realistic three-dimensional geometric model of the vehicle with the external vehicle load is taken into account during route planning. For example, a reduced range of the vehicle due to the external vehicle load comes into play here. Furthermore, the cornering behavior of the vehicle with the external vehicle load, which is restricted by the external vehicle load, can be taken into account.

[0022] In block 18, for example, an adapted range calculation in automated driving mode is taken into account, for example by a slower speed with lower energy consumption.

[0023] In Fig. 4, a block 20 indicates a special mobile application or app on the mobile device. Blocks 21, 22, and 23 symbolize a chassis, seat occupancy, or occupant detection. Block 24 illustrates image processing, possibly LiDAR processing. Block 24 also includes vehicle model recognition and scaling derivation based on vehicle characteristics.

[0024] Block 25 comprises the NeRF method for image processing or image reconstruction. Block 26 comprises the realistic three-dimensional model of the motor vehicle with the external load and, if applicable, with a trailer that also carries an external load. The three-dimensional model is advantageously scaled metrically using calculated scaling factors. A simple projection can be used to calculate the changed flow resistance area compared to the unloaded state.

[0025] Block 27 includes collision avoidance during parking, for example when an automated parking process in front of a garage or underground car park is aborted due to insufficient clearance height or width.

[0026] Block 28 includes a warning message about a changed center of gravity or impaired handling of the vehicle due to an increased roof load.

[0027] Block 29 comprises a preferably simplified three-dimensional flow simulation for range calculation in the vehicle or in the backend with live driving data.

[0028] Block 30 comprises an automated parking function. Block 31 comprises an automated driving function with an adapted range calculation. Block 32 comprises an adapted route planning, advantageously taking into account the ranges of automated driving functions in curves in relation to the external load of the motor vehicle.

[0029] By creating a realistic three-dimensional model of the motor vehicle using a mobile device, in particular a smartphone camera, and processing it using NeRF, and transferring this directly to a metric scale using the known vehicle dimensions, an aerodynamic consumption model with the payload can be calculated.

[0030] To do this, the driver must be asked to launch the app on their smartphone before setting off if a trailer or roof rack is detected. This requires the driver to walk around the vehicle and film the vehicle, including the load and, if applicable, the trailer, from all angles.

[0031] The NeRF model uses the recorded individual frames of the video, calculates the smartphone's movement, and builds a three-dimensional model of the filmed vehicle / trailer combination. With precise knowledge of the vehicle's dimensions, the resulting model can be scaled to a scaled 3D model of the vehicle with its payload / trailer combination. Even with very simplified and estimated calculations of aerodynamic drag, the range calculation becomes more accurate. Reference symbol 1 driver 2 mobile devices 4 Motor vehicle 5 optional trailer 6 roof racks 7 Loading device 8 people 9 external loading 10 special image processing methods 11 blocks 12 aerodynamic consumption model 15 Application level 16 blocks 17 Blocks 18 blocks 20 special mobile applications 21 Block 22 blocks 23 Block 24 blocks 25 blocks 26 realistic three-dimensional model 27 Block 28 blocks 29 Block 30 Driver assistance function 31 Driver assistance function 32 Driver assistance function

Claims

[1] Method for operating a motor vehicle (4) equipped with at least one driver assistance function (30, 31, 32) and having an external vehicle load (9), wherein images or videos are recorded from outside the motor vehicle (4) using a mobile terminal (2), from which image data are obtained, which are processed and evaluated with regard to the external vehicle load (9), wherein a three-dimensional model of the motor vehicle (4) with the external vehicle load (9) is generated, characterized bythat a realistic and true-to-scale three-dimensional geometric model (26) of the motor vehicle (4) with the external vehicle load (9) is generated using a NeRF image processing method (10), wherein not only the geometry of a load-carrying device (7) mounted or attached to the motor vehicle (4) but also an actual external load (9) of the motor vehicle (4) is recorded, wherein the realistic three-dimensional and true-to-scale geometric model (26) of the motor vehicle (4) with the external vehicle load (9), in particular with a weight distribution determined therefrom, is used in the driver assistance function (30, 31, 32). [2] Method according to claim 1, characterized bythat the NeRF image processing method (10) is used to generate an aerodynamic consumption model (12) of the motor vehicle (4) taking into account the additional, external geometric characteristics of the external vehicle load (9), which is used for a range calculation. [3] Method according to one of the preceding claims, characterized by that the realistic three-dimensional geometric model (26) of the motor vehicle (4) with the external vehicle load (9) is used to calculate a clearance height. [4] Method according to one of the preceding claims, characterized by that the realistic three-dimensional geometric model (26) of the motor vehicle (4) with the external vehicle load (9) is used to calculate a passage width. [5] Method according to one of the preceding claims, characterized bythat the realistic three-dimensional geometric model (26) of the motor vehicle (4) with the external vehicle load (9) is used for an automated parking function (30). [6] Method according to one of the preceding claims, characterized by that the realistic three-dimensional geometric model (26) of the motor vehicle (4) with the external vehicle load (9) is used to determine a center of gravity of the motor vehicle (4) with the external vehicle load (9) that has changed due to the vehicle load. [7] Method according to one of the preceding claims, characterized by that the realistic three-dimensional geometric model (26) of the motor vehicle (4) with the external vehicle load (9) is used in route planning (32) for the motor vehicle (4) with the external vehicle load (9). [8] Method according to one of the preceding claims, characterized bythat a driver (1) of the motor vehicle (4) with the external vehicle load (9) records a video of the motor vehicle (4) with the external vehicle load (9) with the mobile terminal (2) using a special mobile application (20) before a journey, wherein the driver (1) walks around the motor vehicle (4) with the external vehicle load (9) with the recording mobile terminal (2) until the mobile application (20) informs the driver (1) that the image data obtained are sufficient to generate the realistic three-dimensional geometric model (26) of the motor vehicle (4) with the external vehicle load (9). [9] Computer program, in particular mobile application (20), comprising instructions which, when the computer program is executed by a computing device, cause the latter to carry out a method according to one of the preceding claims. [10] System, arranged to operate a motor vehicle (4) equipped with at least one driver assistance function (30, 31, 32) with an external vehicle load (9) according to a method according to claim 8, characterized by that the mobile application (20) is linked to the driver assistance function (30,31,32) via the NeRF image processing method (10).

Images