Method and device for estimating eye position of a driver of a vehicle

Abstract

The invention relates to a method for estimating an eye position (20) of a driver of a vehicle (50), wherein state data (10) of the vehicle (50) are detected by means of at least one sensor (51) and / or are queried in a vehicle control device (52), wherein the detected and / or queried state data (10) of the vehicle (50) are input as input data into a model (15) for the driver provided by means of a computing device (2), and wherein the eye position (20) of the driver is estimated and provided by means of the model (15). Furthermore, the invention relates to a device (1) for estimating an eye position (20) of a driver of a vehicle (50) and to a vehicle (50) having such a device (1).

Description

[0001] The invention relates to a method and a device for estimating an eye position of a driver of a vehicle. Furthermore, the invention relates to a vehicle having such a device.

[0002] Increasingly, head-up displays (HUD) are used in modern vehicles, in which information is projected into the field of view of the driver. In future HUDs in vehicles, information about other road users should be displayed spatially directly on the objects. In order to enable the information displayed in the view to be displayed directly on the objects of the environment or on the road plane, the position of the eyes of the driver relative to the vehicle needs to be known. For this purpose, the eyes of the driver are currently detected by means of sensors, for example cameras, and the current eye position is estimated on the basis of the detected data. However, due to the high requirements and only limited availability of computing resources, the view projected into the HUD on the basis of the thus estimated eye position is usually delayed, so that objects in the environment of the vehicle can no longer be correctly marked, since an undesirable latency and a misplacement in the display occur.

[0003] A method and a device for tracking the eyes of a user are known from US 2018 / 0144483 A1.

[0004] The technical problem addressed by the invention is to improve a method and a device for estimating an eye position of a driver of a vehicle.

[0005] The technical problem is solved by the invention by a method having the features of claim 1 and a device having the features of claim 8. Advantageous design solutions of the invention result from the dependent claims.

[0006] In particular, a method for estimating an eye position of a driver of a vehicle is provided, wherein state data of the vehicle are detected by means of at least one sensor and / or are queried in a vehicle control device, wherein the detected and / or queried state data of the vehicle are input as input data into a model for the driver provided by means of a computing device, and wherein the eye position of the driver is estimated and provided by means of the model.

[0007] Furthermore, in particular, a device for estimating an eye position of a driver of a vehicle is implemented, comprising a computing device, wherein the computing device is configured to receive state data of the vehicle detected by means of at least one sensor and / or queried in a vehicle control device, to provide a model for the driver, to input the detected and / or queried state data of the vehicle as input data into the model for the driver, and to estimate and provide the eye position of the driver by means of the model.

[0008] The method and the device implement an estimation of an eye position of a driver based on state data of the vehicle. To this end, a model of the driver is used, which estimates a current eye position based on the state data of the vehicle. Here, the model reflects the influence of the vehicle on the driver or the eye position.

[0009] An advantage of the method and the device is that the estimation of the eye position based on the state data requires less computing power, since the state data is generally not as extensive as the sensor data that is accumulated when detecting the eye position. Thereby, the estimated eye position can be provided with less delay, so that the estimated eye position is closer to the real eye position.

[0010] The state data can in particular comprise a speed, a longitudinal and / or lateral and / or vertical acceleration, a steering angle, a steering angle change, a wheel angle, a wheel angle change, etc. A vertical acceleration can for example occur when driving over a deceleration road arch (road arch) and strongly influences the eye position due to the resulting head movement. The state data can be detected by means of at least one sensor. Alternatively or in addition, the state data of the vehicle can also be queried by a vehicle control device of the vehicle, for example via a controller area network (CAN) bus.

[0011] The components of the device, in particular the computing device, can be designed individually or integrally as a combination of hardware and software, for example as program code that is executed on a microcontroller or microprocessor. It can however also be provided that the components are designed individually or integrally as an application-specific integrated circuit (ASIC).

[0012] The vehicle is in particular a motor vehicle. However, the vehicle can in principle also be another land, water, rail, air or space vehicle.

[0013] In one embodiment it is provided that at least one eye position of the driver is additionally detected by means of at least one sensor, wherein the detected at least one eye position is likewise input as input data into the model, wherein the eye position is estimated taking into account the detected at least one eye position. Although a latency occurs when taking into account the detected eye position or when estimating the current eye position based on the detected eye position, the eye position can be better estimated overall, since both the detected eye position (with latency) and the detected and / or queried vehicle state data are taken into account when estimating the current eye position. Thereby, in particular the viewing direction can be better taken into account. The eye position is thereby estimated taking into account the eye position estimated by means of the state data of the vehicle and the detected at least one eye position. It can in particular be provided that an eye position detected at an earlier point in time is taken into account when estimating.

[0014] In an embodiment it is provided that the model is provided at least partially by means of at least one machine learning method. Machine learning methods include in particular the use of trained artificial neural networks. The neural networks can be deep neural networks. The neural networks contain as input data the detected and / or queried state data of the vehicle and estimate as output data the eye position of the driver based on these input data.

[0015] In an embodiment it is provided that the model is provided at least partially by means of a physical model for the driver's body. The physical model of the driver here reflects in particular the physical or mechanical correlation between the vehicle state described by the state data and the eye position of the driver. Thereby, for example, the human body can be modeled as masses coupled to each other by elastic springs. The masses coupled to each other by springs are mechanically coupled at the lower end by means of the vehicle seat to the vehicle. The eye position is at the upper end. For this model, for illustration, for example, the lateral or longitudinal acceleration of the vehicle can be observed. Due to inertia and the coupling by means of the springs, changes in motion or accelerations of the vehicle are transmitted to the masses representing the driver's head with a time delay and their motion or travel path also influences the eye position. The physical model can be described mathematically by a system of coupled differential equations and solved. The solution is used by the computing device in order to estimate the eye position based on the state data. Furthermore, known human body models can also be used, for example the finite element human body models HUMOS (Human Model for Safety; EU project), THUMS (Total Human Model for Safety) and / or H-Model (ESI Group). The muscle activity within the human body model can be described, for example, by means of Hill's muscle model.

[0016] It can be provided that the physical model contains or models further components in addition to the driver's body. Thereby, for example, also the driver's seat can be taken into account, since this driver's seat is usually not designed fixedly, so that the motion and changes in motion of the vehicle are not immediately transmitted to the driver's body, but rather with a time delay or by means of the damping of the elastic seat cushion. This can be taken into account in the model by means of corresponding dampers or additional springs.

[0017] It can also be provided that the physical model and at least one machine learning method are used jointly in order to estimate the eye position. The respective results can be, for example, weightedly integrated (or rather summarized) and provided.

[0018] It can also be provided that the solution provided by the physical model is input as input data into at least one machine learning method, for example a (trained) neural network. The at least one machine learning method, for example the (trained) neural network, thereby estimates the eye position at least also on the basis of the solution provided by the physical model.

[0019] It is provided in an embodiment that the parameters of the model are determined or have been determined by means of training data, wherein the training data comprise detected and / or queried state data of the vehicle and respectively simultaneously detected eye positions of the driver by means of the sensors. The detected and / or queried state data and respectively the assigned detected eye positions are in particular synchronized with one another in time. The training data are used to determine the parameters of the model. If the model is provided by means of a neural network, the neural network is trained by means of the training data. This is achieved, for example, as follows: For this purpose, the detected and / or queried state data are input as input data into the neural network in a training phase. Here, respectively the detected eye positions which are synchronized in time with the respective state data are used as ground truth (English). For the training data, the deviation (so-called Loss) of the output of the neural network (for a given parameterization) from the respective ground truth is determined. The loss function used here is in particular chosen such that the parameters of the neural network are differentially related to the loss function. In the context of a gradient descent method, the parameters of the neural network are adjusted or adapted in each training step in accordance with the derivative of the (determined in a plurality of examples) deviation. These training steps are repeated very frequently until the loss no longer decreases. The neural network trained in this way (or in another suitable way) is subsequently used by the computing device for estimating the eye position. In the case of the use of a physical model, a classic parameter fitting can be implemented on the basis of the vehicle state data detected and / or queried as training data and respectively the eye positions of the driver synchronized in time therewith.

[0020] The detection and / or querying of the training data can take place, for example, on a test section set or selected for this purpose. As an alternative or in addition, the training data can also be detected or queried continuously, so that continuous or at least regular adjustments or updates of the model are possible.

[0021] In one embodiment it is provided that the parameters of the model are determined or have been determined taking into account at least one driver characteristic and / or the model is provided taking into account at least one driver characteristic. Thereby the model can be provided in particular for a specific driver or driver characteristics. The driver characteristics can be for example one of the following: name, age, sex, height, weight, mass distribution, body type / shape (e.g. slim, athletic, obese), etc. The at least one driver characteristic is likewise input as input data into the model. When determining the parameters, the at least one driver characteristic is accordingly taken into account as input data.

[0022] In one embodiment the model is adjusted or can be adjusted in a fine-tuning phase. Thereby the parameters of the model can be roughly determined in a first training phase in order to subsequently fine-tune in particular for one or more drivers of the vehicle in a fine-tuning phase. Thereby a model can be provided in a new vehicle which has for example already been able to roughly estimate the eye position. For example, the model can initially be parameterized by specifying at least one driver characteristic. After delivery of the vehicle, a fine-tuning phase can be carried out, for example in such a way that the state data of the vehicle are detected and / or queried for the driver of the vehicle over a predetermined period of time and collected as training data. At the same time or in a new training phase, the parameters of the model are fine-tuned on the basis of the collected training data.

[0023] Further features of the design of the device result from the description of the design of the method. The advantages of the device are here the same as in the design of the method, respectively.

[0024] Furthermore, a vehicle is provided which comprises at least one device according to one of the embodiments.

[0025] In one embodiment of the vehicle it is provided that the vehicle comprises a head-up display, wherein the head-up display is designed to use the estimated and provided eye position as input data when generating and providing a view which can be perceived by the driver. The head-up display calculates in particular a perspective-correct view for the driver or a correct view of information in the field of view of the driver on the basis of the estimated and provided eye position.

[0026] The application is explained in more detail below with reference to preferred embodiments according to the drawings. In the drawings:

[0027] Figure 1 A schematic diagram of one embodiment of a device for estimating the eye position of a driver of a vehicle is shown.

[0028] In Figure 1A schematic diagram showing an embodiment of the device 1 for estimating the eye position 20 of a driver of a motor vehicle 50 is shown in Fig. 1. In the example shown, the device 1 is designed in the motor vehicle 50. The device 1 performs the method described in the present disclosure.

[0029] The device 1 comprises a computing means 2 and a memory means 3. The memory means 3 can also be designed as part of the computing means 2. The computing means 2 is designed as a combination of hardware and software, for example as a program code which is executed on a microcontroller or microprocessor. The computing means 2 has access to the memory means 3 and performs computing operations on data stored in said memory means.

[0030] The state data 10 of the motor vehicle 50 is input into the computing means 2, for example via an input interface (not shown) provided for this purpose. The state data 10 is detected by sensors 51 of the motor vehicle 50 and / or queried and transmitted by a vehicle control device 52. The state data can include, inter alia, speed, longitudinal and / or lateral acceleration, steering angle, steering angle change, wheel angle, wheel angle change, etc.

[0031] The computing means 2 provides a model 15 for the driver of the motor vehicle 50. Here, the model 15 describes or reflects an association between the state data 10 of the motor vehicle 10 and the eye position 20 of the driver of the motor vehicle 50. The detected and / or queried state data 10 is input into the model 15 from the computing means 2. The computing means 2 estimates the eye position 20 of the driver by means of the model 15 and provides the estimated eye position 20. The computing means 2 outputs the estimated eye position 20, for example as an analog or digital eye position signal, for example via an output interface (not shown) provided for this purpose. The eye position signal comprises, inter alia, the eye position in the form of three-dimensional coordinates in a vehicle-related coordinate system.

[0032] The estimation of the eye position 20 is repeated, inter alia, continuously or periodically, so that the current eye position 20 of the driver is continuously estimated and provided on the basis of the current detected and / or queried state data 10 of the motor vehicle 50.

[0033] It can be provided that at least one eye position 11 of the driver is additionally detected by means of at least one sensor 53. The detected at least one eye position 11 is also input as input data 10 into the model 15, wherein the eye position 20 is estimated by the model 15 taking into account the detected at least one eye position 11.

[0034] It can be provided that the model 15 is provided at least partially by at least one machine learning method. In particular, the model can be provided by means of a trained neural network.

[0035] It can be provided that the model 15 is provided at least partially by means of a physical model 15 of the driver's body. This physical model comprises, inter alia, a description of masses coupled to one another by means of elastic springs.

[0036] It can be provided that the parameters of the model 15 are determined or have been determined by means of training data, wherein the training data comprise detected and / or queried state data 10 of the motor vehicle 50 and the eye position 11 of the driver detected by means of the sensor 53 respectively at the same time.

[0037] It can be provided that the parameters of the model 15 are determined taking into account at least one driver characteristic 12 and / or the model 15 is provided taking into account at least one driver characteristic 12. It can be provided, for example, that the driver can set his driver characteristics 12 in a first drive through the motor vehicle 50. On the basis of the set driver characteristics 12, the parameters of the model 15 are then selected and set. It can be provided, for example, that the parameters of the model 15 are stored categorically in a database according to the driver characteristics 12 and are queried from the database if required or after setting the driver characteristics and the model 15 is parameterized accordingly.

[0038] It can also be provided that the model 15 is adjusted or can be adjusted in a fine-tuning phase. In the example mentioned previously, this can take place after setting the driver characteristics 12. The model 15 is roughly parameterized by means of the driver characteristics 12 (for example gender: female, age: 25, height: 1.69 m, weight: 60 kg, body type: athletic). It can be provided here, for example, that if there are no known or stored parameters for one driver characteristic or a combination of several driver characteristics, the parameters of the model 15 are interpolated or extrapolated on the basis of the driver characteristics 12. The parameters of the model 15 are then adapted to the driver in a coordinated manner by means of the fine-tuning phase. To this end, the state data of the motor vehicle 50 are detected and / or queried again as training data and the eye position of the driver is detected which is synchronized in time. The model 15 is then adapted to the detected training data. In the case of a neural network being used to provide the model 15, in particular a retraining takes place (also referred to as "fine-tuning").

[0039] It is provided in particular that the estimated eye position 20 is input into a head-up display 54 of the motor vehicle 50. The estimated and provided eye position 20 is used by the head-up display 54 to generate and provide a view which can be perceived by the driver in order to estimate or calculate the viewing angle of the driver.

[0040] Compared to the detection and estimation of the eye position on the basis of sensor data of a sensor which detects the eye position, the device 1 and the method carried out thereby enable the estimation of the eye position 20 of the driver with a significantly reduced latency.

[0041] List of reference signs

[0042] 1 device

[0043] 2 computing device

[0044] 3 memory device

[0045] 10 status data

[0046] 11 detected eye position

[0047] 12 driver characteristic(s)

[0048] 15 model of driver

[0049] 20 estimated eye position

[0050] 50 (motor) vehicle

[0051] 51 sensor

[0052] 52 vehicle control device

[0053] 53 sensor

[0054] 54 head-up display

Claims

1. A method for operating a vehicle (50) having a head-up display (54), in, detecting state data (10) of a vehicle (50) by means of at least one sensor (51) and / or querying the vehicle control unit (52), The method comprises inputting detected and / or queried state data (10) of a vehicle (50) into a model (15) for the driver provided by means of a computing device (2), wherein the model (15) is provided at least partially by means of a physical model of the driver's body, which includes a description of masses coupled to one another by means of elastic springs, and wherein the driver's eye position (20) is estimated and provided with the aid of the model (15), The estimated and provided eye position (20) is used as input data by a head-up display to generate and provide a view that can be perceived by the driver, wherein the head-up display calculates a perspectively correct view for the driver or a correct view of the information in the driver's field of view based on the estimated and provided eye position.

2. The method according to claim 1, characterized in that At least one eye position (11) of the driver is additionally detected by means of at least one sensor (53), wherein the at least one detected eye position (11) is also input into the model (15) as input data, wherein the eye position (20) is estimated taking into account the at least one detected eye position (11).

3. The method according to claim 1 or 2, characterized in that The model (15) is provided at least in part by means of at least one machine learning method.

4. The method according to claim 1, characterized in that The parameters of the model (15) are determined or have been determined with the aid of training data, wherein the training data include detected and / or queried state data (10) of the vehicle (50) and the eye position (11) of the driver detected simultaneously with the aid of a sensor (53).

5. The method according to claim 1, characterized in that The parameters of the model (15) are determined or have been determined taking into account at least one driver characteristic (12) and / or the model (15) is provided taking into account at least one driver characteristic (12).

6. The method according to claim 1, characterized in that The model (15) is adjusted or can be adjusted in a fine-tuning phase.

7. A vehicle (50), comprising: Head-up display (54); and At least one device (1) for estimating the eye position (20) of a driver of a vehicle (50), comprising a computing device (2), wherein the computing device (2) is configured to: - receiving status data (10) of a vehicle (50) detected by means of at least one sensor (51) and / or status data (10) queried in a vehicle control device (52); - providing a model (15) for the driver, wherein the model (15) is provided at least partially with the aid of a physical model for the driver's body, the physical model including a description of masses coupled to one another via elastic springs; - inputting the detected and / or queried state data (10) of the vehicle (50) as input data into a model (15) for the driver; and - estimating and providing the driver's eye position (20) with the aid of the model (15), The head-up display (54) is configured to use the estimated and provided eye position (20) as input data when generating and providing a view perceptible to the driver, wherein the head-up display is further configured to calculate a perspective-correct view for the driver or a correct view of information in the driver's field of view based on the estimated and provided eye position.

Images