Method and device for determining a driver's level of attention while driving a vehicle

Abstract

The invention relates to a method for determining the level of attention of a driver (19) while driving a vehicle (3), in which the driver's (19) level of attention is inferred based on the driver's (19's) eye configuration that develops during the journey. The eye configuration is measured by monitoring the accommodation of the lens of the driver's (19's) respective eye (43) and creating a spatial model of the environment in front of the vehicle (3). The accommodation profile is correlated with the distance profile of an object (29, 39, 45) included in the environment model. Changes in the size of the lens (44) of the respective eye (43) are detected, and the pupil diameter (47) is measured, which must be adjusted to see the object (29, 39, 45) clearly.

Description

[0001] The invention relates to a method for determining the attention of a driver during a journey in a vehicle, in which the level of attention of the driver is inferred depending on an eye configuration of the driver that develops during the journey, and to a device for carrying out the method.

[0002] From DE 11 2019 006 700 T5, a device for determining attention and a method for determining attention are known. The device comprises an image processing unit, a pupillary distance calculation unit, a heterophoria detection unit, and an attention determination unit. The image processing unit determines first reference coordinates, second reference coordinates, first pupil coordinates, and second pupil coordinates from an image of the driver's two eyes. From these coordinates, the pupillary distance calculation unit and the heterophoria detection unit determine the states of the eyeballs of both eyes, from which the driver's attention is inferred.

[0003] US Patent 9,834,221 B2 discloses a method and a device for determining driver attention using at least one digital camera device and a control unit. The camera device detects the eye configurations of a vehicle driver. The control unit compares the detected eye configurations with previously stored models and determines whether the detected eye configurations are looking within or outside a predetermined field of view.

[0004] DE 10 2008 038 855 A1 discloses a system for detecting a hazardous situation, comprising a detection unit designed to detect information from an information system of at least one person, in particular as neuronal impulses and / or hormonal signals and / or metabolic processes in the nervous system, and an evaluation unit designed to detect a hazardous situation from the detected information, in particular a hazardous situation in connection with the movement of a vehicle.

[0005] US Patent 2019 / 0236386A1 describes a computer-implemented method for detecting distracted driving of a motor vehicle.

[0006] The object of the invention is to provide a method and a device for determining the attention of a driver during a journey in a vehicle, with which the accuracy of the attention determination is improved.

[0007] The invention is defined by the features of the independent claims. Advantageous further developments and embodiments are the subject of the dependent claims. Further features, applications, and advantages of the invention will become apparent from the following description and the explanation of exemplary embodiments of the invention illustrated in the figures.

[0008] The problem is solved by the subject matter of claim 1.

[0009] In the method described at the beginning, used to determine a driver's attention while driving, the driver's level of attention is inferred from their eye configuration during the journey. This involves monitoring the accommodation of the eye's lens and creating a spatial model of the environment in front of the vehicle. The accommodation profile is then correlated with the distance profile of an object within the modeled environment to the vehicle. By evaluating the actual accommodation profile of the eye's lens, during which the lens changes its refractive power to focus on objects at different distances, the driver's cognitive attention is assessed with high accuracy while driving.This allows for reliable detection of whether objects viewed by the driver have also been cognitively processed. This also reliably identifies situations where the driver is aware of the driving situation ahead, i.e., has it within their field of vision, but is not focused on it, as can occur with daydreaming or staring, thus limiting the driver's actual attention. By taking lens accommodation into account, a traffic hazard can therefore be prevented.

[0010] In one implementation, the spatial environment model is created using data acquired by environmental sensors located in, on, and / or attached to the vehicle. This can be achieved, for example, using camera sensors and / or lidar sensors. Continuously adapting the environmental model to the surroundings traversed by the vehicle increases the accuracy of the method. Alternatively, the spatial environment model can also be created in an external cloud and cyclically transmitted to the vehicle.

[0011] In a further embodiment, a vehicle trajectory with a tolerance range is inserted into the spatial environment model. This tolerance range serves to define an active control range within which driver attention is to be ensured.

[0012] In a further refinement, the object is defined depending on the driver's viewing angle. This ensures that only objects detected by the driver are evaluated.

[0013] In a further embodiment, a viewing angle and accommodation parameters are determined from images of the eye. For each image, it is checked whether an intersection of the viewing angles of both eyes lies within the tolerance range surrounding the vehicle's trajectory. For intersections that lie within the tolerance range, a distance difference between the object's distance and the accommodation parameters is determined, and from this, the relative course of the eye's accommodation parameters to the object's distance profile is calculated. Depth is not determined from the accommodation; instead, the relative course between accommodation and depth profile within the spatial environment is correlated. Based on the determined viewing angle, it is then established whether it intersects with the tolerance range of the vehicle's trajectory.

[0014] In a further embodiment, it is cyclically checked whether the driver is observing the tolerance range while the accommodation parameters are being determined. The accommodation parameters can then be easily adjusted by evaluating a video stream of the eye lenses.

[0015] In a further embodiment, the detection of the driver's attention is performed by a driver assistance system in the vehicle, which, upon detecting a reduced level of driver attention, issues a warning signal to higher-level vehicle systems. Since the described method is independent of the sensor modalities used and the geometric installation components, no hardware modifications are required in the vehicle, and it can be implemented using standard components. The method can be transferred to the driver assistance system as a purely software-based implementation.

[0016] In a further refinement, all system components involved in attention detection are transferred into a common vehicle coordinate system before the start of the process. This simplifies the subsequent data transformation and saves computing time. The system components include the sensors involved, the driver observation camera, and the electrical components of the driver assistance system.

[0017] Another aspect of the invention relates to a device for determining a driver's level of attention during a journey in a vehicle, comprising a driver observation camera for capturing the driver's eye configuration during the journey, which is connected to a control unit that infers the driver's level of attention based on the eye configuration captured by the driver observation camera.The control unit, connected to at least one environmental sensor of the vehicle, comprises an image processing unit for determining the accommodation parameters of the driver's eye and an environment reconstruction unit for the area in front of the vehicle that is perceptible to the driver's eye. The image processing unit and the environment reconstruction unit are coupled to an attention detection unit for performing a spatial correlation analysis between accommodation data and object distances within the environment model according to at least one method described in this patent application. This enables adaptive attention detection based solely on the results from the driver observation camera and a visual cognitive feedback process. The detection of vital parameters is not required.

[0018] Further advantages, features, and details will become apparent from the following description, in which at least one embodiment is described in detail. The described features can, individually or in any meaningful combination, constitute the subject matter of the invention, optionally also independently of the claims, and can, in particular, also be the subject matter of one or more separate applications.

[0019] This shows: Fig. 1 a principle representation of the device according to the invention, Fig. 2 an embodiment of the method according to the invention, Fig. 3 an example of a visualized process flow.

[0020] In Fig. Figure 1 shows a schematic representation of the device according to the invention, which is to be considered hereafter as a driver assistance system 1 in a vehicle. The driver assistance system 1 comprises a control unit 5, which is connected to a lidar sensor 7 of the vehicle 3. A driver observation camera 11 is arranged on a windshield 9, which also connects to the control unit 5.

[0021] The control unit 5 comprises an image processing unit 13 and an environment reconstruction unit 15, both of which lead to an attention determination unit 17.

[0022] The inventive method is used to determine driver attention during a journey, which is exemplified as a flowchart in Fig. Figure 2 shows the process. In the first step, VS1, the driver assistance system 1 is calibrated, whereby the lidar sensor 7 and the driver observation camera 9 are adjusted to the coordinate system 25 of the vehicle 3. In step VS2, the environment model reconstruction unit 15 performs a 3D reconstruction of the environment in front of the vehicle 3 using the data output by the lidar sensor 7. Subsequently, in step VS3, the environment reconstruction created in step VS2 is restricted depending on the active functions of the driver assistance system 1. From the images provided by the driver observation camera 11, the image processing unit 13 determines the driver's gaze direction as a 3D vector (step VS4). Based on the driver's gaze direction, a precise 3D viewpoint is determined in step VS5 within the environment reconstruction adjusted in step VS3.Subsequently, based on the specific viewing angle, it is checked whether the driver 19 is generally viewing the area 23 of the vehicle environment defined in step VS3 (step VS6). Then, in the attention assessment unit 17, the driver's accommodation is determined in step VS7 using a video stream provided by the driver observation camera 11. During this step, the system repeatedly returns to step VS2 in VS7 to obtain sufficient results for testing at the functional levels of the individual active functions of the driver assistance system 1. In step VS8, it is checked whether the driver 19 repeatedly keeps a specific object or several objects 39 in focus within the area 27 defined in step VS3. This determines whether cognitive feedback has occurred in the driver 19. The driver's attention is calculated in step VS9 based on the previously determined attention status of the driver 17.The described steps VS1 to VS9 are repeated cyclically by going back to step VS2 (VS10).

[0023] In Fig. Figure 3 shows an exemplary embodiment of a visualized process flow, in which the corresponding process steps VS1 to VS9 are assigned to the individual images or representations. Fig. 3a The system components involved, such as lidar sensor 7, driver observation camera 11, and the electronic components of the driver assistance system 1 (not shown), are calibrated so that the extrinsic position of the system components in the vehicle coordinate system 25 is known. The 3D reconstruction of the environment in front of the vehicle 3 is shown in Fig. Figure 3b shows the lidar point cloud recorded by the lidar sensor 7, while figure 3b2 shows a mesh generation from the points of the lidar point cloud.

[0024] The Fig. Figure 3c illustrates the restriction of the environment reconstructed in step VS2 to an area 27 in which functions of the driver assistance system 1 are active. These functions include the adaptive cruise control function 29, the lane keeping function 31, an emergency brake warning function 33, and ULV video visualization 35, which are represented by the reference objects to be controlled.

[0025] The determination of the viewing angle according to VS4 in the vehicle coordinate system 25 is in Fig. 3D visualization. Two possible viewing directions 31 of the driver 19 are shown. In image 3d1, the driver 19 is looking straight ahead in the direction of travel, while in image 3d2 he is looking to the side. The detected viewing direction 31 of the driver 19 is output as a 3D vector and in Fig. 3e was used to determine an intersection point 37 in the sensor-based environmental reconstruction according to VS4. Fig. Figure 3f illustrates the test to determine whether the driver (19) recognizes the restricted area (27) of the environmental scenario. The pixels located within the restricted area (27) are labeled 39, while those outside the area are labeled 41.

[0026] The VS7 step for driver accommodation detection using the video stream recorded by the driver observation camera 11 is in Fig. Figure 3g illustrates this. According to the invention, a change in the size of the lens 44 of the respective eye 43 is detected, and the diameter of the pupil 47 is measured, which must be adjusted to see a viewed object 45 sharply. Figure 3g1 shows sharp detection of the object 45 without any change in lens size. In Figure 3g2, the observed object 45 is perceived as blurry without any change in lens size, as can occur, for example, during driver fatigue. Figure 3g3 illustrates the focusing of the object 45 by constricting the lens 44 (shown with a dashed line).

[0027] To determine whether cognitive feedback has occurred by driver 19, it is checked whether driver 19 keeps the restricted area 27 in focus. The position of the detected gaze point 39 is monitored in a temporal sequence of several frames of the video stream. From the image points 39 that lie sequentially within the selected area 27 of the environmental reconstruction, the following is obtained: Fig. 3h1 a diagram where a difference in the depth representation is shown above each image. Fig. Figure 3h2 shows a corresponding diagram in which the focus of lens 44 is plotted over each image.

[0028] The driver attention ultimately calculated by attention determination unit 17 is in Fig. 3k visualized. This visualizes the difference in area between the in Fig. 3h shown curves K A , K D, for example, based on an offset between the calculated delta area and the total area across the number of images considered. To obtain a reliable statement about the driver's level of attention F A To enable this, it is based on graph comparison in Fig. 3k, compared to a threshold SW: FA=0.12 <SW,FSW_A=0,22

[0029] If the SW threshold is exceeded in a predefined number of images, the active function of the driver assistance system 1 can be deactivated and / or a visual and / or audible warning can be issued to the driver 19. If the SW threshold is not reached, the driver 19 is assumed to be paying attention without any impairment. The SW threshold can be adaptively adjusted or fixed as part of a learning process.

[0030] The described solution enables adaptive attention detection, which can also be used in cases of microsleep or staring. It is independent of weather and lighting conditions as well as driving scenarios and requires no driver input.

Claims

[1] Method for determining the attention of a driver (19) during a journey with a vehicle (3), in which the level of attention of the driver (19) is inferred depending on an eye configuration of the driver (19) that develops during the journey, wherein the eye configuration is an accommodation of the lens (44) of the respective eye (43) of the driver (19) and a spatial model of the environment extending in front of the vehicle (3) is created, wherein an accommodation profile is correlated with a distance profile of an object (29, 39, 45) included in the environment model, characterized by , that a change in the size of the lens (44) of the respective eye (43) is detected and a diameter of the pupil (47) is measured, which must be adjusted in order to see the object (29, 39, 45) sharply. [2] Method according to claim 1 characterized by, that the environment model is created taking into account data determined by environmental sensors (7) located in and / or on the vehicle (3). [3] Method according to claim 2, characterized by , that a vehicle trajectory (3) with a tolerance range (27) is inserted into the spatial environment model. [4] Method according to claim 1, 2 or 3 characterized by , that the object (29, 39, 45) is defined depending on a viewpoint (31, 33) of the driver (19). [5] Method according to claim 4, characterized by, that from images of the eye (43) a viewing angle (31, 33) and the accommodation measures are determined, whereby for each image it is checked whether an intersection point (37) of the viewing angles (31, 33) of both eyes (43) is located in the tolerance range (27) surrounding the driving trajectory and for the intersection points (39) which lie within the tolerance range (27) a distance difference between object distance and accommodation measures is determined and from this the relative course of the accommodation measures of the eye (43) and the distance course of the object (29, 39, 45) is determined. [6] Method according to claim 5, characterized by , that it is checked cyclically whether the driver (19) considers the tolerance range (27) during the determination of the accommodation measures. [7] Method according to at least one of the preceding claims, characterized by, that the detection of the driver's attention (19) is carried out by a driver assistance system (1) of the vehicle (3), which, upon detection of a reduced level of attention of the driver (19), issues a warning signal to higher vehicle systems. [8] Method according to at least one of the preceding claims, characterized by , that before the start of attention detection all system components involved (1, 7, 9) are transferred into a common vehicle coordinate system (25). [9] Device for determining the attention of a driver (19) during a journey in a vehicle (3), comprising a driver observation camera (11) for capturing an eye configuration of the driver (19) during the journey, which is connected to a control unit (5) which infers the attention of the driver (19) based on the eye configuration captured by the driver observation camera (11), characterized by, that the control unit (5) connected to at least one environment sensor (9) of the vehicle (3) comprises an image processing unit (13) for determining the accommodation measures of the eye (43) of the driver (19) and an environment reconstruction unit (15) for an environment located in front of the vehicle (3) and detectable by the driver's eye (43), wherein the image processing unit (13) and the environment model reconstruction unit (15) are coupled with an attention determination unit (17) for performing a spatial correlation analysis between accommodation data and object distances within the environment model according to at least one of the preceding claims.

Images