Corneal reflection-based eye tracking in vehicles

Abstract

The invention relates to a method for determining a viewing direction of a vehicle occupant, comprising the steps of: detecting (S1) a vehicle environment with a sensor unit (1) and generating a spatial model of the environment; by means of an interior camera (5) of the vehicle: recording (S2) the occupant in a camera image, wherein the eye region comprises a cornea region of an eye of the occupant, on which a corneal reflex image is created when the environment of the vehicle is reflected thereon; extracting (S3) an extraction image from the camera image; optimizing (S4) a position and / or viewing direction of a virtual camera, which is virtually directed at the spatial model of the environment, until a predetermined termination condition is met with regard to a match between the extraction image and a reference image; determining (S5) the viewing direction of the occupant from the position and / or viewing direction of the virtual camera when the termination condition is met;

Description

[0001] The invention relates to a method for determining a viewing direction of an occupant of a vehicle, as well as a system for determining a viewing direction of an occupant of a vehicle.

[0002] Today's eye-tracking systems are often based on light sources, often infrared diodes. Based on these projections, the position of the eye is determined, and the person's gaze direction is derived from this. One of the most common methods is so-called "Purkinje Bases Eye Tracking," which evaluates characteristic reflections and their relative positions to one another and, based on this, estimates the position of the eye in relation to the projection. The disadvantages of this approach are, on the one hand, the use of additional infrared diodes, and, on the other, that all so-called Purkinje reflections must be known. This can sometimes be difficult to achieve, especially when the driver's head is rotating.

[0003] In this context, DE 10 2016 013 806 A1 relates to a system for detecting a line of sight of a driver in a vehicle, in which a lighting unit and an image recording unit are arranged directly next to one another, wherein one eye of the driver is located in the detection range of the image recording unit and a reflection point generated by the lighting unit on the eye as a result of corneal reflection is recorded by the image recording unit and evaluated by an evaluation unit for detecting the line of sight, wherein the lighting unit and the image recording unit are arranged in a parting plane between a cockpit and a windscreen of the vehicle, wherein a line of sight in the direction of the cockpit or windscreen can be detected by means of a respective deviation of the reflection point generated on the eye from a reference point.

[0004] DE 10 2023 000 955 B3 discloses a vehicle occupant detection device comprising at least one infrared light source, a camera, and a computing unit for controlling the infrared light source and for evaluating camera images generated by the camera. The infrared light source and the camera are directed toward an observation region, wherein the infrared light source is configured to illuminate at least one part of the face of a person located in the observation region. The camera is configured to capture at least the illuminated part of the face, and the computing unit is configured to recognize a viewing direction of the person in the camera images. The vehicle occupant detection device is characterized in that the infrared light source is integrated into a display device and is formed by a plurality of IR lighting elements arranged in a matrix.The computing unit is further configured to activate a selection of the IR lighting elements for emitting infrared light, wherein the computing unit determines the selection of the IR lighting elements to be activated depending on the evaluation result of a camera image.

[0005] DE 11 2018 006 886 T5 discloses an occupant state detection device comprising a correction parameter setting unit for setting a correction parameter for an image of an occupant of a vehicle captured by a camera for each of a plurality of detection elements in an occupant state detection process using at least one feature amount in a partial face region of the occupant, and an image correction unit for correcting the captured image for each of the detection elements in the occupant state detection unit using the correction parameter set by the correction parameter setting unit. The detection elements in the occupant state detection process include at least a gaze direction, a mouth opening degree, a face orientation, a skin color, and / or an eye opening degree of the occupant.

[0006] From WO 02 / 031 580 A1, an information system is known comprising a signal detection device which detects signals reflected back from at least one eye having a retina; a visual field detection device which detects visible light from a visual field associated with the retina without detecting a retinal reflex image of the retina; an information device and an output device which, in cooperation with the information device, provides information as a function of the detected light and in correlation with the detected signals.

[0007] The object of the invention is to circumvent these disadvantages and to provide an improved method for determining a viewing direction of an occupant of a vehicle.

[0008] The invention is based on the features of the independent claims. Advantageous developments and refinements are the subject of the dependent claims.

[0009] A first aspect of the invention relates to a method for determining a viewing direction of an occupant of a vehicle, comprising the steps: - Detecting the environment of the vehicle by means of an on-board sensor unit and, based thereon, generating a spatial model of the environment by means of a computing unit; - By means of an interior camera of the vehicle: recording the occupant in a camera image, so that an eye area of ​​the occupant is captured in the camera image, wherein the eye area comprises a corneal area of ​​an eye of the occupant on which a corneal reflex image is created when the surroundings of the vehicle are reflected thereon; - Extracting, from the camera image, a zone of the corneal reflex image surrounding a pupil into an extraction image, by the computing unit; - iteratively optimising a position and / or viewing direction of a virtual camera which is virtually directed towards the spatial model of the environment until a predefined termination condition is met with regard to a match between the extraction image and a reference image which is generated on the basis of the position and viewing direction of the virtual camera into the model of the environment, by the computing unit; - Determining the viewing direction of the occupant from the position and / or viewing direction of the virtual camera when the termination condition is met by the computing unit;

[0010] In addition to the viewing direction of the occupant from the position and / or viewing direction of the virtual camera when the termination condition is met by the computing unit, the position of one of the occupant's eyes can also be determined.

[0011] This method makes it possible to perform eye tracking based solely on an existing spatial model of the environment and an interior camera. A virtual camera is used, from whose position and virtual viewing direction into the spatial model, a planar reference image is determined by projection. The position of the virtual camera is computationally iteratively shifted, and its view into the spatial model of the environment is determined until it corresponds sufficiently accurately to the position of the occupant's eye.

[0012] To do this, a sensor unit first records the vehicle's surroundings and then mathematically reconstructs them in a spatial model. The sensor unit can utilize, for example, radar, camera, lidar, or a fusion of various data sources.

[0013] In addition, a corneal reflex image is captured using an interior camera, which includes a reflection of the environment outside the vehicle interior. Regions of interest are extracted and preferably rectified to compensate for the curvature of the cornea, thus enabling comparison with the reference image. This extracted zone extends around the pupil center, preferably concentric with the pupil.

[0014] The position and / or viewing direction of the virtual camera is now changed iteratively in order to generate a reference image that matches the extracted image as closely as possible. The reference image is generated by using a position and / or viewing direction of a respective iteration step to generate a virtual camera image of the virtual camera in the spatial model of the environment that is consistent with the position and viewing direction of the virtual camera. The virtual camera is not a physically existing camera, but rather a starting point of a viewing direction that virtually runs into the virtual environment, implemented by the model of the environment. A scene is simulated in which a real camera is pointed at a real environment, and the camera image is analyzed. However, instead of the real scene and the real camera, the virtual camera is used, which is pointed at the virtual environment of the spatial model.Analogously, however, the image of the virtual camera is compared with the extraction image, for which a coordinate system transformation is preferably carried out in order to have the extraction image and the reference image in the same coordinate system.

[0015] A corresponding similarity measure is calculated, which indicates the correspondence between the reference image and the extracted image. The more similar these are, the closer the position and / or viewing direction of the virtual camera is to that of the observed eye.

[0016] If an appropriate optimization algorithm is used, for example, a gradient-based method, the position and / or gaze direction of the virtual companions of the eye can generally be approximated with increasing iterations until a predefined termination condition is met, indicating sufficient agreement. In each iteration, a match is checked, for example, using a "structural similarity index measure" as a similarity measure between the reference image and the extracted image. This position and / or gaze direction of the virtual camera upon fulfillment of the termination condition is used as the position and / or gaze direction of the eye of the vehicle occupant, for example, in a driver assistance system or similar.A first position as the starting point of the iterative process of the virtual camera is preferably chosen so that it statistically corresponds to that of 95% of occupants, whereby the latter can be known from CAD data.

[0017] This advantageously enables eye tracking without the need for a light source or any other physical technical components other than the interior camera in the vehicle interior. Furthermore, error-prone feature detection is eliminated. The process is independent of lighting and weather conditions.

[0018] According to an advantageous embodiment, the camera image is rectified with the corneal reflex image or the extraction image by the computing unit in order to remove the influence of a corneal curvature from the extraction image.

[0019] The cornea of ​​the eye is naturally convexly curved when viewed from outside the eye, which is why a corneal reflex image is correspondingly spherically distorted. Rectification compensates for the curvature of the cornea, essentially allowing the extracted image to overlap with the reference image.

[0020] According to a further advantageous embodiment, the zone of the corneal reflex image surrounding the pupil is defined as a rectangle or circle with a predetermined number of pixels.

[0021] According to a further advantageous embodiment, an initial position and / or viewing direction of the virtual camera used for the iterative optimization is used, which is determined from a head position or eye position of a 95th percentile of men and / or women.

[0022] If a nonlinear optimization with iterative search steps is subsequently carried out, the use of a promising starting point of the iteration helps to achieve rapid convergence until a termination condition is reached.

[0023] According to a further advantageous embodiment, in order to determine the termination condition, a structural similarity index measure is determined in each iteration and this is compared with a predetermined boundary condition.

[0024] According to a further advantageous embodiment, the iterative optimization comprises the application of a gradient method to maximize the agreement.

[0025] According to a further advantageous embodiment, the determined viewing direction of the occupant is further processed by an assistance system of the vehicle.

[0026] The assistance system can in particular be one of the following: fatigue detection assistant, attention assistant, lane keeping assistant.

[0027] According to a further advantageous embodiment, the determined gaze direction is used by the assistance system as a comparison value to increase reliability or as a replacement value for the gaze direction determined from fixations, saccades or a head orientation.

[0028] According to a further advantageous embodiment, the reference image is generated by projecting the spatial model of the environment into a planar image.

[0029] A further aspect of the invention relates to a system for determining a viewing direction of an occupant of a vehicle, comprising a sensor unit which is configured and designed to detect an environment of the vehicle; a computing unit which is designed to generate a spatial model of the environment based on the data of the sensor unit; an interior camera which is designed to record the occupant in a camera image, so that an eye region of the occupant is detected in the camera image, wherein the eye region comprises a cornea region of an eye of the occupant, on which a corneal reflex image is created when the environment of the vehicle is reflected thereon;and a computing unit designed to extract from the camera image a zone of the corneal reflex image surrounding a pupil into an extraction image, as well as to iteratively optimize a position and / or viewing direction of a virtual camera that is virtually directed at the spatial model of the environment until a predetermined termination condition is met with regard to a match between the extraction image and a reference image that is generated on the basis of the position and viewing direction of the virtual camera into the model of the environment, and to determine the viewing direction of the occupant from the position and / or viewing direction of the virtual camera when the termination condition is met.

[0030] Advantages and preferred developments of the proposed system result from an analogous and analogous transfer of the statements made above in connection with the proposed method.

[0031] Further advantages, features and details emerge from the following description, in which - if necessary with reference to the drawing - at least one embodiment is described in detail.

[0032] They show: Fig. 1: An interior of a vehicle with a system for determining a viewing direction of a vehicle occupant according to an embodiment of the invention. Fig. 2: A method for determining a viewing direction of an occupant of a vehicle according to an embodiment of the invention.

[0033] The representations in the figures are schematic and not to scale.

[0034] Fig. 1 shows the interior of a vehicle designed as a passenger car with an occupant in the role of driver. This vehicle has a sensor unit 1 that is directed towards the surroundings of the vehicle. The sensor unit 1 can have a camera unit, lidar unit, or similar, and transmits its sensor data to a computing unit 3. The computing unit 3 continuously generates an updated model of the surroundings from the data of the sensor unit 1. Because the computing unit 3 can treat the model of the surroundings like a CAD model that is observable for a virtual camera, a virtual image of the model of the surroundings can be generated depending on a positioning from a viewing direction of the virtual camera. This positioning and / or viewing direction is varied in an optimization process executed by the computing unit 3 in order to generate various reference images as optimization variables.The various reference images represent the values ​​of the optimization variables across various search steps, for example search steps of a gradient-based optimization method, for which quadratic nonlinear optimization or other optimization methods such as genetic algorithms or evolutionary algorithms can alternatively be used. During these search steps, the respective reference image, which maps a respective virtual image of the virtual camera from its respective current position and viewing direction onto the virtual spatial model of the environment in a planar manner, i.e. in 2D, is compared with a respective extraction image for a match. The respective extraction image is generated by the computing unit 3 from the image (A) of an interior camera 5. The interior camera 5 is directed at the driver of the vehicle, so that a geometric area around the eyes of the vehicle is contained in the image of the interior camera 5.If a corneal reflex image is also created on the cornea of ​​one of the driver's eyes, in which the surroundings are reflected on the cornea of ​​the eye, this corneal reflex image and thus corresponding information about the surroundings is contained in the image from the interior camera 5. After appropriate rectification of the image (A) from the interior camera 5, a section with the pupil center at the center of the extracted image, if possible, and a coordinate system transformation and, if necessary, a reflection, the extracted image from (A) can be compared with the reference image (B) in each of the search steps. In this process, the position and / or viewing direction of the virtual camera is changed as expediently as possible in discrete steps so that the extracted image and the dependent reference image match as closely as possible.If this is the case with sufficient approximation, a termination condition is met and the position and viewing direction of the virtual camera can be used to determine the position and / or viewing direction of the driver's eye.

[0035] Fig. 2 shows a corresponding procedure for the system of Fig. 1. The following steps are carried out, not necessarily in this exact order: - detecting S1 an environment of the vehicle by means of a vehicle-specific sensor unit 1 and based thereon: generating a spatial model of the environment by means of a computing unit 3; - By means of an interior camera 5 of the vehicle: recording S2 of the occupant in a camera image, so that an eye region of the occupant is captured in the camera image, wherein the eye region comprises a corneal region of an eye of the occupant, on which a corneal reflex image is formed when the surroundings of the vehicle are reflected thereon; - Extracting S3, from the camera image, a zone of the corneal reflex image surrounding a pupil into an extraction image, by the computing unit 3; - iteratively optimizing S4 a position and / or viewing direction of a virtual camera that is virtually directed at the spatial model of the environment until a predetermined termination condition is met with regard to a match between the extraction image and a reference image that is generated on the basis of the position and viewing direction of the virtual camera into the model of the environment, by the computing unit 3; - Determining S5 the viewing direction of the occupant from the position and / or viewing direction of the virtual camera when the termination condition is present by the computing unit 3; wherein the computing unit 3 corrects the camera image with the corneal reflex image or the extraction image in order to remove the influence of a corneal curvature from the extraction image.

[0036] Although the invention has been illustrated and explained in detail by preferred embodiments, the invention is not limited by the disclosed examples, and other variations may be derived therefrom by those skilled in the art without departing from the scope of the invention. It is therefore clear that a multitude of variations exist. It is also clear that exemplary embodiments are truly only examples and should not be construed as limiting the scope, possible applications, or configuration of the invention in any way.Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without departing from the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description.

Claims

[1] A method for determining a viewing direction of an occupant of a vehicle, comprising the steps: - detecting (S1) an environment of the vehicle by means of a vehicle-specific sensor unit (1) and based thereon: generating a spatial model of the environment by means of a computing unit (3); - By means of an interior camera (5) of the vehicle: recording (S2) the occupant in a camera image, so that an eye region of the occupant is captured in the camera image, wherein the eye region comprises a cornea region of an eye of the occupant, on which a corneal reflex image is created when the surroundings of the vehicle are reflected thereon; - extracting (S3) from the camera image a zone of the corneal reflex image surrounding a pupil into an extraction image by the computing unit (3); - iteratively optimizing (S4) a position and / or viewing direction of a virtual camera that is virtually directed at the spatial model of the environment until a predetermined termination condition is met with regard to a match between the extraction image and a reference image that is generated on the basis of the position and viewing direction of the virtual camera into the model of the environment, by the computing unit (3); - determining (S5) the viewing direction of the occupant from the position and / or viewing direction of the virtual camera when the termination condition is present by the computing unit (3); [2] Method according to claim 1, wherein the camera image with the corneal reflex image or the extraction image is rectified by the computing unit (3) in order to remove the influence of a corneal curvature from the extraction image. [3] Method according to one of the preceding claims, wherein the zone of the corneal reflex image surrounding the pupil is defined as a rectangle or circle with a predetermined number of pixels. [4] Method according to one of the preceding claims, wherein an initial position and / or viewing direction of the virtual camera used for the iterative optimization is used, which is determined from a head position or eye position of a 95th percentile of men and / or women. [5] Method according to one of the preceding claims, wherein to determine the termination condition in each iteration a Structural Similarity Index measure is determined and this is compared with a predetermined boundary condition, [6] A method according to any one of the preceding claims, wherein the iterative optimization comprises applying a gradient method to maximize the agreement. [7] Method according to one of the preceding claims, wherein the determined viewing direction of the occupant is further processed by an assistance system of the vehicle. [8] Method according to claim 7, wherein the determined gaze direction is used by the assistance system as a comparison value to increase reliability or as a substitute value for the gaze direction determined from fixations, saccades or a head orientation. [9] Method according to one of the preceding claims, wherein the reference image is generated by projecting the spatial model of the environment into a planar image. [10] A system for determining a viewing direction of an occupant of a vehicle, comprising a sensor unit (1) which is configured and designed to detect an environment of the vehicle; a computing unit (3) which is designed to generate a spatial model of the environment based on the data of the sensor unit (1); an interior camera (5) which is designed to record the occupant in a camera image, so that an eye region of the occupant is detected in the camera image, wherein the eye region comprises a cornea region of an eye of the occupant, on which a corneal reflex image is created when the environment of the vehicle is reflected thereon;and a computing unit (3) designed to extract from the camera image a zone of the corneal reflex image surrounding a pupil into an extraction image, as well as to iteratively optimize a position and / or viewing direction of a virtual camera that is virtually directed at the spatial model of the environment until a predetermined termination condition is met with regard to a match between the extraction image and a reference image that is generated on the basis of the position and viewing direction of the virtual camera into the model of the environment, and to determine the viewing direction of the occupant from the position and / or viewing direction of the virtual camera when the termination condition is met.

Images