Determining the gaze direction

Abstract

1. A method for estimating a user's gaze direction, comprising: acquiring an image of the user's face (S3); determining an approximate gaze direction (56) based on a current head pose and a relationship between the head pose and the gaze direction (S5); determining an estimated gaze direction (57) based on identified eye features (S7); determining a fine gaze direction (58) based on a glint position and the eye features (S8); and combining the approximate gaze direction with at least one of the estimated gaze direction and the fine gaze direction (S10) to provide a revised gaze direction (59).

Description

[Technical field]

[0001] The present invention relates generally to the field of eye tracking, i.e. detecting and monitoring a user's eyes using an image of the user. In particular, the present invention relates to determining and tracking a user's gaze direction, i.e. where the user is currently looking. [Background technology]

[0002] The direction of a user's gaze can be determined from captured images of the user's face by first determining the head pose (in a room coordinate system) and then the eye pose (in a head coordinate system). The eye pose can be determined based on the position of the iris relative to the head. This type of gaze direction detection is sometimes called "estimated gaze detection".

[0003] If more precise gaze direction is required, the face may be illuminated (e.g., with ultraviolet light) while acquiring the images, so that the acquired images contain a reflection (glow) in the cornea of ​​the eye. Using the known geometric relationship between the light source and the image sensor, this glow can be used to make the gaze direction determination more precise. This type of gaze direction detection is sometimes referred to as "precise gaze detection."

[0004] Both types of gaze detection suffer from noise and drift and usually require some kind of online calibration. Various methods have been proposed, including calibration based on statistical analysis of where the user is expected to look. In particular, in vehicular implementations, this approach has proven suitable, since users typically look at a small number of easily identifiable objects such as rear-view mirrors and dashboard controls.

[0005] There is still a need for better calibration techniques. In addition, both of the gaze direction detection methods discussed above (estimated gaze and refined gaze) require the user to look towards the image sensor so that the pupils are captured in the image. It is desirable to provide an estimate of gaze direction even when the eyes are not present in the image.

[0006] Prior art briefly mentions this problem and even more briefly provides solutions. WO 2018 / 002000, for example, discusses the basis of very approximate gaze detection based only on head pose, i.e., only on where the user is pointing. In WO 2018 / 002000, the gaze direction is approximated as being perpendicular to the plane of the user's face. WO 2019 / 147607 discloses a neural network-based approach to gaze detection, which can detect gaze even from images that do not contain eyes. Summary of the Invention [Problem to be solved by the invention]

[0007] It is an object of the present invention to alleviate the above problems and provide an improved gaze estimation. [Means for solving the problem]

[0008] According to a first aspect of the present invention, this and other objects are achieved by a method of estimating a user's gaze direction comprising: acquiring an image of the user's face; determining a current head pose based on the image; accessing a database storing relationships between population-based head poses and gaze directions; determining an approximate gaze direction based on the current head pose and said relationship; identifying a set of eye features in the image and determining an estimated gaze direction based on the eye features; identifying the set of eye features and a glint in the eye in the image and determining a refined gaze direction based on the location of the glint and the eye features; and weighting at least one of 1) the approximate gaze direction, 2) the estimated gaze direction and 3) the refined gaze direction with an adaptive filter, thereby providing a revised gaze direction.

[0009] The present invention is based on the realization that eye gaze approximations based solely on head pose require additional information to be sufficiently accurate: it is simply not sufficient to assume that the gaze direction is perpendicular to the plane of the face.

[0010] Furthermore, the invention is based on the realization that for stationary users (e.g., car drivers, or computer users, but in general, any user who stays in the same place and moves only his / her head), the relationship between a particular head pose and the associated gaze direction is in fact remarkably constant within a population. Thus, by generating a database of such relationships and using it (e.g., a look-up table), it is possible to determine an approximate gaze direction for a given head pose.

[0011] The gaze direction estimate obtained using identified eye features is more accurate than an approximate gaze direction based on head pose alone. However, as the identification of eye features becomes more difficult and uncertain (i.e., further away from the optical axis), the gaze direction estimate also becomes more uncertain (larger standard deviation).

[0012] The precise gaze direction obtained using one or more glares in the eye is more accurate than the estimated gaze direction based on eye features alone. However, as the identification of the glares becomes more difficult and uncertain (i.e., further away from the optical axis), the precise gaze direction also becomes more uncertain (larger standard deviation). At a certain distance from the optical axis, it is no longer possible to identify the glares in the eye and the precise gaze direction is lost.

[0013] According to the present invention, the estimated gaze direction and / or the fine gaze direction are combined with the approximate (head pose based) gaze direction to provide a revised gaze direction. For example, the approximate gaze direction and at least one of the estimated gaze direction and the fine gaze direction are weighted using a suitable adaptive filter, such as a Kalman filter, to provide a revised gaze direction.

[0014] A Kalman filter (or similar filter) can ensure that the relative weights of different gaze directions are adaptively adjusted based on the uncertainty (e.g., standard deviation) of these gaze directions. For example, if the precise gaze direction is stable (e.g., small standard deviation), then the approximation based on head pose need not be given much weight. Conversely, if the estimated and precise gaze directions are uncertain (large standard deviation), then the approximation based on head pose can get more weight.

[0015] Note that precise gaze directions close to the optical axis typically have a small standard deviation, but this standard deviation increases with distance from the optical axis (head or yaw angle). Estimated gaze directions have a larger standard deviation than precise gaze directions close to the optical axis, but the standard deviation does not increase as rapidly as the precise gaze directions. In contrast, approximate gaze directions (based on head pose) have a large but more constant standard deviation that is independent of head angle.

[0016] Based on this, an approximation of the respective standard deviation of each determined gaze direction can be expressed as a function of head angle: In other words, as the user looks away from the optical axis, the precise gaze (and estimated gaze) will have less weight and the approximate gaze will have more weight.

[0017] Furthermore, noise and other imperfections can sometimes prevent accurate determination of the precise and / or estimated gaze direction, and in these circumstances, approximate gaze directions may be given greater weight to ensure that the corrected gaze direction is not lost and is at least approximately accurate.

[0018] Since the (head pose -> gaze direction) relationship is fairly constant within a population, a satisfactory approximation is generated even without user-specific data. However, the quality of the approximation improves significantly if user-specific information is used. Therefore, in one embodiment, the population-based relationship is tailored to a specific user. For example, online calibration based on statistical information about the view pattern is used. Such statistical calibration is particularly useful in a vehicular environment where the driver's gaze direction is restricted to a fairly small number of predefined directions most of the time.

[0019] The above relationship can be expressed in all six degrees of freedom, i.e., it takes the complete head pose, including position and orientation, as input and returns an approximate gaze direction. However, it is advantageous to simplify the above relationship to include fewer degrees of freedom. For example, the above relationship can be expressed using only two degrees of freedom, e.g., a spherical coordinate system with two angular coordinates (yaw / head angle and pitch angle). In this case, head position information is used to adjust the stored relationship.

[0020] The present invention will now be described in more detail with reference to the accompanying drawings, in which preferred embodiments of the invention are shown. [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 is a schematic diagram of an eye-tracking system mounted on the dashboard of a vehicle. [Figure 2a] FIG. 2 shows details of the eye-tracking system of FIG. 1. [Figure 2b] FIG. 2b shows an example of the relative positions of the camera and the light source in FIG. 2a. [Diagram 3] 2 is a flow chart of a method according to an embodiment of the present invention. [Figure 4] FIG. 1 is a diagram of a spherical coordinate system centered on the user's face. [Diagram 5] FIG. 1 is a block diagram of an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Hereinafter, embodiments of the present invention are described in the context of a vehicle eye-tracking system, although it should be noted that other implementations of the present invention are possible as well.

[0023] Figure 1 shows a driver 1 sitting in a driver's seat 2 of a vehicle 3. An eye gaze detection system 4 is mounted in front of the driver 1, here on the dashboard 5. Alternatively, the eye gaze detection system 4 may be mounted on a steering wheel column 6 or fixed to the ceiling 7. Regardless of how it is mounted, the eye gaze tracking system 4 should have a clear view of the head 8 and eyes 9 of the driver 1. The eye gaze detection system 4 is typically positioned at a distance d1 from the driver 1 in the range of 40cm to 120cm, more typically at "arm's length", i.e. approximately 60-70cm.

[0024] With reference to Fig. 2a, the eye tracking system 4 in the illustrated example includes an image sensor 10, e.g. a CMOS image sensor, provided with suitable optics 11. The illustrated system 4 further includes two light sources 12, typically configured to emit light outside the visible range, such as infrared (IR) or near infrared (NIR). The light sources 12 may be solid-state light sources, such as LEDs. In the illustrated example, the light sources 12 are LEDs configured to emit light with a light spectrum centered in a 50 nm band centered around 850 or 940 nm (NIR).

[0025] In Fig. 2a, the light sources 12 are arranged along a straight line on either side of the image sensor 10. In order to employ "dark-pupil" processing, a sufficient distance is required between each light source and the image sensor 10. If it is not desirable to increase the horizontal dimension of the device, i.e. the distance d2 between the light sources, it is advantageous to separate the light sources 12 and the image sensor 10 by a vertical distance d3, as shown in Fig. 2b. In other words, the light sources and the image sensor are no longer in a straight line.

[0026] Returning to FIG. 2a, the image sensor 10 is further provided with an optical bandpass filter 13, for example an interference filter. The filter 13 may be configured to have a passband that substantially corresponds to the emission spectrum of the light sources 12a, 12b. Thus, in the above example, the filter 13 should have a passband of approximately 825-875 nm, or 915-965 nm. The combination of narrowband illumination and narrow passband filtering makes the image acquisition system less susceptible to ambient light, such as sunlight.

[0027] An image acquisition controller 14 (hereafter referred to as "controller" for brevity) is connected to the image sensor 10 and the light sources 12a, 12b and is programmed to control the image sensor 10 to acquire images under the illumination of the light sources 12a, 12b. Typically, to save energy and power consumption, the light sources 12a, 12b are controlled to emit light only when the electronic shutter of the image sensor is open. The light sources 12a, 12b may be driven alternately (successive image frames acquired illuminated by different light sources) or both may be driven for each image frame. By using alternating illumination, disturbing reflexes (e.g. from glass) are suppressed, as disclosed in EP 1,349,487.

[0028] In a practical example, the light sources 12a, 12b are triggered by a pulse having a duty cycle selected to provide sufficient illumination to acquire one image frame. As described above, for each frame, either or both light sources 12a, 12b may be activated. Preferably, the light sources 12a, 12b are controlled by a pulse signal having a duty cycle that ensures that the pulse width corresponds to the period required to acquire one image by the image sensor 10. The image sensor 10 is then synchronized to this pulse signal.

[0029] The eye tracking system 14 further comprises a processing circuit 15 configured to receive images captured by the image sensor 10 and to process these images. The processing circuit 15 may be integrated into the controller 15 or may be separate from the controller 15. The system also comprises a memory, such as a RAM memory, for example to store software executed by the processing circuit 15.

[0030] In use, the eye tracking system 4 is arranged to acquire images of a target area of ​​the face 9 of the user 1. The processing circuitry 15 is configured to track the user's head position and eye movements using the images from the sensor 10 to acquire various information. In particular, here the processing circuitry 15 is programmed to acquire the user's gaze direction.

[0031] Based on the acquired images, the gaze direction can be determined by determining the head pose, i.e. the position and orientation in a head coordinate system 20 in space, and the relative eye pose, i.e. the position and orientation in an eye coordinate system 21 with respect to the coordinate system 20. In the absence of illumination, the eye pose can be determined based on the position of the iris with respect to the head. This approach to determining the gaze direction is usually called "estimated gaze".

[0032] The direction of gaze can also be determined by generating a reflection (glitter) in the cornea of ​​the eye using light sources 12a, 12b. If the position of the light source relative to the image sensor is known, the position of the glitter allows the direction of gaze to be determined. This approach to determining the direction of gaze is usually called "precision gaze". Some eye-tracking systems have two or more light sources, which allows images to be acquired with different illuminations and therefore glitters at different positions.

[0033] Fig. 3 shows a method according to an embodiment of the present invention. First, in step S1, before the actual start of the eye tracking, a relationship between head pose and eye gaze direction for a population is determined and stored in a database in memory 16. Such a relationship can be obtained by collecting data from a sample of people. The relationship can be stored in various ways, but in a practical example, the relationship assumes a fixed head position in space and associates two degrees of freedom head pose orientations (essentially, where the nose is pointing) with the corresponding gaze direction (i.e., where the eyes are looking). Or, in a different expression, for every head pose orientation, a correction vector is associated to arrive at the gaze direction.

[0034] The actual gaze determination begins at step S2, with the head 8 and eyes 9 of the user 1 illuminated. In step S3, an image of the head 8 including the eyes 9 is acquired. In step S4, the user's head pose is determined based on facial features identified by processes known per se.

[0035] Using the determined head pose, and using the relationship stored in memory 16, an approximate gaze direction is determined in step S5. Note that in some situations this very rough approximation of the gaze direction may be the only approximation available.

[0036] Figure 4 shows a spherical coordinate system with origin 41 representing the head position of user 1, and two axes 42, 43 representing pitch and yaw / head angles, respectively. Two different head pose directions are shown in Figure 4 as arrows (vectors) 44 and 45. The relationships in memory 16 provide correction vectors 46, 47 for each head position direction, which are also shown in Figure 4. The endpoints of vectors 46, 47 therefore represent approximate line of sight, based only on the head pose and the pre-stored relationships.

[0037] Then, in step S6, relevant eye features, such as the position of the iris, are identified to enable the determination of the estimated gaze in step S7. In Figure 4, vectors 48 and 49 represent the estimated gaze, which correspond to the approximate gaze represented by the combined vectors 44, 46 and 45, 47, respectively.

[0038] In the following step S8, the glint, i.e. the reflection of the illumination in the cornea of ​​the eye, is identified (if possible) and in step S9, the identified glint is used to determine the refined line of sight. In Fig. 4, vectors 50 and 51 represent the refined line of sight corresponding to the approximate line of sight represented by the combined vectors 44, 46 and 45, 47, respectively.

[0039] Finally, using the various gaze directions determined in steps S5, S7 and S9, i.e. approximate gaze, estimated gaze and fine gaze, a revised gaze direction is determined in step S10 using a Kalman filter (or other similar adaptive weighting filter).

[0040] A schematic of the operation of a Kalman filter is shown in Figure 5. The filter 55 receives as input one, two or three different gaze directions: an approximate gaze 56, an estimated gaze 57 (if available) and a fine gaze (also if available).

[0041] The filter 50 also receives, for each gaze direction, a variable value representing the standard deviation of the individual gaze directions: one value for the approximate gaze, one value for the estimated gaze, and one value for the fine gaze. In one embodiment, the system is configured to determine an estimate of the standard deviation for each of the determined gaze directions. However, in a more practical example, the system makes the assumption that the standard deviation of the individual gaze directions is correlated to the yaw / head angle. This information may be provided as an input to the Kalman filter 55, as shown in FIG. 5.

[0042] Returning to Figure 4, the estimated standard deviation for each gaze direction is represented by dashed circles 46', 47', 48', 49' and 50'. Circle 48' represents the standard deviation for gaze direction 48, and so on.

[0043] To the left near axis 42 is an approximate line of sight, represented by vectors 44, 46 and standard deviation 46'. The corresponding estimated line of sight 48 has a very small standard deviation 48', and the corresponding precise line of sight 50 has an even smaller standard deviation 50'. This corresponds to a situation where the user's line of sight is close to the central axis 42 (yaw / head angle close to 0), the glint is clearly discernible, and the precise line of sight is relatively reliable (small standard deviation). As a result, the precise line of sight is given a high weighting, and the output from Kalman filter 55 will be very close to the precise line of sight.

[0044] However, if noise or some interference (e.g., a sudden burst of sunlight) causes the glow to become undetectable in one or more frames, the Kalman filter 55 can rely on the remaining line of sight directions 56 and 57, thereby continuing to provide a relatively accurate output 59.

[0045] Noise or interference may also cause sudden shifts in the precise line of sight 58. Such sudden shifts may be interpreted by the system as an unreliable determination (large standard deviation), and therefore it again derives a revised line of sight, based mainly on the approximate and estimated lines of sight 56, 57.

[0046] Looking to the right side of FIG. 4, there is an approximate line of sight represented by vectors 45, 47 and standard deviation 47'. The corresponding estimated line of sight 49 has a similar standard deviation 49', and the corresponding fine line of sight 51 also has a similar standard deviation 51'. This corresponds to a situation where the user's line of sight is far from the central axis 42 (yaw / head angle far from 0) and both the estimated line of sight determination and the fine line of sight determination are less accurate and less reliable (large standard deviation). As a result, the approximate line of sight 56, the estimated line of sight 57 and the fine line of sight 58 are given similar weights in the Kalman filter 55, and the revised line of sight 59 is based on all three inputs 56, 57, 58.

[0047] Those skilled in the art will understand that the present invention is in no way limited to the preferred embodiment described above. On the contrary, many modifications and variations are possible within the scope of the appended claims. In particular, the details of the gaze detection system disclosed herein are provided as examples only and do not limit the application of the present invention, which is intended to combine gaze directions determined by various methods. Furthermore, the weighting of various gaze directions does not necessarily require an adaptive filter, and various types of adaptive filters can be used instead of a Kalman filter.

Claims

1. A method for estimating a user's gaze direction, comprising: acquiring an image of the user's face (S3); determining (S4) a current head pose based on the image, accessing a database storing relationships between ensemble-based head poses and gaze directions, and determining (S5) an approximate gaze direction (56) based on the current head pose and the relationships; identifying (S6) a set of eye features in the image and determining (S7) an estimated gaze direction (57) based on the eye features; identifying (S8) a set of eye features and eye glints in the image and determining a refined gaze direction (58) based on the location of the glints and the eye features; combining (S10) the approximate gaze direction with at least one of the estimated gaze direction and the refined gaze direction to provide a revised gaze direction (59); A method for estimating a user's gaze direction, comprising:

2. The method of claim 1 , wherein the approximate gaze direction and at least one of the estimated gaze direction and the fine gaze direction are weighted using an adaptive filter.

3. 3. The method of claim 2, wherein the relative weights of the approximate, estimated and fine gaze directions are adaptively adjusted by a standard deviation of each of the approximate, estimated and fine gaze directions.

4. 4. The method of claim 3, wherein an approximation of the standard deviation of each of the determined gaze directions is expressed as a function of yaw angle (43).

5. The method of claim 4, wherein the adaptive filter is a Kalman filter (55).

6. The method of any one of claims 1 to 5, further comprising illuminating the face of the user while capturing the image.

7. The method of any one of claims 1 to 6, further comprising tailoring the population-based relationships to a particular user using online calibration.

8. The method of claim 7 , wherein the online calibration is based on statistical information about view patterns.

9. The method according to any one of claims 1 to 8, wherein the population-based relationships are expressed in two dimensions of a spherical coordinate system, for example yaw and pitch coordinates (42, 43).

10. A system for estimating a user's gaze direction, comprising: an image sensor (10) for acquiring an image of the face of the user; A processing circuit (15), determining a current head pose based on the acquired images, accessing a database storing relationships between ensemble-based head poses and gaze directions, and determining an approximate gaze direction (56) based on said current head pose and said relationships; identifying a set of eye features in the image and determining an estimated gaze direction (57) based on the eye features; identifying a set of eye features and eye glints in the image and determining a refined gaze direction (58) based on the location of the glints and the eye features; Combining the approximate gaze direction with at least one of the estimated gaze direction and the fine gaze direction to provide a revised gaze direction. A processing circuit configured to A system for estimating a user's gaze direction, comprising:

11. The system of claim 10, further comprising one or more light sources (12a, 12b) for illuminating the face of the user during acquisition of the image.

Images