Eye movement tracking system and method of using same

Abstract

A method for determining a gaze convergence distance of a user of an eye movement tracking system. The method involves identifying a plurality of candidate convergence points along a combined gaze vector; and for each candidate convergence point, calculating a probability that the candidate convergence point is located at the gaze convergence distance based on: a) an angle between the left gaze vector and a straight line passing through the candidate convergence point from the left eye origin; b) an angle between the right gaze vector and a straight line passing through the candidate convergence point from the right eye origin; c) the distance between the left gaze vector and the candidate convergence point; and d) the distance between the right gaze vector and the candidate convergence point. The method then involves calculating a gaze convergence distance based on the candidate convergence point having the highest probability.

Description

Technical Field

[0001] The present disclosure relates generally to the field of eye tracking. In particular, the present disclosure relates to methods and systems for determining the gaze convergence distance of a user of an eye tracking system. Background Art

[0002] In eye tracking applications, digital images of a user's eyes are taken and analyzed to estimate the user's gaze direction. The estimation of the gaze direction can be based on computer-based image analysis of features of the imaged eye. One known example method of eye tracking includes the use of near infrared light and an image sensor. The near infrared light is directed at the user's eyes (binoculars), and the reflection of the light is captured by the image sensor.

[0003] Portable or wearable eye tracking devices have been described previously. One such eye tracking system is described in U.S. Patent No. 9,041,787 and PCT Patent Publication No. WO 2019 / 158709 (the entire contents of both patents are incorporated herein by reference). A wearable eye tracking device that uses an illuminator and a camera to determine gaze direction is described. Summary of the invention

[0004] According to a first aspect of the present disclosure, there is provided a method for determining a gaze convergence distance of a user of an eye tracking system, the method comprising:

[0005] take over:

[0006] a left gaze signal, which represents the left gaze vector from the left eye origin;

[0007] a right gaze signal, which represents the right gaze vector from the right eye origin;

[0008] a combined gaze signal representing the determined combined gaze vector;

[0009] identifying a plurality of candidate convergence points along the combined gaze vector;

[0010] For each candidate convergence point, the probability that the candidate convergence point is at the fixation convergence distance is calculated based on:

[0011] a) the angle between the left gaze vector and the line from the left eye origin through the candidate convergence point;

[0012] b) the angle between the right gaze vector and the line from the right eye origin through the candidate convergence point;

[0013] c) the distance between the left gaze vector and the candidate convergence point; and

[0014] d) the distance between the right gaze vector and the candidate convergence point; and

[0015] The gaze convergence distance is calculated based on the candidate convergence point with the highest probability.

[0016] Advantageously, this approach can improve the accuracy with which the gaze convergence distance of the user's eyes can be determined. This can be achieved by influencing the accuracy of the combined gaze signal by assuming that the convergence corresponds to a certain point on the combined gaze vector.

[0017] The probability of each candidate convergence point being at the fixation convergence can be calculated based on:

[0018] a) applying a probability distribution function to the angle between the left gaze vector and a line from the left eye origin through the candidate convergence point; and

[0019] b) Apply a probability distribution function to the angle between the right gaze vector and the line from the right eye origin through the candidate convergence point.

[0020] The probability of each candidate convergence point being at the fixation convergence can be calculated based on:

[0021] a) applying one of the following probability distribution functions to the angle between the left gaze vector and a line from the left eye origin through the candidate convergence point: a monotonically decreasing function, a normal distribution function, and a directional distribution function (such as a von Mises function); and

[0022] b) Apply one of the following probability distribution functions to the angle between the right gaze vector and the line from the right eye origin through the candidate convergence point: a monotonically decreasing function, a normal distribution function, and a von Mises function.

[0023] The probability of each candidate convergence point being at the fixation convergence can be calculated based on:

[0024] a) applying a three-dimensional probability distribution function to the angle between the left gaze vector and a line from the left eye origin through the candidate convergence point; and

[0025] b) Apply a three-dimensional probability distribution function to the angle between the right gaze vector and the line from the right eye origin through the candidate convergence point.

[0026] The probability of each candidate convergence point being at the fixation convergence can be calculated based on:

[0027] c) applying a probability distribution function to the distance between the left gaze vector and the candidate convergence point; and

[0028] d) Apply a probability distribution function to the distance between the right gaze vector and the candidate convergence point.

[0029] The probability of each candidate convergence point being at the fixation convergence can be calculated based on:

[0030] c) applying a monotonically decreasing probability distribution function to the distance between the left gaze vector and the candidate convergence point; and

[0031] d) Apply a monotonically decreasing probability distribution function to the distance between the right gaze vector and the candidate convergence point.

[0032] The monotonically decreasing function applied to the left gaze vector may be an exponential function with a power of d, where d is the distance between the left gaze vector and the candidate convergence point. The monotonically decreasing function applied to the right gaze vector may be an exponential function with a power of d, where d is the distance between the right gaze vector and the candidate convergence point.

[0033] The probability of each candidate convergence point being at the fixation convergence can be calculated based on:

[0034] c) the Euclidean or Minkowski / Manhattan distance between the left gaze vector and the candidate convergence point; and

[0035] d) Euclidean or Minkowski / Manhattan distance between the right gaze vector and the candidate convergence point.

[0036] The method may further comprise:

[0037] The determined gaze convergence distance is used to control the focal length of a zoom lens of the eye tracking system.

[0038] The method may further comprise:

[0039] presenting a plurality of selectable objects to a user on one or more display screens, wherein each selectable object of the plurality of selectable objects is at an object distance known to the user and at a known position on the one or more display screens;

[0040] determining a user's gaze angle while capturing images of the user's left and right eyes; and

[0041] The calculated gaze convergence distance is used in conjunction with the determined gaze angle to identify one of the selectable objects as a selected object.

[0042] The method may further include, before receiving the combined gaze signal, the left gaze signal, and the right gaze signal:

[0043] presenting a plurality of calibration objects to a user on one or more display screens, wherein the plurality of calibration objects are at known locations on the one or more display screens;

[0044] While rendering multiple calibration objects:

[0045] capturing a calibration image of a user's left eye, and determining calibration left gaze signals based on the captured calibration image of the user's left eye, each calibration left gaze signal representing a left gaze vector and a left eye origin of the user's left eye;

[0046] capturing a calibration image of a user's right eye, and determining calibration right gaze signals based on the captured calibration image of the user's right eye, each calibration right gaze signal representing a right gaze vector and a right eye origin of the user's right eye;

[0047] Sure:

[0048] One or more parameters of a probability distribution function for the user's left eye (e.g., a mean direction and a focus parameter of a von Mises distribution) are based on pairs of: i) a determined calibration left gaze signal; and ii) a known position of a calibration object displayed on one or more display screens while capturing a calibration image associated with the determined calibration left gaze signal; and

[0049] One or more parameters of the probability distribution function of the user's right eye are based on pairs of: i) a determined calibration right gaze signal; and ii) a known position of a calibration object displayed on one or more display screens while capturing a calibration image associated with the determined calibration right gaze signal.

[0050] Also disclosed is an eye tracking system configured to perform any of the methods disclosed herein.

[0051] The eye tracking system may be a head mounted eye tracking system.

[0052] Eye tracking systems can be extended reality systems.

[0053] A computer program may be provided that, when run on a computer, causes the computer to configure any device (including the controller, apparatus, or system disclosed herein), or to perform any method disclosed herein. The computer program may be a software implementation, and the computer may be considered to be any suitable hardware, including a digital signal processor, a microcontroller, and an implementation of a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or an electrically erasable programmable read-only memory (EEPROM), as non-limiting examples. The software may be an assembler.

[0054] The computer program may be provided on a computer readable medium, which may be a physical computer readable medium such as a disk or a memory device, or may be embodied as a transient signal. Such a transient signal may be downloaded over a network, including the Internet. One or more non-transitory computer readable storage media may be provided, storing computer executable instructions that, when executed by a computing system, cause the computing system to perform any of the methods disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] One or more embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0056] Figure 1 shows a simplified view of an eye tracking system in a head mounted device;

[0057] Figure 2 shows the movement of the eye by an eye tracking system (e.g. Figure 1 A simplified example of an image of a pair of eyes captured by a system;

[0058] Figure 3 shows a head mounted device worn on a user's head;

[0059] Figure 4 shows various properties associated with the user's eyes when the user looks at an object;

[0060] Figure 5a schematically illustrates representations of the following signals that may be provided by an eye tracking system: a left gaze signal, a right gaze signal, and a combined gaze signal;

[0061] Figure 5b An example embodiment of a controller is presented that may be part of an eye tracking system that processes a left gaze signal, a right gaze signal, and a combined gaze signal to calculate a gaze convergence distance of a user;

[0062] Figure 5c Various parameters used to calculate the probability of a particular convergence point among the candidate convergence points are shown;

[0063] Figure 6a We show how regions of candidate convergence points can have a constant combined confidence when the confidence is based only on angle;

[0064] Figure 6b Shown with Figure 6a the same vector and the candidate convergence point, and the shortest distance between each of the left gaze vector and the right gaze vector and the candidate convergence point;

[0065] Figure 7 Some simulation results showing an exemplary eye tracking system of the present disclosure; and

[0066] Figure 8 An example embodiment of a method for determining a gaze convergence distance of a user of an eye tracking system is shown. DETAILED DESCRIPTION

[0067] Figure 1 A simplified view of an eye tracking system 100 (which may also be referred to as a gaze tracking system) in a head mounted device in the form of a virtual reality or augmented reality (VR or AR) device, or VR or AR glasses, or anything related, such as an extended reality (XR) or mixed reality (MR) headset. The system 100 includes an image sensor 120 (e.g., a camera) for capturing an image of a user's eye. The system may optionally include one or more illuminators 110-119 (also referred to herein as light sources) for illuminating the user's eye, which may be, for example, light emitting diodes (LEDs) that emit light in an infrared band or a near infrared band, and may be physically arranged in various configurations. The image sensor 120 may be, for example, any type of image sensor, such as a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor. The image sensor may be comprised of an integrated circuit comprising an array of pixel sensors, each pixel comprising a photodetector and an active amplifier. The image sensor may be capable of converting light into a digital signal. In one or more examples, the image sensor may be an infrared image sensor or IR image sensor, an RGB sensor, an RGBW sensor, or an RGB or RGBW sensor with an IR filter.

[0068] Eye tracking system 100 may include circuitry or one or more controllers 125 for receiving and processing images captured by image sensor 120, for example, circuitry or one or more controllers including receiver 126 and processing circuitry 127. Circuitry 125 may be connected to image sensor 120 and optionally one or more illuminators 110-119, for example, via a wired connection or a wireless connection, and may be co-located with image sensor 120 and one or more illuminators 110-119, or may be located at a distance therefrom (e.g., in a different device). In another example, circuitry 125 may be disposed in one or more stacked layers below the photosensitive surface of light sensor 120.

[0069] The eye tracking system 100 may include a display (not shown) for presenting information and / or visual stimuli to the user. The display may include a VR display that presents imagery and substantially blocks the user's real-world view, or an AR display that presents imagery that is perceived as superimposed on the user's real-world view.

[0070] In such a system 100, the image sensor 120 for one eye is typically positioned away from the user's line of sight so as not to obstruct the display for that eye. This configuration can be achieved, for example, with the aid of a so-called hot mirror, which reflects a portion of the light and allows the rest of the light to pass through, e.g. infrared light is reflected while visible light is allowed to pass through.

[0071] Although in the above example, the image of the user's eyes is captured by the head-mounted image sensor 120, in other examples, the image may be captured by a non-head-mounted image sensor. Such a non-head-mounted system may be referred to as a remote system.

[0072] Figure 2 shows the movement of the eye by an eye tracking system (e.g. Figure 1 2 is a simplified example of an image 229 of a pair of eyes captured by a system (e.g., a pair of eyes captured by a pair of eyes). Image 229 can be considered to include a right eye image 228 of a person's right eye and a left eye image 234 of a person's left eye. In this example, right eye image 228 and left eye image 234 are two larger image portions of a person's two eyes. In other examples, separate image sensors can be used to acquire right eye image 228 and left eye image 234. In other examples, multiple image sensors can be used to acquire images that capture both eyes.

[0073] The system may employ image processing (such as digital image processing) to extract features in the image. The system may, for example, identify the location of the pupil 230 in one or more images captured by the image sensor. The system may determine the location of the pupil 230 using a pupil detection process. The system may also identify a corneal reflection (also referred to as a glint) 232 that is immediately adjacent to the pupil 230. The system may estimate the corneal center and / or the distance to the user's eye based on the corneal reflection 232. For example, the system may match each of the individual corneal reflections 232 of each eye with a corresponding illuminator and determine the corneal center of each eye and / or the distance to the user's eye based on the match. In general, the eye tracking system may determine the optical axis of the user's eye as a vector passing through the center of the pupil 230 and the center of the cornea. The gaze direction corresponds to the axis from the fovea of ​​the eye to the center of the cornea (the visual axis). The angle between the optical axis and the gaze direction is the foveal offset, which typically varies from user to user and is in the range of several degrees. The eye tracking system may perform a calibration procedure to instruct the user to gaze in a series of predetermined directions (e.g., via on-screen instructions) to determine the foveal offset. The determination of the optical axis described above is known to those skilled in the art and is generally referred to as the pupil center corneal reflection (PCCR). The PCCR will not be discussed in further detail herein.

[0074] Figure 3A head mounted device 337 is shown worn on the head of a user 335. The head mounted device 337 in this example is an extended reality (XR) head mounted device. XR head mounted devices include virtual reality (VR) head mounted devices, augmented reality (AR) head mounted devices, and mixed reality (MR) head mounted devices. The head mounted device 337 includes a 3-dimensional (3D) display screen 336, which is capable of visualizing an object and making the object appear to be at a certain distance from the user in response to control signals received from a computer. The head mounted device 337 can also typically use one or more gaze tracking sensors to determine the gaze angle, as is known in the art.

[0075] The 3D display screen 336 may be, for example, a stereoscopic display screen. Alternatively, the 3D display screen 336 may be a stereoscopic 3D display screen that is autostereoscopic or automulti-vision. This means that they can form 3D images visible to the naked eye without the need for stereo goggles or a stereo head-mounted display. Therefore, the various eye tracking systems described herein may be provided as a head-mounted device or a remote system that does not require stereo goggles or a stereo head-mounted display. In another example, the 3D display screen may be a remote display screen where stereo glasses are required to allow the user to see the 3D effect.

[0076] Figure 4 Various properties associated with the user's eyes when the user looks at an object are shown, which will be mentioned below.

[0077] exist Figure 4 , a user's right eye 442a and a user's left eye 442b are shown. A right gaze vector 439a of the user's right eye 442a is shown, which originates from the center of the user's right pupil 438a (which may also be referred to as the right eye origin). A left gaze vector 439b of the user's left eye 442b is shown, which originates from the center of the user's left pupil 438b (which may also be referred to as the left eye origin). The convergence point 440 can be determined as the intersection of the gaze vectors 439a, 439b from each eye. The gaze convergence distance is the distance between the user and the convergence point 440. The gaze convergence distance can be calculated as the distance from the left eye 442b to the convergence point 440 (i.e., along the left gaze vector 439b of the left eye 442b), or the gaze convergence distance can be calculated as the distance from the right eye 442a to the convergence point 440 (i.e., along the right gaze vector 439a of the right eye 442a), or the gaze convergence distance can be calculated as the distance 441 from the normal between the left eye 442b and the right eye 442a to the convergence point 440.

[0078] For Figure 3For the VR headset shown, the gaze convergence distance can be defined as the distance between the system origin of the VR headset and the intersection of the left gaze vector 439a and the right gaze vector 439b. Two vectors in 3D space intersect only if they are on the same plane, and the probability of two vectors in 3D space intersecting in practice is very small. Therefore, one way to determine the intersection of two vectors in 3D space is to calculate the closest point between the two vectors. However, this method is susceptible to noise when determining the gaze origin (cornea center).

[0079] The stereoscopic scene in the VR headset is presented at a fixed depth (focal plane) away from the eyes. If the vergence of the eyes is not in the same position as the focal plane at this time, the user may feel blur when looking at different objects in the scene. This phenomenon is the so-called vergence-accommodation conflict problem. In addition to focusing problems, vergence-accommodation conflict can also cause visual fatigue and eye strain.

[0080] As will be discussed in detail below, the examples disclosed herein can provide an improved method of determining the vergence distance of a user's eyes. The determined vergence distance can then be used to automatically adjust the zoom VR headset to set the correct focal plane of the stereoscopic scene for the user.

[0081] Figure 5a Schematically illustrated are representations of the following signals that may be provided by an eye tracking system: left gaze signal 550 , right gaze signal 553 , and combined gaze signal 556 . Figure 5b An example embodiment of a controller 560 is shown, which may be part of an eye tracking system, processing a left gaze signal 550 , a right gaze signal 553 , and a combined gaze signal 556 to calculate a gaze convergence distance 559 of a user.

[0082] Left gaze signal 550 represents left gaze vector 551 from left eye origin 552. Right gaze signal 553 represents right gaze vector 554 from right eye origin 555. Combined gaze signal 556 represents combined gaze vector 557 from combined origin 558. One or more of the gaze vectors and signals described herein may be provided by an eye tracking system as an output signal representing the direction in which the user is looking.

[0083] Eye tracking systems known in the art can provide a left gaze signal 550 and a right gaze signal 553. In addition, known eye tracking systems can combine the left gaze signal 550 with the right gaze signal 553 to provide a combined gaze signal 556. The combination of the left gaze signal 550 and the right gaze signal 553 can produce a low-noise combined gaze signal 556. For example, the combined gaze signal 556 can be a weighted combination of the left gaze signal 550 and the right gaze signal 553. The weights can be calculated by estimating the noise in the gaze calculation algorithm for each side (left / right). It has been found through testing that such a combined gaze signal 556 has a high degree of accuracy. As will be discussed below, examples of the present disclosure can affect the accuracy of the combined gaze signal 556 by assuming that the convergence point is located on the combined gaze vector 557.

[0084] Figure 5b The controller 560 identifies a plurality of candidate convergence points 562 along the combined gaze vector 557. The controller 560 processes each of these candidate convergence points 562 to determine whether it should be identified as corresponding to the user's convergence distance. It will be understood from the following description that there may be a finite number of candidate convergence points 562 (e.g., Figure 5a 55), or a continuous function may be applied along the combined gaze vector 557. The application of such a continuous function may be considered to identify an infinite number of candidate convergence points 562. The selection of a finite number of candidate convergence points 562 may be performed in any suitable manner, such as at regular intervals between two endpoint candidate convergence points. The endpoint candidate convergence points may be predefined for a given application, may be fixed values ​​for any application, or may be set in any other manner.

[0085] For each candidate convergence point 562, the controller 560 calculates the probability that the candidate convergence point 562 is located at the gaze convergence distance. The controller 560 calculates the probability of each candidate convergence point 562 based on the following:

[0086] a) the angle between the left gaze vector 551 and the straight line from the left eye origin 552 to the candidate convergence point 562;

[0087] b) the angle between the right gaze vector 554 and the line from the right eye origin 555 to the candidate convergence point 562;

[0088] c) the distance between the left gaze vector 551 and the candidate convergence point 562; and

[0089] d) The distance between the right gaze vector 554 and the candidate convergence point 562 .

[0090] Controller 560 may determine each of these probabilities by applying a probability distribution function.

[0091] Figure 5c The following parameters used to calculate the probability of a particular convergence point 562a among the candidate convergence points are shown:

[0092] a) Angle 563 between left gaze vector 551 and straight line 561 from left eye origin 552 to candidate convergence point 562a;

[0093] b) Angle 564 between right gaze vector 554 and line 569 from right eye origin 555 to candidate convergence point 562a;

[0094] c) the distance 565 between the left gaze vector 551 and the candidate convergence point 562a; and

[0095] d) The distance 566 between the right gaze vector 554 and the candidate convergence point 562a.

[0096] exist Figure 5c , the distances 565, 566 between the left gaze vector 551 and the right gaze vector 554 and the candidate convergence point 562a are the shortest distances between the candidate convergence point 562a and any point on the associated gaze vectors 551, 554. However, it will be appreciated that in other examples, the shortest distance need not be used as long as a consistent measure of the distances between the left gaze vector 551 and the right gaze vector 554 and the candidate convergence point 562a is used. Thus, the Euclidean or Minkowski / Manhattan distances between the candidate convergence point 562 and the corresponding left gaze vector 551 and the right gaze vector 554 may be used.

[0097] Once the controller 560 has calculated the probabilities of the candidate convergence points 562 , the controller may calculate the gaze convergence distance 559 based on the candidate convergence point 562 having the highest probability for each candidate convergence point 562 .

[0098] It has been found that calculating the gaze convergence distance 559 in this manner results in an accurate value for the gaze convergence distance 559. This is because the high accuracy of the combined gaze signal 556 can be exploited so that the gaze convergence distance 559 is accurately calculated in a computationally efficient manner.

[0099] Now, we will refer to Figure 5c The candidate convergence point 562a shown in FIG. 5 is examined in more detail. Figure 5b Examples of calculations that controller 560 may perform.

[0100] The controller 560 determines a probability distribution around each eye's gaze vector 551, 554. These probability distributions can be thought of as rays 561, 569 emanating from each eye's gaze origin 552, 555, where the angle 563, 564 between the ray 561, 569 and each eye's gaze direction 551, 554 has an effect on the probability that the actual convergence point is on that particular ray 561, 569. The larger the angle 563, 564, the more candidate convergence points deviate from the associated gaze direction 551, 554, and therefore the lower the probability that the candidate convergence point is the actual convergence point.

[0101] exist Figure 5c The following are known:

[0102] 1. Left gaze origin 552 , right gaze origin 555 , and combined gaze origin 558 .

[0103] 2. Left gaze vector 551 , right gaze vector 554 , and combined gaze vector 557 .

[0104] The candidate convergence point 562a under consideration is P. Let P be the actual convergence point of the probability distribution on each side, then its respective probability is f l (P) and f r (P). Let P be the convergence point of the two sides, then the joint probability is f(P).

[0105] We consider the individual probability distributions to be independent of each other, so we calculate the joint probability distribution as

[0106] f(P)=f l (P)*f r (P)

[0107] As indicated above, the respective probability distributions are based on the angular deviation from the respective gaze vectors. The angular deviation 563 of the left gaze vector 551 is α l The angle deviation 564 of the right gaze vector 551 is α r Substituting these into the equation above, we get,

[0108] f(P)=f l (α l )*f r (α r )

[0109] One method of modeling the probability distribution based on angle is to model the user's gaze distribution during a calibration procedure. That is, before receiving the combined gaze signal 556, the left gaze signal 550, and the right gaze signal 553, one or more parameters of the probability distribution function can be calibrated for the user.

[0110] Such a calibration procedure may involve presenting a plurality of calibration objects (also referred to as stimulus points) to the user on one or more display screens, wherein the plurality of calibration objects are at known locations on the one or more display screens. When presenting the plurality of calibration objects: a calibration image of the user's left eye is captured, so that the controller can determine a calibration left gaze signal based on the captured calibration image of the user's left eye, each calibration left gaze signal representing a left gaze vector and a left eye origin of the user's left eye. In addition, when presenting the plurality of calibration objects: a calibration image of the user's right eye may be captured, so that the controller can determine a calibration right gaze signal based on the captured calibration image of the user's right eye, each calibration right gaze signal representing a right gaze vector and a right eye origin of the user's right eye. In this way, gaze direction samples may be collected for each stimulus point, so that the controller can model how the gaze direction samples are distributed around the ground truth gaze direction.

[0111] The controller may then determine one or more parameters of a probability distribution function for the user's left eye based on pairs of: i) the determined calibration left gaze signal; and ii) a known position of a calibration object displayed on one or more display screens while capturing a calibration image associated with the determined calibration left gaze signal. The controller may also determine one or more parameters of a probability distribution function for the user's right eye based on pairs of: i) the determined calibration right gaze signal; and ii) a known position of a calibration object displayed on one or more display screens while capturing a calibration image associated with the determined calibration right gaze signal. In one example described below, the parameters may be the mean direction and focus parameters of a von Mises distribution function.

[0112] The gaze direction distribution can be modeled using a directional distribution such as the von Mises distribution by:

[0113] 1. Collect samples over a period of time during the calibration procedure and normalize the relative gaze angles.

[0114] 2. Calculate the sample mean and variance. The mean direction of the von Mises distribution will correspond to the sample mean, and the focus parameter of the von Mises distribution will correspond to the sample variance.

[0115] 3. Once these two parameters are determined, draw samples from the von Mises distribution using the calculated mean direction and focus parameters. Samples can be drawn for each data point of the target data.

[0116] 4. Perform a two-sample Kolmogorov–Smilov test between samples from the target data and samples collected from the von Mises distribution.

[0117] Since the gaze vector can be three-dimensional, it will be understood that the probability distribution function can be a three-dimensional probability distribution defining a confidence cone, with its vertex located at the gaze origin. Such a function defines the probability that the user's gaze is along the axis of the confidence cone for a given angular offset from the axis of the confidence cone for the gaze vector. If the gaze vector is aligned with the axis of the confidence cone, the gaze vector has a maximum probability value that is aligned with the user's actual gaze. If there is an offset between the gaze vector and the axis of the confidence cone, the probability that the user's gaze is actually aligned with the axis of the confidence cone decreases as the size of the offset increases.

[0118] Although the von Mises distribution is described above, in other examples, one of the following probability distribution functions may be used instead to define the probability value of a given angle 563, 564 between a respective one of the left gaze vector 551 and the right gaze vector 554 and a straight line from the corresponding eye origin 552, 555 to the candidate convergence point 562a:

[0119] Monotonically decreasing function;

[0120] • A normal distribution function, which may be centered on a respective one of the left gaze vector 551 and the right gaze vector 554 such that it has a maximum value for an angle of zero degrees.

[0121] However, it has been found that using only the probability distribution based on the angle may result in inaccurate calculated convergence distance. More specifically, by modeling only the distribution of the angle deviation, it has been found that there is a continuous region along the combined gaze vector where the combined confidence from the left and right confidence cones is constant and reaches a maximum value. Within this region, the confidence from each confidence cone complements each other. This is because the angle relative to one confidence cone decreases as the angle relative to the other confidence cone increases.

[0122] Figure 6a It shows how a region of candidate convergence points 662 has a constant combined confidence when the confidence is based only on angle. The figure shows an exaggerated divergence between the left gaze vector 651 and the right gaze vector 654 for ease of explanation.

[0123] The figure shows the left gaze vector (g l )651, right gaze vector (g r )654 and the combined gaze vector (g c )657. We consider the combined confidence at points (P1, P2, P3, P4) on the combined gaze vector 657. The angles of the left gaze vector 651 and the right gaze vector 654 pointing to these points are shown in the figure, so that

[0124] α1>α2>α3>α4 and β1<β2<β3<β4

[0125] If we consider the probability distribution based only on the angle, the corresponding probabilities have the same relationship. That is,

[0126] P(α1)>P(α2)>P(α3)>P(α4) and P(β1)>P(β2)>P(β3)>P(β4)

[0127] Then the probability distributions of the left and right confidence cones may result in the combined probabilities of these points being equal. That is,

[0128] P(α1)*P(β1)=P(α2)*P(β2)=P(α3)*P(β3)=P(α4)*P(β4)

[0129] However, as will now be described, it has been found that if we include the distances (in this example, the shortest distances) from each of these points (P1, P2, P3, P4) to the left gaze vector 651 and the right gaze vector 654, then these distances, and therefore any probabilities based on these distances, will be less likely to have a similar relationship. Figure 6b As shown, it shows that Figure 6a The same vector and candidate convergence point. Figure 6b In FIG. 6 , the shortest distance between each of the left gaze vector 651 and the right gaze vector 654 and the candidate convergence points ( P1 , P2 , P3 , P4 ) is marked.

[0130] If the left gaze vector 651 and the right gaze vector 654 are completely parallel, an exception will occur. In this case, the convergence distance is infinite. However, the controller can handle this as a unique edge case. For example, if the controller determines that the left gaze vector 651 and the right gaze vector 654 are parallel to each other, the convergence distance can be automatically set to a predetermined value. Such a predetermined value can be the maximum value allowed by the application that will use the convergence distance.

[0131] Back to Figure 5c , the examples described herein therefore include a distance term in the probability distribution. In this example, the distance term is based on the shortest distance from the candidate convergence point P to each gaze vector 551, 554 (although as indicated above, using the shortest distance is not required). Figure 5c , the distance 565 between the left gaze vector 551 and the candidate convergence point 562a will be referred to as d l The distance 566 between the right gaze vector 554 and the candidate convergence point 562a will be referred to as d r Therefore, we determine the formula for this modified probability distribution based on angle and distance. Let this function be h(α, d).

[0132] Then the combined probability distribution is:

[0133] f(P)=h l (α l , d l )*h r (α r , d r )

[0134] In this example, the distance-based probability distribution term we use is

[0135]

[0136] where d is the shortest distance to the relevant left or right gaze vector, and C d is a constant that can be manually adjusted during the calibration routine.

[0137] Combining this formula with the probability distribution based on the angle deviation α, we obtain

[0138]

[0139] where V is the von Mises distribution described above.

[0140] In this manner, the controller calculates the probability that the candidate convergence point 562a is located at the gaze convergence based on:

[0141] The distance d between the left gaze vector 551 and the candidate convergence point 562a l applying a monotonically decreasing probability distribution function; and

[0142] The distance d between the right gaze vector 564 and the candidate convergence point 562a r Applies a monotonically decreasing probability distribution function.

[0143] In this example, the monotonically decreasing function is an exponential function raised to a power of d, where d is the distance between the candidate convergence point 562a and a corresponding one of the left gaze vector 551 and the right gaze vector 554. The exponential function is a convenient function because it provides a value of 1 (to represent a maximum probability value) when the distance d is zero. In other examples, the probability distribution function applied to the distance between the left gaze vector 561 or the right gaze vector 564 and the candidate convergence point 562 may be a normal distribution or any other suitable function.

[0144] Figure 7 Some simulation results of an exemplary eye tracking system of the present disclosure are shown. The horizontal axis represents the distance of the candidate convergence point from the combined fixation origin. The vertical axis represents the probability of the point being the convergence point. Figure 7 The peak of each curve shown in indicates the most likely convergence distance of the information represented by that curve.

[0145] The simulation results are for the following set of signals:

[0146] A left gaze signal, which has a left eye origin of [30 0 0] and a left gaze vector / direction of [-70 0 250];

[0147] a right gaze signal with a right eye origin of [-30 0 0] and a right gaze vector / direction of [50 0 250]; and

[0148] A combined gaze signal with a combined eye origin of [0 0 0] and a combined gaze vector / direction of [0 0 21];

[0149] Figure 7 A curve of probability versus convergence distance is shown based on:

[0150] The angle between the left gaze vector only and the line from the left eye origin through the candidate convergence point (curve 770);

[0151] The angle between the right gaze vector only and the line from the right eye origin through the candidate convergence point (curve 771);

[0152] • Both the distance between the left gaze vector and the candidate convergence point; and the angle between the left gaze vector and a line from the left eye origin through the candidate convergence point (curve 772);

[0153] • Both the distance between the right gaze vector and the candidate convergence point; and the angle between the right gaze vector and a line from the right eye origin through the candidate convergence point (curve 773);

[0154] The combination of the angles of the left and right gaze vectors only (curve 774); and

[0155] • The combination of the angle and distance of the left and right gaze vectors (curve 775).

[0156] By constructing this test case in simulation, a case with a constant probability region can be generated when a probability function based only on angle is used. This can be seen by the plateau at approximately 100-200 mm in curve 774, which is parallel to the horizontal axis and represents multiple convergence distances, each of which is equally likely to be the true convergence distance. However, curve 775 shows that a single maximum probability point is generated at approximately 100 mm using a probability function based on both angle and distance. That is, the flat area that can be seen in curve 775 is no longer parallel to the horizontal axis, so that in this example, the curve has a maximum at the beginning of the flat area. Therefore, using a probability function for both angle and distance simultaneously resolves the ambiguity that occurs when only a probability function for angle is used.

[0157] Figure 8 An example embodiment of a method for determining a gaze convergence distance of a user of an eye tracking system is shown.

[0158] At step 880, the method involves receiving a left gaze signal, a right gaze signal, and a combined gaze signal. As discussed above: the left gaze signal represents a left gaze vector from a left eye origin; the right gaze signal represents a right gaze vector from a right eye origin; and the combined gaze signal represents the determined combined gaze vector.

[0159] At step 881, the method involves identifying a plurality of candidate convergence points along the combined gaze vector. Then, at step 882, for each candidate convergence point, the method involves calculating a probability that the candidate convergence point is located at the gaze convergence distance. As discussed above, the probability is calculated based on:

[0160] a) the angle between the left gaze vector and the line from the left eye origin through the candidate convergence point;

[0161] b) the angle between the right gaze vector and the line from the right eye origin through the candidate convergence point;

[0162] c) the distance between the left gaze vector and the candidate convergence point; and

[0163] d) The distance between the right gaze vector and the candidate convergence point.

[0164] Then, at step 883, the method involves calculating a gaze convergence distance based on the candidate convergence point with the highest probability.

[0165] Once the gaze convergence distance has been determined, the method may proceed to control the focal length of a zoom lens of the eye tracking system by using the determined gaze convergence distance. For example: a voltage may be applied to an electrically controlled zoom lens; or a mechanical actuator may be activated to move the lens and change the position of the associated focal plane. In this way, the eye tracking system may be automatically controlled to improve the viewing experience of the user. Such functionality may be particularly useful in extended reality systems such as augmented reality systems and virtual reality systems.

[0166] Additionally or alternatively, the determined gaze convergence distance can be used by the eye tracking system as part of an algorithm to more accurately identify which of a plurality of selectable objects a user is looking at. This can involve: the method presenting a plurality of selectable objects to a user on one or more display screens; each of the plurality of selectable objects is at a known object distance from the user and at a known position on the one or more display screens. The method can determine the user's gaze angle while capturing images of the user's left and right eyes, and then use the determined gaze convergence distance in combination with the determined gaze angle to identify one of the selectable objects as a selected object. For example, a cost function can be applied that seeks to minimize the difference between: i) the known object distance of each selectable object and the determined gaze convergence distance; and ii) the known position of each selectable object in the display screen and the determined gaze angle. In this way, the determined gaze convergence distance is used to supplement the determined gaze angle as part of the eye tracking operation.

Claims

1. A method for determining a gaze convergence distance of a user of an eye tracking system, the method comprising: take over: a left gaze signal, which represents the left gaze vector from the left eye origin; a right gaze signal, which represents the right gaze vector from the right eye origin; a combined gaze signal representing the determined combined gaze vector; identifying a plurality of candidate convergence points along the combined gaze vector; For each candidate convergence point, the probability that the candidate convergence point is located at the gaze convergence distance is calculated based on: a) the angle between the left gaze vector and a straight line from the left eye origin through the candidate convergence point; b) the angle between the right gaze vector and a straight line from the right eye origin through the candidate convergence point; c) the distance between the left gaze vector and the candidate convergence point; as well as d) the distance between the right gaze vector and the candidate convergence point; as well as The gaze convergence distance is calculated based on the candidate convergence point with the highest probability.

2. The method of claim 1, wherein: The probability that each candidate convergence point is located at the fixation convergence is calculated based on: a) applying a probability distribution function to the angle between the left gaze vector and a line from the left eye origin through the candidate convergence point; and b) applying a probability distribution function to the angle between the right gaze vector and a line from the right eye origin through the candidate convergence points.

3. The method of claim 2, wherein: The probability that each candidate convergence point is located at the fixation convergence is calculated based on: a) applying one of the following probability distribution functions: a monotonically decreasing function, a normal distribution function, and a von Mises function to the angle between the left gaze vector and a straight line from the left eye origin through the candidate convergence point; and b) applying one of the following probability distribution functions: a monotonically decreasing function, a normal distribution function, and a von Mises function to the angle between the right gaze vector and a straight line from the right eye origin to the candidate convergence point.

4. The method according to claim 2 or claim 3, wherein: The probability that each candidate convergence point is located at the fixation convergence is calculated based on: a) applying a three-dimensional probability distribution function to the angle between the left gaze vector and a line from the left eye origin through the candidate convergence point; and b) applying a three-dimensional probability distribution function to the angle between the right gaze vector and a line from the right eye origin through the candidate convergence points.

5. A method as claimed in any preceding claim, wherein: The probability that each candidate convergence point is located at the fixation convergence is calculated based on: c) applying a probability distribution function to the distance between the left gaze vector and the candidate convergence point; and d) applying a probability distribution function to the distance between the right gaze vector and the candidate convergence point.

6. The method of claim 5, wherein: The probability that each candidate convergence point is located at the fixation convergence is calculated based on: c) applying a monotonically decreasing probability distribution function to the distance between the left gaze vector and the candidate convergence point; and d) applying a monotonically decreasing probability distribution function to the distance between the right gaze vector and the candidate convergence point.

7. The method of claim 6, wherein: The monotonically decreasing function applied to the left gaze vector is an exponential function with a power of d, where d is the distance between the left gaze vector and the candidate convergence point; and The monotonically decreasing function applied to the right gaze vector is an exponential function with a power of d, where d is the distance between the right gaze vector and the candidate convergence point.

8. The method of claim 7, wherein: The probability that each candidate convergence point is located at the fixation convergence is calculated based on: c) the Euclidean or Minkowski distance between the left gaze vector and the candidate convergence point; and d) The Euclidean or Minkowski distance between the right gaze vector and the candidate convergence point.

9. The method of any preceding claim, further comprising: The determined gaze convergence distance is used to control the focal length of a zoom lens of the eye tracking system.

10. The method of any preceding claim, further comprising: presenting a plurality of selectable objects to the user on one or more display screens, wherein each selectable object of the plurality of selectable objects is at a known object distance from the user and at a known position on the one or more display screens; determining a gaze angle of the user while capturing images of the user's left and right eyes; and One of the selectable objects is identified as a selected object using the calculated gaze convergence distance in combination with the determined gaze angle.

11. The method of any preceding claim, further comprising, prior to receiving the combined gaze signal, the left gaze signal and the right gaze signal: A plurality of calibration objects are presented to the user on one or more display screens, wherein: the plurality of calibration objects being at known locations on the one or more display screens; When presenting the plurality of calibration objects: Capturing a calibration image of the user's left eye, and determining calibration left gaze signals based on the captured calibration image of the user's left eye, each calibration left gaze signal representing a left gaze vector and a left eye origin of the user's left eye; Capturing a calibration image of the user's right eye, and determining calibration right gaze signals based on the captured calibration image of the user's right eye, each calibration right gaze signal representing a right gaze vector and a right eye origin of the user's right eye; Sure: one or more parameters of a probability distribution function of a left eye of the user, the determination being based on a pair of: i) a determined calibration left gaze signal; and ii) a known position of the calibration object displayed on the one or more display screens while capturing the calibration image associated with the determined calibration left gaze signal; as well as one or more parameters of a probability distribution function of the user's right eye, the determination being based on pairs of: i) the determined calibrated right gaze signal; and ii) displaying the known position of the calibration object on one or more display screens while capturing the calibration image associated with the determined calibration right gaze signal.

12. An eye tracking system configured to perform the method of any preceding claim.

13. The eye tracking system of claim 12, wherein: The eye tracking system is a head-mounted eye tracking system.

14. An eye tracking system as claimed in claim 12 or claim 13, wherein: The eye tracking system is an extended reality system.

Images