Eye gaze tracking systems, related methods and computer programs

Abstract

An eye-tracking system is configured to: receive multiple right-eye images of a user's right eye; receive multiple left-eye images of a user's left eye, each left-eye image corresponding to a certain right-eye image; for each right-eye image and each left-eye image among the multiple right-eye images, detect pupils and determine associated pupil signals; calculate right-eye pupil changes in the pupil signals of the multiple right-eye images and left-eye pupil changes in the pupil signals of the multiple left-eye images; and determine right-eye weights and left-eye weights based on the right-eye pupil changes and left-eye pupil changes. For one or more right-eye images and one or more corresponding left-eye images, the eye-tracking system can: determine at least one right-eye gaze signal based on the right-eye image and at least one left-eye gaze signal based on the corresponding left-eye image; and calculate a combined gaze signal based on a weighted sum of the right-eye gaze signal and the left-eye gaze signal using the right-eye weight and the left-eye weight.

Description

Technical Field

[0001] This disclosure generally relates to the field of eye tracking. Specifically, this disclosure relates to systems and methods that use pupil detection in eye images as part of an eye tracking system. Background Technology

[0002] In eye-tracking applications, digital images of the user's eyes are acquired and analyzed to estimate the user's gaze direction. Gazing direction estimation can be based on computer-based image analysis of the features of the imaged eye. Many eye-tracking systems estimate gaze direction based on the identification of pupil position and flashes of light or corneal reflexes. Therefore, the accuracy of gaze direction estimation may depend on the accuracy of pupil position and / or corneal reflex identification or detection. Digital images may contain one or more spurious image features, such as stray reflections, which can adversely affect eye feature recognition.

[0003] One known example of eye-tracking involves using infrared light and an image sensor. The infrared light is directed toward the user's pupil, and the reflection of the light is captured by the image sensor.

[0004] Portable or wearable eye-tracking devices have been previously described. One such eye-tracking system is described in U.S. Patent No. 9,041,787 (the entire contents of which are incorporated herein by reference). A wearable eye-tracking device is described that uses an emitter and an image sensor to determine the gaze direction. The processing power of the hardware in a portable wearable eye-tracking device may be limited compared to a standalone or remote system. Summary of the Invention

[0005] According to a first aspect of the present invention, an eye-tracking system is provided, the eye-tracking system being configured to:

[0006] Receive multiple right-eye images from the user's right eye;

[0007] Receive multiple left-eye images of the user's left eye, each left-eye image corresponding to a specific right-eye image among multiple right-eye images;

[0008] For each right-eye image in multiple right-eye images and each left-eye image in multiple left-eye images, detect the pupil and determine the associated pupil signal;

[0009] Calculate the right eye pupil changes from multiple right eye images and the left eye pupil changes from multiple left eye images;

[0010] The weights for the right and left eyes are determined based on the changes in the right and left pupils; and

[0011] For one or more right-eye images and one or more corresponding left-eye images:

[0012] Determine at least one right-eye gaze signal based on the right-eye image and at least one left-eye gaze signal based on the corresponding left-eye image; and

[0013] The combined gaze signal is calculated by weighting the right-eye gaze signal and the left-eye gaze signal using the right-eye weight and the left-eye weight.

[0014] Such combined gaze signals can have improved accuracy and / or precision. It has been found that particularly good combined gaze signals can be obtained by determining weights using variations in the pupil signal. Specifically, in determining the combined gaze signal, the pupil signal can be considered an intermediate signal. Therefore, it has been found that calculating weights based on the (intermediate) pupil signal can produce particularly good improvements to the combined gaze signal.

[0015] The pupil signal may include pupil position and / or pupil radius.

[0016] The right eye gaze signal may include the right eye gaze origin signal. The left eye gaze signal may include the left eye gaze origin signal. The combined gaze signal may include the combined gaze origin signal.

[0017] Determining the right-eye gaze signal and the left-eye gaze signal may include determining the right-eye gaze origin signal based on the right-eye image and determining the left-eye gaze origin signal based on the corresponding left-eye image. Calculating the combined gaze signal may include calculating the combined gaze origin signal by weighting the right-eye gaze origin signal and the left-eye gaze origin signal using the right-eye weight and the left-eye weight.

[0018] The eye-tracking system can be configured to: determine the right eye gaze origin signal by detecting one or more flashes in the right eye image and calculating the right eye gaze origin signal based on the one or more flashes; and determine the left eye gaze origin signal by detecting one or more flashes in the left eye image and calculating the left eye gaze origin signal based on the one or more flashes.

[0019] The eye-tracking system can be configured to: set the right eye weight to zero if the right eye gaze origin signal is outside the expected right eye origin region; and / or set the left eye weight to zero if the left eye gaze origin signal is outside the expected left eye origin region.

[0020] The right eye gaze signal may include a right eye gaze direction signal. The left eye gaze signal may include a left eye gaze direction signal. The combined gaze signal may include a combined gaze direction signal.

[0021] Determining the right-eye gaze signal and the left-eye gaze signal may include determining the right-eye gaze direction signal based on the right-eye image and determining the left-eye gaze direction signal based on the corresponding left-eye image. Calculating the combined gaze signal may include calculating the combined gaze direction signal by weighting the right-eye gaze direction signal and the left-eye gaze direction signal using the right-eye weight and the left-eye weight.

[0022] The eye-tracking system can be configured to:

[0023] Determine the right eye gaze direction signal using the following steps:

[0024] Detect one or more flashes of light in the right eye image;

[0025] The right eye fixation origin signal is calculated based on the one or more flashes;

[0026] The position of the right eye pupil detected in the right eye image is determined; and the right eye gaze direction signal is calculated based on the right eye gaze origin signal and the pupil position.

[0027] The eye-tracking system can be configured to:

[0028] Determine the left eye's gaze direction signal using the following steps:

[0029] Detect one or more flashes in the left eye image;

[0030] The left eye fixation origin signal is calculated based on the one or more flashes;

[0031] The position of the left eye pupil detected in the left eye image is determined; and the left eye gaze direction signal is calculated based on the left eye gaze origin signal and the pupil position.

[0032] The eye-tracking system can be configured to determine the right eye weights and the left eye weights such that they are normalized, for example, such that their sum is 1.

[0033] Changes in the right eye pupil can be based on the standard deviation of pupil signals from multiple right eye images. Changes in the left eye pupil can be based on the standard deviation of pupil signals from multiple left eye images.

[0034] The eye-tracking system can be configured to:

[0035] The right eye weight is determined based on a nonlinear function that correlates the right eye pupil change with the right eye weight; and

[0036] The left eye weight is determined based on a nonlinear function that correlates the change in the left eye pupil with the left eye weight.

[0037] The eye-tracking system can be configured to:

[0038] For each right-eye image in the plurality of right-eye images and each left-eye image in the plurality of left-eye images, determine one or more additional eye feature signals;

[0039] For each of the one or more additional eye feature signals:

[0040] The changes in right-eye features are calculated based on the eye feature signals of the multiple right-eye images; and the corresponding changes in left-eye features are calculated based on the eye feature signals of the multiple left-eye images; and

[0041] The weights of the right eye and the left eye are determined based on the changes in the right pupil, the changes in the left pupil, the changes in one or more right eye features, and the changes in one or more left eye features.

[0042] The eye-tracking system can be configured to determine the right eye weight and the left eye weight by the following steps:

[0043] Determine pupil weight pairs, the pupil weight pairs including right eye pupil weight based on the change of the right eye pupil and left eye pupil weight based on the change of the left eye pupil;

[0044] For each of the one or more additional eye feature signals:

[0045] Determine feature weight pairs, the feature weight pairs including right eye feature weights based on the right eye feature changes and left eye feature weights based on the left eye feature changes;

[0046] Normalize the pupil weight pairs and the one or more feature weight pairs;

[0047] Determine the minimum normalized weight among all normalized weights in the normalized pupil weight pair and the one or more normalized feature weight pairs; and

[0048] The right eye weight and the left eye weight are determined as a normalized pupil weight pair or a normalized feature weight pair that includes the minimum normalized weight.

[0049] The eye-tracking system can be configured to:

[0050] The right eye weight is determined based on a nonlinear function that correlates the right eye pupil change with the right eye weight.

[0051] The left eye weight is determined based on a nonlinear function that correlates the change in the left eye pupil with the left eye weight.

[0052] The one or more right-eye feature weights are determined based on a nonlinear function that associates the associated right-eye feature weights with the one or more right-eye feature weights; and / or

[0053] The one or more left-eye feature weights are determined based on a nonlinear function that associates the associated left-eye feature weights with the one or more left-eye feature weights.

[0054] The one or more additional eye feature signals may represent one or more of the following:

[0055] Pupil position;

[0056] Pupil radius;

[0057] Focus on the origin;

[0058] Corneal features (e.g., those manifested by corneal signals), such as corneal position and / or one or more corneal reflexes;

[0059] Iris position;

[0060] Iris radius;

[0061] The corner of the eye; and

[0062] Number of matched emitters-flash pairs.

[0063] The right eye gaze signal may include the right eye gaze origin signal. The left eye gaze signal may include the left eye gaze origin signal. The combined gaze signal may include the combined gaze origin signal.

[0064] The eye-tracking system can be configured to:

[0065] If the right eye fixation origin signal is located outside the expected right eye origin region, then the right eye weight is set to zero; and / or

[0066] If the left eye gaze origin signal is outside the expected left eye origin region, then the left eye weight is set to zero.

[0067] The eye-tracking system may further include a buffer configured to store multiple pupil signals corresponding to the most recent right-eye image and the most recent left-eye image. The eye-tracking system may be configured to: calculate the right-eye pupil change of the pupil signal of the right-eye image stored in the buffer; and calculate the left-eye pupil change of the pupil signal of the left-eye image stored in the buffer.

[0068] The eye-tracking system can be configured to use a moving average filter to update the right eye pupil change and the left eye pupil change when the contents of the buffer are updated.

[0069] According to another aspect, a head-mounted device is provided that includes any eye-tracking system disclosed herein.

[0070] According to another aspect, a method is provided for weighting right-eye and left-eye images to calculate a combined gaze signal, the method comprising:

[0071] Receive multiple right-eye images from the user's right eye;

[0072] Receive multiple left-eye images of the user's left eye, each left-eye image corresponding to a certain right-eye image among the multiple right-eye images;

[0073] For each right-eye image in the plurality of right-eye images and each left-eye image in the plurality of left-eye images, the pupil is detected and the associated pupil signal is determined;

[0074] Calculate the right eye pupil change of the pupil signals of the multiple right eye images and the left eye pupil change of the pupil signals of the multiple left eye images;

[0075] The weight of the right eye is determined based on the change in the right pupil, and the weight of the left eye is determined based on the change in the left pupil; and

[0076] For one or more right-eye images and one or more corresponding left-eye images:

[0077] Determine the right eye gaze signal based on the right eye image and determine the left eye gaze signal based on the corresponding left eye image; and

[0078] The combined gaze signal is calculated by weighting the right-eye gaze signal and the left-eye gaze signal using the right-eye weight and the left-eye weight.

[0079] A computer program may be provided that, when executed on a computer, causes the computer to configure any device (including the circuits, controllers, converters, or devices disclosed herein) or to perform any of the methods disclosed herein. The computer program may be a software implementation, and the computer may be considered any suitable hardware, including a digital signal processor, a microcontroller, and may be implemented as a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or an electrically erasable programmable read-only memory (EEPROM), as a non-limiting example. The software may be an assembler.

[0080] A computer program may be provided on a computer-readable medium, which may be a physical computer-readable medium such as a disk or storage device, or may embody a transient signal. Such a transient signal may be a network download, including an internet download. One or more non-transitory computer-readable storage media may be provided storing computer-executable instructions that, when executed by a computing system, cause the computing system to perform any of the methods disclosed herein.

[0081] An eye-tracking system is also disclosed, which is configured to:

[0082] Receive multiple right-eye images from the user's right eye;

[0083] Receive multiple left-eye images of the user's left eye, each left-eye image corresponding to a certain right-eye image among the multiple right-eye images;

[0084] For each right-eye image in the plurality of right-eye images and each left-eye image in the plurality of left-eye images, image features are detected and associated eye feature signals are determined;

[0085] Calculate the right-eye feature changes of the eye feature signals of the multiple right-eye images and the left-eye feature changes of the eye feature signals of the multiple left-eye images;

[0086] The weight of the right eye is determined based on the changes in the right eye features, and the weight of the left eye is determined based on the changes in the left eye features.

[0087] For one or more right-eye images and one or more corresponding left-eye images:

[0088] Determine the right eye gaze signal based on the right eye image and determine the left eye gaze signal based on the corresponding left eye image; and

[0089] The combined gaze signal is calculated by weighting the right-eye gaze signal and the left-eye gaze signal using the right-eye weight and the left-eye weight.

[0090] The eye feature signal may be a pupil signal or a corneal signal, or any other eye feature signal disclosed herein. The image feature may be a pupil or a flash. Attached Figure Description

[0091] Embodiments of the present invention will now be described in detail by way of example only, with reference to the following accompanying drawings, in which:

[0092] Figure 1 A schematic diagram of an eye-tracking system is shown, which can be used to capture image sequences that can be used by the example embodiment.

[0093] Figure 2An example image of a pair of eyes is shown;

[0094] Figure 3 An example of an eye-tracking system according to an embodiment of the present disclosure is shown;

[0095] Figure 4 An example of an sigmoid function is shown graphically;

[0096] Figure 5 The computer-implemented method according to embodiments of the present disclosure is illustrated schematically; and

[0097] Figure 6 A module of an eye-tracking system according to another embodiment of the present disclosure is shown.

[0098] All accompanying drawings are schematic and generally only show the parts necessary to illustrate the various embodiments, while other parts may be omitted or only implied. Detailed Implementation

[0099] Figure 1 A simplified view of an eye-tracking system 100 (which may also be referred to as a gaze tracking system) in a head-mounted device, in the form of a virtual or augmented reality (VR or AR) device, VR or AR glasses, or anything related to it, such as an extended reality (XR) or mixed reality (MR) head-mounted device. System 100 includes an image sensor 120 (e.g., a camera) for capturing images of a user's eyes. The system may optionally include one or more emitters 110 to 119 for illuminating the user's eyes; these emitters may be, for example, light-emitting diodes emitting light in the infrared or near-infrared bands, and may be physically arranged in various configurations. Image sensor 120 may be, for example, any type of image sensor, such as a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor. The image sensor may consist of an integrated circuit containing an array of pixel sensors, each pixel containing a photodetector and an active amplifier. The image sensor is capable of converting light into digital signals. In one or more examples, the image sensor may be an infrared image sensor or IR image sensor, an RGB sensor, an RGBW sensor, or an RGB or RGBW sensor with an IR filter.

[0100] The eye-tracking system 100 may include a circuitry or one or more controllers 125 for receiving and processing images captured by the image sensor 120, such as a receiver 126 and a processing circuitry 127. The circuitry 125 may be connected to the image sensor 120 and optionally one or more emitters 110-119, for example via a wired or wireless connection, and may be located in the same location as or at a distance from the image sensor 120 and the one or more emitters 110-119 (e.g., in different devices). In another example, the circuitry 125 may be disposed in one or more stacked layers beneath the photosensitive surface of the light sensor 120.

[0101] The eye-tracking system 100 may include a display (not shown) for presenting information and / or visual stimuli to the user. The display may include a VR display that presents images and substantially blocks the user's field of vision of the real world, or an AR display that presents images that are perceived as superimposed on the user's real-world field of vision.

[0102] In such a system 100, the image sensor 120 for one eye is typically positioned away from the user's line of sight so as not to obstruct the display for that eye. This configuration can be achieved, for example, by means of a so-called thermal mirror, which reflects a portion of the light while allowing the rest of the light to pass through; for example, infrared light is reflected while visible light is allowed to pass through.

[0103] While in the example above, the image of the user's eyes is captured by a head-mounted image sensor 120, in other examples, the image may be captured by a non-head-mounted image sensor. Such a non-head-mounted system may be referred to as a remote system.

[0104] In eye-tracking systems, gaze signals can be calculated for each of the user's eyes (left and right). The quality of these gaze signals can be degraded by interference in the input image (such as image noise) and incorrect algorithmic behavior (such as incorrect predictions). The goal of eye-tracking systems is to provide gaze signals that are as good as possible in terms of accuracy (bias error) and precision (variance error). For many applications, providing only one gaze signal at each moment may be sufficient, rather than providing gaze signals for the left and right eyes separately. Furthermore, combined gaze signals can be provided in combination with the left and right signals. Such gaze signals can be called combined gaze signals.

[0105] Figure 2 This illustrates the use of eye-tracking systems (such as...) Figure 1A simplified example of a system capturing images 229 of a pair of eyes. Image 229 can be considered to include a right-eye image 228 of a person's right eye and a left-eye image 234 of a person's left eye. In this example, the right-eye image 228 and the left-eye image 234 are two parts of a larger image of a person's two eyes. In other examples, separate image sensors can be used to acquire the right-eye image 228 and the left-eye image 234. The system can employ image processing (such as digital image processing) to extract features from the images. The system can, for example, identify the location of pupils 230, 236 in one or more images captured by the image sensor. The system can use a pupil detection process to determine the location of pupils 230, 236. The system can also identify corneal reflections 232, 238 adjacent to pupils 230, 236. The system can estimate the corneal center or the center of the eyeball based on corneal reflections 232, 238.

[0106] Figure 3 An example of an eye-tracking system 340 according to an embodiment of the present disclosure is shown. Figure 3 The functionality shown can be provided by one or more controllers. The eye-tracking system can be part of, or associated with, a head-mounted device or a remote system. Eye-tracking system 340 calculates a combined gaze signal 350 representing a person's gaze. The combined gaze signal 350 can include one or more user gaze parameters, such as a combined gaze origin signal and / or a combined gaze direction signal. It should be understood that the various modules of the eye-tracking system 340 described below can be implemented in software or hardware.

[0107] The eye-tracking system 340 receives multiple right-eye images of the user's right eye and multiple left-eye images of the user's left eye. The multiple right-eye and left-eye images can be provided as an image sequence of the eyes, either one at a time or together as a batch of images. The multiple images can be a temporal sequence of images, where the images in the sequence are spaced apart from each other in time.

[0108] As discussed above, the right-eye and left-eye images can be part of a larger common image acquired by a single camera, or they can be acquired by different cameras. Several left-eye images correspond to several right-eye images among multiple right-eye images. That is, a left-eye image can have a corresponding right-eye image of the same person. The corresponding left-eye and right-eye images may be acquired at the same time (e.g., as part of the same original image of both eyes) or at least when the human eye is expected to gaze in the same direction (e.g., because a stationary stimulus point is shown to the user). Therefore, it should be understood that the left-eye and right-eye images do not necessarily need to be synchronized with each other, as they do not necessarily need to be acquired at the same time. In some applications, the left-eye and right-eye images may be staggered in time but are still considered to correspond to each other.

[0109] The eye-tracking system 340 includes a pupil detector 342 that detects pupils in images and determines associated pupil signals. In this example, the pupil detector 342 determines a right-eye pupil signal for each of a plurality of right-eye images, and it also determines a left-eye pupil signal for each of a plurality of left-eye images. The pupil signal may include one or more signals describing the position and shape of the pupil, such as pupil position and / or pupil radius. In this way, the pupil signal may include data related to the pupil position and / or pupil radius of the eye. The pupil detector 342 may use any known pupil detection method to determine the left-eye and right-eye pupil signals. For example, the pupil detector 342 may apply detection and fitting algorithms. The pupil detector 342 may perform any known pupil detection procedure.

[0110] Due to the size of the input space represented by the left and right eye images, and the possibility of unexpected interference, such pupil detection algorithms may have flaws, such as features of false detection. Figure 3 The subsequent processing performed by the eye-tracking system 340 can mitigate these deficiencies and enable the computation of an improved combined gaze direction signal 350. This is particularly useful for computing combined gaze direction signals. Even small shifts in the detected pupil position can significantly affect the quality of the combined gaze direction signal.

[0111] The eye-tracking system 340 includes a variation calculator 344 that calculates the right-eye pupil variation of the right-eye pupil signal in multiple right-eye images and the left-eye pupil variation of the left-eye pupil signal in multiple left-eye images. As will be discussed in detail below, these variations can be correlated with the degree of change in the pupil signal over a period of time (e.g., represented by multiple images). Variation can be represented by statistical dispersion, such as one or more of the following: standard deviation, mean absolute deviation, interquartile range (IQR), range, mean absolute difference, median absolute deviation, mean absolute deviation, distance standard deviation, coefficient of variation, interquartile range coefficient of variation, relative mean difference, entropy, variance, and variance-to-mean ratio, as non-limiting examples.

[0112] The eye-tracking system 340 includes a weight calculator 346 that determines the weights of the right eye based on changes in the right pupil and the left eye based on changes in the left pupil. The weight calculator 346 can apply equations to the received changes to determine the associated weights, or it can use a database / lookup table to determine the associated weights. The eye-tracking system 340 can determine the right eye weights based on a nonlinear function that associates the right pupil changes with the right eye weights, and it can also determine the left eye weights based on a nonlinear function that associates the left pupil changes with the left eye weights. As a non-limiting example, the nonlinear function can be a sigmoid function or a modified linear unit function. This function can define an inverse relationship between the weights and the changes. In this way, as the value of the change increases, the value of the weight decreases.

[0113] In this example, the weight calculator 346 applies an S-shaped function to the received changes to determine the weights. Optionally, the weight calculator 346 normalizes the determined right-eye and left-eye weights so that the sum of the right-eye and left-eye weights is 1.

[0114] Figure 4 An example of an sigmoid function is shown graphically, which can be applied to received variations (such as standard deviation σ) as part of determining the appropriate weight w.

[0115] Return to Figure 3 The eye-tracking system 340 includes a gaze calculator 348 that calculates a combined gaze signal 350 for one or more right-eye images and one or more corresponding left-eye images. As described above, the combined gaze signal 350 may include a combined gaze origin signal and / or a combined gaze direction signal.

[0116] The gaze calculator 348 can determine the right-eye gaze signal based on the right-eye image. The gaze calculator 348 can also determine the left-eye gaze signal based on the corresponding left-eye image. The gaze calculator 348 can determine such gaze signals in any manner known in the art. For example, using the processing described in US 7,572,008 B2, or as described in Elias Daniel Guestrin and Moshe Eizenman's "General Theory of Remote Gaze Estimation Using the PupilCenter and Corneal Reflections" (IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, Vol. 53, No. 6, June 2006).

[0117] In some examples, gaze calculator 348 can determine the right eye gaze origin signal by detecting one or more flashes in the right eye image and calculating the right eye gaze origin signal based on the one or more flashes. Similarly, gaze calculator 348 can determine the left eye gaze origin signal by detecting one or more flashes in the left eye image and calculating the left eye gaze origin signal based on the one or more flashes. Detecting and processing flashes in this way is well known in the art.

[0118] The gaze calculator 348 can determine or receive the position of the right pupil detected in the right eye image; and calculate the right eye gaze direction signal based on the right eye gaze origin signal and the pupil position. Similarly, the gaze calculator 348 can determine or receive the position of the left pupil detected in the left eye image; and calculate the left eye gaze direction signal based on the left eye gaze origin signal and the pupil position.

[0119] The gaze calculator 348 can then calculate a combined gaze signal 350 based on a weighted sum of the right-eye gaze signal and the left-eye gaze signal, using right-eye weights and left-eye weights. In this example, the gaze calculator 348 calculates the right-eye gaze origin signal (O... R ), left eye fixation origin signal (O L ), right eye gaze direction signal (v R ) and left eye gaze direction signal (v L The gaze calculator 348 can then determine the combined gaze origin signal (O) as follows. C ) and combined gaze direction signal (v C ):

[0120] O C =w L *O L +w R *O R

[0121] v C =w L *v L +w R *v R

[0122] in:

[0123] w L It is the weight of the left eye; and

[0124] w R It's the weight of the right eye.

[0125] Advantageously, such a combined gaze signal 350 can have improved accuracy and / or precision compared to other solutions using left and right signals. Accuracy can be a measure of statistical bias (or systematic error) and can be expressed as the estimated gaze point closely matching the person's true gaze point. Precision can be a measure of statistical uncertainty, or the degree of convergence / clustering of the estimated gaze points. The position of the estimated gaze point relative to the person's true gaze point is independent of the precision of the combined gaze signal 350. It has been found that particularly good combined gaze signals 350 can be obtained by determining weights using variations in the pupil signal. In particular, the pupil signal can be considered as an intermediate signal in the determination of the combined gaze signal. Therefore, it has been found that calculating weights based on the (intermediate) pupil signal can produce particularly good improvements to the combined gaze signal. This results in improved combined gaze signals, at least in some applications, compared to calculating variations in the left and right gaze signals themselves.

[0126] Figure 5 A computer-implemented method according to an embodiment of the present disclosure is illustrated schematically.

[0127] At step 552, the method detects the pupil P in one or more images i = t, ..., tN of the left eye. L i =(x L i ,y L i ), where x L i Let y represent the position of the pupil of the left eye in image i, and y represent the position of the pupil of the left eye in image i. L i This represents the pupil radius of the left eye in image i. At step 554, the method detects the pupil P of the right eye in one or more images i = t, ..., tN. R i =(x R i ,y R i ), where x R i Let represent the position of the pupil of the right eye in image i, and y R i Let represent the pupil radius of the right eye in image i. As discussed above, these images can be provided as a series of images over time.

[0128] Pupil matrix P L i and P R iThis represents one way to store the history of pupil detection across multiple images. In some instances, a batch of N images can be processed at steps 552 and 554 to generate matrix P before the method proceeds to subsequent processing steps. L i and P R i In some examples, method steps 552 and 554 can each process the received image i one at a time, and in the matrix (P) of each image... L i ,P R i The data includes new data (x). L i ,y L i (x) R i ,y R i When the number of data entries in the matrix reaches a predetermined number (N), the method will use the data from the most recent image (x). L i ,y L i (x) R i ,y R i Replace the oldest data entry in the matrix. This is one way to limit the size of the dataset in the matrix so that it stores data representing the N most recent images.

[0129] In this way, the method can utilize a buffer that stores multiple pupil signals derived from the most recent right-eye image and the most recent left-eye image. The buffer can store the received pupil signals in a first-in, first-out (FIFO) manner, such that only the most recent pupil signal is stored in the buffer. In some examples, two separate buffers can be used—one for the pupil signal associated with the right-eye image and another for the pupil signal associated with the left-eye image. At step 558, the method can then calculate the right-eye pupil change of the right-eye image pupil signal stored in the buffer. Similarly, at step 556, the method can then calculate the left-eye pupil change of the left-eye image pupil signal stored in the buffer. As discussed above, this can involve processing the data (x...) stored in the buffer... L i ,y L i (x) R i ,y R i Perform statistical operations.

[0130] Alternatively, the method can utilize a buffer that stores multiple right-eye images and multiple left-eye images. The multiple right-eye images and multiple left-eye images can be buffer portions of corresponding sequences of right-eye and left-eye images. The buffer portions can be continuously updated in a first-in-first-out manner by new images received by the eye-tracking system. Steps 552 and 554 can then be used to detect pupils in the images stored in the buffer.

[0131] In the example using the buffer, the method can use a moving average filter to update the right eye pupil change and the left eye pupil change as the contents of the buffer are updated.

[0132] Also at step 556, the method is based on the pupil position (x) in the captured left-eye image. L i The weight w of the left eye is calculated using the variance of the variance. L (This can also be referred to as the confidence level). At step 558, the method is based on the pupil position (x) in the captured right-eye image. R i The weight w of the right eye is calculated using the variance of the variance of the right eye. R (This can also be referred to as the confidence level). In other embodiments, the method can be based on the pupil radius of each eye (y). L i ,y R i The weights w are calculated using the variance of the variance. L w R This could be the pupil position (x) L i ,x R i This method replaces or supplements the variance of the function. As discussed above, the method can calculate the weights by applying the inverse function to the appropriate variance value. Furthermore, the weights w can be... L and w R Normalize them so that their sum is 1.

[0133] At step 560, the method calculates the fixation origin (O) of the left eye. L ) and gaze direction (v L In step 562, the method calculates the gaze origin (O) of the right eye. R ) and gaze direction (v R ).

[0134] At step 564, the method generates a combined gaze origin (O). C As a weighted sum of the left and right gaze origins calculated for image i: O C =w L *O L +w R *OR At step 566, the method generates a combined gaze direction (v C As a weighted sum of the left and right gaze directions calculated for image i: v C =w L *v L +w R *v R .

[0135] At step 568, the method output includes the combined gaze origin (O). C ) and combined gaze direction (v C (A combination of gaze signals)

[0136] It should be understood that Figure 5 The various method steps shown do not need to be performed in the order shown. For example, the processing of signals related to the left and right eyes can be performed in parallel.

[0137] Figure 5 The method advantageously accounts for the possibility of unintended interference and defects in pupil detection by calculating the weights (confidence levels) of the gaze signals for each eye. This can be considered as anticipating suboptimal performance in pupil detection, combining it with the physical constraints of the human eye, and assuming that the movements between the user's left and right eyes are (at least to some extent) correlated. This can be achieved by measuring the positional distribution (variance) of the pupils detected for each eye and correlating them with each other (e.g., by normalizing the weights so that their sum is 1).

[0138] Figure 6 A module of an eye-tracking system according to another embodiment of the present disclosure is shown. In this embodiment, (in addition to the pupil signals discussed above) the eye-tracking system receives one or more eye feature signals 681, 682 and calculates associated feature changes for multiple left-eye and right-eye images. The eye-tracking system can then determine a right-eye weight 687 and a left-eye weight 686 based on the pupil changes (calculated using pupil signals 677, 678) and the feature changes (calculated using eye feature signals 681, 682). In this way, additional parameters (eye features) can be used to determine the weights. This can lead to even further improved combined gaze signals. Further details are provided below.

[0139] Figure 6A pupil position weighting module 671 is shown, which receives multiple left-eye pupil signals 677 and multiple right-eye pupil signals 678. These signals can be calculated by a pupil detector module (not shown) based on multiple left-eye and right-eye images, respectively. In this example, the left-eye pupil signal 677 is the left-eye pupil position signal, and the right-eye pupil signal 678 is the right-eye pupil position signal. The pupil position weighting module 671 calculates the right-eye pupil weight and the left-eye pupil weight in the same manner as discussed above, and these weights are normalized such that their sum is 1. That is, the right-eye pupil weight and the left-eye pupil weight are based on the calculated right-eye pupil change and the calculated left-eye pupil change. In this way, the pupil position weighting module 671 can determine a pair of pupil weights including a right-eye pupil weight 692 (based on the right-eye pupil change) and a left-eye pupil weight 691 (based on the left-eye pupil change). The right eye pupil weight 692 and the left eye pupil weight 691 are processed by the minimum value function block 674, which provides the minimum normalized pupil weight 683 to the weight determination block 676. The minimum normalized pupil weight 683 represents the minimum value of the pupil weight pair. For example, if the pupil weight pair consists of values ​​of 0.7 and 0.3, then the minimum normalized pupil weight 683 will be 0.3.

[0140] Figure 6 The system also includes a gaze origin weighting module 670, which receives multiple left-eye gaze origin signals 679 and multiple right-eye gaze origin signals 680. These signals can be calculated by a gaze calculator module (not shown) based on multiple left-eye and right-eye images, respectively. The gaze origin weighting module 670 calculates the right-eye gaze origin weight 694 and the left-eye gaze origin weight 693 in the same manner discussed above, and these weights are normalized so that their sum is 1. In this way, the gaze origin weighting module 670 can determine a gaze origin weight pair including the right-eye gaze origin weight 694 (based on changes in the right-eye gaze origin) and the left-eye gaze origin weight 693 (based on changes in the left-eye gaze origin). The right-eye gaze origin weight and the left-eye gaze origin weight are processed by a minimum value function block 673, and the minimum value function block 673 provides the minimum normalized gaze origin weight 684 to the weight determination block 676. The minimum normalized gaze origin weight of 684 represents the minimum value of the gaze origin weight pair.

[0141] Figure 6A more general eye feature weighting module 672 is also shown. Examples of various types of eye features are provided below. The eye feature weighting module 672 receives multiple left-eye feature signals 681 and multiple right-eye feature signals 682. The left-eye feature signals 681 and right-eye feature signals 682 can be provided by another module (not shown) of the eye-tracking system, which processes multiple left-eye images and multiple right-eye images to determine the left-eye feature signals 681 and right-eye feature signals 682, respectively.

[0142] The eye feature weighting module 672 can calculate right-eye feature changes based on right-eye feature signals 682 from multiple right-eye images. The eye feature weighting module 672 can also calculate corresponding left-eye feature changes based on left-eye feature signals 681 from multiple left-eye images. The eye feature weighting module 672 can then determine feature weight pairs including right-eye feature weight 696 (based on right-eye feature changes) and left-eye feature weight 695 (based on left-eye feature changes). In the same manner as described above regarding pupil changes, the eye feature weighting module 672 can apply a nonlinear function to the left-eye and right-eye feature changes, and then normalize the feature weight pairs so that their sum is 1, to determine the left-eye feature weight 695 and the right-eye feature weight 696.

[0143] The right eye feature weight 696 and the left eye feature weight 695 are processed by the minimum value function block 675, and the minimum value function block 675 provides the minimum normalized eye feature weight 685 to the weight determination block 676. The minimum normalized eye feature weight 685 represents the minimum value of the feature weight pair.

[0144] An eye-tracking system may include one or more eye feature weighting modules 672, each of which processes eye feature signals representing one or more of the following:

[0145] • Pupil position (in the same way as pupil position weight module 671);

[0146] • Pupil radius;

[0147] • Gaze origin (in the same way as the gaze origin weighting module. That is, gaze origin signals 679 and 680 are examples of eye feature signals);

[0148] • Corneal position;

[0149] • Iris position;

[0150] • Iris radius;

[0151] • The corner of the eye; and

[0152] • The number of matched emitter-flash pairs. An example system for matching emitter-flash pairs is described in WO 2019 / 185136 A1.

[0153] In some examples, eye feature signals can be provided by any image-based method / computer vision method or machine learning-based algorithm. For example, Diana Borza et al. (Sensors (Basel) 2016 July; 16(7):1105. Published online 2016 July 16. doi:10.3390 / s16071105) “Real-Time Detection and Measurement of Eye Features from Color Images” describes a variety of eye features, including some iris-related features.

[0154] Weight determination block 676 can then determine the minimum normalized weight among all normalized weights in the normalized pupil weight pair (provided in this example by pupil position weight module 671) and one or more normalized eye feature weight pairs (provided in this example by gaze origin weight module 671 and eye feature weight module 672). Weight determination block 676 then determines the left eye weight 686 and right eye weight 687 as the normalized pupil weight pair, normalized gaze origin weight pair, or normalized feature weight pair containing the minimum normalized weight.

[0155] As a numerical example, if: the value of the minimum normalized gaze origin weight 684 is 0.2; the value of the minimum normalized pupil weight 683 is 0.5; and the value of the minimum normalized eye feature weight 685 is 0.25; then the minimum normalized weight will be 0.2. The weight determination block 676 then determines the left eye weight 686 and the right eye weight 687 to have values ​​that match the right eye gaze origin weight and the left eye gaze origin weight (as determined by the gaze origin weight module 670), because it is this weight pair that has the minimum value of 0.2.

[0156] It should be understood that the same result can be obtained if the minimum value functions 673, 674, 675 and the weight determination block 676 all identify the maximum value of the signal they are processing, rather than the minimum value. This is because the sum of the normalized weight pairs will always be 1, and therefore identifying the pair with the highest weight will also identify the pair with the lowest weight.

[0157] In this way, the weight determination block 676 can determine the right eye weight 687 and the left eye weight 686 based on changes in the right eye pupil, changes in the left eye pupil, one or more right eye feature changes, and one or more left eye feature changes. The right eye weight 687 and the left eye weight 686 can then be determined by a fixation calculator (such as...). Figure 3 The gaze calculator in the program is used to determine combined gaze signals.

[0158] Figure 6 The module can advantageously identify which of several weight pairs is likely to provide the best combined gaze signal. This is because when the pupil signal / eye features computed from the left or right eye image are particularly poor, low weights can be used to compute the combined gaze signal from those left or right eye images. In this way, the influence of those poorer images on the computed combined gaze signal is reduced.

[0159] In one or more examples disclosed herein, if the right eye gaze origin signal is outside the expected right eye origin region, the eye-tracking system may set the right eye weight to zero. Similarly, if the left eye gaze origin signal is outside the expected left eye origin region, the eye-tracking system may set the left eye weight to zero. The expected right eye origin region and the expected left eye origin region can define two-dimensional or three-dimensional spatial regions where the expected gaze origin resides. These regions can be predefined based on known physical relationships between the image sensors in the eye-tracking system and the expected position of the human eye. These regions are particularly well-defined in head-mounted display systems. If the calculated gaze origin signal is outside its associated expected origin region, the eye-tracking system may interpret it as a signal derived from the associated eye image that is unreliable and therefore should not be used to calculate an indication of the combined gaze signal.

[0160] The systems disclosed in this article (including) Figure 3 and Figure 6 The systems described herein can be modified so that they do not necessarily need to detect the pupil in the eye image. Nor do they necessarily need to determine the corresponding pupil signal. Instead, the eye-tracking system can detect image features and determine associated eye feature signals for each right-eye image in multiple right-eye images and each left-eye image in multiple left-eye images. The eye-tracking system can then calculate the right-eye feature changes of the eye feature signals in the multiple right-eye images and the left-eye feature changes of the eye feature signals in the multiple left-eye images, and then determine the right-eye weights based on the right-eye feature changes and the left-eye weights based on the left-eye feature changes. The eye-tracking system can then use the right-eye weights and left-eye weights to calculate a combined gaze signal in the same manner as described above. The eye feature signal can be a pupil signal, a corneal signal, or any other suitable signal disclosed herein. For example, the image feature can be a pupil or a flash of light.

Claims

1. An eye-tracking system for calculating a user's eye gaze signal, the eye-tracking system comprising one or more controllers configured to: Receive multiple right-eye images from the user's right eye; Receive multiple left-eye images of the user's left eye, each left-eye image corresponding to a certain right-eye image among the multiple right-eye images; For each right-eye image in the plurality of right-eye images and each left-eye image in the plurality of left-eye images, the pupil is detected and the associated pupil signal is determined; Calculate the right eye pupil change of the pupil signal over a period of time in the multiple right eye images and the left eye pupil change of the pupil signal over a period of time in the multiple left eye images; The weights for the right eye and the left eye are determined based on the changes in the right pupil and the changes in the left pupil. as well as For one or more right-eye images and one or more corresponding left-eye images: At least one right-eye gaze signal is determined based on the right-eye image, and at least one left-eye gaze signal is determined based on the corresponding left-eye image; as well as The combined gaze signal is calculated by weighting the right-eye gaze signal and the left-eye gaze signal using the right-eye weight and the left-eye weight.

2. The eye-tracking system as described in claim 1, wherein, The pupil signal includes pupil position and / or pupil radius.

3. The eye-tracking system as described in claim 1, wherein: The right eye gaze signal includes the right eye gaze origin signal; The left-eye fixation signal includes the left-eye fixation origin signal; and The combined gaze signal includes the combined gaze origin signal.

4. The eye-tracking system as described in claim 3, wherein, The one or more controllers are further configured to: The right eye fixation origin signal is determined by detecting one or more flashes in the right eye image and calculating the right eye fixation origin signal based on the one or more flashes. as well as The left eye gaze origin signal is determined by detecting one or more flashes in the left eye image and calculating the left eye gaze origin signal based on the one or more flashes.

5. The eye-tracking system as described in claim 3, wherein, The one or more controllers are further configured to: If the right eye fixation origin signal is located outside the expected right eye origin region, then the right eye weight is set to zero; and If the left eye gaze origin signal is outside the expected left eye origin region, then the left eye weight is set to zero.

6. The eye-tracking system as described in claim 1, wherein: The right eye gaze signal includes the right eye gaze direction signal; The left-eye gaze signal includes the left-eye gaze direction signal; and The combined gaze signal includes a combined gaze direction signal.

7. The eye-tracking system as described in claim 6, wherein, The one or more controllers are further configured to: Determine the right eye gaze direction signal using the following steps: Detect one or more flashes of light in the right eye image; The right eye fixation origin signal is calculated based on the one or more flashes; Determine the position of the right eye pupil detected in the right eye image; as well as The right eye gaze direction signal is calculated based on the right eye gaze origin signal and the pupil position. and Determine the left eye's gaze direction signal using the following steps: Detect one or more flashes in the left eye image; The left eye fixation origin signal is calculated based on the one or more flashes; Determine the position of the left eye pupil detected in the left eye image; as well as The left eye gaze direction signal is calculated based on the left eye gaze origin signal and the pupil position.

8. The eye-tracking system as described in claim 1, wherein, The one or more controllers are configured to determine the right eye weights and the left eye weights such that they are normalized.

9. The eye-tracking system as described in claim 1, wherein, The one or more controllers are configured to: The right eye weight is determined based on a nonlinear function that correlates the right eye pupil change with the right eye weight. as well as The left eye weight is determined based on a nonlinear function that correlates the change in the left eye pupil with the left eye weight.

10. The eye-tracking system as claimed in claim 1, wherein, The one or more controllers are further configured to: For each right-eye image in the plurality of right-eye images and each left-eye image in the plurality of left-eye images, determine one or more further eye feature signals; For each of the one or more further eye feature signals: The changes in right-eye features are calculated based on the eye feature signals of the multiple right-eye images; as well as The corresponding changes in left-eye features are calculated based on the eye feature signals of the multiple left-eye images; as well as The right eye weight and the left eye weight are determined based on the changes in the right pupil, the changes in the left pupil, the changes in one or more right eye features, and the changes in one or more left eye features.

11. The eye-tracking system of claim 10, wherein, The one or more controllers are configured to determine the right eye weight and the left eye weight by the following steps: Determine a pair of pupil weights, including the right pupil weight based on the change in the right pupil and the left pupil weight based on the change in the left pupil; For each of the one or more further eye feature signals: Determine feature weight pairs including right eye feature weights based on the changes in right eye features and left eye feature weights based on the changes in left eye features; Normalize the pupil weight pairs and the one or more feature weight pairs; Determine the minimum normalized weight among all weights in the normalized pupil weight pair and the one or more normalized feature weight pairs; as well as The right eye weight and the left eye weight are determined as a normalized pupil weight pair or a normalized feature weight pair that includes the minimum normalized weight.

12. The eye-tracking system of claim 11, wherein, The one or more controllers are configured to: The right eye pupil weight is determined based on a nonlinear function that correlates the right eye pupil change with the right eye weight. The left eye pupil weight is determined based on a nonlinear function that correlates the change in the left eye pupil with the left eye weight. The one or more right-eye feature weights are determined based on a nonlinear function that associates the associated right-eye feature weights with the one or more right-eye feature weights. as well as The one or more left-eye feature weights are determined based on a nonlinear function that associates the associated left-eye feature weights with the one or more left-eye feature weights.

13. The eye-tracking system of claim 10, wherein, The one or more further eye feature signals represent one or more of the following: Pupil position; Pupil radius; Focus on the origin; Corneal position; Iris position; Iris radius; The corner of the eye; and Number of matched emitters-flash pairs.

14. The eye-tracking system of claim 10, wherein: The right eye gaze signal includes the right eye gaze origin signal; The left-eye fixation signal includes the left-eye fixation origin signal; and The combined gaze signal includes the combined gaze origin signal.

15. The eye-tracking system of claim 14, wherein, The one or more controllers are further configured to: If the right eye fixation origin signal is located outside the expected right eye origin region, then the right eye weight is set to zero; and If the left eye gaze origin signal is outside the expected left eye origin region, then the left eye weight is set to zero.

16. The eye-tracking system of claim 1, further comprising a buffer configured to store a plurality of pupil signals corresponding to the most recent right-eye image and the most recent left-eye image, and the eye-tracking system being configured to: Calculate the right eye pupil change of the pupil signal of the right eye image stored in the buffer; and Calculate the left eye pupil change of the pupil signal of the left eye image stored in the buffer.

17. The eye-tracking system of claim 16, wherein, The one or more controllers are configured to use a moving average filter to update the right eye pupil change and the left eye pupil change when the contents of the buffer are updated.

18. A head-mounted device comprising the eye-tracking system as claimed in claim 1.

19. A method for weighting right-eye and left-eye images to calculate a combined gaze signal, the method comprising: Receive multiple right-eye images from the user's right eye; Receive multiple left-eye images of the user's left eye, each left-eye image corresponding to a certain right-eye image among the multiple right-eye images; For each right-eye image in the plurality of right-eye images and each left-eye image in the plurality of left-eye images, the pupil is detected and the associated pupil signal is determined; Calculate the right eye pupil change of the pupil signal over a period of time in the multiple right eye images and the left eye pupil change of the pupil signal over a period of time in the multiple left eye images; The weight of the right eye is determined based on the change in the right pupil, and the weight of the left eye is determined based on the change in the left pupil; as well as For one or more right-eye images and one or more corresponding left-eye images: The right eye gaze signal is determined based on the right eye image, and the left eye gaze signal is determined based on the corresponding left eye image; as well as The combined gaze signal is calculated by weighting the right-eye gaze signal and the left-eye gaze signal using the right-eye weight and the left-eye weight.

20. One or more non-transitory computer-readable storage media storing computer-executable instructions that, when executed by a computing system, cause the computing system to perform the method of claim 19.

Images