Gaze defect compensation

Abstract

An eye-tracking system includes a controller configured to target one or both of a user's left and right eyes: receiving eye measurement data associated with the eye; determining the optical axis of the eye based on the eye measurement data; selecting one of a plurality of eye models based on the direction of the optical axis; and determining the gaze vector of the eye by applying the selected eye model to the eye measurement data.

Description

Technical Field

[0001] This disclosure generally relates to the field of eye tracking. In particular, this disclosure relates to eye tracking systems and methods for providing accurate eye tracking in the presence of health-related eye defects. Background Technology

[0002] In eye-tracking applications, digital images of the user's eyes are acquired and analyzed to estimate the user's gaze direction. The gaze direction estimation can be based on computer-based image analysis of the features of the imaged eye. A known example of eye-tracking involves using infrared light and an image sensor. The infrared light is directed toward the user's pupil, and the reflection of the light is captured by the image sensor.

[0003] Many eye-tracking systems estimate gaze direction based on the recognition of pupil position and flashes of light or corneal reflexes. Eye-tracking systems may include calibration sequences for defining an eye-tracking model that maps pupil position and flashes of light to gaze direction. However, such eye-tracking models may perform poorly for users with health-related eye defects.

[0004] Portable or wearable eye-tracking devices have been previously described. One such eye-tracking system is described in U.S. Patent No. 9,041,787 (the entire contents of which are incorporated herein by reference). A wearable eye-tracking device is described that uses a light source and an image sensor to determine the direction of gaze. Summary of the Invention

[0005] According to a first aspect of this disclosure, an eye-tracking system is provided, the eye-tracking system including a controller configured to target one or both of a user's left and right eyes:

[0006] Receive eye measurement data associated with the eye;

[0007] Determine the optical axis of the eye based on ocular measurement data; and

[0008] Selecting one of multiple eye models based on the direction of the optical axis; and

[0009] The eye gaze vector is determined by applying the selected eye model to eye measurement data.

[0010] Eye-tracking systems can advantageously implement gaze-dependent eye models, which apply different computational processes depending on the optical axis / direction of the user's gaze. By selecting one of multiple eye models based on the direction of the optical axis, eye-tracking systems can advantageously compensate for strabismus or other gaze-related defects in one or both eyes.

[0011] The controller can be configured to select one of multiple predefined eye models based on the direction of the optical axis.

[0012] The controller can be configured to select one of multiple eye models by: selecting a gaze offset value from multiple gaze offset values ​​based on the direction of the optical axis; and applying the selected gaze offset value to a reference eye model.

[0013] The controller can be configured to select one of multiple eye models by: selecting a gaze offset value from multiple gaze offset values ​​based on the direction of the optical axis; and applying the selected gaze offset value to the selected predetermined eye model.

[0014] The controller can be configured to determine multiple eye models by defining multiple eye modeling parameter sets for corresponding multiple predetermined eye models during the calibration process; a reference eye model and multiple gaze offset values ​​for application to the reference eye model; or defining multiple eye modeling parameter sets for corresponding multiple predetermined eye models and multiple gaze offset values ​​for application to one of the predetermined eye models.

[0015] The controller can be configured to determine multiple eye models by defining the following during the calibration process: defining multiple sets of eye modeling parameters for corresponding multiple predetermined eye models for multiple stimulus points or stimulus point regions; a reference eye model and multiple gaze offset values ​​for corresponding multiple stimulus points or stimulus point regions, wherein the gaze offset values ​​are applied to the reference eye model; or defining multiple sets of eye modeling parameters for corresponding multiple predetermined eye models for corresponding multiple stimulus point regions and multiple gaze offset values ​​for corresponding subsets of stimulus point regions, wherein the gaze offset values ​​are applied to one of the predetermined eye models.

[0016] The controller can be configured to display multiple stimulation points to the user one at a time, and to receive eye measurement data for each stimulation point.

[0017] Multiple stimulation points can include six or more stimulation points.

[0018] Each of the following: multiple sets of eye modeling parameters; and / or multiple gaze offset values, which may correspond to: each stimulus point; or a stimulus point region.

[0019] The controller can be configured to: determine the user's left eye gaze vector; determine the user's right eye gaze vector; determine the weight of each of the left and right eye gaze vectors based on the eye models selected for the respective left and right eyes; and apply the weights to the gaze vectors of each of the left and right eyes to provide a combined gaze vector.

[0020] The controller can be configured to determine weights based on the magnitude of the gaze offset value associated with the selected eye model.

[0021] The controller can be configured to determine weights based on the variation of the gaze offset value associated with the selected eye model relative to neighboring values ​​of multiple gaze offset values.

[0022] The controller can be configured to determine weights based on the variation of the values ​​of the eye modeling parameter set associated with the selected eye model relative to neighboring values ​​of multiple eye modeling parameter sets.

[0023] The controller can be configured to determine multiple weights during the calibration process, each weight corresponding to each of the following: multiple sets of eye modeling parameters; multiple gaze offset values; and / or multiple stimulus points.

[0024] Each weight can include a value from 0 to 1.

[0025] The controller can be further configured to: re-display multiple stimulus points to the user one at a time; and for each stimulus point displayed: receive eye measurement data; select an eye model corresponding to the stimulus point; calculate a gaze vector using the selected eye model and eye measurement data; calculate the difference between the calculated gaze vector and a known gaze vector corresponding to the stimulus point; and determine a weight based on the difference.

[0026] According to a second aspect of this disclosure, a head-mounted device is provided, including any eye-tracking system as disclosed herein.

[0027] According to a third aspect of this disclosure, an eye-tracking method is provided, the method comprising:

[0028] Receive eye measurement data associated with the user's eyes;

[0029] Determine the optical axis of the eye based on ocular measurement data; and

[0030] Selecting one of multiple eye models based on the direction of the optical axis; and

[0031] The eye gaze vector is determined by applying the selected eye model to eye measurement data.

[0032] According to a fourth aspect of this disclosure, a method for calibrating an eye-tracking system is provided, the method comprising:

[0033] This allows multiple stimulus points to be displayed to the user one at a time.

[0034] Receive eye measurement data for each stimulation point;

[0035] Sure:

[0036] Define multiple sets of eye modeling parameters for multiple predefined eye models;

[0037] A baseline eye model and multiple gaze offset values ​​applied to the baseline eye model; or

[0038] Define multiple sets of eye modeling parameters for multiple predefined eye models and multiple gaze offset values ​​for applying to one of the predefined eye models.

[0039] Each of the following:

[0040] Multiple eye modeling parameter sets; and / or

[0041] Multiple gaze offset values,

[0042] Corresponding to:

[0043] Each stimulation point; or

[0044] Stimulation point area.

[0045] According to a fifth aspect of this disclosure, one or more non-transitory computer-readable storage media are provided storing computer-executable instructions that, when executed by a computing system, cause the computing system to perform any of the methods disclosed herein.

[0046] A computer program may be provided that, when executed on a computer, causes the computer to configure any device (including the circuits, controllers, or apparatuses disclosed herein) or to perform any of the methods disclosed herein. The computer program may be a software implementation, and the computer may be considered any suitable hardware, including digital signal processors, microcontrollers, and implementations of read-only memory (ROM), erasable programmable read-only memory (EPROM), or electrically erasable programmable read-only memory (EEPROM), as a non-limiting example. The software may be an assembler.

[0047] A computer program may be provided on a computer-readable medium, which may be a physical computer-readable medium such as a disk or storage device, or may be embodied as a transient signal. Such a transient signal may be content downloaded from a network, including content downloaded from the Internet. One or more non-transitory computer-readable storage media may be provided, storing computer-executable instructions that, when executed by a computing system, cause the computing system to perform any of the methods disclosed herein. Attached Figure Description

[0048] One or more embodiments will now be described by way of example only with reference to the accompanying drawings, in which:

[0049] Figure 1 A schematic diagram of an eye-tracking system is shown, which can be used to capture image sequences that can be used by example embodiments of the present disclosure;

[0050] Figure 2 An example image of a pair of eyes is shown;

[0051] Figure 3 This schematically illustrates the difference between the eye's optical axis and the gaze vector; and

[0052] Figure 4 The measured gaze direction of the left eye of a user with severe strabismus in response to various stimuli is shown.

[0053] Figure 5 An eye-tracking system according to an embodiment of the present disclosure is shown;

[0054] Figures 6A to 6C The calibration routines for determining multiple eye models are shown;

[0055] Figure 7 The method according to an embodiment of this disclosure is illustrated schematically; and

[0056] Figure 8 A method for calibrating an eye-tracking system according to an embodiment of the present disclosure is illustrated schematically. Detailed Implementation

[0057] Figure 1A simplified view of an eye-tracking system 100 (also referred to as a gaze tracking system) in a head-mounted device, in the form of a virtual or augmented reality (VR or AR) device, VR or AR glasses, or anything related to it, such as an extended reality (XR) or mixed reality (MR) head-mounted device. System 100 includes an image sensor 120 (e.g., a camera) for capturing images of a user's eyes. The system may optionally include one or more illuminators 110 to 119 for illuminating the user's eyes; these illuminators may be, for example, light-emitting diodes emitting light in the infrared or near-infrared bands, and may be physically arranged in various configurations. Image sensor 120 may be, for example, any type of image sensor, such as a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor. The image sensor may be constructed from an integrated circuit comprising an array of pixel sensors, each pixel containing a photodetector and an active amplifier. The image sensor may be able to convert light into digital signals. In one or more examples, the image sensor may be an infrared image sensor or IR image sensor, an RGB sensor, an RGBW sensor, or an RGB or RGBW sensor with an IR filter.

[0058] The eye-tracking system 100 may include a circuitry or one or more controllers 125 for receiving and processing images captured by the image sensor 120, such as a receiver 126 and a processing circuitry 127. The circuitry 125 may be connected to the image sensor 120 and optionally one or more light sources 110-119, for example via a wired or wireless connection, and may be located in the same location as or at a distance from the image sensor 120 and the one or more light sources 110-119 (e.g., in different devices). In another example, the circuitry 125 may be disposed in one or more stacked layers beneath the photosensitive surface of the light sensor 120.

[0059] The eye-tracking system 100 may include a display (not shown) for presenting information and / or visual stimuli to the user. The display may include a VR display that presents images and substantially obstructs the user's real-world view, or an AR display that presents images that are perceived as being superimposed on the user's real-world view.

[0060] In such a system 100, the image sensor 120 for one eye is typically positioned away from the user's line of sight so as not to obstruct the display for that eye. This configuration can be achieved, for example, by means of a so-called thermal mirror, which reflects a portion of the light while allowing the rest of the light to pass through; for example, infrared light is reflected while visible light is allowed to pass through.

[0061] While in the example above, the image of the user's eyes is captured by a head-mounted image sensor 120, in other examples, the image may be captured by a non-head-mounted image sensor. Such a non-head-mounted system may be referred to as a remote system.

[0062] In eye-tracking systems, gaze signals can be calculated for each of the user's eyes (left and right). The quality of these gaze signals can be degraded by interference in the input image (such as image noise) and incorrect algorithmic behavior (such as incorrect predictions). The goal of eye-tracking systems is to provide gaze signals that are as good as possible in terms of accuracy (bias error) and precision (variance error). For many applications, providing only one gaze signal at each moment may be sufficient, rather than providing the left and right eye gazes separately. Furthermore, the left and right eye signals can be combined to provide a combined gaze signal. Such a gaze signal can be called a combined gaze signal.

[0063] Figure 2 This demonstrates the use of eye-tracking systems (such as...) Figure 1 This is a simplified example of an image 229 of a pair of eyes captured by a system. Image 229 can be considered to include a right-eye image 228 of a person's right eye and a left-eye image 234 of a person's left eye. In this example, the right-eye image 228 and the left-eye image 234 are two larger image portions of a person's two eyes. In other examples, a separate image sensor could be used to acquire the right-eye image 228 and the left-eye image 234.

[0064] The system can employ image processing (such as digital image processing) to extract features from images. For example, the system can identify the location of a pupil 230 in one or more images captured by an image sensor. The system can use a pupil detection process to determine the location of the pupil 230. The system can also identify corneal reflections 232 adjacent to the pupil 230. The system can estimate the corneal center or eye center based on the corneal reflections 232. For example, the system can match each individual corneal reflection 232 of each eye with a corresponding illuminator and determine the corneal center of each eye based on this matching.

[0065] Figure 3 It demonstrates the basis, such as through about Figure 2The described image processing describes an example calculation of the user's gaze vector 338 based on the position of the corneal reflection 332 and the pupil 330. As a first approximation, the eye-tracking system can determine the optical axis 336 of the user's eye as a vector passing through the center of the pupil 330 and the center of the cornea 333. However, due to limitations of the imaging system and / or anatomical differences in the shape and geometry of the human eye, the optical axis 336 is typically not aligned with the gaze vector 338. The gaze vector 338 passes through the center of the pupil 330 from the fovea 337 of the eye. The difference between the gaze vector 338 and the optical axis can depend on the pupillary plane offset (PPO) 340, which defines the distance from the plane of the pupil to the tangential plane at the anterior surface of the cornea / eye (in other words, the depth of the pupil in the eye). The PPO can vary depending on the user and may not be directly measurable by the eye-tracking system. The PPO 340 can act as a scalar multiplier component between the optical axis 336 and the gaze vector 338. Furthermore, as shown in the figure, the position of the central concave 337 may not be located on the optical axis, but is defined by a central concave offset (FO) 342 relative to the optical axis 336. FO 342 may include a vertical component (FO). y ) and horizontal component (FO) x However, we will refer to them simply as FO 342 in this paper. FO can serve as a scalar offset component between the optical axis 336 and the gaze vector 338.

[0066] Prior to eye tracking, the eye-tracking system can perform a calibration routine to estimate the PPO and FO for each individual user. During calibration, one or more stimulus points (e.g., five) at known locations / gaze angles can be displayed sequentially on the screen, and the user can be instructed to gaze at each point as it is displayed. The eye-tracking system can then compare the determined optical axis 336 with the true known gaze direction 338 (assuming the user is looking at the correct stimulus point) to determine the PPO 340 and FO 342. In this way, the eye-tracking system can determine a personal eye gaze tracking model (referred to herein as an eye model) for each user based on the determined constants PPO 340 and FO 342.

[0067] Next, the system can determine the gaze vector (also known as the line of sight) for each eye. In some examples, the gaze vector can be based on the gaze origin and gaze direction, which can be determined according to the corresponding flash and illuminator matching / corneal center, the determined pupil position, and calibration constants (as defined by the eye-tracking model). The gaze vectors of each eye can be combined to provide a combined gaze vector.

[0068] The above-described process for determining the gaze vector is an example of the pupil-central corneal reflex (PCCR) method. It should be understood that variations and alternatives to the above-described method for determining an eye model of an image of a user's eye are known in the art, and this disclosure is not limited to any particular method.

[0069] When users have health-related eye defects, the methods described above for determining a single calibrated eye model for each user may encounter performance issues. One such eye defect is strabismus, or "amblyopia." Strabismus affects 4% of the population and is characterized by a persistent inability to turn both eyes toward a focal point. This effect can vary from person to person, adding another layer of difficulty to eye tracking. This effect may manifest as: incorrect movement of one or both eyes in one or more directions; limited range of movement of one or both eyes; proportional movement of one or both eyes relative to the focal point; constant gaze deviation of one eye; different constant deviations between the two eyes depending on whether the user is looking left or right; cross-eyedness (too close convergence), non-constant deviations, and / or many other effects. In some examples, strabismus is non-uniform, with its effects varying significantly across the range of eye movement. In some examples, strabismus is inconsistent because the same stimulus may not necessarily elicit the same eye movement.

[0070] Figure 4 This diagram shows the measured fixation direction of the left eye of a user with severe strabismus for various stimulus points. Each quadrant's stimulus point is represented as a circle 444, corresponding to fixed angular deviations relative to the eye center in eight different directions. As marked, the angular deviation increases in each quadrant. The measured fixation direction is represented as point 446. For a stimulus point of 10 degrees (upper left quadrant), the user's left eye exhibits leftward tilt by moving only within a small area to the left of the center. However, for a stimulus point of 20 degrees (upper right quadrant), a larger movement was recorded, but with a rightward tilt towards the right of the center. Generally, the left eye movement is independent of where the user intends to look. Furthermore, the left eye does not move beyond approximately 15 degrees.

[0071] Due to the nature of the variations in strabismus, the calibrated eye model of the individuals described above has performance issues for users with strabismus. One approach to address strabismus is to have the user disable tracking of the affected eye. However, this approach severely limits performance in the unaffected gaze direction, as combined gaze performance is superior to monocular performance in most cases. This approach also cannot compensate for users with binocular strabismus.

[0072] The systems and methods disclosed below can provide eye tracking that overcomes the limitations described above and can compensate for strabismus or other individual gaze deficits. The disclosed systems and methods can advantageously implement gaze-dependent eye models that apply different computational processes depending on the optical axis / direction of the user's gaze.

[0073] Figure 5 An eye-tracking system 550 according to an embodiment of this disclosure is illustrated. The eye-tracking system 550 includes a controller 552. The controller 552 is configured to receive eye measurement data associated with a user's eyes (for one or both eyes). The eye measurement data may include images of one or both eyes, or may be derived from such images, as will be discussed below. The controller 552 includes an optical axis detector 553 configured to receive the eye measurement data and determine the optical axis of the eye based on the eye measurement data. The controller 552 also includes a gaze estimation module 554 configured to receive the optical axis from the optical axis detector 553 and select one of a plurality of eye models based on the direction of the optical axis. The gaze estimation module 554 is further configured to receive the eye measurement data and determine the gaze vector (gaze) of the eye by applying the selected eye model to the eye measurement data.

[0074] Eye measurement data may include one or more parameters determined based on an image of the user's eye. For example, eye measurement data may include the location of the corneal reflection, the calculated corneal center, the determined pupil position, etc., as described above. In other examples, eye measurement data may include a raw image of the eye received from an image sensor. In such an example, controller 552 is further configured to determine the corneal center and pupil position based on the raw image, as described above and known in the art.

[0075] By selecting one of multiple eye models based on the direction of the optical axis, eye-tracking systems can advantageously compensate for strabismus in one or both eyes. In particular, eye-tracking systems can employ different eye models for areas of strabismus and areas of normal eye behavior (and thus perform different gaze calculation processes).

[0076] The method described above uses several (e.g., 5) known stimulus points to calibrate a single eye model using eye model parameters (e.g., PPO, FO). As a result, the same eye model is used for all gaze angles. Figure 5 The eye-tracking system 550 can select one of several eye models, which changes the mapping of its pupil center and corneal center to the gaze vector depending on the direction the user is looking (as determined by the detected optical axis). This can advantageously compensate for the effects of strabismus in one or more directions, as well as in one or both eyes.

[0077] Multiple eye models can be based on calibration routines (e.g.) Figures 6A to 6C The calibration routine shown is determined by the specific calibration procedure. The calibration routine may include presenting a series of stimulus points 660 (also called calibration points) to the user 662 on a display. For remote eye-tracking systems, the user can be positioned at a fixed distance from a display screen, such as a computer screen. This fixed distance can range from 50cm to 200cm. For head-mounted eye-tracking systems, the stimulus points 660 may be presented at a distance of approximately 2 meters in virtual space. In other remote or head-mounted examples, this distance can be greater than 2 meters and can reach infinity in virtual space. In this example, the stimulus points 660 are arranged in an 8×8 square grid. However, any number of stimulus points 660 can be displayed. The number of stimulus points can be greater than 5 (the number of points that can be used in a single model calibration). A large number of points is desirable for better characterizing the user's strabismus; however, a smaller number of points is desirable to avoid time-consuming calibration routines. The stimulus points 660 can be arranged in a 4×4 grid, a 4×5 grid, a 5×5 grid, a 5×6 grid, or a 6×6 grid. In other examples, the stimulus points 660 can be positioned as a non-grid pattern. The size of the stimulus point pattern and the distance between the user 662 and the display can be selected such that the stimulus pattern forms an angle of approximately 30 to 60 degrees with the user's eyes in both the horizontal and vertical directions. The stimulus points 662 can be displayed to the user 662 sequentially (one at a time). Each stimulus point 660 can be displayed for 0.5 to 1.0 seconds. The stimulus points 660 can be displayed in any order.

[0078] In the first example, the multiple eye models may include multiple predetermined eye models determined by the controller 552 as part of the calibration process. These eye models can be considered predetermined because they are determined during the calibration process before the eye-tracking system uses “real-time” (i.e., uncalibrated) eye measurement data. As an example, during calibration, the controller 552 may determine a separate eye model for each stimulus point 660 based on eye measurement data recorded when the user is instructed to look at each corresponding stimulus point 660. As another example, the controller 552 may determine separate eye models for multiple stimulus point 660 regions. For example, as... Figure 6CAs shown, controller 552 can determine four separate eye models, each corresponding to each quadrant 664a, 664b, 664c, 664d (upper left quadrant, upper right quadrant, lower left quadrant, lower right quadrant, collectively referred to as quadrant 664) as viewed by the user. Each individual (predetermined) eye model (determined during the calibration process) can have its own separate set of model parameters (PPO, FO, etc.), which are calculated during calibration for each stimulus point or quadrant / region as appropriate. In this way, the controller is configured to determine multiple sets of eye modeling parameters that define multiple predetermined eye models corresponding to each other during the calibration process.

[0079] During use of the eye-tracking system 550 (after calibration), the optical axis detector 553 can determine the direction of the eye's optical axis in a normal manner (e.g., based on the PCCR method) to determine the user's approximate gaze angle. In this way, the controller 552 can determine which stimulus point 660 or stimulus point region (e.g., which quadrants 664a to 664d) is associated with the determined optical axis. The gaze estimation model 554 can then select an eye model corresponding to the determined region / quadrants 664a to 664d / stimulus point 660. In this way, the controller 552 can use eye models with different eye model parameters (PPO, FO, etc.) based on the angle of the determined optical axis to address strabismus effects or similar eye gaze deficits. If the direction of the optical axis lies between two or more stimulus points 660, the controller 552 can interpolate the model parameters of the eye model corresponding to the nearest stimulus point 660. The interpolation can depend on the distance to each stimulus point 660, such that the closer stimulus point 660 has a larger weight in the combination.

[0080] In the second example, the multiple eye models may include a reference eye model and multiple gaze offset values. As part of the calibration process, controller 552 can determine the reference eye model and the multiple gaze offset values. Controller 552 can determine the reference eye model using conventional methods based on one or more stimulus points 660, for example, based on five stimulus points 660 or all stimulus points 660. Each of the multiple gaze offset values ​​may correspond to a corresponding stimulus point 660 or a region of stimulus point 660. Controller 552 can determine each gaze offset value as an error value between a reference gaze angle calculated using the reference eye model and the expected gaze angle of the corresponding one or more stimulus points 660. Each gaze offset value may include an error value in the vertical direction (y-axis) and an error value in the horizontal direction (x-axis). The multiple gaze offset values ​​may be referred to as an error value matrix (EVM). The size of the EVM may correspond to the size of the stimulus point grid or the number of different stimulus point regions. In this way, the controller is configured to determine the multiple gaze offset values ​​to be applied to the reference eye model during the calibration process.

[0081] During use, after calibration, the optical axis detector 553 can determine the direction of the optical axis. Then, the gaze estimation module 552 can determine the stimulus point 660 (or stimulus point region) closest to the determined optical axis direction. Next, the gaze estimation module 552 can select one of multiple eye models by selecting the gaze offset value corresponding to the stimulus point 660 closest to the optical axis direction and applying the gaze offset value to a reference model. In this way, the controller 552 can use a single eye model with the same eye model parameters (PPO, FO, etc.) and apply a direction-dependent gaze angle offset to address strabismus or similar defects. If the direction of the optical axis lies between two or more stimulus points 660 (or stimulus point regions), then the controller 552 can interpolate the error vector applied to the calculated reference gaze angle based on the gaze offset value from the nearest stimulus point 660. The interpolation may depend on the distance to each stimulus point 660 or stimulus point region.

[0082] The second example maintains a single value for each of the PPO and FO in the baseline model, providing a single representation of the eye's true physical parameters. This advantageously avoids an unusually large number of PPOs or FOs that could lead to undesirable noise sensitivity. By providing a single eye model along with multiple scalar offsets, this method also advantageously offers a simpler approach than the first example.

[0083] The third example may include a combination of the first and second examples. In other words, the controller 552 may determine multiple predetermined eye models and multiple gaze offset values ​​during the calibration process. Each predetermined eye model may correspond to a different calibration / stimulus point region; for example, there may be a separate eye model for each of the four quadrants 664. The controller 552 may determine each gaze offset value for each individual stimulus point in that region as an error value between the gaze angle calculated using the appropriate predetermined eye model and the known gaze angle of the corresponding stimulus point 660. For example, the controller may determine a separate gaze offset value for each stimulus point in a region / quadrant relative to the predetermined eye model of that region / quadrant. In this way, the controller is configured to determine, during the calibration process, multiple sets of eye modeling parameters defining the multiple predetermined eye models corresponding to each other; and multiple gaze offset values ​​for application to one of the multiple predetermined eye models.

[0084] During subsequent use, the gaze estimation module 554 can select one of a plurality of eye models by: selecting one of a plurality of predetermined eye models based on the direction of the optical axis; selecting one of a plurality of gaze offset values ​​based on the direction of the optical axis; and applying the selected gaze offset value to the selected predetermined eye model.

[0085] In some cases of strabismus, eye behavior may be inconsistent for one or more gaze angles. In some examples, the eye-tracking system 550 can be configured to detect such inconsistent eye behavior so that eye tracking can be disabled for the affected gaze angle of the affected eye.

[0086] It should be understood that the controller can perform the aforementioned calibration process, model selection, and gaze vector determination for both the user's left and right eyes. The controller 552 can combine the resulting eye vectors for each eye to provide a combined gaze vector. In some examples, when providing a combined gaze vector, the controller 552 can apply weights to the left-eye gaze vector and the right-eye gaze vector. The controller can determine the weights for each eye based on values ​​associated with the selected eye model for the respective eye. For example, for the second and third examples described above, the weights can be based on the magnitude of the gaze offset value, which represents the error of the user's gaze relative to a reference or predetermined model. The weights can also be based on the variation of the gaze offset value relative to adjacent gaze offset values ​​(associated with adjacent stimulus points). A high variation in the gaze offset value for a specific region (stimulus point 660) can indicate that the eye gaze vector is unstable in that region and can therefore be an indication of strabismus or other gaze-related defects. For the first and third examples, the weights can be based on the variation of the values ​​of the eye modeling parameter set (e.g., PPO, FO) relative to adjacent values ​​of multiple eye modeling parameter sets (associated with adjacent stimulus points / stimulus point regions). The weights can include values ​​from 0 to 1. The sum of these weights (one for each eye) can be equal to 1. The weights can depend on the magnitude of the gaze offset value or any variation in it. For regions with particularly high gaze offset values ​​(above a threshold) or particularly high variations in the gaze offset value and / or the set of eye modeling parameters (above a threshold variation), controller 552 can set the corresponding weight to zero to disable eye gaze tracking in that region. In some examples, both eyes may experience the same poor performance in certain regions, and the controller can set the corresponding weights for both eyes to zero, thus completely disabling eye tracking. In this way, eye tracking system 550 can advantageously disable eye tracking instead of providing inaccurate gaze values ​​in regions of severe strabismus. In other examples, controller 552 can set the weight to zero only for the eye with the highest error / variation.

[0087] In some examples, controller 552 may determine multiple weights (also referred to as paired weight values ​​– one weight value per eye) during the calibration process. Each of the multiple weights may correspond to: each of multiple predetermined model / eye modeling parameter sets (Examples 1 and 3); each of multiple gaze offset values ​​(Examples 2 and 3); and / or each stimulus point 660 (Examples 1, 2, and 3).

[0088] Multiple weights can be referred to as the Eye Weight Matrix (EWM), the size of which corresponds to the number of different eye models (which can be implemented as multiple different eye models and / or multiple different gaze offset values), and the size of the eye weight matrix can be the same as the size of the stimulus point grid.

[0089] In some examples, as part of a two-stage calibration process, controller 552 may determine multiple weight values. During the first stage, multiple stimulus points 660 may be displayed to user 662 one at a time, and controller 552 may determine multiple sets of eye modeling parameters (for each eye); and / or multiple gaze offset values ​​corresponding to each stimulus point or stimulus point region, as described above with respect to Examples 1 through 3. During the second stage, multiple stimulus points 660 may be re-displayed to user 662 one at a time. When each stimulus point is displayed, controller 552 may receive eye measurement data for each eye, select an eye model corresponding to the stimulus point, and use the selected eye model and eye measurement data to determine a calculated gaze vector. Controller 552 may then compare the calculated eye vector for each eye with a known eye vector corresponding to the stimulus point and determine the weight of each eye based on the magnitude of the difference between the calculated eye vector and the known eye vector. In this way, selected eye models that cannot accurately calculate gaze vectors may be given lower weights. This can be useful when subsequently combining the gaze vectors of the left and right eyes, so that the less accurate gaze vectors expected to have a smaller impact on the combined gaze signal. As mentioned above, when the user's gaze corresponds to the direction of the relevant stimulus point, one or more weight values ​​for one or both eyes can be set to zero to disable gaze tracking for the relevant eye during subsequent use. If the difference between the calculated eye vector and the known eye vector is greater than an acceptable difference threshold, then the weight values ​​can be set to zero.

[0090] In a similar example, the first stage of the calibration process may include controller 552 (for each eye) determining a first plurality of gaze offset values ​​for a plurality of stimulus points 660 to be applied to a baseline model (Example 2) or one of a plurality of predetermined models (Example 3). The second stage may include controller 552 (for each eye) determining a second plurality of gaze offset values ​​for the same plurality of stimulus points to be applied to the same baseline model or the same predetermined model among the plurality of predetermined models. The difference between each of the first plurality of gaze offset values ​​and its corresponding counterpart in the second plurality of gaze offset values ​​represents the consistency of the user's eye gaze toward that stimulus point / stimulus point region. Then, controller 552 may determine multiple weights (for each eye) based on the differences between corresponding values ​​of the gaze offset values ​​in the first plurality of gaze offset values ​​and the second plurality of gaze offset values.

[0091] Figure 7 An eye-tracking method according to embodiments of the present disclosure is illustrated. This method can provide eye tracking that advantageously addresses eye fixation defects such as strabismus.

[0092] Step 770 includes receiving eye measurement data associated with the user's eye. Step 772 includes determining the optical axis of the eye based on the eye measurement data. Step 774 includes selecting one of a plurality of eye models based on the direction of the optical axis. Step 776 includes determining the gaze vector of the eye by applying the selected eye model to the eye measurement data.

[0093] Figure 8 A method for calibrating an eye-tracking system according to embodiments of the present disclosure is illustrated. This method enables an eye-tracking system that advantageously addresses eye fixation defects such as strabismus.

[0094] Step 880 includes displaying the multiple stimulus points to the user one at a time. Step 882 includes receiving eye measurement data for each stimulus point. Step 884 includes determining: multiple sets of eye modeling parameters defining multiple predetermined eye models; a reference eye model and multiple gaze offset values ​​applied to the reference eye model; or defining multiple sets of eye modeling parameters defining multiple predetermined eye models and multiple gaze offset values ​​applied to one of the multiple predetermined eye models. Each of the following: multiple sets of eye modeling parameters; and / or multiple gaze offset values, corresponding to: each stimulus point; or a stimulus point region.

[0095] The disclosed eye-tracking system can provide strabismus compensation modes by implementing the above calibration process, model definition, and model selection. Users with gaze deficits can choose a strabismus compensation mode instead of the traditional single-model individual calibration.

[0096] The disclosed systems and methods provide users with functional eye tracking solutions that address the effects of their individual strabismus.

[0097] Throughout this specification, descriptive terms relating to relative orientation and position, such as “horizontal,” “vertical,” “top,” “bottom,” and “side,” are used in the sense of the orientation of the device / equipment as presented in the accompanying drawings. However, these descriptive terms are not intended to limit in any way the intended use of the described or claimed invention.

[0098] It should be understood that, depending on the context, any reference to “near,” “before,” “shortly before,” “after,” “shortly after,” “above,” or “below,” etc., may refer to the parameter in question being less than or greater than a threshold, or between two thresholds.

Claims

1. An eye-tracking system comprising a controller, the controller including an optical axis detector and a gaze estimation module, wherein the controller is configured to: The optical axis detector is used to receive eye measurement data associated with the user's eyes; The optical axis detector is used to determine the optical axis of the eye based on the eye measurement data; The gaze estimation module is used to select one of multiple eye models based on the direction of the optical axis, thereby compensating for gaze defects characterized by eye misalignment. as well as The gaze estimation module is used to determine the gaze vector of the eye by applying the selected eye model to the eye measurement data.

2. The eye-tracking system as described in claim 1, wherein, The controller is configured to select one of a plurality of predetermined eye models based on the direction of the optical axis.

3. The eye-tracking system as described in claim 1, wherein, The controller is configured to select one of a plurality of eye models in the following manner: Based on the direction of the optical axis, a gaze offset value is selected from multiple gaze offset values; as well as The selected gaze offset value is applied to the baseline eye model.

4. The eye-tracking system as described in claim 2, wherein, The controller is configured to select one of a plurality of eye models in the following manner: Based on the direction of the optical axis, a gaze offset value is selected from multiple gaze offset values; as well as The selected gaze offset value is applied to the selected predefined eye model.

5. The eye-tracking system as described in claim 1, wherein, The controller is configured to determine the plurality of eye models by determining at least one of the following during the calibration process: Define multiple sets of eye modeling parameters for multiple predefined eye models; A reference eye model and multiple gaze offset values ​​applied to the reference eye model; or Define a plurality of eye modeling parameter sets for the corresponding plurality of predetermined eye models and a plurality of gaze offset values ​​for applying to one of the plurality of predetermined eye models, wherein the plurality of eye modeling parameter sets or the plurality of gaze offset values ​​correspond to a plurality of different regions of the stimulus point, and each region is associated with a different quadrant of the user’s field of vision.

6. The eye-tracking system of claim 5, wherein, The controller is configured to: This allows multiple stimuli to be displayed to the user one at a time; and Receive eye measurement data for each stimulation point.

7. The eye-tracking system of claim 6, wherein, The plurality of stimulation points includes six or more stimulation points.

8. The eye-tracking system of claim 6, wherein, Each of the plurality of eye modeling parameter sets and the plurality of gaze offset values ​​corresponds to: Each stimulation point; or The area of ​​the stimulation point.

9. The eye-tracking system of claim 5, wherein, The controller is configured to: Determine the user's left eye gaze vector; Determine the right eye gaze vector of the user; The weights of each of the left-eye gaze vector and the right-eye gaze vector are determined based on the left-eye model selected for the left eye and the right-eye model selected for the right eye, wherein the weights are based on the magnitude of the gaze offset value associated with the selected eye model or the variation in the set of eye modeling parameters. as well as Weights for the left eye gaze vector are applied to the left eye gaze vector, and weights for the right eye gaze vector are applied to the right eye gaze vector to provide a combined gaze vector.

10. The eye-tracking system of claim 9, wherein, The controller is configured to determine the weights for each of the left-eye gaze vector and the right-eye gaze vector based on the magnitude of the gaze offset value associated with the selected eye model.

11. The eye-tracking system of claim 10, wherein, The controller is configured to determine the weights based on the variation of the gaze offset value associated with the selected eye model relative to adjacent values ​​of the plurality of gaze offset values.

12. The eye-tracking system of claim 9, wherein, The controller is configured to determine the weights based on the changes in the values ​​of the eye modeling parameter set associated with the selected eye model relative to adjacent values ​​of the plurality of eye modeling parameter sets.

13. The eye-tracking system of claim 9, wherein, The controller is configured to determine a plurality of weights during the calibration process, each of the plurality of weights corresponding to at least one of the following: The multiple sets of eye modeling parameters, The multiple gaze offset values, or The aforementioned multiple stimulation points.

14. The eye-tracking system of claim 9, wherein, Each weight includes a value from 0 to 1.

15. The eye-tracking system of claim 9, wherein, The controller is further configured to: The plurality of stimulation points are then redisplayed to the user one at a time; and For each of the plurality of stimulus points that are re-displayed: Receive eye measurement data; Select the eye model corresponding to the stimulation point; The gaze vector is calculated using the selected eye model and the eye measurement data; Calculate the difference between the calculated gaze vector and the known gaze vector corresponding to the stimulus point; as well as Weights are determined based on the differences.

16. A head-mounted device comprising an eye-tracking system as described in any of the preceding claims.

17. An eye-tracking method, the method comprising: Use an optical axis detector to receive eye measurement data associated with the user's eyes; The optical axis detector is used to determine the optical axis of the eye based on the eye measurement data; as well as The gaze estimation module is used to select one of multiple eye models based on the direction of the optical axis, thereby compensating for gaze defects characterized by eye misalignment; as well as The gaze estimation module is used to determine the gaze vector of the eye by applying the selected eye model to the eye measurement data.

18. A method for calibrating an eye-tracking system, the method comprising: The controller displays multiple stimulation points to the user's eyes one at a time. The optical axis detector is used to receive eye measurement data for each stimulation point; Based on the eye measurement data, determine at least one of the following: Define multiple sets of eye modeling parameters for multiple predefined eye models; A reference eye model and multiple gaze offset values ​​applied to the reference eye model; or Define the multiple eye modeling parameter sets for the corresponding multiple predetermined eye models, and multiple gaze offset values ​​for applying to one of the multiple predetermined eye models. in, The multiple eye modeling parameter sets and / or the multiple gaze offset values ​​correspond to: Each of the plurality of stimulation points or a region of the plurality of stimulation points, wherein the calibration is performed to define a strabismus compensation mode for the eye-tracking system.

19. One or more non-transitory computer-readable storage media storing computer-executable instructions that, when executed by a computing system, cause the computing system to perform the method of claim 17 or claim 18.

Images