Gaze tracking using a map of pupil center position

Abstract

An image of an eye (100) is captured by a camera (302). For each of the images, gaze data (402) is obtained, and the position of a pupil center (103) in the image is estimated (403). The gaze data indicates a gaze point (111) and / or a gaze direction (107) of the eye (100) when the image was captured. A mapping (510, 520, 530) is calibrated (404) using the obtained gaze data and the estimated position of the pupil center. The mapping maps the position of the pupil center in an image (501) captured by the camera to a gaze point (511, 521) on a surface (512, 522), or to a gaze direction (531). Another image of the eye is captured by the camera. The position of the pupil center in the other image is estimated (406). Gaze tracking is performed (407) using the calibrated mapping and the estimated position of the pupil center in the other image. These steps may be performed, for example, on an HMD (1501).

Description

[0001] This application is a divisional application of the invention patent application with application date of October 30, 2019, application number 201911044668.5, and invention name “Gaze tracking using mapping of pupil center position”. Technical Field

[0002] The present disclosure relates generally to gaze tracking. Background Art

[0003] Different technologies have been developed for monitoring which direction (or which point on the display) a user is looking at. This is often referred to as gaze tracking or eye tracking. One or more cameras are typically used to capture an image of the user's eyes. Image processing is used to detect various features of the eye in the image (e.g., the center of the pupil and the reflection of the illuminator at the cornea). These detected features are then used to estimate where the user is looking. For example, gaze tracking can be used to allow a user to control a computer (or some other type of system) through eye movements, or gaze tracking can be used to enhance the virtual reality (VR) experience in a head-mounted device such as a head-mounted display (HMD). The accuracy and precision of gaze tracking may be important for providing a good user experience. Noise in the image may affect gaze tracking performance. Another issue that may affect gaze tracking performance is that it may be difficult to detect features in the image captured by (multiple) cameras at certain gaze angles. Features such as the center of the pupil and / or corneal reflection (also known as glint) may be difficult to accurately detect in certain images, for example. This may have a negative impact on gaze tracking performance. One way to solve this problem is to equip the gaze tracking system with additional cameras and illuminators arranged at appropriate positions relative to the user's head to ensure that the cameras and illuminators can be used for as many gaze angles as possible. However, such additional equipment may increase the cost of the gaze tracking system and may occupy space that should be occupied by other components other than the gaze tracking device. It is desirable to provide new methods to solve one or more of the above problems. Summary of the invention

[0004] A method, a system and a computer readable storage medium having the features defined in the independent claims are provided to solve one or more of the above mentioned problems. Preferred embodiments are defined in the dependent claims.

[0005] Therefore, a first aspect provides an embodiment of a gaze tracking method. The gaze tracking method comprises: obtaining a set of eye images captured by a camera. For each image from the set of images, the gaze tracking method comprises: obtaining gaze data indicating a point at which the eye was looking when the image was captured and / or a direction in which the eye was looking when the image was captured, and estimating the position of the center of the pupil of the eye in the image. The gaze tracking method comprises: calibrating a mapping using the obtained gaze data and the estimated position of the center of the pupil. The mapping is suitable for mapping the position of the center of the pupil of the eye in the image captured by the camera to an eye gaze point on a surface, or to certain gaze directions of the eye. The gaze tracking method comprises: obtaining another eye image captured by the camera; estimating the position of the center of the pupil in the other image; and performing gaze tracking using the calibrated mapping and the estimated position of the center of the pupil in the other image.

[0006] Gaze tracking based on pupil center corneal reflection (PCCR) has been widely used for remote gaze tracking and gaze tracking of head-mounted devices such as virtual reality (VR) head-mounted devices or augmented reality (AR) head-mounted devices. In gaze tracking of head-mounted devices, PCCR-based methods usually provide lower accuracy and precision at large gaze angles than at small gaze angles, mainly due to the noise generated when detecting corneal reflections in images. In gaze tracking of head-mounted devices, the head-mounted device and the camera mounted thereon are usually completely stationary, and the movement of the pupil reflects the movement of the eyeball and the gaze direction. Therefore, the position of the pupil center is useful for estimating gaze. In other words, as described above with respect to the embodiment of the method according to the first aspect, gaze tracking can be performed using a calibrated mapping and an estimated position of the pupil center in the image. For example, such gaze tracking can be used as an alternative or supplement to PCCR-based gaze tracking to improve gaze tracking performance (such as accuracy and / or precision) for at least some gaze angles or in cases where PCCR-based gaze tracking is not reliable. Some of the problems mentioned above may occur in remote gaze tracking, and therefore the present invention is also applicable to remote gaze tracking.

[0007] A second aspect provides an embodiment of a gaze tracking system, the gaze tracking system comprising a processing circuit system (or one or more processors) configured to obtain a set of eye images captured by a camera. For each image from the set of images, the processing circuit system is configured to: obtain gaze data indicating the point at which the eye was looking when the image was captured and / or the direction in which the eye was looking when the image was captured, and estimate the position of the center of the pupil of the eye in the image. The processing circuit system is configured to: calibrate a mapping using the obtained gaze data and the estimated position of the center of the pupil. The mapping is suitable for mapping the position of the center of the pupil of the eye in the image captured by the camera to the eye gaze point on a surface, or to the gaze direction of the eye. The processing circuit system is configured to: obtain another eye image captured by the camera; estimate the position of the center of the pupil in the other image; and perform gaze tracking using the calibrated mapping and the estimated position of the center of the pupil in the other image.

[0008] The processing circuit system (or one or more processors) can, for example, be configured to perform the method defined in any embodiment of the first aspect as disclosed herein (in other words, in the claims, the inventive content or the detailed description). The system can, for example, include one or more non-transitory computer-readable storage media (or one or more memories) storing instructions, which, when executed by the processing circuit system (or one or more processors), cause the gaze tracking system to perform the method defined in any embodiment of the first aspect as disclosed herein.

[0009] The effects and / or advantages presented in the present disclosure for the embodiments of the method according to the first aspect may also be applicable to the corresponding embodiments of the system according to the second aspect.

[0010] A third aspect provides an embodiment of a non-transitory computer-readable storage medium storing instructions that, when executed by a gaze tracking system, cause the gaze tracking system to:

[0011] a) obtaining a set of images of the eye captured by a camera;

[0012] b) For each image in the set of images:

[0013] - obtaining gaze data indicating the point at which the eye was looking when the image was captured and / or the direction in which the eye was looking when the image was captured, and

[0014] - estimating the position of the pupil centre of the eye in the image;

[0015] c) using the obtained gaze data and the estimated position of the pupil centre to calibrate a mapping, wherein the mapping is adapted to map the position of the pupil centre of the eye in an image captured by the camera to:

[0016] - the point where the eyes are looking on a surface, or

[0017] - direction of eye gaze;

[0018] d) obtaining another image of the eye captured by the camera;

[0019] e) estimating the position of the pupil centre in the further image; and

[0020] f) performing gaze tracking using the calibrated map and the estimated position of the pupil center in the further image.

[0021] The non-transitory computer-readable storage medium may, for example, store instructions that, when executed by a gaze tracking system (or by a processing circuit system included in the gaze tracking system), cause the gaze tracking system to perform a method as defined in any embodiment of the first aspect disclosed herein (in other words, in the claims, summary of the invention, or specific embodiments).

[0022] The non-transitory computer-readable storage medium may be provided, for example, in a computer program product. In other words, the computer program product may, for example, include a non-transitory computer-readable storage medium storing instructions that, when executed by the gaze tracking system, cause the gaze tracking system to perform the method as defined in any embodiment of the first aspect disclosed herein.

[0023] The effects and / or advantages presented in the present disclosure for the embodiments of the method according to the first aspect may also be applicable to the corresponding embodiments of the non-transitory computer-readable storage medium according to the third aspect.

[0024] It is noted that embodiments of the present disclosure relate to all possible combinations of features recited in the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In the following, example embodiments will be described in more detail with reference to the accompanying drawings, in which:

[0026] Figure 1 It is a front view of the eye;

[0027] Figure 2 It is observed from the side of the eye. Figure 1 A cross-sectional view of the eye;

[0028] Figure 3 is a schematic overview diagram of a gaze tracking system according to an embodiment;

[0029] Figure 4 is a flow chart of a gaze tracking method according to an embodiment;

[0030] Figure 5 It is shown that according to some embodiments, Figure 4 Example mappings used in the method;

[0031] Figure 6 It is shown that according to the embodiment, Figure 4 The pattern of known stimulation points used in the method;

[0032] Figure 7 Different areas in the eye's field of view are shown;

[0033] Figure 8 According to an embodiment, Figure 7 A flowchart of a gaze tracking method for different types of gaze tracking in different regions of the image;

[0034] Fig. 9 is a flow chart of a gaze tracking method involving mapping correction for gaze tracking according to an embodiment;

[0035] Fig.10 According to some embodiments, Figures 8 to 9 Flowchart of PCCR-type gaze tracking used in the method;

[0036] Fig.11 is a flow chart of how to determine a radial correction for a mapping employed in gaze tracking according to an embodiment;

[0037] Figure 12 to Figure 13 Different methods of how gaze direction may be determined according to embodiments are shown;

[0038] Fig.14 According to the embodiment Figure 4 , Figure 8 and / or Fig. 9 A flowchart of a method for performing gaze tracking in the method of

[0039] Fig.15 A head mounted device according to an embodiment is shown.

[0040] All the drawings are schematic and not necessarily drawn to scale and generally show only parts necessary for illustrating the various embodiments, while other parts may be omitted or merely suggested. Unless otherwise indicated, any reference numeral appearing in multiple drawings refers to the same object or feature in all drawings. DETAILED DESCRIPTION

[0041] Throughout this disclosure, the term "head-mounted display" or "HMD" refers to a display device that is adapted to be worn on the head of a user. In addition to the actual display optics, an HMD typically includes other components. Such other components may, for example, include circuits for powering the HMD, sensors for detecting movement of the HMD, gaze tracking devices, or a housing for protecting components of the HMD. In other words, the term "head-mounted display" or "HMD" is not necessarily to be interpreted as referring only to the actual display optics that are intended to be arranged in front of one eye of a user or in front of both eyes of a user.

[0042] Throughout this disclosure, the term "head-mounted device" refers to a device adapted to be worn on a user's head. The head-mounted device may, for example, be a head-mounted display (HMD). The head-mounted device may, for example, include any type of display. However, there are also head-mounted devices that do not include a display.

[0043] The gaze tracking method, the gaze tracking system and the associated storage medium will be referred to below. Figures 3 to 15 First, we will refer to Figure 1 to Figure 2 Describe some features of the eye.

[0044] Figure 1 is a front view of the eye 100 . Figure 2 is a cross-sectional view of the eye 100 as viewed from the side of the eye 100. Figure 2 More or less the entire eye 100 is shown, but Figure 1 1 shows only those portions of the eye 100 that are typically visible from the front of a person's face. The eye 100 has a cornea 101 and a pupil 102 with a pupil center 103. The cornea 101 is arc-shaped and has a center of curvature 104, which is referred to simply as the cornea center 104. The cornea 101 has a radius of curvature referred to as the cornea 101 radius 105, or simply as the cornea radius 105. The eye 100 has a center 106. The visual axis of the eye 100 passes through the center 106 of the eye 100 to the fovea 108 of the eye 100. The visual axis forms an angle 109 with respect to the optical axis 110 of the eye 100. The deviation or offset between the visual axis and the optical axis 110 is generally referred to as the foveal offset 109. Figure 1 Also shown is a reflection 112 of the illuminator at the cornea 101. Such a reflection 112 is also called a glint.

[0045] Figure 33 is a schematic overview of a gaze tracking system 300 according to an embodiment. The system 300 includes one or more illuminators 301 for illuminating the eye 100, and one or more cameras 302 for capturing images of the eye 100 when the eye 100 looks at a display 303. The system 300 also includes a processing circuit system 304 configured to estimate where the eye 100 is looking. The processing circuit system 304 can, for example, estimate the gaze direction (or gaze vector) of the eye 100 (corresponding to the direction of the visual axis), or estimate the gaze point 111 of the eye 100 on the display 303 (e.g., Figure 2 shown).

[0046] Processing circuit system 304 is communicatively connected to illuminator 301 and camera 302, for example via a wired or wireless connection. Processing circuit system 304 may also be communicatively connected to display 303, for example for controlling (or triggering) display 303 to show test stimulus points 305 for calibrating gaze tracking system 300.

[0047] The illuminator 301 may for example be an infrared or near infrared illuminator, for example in the form of a light emitting diode (LED).However, other types of illuminators are also conceivable. Figure 3 Example illuminators 301 are shown located on either side of the display 303, but illuminators 301 may be located elsewhere as well.

[0048] The camera 302 may be, for example, a Charge Coupled Device (CCD) camera or a Complementary Metal Oxide Semiconductor (CMOS) camera. However, other types of cameras are also contemplated. Figure 3 An example camera 302 is shown located above the display 303 , but the camera 302 may be located elsewhere, such as below the display 303 .

[0049] The display 303 may, for example, be included in the gaze tracking system 300, or may be considered separate from the gaze tracking system 300. The display 303 may, for example, be a liquid crystal display (LCD) or an LED display. However, other types of displays may also be contemplated. The display 303 may, for example, be a flat display or a curved display. The display 303 may, for example, be a television screen, a computer screen, or may be part of a head-mounted device such as a virtual reality (VR) or augmented reality (AR) device. The display 303 may, for example, be placed in front of one eye of the user. In other words, separate displays 303 may be used for the left eye and the right eye. For example, separate gaze tracking devices (such as an illuminator 301 and a camera 302) may be used for the left eye and the right eye.

[0050] Gaze tracking of both eyes may be performed using processing circuit system 304, or the left and right eyes may use separate processing circuit systems 304. Gaze tracking system 300 may, for example, perform gaze tracking for the left and right eyes separately, and then may determine a combined gaze point as an average of the gaze points of the left and right eyes.

[0051] The processing circuit system 304 may, for example, include one or more processors 306. The processor(s) 306 may, for example, be an application specific integrated circuit (ASIC) configured to perform a particular gaze tracking method. Alternatively, the processor(s) 306 may be configured to execute instructions stored in one or more memories 307 (e.g., in the form of a computer program). Such a memory 307 may, for example, be included in the circuit system 304 of the gaze tracking system 300, or may be external to the gaze tracking system 300 (e.g., remote from the gaze tracking system). The memory 307 may store instructions for causing the gaze tracking system 300 to perform the gaze tracking method.

[0052] It should be understood that the above reference Figure 3 The described gaze tracking system 300 is provided as an example, and many other gaze tracking systems are contemplated. For example, the illuminator(s) 301 and / or the camera(s) 302 are not necessarily considered part of the gaze tracking system 300. The gaze tracking system 300 may, for example, consist of only the processing circuitry 304. There are even gaze tracking systems that do not employ illuminators at all. As described below with reference to Figures 4 to 5 As described, the gaze direction 107 or the gaze point 111 can be estimated without using the illuminator 301. Fig.15 As described, system 300 can be used, for example, to perform gaze tracking on a head mounted device.

[0053] Figure 4 4 is a flow chart of a gaze tracking method 400 according to an embodiment. The gaze tracking method 400 includes obtaining 401 (or receiving) a set of images of the eye 100 captured by a camera 302. The images may be obtained, for example, in the form of snapshots or as part of a video sequence. The images may be received, for example, from the camera 302, either directly from the camera 302 or indirectly (e.g., via one or more other components of the gaze tracking system 300). The camera 302 may, for example, be arranged at a fixed position relative to the user's head (or relative to the user's face), such as at a head-mounted device (e.g., an HMD). The images captured by the camera 302 may, for example, capture the same view of the user's face, but captured at different time instances (or points in time).

[0054] The method 400 includes obtaining 402 (or receiving) gaze data for each image. The gaze data indicates a point 111 at which the eye 100 was looking (or assumed / thought to have been looking) when the image was captured and / or a direction 107 at which the eye 100 was looking (or assumed / thought to have been looking) when the image was captured. As described below with reference to Figure 6 As described, the gaze data obtained 402 may, for example, indicate a known position of a reference stimulus used for calibration. Alternatively, the gaze data obtained 402 may have been determined (or calculated or estimated) by some gaze tracking technology such as PCCR. The step 402 of obtaining gaze data may, for example, include receiving gaze data or calculating gaze data.

[0055] The method 400 includes estimating 403 the position of the center 103 of the pupil 102 of the eye 100 in each image. The estimation 403 can be performed, for example, via image analysis. For example, the edge of the pupil 102 (or at least a portion of the edge) can be detected in the image, and the position of the pupil center 103 can be estimated based on the detected pupil edge. In some images, it may be difficult to detect the edge of the pupil 102. For example, this may happen if the iris is so dark that its color is similar to the pupil 102. Then the edge of the iris (or at least a portion of the edge) can be detected in the image, and the position of the pupil center 103 can be estimated based on the detected iris edge. The image captured by the camera 302 can be, for example, a digital image. The position of the pupil center 103 in the image captured by the camera 302 can be estimated as one or more pixels in the image.

[0056] The method 400 comprises calibrating 404 a mapping (or a model or a function) using the obtained gaze data and the estimated position of the center 103 of the pupil 102. The mapping (or the model or the function) is adapted to map the position of the center 103 of the pupil 102 of the eye 100 in the image captured by the camera 302 to a gaze point 111 of the eye 100 on a surface or to a gaze direction 107 of the eye 100. In other words, the mapping (or the model or the function) is adapted to predict the gaze point 111 of the eye 100 on a surface and / or the gaze direction 107 of the eye 100 using the position of the center 103 of the pupil 102 of the eye 100 in the image captured by the camera 302. The calibration 404 may, for example, comprise setting the values ​​of one or more parameters of the mapping (or the model or the function) using the obtained gaze data and the estimated position of the center 103 of the pupil 102.

[0057] The image captured by the camera 302 provides a two-dimensional representation of the area imaged by the camera 302. The two-dimensional representation can be considered as a plane, referred to herein as the image plane. In other words, the image captured by the camera 302 can be considered to represent the image plane, and the mapping employed in the method 400 can be considered to be suitable for mapping the position of the center 103 of the pupil 102 in the image plane to a gaze point 111 of the eye 100 on a plane different from the image plane, or to a gaze direction 107 of the eye 100. Therefore, for the position of the center 103 of the pupil 102 of the eye 100 in the image plane (in other words, in the image captured by the camera 302), the mapping is suitable for providing (or outputting) a gaze point 111 of the eye 100 on a surface different from the image plane, or for providing (or outputting) a gaze direction 107 of the eye 100.

[0058] Figure 5 4 shows an example mapping that may be employed in method 400 according to some embodiments. Figure 5 As shown, a first example mapping 510 maps the location of the pupil center 103 in the image 501 captured by the camera 302 to a gaze point 511 on a flat surface 512 (or plane). The flat surface 512 may be, for example, a flat display, such as a flat display of a head mounted device. The first example mapping 510 may be considered as a mapping from the location of the pupil center in the images captured by the camera 302 to a gaze point on a plane other than the image plane represented by these images. Figure 5 The second example mapping 520 shown in maps the position of the pupil center 103 to a gaze point 521 on a curved surface 522. The curved surface 522 may be, for example, a curved display, such as a curved display of a head mounted device. Figure 5 The curved surface 522 depicted in FIG. 5 is concave, but embodiments are also contemplated in which the surface 522 is convex or has some other non-flat shape. The curved surface 522 may, for example, be parabolic. The second example mapping 520 may be considered to be a mapping from the location of the pupil center in the images captured by the camera 302 to a gaze point on a surface other than the image plane represented by those images. Figure 5 The third example mapping 530 shown in maps the position of the pupil center 103 to a gaze direction 531. The gaze direction 531 may be provided, for example, in the form of a vector or ray having a specific position in space, or may be provided in the form of a direction without a spatial position.

[0059] The method 400 includes obtaining 405 another image of the eye 100 captured by the camera 302. The another image may be obtained, for example, in the form of a snapshot or as part of a video sequence. The another image may be received, for example, from the camera 302, either directly from the camera 302 or indirectly (e.g., via one or more other components of the gaze tracking system 300) from the camera 302. The another image may, for example, capture the same user facial view as the other image captured by the camera 302, but captured at a different time instance (or point in time).

[0060] The method 400 comprises estimating 406 the position of the center 103 of the pupil 102 in the other image. The estimation 406 can be performed, for example, via image analysis. For example, the edge of the pupil 102 (or at least a portion of the edge) can be detected in the other image, and the position of the pupil center 103 can be estimated based on the detected pupil edge. It may be difficult to detect the edge of the pupil 102. For example, this may happen if the iris is so dark that its color is similar to the pupil 102. Then the edge of the iris (or at least a portion of the edge) can be detected in the other image, and the position of the pupil center 103 can be estimated based on the detected iris edge. The other image can be, for example, a digital image. The position of the pupil center 103 in the other image can, for example, be estimated as one or more pixels in the other image.

[0061] The method 400 includes performing 407 gaze tracking using the calibrated mapping and the estimated position of the center 103 of the pupil 102 in the other image. The gaze tracking 407 may, for example, involve estimating the gaze point 111 and / or the gaze direction 107. Example implementations of the gaze tracking 407 will be described below with reference to Figure 12 to Figure 14 Give a description.

[0062] According to some embodiments, the mapping employed in method 400 is suitable for mapping the location of pupil center 103 in the image captured by camera 302 to a surface (eg Figure 5 The eye 100 is now referred to as the point of gaze on the plane 512 and curved surface 522 shown in FIG. 4 , and the gaze data obtained at step 402 for each image indicates the point at which the eye 100 was looking when the image was captured. Figure 6 To describe such an embodiment, Figure 6 A pattern of known stimulation points 601 that may be used for calibration 404 in method 400 is shown.

[0063] In this embodiment, the Figure 5A first example mapping 510 of . During a personal calibration process, the user is prompted (or instructed or requested) to look at a plurality of stimulation points 601. The gaze tracking system 300 may provide instructions to the user, for example, via a text message on the display 303 or via an audio message. The pupil center position 103 is estimated 403, and the mapping 510 is calibrated 404 using the corresponding known stimulation point positions (601). Assume that K stimulation points are displayed to the user during calibration

[0064] {S0=(x g,0 ,y g,0 ),...,S K-1 =(x g,K-1 ,y g,K-1 )}.

[0065] For each stimulus point S k , collect N for each time instance k For these K stimulation points, a total of

[0066] N=N0+N1+...+N K-1

[0067] Pupil center position

[0068] {(x p,0 ,y p,0 ),...,(x p,N-1 ,y p,N-1 )}

[0069] Each pupil center position has a corresponding stimulus point position. The two functions f and g in the following equations (1) and (2) are then fitted to the calibration data.

[0070] x g =f(x p ,y p ) (1)

[0071] y g =h(x p ,y p ) (2)

[0072] These two functions f and g define a mapping 510 and are used to perform gaze tracking in step 407. In an example implementation, functions f and g are linear functions according to the following equations (3) and (4):

[0073] x g =ax p +by p +c (3)

[0074] y g =dx p +eyp +f (4)

[0075] Where a, b, d and e are slopes, and c and f are intercepts of the linear function. These six parameters are estimated (or calculated) based on the calibration data. Methods such as the least squares method can be used to estimate the values ​​of the parameters a, b, c, d, e and f.

[0076] Functions f and g may also be polynomials of higher order, as long as N is at least as large as the number of parameters to be estimated in such higher order polynomials. However, embodiments are also conceivable in which functions f and g are not polynomials.

[0077] After calibration 404 is completed via known stimulation points 601 on surface 512, functions f and g may be used for new images captured by camera 302 by replacing x in equations (1) and (2) with the position of pupil center 103 detected in these new images. p and p To estimate the gaze point (x g ,y g ).

[0078] As an alternative to the above equations (1) to (4), the mapping 510 may be defined via a homography according to the following equation:

[0079] [x g y g 1] T =H[x p y p 1] T

[0080] A first example mapping 510 maps the cornea center position from the image plane to a flat surface 512, and may be linear (as illustrated in equations (3) and (4) above) or non-linear. A second example mapping 520 maps the cornea center position from the image plane to a curved surface 522, and is generally non-linear. A third example mapping 530 maps the cornea center position from the image plane to a gaze direction, and may be linear or non-linear.

[0081] Therefore, according to some embodiments, the mapping employed in method 400 may be, for example, a linear mapping, a non-linear mapping, or a homography mapping.

[0082] As mentioned above Figure 6As described, the known stimulation point 601 may be used for calibration 404. Thus, according to some embodiments, at least one image in the set of images obtained 401 is captured when the eye 100 is gazing at the stimulation point 601. The gaze data obtained 402 for the image indicates a known position 601 of the stimulation point and / or a known direction from the eye 101 towards the stimulation point 601.

[0083] References Figure 4 The described gaze tracking method 400 can be used, for example, as an alternative or supplement to PCCR-based gaze tracking to improve gaze tracking performance (e.g., accuracy and / or precision) for at least some gaze angles. The performance of PCCR-based gaze tracking is generally lower at large gaze angles than at small gaze angles. Figure 4 The described method 400 can be used, for example, to improve gaze tracking performance at large gaze angles. Figures 7 and 8 Such an embodiment is described below.

[0084] Figure 7 Different regions in the field of view of the eye 100 are shown. The field of view includes a central region 701 and an outer region 702. The central region 701 may, for example, correspond to a gaze angle 703 that is at most a certain threshold, such as a gaze angle of at most 5 degrees, or at most 10 degrees, or at most 25 degrees. The outer region 702 may, for example, correspond to a gaze angle 703 that is greater than a threshold, such as a gaze angle greater than 5 degrees, or greater than 10 degrees, or greater than 25 degrees. The gaze angle 703 may, for example, be measured as an angle between a forward direction 704 and a gaze direction 705 of the eye 100. The gaze angle 703 may, for example, be measured as an angle between a forward direction 704 of the user's head and a gaze direction 705 of the eye 100. The gaze angle 703 may, for example, be measured as an angle between a forward direction 704 of the user's head and a gaze direction 705 of the eye 100. The gaze angle 703 may, for example, be measured as an angle between a forward direction 704 in a coordinate system of a gaze tracking system and a gaze direction 705 of the eye 100.

[0085] Figure 8 According to an embodiment, Figure 7 Flow chart of a gaze tracking method 800 for different types of gaze tracking of different regions shown in FIG. The method 800 includes the same method as described above with reference to FIG. Figure 4 The method 400 described has the same six initial steps 401 to 406. The description of these steps will not be repeated here.

[0086] The method 800 comprises estimating 801 whether the eye 100 is oriented toward (or looking toward) a central region 701 of the visual field of the eye 100. This estimation 801 may be performed, for example, via PCCR (as described below with reference to Fig.10 described), but may be performed in some other way.

[0087] If the eye 100 is directed toward an area outside of the central area 701 (e.g., an outer area 702, which may correspond to a gaze angle greater than 5 degrees, or greater than 10 degrees, or greater than 25 degrees, for example), the method 800 proceeds to output 802 gaze data (e.g., gaze data indicating a gaze direction or gaze point) determined by performing 407 gaze tracking using the calibrated mapping and the method described above with reference to FIG. Figure 4 The estimated position of the center 103 of the pupil 102 in the other image obtained at step 405 described above. The step 407 of performing gaze tracking using the calibrated map can, for example, be performed only when the eye 100 is directed toward an area 702 outside the central area 701 (e.g. Figure 8 As shown), or it can be executed regardless of where the eye 100 is facing.

[0088] If the eye 100 is directed toward the central area 701 (e.g., corresponding to a gaze angle of up to 5 degrees, or up to 10 degrees, or up to 25 degrees), the method 800 proceeds to output 804 gaze data (e.g., gaze data indicating a gaze direction or gaze point), the gaze data being determined by performing 803 a second type of gaze tracking different from the gaze tracking performed in the gaze tracking step 407. The step 803 of performing the second type of gaze tracking may, for example, be performed only when the eye 100 is directed toward the central area 701 (e.g., corresponding to a gaze angle of up to 5 degrees, or up to 10 degrees, or up to 25 degrees). Figure 7 As shown), or it can be executed regardless of where the eye 100 is facing.

[0089] In this embodiment, the second type of gaze tracking 803 is a PCCR-based scheme that uses the estimated position of the reflection 112 (also called the glint) of the illuminator 301 at the cornea 101 of the eye 100 and the estimated position of the pupil center 103 of the eye 100. Due to the noise in estimating the glint position, the gaze tracking performance of PCCR-based gaze tracking is generally lower at large gaze angles than at small gaze angles. In an arrangement where the camera 302 is arranged at a fixed position relative to the user's head (such as in a head-mounted device), the first type of gaze tracking 407 (which involves using the above reference Figures 4 to 5The method 800 may provide better gaze tracking performance than the PCCR-based gaze tracking 803. The ability of the method 800 to output 802 gaze data from the first type of gaze tracking 407 (involving the use of mapping) for larger angles and output 804 gaze tracking data from the PCCR-based gaze tracking 803 for smaller gaze angles (where the PCCR-based gaze tracking performs well) allows for improved gaze tracking performance for at least some of the gaze angles in the outer region 702. This advantage may also be obtained in embodiments where the second type of gaze tracking 803 is not PCCR-based, as long as the first type of gaze tracking 407 can perform better than the second type of gaze tracking for at least some of the gaze angles in the outer region 702.

[0090] In a PCCR-based gaze tracker that does not employ the first type of gaze tracking 407, additional cameras 302 and / or illuminators 301 may be placed at appropriate locations in an attempt to improve gaze tracking performance for large gaze angles. However, such additional equipment may increase the cost of the gaze tracking system and may occupy space that should be occupied by other components other than the gaze tracking device. Figure 8 The described method 800 provides an alternative method of coping with large viewing angles that does not require the use of additional cameras 302 or illuminators 301. This may reduce power consumption and / or improve eye safety (due to a reduced number of illuminators).

[0091] It should be understood that the step 801 of estimating whether the eye 100 is toward the central region 701 or the outer region 702 does not necessarily need to be a high-quality (or high-precision / accuracy) estimate. Typically, a rough estimate is sufficient because both the first type of gaze tracking 407 and the second type of gaze tracking 803 may perform relatively well for gaze angles close to the border between the central region 701 and the outer region 702. In other words, for gaze angles close to the border between the central region 701 and the outer region 702, the gaze tracking performance may not be greatly affected if the "wrong" type of gaze tracking is accidentally used.

[0092] By reference above Figure 8 The gaze data output by the described method 800 may, for example, indicate a gaze point 111 and / or a gaze direction 107. The gaze data may, for example, be provided as a signal from a gaze tracking system (e.g., as described above with reference to Figure 3 The gaze tracking system 300 described herein may be an output of the gaze tracking system 300 , or may be provided as an output from a subsystem or portion of a gaze tracking system to another subsystem or portion of the gaze tracking system.

[0093] Fig. 9 According to an embodiment, the correction of the above reference Figures 4 to 5Flowchart of gaze tracking method 900 for mapping described herein. If camera 302 moves or is repositioned relative to the user's head after calibration mapping, the mapping may need to be corrected (or recalibrated) to take this repositioning into account. Even if camera 302 is mounted on a head mounted device, camera 302 may change position if the head mounted device slips. When the eyes are directed toward central region 701 (corresponding to a small gaze perspective), a gaze tracking method that performs well for small gaze perspectives (such as PCCR-based gaze tracking) can be used to detect the repositioning of camera 302, and this gaze tracking method can also be used to determine the gaze tracking method to be applied to the above reference. Figures 4 to 5 Appropriate correction of the described mapping. This idea is adopted in method 900.

[0094] Method 900 includes the above reference Figure 4 The initial seven steps 401 to 407 of the method 400 are described. The description of these steps will not be repeated here. However, it is recalled that the seventh step 407 involves performing a first type of gaze tracking that employs a calibrated map and an estimated position of the pupil center 103 in the image.

[0095] The method 900 includes performing 901 a second type of gaze tracking that is different from the first type of gaze tracking (which uses the calibrated mapping and the estimated position of the center 103 of the pupil 102 in another image). In the present embodiment, the second type of gaze tracking is a PCCR-based gaze tracking scheme that uses the estimated position of the reflection 112 (also referred to as the glint) of the illuminator 301 at the cornea 101 of the eye 100 and the estimated position of the pupil center 103 of the eye 100. However, the second type of gaze tracking may be some other type of gaze tracking.

[0096] The method 900 comprises estimating 902 whether the eye 100 is oriented towards the central area 701 of the visual field of the eye 100. This estimation 901 may be performed, for example, via PCCR (as described below with reference to Fig.10 described), but may be performed in some other way.

[0097] If the eye 100 is oriented toward the central region 701 (e.g., corresponding to a gaze angle of up to 5 degrees, or up to 10 degrees, or up to 25 degrees), the method 900 continues mapping based on the gaze data obtained via the second type of gaze tracking (in other words, the method 900 further includes the method 901 of FIG. 1 ). Figures 4 to 5The second type of gaze tracking may be applied 903 to the mapping described above. The gaze data obtained via the second type of gaze tracking may, for example, indicate a different gaze direction and / or gaze point than the gaze data obtained via the first type of gaze tracking, which may indicate that the mapping employed by the first type of gaze tracking needs to be corrected. For example, the correction applied to the mapping in step 903 may be applied to the mapping until a new / updated correction is deemed necessary, or until the eye 100 is again directed toward the central area 701. In other words, the correction determined when the eye 100 was directed toward the outer area 702 may also continue to be applied when the eye 100 is directed toward the central area 701.

[0098] If the eye 100 is directed toward an area 702 outside the central area 701 (e.g., corresponding to a gaze angle greater than 5 degrees, or greater than 10 degrees, or greater than 25 degrees), the method 900 can, for example, proceed to step 904, where the previously determined correction can be maintained, or (e.g., if no correction has been determined) an uncorrected mapping can be employed. Since PCCR-based gaze tracking performs better for smaller gaze angles, the correction for determining the mapping when the eye 100 is directed toward the central area 701 may be better than when the eye 100 is directed toward the outer area 702.

[0099] A clear example of how the correction is applied in step 903 will now be described. For simplicity, consider the following example where the first type of gaze tracking 407 employs a reversible mapping M that transforms the pupil center position (x p ,y p ) is mapped to the fixation point (x g ,y g ):

[0100] (x g ,y g )=M(x p ,y p )

[0101] Assume that the second type of gaze tracking 901 shows that the gaze point should actually be (x' g ,y' g ), (x' g ,y' g ) relative to the estimated fixation point (x g ,y g ) has a deviation. -1 Applied to the "real" fixation point, we get the point

[0102] (x' p ,y' p )=M -1 (x' g ,y'g )

[0103] If the mapping M is performed, this will give the "real" fixation point (x' g ,y' g ). Point (x p ,y p ) and (x' p ,y' p ) indicates that the camera 302 that captured the image has been repositioned after calibration of the map M. If the camera 302 is mounted on a head-mounted device, the repositioning may be caused by slippage of the head-mounted device. To compensate for the repositioning, we can determine a correction map R that accounts for the potential translation and / or reorientation of the camera 302, such that

[0104] (x' p ,y' p )=R(x p ,y p )

[0105] And therefore

[0106] (x' g ,y' g )=M(R(x p ,y p )).

[0107] We want to map R to all points (x p ,y p ) works, so it may take several pairs of (x p ,y p ) and (x' p ,y' p ) to determine an appropriate correction mapping R to be used with M. Thus, after a set of data samples with sufficient variation has been collected, the correction mapping R can be determined. If, for example, the camera 302 has slipped to the left relative to the user's head, the correction mapping R can be a simple translation. For example, if the mapping M is linear and defined by equations (3) and (4), then applying the correction function R and the mapping M may correspond only to recalibrating the six parameters a, b, c, d, e, and f. In other words, instead of using R and M, one can simply use a new mapping M' with new values ​​for the six parameters a, b, c, d, e, and f. On the other hand, if the mapping M is a more complex, non-linear mapping with many parameters, then determining the correction mapping R may be easier (and / or require fewer data samples) than recalibrating the entire mapping M.

[0108] Another example of how to apply the correction in step 903 is based on the deviation (x' g ,y' g )-(xg ,y g ) samples to estimate the compensation (Δ c x ,Δ c y ). For example, the deviation is mean filtered. Then it is applied such that:

[0109] (x g ,y g )=M(x p ,y p )+(Δ c x ,Δ c y )

[0110] In one example, the compensation is updated only when the eye 100 is directed toward the central area 701 .

[0111] The second type of gaze tracking 901 may, for example, be performed only when the eye 100 is directed toward the central area 701, or may be performed regardless of where the eye 100 is directed (e.g., Fig. 9 shown).

[0112] It should be understood that the step 902 of estimating whether the eye 100 is facing the central region 701 or the outer region 702 does not necessarily need to be a high-quality (or high-precision / accuracy) estimate. Typically, a rough estimate is sufficient because the second type of gaze tracking 901 can also perform relatively well for gaze angles close to the boundary between the central region 701 and the outer region 702. In other words, even if the gaze angle is slightly outside the central region 701 where PCCR-based gaze tracking performs best, the gaze tracking performance may not be greatly affected if the new correction is applied 903.

[0113] Fig.10 According to some embodiments, Figures 8 to 9 Flowchart of PCCR-based gaze tracking scheme 1000 employed in method 800. For example, PCCR gaze tracking 1000 can be employed in gaze tracking step 803 of method 800 or in gaze tracking step 901 of method 900. For example, PCCR gaze tracking 1000 can be employed in step 801 of method 800 or in step 902 of method 900 to estimate whether eye 100 is directed toward central region 701 of visual field of eye 100.

[0114] If PCCR gaze tracking 1000 is employed, another image for gaze tracking is captured while the eye 100 is illuminated by one or more illuminators 301. PCCR gaze tracking 1000 includes estimating 1001 one or more positions of one or more reflections 112 of the one or more illuminators 301 at the cornea 101 of the eye 100 in the another image, and estimating 1002 the position of the center 103 of the pupil 102 of the eye 100 in the another image.

[0115] Gaze tracking 1003 is performed using the estimated position(s) of the corneal reflection(s) 112 and the estimated position 103 of the pupil center. Gaze tracking 1003 may provide gaze point 111 and / or gaze direction 107, for example.

[0116] Fig.11 How to determine the above reference according to the embodiment Figures 4 to 5 Flowchart of radial correction of the mapping described. For large viewing angles, the above reference Figure 4 The gaze point estimated by the described gaze tracking step 407 is usually slightly deviated from the true gaze point in the radial direction. The radial direction refers to the direction extending radially outward from the center of the field of view of the eye 100. A radial correction can be applied to compensate for this deviation. Therefore, in this embodiment, the step 407 of performing gaze tracking using a calibrated mapping and an estimated position of the center 103 of the pupil 102 in another image includes applying a radial correction to the mapping. The radial correction involves the use of a correction function. The size of the correction function depends on the gaze angle of the eye 100. For example, when the gaze angle increases, the size of the correction function can increase. The correction function can, for example, be linear or non-linear.

[0117] For simplicity, consider the following situation: the mapping maps the pupil center position 103 to the gaze point 111 in a certain plane (such as Figure 5 In this setting, the radial correction applied to the mapping can be expressed, for example, as

[0118] f(α)v

[0119] Where α is the gaze angle, f is a scalar function, and v is a vector directed radially outward from the center of the field of view of the eye 100. Thus, if the uncorrected radial mapping is represented by M, and the coordinates of the pupil center position are represented by x p and p Given, the coordinates of the gaze point x g and g It can be calculated as

[0120]

[0121] The function may be, for example, non-negative or positive. For a small gaze angle, the function f may be, for example, zero. For example, when the gaze angle α increases, the function f may increase. The function f may be, for example, linear or non-linear. The vector v may, for example, have a length of 1 (in other words, v may have a unit length).

[0122] The correction function can be used as a reference above Figure 4 This may be calibrated as part of the described calibration step 404 or may be calibrated via a separate calibration process. Fig.11 is a flow chart for such a calibration process 1100. The calibration process 1100 is performed before applying radial correction to the mapping.

[0123] The calibration process 1100 includes obtaining 1101 a plurality of images of the eye 100 captured by the camera 302 while the eye 100 is gazing at various known stimulus points. Figure 6 Examples of such stimulation points 601, 602, 604 are shown. In this embodiment, at least one stimulation point corresponds to a gaze angle of at least 5 degrees, or at least 10 degrees, or at least 25 degrees. Figure 6 Examples of such stimulation points corresponding to large gaze angles are provided in FIG. 6 , such as stimulation point 602 on the left 603 and stimulation point 604 on the right 605 .

[0124] The calibration process 1100 includes estimating 1102 the location of the center 103 of the pupil 102 of the eye 100 in a plurality of images.

[0125] The calibration process includes using the locations of the known stimulus points 601, 602, 604, the estimated location of the center 103 of the pupil 102 of the eye 100 in multiple images, and the mapping (see above for reference). Figure 4 Describe and Figure 5 The correction function is calibrated 1103 by mappings 510, 520 and 530).

[0126] exist Figure 6 In the example stimulation point pattern shown, the stimulation points in the central region 606 are used for calibration mapping (see above for reference). Figure 4 Describe and Figure 5510, 520 and 530 in ), while the stimulus point 602 at the far left 603 and the stimulus point 604 at the far right 605 are used to calibrate the radial correction function. Thus, a relatively simple mapping, such as a linear mapping, can be used for gaze tracking and supplemented with a relatively simple radial correction for large gaze angles. A single camera 302 may be sufficient and an illuminator 301 may not be required. In contrast, if a purely PCCR-based gaze tracking system is to be designed to cope with such large gaze angles, multiple illuminators 301 and / or cameras 302 may have to be distributed at various locations to ensure that illuminators 301 and cameras 302 are available for all gaze angles.

[0127] Fig.14 How the above reference may be performed according to an embodiment Figure 4 The method 400 described above (or Figures 8 to 9 Flowchart of gaze tracking step 407 in method 800 and / or 900 described herein. Fig.12 12. This embodiment is described below, which shows how line of sight 1201 may be determined. Fig.12 The coordinate system of the gaze tracking system is shown. The coordinate system includes axes x, y and z, and an origin O of the coordinate system. Axis z is toward the user's face. User distance (or depth) is measured along the z axis.

[0128] In this embodiment, the above reference Figure 4 The described mapping is suitable for mapping the position of the center 103 of the pupil 102 of the eye 100 in the image captured by the camera 302 to the gaze point 1202 of the eye 100 on a surface 1203 (similar to Figure 5 14. In the embodiment, step 407 of performing gaze tracking using the calibrated mapping and the estimated position of the center 103 of the pupil 102 in the other image comprises estimating 1401 a gaze point 1202 on the surface 1203 by applying the mapping to the estimated position of the center 103 of the pupil 102 in the other image, and estimating 1402 a gaze direction 1201 (or line of sight) as a straight line or vector passing through a reference point 1204 of the eye 100 and passing through the estimated gaze point 1202 on the surface 1203.

[0129] The reference point 1204 is a point through which the line of sight 1201 passes, and can be considered, for example, as the line of sight starting point of the line of sight 1201. The center of corneal curvature 104 (also referred to as the corneal center 104, in Figure 2 ) can be used as a reference point 1204, for example. Thus, the reference point 1204 for gaze tracking using another image can be the center 104 of the corneal curvature estimated based on the other image. The corneal center 104 can be estimated, for example, via PCCR.

[0130] In the example implementations described above, the reference point (or gaze origin) 1204 is a variable position determined for each image during gaze tracking. However, embodiments are also contemplated where the reference point 1204 is a fixed position that is used to determine the gaze for multiple images and time instances. For example, the reference point 1204 may be a variable position that is determined for each image during gaze tracking. Figure 4 Each image in a set of images used in the calibration step 404 in the described method 400 estimates the center 104 of the corneal curvature. The average of these centers 104 of corneal curvature can be used as a reference point 1204 for gaze tracking 407 using another image. The corneal center 104 can be estimated, for example, via PCCR. As described above, PCCR can perform better at small gaze angles than at large gaze angles. Therefore, the corneal center position 104 estimated for small gaze angles where PCCR performs well can be used to calculate a fixed reference point 1204, and this reference point 1204 can also be used for larger gaze angles where PCCR may have lower accuracy / precision. Therefore, by using a fixed reference point 1204, potential errors caused by noise in the PCCR calculation for large gaze angles can be avoided.

[0131] Fig.13 An alternative way of how line of sight 1301 may be determined is shown. Fig.13 The coordinate system of the gaze tracking system is shown. The coordinate system includes axes x, y and z, and an origin O of the coordinate system. Axis z points toward the user's face. Figure 4 The described mapping is suitable for mapping the position of the center 103 of the pupil 102 of the eye 100 in the image captured by the camera 302 to the gaze point 1302 of the eye 100 on the surface 1303 (similar to Figure 5 1402 ). In this embodiment, step 407 of performing gaze tracking using the calibrated mapping and the estimated position of the center 103 of the pupil 102 in the other image comprises estimating 1401 a gaze point 1302 on the surface 1303 by applying the mapping to the estimated position of the center 103 of the pupil 102 in the other image, and estimating 1402 a gaze direction 1301 (or line of sight) as a straight line or vector passing through a reference point (or line of sight starting point) 1304 and passing through the estimated gaze point 1302 on the surface 1303.

[0132] In this embodiment, reference point 1304 is located in corneal center plane 1305. Corneal center plane 1305 has been used for the above reference Figure 4The corneal center plane 1305 is determined by the estimated corneal curvature center 104 for each image in the set of images used in the calibration step 404 in the described method 400. The corneal center plane 1305 is a plane orthogonal to the z-axis and has a z-coordinate that is the average of the z-coordinates of the corneal curvature center 104 estimated for each image in the set of images used in the calibration step 404. The corneal curvature center 104 is also estimated for another image. The reference point 1304 for gaze tracking 407 using another image is calculated as the projection of the corneal curvature center 104 estimated based on the other image in the corneal center plane 1305. As described above, PCCR may perform better at small gaze angles than at large gaze angles. Therefore, the corneal center position 104 estimated for small gaze angles at which PCCR performs well can be used to calculate a fixed corneal center plane 1305, and the corneal center plane 1305 can also be used for larger gaze angles at which PCCR may have lower accuracy / precision. Therefore, potential errors due to noise in PCCR calculations for large fixation angles may be reduced by projecting the estimated corneal center position 104 in the corneal center plane 1305 .

[0133] References in the above Figure 12 to Figure 13 In the described example, the reference points 1204 and 1304 used for gaze tracking are based on the position of the center of corneal curvature 104. However, the reference points 1204 and 1304 may be based on the position of the center 106 of the eye 100, for example, rather than the position of the center of corneal curvature 104. In other words, Fig.12 The reference point 1204 (or the starting point of sight) used in may be the eye center 106 estimated using another image obtained at step 405, or may be the average value of the position of the eye center 106 estimated using a set of images obtained at step 401.

[0134] Fig.15 A head mounted device 1501 according to an embodiment is shown. The head mounted device 1501 may, for example, include the Figure 3 The gaze tracking system 300 described above, or a portion of the gaze tracking system 300, may be a head mounted device 1501, for example, including the camera 302, but the processing circuit system 304 may be located, for example, remote from the head mounted device 1501. The processing circuit system 304 may communicate with the camera 302 in the head mounted device 1501, for example, via wired or wireless communication. For example, the above reference may be performed. Figure 4 The method 400 is described for performing gaze tracking on the head mounted device 1501. The method 400 may be performed in the head mounted device 1501, for example, using images captured by the camera 1302 in the head mounted device 1501, or by a processing circuit system located remotely from the head mounted device 1501.

[0135] When user 1502 wears head mounted device 1501, camera 1302 is located at a fixed position relative to user 1502. When head mounted device 1501 moves relative to user 1502 (e.g., due to slippage), the mapping used for gaze tracking can be updated to compensate for the repositioning of head mounted device 1501. For example, the above reference to Figure 4 Another way to compensate for the repositioning of the head mounted device 1501 is to apply a correction to the mapping used for gaze tracking, as described above with reference to Fig. 9 Described.

[0136] The head-mounted device 1501 may be, for example, a head-mounted display (HMD) such as a VR head-mounted device, an AR head-mounted device (such as AR glasses), or a mixed reality (MR) head-mounted device.

[0137] References Figures 4 to 14 The methods and schemes described represent a first aspect of the present disclosure. Figure 3 The gaze tracking system 300 described above and the Fig.15 The head mounted device 1501 described represents the second aspect of the present disclosure. The system 300 (or the processing circuit system 304 of the system 300) can, for example, be configured to perform the gaze tracking method of any embodiment of the first aspect described above. The system 300 can, for example, be configured to perform the gaze tracking method of any embodiment of the first aspect described above. Figure 4 The method 400 described above with reference to Figure 8 Method 800 described above, or Fig. 9 Method 900 is described.

[0138] According to an embodiment, the gaze tracking system 300 includes processing circuitry 304 configured to:

[0139] a) obtaining a set of images of the eye 100 captured by the camera 302;

[0140] b) for each image from the set of images:

[0141] - obtaining gaze data indicating the point 111 at which the eye 100 was looking when the image was captured and / or the direction 107 in which the eye 100 was looking when the image was captured, and

[0142] - estimating the position of the center 103 of the pupil 102 of the eye 100 in the image;

[0143] c) using the obtained gaze data and the estimated position of the center 103 of the pupil 102 to calibrate a mapping, wherein the mapping is suitable for mapping the position of the center 103 of the pupil 102 of the eye 100 in the image captured by the camera 302 to:

[0144] - the gaze point 111 of the eye 100 on a surface, or

[0145] - gaze direction 107 of the eye 100;

[0146] d) obtaining another image of the eye 100 captured by the camera 702;

[0147] e) estimating the position of the center 103 of the pupil 102 in the further image; and

[0148] f) Performing gaze tracking using the calibrated map and the estimated position of the center 103 of the pupil 102 in the further image.

[0149] As mentioned above Figure 3 As described, the gaze tracking system 300 does not necessarily include Figure 3 All components shown in .

[0150] A third aspect of the present disclosure is represented by an embodiment of a non-transitory computer-readable storage medium 307 storing instructions that, when executed by a gaze tracking system 300, cause the gaze tracking system 300 to perform the method of any embodiment of the first aspect described above (as described above with reference to Figure 4 The method 400 described above with reference to Figure 8 Method 800 described above, or Fig. 9 Method 900 described).

[0151] According to an embodiment, the non-transitory computer-readable storage medium 307 stores instructions that, when executed by the gaze tracking system 300, cause the gaze tracking system 300 to perform the following operations:

[0152] a) obtaining a set of images of the eye 100 captured by the camera 302;

[0153] b) for each image from the set of images:

[0154] - obtaining gaze data indicating the point 111 at which the eye 100 was looking when the image was captured and / or the direction 107 in which the eye 100 was looking when the image was captured, and

[0155] - estimating the position of the center 103 of the pupil 102 of the eye 100 in the image;

[0156] c) using the obtained gaze data and the estimated position of the center 103 of the pupil 102 to calibrate a mapping, wherein the mapping is suitable for mapping the position of the center 103 of the pupil 102 of the eye 100 in the image captured by the camera 302 to:

[0157] - the gaze point 111 of the eye 100 on a surface, or

[0158] - gaze direction 107 of the eye 100;

[0159] d) obtaining another image of the eye 100 captured by the camera 302;

[0160] e) estimating the position of the center 103 of the pupil 102 in the further image; and

[0161] f) Performing gaze tracking using the calibrated map and the estimated position of the center 103 of the pupil 102 in the further image.

[0162] As mentioned above Figure 3 As described, storage medium 307 is not necessarily included in system 300 .

[0163] Those skilled in the art will appreciate that the present invention is by no means limited to the preferred embodiments described above. On the contrary, many modifications and variations are possible within the scope of the appended claims. For example, the above reference Figures 4 to 14 The described methods and schemes may be combined to form further embodiments. In addition, it should be understood that the gaze tracking system 300 shown in FIG. 300 is intended only as an example, and other gaze tracking systems may also perform the above referenced Figures 4 to 14 It should also be understood that reference Figure 4 , Figure 8 , Fig. 9 , Fig.10 , Fig.11 and Fig.14 The method steps described are not necessarily performed in the exact order shown in the figures.

[0164] It should be understood that the processing circuit system 304 (or processor) may include a combination of one or more of the following: a microprocessor, a controller, a microcontroller, a central processing unit, a digital signal processor, an application specific integrated circuit, a field programmable gate array, or any other suitable computing device, resource, or combination of hardware, software and / or encoded logic that is operable to provide computer functions alone or in combination with other computer components (such as memory or storage media).

[0165] It should also be understood that the memory or storage medium 307 (or computer-readable medium) may include any form of volatile or non-volatile computer-readable memory, including but not limited to persistent storage devices, solid-state memory, remotely mounted memory, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), mass storage media (e.g., a hard disk), removable storage media (e.g., a flash drive, a compact disk (CD) or a digital video disk (DVD)), and / or any other volatile or non-volatile, non-transitory device-readable and / or computer-executable memory device that stores information, data and / or instructions that can be used by a processor or processing circuit system.

[0166] In addition, when practicing the claimed invention, those skilled in the art can understand and realize the variations of the disclosed embodiments by studying the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude pluralities. In the claims, the word "or" is not interpreted as an exclusive or (sometimes referred to as "XOR"). On the contrary, unless otherwise indicated, expressions such as "A or B" cover all situations in "A but not B", "B but not A", and "A and B". The simple fact that certain measures are stated in mutually different dependent claims does not indicate that a combination of these measures cannot be used to achieve a favorable result. Any figure mark in the claims should not be understood as limiting the scope.

Claims

1. A gaze tracking method, comprising: obtaining a set of images of the eye captured by a camera; For each image from the set of images: obtaining gaze data indicating a point at which the eye was looking when the image was captured and / or a direction in which the eye was looking when the image was captured, and estimating the position of the center of the pupil of the eye in the image; calibrating a mapping using the obtained gaze data and the estimated position of the center of the pupil, wherein the mapping is adapted to map the position of the center of the pupil of the eye in an image captured by the camera to: the point at which the eye is looking on the surface it is looking at, or the gaze direction of the eye; obtaining another image of the eye captured by the camera while the eye is illuminated by the illuminator; estimating a position of a reflection of the illuminator at the cornea of ​​the eye in the further image; estimating the position of the center of the pupil in the other image; and performing gaze tracking using the calibrated map and an estimated position of a center of the pupil in the other image; The method further comprises the following steps: estimating whether the eye is facing towards a central area of ​​the eye's field of view by using the estimated position of the reflection of the illuminator and the estimated position of the center of the pupil in the other image; and In response to the eye being directed toward an area outside of the central area, gaze data determined by performing gaze tracking using the calibrated map and the estimated position of the center of the pupil in the other image is output.

2. The method of claim 1, wherein: An image from the set of images is captured when the eye is gazing at a stimulation point, and wherein the gaze data obtained for the image indicates a known position of the stimulation point and / or a known direction from the eye towards the stimulation point.

3. The method of claim 1, wherein: When the surface is a plane, the mapping is adapted to map the position of the center of the pupil of the eye in the image captured by the camera to a gaze point of the eye on the plane.

4. The method of claim 1, wherein: The mapping is adapted to map the position of the center of the pupil of the eye in an image captured by the camera to a gaze point of the eye on the surface, and wherein the obtained gaze data indicates the point on the surface at which the eye was looking when the image was captured.

5. The method of claim 1, comprising: estimating whether the eye is oriented toward a central region of the eye's field of view; as well as In response to the eye being directed toward the central region, outputting gaze data determined by performing a second type of gaze tracking different from gaze tracking using the calibrated mapping and the estimated position of the center of the pupil in the other image.

6. The method of claim 1, comprising: performing a second type of gaze tracking different from the gaze tracking using the calibrated map and the estimated position of the center of the pupil in the other image; estimating whether the eye is oriented toward a central region of the eye's field of view; as well as In response to the eye being directed toward the central region, applying a correction to the mapping based on gaze data obtained via the second type of gaze tracking.

7. The method of claim 6, wherein: The second type of gaze tracking uses an estimated position of a reflection of an illuminator at the cornea of ​​the eye in the other image and an estimated position of a center of the pupil in the other image.

8. The method of claim 1, comprising: In response to the eye being directed toward the central region, gaze data determined by performing gaze tracking using the estimated position of the reflection of the illuminator and the estimated position of the center of the pupil in the other image is output.

9. The method of claim 1, wherein: The central area corresponds to a gaze angle less than a threshold value.

10. The method of claim 9, wherein: The central area corresponds to a gaze angle of less than 25 degrees.

11. The method of claim 9, wherein: The central area corresponds to a gaze angle of less than 10 degrees.

12. The method of claim 9, wherein: The central area corresponds to a gaze angle of less than 5 degrees.

13. The method of claim 1, wherein: Performing gaze tracking using the calibrated map and the estimated position of the center of the pupil in the other image comprises: A radial correction is applied to the mapping, wherein the radial correction involves using a correction function, wherein a magnitude of the correction function depends on a gaze angle of the eye.

14. The method of claim 13, comprising: Before applying the radial correction to the map: obtaining a plurality of images of the eye captured by the camera while the eye is gazing at respective known stimulus points, wherein at least one of the stimulus points corresponds to a gaze angle of at least 10 degrees; estimating the location of the center of the pupil of the eye in the plurality of images; and The correction function is calibrated using the locations of the known stimulus points, the estimated locations of the centers of the pupils of the eyes in the plurality of images, and the mapping.

15. The method of claim 13, comprising: Before applying the radial correction to the map: obtaining a plurality of images of the eye captured by the camera while the eye is gazing at respective known stimulus points, wherein at least one of the stimulus points corresponds to a gaze angle of at least 25 degrees; estimating the location of the center of the pupil of the eye in the plurality of images; and The correction function is calibrated using the locations of the known stimulus points, the estimated locations of the centers of the pupils of the eyes in the plurality of images, and the mapping.

16. The method of claim 1, wherein: The mapping is adapted to map a position of a center of the pupil of the eye in an image captured by the camera to a gaze point of the eye on the surface, and wherein performing gaze tracking using the calibrated mapping and the estimated position of the center of the pupil in the further image comprises: estimating a gaze point on the surface by applying the mapping to an estimated position of a center of the pupil in the other image; and The gaze direction is estimated as a straight line or vector passing through the eye's reference point and through the estimated gaze point on the surface.

17. The method of claim 1, wherein: The reference points of the eye are: the centre of corneal curvature estimated based on said further image; or an average value of the centre of corneal curvature estimated based on said images in said set of images; or a projection of the center of corneal curvature estimated based on the other image in a corneal center plane, wherein the corneal center plane is estimated based on the center of corneal curvature estimated from the image in the set of images; or A center of the eye is estimated based on the image in the set of images and / or based on the further image.

18. A gaze tracking system comprising a processing circuit system configured to: obtaining a set of images of the eye captured by a camera; For each image from the set of images: obtaining gaze data indicating a point at which the eye was looking when the image was captured and / or a direction in which the eye was looking when the image was captured, and estimating the position of the center of the pupil of the eye in the image; calibrating a mapping using the obtained gaze data and the estimated position of the center of the pupil, wherein the mapping is adapted to map the position of the center of the pupil of the eye in an image captured by the camera to: the point at which the eye is looking on the surface it is looking at, or the gaze direction of the eyes; obtaining another image of the eye captured by the camera while the eye is illuminated by the illuminator; estimating a position of a reflection of the illuminator at the cornea of ​​the eye in the further image; estimating the position of the center of the pupil in the other image; and performing gaze tracking using the calibrated map and an estimated position of a center of the pupil in the other image; The processing circuitry is further configured to: estimating whether the eye is directed toward a central area of ​​the eye's field of view by using the estimated position of the reflection of the illuminator and the estimated position of the center of the pupil in the other image; and In response to the eye being directed toward an area outside of the central area, gaze data determined by performing gaze tracking using the calibrated map and the estimated position of the center of the pupil in the other image is output.

19. The system of claim 18, wherein: The mapping is adapted to map the location of the center of the pupil of the eye in the image captured by the camera to a gaze point of the eye on the surface, and wherein the processing circuitry is configured to perform gaze tracking using the calibrated mapping and the estimated location of the center of the pupil in the other image by at least: estimating a gaze point on the surface by applying the mapping to an estimated position of a center of the pupil in the other image; and The gaze direction is estimated as a straight line or vector passing through the eye's reference point and through the estimated gaze point on the surface.

20. A head mounted device comprising the system of claim 18.

21. A non-transitory computer-readable storage medium storing instructions, which, when executed by a gaze tracking system, cause the gaze tracking system to: obtaining a set of images of the eye captured by a camera; For each image from the set of images: obtaining gaze data indicating a point at which the eye was looking when the image was captured and / or a direction in which the eye was looking when the image was captured, and estimating the position of the center of the pupil of the eye in the image; calibrating a mapping using the obtained gaze data and the estimated position of the center of the pupil, wherein the mapping is adapted to map the position of the center of the pupil of the eye in an image captured by the camera to: the point at which the eye is looking on the surface it is looking at, or the gaze direction of the eye; obtaining another image of the eye captured by the camera while the eye is illuminated by the illuminator; estimating a position of a reflection of the illuminator at the cornea of ​​the eye in the further image; estimating the position of the center of the pupil in the other image; and performing gaze tracking using the calibrated map and an estimated position of a center of the pupil in the other image; The gaze tracking system further performs the following steps: estimating whether the eye is facing towards a central area of ​​the eye's field of view by using the estimated position of the reflection of the illuminator and the estimated position of the center of the pupil in the other image; and In response to the eye being directed toward an area outside of the central area, gaze data determined by performing gaze tracking using the calibrated map and the estimated position of the center of the pupil in the other image is output.

Images