Systems and methods for determining reference gaze data

Abstract

An eye-tracking system includes processing circuitry configured to acquire a first eye image captured during a first time period, including a user's pupil; determine a first pupil size based on the first eye image; acquire a second eye image captured during a second time period; determine a second pupil size based on the second eye image; acquire scene information of the scene in which the pupil is exposed, including a first brightness level, a second brightness level, and spatial information of a second region within the second time period; determine a pupil size change between the first and second pupil sizes, indicating that the user is looking at the second region; and determine reference gaze data of the user during the second time period if the pupil size change is greater than a pupil size change threshold. The invention also relates to a head-mounted device, method, computer program, and carrier.

Description

Technical Field

[0001] This disclosure generally relates to the field of eye tracking. Specifically, this disclosure relates to systems and methods for determining reference gaze data of a user while exposing the user's pupil. Background Technology

[0002] Eye / gaze tracking is the process of measuring eye movement relative to the head or a point of gaze. An eye tracker is a device used to measure eye position and eye movement. Eye trackers are used in many different applications. Several different eye tracking systems are known in the art. For example, a system could be employed to allow a user to indicate the location of a point on a computer monitor by looking at it. An eye tracking system can capture an image of the user's face and then use image processing to extract key features from the user's face, such as the pupil center and potential flashes from a light source illuminating the user's face. The extracted features can then be used to determine where the user is looking on the display.

[0003] To provide users with an accurate and precise eye-tracking experience, the eye tracker needs to be calibrated. This is typically done by looking at a given set of reference points over a certain amount of time and acquiring the user's eye data related to the user's gaze point or gaze line. Calibration can also be performed continuously as the user uses the eye tracker. Furthermore, users may want to interact with the object they are looking at by using their gaze. Additionally, gaze can be used to provide a better experience for the user by implementing foveated rendering. For all of the above purposes, it is important to be able to determine when the user is actually looking at the calibration target or interactive object. Current solutions include using external input devices such as a mouse, keyboard, microphone, or clicking a button. Alternatively, looking at the same target or determining the minimum amount of time a blink occurs can be used as an indication that the user is looking at the intended location. However, external input is often slow and cumbersome for users, and gaze determination can become less accurate without it.

[0004] Therefore, there is a need to find a faster, less cumbersome, and more accurate way to determine where a user is looking. Summary of the Invention

[0005] According to a first aspect, an eye-tracking system is provided for determining reference gaze data of a user in the case of an exposed user's pupil, the scene including at least a first region and a second region having a first brightness level, the second region having a second brightness level that differs from the first brightness level by a brightness level threshold, the eye-tracking system including a processing circuit configured to:

[0006] A first eye image including the user's pupil is obtained, the first eye image being captured during a first time period;

[0007] Determine the first pupil size based on the first eye image;

[0008] A second eye image, including the user's pupil, is obtained, the second eye image being captured during a second time period;

[0009] Determine the size of the second pupil based on the second eye image;

[0010] Acquire scene information of the scene in which the user's pupil is exposed, the scene information including at least a first brightness level, a second brightness level, and spatial information of a second region during a second time period;

[0011] Determine the pupil size change between the first pupil size and the second pupil size, wherein the pupil size change indicates that the user is looking at the second region;

[0012] If the pupil size change is greater than the pupil size change threshold, then the user's reference fixation data for the second time period is determined.

[0013] The advantage of determining a user's reference gaze data based on pupil size changes is that it doesn't require external input to determine if the user is looking at a second region, while still maintaining high accuracy. Therefore, it's possible to determine if the user is looking at a second region without external input indicating where they are looking. Reference gaze data can be used as input during the calibration process. Another advantage in this example is that it's not necessary to instruct the user when they are looking at the first calibration point, as the eye-tracking system determines the reference gaze data only when the user is looking at the second region (which is the first calibration point in this example). Furthermore, reference gaze data can be used for interaction with the object the user is looking at. Again, in this example, the advantage is that the reference gaze data will only be determined when the user is looking at the second region, which in this example is an interactive object. In a broader interpretation, gaze data can include confirmation that the user is looking at a second region / interactive object. Additionally, gaze data can include information about where the user is looking, such as the gaze point, gaze line, or focus point / region. Reference gaze data can also be used for gaze rendering, i.e., rendering different parts of the display differently depending on where the user is looking.

[0014] In one example, the processing circuitry is further configured to obtain the position of the pupil during the second time period; and wherein the determination of the user's reference gaze data is based at least in part on the obtained pupil position. In this example, the pupil position can be determined from the pupil position in a second eye image, but it can also be determined from other eye images or by other means used to determine the pupil position during the second time period. In this example, the gaze data includes information about where the user is looking, such as the gaze point, gaze line, or point of interest / region.

[0015] In another example, the pupil position is obtained at least in part based on the pupil position in a second eye image. Therefore, the second eye image is used both to determine pupil size changes and to determine reference gaze data. In this example, gaze data includes information about where the user is looking, such as the gaze point, gaze line, or point of interest / region.

[0016] In another example, the determination of the user's reference gaze data is based at least in part on the acquired scene information. By taking scene information into account, gaze data, such as gaze points or gaze lines, can be determined with greater precision because the scene information includes spatial information (such as location) of the second region during a second time period. Therefore, if the pupil position indicates a gaze point / line that is inconsistent with the location of the second region, calibration can be performed based on the location of the second region. The reference gaze data can be set to be consistent with the location of the second region. Alternatively, the reference gaze data can be set to be consistent with the location between the location of the second region and the gaze point / line indicated by the pupil position.

[0017] Furthermore, the processing circuitry can be configured to associate reference gaze data with a second region based on the acquired scene information. Associating a gaze location / line with a region / object is called gaze-object mapping. Gaze-object mapping can be used to determine which object to interact with and / or whether interaction with a particular object should be permitted.

[0018] Furthermore, the processing circuitry can be configured to: determine the distance difference between the reference gaze data and the spatial information of the second region; and if the distance difference exceeds an error threshold, send an error signal to the user. The error signal can indicate what is wrong with the eye-tracking system. For example, if the eye-tracking system is included in a head-mounted device, the error signal can indicate that the head-mounted device is slipping and that its position on the user's head needs to be corrected. In another example, there may be a hardware error in the eye image sensor or another component of the eye-tracking system, such as the display or the position of the display relative to the eye image sensor or the user's pupil. There may also be unexpected problems related to the function of the light emitters in the eye-tracking system or related to external lighting, which may cause unexpected reflections that could lead to uncertainty in the determination of the gaze data.

[0019] In another example, the processing circuitry is also configured to update the user's gaze calibration settings based on reference gaze data and scene information. Therefore, the reference gaze data can be set to correspond to the position of the second region and the position between the gaze point / line indicated by the pupil position. Alternatively, the reference gaze data can be set to correspond to the position of the second region. In other words, when reference gaze data is obtained, the user's gaze calibration settings are updated based on the user's awareness that they are looking at the second region.

[0020] In yet another example, the reference gaze data includes gaze convergence depth of field, and the spatial information of the second region includes depth information, and updating the user's gaze calibration settings also includes updating the user's gaze convergence depth of field based on the depth information. Therefore, the reference gaze data can be set to be consistent with the gaze convergence depth of field between the depth of field of the second region and the user's gaze convergence depth of field. Alternatively, the gaze convergence depth of field can be set to the depth of field of the second region. In other words, when the gaze convergence depth of field is obtained, the user's gaze calibration settings are updated based on the user's awareness that they are looking at the second region.

[0021] In another example, the processing circuitry is further configured to: obtain a first total light level of the user's pupil exposed during the first time period; obtain a second total light level of the user's pupil exposed during the second time period; determine the total light level change between the first and second total light levels; and determine reference fixation data only if the total light level change is less than a total light level change threshold. Therefore, a check is introduced to verify that pupil size changes are not caused by changes in the total light level of the exposed pupil.

[0022] In yet another example, the processing circuitry is further configured to: obtain a first scene image exposing the user's pupil during the first time period; obtain a second scene image exposing the user's pupil during the second time period; determine a scene change between the first and second scene images; and determine the user's reference gaze data only if the scene change is less than a scene change threshold. Therefore, a check is introduced to verify that the pupil size change is not caused by a change in the scene image exposing the pupil.

[0023] In yet another example, the processing circuitry is further configured to: determine a first pupil position based on the first eye image; determine a second pupil position based on the second eye image; determine a pupil position change between the first and second pupil positions; and determine the user's reference gaze data only if the pupil position change is greater than a pupil position change threshold. Therefore, a check is introduced to verify that the pupil size change is caused by a pupil position change between a first time period and a second time period. If the scene exposing the user's pupil remains the same or at least similar between the first and second time periods, a pupil position change is required to cause a pupil size change. This is due to the local brightness level, i.e., where the user is looking, or the perceived brightness level, rather than due to the global brightness level, i.e., the total brightness level of the exposed pupil.

[0024] In addition, reference gaze data may include at least one of a gaze point and a gaze line. A gaze point is a 2D point on a surface such as a 2D display. A gaze line is a 3D line originating from the user's eye / pupil that passes through a surface such as a 2D display or through space such as a 3D display or a semi-transparent display.

[0025] Furthermore, the eye-tracking system may include a display configured to show at least a portion of a scene that exposes the user's pupils. The display may be a 2D display positioned within a head-mounted device or located at a distance from the user (i.e., not attached to the user). The display can be any type of display, such as OLED or LED. Alternatively, the display may be a 3D display positioned within a head-mounted device or located at a distance from the user (i.e., not attached to the user). Furthermore, the display may be semi-transparent, such as a display used for extended reality or XR applications, or non-transparent, such as a standard desktop display or a display used for virtual reality or VR applications.

[0026] Furthermore, the eye-tracking system may include an outward-facing scene sensor arranged to acquire at least a portion of the scene exposing the user's pupil. By using the outward-facing scene sensor, information about the real world can be obtained. This information about the real world is necessary in extended reality, XR, or real-world applications where the eye-tracking system does not include a display.

[0027] Furthermore, the eye-tracking system may include an eye image sensor arranged to capture a first eye image and a second eye image. Alternatively, the first eye image and the second eye image may be captured by one or more external eye image sensors. In any case, the processing circuitry is configured to obtain the first eye image and the second eye image by acquiring images from the eye image sensor of the eye-tracking system or from one or more external eye image sensors.

[0028] Furthermore, the eye-tracking system may include at least one emitter, each emitter being arranged to cause a flash in the user's eye, the flash being detectable in the second eye image, wherein the determination of the user's reference gaze data is based at least in part on the position of the at least one flash relative to the pupil. This technique is known as central pupillary corneal reflex, PCCR, eye tracking, and is a well-known technique for improving the accuracy of gaze data determination.

[0029] According to a second aspect, a head-mounted device is provided, the head-mounted device including an eye-tracking system for determining reference gaze data of a user in a scene exposing the pupil of a user, the scene including at least a first region having a first brightness level and a second region having a second brightness level, the second brightness level differing from the first brightness level by a brightness level threshold, the eye-tracking system including processing circuitry configured to:

[0030] A first eye image including the user's pupil is obtained, the first eye image being captured during a first time period;

[0031] Determine the first pupil size based on the first eye image;

[0032] A second eye image, including the user's pupil, is obtained, the second eye image being captured during a second time period;

[0033] Determine the size of the second pupil based on the second eye image;

[0034] Acquire scene information of the scene in which the user's pupil is exposed, the scene information including at least a first brightness level, a second brightness level, and spatial information of a second region during a second time period;

[0035] Determine the pupil size change between the first pupil size and the second pupil size, wherein the pupil size change indicates that the user is looking at the second region;

[0036] If the pupil size change is greater than the pupil size change threshold, then the user's reference fixation data for the second time period is determined.

[0037] The second aspect may include any additional features described in relation to the first aspect.

[0038] According to a third aspect, a method performed by an eye-tracking system is provided for determining reference gaze data of a user in a scene exposing the pupil of a user, the scene including at least a first region having a first brightness level and a second region having a second brightness level, the second brightness level differing from the first brightness level by a brightness level threshold, the method comprising the following steps:

[0039] A first eye image including the user's pupil is obtained, the first eye image being captured during a first time period;

[0040] Determine the first pupil size based on the first eye image;

[0041] A second eye image, including the user's pupil, is obtained, the second eye image being captured during a second time period;

[0042] Determine the second pupil size based on the second eye image;

[0043] Acquire scene information of the scene in which the user's pupil is exposed, the scene information including at least a first brightness level, a second brightness level, and spatial information of a second region during a second time period;

[0044] Determine the pupil size change between the first pupil size and the second pupil size, wherein the pupil size change indicates that the user is looking at the second region;

[0045] If the pupil size change is greater than the pupil size change threshold, then the user's reference fixation data for the second time period is determined.

[0046] The third aspect may include any additional features described in relation to the first aspect.

[0047] According to a fourth aspect, a computer program including computer-readable code means is provided, the computer-readable code means being intended to run in an eye-tracking system to determine reference gaze data of a user in a scene exposing the pupil of a user, the scene including at least a first region having a first brightness level and a second region having a second brightness level, the second brightness level differing from the first brightness level by a brightness level threshold, wherein the computer-readable code means, when running in the system, causes the system to perform the following steps:

[0048] A first eye image including the user's pupil is obtained, the first eye image being captured during a first time period;

[0049] Determine the first pupil size based on the first eye image;

[0050] A second eye image, including the user's pupil, is obtained, the second eye image being captured during a second time period;

[0051] Determine the second pupil size based on the second eye image;

[0052] Acquire scene information of the scene in which the user's pupil is exposed, the scene information including at least a first brightness level, a second brightness level, and spatial information of a second region during a second time period;

[0053] Determine the pupil size change between the first pupil size and the second pupil size, wherein the pupil size change indicates that the user is looking at the second region;

[0054] If the pupil size change is greater than the pupil size change threshold, then the user's reference fixation data for the second time period is determined.

[0055] The fourth aspect may include any additional features described in relation to the first aspect.

[0056] According to the fifth aspect, a carrier comprising the computer program according to the fourth aspect is provided. Attached Figure Description

[0057] Specific embodiments will now be described in detail with reference to the accompanying drawings, in which:

[0058] Figure 1 This is a block diagram of an eye-tracking system;

[0059] Figure 2A A cross-sectional view of the user's eye is shown;

[0060] Figure 2B An example of the front view of the user's eye is shown;

[0061] Figure 3 illustrates the effect of visual awareness on pupillary light response according to existing technology;

[0062] Figure 4A and Figure 4B An example of eye images captured during two different time periods is shown, where pupil position is ignored;

[0063] Figure 4C It shows Figure 4A and Figure 4B How an eye image is associated with a scene comprising two regions.

[0064] Figure 5A and Figure 5B An example of eye images captured during two different time periods is shown, taking pupil position into account;

[0065] Figure 5C It shows Figure 5A and Figure 5B How an eye image is associated with a scene comprising two regions.

[0066] Figure 6 An example is shown of eye images captured during two different time periods and how these eye images are correlated with a scene comprising three regions.

[0067] Figure 7 An example of a 3D view including multiple objects at different depths of field is shown;

[0068] Figure 8 A method for determining a user's reference gaze data is shown;

[0069] Figure 9A A head-mounted device according to one or more embodiments is shown;

[0070] Figure 9B A remote display system according to one or more embodiments is shown. Detailed Implementation

[0071] The term "gaze data" should be broadly interpreted as data describing where a user is looking or what object / area the user is focusing on. When knowing the object / area of ​​focus is sufficient, gaze data can refer to that object or area. When knowing more precisely where the user is looking is crucial, gaze data can refer to the gaze point, i.e., a 2D point on a plane or display, or a gaze line or gaze vector, i.e., a 3D line or vector extending from, for example, the eye, the center of the eyeball, or the pupil, through the gaze point on the plane or display. Furthermore, gaze data can also include information about the user's gaze convergence depth of field, i.e., the depth of field at which the user's attention or focus is located and / or the depth of field at which the gaze lines of the left and right eyes coincide.

[0072] The term "reference gaze data" should be interpreted as gaze data determined based on pupil data and / or eye images obtained during periods when the user's pupils are exposed to changes in local or perceived brightness levels. During these periods, pupil size changes are used to aid in determining what area or object the user is looking at.

[0073] The term "luminance level" refers to a range of luminous emission or light intensity. This disclosure describes a first luminance level and a second luminance level. In the broadest definition, it is sufficient that a measurable difference exists between the first luminance level and the second luminance level. Furthermore, the ranges of the first luminance level and the second luminance level may be separated by a range of luminous emission or light intensity, such that a safety margin can exist between the ranges of luminous emission or light intensity of the first luminance level and the second luminance level.

[0074] The term "scene" should be interpreted as anything that exposes a user's pupils. This scene can include a display, or a display provided for each eye, to expose the user's pupils. However, a scene can also include the real world that exposes a user's pupils.

[0075] The term "scene information" should be interpreted as information about anything that exposes the user's pupil. Scene information may include the brightness or light level of the area or object exposing the user's pupil. Furthermore, scene information may include spatial information, such as the location and / or size of the area or object. This location can be interpreted as x / y coordinates on a plane, surface, or display. Scene information may also include depth information about the area or object in a relative sense, such as relative to the user's pupil, or depth information in an absolute sense, such as 3D location world space.

[0076] Figure 1 This is a block diagram of eye-tracking system 10. Eye-tracking system 10 is capable of determining the user's gaze data. (Reference) Figure 2A and Figure 2BThe user 230's eye 100 is described. The eye-tracking system 10 includes processing circuitry 11 configured to process eye images and determine gaze data, at least in part, based on gaze data. Furthermore, the eye-tracking system 10 may include a display 12. Additionally, the eye-tracking system 10 may include an eye image sensor 13 arranged to capture eye images. Furthermore, the eye image sensor 13 may be configured to determine the light level or brightness level exposing the user 230 or the user 230's pupil 102. The eye image sensor 13 can be any type of imaging device. However, the eye-tracking system 10 may also include a light sensor arranged to determine the light level or brightness level exposing the user 230 or the user 230's pupil 102. Furthermore, the eye-tracking system 10 may include at least one emitter 14 illuminating at least one of the user's eyes. Additionally, the eye-tracking system 10 may include a graphics processing device 15 configured to process graphics, for example, presented by the display 12. The display 12 can be any type of display, such as OLED or LED. Furthermore, the graphics processing device 15 may be configured to determine the light level or brightness level of the display 12, which can be used to determine the light level or brightness level that exposes the user 230 or the user 230's pupil 102. Additionally, the eye-tracking system 10 may include an outward-facing scene sensor 16 configured to capture information about the scene 20 exposing the user 230 and / or the user 230's pupil 102. The scene sensor 16 can be any type of imaging device. By using the outward-facing scene sensor 16, information about the real world can be obtained. Information about the real world is necessary in extended reality, XR, applications, or real-world applications where the eye-tracking system 10 does not include any display. The outward-facing scene sensor 16 can be an image sensor, such as a camera or a LiDAR-based sensor. Furthermore, the outward-facing scene sensor 16 may be arranged to determine the light level or brightness level that exposes the user 230 or the user 230's pupil 102.

[0077] Figure 2AA cross-sectional view of the eye 100 of the user 230 of the eye-tracking system 10 is shown. The eye 100 has a cornea 101 and a pupil 102 with a pupil center 103. The cornea 101 is curved and has a center of curvature 104, which is referred to as the corneal curvature center 104, or simply the corneal center 104. The radius of curvature of the cornea 101 is referred to as the radius 105 of the cornea 101, or simply the corneal radius 105. The eye 100 has a center 106, which is also referred to as the center of the eyeball 106, or simply the center of the eyeball 106. The visual axis 107 of the eye 100 passes through the center 106 of the eye 100 and reaches the fovea 108 of the eye 100. The optical axis 110 of the eye 100 passes through the pupil center 103 and the center 106 of the eye 100. The visual axis 107 forms an angle 109 with respect to the optical axis 110. The deviation or offset between the visual axis 107 and the optical axis 110 is commonly referred to as foveal offset 109. During the calibration of the eye-tracking system 10, the foveal offset 109 can be determined for one or more fixated targets. Figure 2A In the example shown, eye 100 is looking toward display 111, and eye 100 is fixating on fixation point 112 on display 111. Visual axis 107 can be considered as a gaze line passing through a point in eye 100 and fixation point 112. However, a gaze line can also be defined for a combination of left and right eyes, passing through a point between the left and right eyes and fixation point 112.

[0078] Figure 2B An example of a front view of the eyes 100 of a user 230 in an eye-tracking system 10 is shown. Figure 2B This is an example of a user 230's eye image. Figure 2BAn eye 100, cornea 101, pupil 102, and iris 113 are disclosed. Furthermore, in this example, the eye 100 has a reflection 115 at the cornea 101, which is caused by a light emitter 114. This reflection 115 is also referred to as a flash 115. The light emitter 114 can be any light source, such as an LED or a laser. The processing circuitry 11 can employ image processing (e.g., digital image processing) to extract features from the image. The processing circuitry 11 can employ, for example, central pupillary corneal reflection, PCCR, and eye tracking to determine where the eye 100 is looking. In PCCR eye tracking, the processing circuitry 11 estimates the position of the pupil 102 or the center 103 of the pupil 102, and the position of the flash 115 or the center of the flash 115 at the eye 100. The processing circuitry 11 uses the flash 115 to calculate the user's position in space and uses the position of the pupil 102 to calculate the direction the user's eye 100 is facing. Eye tracking can also be performed on the eye image without the flash 115. For both PCCR and non-PCCR eye tracking, machine learning can be applied to increase the accuracy of gaze position estimation. Since there is typically an offset between the optical axis 110 and the visual axis 107, the processing circuitry 11 performs a foveal offset 109 calibration to determine where the user is looking. The gaze point 112, or gaze line, can be determined based on one or both eyes of the user of the eye-tracking system 10. In the latter case, the gaze lines obtained from the left and right eyes, respectively, can be combined to form a combined estimated gaze point 112 or gaze line.

[0079] The pupil responds to many different stimuli, such as ambient light, changes in depth of field, and cognitive triggers, such as emotional arousal or cognitive load. However, the pupil responds not only to global brightness changes but also to local and perceived brightness levels. Therefore, when a user shifts their gaze from a dark object to a bright object within the same scene, the pupil constricts even before the gaze lands on the target. This is illustrated in Figure 3, which shows that since 2015… This was discovered in the article "New Light on the Mind's Eye: The Pupillary Light Response as ActiveVision" by [author's name]. Historically, pupillary light response was described as a low-level reflex devoid of any cognitive component. Recent research, which has significantly altered this view, is reviewed: the light response depends not only on the brightness of the stimulus, but also on whether you are aware of the stimulus, whether you notice it, and even whether you think about it. The authors emphasize the link between pupillary light response and eye movement preparation: when you intend to look at a bright stimulus, pupillary constriction prepares along with eye movement before the eye begins to move. This preparation allows the pupil to rapidly change its size as the eye moves from a bright object to a dark object and back again. Figure 3 illustrates the effect of visual awareness on pupillary light response. When awareness switches from a dark stimulus to a bright stimulus, the pupil constricts (brighter line). Conversely, when awareness switches from a bright stimulus to a dark stimulus, the pupil dilates (darker line). Error bands represent standard error.

[0080] Figures 4-7 illustrate how information about changes in pupil 102 due to local and perceived brightness levels is used to determine different aspects of reference gaze data for user 230. Gaze data can generally be acquired and / or determined at a much higher frequency. As an example only, an eye tracker can determine gaze data 50 to 500 times per second. However, the term reference gaze data should be interpreted as gaze data determined based on pupil data and / or eye images acquired during periods of exposure of the user's pupils to changes in local or perceived brightness levels. During these periods, pupil size changes serve as an aid in determining what area or object the user is looking at.

[0081] The eye-tracking system 10 is capable of determining reference gaze data for the user 230 of the eye-tracking system 10. The processing circuit 11 is configured to acquire an eye image of the user 230, including the pupil 102 of the user 230. Furthermore, the processing circuit 11 is configured to determine the pupil size of the user 230's pupil 102 based on the eye image. Additionally, the processing circuit 11 is configured to acquire scene information of a scene 20 exposing the user 230's pupil 102. Scene 20 includes at least a first region 30 having a first brightness level and a second region 40 having a second brightness level, the second brightness level differing from the first brightness level by a brightness level threshold. The scene information of scene 20 includes at least the first brightness level, the second brightness level, and spatial information of the second region 40 during a second time period. Scene 20 may include a display 12, or one display for each eye, to expose the user 230's pupil 102. However, scene 20 may also include real-world information exposing the user 230's pupil 102. In short, scene 20 can include anything that exposes the pupil 102 of user 230. The second time period represents the period during which user 230 has switched focus from the first region 30 to the second region 40 and therefore user 230 has been exposed to a change in local brightness level. Furthermore, processing circuitry 11 is configured to determine the pupil size change between the first and second pupil sizes. The pupil size change indicates that user 230 is looking at the second region 40. Additionally, processing circuitry 11 is configured to determine reference fixation data for user 230 during the second time period if the pupil size change is greater than a pupil size change threshold.

[0082] Pupil size can be defined as the radius, diameter, and / or area of ​​the pupil. Furthermore, pupil size can be determined using relative terms, such as the radius, diameter, and / or area of ​​pupil 102 in an eye image. Alternatively, pupil size can be determined using absolute terms, such as the actual radius, diameter, and / or area of ​​pupil 102. A pupil size change threshold can be associated with any of the above definitions. If the pupil size change is below the pupil size change threshold, the reference fixation data is not determined.

[0083] In one example, processing circuitry 11 is configured to obtain the position of the pupil during a second time period t2, and the determination of reference gaze data for user 230 is based at least in part on the obtained position of pupil 102. It is not necessary that the position of pupil 102 has already been determined from one of the eye images. The position of pupil 102 during the second time period t2 can also be obtained in another manner. One option is that the position coordinates of pupil 102 during the second time period t2 have already been obtained by processing circuitry 11. Another option is to use another eye image captured during the second time period t2 to obtain the position of pupil 102 during the second time period t2.

[0084] In the following example, a first time period t1 and a second time period t2 are described. The first time period t1 occurs before the pupil size change, and the second time period t2 occurs after the pupil size change. Therefore, the second time period t2 follows the first time period t1. In one example, the second time period t2 arrives immediately after the first time period t1. In another example, there is a time gap between the first time period t1 and the second time period t2. Each of the first time period t1 and the second time period t2 can have any duration. In one example, the first time period t1 and the second time period t2 are the reciprocals of the frequency of the eye tracker, i.e., each of the first time period t1 and the second time period t2 includes only one eye image. However, each of the first time period t1 and the second time period t2 can include several consecutive eye images, where each of the first eye image 35 and the second eye image 45 is only one eye image among consecutive eye images within that time period. It is also possible that each of the first eye image 35 and the second eye image 45 represents a combination of multiple eye images within the corresponding time period. Each of the first and second time periods can be of any duration from milliseconds to seconds.

[0085] Figure 4A and Figure 4B Examples of eye images 35 and 45 are shown during two different time periods, where pupil position is ignored. Furthermore, Figure 4A A first eye image 35 of a user 230's eye 100, captured during a first time period t1, is shown. The first eye image 35 includes the pupil 102 of the user 230. The first eye image 35 may also include the iris 113 of the eye 100. Since the size of the iris 113 is constant, it facilitates the determination of the pupil size. It can be seen that the first eye image 35 does not include any other eye features. This is to illustrate that, in the widest embodiment of the invention, it is not necessary to determine the pupil position of the first eye image 35. Instead, the first pupil size is sufficient to be determined based on the first eye image 35.

[0086] Figure 4B A second eye image 45 of user 230's eye 100, captured during a second time period t2, is shown. The second time period t2 follows the first time period t1. Furthermore, the second eye image 45 of eye 100 includes the pupil 102 of user 230. Additionally, the second eye image 45 may also include the iris 113 of eye 100. As can be seen, the second eye image 45 does not include any other eye features. This is to illustrate that, in the widest embodiment of the invention, it is not necessary to determine the pupil position of the second eye image 45. Instead, it is sufficient to determine the second pupil size based on the second eye image 45.

[0087] Figure 4C It shows Figure 4A and Figure 4B The first and second eye images 35 and 45 are related to scene 20, which includes two regions, during two different time periods. As described above, scene 20 includes at least a first region 30 with a first brightness level and a second region 40 with a second brightness level, the second brightness level differing from the first brightness level by a brightness level threshold. In this example, the second region 40 is the right half of scene 20, while the first region 30 is the left half of the scene. However, each of the first region 30 and the second region 40 can have any other shape and / or size. However, if the second region 40 is used for calibration purposes, it is advantageous that the second region 40 is small in size or has an indicator at its center, so that it can be assumed with high probability where the user is looking when the pupil size changes. In this example, scene 20 is constant during the first and second time periods t1 and t2. That is, the first region 30 and the second region 40 do not change during the first time period t1 and the second time period t2. Furthermore, the first brightness level is higher than the second brightness level. Because the local brightness level in the second region 40 is lower than that in the first region 30, the pupil size will increase when the user 230 looks at the second region 40 after viewing the first region 30. Therefore, it is possible to determine whether the user 230 is looking at the brighter first region 30 or the darker second region 40 simply by observing the pupil size of the user 230. Since the pupil size is larger during the second time period t2, it is obvious that the user is looking at the darker second region 40 during the second time period t2. It can also be assumed that the user 230 is looking at the brighter first region 30 during the first time period t1. Therefore, the reference gaze data can be set as the gaze point within the second region 40. In addition, the reference gaze data can be set as a gaze line extending through the second region 40. Therefore, in both cases, the determination of the user's reference gaze data is at least partially based on the obtained scene information. In this case, the obtained scene information is represented by the position of the second region 40. Furthermore, the reference gaze data is associated with the second region based on the obtained scene information because no additional gaze point or gaze line is determined based on the pupil position in the eye image. This information can be used to interact with objects in or within the second region 40. Associating a gaze location / line with a region / object is called gaze-to-object mapping.

[0088] In the example above, the first brightness level is higher than the second brightness level. However, the opposite can also apply; that is, if the first brightness level is lower than the second brightness level, then when user 230 looks at the second region 40 after looking at the first region 30, the pupil size will decrease.

[0089] Figure 5A and Figure 5BExamples of eye images 35 and 45 are shown during two different time periods, taking pupil position into account. (As above...) Figure 4A In that way, Figure 5A A first eye image 35 of user 230's eye 100, captured during a first time period t1, is shown. The first eye image 35 includes the pupil 102 of user 230. The first eye image 35 may also include the iris 113 of eye 100. Since the size of the iris 113 is constant, the iris 113 facilitates the determination of the pupil size. Additionally, Figure 5A Further details of the eye 100 are shown, allowing the fixation point or fixation line to be determined based on the first eye image 35. Furthermore, the first pupil size can be determined based on the first eye image 35.

[0090] In addition, as mentioned above Figure 4B As shown, Figure 5B A second eye image 45 of user 230's eye 100, captured during a second time period t2, is shown. The second time period t2 follows the first time period t1. Furthermore, the second eye image 45 of eye 100 includes the pupil 102 of user 230. Additionally, the second eye image 45 may also include the iris 113 of eye 100. Figure 5B Further details of the eye 100 are shown, allowing the fixation point or fixation line to be determined based on the second eye image 45. Furthermore, the second pupil size can be determined based on the second eye image 45.

[0091] also, Figure 5C It shows Figure 5A and 5B How do eye images relate to scene 20, which includes two regions, during two different time periods? Similar to... Figure 4C , Figure 5C It shows in Figure 5A and 5BIn this case, how do the first and second eye images 35 and 45 relate to scene 20, which includes two regions, during two different time periods? As described above, scene 20 includes at least a first region 30 with a first brightness level and a second region 40 with a second brightness level, the second brightness level differing from the first brightness level by a brightness level threshold. In this example, the second region 40 is a smaller region indicated by a circle or dot. However, the second region 40 can have any other shape and / or size. However, if the second region 40 is used for calibration purposes, it is advantageous that the second region 40 is smaller or has an indication at its center, making it possible to assume with high probability where the user is looking when the pupil size changes. In this example, scene 20 is constant during the first and second time periods t1 and t2. That is, the first region 30 and the second region 40 do not change during the first time period t1 and the second time period t2. Furthermore, the first brightness level is lower than the second brightness level. Since the local brightness level in the second region 40 is higher than that in the first region 30, the pupil size will decrease when the user 230 looks at the second region 40 after looking at the first region 30. Therefore, simply by observing the pupil size of user 230, it is possible to determine whether user 230 has switched their attention (i.e., fixation) to the brighter second region 40. Since the pupil size is smaller during the second time period t2, it is clear that the user is looking at the second region 40. It can also be assumed that user 230 is looking at the brighter first region 30 during the first time period t1.

[0092] In addition, due to Figure 5A and Figure 5B Including additional information about the pupil position in relation to other features in or around the eye 100, the gaze point or gaze line in scene 20 can be determined based on the first eye image 35 and the second eye image 45, respectively. That is, the first gaze point 212 can be determined based on the first eye image 35. t1 Or through the first fixation point 212 t1 The first gaze line. Furthermore, based on the second eye image 45, the second gaze point 212 can be determined. t2 Or through the second gaze point 212 t2The second gaze line. Eye images may include the cornea 101 and / or the entire eye 100 and / or even other facial features such as eyelashes, eyebrows, or nose, which allow for determination that the pupil 102 has moved. However, if an eye image is captured from a fixed position relative to the eye, it is sufficient to track the position of the eye 102 to determine the gaze point 212 or gaze line. Furthermore, there are other ways to improve the determination of reference gaze data (such as the gaze point 212 or gaze line) based on machine learning and / or flash 115. Additionally, 3D models can be used to determine the gaze point 212 or gaze line based on each of the eye images 35, 45. However, details of the different ways of determining gaze data based on eye images will not be given here. Instead, the explanation here will remain at a higher level. By comparison... Figures 5A to 5C The first and second eye images 35 and 45 clearly show that not only has the pupil size changed, but the pupil position within eye 100 has also shifted to the right. Therefore, in addition to determining that the pupil size has changed (in this case, pupil 102 has become smaller), it is clear that user 230 is looking further to the right in scene 20, where a brighter second region 40 exists. This reference gaze data matches the user looking at the second region 40 during the second time period t2. However, the reference gaze data is not only used to confirm that the user is actually looking at the second region 40.

[0093] The reference gaze data may be based on gaze data determined from the second eye image 45. However, the reference gaze data may also be based on other data, such as the average of earlier and / or later gaze data obtained from eye images captured during the second time period t2. Furthermore, the reference gaze data may be based on the location of the second region 40 during the second time period t2.

[0094] For example, the processing circuit 11 can also be configured to associate reference gaze data with the second region based on the acquired scene information. That is, if the reference gaze data from the position of the pupil 102 in the second eye image 45 determines that the gaze point is in the second region 40 or the gaze line coincides with the second region 40, the user can interact with the second region 40 or objects within the second region 40.

[0095] In another example, processing circuitry 11 may also be configured to determine the distance difference between the reference gaze data and the position of the second region 40, and to send an error signal to the user if the distance difference exceeds an error threshold. The error signal can indicate what is wrong with the eye-tracking system 10. For example, if the eye-tracking system 10 is included in a head-mounted device, the error signal could indicate that the head-mounted device is slipping and needs correction. In another example, there may be a hardware error, such as with the eye image sensor 13 or another component of the eye-tracking system 10 (e.g., the display 12 or the position of the display 12 relative to the eye image sensor 13 or the pupil 102 of the user 230). There may also be unexpected problems related to the function of the emitter 14 of the eye-tracking system 10 or related to external lighting, which could cause unexpected reflections that could lead to uncertainty in the determination of the gaze data.

[0096] In yet another example, processing circuitry 11 is also configured to update the gaze calibration settings of user 230 based on reference gaze data and scene information. In this case, the position of the second region 40 can be used to update the gaze calibration settings of user 230. Therefore, the reference gaze data can be set to correspond to the position of the second region 40 and the position between the gaze point / line indicated by the pupil position. Alternatively, the reference gaze data can be set to correspond to the position of the second region 40. In other words, when reference gaze data is obtained, the user's gaze calibration settings are updated based on the user's perception that they are looking at the second region 40.

[0097] Figure 6 An example is shown of first and second eye images 35 and 45 during first and second time periods t1 and t2, and how these images relate to scene 20, which includes three regions. This example is similar to... Figures 5A to 5C The example explained. However, Figure 5 also includes a third region 50, which has the same brightness level as the second region 40, i.e., has the second brightness level. Furthermore, in this example, the first brightness level is higher than the second brightness level (therefore, with). Figure 5C(Conversely). Because the local brightness levels in the second region 40 and the third region 50 are lower than in the first region 30, the pupil size will increase when the user 230 looks at the second region 40 or the third region 50 after viewing the first region 30. Since scene 20 includes several regions within the second brightness level, in addition to determining the increase in pupil size, gaze data obtained from the second eye image 45 is also needed to determine whether the user 230 is looking at the second region 40 or the third region 50. That is, based on the first eye image 35, a first gaze point 212t1 or a first gaze line passing through the first gaze point 212t1 can be determined. Furthermore, based on the second eye image 45, a second gaze point 212t2 or a second gaze line passing through the second gaze point 212t2 can be determined. In this case, based on the second gaze point 212t2 or the second gaze line passing through the second gaze point 212t2 and the increase in pupil size, the processing circuit is configured to determine that the user is looking at the second region 40.

[0098] In Figure 4A-6 In the relevant example above, scene 20 is constant during the first and second time periods t1 and t2, and therefore the total illuminance is the same during both periods. Consequently, the pupil size is unaffected by anything other than the local / perceived luminance level. This is common in virtual reality (VR) applications using head-mounted devices, where no external light enters the user's field of view or scene 20. Therefore, the total luminance level can be controlled. In particular, it is easy to maintain a constant total luminance level during the calibration process.

[0099] However, in applications where the total light level of the user's pupil changes over time, it may be advantageous to introduce one or more checks to ensure that the pupil 102 changes size due to changes in local brightness levels rather than due to changes in global brightness levels (i.e., the total brightness level of the exposed pupil 102). This may be important in applications where ambient light from the real world enters the user's field of view or scene 20, or in applications where the total brightness level of the display changes over time.

[0100] In one example, the processing circuit 11 is further configured to: obtain a first total light level of the pupil 102 of the user being exposed during a first time period t1; obtain a second total light level of the pupil 102 of the user 230 being exposed during a second time period t2; determine the total light level change between the first and second total light levels; and determine reference fixation data only if the total light level change is less than a total light level change threshold. Therefore, a check is introduced to verify that the pupil size does not change due to variations in the total light level of the exposed pupil.

[0101] In another example, the processing circuit 11 is further configured to: obtain a first scene image of the user's pupil exposed during a first time period t1; obtain a second scene image of the user's pupil exposed during a second time period t2; determine a scene change between the first scene image and the second scene image; and determine the user's reference gaze data only if the scene change is less than a scene change threshold. Therefore, a check is introduced to verify that the pupil size change is not caused by a change in the scene image exposing the pupil.

[0102] In another example, the processing circuit 11 is further configured to: determine a first pupil position based on a first eye image 35; determine a second pupil position based on a second eye image 45; determine a pupil position change between the first and second pupil positions; and determine the user's reference gaze data only if the pupil position change is greater than a pupil position change threshold. Therefore, a check is introduced to verify that the pupil size change is caused by a pupil position change between a first time period t1 and a second time period t2. If the scene 20 exposing the user's pupils remains the same or at least similar between the first and second time periods, a pupil position change is required to cause a pupil size change due to local brightness levels (i.e., where the user is looking) or perceived brightness levels, rather than due to global brightness levels.

[0103] Figure 7Scene 20 is shown, comprising multiple objects 1011, 1021, and 1031 presented by a 3D display 1000, wherein objects 1011, 1021, and 1031 are located at different depths of field. The display of objects by the 3D display 1000 depicts a scene 20 in which a bottle is placed on various other objects or items. In one example, the 3D position of the object is converted to its position in the display for the left eye using a first projection, and the 3D position of the object is converted to its position in the second display for the left eye using a second projection, thereby presenting the object to the user 230 in 3D. Based on the above method and the eye-tracking system 10, object 1011 among the displayed objects 1011, 1021, and 1031 that the user is viewing is determined. In this example, it can be determined that after a change in pupil size, the user 230 is viewing bottle 1011, which in this example corresponds to the second region 40. Therefore, gaze-to-object mapping is performed. The gaze-converging depth of field representing the depth of field that the user is viewing is then obtained. The gaze convergence depth of field can be determined by comparing the pupil positions of the left and right eyes. For example, the gaze convergence depth of field can be obtained by comparing the current interpupillary distance (IPD) between the left and right eyes with the interpupillary distance (IOD) defined as when the user is looking at infinity. The object depth of field is then obtained based on the determined 3D position of the object 1011. The updated gaze convergence depth of field is then calculated using the obtained gaze convergence depth of field and the object depth of field, i.e., by updating the obtained gaze convergence depth of field with the object depth of field or a value between the obtained gaze convergence depth of field and the object depth of field. Therefore, in this case, the reference gaze data includes the gaze convergence depth of field, and the spatial information of the second region 40 includes depth information.

[0104] Furthermore, updating the user 230's gaze calibration settings also includes updating the user's gaze convergence depth of field based on depth information. Therefore, the reference gaze data can be set to be consistent with the gaze convergence depth of field between the depth of field of the second region 40 and the user's gaze convergence depth of field. Alternatively, the gaze convergence depth of field can be set to be consistent with the depth of field of the second region 40. In other words, when the gaze convergence depth of field is obtained, the user's gaze calibration settings are updated based on the user's awareness that they are looking at the second region 40.

[0105] Figure 8 A method performed by an eye-tracking system 10 is shown for determining reference gaze data of a user 230 in a scene 20 exposing the pupil 102 of a user. Scene 20 includes at least a first region 30 having a first brightness level and a second region 40 having a second brightness level that differs from the first brightness level by a brightness level threshold. The method includes the following steps:

[0106] S1 obtains a first eye image 35 including the pupil 102 of user 230, which is captured during a first time period t1;

[0107] The size of the first pupil in S2 is determined based on the first eye image 35;

[0108] S3 obtains a second eye image 45 including the pupil 102 of user 230, which is captured during the second time period t2;

[0109] The size of the second pupil in S4 is determined based on the second eye image 45;

[0110] Obtain scene information of the scene in which S5 exposes the pupil 102 of user 230, the scene information including at least a first brightness level, a second brightness level, and spatial information of the second region 40 during the second time period t2;

[0111] Determine the pupil size change between the first pupil size and the second pupil size in S6, which indicates that the user 230 is looking at the second region 40;

[0112] If the pupil size change is greater than the pupil size change threshold, then the reference fixation data for user 230 during the second time period t2 is determined.

[0113] In addition, the method may include any additional steps that the eye-tracking system 10 is capable of performing, as disclosed in this specification.

[0114] Figure 9A A head-mounted device 210 according to one or more embodiments is shown. The head-mounted device 210 is a device optionally adapted to be mounted (or arranged) on the head of a user 230, such as... Figure 9A As shown. The head-mounted device 210 may include, for example, a head-mounted display (HMD) and / or be included in a head-mounted display (HMD), such as virtual reality, VR, headphones, augmented reality, AR, headphones or mixed reality, MR, headphones. The head-mounted device 210 may include information about Figure 1 The described eye-tracking system 10 includes all or some of its features. The head-mounted device 210 or head-mounted display (HMD) includes a 3D display 311 that can display multiple objects 1011, 1021, 1031 in response to control signals received from a computer. The head-mounted device 210 is typically further configured to use one or more eye image sensors 13 to provide gaze data, such as an indication of the gaze point and / or convergence distance. In other words, the head-mounted device 210 is configured to provide an indication of the object or area that the user 230 is looking at and / or the depth of field that the user 230 is looking at / viewing.

[0115] The 3D display 311 can be, for example, a stereoscopic display. The 3D display 311 can, for example, include glasses equipped with AR functionality. Furthermore, the 3D display 311 can be a volumetric 3D display, which can be free-standing stereoscopic or automated multi-view, meaning it produces 3D images visible to the naked eye without requiring stereoscopic goggles or a stereoscopic head-mounted display. Therefore, as regarding... Figure 9A As described, the 3D display 311 can be part of the head-mounted device 210. However, the 3D display 311 can also be a remote display that does not require stereoscopic goggles or a stereoscopic head-mounted display. In the third example, the 3D display 311 is a remote display in which stereoscopic glasses are required to display the 3D effect to the user.

[0116] Figure 9B A remote display system 220 according to one or more embodiments is illustrated. The remote display system 220 may include information about... Figure 1 The eye-tracking system 10 is described, including all or some of its features. The remote display system 220 typically includes a remote 3D display 311, as described above. Figure 9A The 3D display 311 is remote because it is not located near the user 230. The remote display system 220 is also typically configured to provide gaze tracking signals using one or more eye image sensors 312, 313, for example, indicating the gaze point and / or convergence distance. In other words, the remote display system 220 is configured to provide an indication of the object the user 230 is looking at and / or the depth of field the user is looking at / viewing. Figure 9B As shown, the remote 3D display 311 does not require stereoscopic / stereoscopic goggles or a stereoscopic / stereoscopic head-mounted display. In another example, the 3D display is a remote display, where stereoscopic glasses are required to present the 3D effect to the user.

Claims

1. An eye tracking system (10) for determining reference gaze data of a user (230) in a scene (20) exposing a pupil (102) of the user, the scene comprising at least a first region (30) having a first luminance level and a second region (40) having a second luminance level, the second luminance level differing from the first luminance level by more than a luminance level threshold, the eye tracking system comprising processing circuitry (11), the processing circuitry being configured to: obtain a first eye image (35) comprising the pupil of the user, the first eye image being captured during a first time period (ti); determine a first pupil size based on the first eye image; obtain a second eye image (45) comprising the pupil of the user, the second eye image being captured during a second time period (t2); determine a second pupil size based on the second eye image; obtain scene information of the scene exposing the pupil of the user, the scene information comprising at least the first luminance level, the second luminance level, and spatial information of the second region during the second time period; determine a pupil size change between the first pupil size and the second pupil size, the pupil size change being indicative of the user looking at the second region; determine reference gaze data of the user during the second time period if the pupil size change is greater than a pupil size change threshold; and further update gaze calibration settings of the user based on the reference gaze data and the scene information, wherein the reference gaze data comprises a gaze convergence depth of field, and the spatial information of the second region comprises depth of field information, and wherein the update of the gaze calibration settings of the user is further configured to update the gaze convergence depth of field of the user based on the depth of field information.

2. The eye tracking system of claim 1, the processing circuitry being further configured to: obtain a position of the pupil during the second time period; and wherein the determination of reference gaze data of the user is based at least in part on the obtained position of the pupil.

3. The eye tracking system of claim 2, wherein the obtaining of the position of the pupil is based at least in part on a position of the pupil in the second eye image.

4. The eye tracking system of claim 1, wherein the determination of reference gaze data of the user is based at least in part on the obtained scene information.

5. The eye tracking system of claim 1, the processing circuitry being further configured for: determining a distance difference between the reference gaze data and the spatial information of the second region; and sending an error signal to the user if the distance difference is above an error threshold.

6. The eye tracking system of claim 1, the processing circuitry being further configured for: obtaining a first total light level exposing the pupil of the user during the first time period; obtaining a second total light level exposing the pupil of the user during the second time period; determining a total light level change between the first total light level and the second total light level; and determining reference gaze data only if the total light level change is less than a total light level change threshold.

7. The eye tracking system of claim 1, the processing circuitry being further configured to: obtain a first scene image exposing the pupil of the user during the first time period; obtain a second scene image exposing the pupil of the user during the second time period; determine a scene change between the first scene image and the second scene image; and determine reference gaze data of the user only if the scene change is less than a scene change threshold.

8. The eye tracking system of claim 1, the processing circuitry being further configured to: determine a first pupil position based on the first eye image; determine a second pupil position based on the second eye image; determine a pupil position change between the first pupil position and the second pupil position; and determine reference gaze data of the user only if the pupil position change is greater than a pupil position change threshold.

9. The eye tracking system of claim 1, wherein the reference gaze data comprises at least one of a gaze point and a gaze line.

10. The eye tracking system of claim 1, further comprising a display (12; 111; 311; 1000) arranged to display at least a portion of the scene exposing the pupil of the user.

11. The eye tracking system of claim 1, further comprising an outward-facing scene sensor (16) arranged to obtain at least a portion of the scene exposing the pupil of the user.

12. The eye tracking system of claim 1, further comprising an eye image sensor (13; 312; 313) arranged to capture the first eye image and the second eye image.

13. The eye tracking system of claim 12, further comprising at least one light emitter (14), wherein each light emitter is arranged to cause a glint in the eye of the user, the glint being detectable in the second eye image, wherein the determination of reference gaze data of the user is based at least in part on a position of the at least one glint relative to a position of the pupil.

14. A head-mounted device (210) comprising an eye tracking system (10) for determining reference gaze data of a user (230) in a scene (20) exposing a pupil (102) of the user, the scene comprising at least a first region (30) having a first luminance level and a second region (40) having a second luminance level, the second luminance level differing from the first luminance level by more than a luminance level threshold, the eye tracking system comprising processing circuitry (11) configured to: obtain a first eye image (35) comprising the pupil of the user, the first eye image being captured during a first time period (tl); obtain a second eye image (45) comprising the pupil of the user, the second eye image being captured during a second time period (t2); determining a first pupil size based on the first eye image; obtaining a second eye image (45) comprising the pupil of the user, the second eye image being captured during a second time period (t2); determining a second pupil size based on the second eye image; obtaining scene information of the scene exposing the pupil of the user, the scene information comprising at least the first luminance level, the second luminance level and spatial information of the second region during the second time period; determining a pupil size change between the first pupil size and the second pupil size, the pupil size change being indicative of the user looking at the second region; determining reference gaze data of the user during the second time period if the pupil size change is greater than a pupil size change threshold; and further updating gaze calibration settings of the user based on the reference gaze data and the scene information, wherein the reference gaze data comprises a gaze convergence depth of field and the spatial information of the second region comprises depth of field information, and wherein the updating of the gaze calibration settings of the user is further configured to update the gaze convergence depth of field of the user based on the depth of field information.

15. A method performed by an eye tracking system (10) for determining reference gaze data of a user (230) in a scene (20) exposing a pupil (102) of the user, the scene comprising at least a first region (30) having a first luminance level and a second region (40) having a second luminance level, the second luminance level differing from the first luminance level by more than a luminance level threshold, the method comprising the steps of: obtaining (S1) a first eye image (35) comprising the pupil of the user, the first eye image being captured during a first time period (tl); determining (S2) a first pupil size based on the first eye image; obtaining (S3) a second eye image (45) comprising the pupil of the user, the second eye image being captured during a second time period (t2); determining (S4) a second pupil size based on the second eye image; obtaining (S5) scene information of the scene exposing the pupil of the user, the scene information comprising at least the first luminance level, the second luminance level and spatial information of the second region during the second time period; determining (S6) a pupil size change between the first pupil size and the second pupil size, the pupil size change being indicative of the user looking at the second region; determining reference gaze data of the user during the second time period if the pupil size change is greater than a pupil size change threshold; and further updating gaze calibration settings of the user based on the reference gaze data and the scene information, wherein the reference gaze data comprises a gaze convergence depth of focus, and the spatial information of the second region comprises depth of focus information, and wherein the updating of the gaze calibration settings of the user is further configured to update the gaze convergence depth of focus of the user based on the depth of focus information.

16. A computer-readable storage medium comprising a computer program, the computer program comprising computer readable code means which, when run in an eye tracking system (10), are arranged to determine reference gaze data of a user (230) in a scene (20) exposing a pupil (102) of the user, the scene comprising at least a first region (30) having a first level of luminance and a second region (40) having a second level of luminance, the second level of luminance differing from the first level of luminance by more than a luminance level threshold, wherein the computer readable code means, when run in the system, are arranged to cause the system to perform the following steps: obtaining (SI) a first eye image (35) comprising the pupil of the user, the first eye image being captured during a first time period (ti); determining (S2) a first pupil size based on the first eye image; obtaining (S3) a second eye image (45) comprising the pupil of the user, the second eye image being captured during a second time period (t2); determining (S4) a second pupil size based on the second eye image; acquiring (S5) scene information of the scene exposing the pupil of the user, the scene information comprising at least the first level of luminance, the second level of luminance and spatial information of the second region during the second time period; determining (S6) a pupil size change between the first pupil size and the second pupil size, the pupil size change being indicative of the user looking at the second region; if the pupil size change is greater than a pupil size change threshold, determining reference gaze data of the user during the second time period; and further updating gaze calibration settings of the user based on the reference gaze data and the scene information, wherein the reference gaze data comprises a gaze convergence depth of focus, and the spatial information of the second region comprises depth of focus information, and wherein the updating of the gaze calibration settings of the user is further configured to update the gaze convergence depth of focus of the user based on the depth of focus information.

Images