Methods and systems for flash classification

Abstract

An eye-tracking system includes at least one emitter and at least one image sensor configured to generate an image of a user's eye, the image including light reflected from the user's eye from the emitter. A Fresnel lens is positioned between the image sensor and the user's eye, through which the image sensor observes the eye. A processing circuitry system receives the image from the image sensor, identifies flashes of light in the image, assigns an angular position to each flash based on the angular relationship between each flash and the center of the Fresnel lens, and determines how many flashes have the same angular position. If more than a predetermined number of flashes have the same angular position, the flashes are classified as false flashes.

Description

Technical Field

[0001] This disclosure generally relates to the field of eye tracking. In particular, this disclosure relates to systems and methods for identifying reflections from optical devices in an eye tracking system. Background Technology

[0002] In eye-tracking applications, digital images of the user's eyes are acquired and analyzed to estimate the user's gaze direction. Different methods exist to achieve this estimation. In some methods, ambient light is used when acquiring images of the user's eyes, while in others, an additional light source, often referred to as a "light emitter," is used to illuminate the eyes to acquire the image. Typically, gaze estimation is based on the identification of the user's pupils and the corneal reflection (referred to as a "flash") in the user's eyes.

[0003] One known eye-tracking method involves using infrared light and an image sensor. The infrared light is directed toward the user's pupil, and the reflection of the light is captured by the image sensor. By analyzing the reflection points, the direction of the user's gaze can be calculated. Such a system is described in US 7,572,008 (which is incorporated herein by reference in its entirety).

[0004] Portable or wearable eye-tracking devices have been previously described. One such eye-tracking system is described in US 9,041,787 (which is incorporated herein by reference in its entirety). A wearable eye-tracking device is described that uses a light emitter and an image sensor to determine the direction of gaze.

[0005] In some cases, problems may arise due to reflections caused by the hardware within the eye-tracking device itself. For example, when a lens is used in an eye-tracking device, the lens itself may cause reflections in the image received by the image sensor.

[0006] It is desirable to provide an eye-tracking device that can cope with this situation and can reduce reflections introduced by the device hardware to improve the accuracy and / or precision of eye tracking. Summary of the Invention

[0007] According to a first aspect, an eye-tracking system is provided, comprising: at least one emitter; and at least one image sensor configured to generate an image of a user's eye, the image including light received from the at least one emitter and reflected from the user's eye; the eye-tracking system further comprising: a Fresnel lens positioned between the image sensor and the user's eye, the image sensor observing the eye through the Fresnel lens; and a processing circuitry system configured to: receive the image from the image sensor; identify flashes in the image; assign an angular position to each flash based on an angular relationship between each flash and the center of the Fresnel lens; determine how many flashes have the same angular position; and classify the flashes as false flashes if more than a predetermined number of flashes have the same angular position.

[0008] In this context, a flash is defined as a bright spot in the image caused by light reflected from the emitter. This bright spot can be a direct reflection of the emitter from the user's cornea, which is the desired flash, or it can be the reflection or refraction of light from the emitter within the lens itself, resulting in an unwanted or false flash. It is these false flashes that need to be categorized so that they can be ignored during further processing by the eye-tracking system. Specifically, false flashes can be caused by light from the emitter that is reflected in the glasses worn by the user and then reflected and / or refracted within the lens.

[0009] This provides a system that implements a computationally lightweight approach in which flashes are classified as false flashes through simple processing of the images captured by the image sensor.

[0010] For example, the processing circuitry system can be provided as one or more controllers or processors within a device containing the image sensor and the emitter. Therefore, the processing operations performed by the processing circuitry system can be executed locally on the device. Alternatively, the processing circuitry system can be located remotely from the device containing the image sensor and the emitter. In this case, one or more host devices can be included within the processing circuitry system in which the processing operations are performed. In another example, the processing circuitry system can be separately configured across the device containing the image sensor and the emitter and one or more remote devices, thereby allowing the processing steps to be distributed across the device and the remote devices.

[0011] The angular position can be defined relative to a predetermined baseline. Providing a predetermined baseline allows for easy calculation of the angular position of each flash. The position of this predetermined baseline can be constant in each image captured by the image sensor.

[0012] Each angular position can be provided as a discrete angular range. By making the angular position a discrete angular range, it can be ensured that the flash does not need to have the exact same angular relationship as the lens center to be classified as being at the same angular position. This allows manufacturing tolerances and slight inaccuracies in processing and image capture to be ignored in the calculations.

[0013] Each angle can range from 5 degrees or less, 3 degrees or less, or 1 degree or less. In one embodiment, a full circle of angles is divided into 360 integer angle bins. The integer value of each angle provides simple processing and an angle mapping to an index in a data structure.

[0014] The predetermined number of flashes can be 3. Alternatively, the predetermined number of flashes can be 2, 4, 5, 6, or 7. The predetermined number of flashes can be any integer greater than 1, selected by the system operator. The higher the predetermined number of flashes used, the less likely a genuine flash will be incorrectly classified as a false flash.

[0015] Flashes occurring within a predetermined area around the center of the lens can be excluded from those classified as false flashes. It has been found that flashes within this predetermined area are less likely to be false flashes than flashes outside this area. Therefore, by excluding these flashes from those classified as false flashes, incorrect identification of false flashes can be limited or prevented.

[0016] The predetermined region surrounding the center of the lens can be a circular region with a predetermined or parameterized radius centered on the center of the lens. For example, the predetermined region can be a circle with a radius of 50 pixels.

[0017] Flashes occurring within a predetermined area around the center of the pupil can be excluded from those classified as false flashes. It has been found that flashes within this predetermined area around the pupil center are less likely to be false flashes than flashes outside this predetermined area. Therefore, by excluding these flashes from those classified as false flashes, incorrect identification of false flashes can be limited or prevented.

[0018] The predetermined region surrounding the center of the pupil can be a circular region with a predetermined or parameterized radius centered on the center of the pupil. For example, the predetermined region can be a circle with a radius of 50 pixels.

[0019] Flashes that are excluded from being classified as false flashes (within a predetermined area around the center of the lens or around the center of the pupil) can be excluded from the determination of how many flashes have the same angular position.

[0020] Alternatively, flashes excluded from being classified as false flashes (within a predetermined area around the center of the lens or around the center of the pupil) can be included in the determination of how many flashes have the same angular position.

[0021] This eye-tracking system identifies the presence of a false flash by measuring the differences between flashes considered false. If the flashes are within a certain distance of each other, they can be identified as false. Otherwise, the flash can be left unclassified as false.

[0022] According to a second aspect, a head-mounted device is provided, comprising: a display; and an eye-tracking system including: at least one emitter; and at least one image sensor configured to generate an image of a user's eye, the image including light received from the at least one emitter and reflected from the user's eye; the eye-tracking system further comprising: a Fresnel lens positioned between the image sensor and the user's eye, the image sensor viewing the eye through the Fresnel lens, and the user viewing the display through the Fresnel lens; and a processing circuitry system configured to: receive an image from the image sensor; identify flashes in the image; assign an angular position to each flash based on an angular relationship between each flash and the center of the Fresnel lens; determine how many flashes have the same angular position; and classify flashes as false flashes if more than a predetermined number of flashes have the same angular position.

[0023] The head-mounted device may include any one or more additional features described with respect to the first aspect.

[0024] According to a third aspect, a method for identifying false flashes in an eye-tracking system is provided, the method comprising: capturing an image of a user's eye from an image sensor, wherein the image sensor observes the user's eye through a Fresnel lens, the image including light received from at least one emitter and reflected from the user's eye; identifying flashes in the image; assigning an angular position to each flash based on an angular relationship between each flash and the center of the Fresnel lens; determining how many flashes have the same angular position; and classifying the flashes as false flashes if more than a predetermined number of flashes have the same angular position.

[0025] The position of this angle can be defined relative to a predetermined baseline.

[0026] Each angular position can be provided as a discrete angular range. Each angular range can be 5 degrees or less, or 3 degrees or less, or 1 degree or less.

[0027] The pre-ordered number of flashes can be 3.

[0028] Flashes appearing within a predetermined area around the center of the lens can be excluded from those classified as false flashes.

[0029] Flashes within a predetermined area around the center of the pupil can be excluded from those classified as false flashes.

[0030] Flashes excluded from being classified as false flashes can be excluded from the determination of how many flashes have the same angular position.

[0031] Flashes excluded from being classified as false flashes can be included in the determination of how many flashes have the same angular position.

[0032] According to a fourth aspect, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by an eye-tracking system, cause the eye-tracking system to perform the method of the third aspect. Attached Figure Description

[0033] Specific embodiments will now be described in detail with reference to the accompanying drawings, in which:

[0034] Figure 1 It is an image of the user's eye, characterized by flashes of light created by interaction with a Fresnel lens;

[0035] Figure 2 This is a simplified schematic diagram of the eye-tracking system based on the first aspect;

[0036] Figure 3 This is an example depiction of classifying flashes using an eye-tracking system of the first aspect;

[0037] Figure 4 It is based on the flowchart of the third aspect of the method, and

[0038] Figure 5a and Figure 5b It shows how a predetermined area, including the center of the lens and the area around the pupil, affects the classification of flashes. Detailed Implementation

[0039] The applicant has noted that, when the image sensor observes the eye through a Fresnel lens, the emitter can cause characteristic patterns to appear in the image. These patterns 10 in Figure 1 As can be seen, pattern 10 is the result of multiple flashes formed due to the interaction of reflected light with the ring of Fresnel lens 12. These flashes caused by the interaction with lens 12 are detrimental to eye tracking because they may be misinterpreted as the expected flashes and used to calculate the user's gaze direction and / or gaze origin.

[0040] The characteristic pattern is that false flashes (i.e., flashes that are not direct reflections from the user's eye) tend to appear in straight lines radiating radially from the center of the Fresnel lens. This is because a Fresnel lens consists of multiple concentric rings around its center. Therefore, this invention utilizes this phenomenon to detect and classify false flashes within eye images generated by an eye tracker.

[0041] Figure 2 An embodiment of an eye-tracking system 100, which may also be referred to as a gaze tracking system, is depicted. The eye-tracking system 100 is shown in a simplified form, and additional components may be present in other embodiments. The eye-tracking system 100 includes an image sensor 102, two emitters 104, a Fresnel lens 106 (hereinafter referred to as the "lens"), and a processing circuitry system 108. The lens 106 is positioned between the image sensor 102 and the user's eye 110 such that the image sensor 102 observes the user's eye 110 through the lens 106. The processing circuitry system 108 communicates with the image sensor 102, enabling it to receive images of the user's eye 110 from the image sensor 102. The processing circuitry system 108 also controls the emitters 104, although in other embodiments these emitters may be controlled by a separate processing circuitry system or may not be controlled at all.

[0042] The emitter 104 is typically a light-emitting diode (LED) configured to emit infrared (IR) or near-infrared (NIR) light. These wavelengths of light can be invisible to the user, partially visible, or not distracting to the user. For illustrative purposes, Figure 2 Two light emitters 104 are shown, but in any embodiment of the invention, there may be more or fewer light emitters 104. For example, a single light emitter 104 may be provided, or multiple light emitters 104 may be provided. In common embodiments, eight, ten, or twelve light emitters 104 may be arranged around the periphery of the eye.

[0043] Image sensor 102 can be any type of image sensor, such as a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor. Image sensor 102 may include an integrated circuit containing a pixel sensor array, each pixel containing a photodetector and an active amplifier. The image sensor is capable of converting light into a digital signal. In some examples, the image sensor may be an infrared image sensor, an RGB sensor, an RGBW sensor, or an RGB or RGBW sensor with an IR filter.

[0044] The processing circuitry 108 includes a receiver for receiving images from the image sensor 102. For example, the processing circuitry 108 may be connected to the image sensor 102 and the emitter 104 via a wired and / or wireless connection. Furthermore, the processing circuitry 108 may be located in the same location as the image sensor 102 and the emitter 104 or it may be located remotely, such as in different devices. In one embodiment, the processing circuitry 108 may be provided together with the image sensor 102.

[0045] It should be noted that Figure 2 The location of image sensor 102 is for illustrative purposes only. In virtual reality (VR) devices (such as VR headsets), the image sensor for the eye is typically positioned away from the user's line of sight so as not to obstruct the VR display positioned within the VR device for that eye. This is achieved, for example, by means of a so-called "thermal mirror," which reflects a portion of the light while allowing the rest to pass through. In some VR devices, the thermal mirror is used to reflect infrared light to the image sensor while allowing visible light to pass through, thus allowing the user to see the display.

[0046] Figure 3 Describing, for example Figure 2 The illustrated eye-tracking system 100 classifies flashes of light. Any eye-tracking system 100 can be used to generate flashes, but for reference, the example shown is based on a system similar to [the one described above]. Figure 2 The eye-tracking system 100 generates this. The only difference is that it uses a total of six emitters 204, instead of... Figure 2 The two shown. Figure 3 It can be considered to show something similar to Figure 2 Image 212 generated by the image sensor in the eye-tracking system. However, Figure 3 The light emitter 204 shown is for illustrative purposes only and is not visible in the actual image generated by the image sensor. Figure 3 The depiction of the light emitter 204 in the image attempts to illustrate the arrangement of the light emitter around the lens, thus explaining the location of the characteristic refractive pattern present in image 212.

[0047] As from Figure 3 As can be seen, lens 206 is shown together with the user's eye 210 (including pupil 214). The center of the lens, or lens center 216, is shown as "+". Due to the fixed relationship between the image sensor and lens 206, the lens center 216 will become known during the hardware calibration process, and therefore will be in a fixed position for the image captured by the image sensor. Numerous flashes 218 are shown. Although a large number of flashes are shown, only some of them are labeled to avoid cluttering the diagram. For ease of depiction, Figure 3Only false flashes are shown in the image, but it is clear that in real-world examples, false flashes will be accompanied by “real” flashes, which are the desired flashes required for eye-tracking purposes. However, the detection and classification of false flashes occur in the same way, regardless of whether real flashes also appear in the image.

[0048] Due to the circular shape of the ring in lens 206, false flashes generated by reflection or refraction within lens 206 (rather than reflection leaving the cornea of ​​eye 210) form clusters of flashes extending radially from the lens center 216. Therefore, in order to classify the flashes 218, this embodiment must first place the flashes 218 into multiple groups or clusters. Groups or clusters of false flashes will extend radially outward from the lens center 216. Typically, false flashes occur when some light generated by the emitter is reflected into the Fresnel lens from another structure (which may or may not be part of the eye-tracking system itself). For example, if the user wears glasses, light from the emitter can be reflected into or onto the lens within the glasses.

[0049] To aid in grouping the flashes 218, the depicted embodiment utilizes a reference position or reference line 220. The reference line 220 can be arbitrarily set at any radius of the lens 206, but must extend radially from the lens center 216. Figure 3 In the image 212, reference line 220 extends horizontally to the right from the lens center 216, as shown. The radial position of reference line 220 is not important, as it is only used as a reference for future grouping of flashes 218 in the image 212. Although the position of reference line 220 will affect the relative position of flashes 218, the relative position is arbitrary.

[0050] The processing circuitry acquires the two-dimensional position of each flash 218 within the image, and then assigns an angular position relative to the baseline 220 to each flash 218 based on its position. For example, a flash 218a can be provided with an angular position of 45° (or...) The position of the flash (radius), while another flash 218b can be provided with 135° (or The position (in radians). Therefore, each flash 218 will be characterized as having an angular position somewhere between 0° and 360° (or between 0 and 2π radians) relative to the reference line 220. It is obvious that the distance from the lens center 216 to each flash 218 is irrelevant to this process. Once the angular position of each flash 218 has been determined, the flashes 218 can be grouped. Any measurement of the angular position can be used. In this embodiment, the measurement is in degrees or radians, but any other metric can be used alternatively. For example, it is preferable to divide the circle into 256 parts so that the angular position can be provided as an eight-bit value. Other options will be apparent to those skilled in the art and are not intended to be limited to the examples described herein.

[0051] Grouping the flashes 218 can be achieved using their discrete angular positions. For example, all flashes 218 calculated to have an angular position of 45° can be grouped together. However, in practice, it may be advantageous to allow grouping flashes 218 that differ from each other by small angular amounts. For example, flashes 218 with angular positions of 45° ± 2.5° (or 42.5° to 47.5°) can be grouped together, and separate groups can be provided for flashes 218 with angular positions of 40° ± 2.5° (or 37.5° to 42.5°) and 50° ± 2.5° (or 47.5° to 52.5°). It may be desirable for each flash 218 to be placed in only a single group, in which case the angular ranges will not overlap. However, in other embodiments, overlapping angular ranges can be allowed, and the algorithm used to classify the flashes can be sophisticated enough to handle this situation.

[0052] In some cases, it is not necessary to provide an angular range for the process of grouping the flashes 218. This can depend on the resolution used to determine the angular position of each flash 218. For example, if the resolution of the angular position is only within 5°, then all flashes 218 at similar angular positions will be grouped together regardless. However, if the resolution is 0.1°, to take an extreme example, it might be desirable to group the flashes 218 to allow for slight differences in their angular positions to be accounted for. The grouping of the flashes 218 should be small enough to ensure that a flash caused by one emitter is not grouped with a flash 218 caused by an adjacent emitter. Due to tolerances in the components and inaccuracies in the calculation and processing steps, the flashes may not be perfectly aligned on a radial line from the center of the lens. This can also be affected by the specific design of the lens used.

[0053] Once grouped, the total number of flashes 218 within each group can be counted, or, in the absence of grouping, the number of flashes 218 at each angular position can be counted. As is known to those skilled in the art, each emitter will produce a real or desired flash, but a single desired flash will only be produced by a given emitter and therefore at a given angle. Therefore, in the case of multiple flashes 218 at any given angular position, these flashes must necessarily be false flashes produced by reflection or refraction through a lens. Thus, the processing circuitry is configured to count the number of flashes 218 within each group, determine if this number is greater than one, and if so, classify the flashes 218 in this group as false flashes. Flashes 218 classified as false flashes will then be excluded from further processing so that they do not affect the determination of the user's gaze position or direction. Although the presence of false flashes can be determined by the presence of two flashes at an angular position, by preventing flashes from being classified as false flashes until a large number of flashes (e.g., three) occur, the system is less likely to mistakenly classify real flashes as false flashes.

[0054] In some cases, the system may require further steps before classifying a flash as a false flash. In such cases, the distance between flashes detected at a certain angle can be measured. If this distance is less than a predetermined value (i.e., the flashes are very close together), the flash can be confirmed as false. Alternatively, if the distance between the flashes is greater than the predetermined value, classifying the flash as false can be prevented.

[0055] As from Figure 3 As can be seen in the depicted example, due to the fact that multiple flashes 218 are found at specific or very similar angular positions, three sets or groups of flashes 222a, 222b, 222c can be classified as false flashes: the first group of flashes 222a is found at an angular position of 0° with respect to the baseline 220; the second group of flashes 222b is found at an angular position of 45° with respect to the baseline 220; and the third group of flashes 222c is found at an angular position of 135° with respect to the baseline 220. Each flash 218 within a group of flashes 222a, 222b, 222c (since there are more than three flashes 218 in each group) is classified as a false flash by the processing circuitry system. The choice of the number three as the value for determining a flash as false is a value that can be chosen according to the user's or programmer's preference. For example, the value used to determine a flash can be one (i.e., if more than one flash occurs at a given angular position, then the flash is false), or it can be two, three, or any other integer value.

[0056] It should be noted that the angular position measurement in this example is performed clockwise from baseline 220. This measurement is arbitrary, and the angular position can also be measured counterclockwise.

[0057] Figure 3 Each group of flashes 222a, 222b, and 222c shown consists of a different number of false flashes. The first group 222a consists of five flashes 218, the second group 222b consists of four flashes 218, and the third group 222c consists of six flashes 218. Any number of flashes can be found within any group, and any group of flashes consisting of more than three flashes will result in the flash being classified as a false flash. It should be noted that there may be some exceptions to this rule, where additional criteria are considered before classifying a flash as a false flash, and these will be discussed in more detail below.

[0058] In some cases, false flashes may not be clearly defined and are not separate from each other, and therefore each individual flash can effectively merge with adjacent flashes, thus forming a single luminous block on the image. An example of this can be found in... Figure 1 As can be seen in the images, it is clear that many of the false flashes present in each group do not have easily distinguishable boundaries.

[0059] In this scenario, the processing circuitry can determine the number of flashes in a group by using different metrics. For example, the system can determine the size of the flash, such as by using intensity blocks and dividing it by an amount indicating the flash. This amount can be a predetermined quantity or can be determined by analyzing other flashes in the image. Alternatively, the system can determine the number of flashes by measuring their shape; for example, if the aspect ratio of the image (i.e., its length relative to its width) is higher than a certain value, the flashes on the image can be identified as multiple flashes. For example, this specific value could be 2, 3, or higher. Alternatively, when large bright areas are detected, these may not be classified as flashes at all. The algorithm used by the system could simply reject large bright areas as flashes from the outset.

[0060] In the context of this disclosure, other methods for determining the existence of multiple flashes are known to those skilled in the art.

[0061] In some embodiments, it may be desirable to provide additional steps to increase system stability. These steps are not essential for the system to function, but can help the system detect false flashes without incorrectly classifying genuine flashes as false flashes.

[0062] If a flash occurs within a predetermined area near the lens center, it may be desirable to reject the classification of the flash as false. The applicant has noted that when a flash is detected near the lens center, it is likely a genuine, desirable flash rather than a false flash. Therefore, while other flashes may occur at the same angular location as this flash and can be automatically classified as false flashes, flashes within the predetermined area near the lens center are unlikely to be false flashes and should not be classified as such. Therefore, by providing a predetermined area around the lens center (where no flash is identified as false), incorrect classification can be limited or avoided. In some embodiments, for example, this predetermined area around the lens center may be a circle with a radius of 50 pixels.

[0063] Even when a predetermined area is defined around the center of the lens, flashes within that predetermined area can still be included in the determination of whether more than a predetermined number of flashes exist at any angular location. Conversely, flashes within that predetermined area can be excluded from the determination of whether more than a predetermined number of flashes exist at any angular location.

[0064] Additionally or alternatively, if the flash is within a predetermined area around the pupil of the user's eye, it may be desirable to reject the classification of the flash as false. The applicant has noted that when a flash is found near the pupil, the flash is likely a genuine, desirable flash rather than a false flash. Therefore, while other flashes may exist at the same angular location as such flashes and can be automatically classified as false flashes, flashes within the predetermined area around the pupil are unlikely to be false flashes and should not be classified in this way. Therefore, by providing a predetermined area around the pupil (where no flash is identified as a false flash), incorrect classification can be limited or avoided. In some embodiments, for example, the predetermined area around the pupil of this user's eye may be a circle with a radius of 50 pixels.

[0065] Even when a predetermined area around the pupil is defined, flashes within that area can still be included in the determination of whether more than a predetermined number of flashes exist at any angular location. Conversely, flashes within that predetermined area can be excluded from the determination of whether more than a predetermined number of flashes exist at any angular location.

[0066] Multiple embodiments may include predetermined regions around both the lens center and the pupil, or may include only one of these predetermined regions, or may not include any predetermined regions.

[0067] The main advantage of this disclosure is that the computational cost of determining false flashes by counting and recording their locations is lower than that of other known methods for classifying flashes. Therefore, this invention can be implemented to reduce processor workload or provide other comparable benefits to the implementation of eye-tracking systems.

[0068] Now we will describe the process in conjunction with the previously mentioned methods and steps. Figure 4 .

[0069] In the first step, the image sensor captures an image of the user's eye in step S102. As can be clearly seen from the previous description, the image sensor observes the eye through a lens. Furthermore, the image sensor receives reflected light incident on the eye by one or more light emitters located around the lens. Therefore, the image includes the eye and the reflected light from the light emitters reflected within the eye, as well as the light from the light emitters reflected or refracted by the lens.

[0070] Flashes in the S104 image can then be identified. In the illustrated embodiment, each processing step (e.g., flash identification) is performed by a processing circuitry system. Flashes in the image may be caused by direct reflection from the emitter onto the cornea of ​​the eye, or by reflection or refraction through a lens.

[0071] Each flash is assigned an angular position in S106 based on its angular relationship with the center of the lens. In the earlier described embodiment, this angular relationship was determined with reference to an arbitrarily defined reference line. Other references for measuring angular positions may be known to those skilled in the art.

[0072] In one embodiment, the angular position of each flash can be discretely set within a certain angular range S108 to simplify subsequent processing steps. For example, the angular position can be discretely set within an angular range of 5 degrees or 1 degree, so that similar angular positions are grouped together.

[0073] Flashes detected by the image sensor and identified near the center of the lens or the center of the pupil can be omitted from further classification. Essentially, due to their location, these flashes are considered real, and therefore it may be desirable to enforce measures to prevent them from being designated as false flashes. This can be performed as part of step S108 or in a separate step. Regarding... Figure 5a and Figure 5b This step is described in more detail.

[0074] It can now be determined how many flashes of S110 have the same angular position. Figure 3 In the example, five flashes have an angular position of 0°, four flashes have an angular position of 45°, and six flashes have an angular position of 135°.

[0075] Finally, if more than a predetermined number of flashes have the same angular position (i.e., if more than a predetermined number of flashes are in the same group), then flashes can be classified as false in S112. See again. Figure 3If the predetermined quantity is three, then each of the three groups has more than three flashes, and therefore all the flashes shown in the image are classified as false flashes.

[0076] Figure 5a and Figure 5b Predetermined regions 324 and 424, located at the center of the lens and around the pupil, are depicted respectively. These images illustrate how the positions of the predetermined regions 324 and 424 affect whether flashes are classified as false flashes. Figure 5a and Figure 5b Each figure in the diagram illustrates a depiction of what the image sensor will see, including Fresnel lenses 306 and 406, eyes 310 and 410 (including pupils 314 and 414), and numerous flashes 318 and 418. Predetermined regions 324 and 424 are each shown as dashed lines. Baselines 320 and 420 are also overlaid on the images to aid in the description.

[0077] In both figures, three sets of flashes 318 and 418 are shown. A set of four flashes 322a and 422a are shown at an angle of approximately 45° to the baseline, a set of three flashes 322b and 422b are shown at an angle of approximately 135° to the baseline, and a single flash 322c and 422c are shown at an angle of approximately 225° to the baseline. Since flashes 318 and 418 are located in the same position in each image, the difference in classification depends only on the position of predetermined regions 324 and 424. Each system is configured to classify a flash as false when three or more flashes 318 and 418 are present at a specific angle.

[0078] exist Figure 5a In this configuration, a predetermined area 324 is located around the lens center 316. Four flashes 318 appear at a 45° angle to the reference line 320. However, the flash closest to the lens center among these flashes 318 is within the predetermined area 324 surrounding the lens center 316. Therefore, this flash 318 is prevented from being classified as false. Since there are still three flashes 318 at the same location that could be classified as false, the system classifies these flashes 318 as false. In other embodiments, the system could still count a fourth flash 318 (to prevent it from being classified as false) out of the total number of flashes 318 at a given location, thus counting four flashes 318. This would not affect the present situation.

[0079] The three flashes 318 at an angle of 135° to the baseline 320 are all outside the predetermined area 324 around the lens center 316. Since there are three flashes 318 at the same angle, these flashes 318 will be classified as false.

[0080] The single flash 318 at 225° to the reference line 320 is within a predetermined area 324 around the lens center 316, and is also the only flash 318 at that given location. Therefore, this flash 318 will not be classified as false.

[0081] exist Figure 5b In this configuration, the predetermined area 424 is located around the pupil 414. Four flashes 418 appear at a 45° angle to the baseline 420, and all of these flashes are outside the predetermined area 424 surrounding the pupil 414. Therefore, all of these flashes 418 will be classified as false.

[0082] The three flashes 418 at a 135° angle to the baseline 420 are all outside the predetermined area 424 around the pupil 414. Since there are three flashes 418 at the same angle, these flashes 418 will be classified as false.

[0083] A single flash 418 at a 225° angle to the baseline 420 is within a predetermined area 424 around the pupil 414 and is also the only flash 418 at that given location. Therefore, this flash 418 will not be classified as false.

[0084] In both examples, the predetermined regions 324 and 424 have been described separately. However, in some embodiments, it may be preferable to provide two predetermined regions 324 and 424 with respect to a single system, thus having two predetermined regions 324 and 424 in which flashes are prevented from being classified as false. Given the above explanation, the operation of this embodiment is clear.

Claims

1. An eye tracking system, comprising: at least one light emitter; and at least one image sensor configured to produce an image of a user's eye, the image comprising light received from the at least one light emitter reflected from the user's eye; the eye tracking system further comprising: a Fresnel lens positioned between the image sensor and the user's eye, the image sensor viewing the eye through the Fresnel lens; and processing circuitry configured to: receive an image from the image sensor; identify glints in the image; assign an angular position to each glint based on an angular relationship between each glint and a center of the Fresnel lens; determine how many glints have the same angular position; classify a glint as a false glint if more than a predetermined number of glints have the same angular position; and exclude the false glint from further processing by the eye tracking system.

2. The eye tracking system of claim 1, wherein, the angular position is defined relative to a predetermined reference line.

3. The eye tracking system of claim 1 or claim 2, wherein, each angular position is provided as a discrete angular range.

4. The eye tracking system of claim 3, wherein, each angular range is 5 degrees or less, or 3 degrees or less, or 1 degree or less.

5. The eye tracking system of claim 1, wherein, the predetermined number of glints is three.

6. The eye tracking system of claim 1, wherein, glints occurring within a predetermined region around the center of the Fresnel lens are excluded from being classified as false glints.

7. The eye tracking system of claim 1, wherein, glints occurring within a predetermined region around the center of the pupil are excluded from being classified as false glints.

8. The eye tracking system of claim 6 or claim 7, wherein, glints excluded from being classified as false glints are excluded from the determination of how many glints have the same angular position.

9. The eye tracking system of claim 6 or claim 7, wherein, glints excluded from being classified as false glints are included in the determination of how many glints have the same angular position.

10. A head-mounted device, comprising: a display; and an eye tracking system, comprising: at least one light emitter; and at least one image sensor configured to produce an image of a user's eye, the image comprising light received from the at least one light emitter reflected from the user's eye; the eye tracking system further comprising: a Fresnel lens positioned between the image sensor and the user's eye, the image sensor viewing the eye through the Fresnel lens, and the user viewing the display through the Fresnel lens; and processing circuitry configured to: receive an image from the image sensor; identify glints in the image; assign an angular position to each glint based on an angular relationship between each glint and a center of the Fresnel lens; determine how many glints have the same angular position; classify a glint as a false glint if more than a predetermined number of glints have the same angular position; and exclude the false glint from further processing by the eye tracking system.

11. A method of identifying false glints in an eye tracking system, the method comprising: capturing an image of a user's eye from an image sensor that views the user's eye through a Fresnel lens, the image including glints reflected from the user's eye from light received from at least one light emitter; identifying glints in the image; assigning an angular position to each glint based on an angular relationship between each glint and a center of the Fresnel lens; determining how many glints have the same angular position; if more than a predetermined number of glints have the same angular position, classifying the glints as spurious glints; and excluding the spurious glints from further processing by the eye tracking system.

12. The method of claim 11, wherein, the angular positions are defined relative to a predetermined reference line.

13. The method of claim 11 or claim 12, wherein, each angular position is provided as a discrete angular range.

14. The method of claim 13, wherein, each angular range is 5 degrees or less, or 3 degrees or less, or 1 degree or less.

15. The method of claim 11, wherein, the predetermined number of glints is three.

16. The method of claim 11, wherein, glints occurring within a predetermined region around the center of the Fresnel lens are excluded from being classified as spurious glints.

17. The method of claim 11, wherein, glints occurring within a predetermined region around the center of the pupil are excluded from being classified as spurious glints.

18. The method of claim 16, wherein, glints excluded from being classified as spurious glints are excluded from the determination of how many glints have the same angular position.

19. The method of claim 16, wherein, glints excluded from being classified as spurious glints are included in the determination of how many glints have the same angular position.

20. A computer-readable storage medium storing computer-executable instructions that, when executed by an eye tracking system, cause the eye tracking system to perform the method of any one of claims 11 to 19.

Images