Classifying Flashes Using an Eye Tracking System

Abstract

A method, system, and non-transitory computer-readable storage medium for classifying flashes using an eye-tracking system of a head-mounted device are provided by the following steps: obtaining a current image of a first eye (any position of a flash present in the image t+1 ); obtaining a data set that indicates the respective positions of at least one flash detected in one or more previously captured images of the first eye; and for each flash present in the current image (image t+1 ), determining whether the position of the flash corresponds to the position of a flash in the data set; and if the positions correspond, classifying the flash as a static flash. Static flashes can be excluded from further processing in the eye-tracking system. If the eyes of the user of the head-mounted device have moved, the embodiment may further include updating the data set based on the current image (image t+1 ).

Description

[0001] This application is a divisional application of the invention patent application with the application date of December 20, 2019, the application number of 201911325884.7, and the invention name of "Classifying Flashes Using an Eye Tracking System". Technical Field

[0002] The present disclosure generally relates to classifying flashes among flashes identified by eye tracking in a head-mounted device. The present disclosure also relates to corresponding methods, systems, and storage media. Background Art

[0003] Many existing eye tracking and gaze tracking solutions use flash detection as part of the tracking. For example, the main components of a corneal reflection eye or gaze tracking system can include a video camera sensitive to near-infrared (NIR) light, an NIR illuminator (usually a light-emitting diode) typically mounted to emit light along the optical axis of the camera, and a processing circuit for analyzing the images captured by the camera. NIR light is typically used because users cannot see this light and are thus not distracted by it. A corneal reflection eye or gaze tracking system typically projects light from the illuminator towards the user's eyes and monitors the angular difference between the pupil position and the beam reflection. The light reflected from the eye includes "flashes", which are very small and very bright virtual images of the illuminator reflected from the cornea (usually the front surface of the corneal bulge of the eye). After an image of the illuminated eye has been captured, the processing circuit is typically configured to perform image processing (such as including intensity threshold determination and edge detection) to identify flashes and pupils from the images captured by the camera. Using the information from the image processing, eye tracking and / or gaze tracking can then be performed.

[0004] If the flashes are not accurately detected, this will introduce errors in the eye tracking and / or gaze tracking results. A common problem in flash detection is that when the user of an eye tracking or gaze tracking system is wearing glasses, the glasses provide a reflective surface from which light from the illuminator is reflected towards the camera. As a result, additional flashes are introduced into the captured images, and these additional flashes are not caused by reflections from the cornea of the user's eyes. Flashes caused by the reflection of light from glasses or any other reflective surface present between the camera and the user's eyes may hereinafter be referred to as "false flashes". Flashes caused by reflections from the cornea of the user's eyes may hereinafter be referred to as "true flashes".

[0005] False flashes can cause serious eye tracking or gaze tracking problems, thus affecting the accuracy and precision of the eye tracking or gaze tracking system.

[0006] Solutions are needed for one or more of the identified problems. Summary of the Invention

[0007] There are provided a method, a system, and a computer-readable storage medium having the features defined in the independent claims, for solving or at least ameliorating one or more of the discovered problems. Preferred embodiments are defined in the dependent claims.

[0008] Specifically, the embodiments presented herein are aimed at achieving an optimized visual experience for the user and at achieving the best possible prerequisites for the performance of eye tracking or gaze tracking. The term "eye tracking" as used herein can be understood to include: tracking or observing the real parts of the eyes in the real world, in a 3D model of the eyes, or in a 2D image depicting the eyes; or determining the object that the eyes are tracking or gazing at. Determining the object that the eyes are tracking or gazing at can also be referred to as gaze tracking. If only this term is used, the term "eye tracking" can thus be understood to include gaze tracking.

[0009] These objectives are achieved by providing a solution for classifying flashes, specifically static flashes (i.e., flashes that do not move from one captured image frame to the next even when the eyes move) presented in the user's eye images. As described herein, a static flash is a special type of false flash, i.e., a flash that is not caused by the reflection of the user's eye's cornea or any other part of the eye. In other words, all static flashes are false flashes, but not all false flashes are static flashes.

[0010] Use any type of imaging sensor, imaging device, camera, etc. included in an eye tracking or gaze tracking system to capture an image of the user's eyes. Hereinafter, the term "camera" will be used, but it should be understood that, in the context of the present disclosure, the term "camera" can refer to any type of imaging sensor, imaging device, etc. configured to generate an image based on incident light. In one or more embodiments, the eye tracking or gaze tracking system is included in a head-mounted device.

[0011] In a first aspect, there is provided an embodiment of a method for classifying flashes using an eye tracking system of a head-mounted device, the method comprising:

[0012] - Using a processing circuit to obtain the respective positions of any flashes present in a current image of a first eye of a user of the head-mounted device that is being illuminated by one or more illuminators, wherein the current image is captured at a subsequent moment;

[0013] - Using the processing circuit to obtain a data set that indicates the respective positions of at least one flash detected in one or more previously captured images of the first eye being illuminated by one or more illuminators;

[0014] Wherein, for each obtained flash present in the current image, the method further comprises:

[0015] - Using a processing circuit to determine whether the position of the flash in the current image corresponds to the position of any of the detected flashes in the dataset; and

[0016] - If the position of the flash in the current image corresponds to the position of any of the detected flashes in the dataset, using the processing circuit to classify the flash as a static flash.

[0017] In some embodiments, the method may further comprise: before using the processing circuit to determine whether the position of the flash in the current image corresponds to any of the detected flashes in the dataset, using the processing circuit to perform eye tracking to determine whether the first eye of the user of the head-mounted device has moved between a previous moment and a subsequent moment; if it is determined that the first eye has moved between the previous moment and the subsequent moment, using the processing circuit to update the dataset based on the corresponding positions of any flashes present in the current image.

[0018] The method may further comprise using the processing circuit to exclude any static flashes for further processing in the eye tracking system.

[0019] According to a second aspect, there is provided an embodiment of an eye tracking system for a head-mounted device for classifying flashes, the system comprising a processing circuit configured to:

[0020] - Obtain the corresponding positions of any flashes present in a current image of the first eye of the user of the head-mounted device being illuminated by one or more illuminators, wherein the current image is captured at a subsequent moment;

[0021] - Obtain a dataset indicating the corresponding positions of at least one flash detected in one or more previously captured images of the first eye being illuminated by one or more illuminators,

[0022] Wherein, for each of the at least one obtained flash present in the current image, the processing circuit is further configured to:

[0023] - Determine whether the position of the flash in the current image corresponds to the position of any of the detected flashes in the dataset; and

[0024] - If the position of the flash in the current image corresponds to the position of any of the detected flashes in the dataset, classify the flash as a static flash.

[0025] In some embodiments, the processing circuitry may further be configured to: perform eye tracking to determine whether a first eye of a user of the head-mounted device has moved between a previous moment and a subsequent moment if a position of a flash in the current image corresponds to a position of any of the detected flashes in the dataset; and update the dataset based on a corresponding position of any flash present in the current image if it is determined that the first eye has moved between the previous moment and the subsequent moment.

[0026] The processing circuitry may further be configured to exclude any static flashes for further processing in the eye tracking system.

[0027] In a third aspect, there is provided a head-mounted device comprising an eye tracking system according to any of the embodiments given herein.

[0028] According to a fourth aspect, there is provided a non-transitory computer-readable storage medium storing instructions that, when executed by a processing circuitry of a system, cause the system to:

[0029] - obtain, using the processing circuitry, a corresponding position of any flash present in a current image of a first eye of a user of the head-mounted device being illuminated by one or more illuminators, wherein the current image is captured at a subsequent moment;

[0030] - obtain, using the processing circuitry, a dataset indicating corresponding positions of at least one flash detected in one or more previously captured images of the first eye being illuminated by one or more illuminators,

[0031] wherein the non-transitory computer-readable storage medium further stores instructions that, when executed by the processing circuitry (210) of the system (200), cause the system, for each obtained flash present in the current image, to:

[0032] - determine, using the processing circuitry, whether a position of the flash in the current image corresponds to a position of any of the detected flashes in the dataset; and

[0033] - classify the flash as a static flash using the processing circuitry if a position of the flash in the current image corresponds to a position of any of the detected flashes in the dataset.

[0034] The non - transitory computer - readable storage medium may further store instructions that, when executed by the processing circuitry of the system, cause the system to: perform eye tracking to determine whether a first eye of a user of the head - mounted device has moved between a previous moment and a successor moment if a position of a flash in the current image corresponds to the position of any of the detected flashes in the data set; and update the data set based on the respective positions of any flashes present in the current image if it is determined that the first eye has moved between the previous moment and the successor moment.

[0035] The non - transitory computer - readable storage medium may further store instructions that, when executed by the processing circuitry of the system, cause the system to exclude any static flashes for further processing in the eye - tracking system.

[0036] The effects and / or advantages presented in the present disclosure for embodiments of the method according to the first aspect may also apply to corresponding embodiments of the non - transitory computer - readable storage medium according to the third aspect.

[0037] It should be noted that embodiments of the present disclosure relate to all possible combinations of the features recited in the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Hereinafter, example embodiments will be described in more detail with reference to the accompanying drawings, in which:

[0039] Figure 1 An image of a user's eye is shown;

[0040] Figure 2 is a schematic overview of an eye - tracking system for classifying flashes according to one or more embodiments;

[0041] Figure 3 is a flowchart of a method for classifying flashes according to one or more embodiments;

[0042] Figure 4a A schematic illustration of a user wearing a head - mounted device is shown;

[0043] Figure 4b A schematic illustration of a head - mounted device according to one or more embodiments is shown;

[0044] Figure 5 A schematic overview of a user's eye, a reflective surface (e.g., which represents glasses worn by the user), and components of a head - mounted device including eye - tracking equipment is shown;

[0045] Figure 6 Examples of true and false flashes captured in an image of a user's eye are shown;

[0046] Figure 7 shows a static flash image, a current image, and an updated static flash image according to one or more embodiments; and

[0047] Figure 8 shows an adjustment of a current static flash image according to one or more embodiments.

[0048] All the figures are schematic, not necessarily drawn to scale, and generally show only the parts necessary to illustrate the embodiments, while other parts may be omitted or only implicitly shown. Unless otherwise indicated, any reference numerals that appear in multiple figures refer to the same object or feature in all figures. Detailed Description

[0049] Introduction

[0050] Throughout this disclosure, the term "head-mounted device" refers to a device adapted to be worn on a user's head. The head-mounted device of the present invention has an eye tracking system. Generally, the head-mounted device is configured to be arranged on the user's head such that each of one or more cameras of the eye tracking system of the head-mounted device is positioned in front of the user's face and oriented towards one or both of the user's eyes, such that the one or more cameras can capture images including one or both of the user's eyes.

[0051] It is assumed that the head-mounted device is fixedly mounted on the user's head during use, which means that the head-mounted device is not configured or intended to move relative to the user's head during use. Thus, a known fixed relationship is provided between at least one camera of the eye tracking system of the head-mounted device and any glasses worn by the user wearing the head-mounted device. In one or more embodiments, a known fixed relationship can be provided between at least one camera of the eye tracking system of the head-mounted device, at least one illuminator of the eye tracking system of the head-mounted device, and any glasses worn by the user wearing the head-mounted device. In some embodiments, the head-mounted device can be a pair of glasses configured to allow any personal glasses of the user to be worn under the head-mounted device. In all embodiments herein, it is assumed that any personal glasses worn by the user of the head-mounted device will be positioned between the user's eyes and one or more cameras of the eye tracking system of the head-mounted device during use. Therefore, if a user of a head-mounted device for XR wears glasses when using the head-mounted device, the problems described in the background art may occur because light reflections from the glasses may cause false flashes to be detected by the eye tracking or gaze tracking system of the head-mounted device.

[0052] As long as the user's head remains stationary and the corneal globe rotates around a fixed point, the flash position remains relatively fixed in the observer's image domain. However, if the user's head does not remain stationary or the corneal globe does not rotate around a fixed point due to the user moving both eyes or the line of sight, the flash position will no longer remain relatively stationary in the observer's image domain. However, the inventors have recognized that since the glasses of a user wearing a head-mounted device are stationary relative to any camera included in the eye tracking or line of sight tracking system of the head-mounted device, any flash caused by light reflected in the user's glasses from a designated illuminator included in the eye tracking or line of sight tracking system of the head-mounted device will be in the same position in any image captured by the designated camera of the eye tracking or line of sight tracking system, even if the user's eyes or line of sight move. A flash having the following flash position is hereinafter referred to as a "static" flash: the flash position remains relatively stationary in the observer's image domain even if the user's eyes or line of sight move. Thus, a static flash according to the present invention is a "false" flash, i.e., a flash not caused by reflection from the eyes, and thus this static flash should not be included in the processing of the captured images for eye tracking and / or line of sight tracking purposes.

[0053] The inventors have further recognized that if it is first determined that the user's eyes or line of sight have moved between a previous moment and a subsequent moment, any detected flash found in the same corresponding position in the image captured at the subsequent moment and also in the image captured at the previous moment can be classified as a static flash caused by the glasses and thus not a static flash caused by the user's eyes.

[0054] Existing solutions for detecting false flashes are generally complex because these solutions may require one or more illuminators configured to emit light at alternating moments and one or more cameras configured to capture images at the same alternating moments, plus computationally intensive image processing taking into account parameters such as an eye model, a camera model, a line of sight direction, and / or a light model. Turning on or off a single or multiple LEDs to identify false flashes is sometimes referred to as using LED logic. Examples of solutions using LED logic disclosed in the prior art can be found, for example, in patent document US 9338382 A1.

[0055] An advantage of the solution according to the embodiments given herein is that it does not require the use of LED logic to identify static flashes. According to any of the embodiments given herein, by classifying the flashes caused by the user's glasses as static flashes, a reliable solution for detecting false flashes is obtained, which advantageously has significantly less computational load, less time consumption, and lower complexity compared to existing solutions for detecting false flashes.

[0056] A further object of embodiments of the present disclosure is to solve or at least ameliorate the following problem: false flashes (represented herein by static flashes) are treated as true flashes in an eye tracking or gaze tracking system of a head-mounted device, thereby negatively affecting the results of eye or gaze tracking. This is achieved by excluding flashes classified as static from further processing in the eye tracking or gaze tracking system of the head-mounted device.

[0057] Accordingly, an obvious advantage of identifying static flashes and excluding them from further processing is that the eye tracking or gaze tracking results become more accurate.

[0058] Another advantage is that if the eye tracking or gaze tracking results become more accurate, the visual results and user experience are improved.

[0059] Yet another advantage is that image processing and / or computations including the flash detection results become less computationally intensive, faster, and more efficient because the processing circuitry will not have to consider static flashes.

[0060] In some embodiments, where the optics of the head-mounted device include a Fresnel lens, in these embodiments, secondary reflections may additionally be caused by reflections in the recesses of the Fresnel lens, which results in multiple false flashes being possibly caused in the captured image by each light reflection from the glasses. This additional plurality of false flashes caused by reflections from the glasses is also addressed by embodiments of the present disclosure.

[0061] The embodiments given herein are best implemented in an environment where there are no false flashes caused by any external light source in the images captured by one or more cameras of an eye tracking or gaze tracking system, because such false flashes cannot be actively classified as static or non-static flashes according to the embodiments presented herein. This preferred condition is, for example, achieved in a head-mounted device that is configured to prevent light from any external light source from entering, such as by including a housing designed to prevent light from external light sources from entering. Such a head-mounted device is hereinafter referred to as a "closed" head-mounted device. However, the same preferred condition can also be obtained in a head-mounted device that is at least partially "open" (which allows light from external light sources to enter to a certain extent) or in other system configurations where if the system is used in an environment where there is no external light source that emits light in a manner that could cause false flashes in the images captured by any of the cameras in an eye tracking or gaze tracking system and there is a fixed relationship between the user's glasses and one or more cameras of the eye tracking or gaze tracking system being used.

[0062] For an "open" head-mounted device that employs eye tracking or gaze tracking or even potentially remote eye tracking or gaze tracking systems (which to some extent allow light from external light sources to enter), the embodiments herein can also work very well. In these cases, false flashes that are static according to the definitions herein (i.e., caused by light from the illuminator of the eye tracking or gaze tracking system that is reflected in the user's glasses) will still be removed, thereby improving eye tracking or gaze tracking results and making image processing and / or calculations that include flash detection results less computationally intensive, faster, and more efficient.

[0063] In some embodiments, the head-mounted device may be configured for extended reality (XR) applications. Hereinafter, when the term "XR" is used, this should be interpreted as including at least the selection of these technologies known as VR, AR, and MR. Nowadays, eye tracking and gaze tracking solutions are increasingly commonly used in head-mounted devices for virtual reality (VR), augmented reality (AR), mixed reality (MR), or other extended reality (XR) experiences.

[0064] The term "eye tracking" as used herein can be understood to include: tracking or observing the physical parts of an eye in the real world, in a 3D model of the eye, or in a 2D image depicting the eye; or determining the object that the eye is tracking or fixating on. Determining the object that the eye is tracking or fixating on can also be referred to as gaze tracking.

[0065] Of course, any embodiment described herein as relating to one eye of a user (e.g., the first eye) is equally applicable to any eye of the user and can also be performed in parallel or sequentially for both eyes of the user.

[0066] Throughout this disclosure, the term "obtaining information" can be understood to receive information in a push manner and / or retrieve information in a pull manner.

[0067] Reference will be made below to Figures 1 to 8 Describe a method, system, and associated storage medium for classifying flashes using an eye tracking system (200) of a head-mounted device (260). First, reference will be made to Figure 1 Describe certain features of the eye.

[0068] Figure 1 is a front view of the eye 100. Figure 5 Includes a cross-sectional view of the eye 100 observed from the side of the eye 100. Although Figure 5 almost shows the entire eye 100, Figure 1The front view presented only shows those parts of the eye 100 that are generally visible from the front of the human face. The eye 100 has a cornea 101 and a pupil 102 with a pupil center 103 and a pupil edge 104. In Figure 1 a flash 105 is shown, in this case a "real" or "non-static" flash 106 caused by reflection from the eye 100.

[0069] System architecture

[0070] Reference will now be made to Figure 2 and Figure 4a and Figure 4b and Figure 5 to describe system embodiments.

[0071] Figure 2 is a schematic overview of an eye tracking system 200 of a head-mounted device 260 for classifying a flash 105 according to one or more embodiments. In Figure 4a a user 270 wearing the head-mounted device 260 is schematically illustrated. In Figure 4b the head-mounted device 260 according to embodiments herein is schematically illustrated. The system 200 according to any of the embodiments given herein can be incorporated into or communicatively connected to such a head-mounted device 260. Figure 5 shows a schematic overview of the eye 100 of the user 270, a reflective surface 108 (such as representing glasses worn by the user), components of the head-mounted device 260 including the eye tracking equipment 220, and the processing circuit 210. First referring to Figure 2 and Figure 4b , the system 200 includes a processing circuit 210 configured to obtain a current image image of the first eye 100 of the user 270 of the head-mounted device 260 t+1 of the respective positions of any flash 105 present in, the first eye 100 being illuminated by one or more illuminators 112, 113 at the time of capture, wherein the current image (image t+1 ) is captured at a subsequent time t+1.

[0072] The processing circuit 210 is further configured to: obtain a data set indicating the respective positions of at least one flash 105 detected in one or more previously captured images of the first eye 100 illuminated by one or more illuminators 112, 113; and for each flash 105 of the at least one flash 105 obtained that is present in the current image (image t+1 ), determine whether the position of the flash 105 in the current image (image t+1 ) corresponds to the position of any flash 105 detected in the data set. If the current image (imaget+1 ) if the position of the detected flash 105 in the dataset corresponds to the position of any flash in the flash 105 in the dataset, the processing circuit 210 is configured to classify the flash 105 as a static flash 107.

[0073] According to any of the embodiments described herein, in order to always present the most recently available information in the dataset for classifying flashes in the current image (image t+1 ), the processing circuit 210 may further advantageously be configured to, before determining whether the position of the flash 105 in the current image (image t+1 ) corresponds to the position of any flash in the detected flash 105 in the dataset for each of the at least one flash 105 obtained in the current image (image t+1 ): perform eye tracking to determine whether the first eye 100 of the user 270 of the head-mounted device 260 has moved between a previous time t and a subsequent time t + 1, and if it is determined that the first eye 100 has moved between the previous time t and the subsequent time t + 1, update the dataset based on the corresponding position of the flash 105 present in the current image (image t+1 ).

[0074] In one or more embodiments, where the eye tracking system 200 is configured to perform eye tracking based on the detected flash to track the eyes or line of sight of the user 270, the processing circuit 210 may further be configured to exclude any static flash 107, i.e., any flash 105 classified as static, from further processing in the eye tracking system 200. In other words, in these embodiments, any flash classified as static will not be used in the process of performing eye tracking based on the detected flash to track the eyes or line of sight of the user 270.

[0075] As described herein, Figure 4a , Figure 4b and Figure 5The head-mounted device 260 illustrated in the figures may include or alternatively be communicatively coupled to a processing circuit 210 configured according to any of the embodiments presented herein. The head-mounted device 260 may include display optics. The display optics should be understood to include any optics suitable for generating and / or displaying 2D image data, 3D image data, graphics data, holographic data, or other content that may be presented to the user / wearer of the head-mounted device to provide a VR, AR, MR, or other XR experience. The display optics may include one or more displays 111, such as a single display 111 positioned in front of the eyes of the user 270, or one display 111 positioned in front of the first eye 100 of the user 270 and a second display positioned in front of the second eye of the user 270. In some embodiments, the display optics may include a lens 109. As described herein, if the lens is a Fresnel lens, secondary reflections may additionally be caused by reflections in the recesses of the Fresnel lens, which results in multiple false flashes in the captured image for each light reflection from glasses or other reflective surfaces 108. Such additional multiple false flashes caused by reflections from glasses are also addressed by embodiments of the present disclosure. In addition to the actual display optics, the head-mounted device typically also includes other components. These other components may include, for example, a circuit for powering the head-mounted device, sensors for detecting the movement of the head-mounted device, eye or gaze tracking equipment, or a housing for protecting the components of the head-mounted device. In other words, the term "head-mounted device" may, but does not necessarily, be interpreted to refer only to the actual display optics intended to be disposed in front of the eyes of the user or in front of the user's binocular eyes.

[0076] As Figure 4b and Figure 5 shown, in some non-limiting embodiments, the processing circuit 210 may include or be communicatively coupled to an eye tracking equipment 220, which is, for example, in the form of one or more illuminators 112, 113 for illuminating the eyes 100 of the user 270 and one or more cameras 110 for capturing images of the eyes 100 when the eyes 100 are looking / gazing at the display 111 or looking / gazing in another direction at the display 111, or any other suitable eye tracking equipment known in the art. The processing circuit 210 may be communicatively coupled to the eye tracking equipment 220, for example, via a wired or wireless connection. The illuminators 112, 113 may be, for example, infrared or near-infrared illuminators, such as in the form of light-emitting diodes (LEDs). However, other types of illuminators may also be contemplated. The one or more cameras 110 may be, for example, charge-coupled device (CCD) cameras or complementary metal-oxide-semiconductor (CMOS) cameras. However, other types of cameras may also be contemplated.

[0077] The system 200 may further include a display 111 or be communicatively connected to the display 111. The display 111 may be, for example, a liquid crystal display (LCD) or an LED display. However, other types of displays are also conceivable. The display may be, for example, a flat display or a curved display. The display 111 may be placed, for example, in front of one of the user's eyes. For any or all of the embodiments herein, separate displays 111 may alternatively be employed for the left and right eyes. Similarly, separate eye tracking devices 220 (such as illuminators and cameras) may be employed for the left and right eyes, for example.

[0078] The processing circuit 210 may be used for binocular eye tracking, or separate processing circuits 210 may exist for the left and right eyes. The system 200 may perform eye tracking, for example, to determine the movement of at least one or both of the left and right eyes, and then may detect and classify the glints in the image of at least one eye in which movement has been detected.

[0079] The processing circuit 210 may include, for example, one or more processors. The processor may be, for example, an application specific integrated circuit (ASIC) configured to execute specific eye tracking and position determination methods. Alternatively, the processor may be configured to execute instructions (e.g., in the form of a computer program) stored in one or more memories 240. Such a memory 240 may be included in the system 200, for example, or may be external to the system 200 (e.g., located at a position remote from the system 200). The memory 240 may store instructions for causing the system 200 to execute the method according to any Figure 3 of the embodiments given in connection with

[0080] In one or more embodiments, the processing circuit 210 may be configured to execute any or all of the method embodiments described in connection with Figure 3

[0081] It will be understood that the system 200 described above with reference to Figures 2 to 4b is provided as an example, and many other systems are conceivable. For example, the system 200 may consist only of the processing circuit 210. The display 111 may be included in the system 200, for example, or may be considered separate from the system 200.

[0082] Eye tracking performed by systems such as the system 200 in Figure 2 typically employs an eye model. This eye model is calibrated according to the characteristics of one or both eyes of each user.

[0083] Figure 6 Examples of true glints and static glints captured in an image of the user 270's eye 100 are shown. InFigure 6 In which, this is illustrated by a previous image (image t ) captured at times t and t + 1 respectively, and a current image (image t+1 ). Each of the previous image (image t ) and the current image (image t+1 ) includes at least a portion of the cornea 101 of the eye 100 of the user 270. The positions of the cornea 101 in the previous image (image t ) and the current image (image t+1 ) are indicated by dashed outlines. These positions are indicated for illustrative purposes and do not need to be determined by the method and system embodiments given herein. As can be seen from Figure 6 , the previous image (image t ) includes two flashes 105 and 105', and the current image (image t+1 ) includes two flashes 105'' and 105'''. As can be further seen from Figure 6 , the flashes 105 and 105''' are reflected from the cornea 101 of the eye 100 of the user 270, so they are true flashes 106, while the flashes 105' and 105'' are not caused by reflections from the cornea 101 of the eye 100 of the user 270, so they are false flashes 107. The false flashes should be excluded from further processing of the eye tracking system 200 in order to improve accuracy, save time, and save the computing power of the eye tracking system 200. Advantageously, according to the embodiments described herein, the flashes 105' and 105'' will be classified as static flashes 107 (representing the same static flashes in both the previous image (image t ) and the current image (image t+1 ). In some embodiments given herein, the flashes 105 and 105''' may be further classified as non-static flashes.

[0084] Method embodiments

[0085] Hereinafter, method embodiments will be described in conjunction with Figure 3 , Figure 7 and Figure 8 .

[0086] Figure 3 An embodiment of a method for classifying flashes using the eye tracking system 200 of the head-mounted device 260 is shown, the method comprising:

[0087] In step 310: using the processing circuit 210 to obtain the corresponding positions of any flashes 105 present in the current image (image t+1 ) of the first eye 100 of the user 270 of the head-mounted device 260 being illuminated by one or more illuminators 112, 113, wherein the current image (imaget+1 ) is captured at a subsequent time t+1.

[0088] The current image (image t+1 ) may or may not be the same image as the second image (image 2).

[0089] In one or more embodiments, the position of the flash 105 may be defined as a pixel coordinate or a sub-pixel coordinate P i,j , representing the approximate center of the detected flash 105. Optionally, if the detected flash 105 is approximated as a circle or an ellipse, the information representing the position of the flash 105 may further include one or more radius parameters. In other embodiments, the position of the flash 105 may be defined as a set of connected pixels having any suitable shape and size.

[0090] In some embodiments, obtaining the corresponding position of any flash 105 present in the current image (image t+1 ) includes: using the processing circuit 210 to perform eye tracking in any known manner in the art to detect any flash 105 present in the current image (image t+1 ). In these embodiments, if the flash position is defined as more than one pixel or sub-pixel coordinate (e.g., in the case of a set of connected pixels in a circular, elliptical or other shape), the method step may further include: comparing the distance between two detected flashes, and if the distance is less than a preset threshold T EQUAL , determining that the detected flashes 105 are part of the same flash 105 and treating them as a single flash 105 in the subsequent method steps.

[0091] In other embodiments, obtaining the corresponding position of any flash 105 present in the current image image t+1 includes: retrieving or receiving in the processing circuit 210 the detected corresponding positions of any flash 105 present in the current image (image t+1 ) from a memory (such as the memory 240).

[0092] In some embodiments, the current image (image t+1 ) of the first eye 100 may be the same image as the second image (image 2) described in the embodiment of step 330.

[0093] In step 320: using the processing circuit 210 to obtain a data set that indicates the corresponding positions of at least one flash 105 detected in one or more previously captured images of the first eye 100 illuminated by one or more illuminators 112, 113.

[0094] In one or more embodiments, each of one or more previously captured images of the first eye 100 illuminated by one or more illuminators 112, 113 is captured at a corresponding time that occurs before a subsequent time t+1.

[0095] In one or more embodiments of step 330, obtaining a data set using the processing circuit 210 can be understood as receiving or retrieving the data set from a memory (e.g., from the memory 240), the data set indicating the respective positions of at least one flash 105 detected in one or more previously captured images of the first eye 100 illuminated by one or more illuminators 112, 113.

[0096] In some non-limiting embodiments, obtaining a data set (the data set indicating the respective positions of at least one flash 105 detected in one or more previously captured images of the first eye 100 illuminated by one or more illuminators 112, 113) using the processing circuit 210 can include: using the processing circuit 210 to generate a data set based on one or more previously captured images of the first eye 100 illuminated by one or more illuminators 112, 113, the data set indicating the respective positions of at least one flash 105 detected in one or more previously captured images of the first eye 100 illuminated by one or more illuminators 112, 113.

[0097] In one or more embodiments, the data set indicating at least one flash 105 detected in one or more previously captured images of the first eye 100 is a static flash image (image 静态 )), where the luminance value I' associated with each corresponding pixel or sub-pixel coordinate P in the static flash image (image 静态 i,j i,j corresponds to the minimum luminance value among the luminance values of the pixel or sub-pixel coordinate P i,j in one or more previously captured images.

[0098] Of course, the static flash image (image 静态 ) does not need to have the same size and / or resolution as the current image (image t+1 ), but can alternatively have a different size and / or resolution. In a non-limiting example, the static flash image (image 静态 ) can be smaller than the current image (image t+1 ), and include a subset of the current image (image t+1 ), such as the eye 100 of the user 270 or a part of the eye 100 of the user 270, while the current image (image t+1)More portions around the eye 100, including the eye 100 or a part of the eye 100 and the face of the user 270. In such embodiments, in one aspect, the pixel or sub-pixel coordinates P of the current image (image t+1 ) and, on the other hand, the mapping function, mapping relationship, etc. between the pixel or sub-pixel coordinates of the static flash image (image i,j ) can be predefined or calculated. In some embodiments, the method may include: determining, based on such mapping function, mapping relationship, etc., which pixel or sub-pixel coordinates of the static flash image (image 静态 ) are associated with the specified pixel or sub-pixel coordinates P of the current image (image 静态 ). t+1 ) i,j are associated.

[0099] In other embodiments, the data set indicating at least one flash 105 detected in one or more previously captured images of the first eye 100 may be a list that includes one or more pixel or sub-pixel coordinates P associated with each detected flash 105 in each of the one or more previously captured images of the first eye 100. i,j . One or more previously captured images may include one, all, or a selection of the previously captured images. The selection may be based on a preset criterion. In a non-limiting example, the last X captured frames (X is a suitably selected integer greater than 0) are included in the data set, where the previously captured images used to generate the data set are cached in a first-in, first-out (FIFO) manner of the captured images.

[0100] In optional step 330: Use the processing circuit 210 to perform eye tracking to determine whether the first eye 100 of the user 270 of the head-mounted device 260 has moved between a previous time t and a subsequent time t + 1.

[0101] Performing eye tracking using the processing circuit 210 to determine whether the first eye 100 has moved can be done in any known manner in the art.

[0102] Determining whether the eye has moved may include determining movement in a two-dimensional space (2D) by: performing eye tracking to estimate the 2D positions of the identified eye features (e.g., the center of the eyeball, the cornea, the center of the cornea, the pupil, the center of the pupil, the edge of the pupil, etc.) in the image captured at the previous time t and in the image captured at the subsequent time t + 1 respectively; and comparing the positions of the identified eye features in the image captured at the previous time t and the image captured at the subsequent time t + 1.

[0103] In a non-limiting example, determining whether a first eye 100 has moved between a previous time t and a successor time t+1 using processing circuitry 210 may include: obtaining, using processing circuitry 210, a first image (Image 1) of a first eye 100 of a user 270 of a head-mounted device 260, where the first image (Image 1) is captured at the previous time t, and obtaining, using processing circuitry 210, a second image (Image 2) of the first eye 100, where the second image (Image 2) is captured at the successor time t+1. Thereafter, the method may include: determining, using processing circuitry 210 and based on the first image (Image 1) and the second image (Image 2), whether the first eye 100 has moved between the previous time t and the current time t+1.

[0104] In any of the embodiments described herein, obtaining an image may include: receiving or retrieving an image from a camera 110 or a memory 240 of an eye tracking system 200 or from an external memory, or capturing an image using a camera 110 of the eye tracking system 200.

[0105] Movement between images captured at different times may be determined in any known manner, such as by determining a pixel distance between positions of eye features detected in the images and comparing the pixel distance with a predetermined threshold T 距离 and comparing. Alternatively or in combination with the above option, determining whether an eye has moved may include: determining whether the image content has changed from time t to time t+1 based on an image of the eye captured using a camera of the eye tracking system 200 at time t and an image of the eye captured using the same camera at time t+1. In some embodiments, determining whether the image content has changed may include: comparing pixel values of corresponding pixel coordinates in two captured images, optionally including generating a difference image based on the two captured images. Based on the comparison or based on the difference image, determining whether the image content has changed from time t to time t+1 may include: for each pixel or a predetermined number of pixels, determining whether there is a difference between pixel values of corresponding pixels in the two captured images. In some embodiments, the determination further includes: determining that the eye has moved from time t to time t+1 if there is a difference exceeding a predetermined threshold between pixel values of the two captured images for a plurality of pixel coordinates.

[0106] Alternatively or in combination with any one or both of the above options, determining whether an eye has moved may include: determining movement of the eye in three-dimensional space based on a comparison of estimated positions of eye features (such as the center of the eyeball, the cornea, the center of the cornea, the pupil, the center of the pupil, the edge of the pupil, etc.) at times t and t+1, respectively.

[0107] The processing circuit 210 can accordingly be configured to determine whether the first eye 100 has moved between a previous time t and a successor time t + 1 by performing any of the method embodiments described in connection with step 320.

[0108] If it is determined that the first eye 100 has moved between the previous time t and the successor time t + 1, the method continues with optional method step 331.

[0109] In optional step 331: The dataset is updated using the processing circuit (210) based on the respective positions of any flashes (105) present in the current image (image t+1 ).

[0110] Performing the optional method step 330 of checking for eye movement and, if eye movement is detected, performing step 331 of updating the dataset with the positions of any flashes 105 present in the current image (image t+1 ) has the advantage that the execution of these steps ensures that the dataset used for classifying the flashes in the current image (image t+1 ) contains the latest available information. Thereby, the classification is further improved.

[0111] In an embodiment where the dataset is a static flash image (image 静态 ), updating the dataset based on the respective positions of any flashes 105 present in the current image (image t+1 ) can include: using the processing circuit 210, by setting the static flash image (image 静态 ) according to the following equation for each pixel or sub - pixel coordinate P i,j associated with the respective luminance value I' i,j in the static flash image (image 静态 ):

[0112]

[0113] where is the updated current luminance value of the pixel or sub - pixel having pixel or sub - pixel coordinate P 静态 in the updated static flash image (image i,j '); is the luminance value of the pixel or sub - pixel having pixel or sub - pixel coordinate P 静态 in the static flash image (image i,j ) at time t; and is the luminance value of the pixel or sub - pixel having pixel or sub - pixel coordinate P t+1 in the current image (image i,jThe brightness value of a pixel or sub - pixel. In these embodiments, the processing circuit 210 may be configured to, after obtaining a data set, based on a static flash image (image 静态 ), by setting, according to Equation 1, the corresponding brightness value I' 静态 associated with each pixel or sub - pixel coordinate P i,j in the current static flash image (image' i,j ) to generate the current static flash image (image' 静态 ).

[0114] Thus, any pixel or sub - pixel representing the flash present in all the captured images, i.e., the pixels that have been processed to generate the static flash image (image 静态 ) and the current image (image t+1 ), will be bright, i.e., have a high brightness value, while all other pixels will be much darker, i.e., have a low brightness value. This is illustrated in Figure 7 . Figure 7 Shows the static flash image (image 静态 ), the current image (image t+1 ), and the updated static flash image (image 静态 '), for ease of illustration, they all have the same resolution in the figure. High - intensity pixels are illustrated as white pixels in Figure 7 . Pixels having a high brightness value at the corresponding pixel positions in both the static flash image (image 静态 ) and the current image (image t+1 ), which are pixels having the same pixel positions due to the same image size in this example, will also have a high brightness value in the updated static flash image (image 静态 '). For example, in the illustration of Figure 7 , the pixel or sub - pixel having the pixel or sub - pixel coordinate P 4,2 is such a case. On the other hand, pixels having corresponding pixel positions in the static flash image (image 静态 ) and the current image (image t+1 ) but having a high brightness value (shown as white in the figure) only in one of those images will be dark because it is given the lower of the two pixel brightness values in the updated static flash image (image 静态 '). For example, in the illustration of Figure 7 , the pixel or sub - pixel having the pixel or sub - pixel coordinate P 1,1 is such a case.

[0115] After step 320 or after optional steps 330 and 331, the method further includes: using processing circuit 210 to classify each of the at least one flash 105 obtained in the current image (image t+1 ) according to the following method steps 340, 350 and also optional step 360.

[0116] In an embodiment where the data set is a list, updating the data set based on the respective positions of any flashes 105 present in the current image (image t+1 ) may include: adding one or more pixel or sub-pixel coordinates P t+1 associated with each of the at least one flash 105 obtained in the current image (image i,j ).

[0117] In some embodiments, the addition of flash information related to the current image (image t+1 ) may result in the deletion of the flash information of the oldest image (the respective positions of any flashes 105 present in the image are currently stored in the list). Thus, the manageable size of the list is maintained.

[0118] In step 340: determine whether the position of the flash 105 in the current image (image t+1 ) corresponds to the position of any of the detected flashes 105 in the data set.

[0119] In some embodiments, the current image (image t+1 ) and one or more previously captured images each include i*j pixels, where the position of the flash 105 is defined as one or more pixel or sub-pixel coordinates P i,j associated with each of the respective detected flashes 105, where the data set includes one or more pixel or sub-pixel coordinates P t+1 associated with each of the respective at least one flash 105 obtained in the current image (image i,j ) and the brightness value I i,j of each pixel or sub-pixel coordinate. In some embodiments, the brightness value I t+1 of each pixel or sub-pixel coordinate included in the data set and associated with each of the respective at least one flash 105 obtained in the current image (image i,j ) may include the individual brightness value I i,j of the pixel or sub-pixel coordinate at each moment of capturing the previous image. This is usually the case when the data set is a list. In other embodiments, the brightness value I i,j of each pixel or sub-pixel coordinate included in the data set and associated with each of the respective at least one flash 105 obtained in the current image (imaget+1 ) each of the corresponding obtained at least one flash 105 in is associated with one or more pixel or sub - pixel coordinates P i,j the luminance value I of each pixel or sub - pixel coordinate in i,j may include a single luminance value I of a pixel or sub - pixel coordinate i,j . The single luminance value I i,j may be based on the luminance value of that pixel or sub - pixel coordinate in a previously captured image according to a predefined set of rules, for example, as the minimum of the luminance values of that pixel or sub - pixel coordinate in a previously captured image. This is usually the case when the data set is a static flash image.

[0120] In one or more embodiments, for each flash 105 in the current image (image t+1 ), determining whether the position of the flash 105 in the current image (image t+1 ) corresponds to the position of any of the detected flashes 105 in the data set includes:

[0121] · For each pixel or sub - pixel coordinate P associated with the flash 105 i,j in:

[0122] ο Determine the luminance value I of the pixel or sub - pixel having one or more pixel or sub - pixel coordinates P t+1 in the current image (image i,j ) i,j and the luminance value I of the pixel or sub - pixel having pixel or sub - pixel coordinates P i,j in the data set i,j and the difference I 差 .

[0123] If the difference I 差 is below a predetermined threshold T, then determine that the position of the flash 105 in the current image (image t+1 ) corresponds to the position of the contrast flash 105 in the data set.

[0124] In these embodiments, the processing circuit 210 may be correspondingly configured to, for each obtained flash 105 present in the current image (image t+1 ), determine whether the position of any obtained flash 105 present in the current image (image t+1 ) corresponds to the position of any of the detected flashes 105 in the data set by: for each pixel or sub - pixel coordinate P associated with the flash 105 i,j in, determine that in the current image (image t+1 ) the pixel or sub - pixel has pixel or sub - pixel coordinates Pi,j The luminance value I of a pixel or sub - pixel i,j and the luminance value I' associated with the pixel or sub - pixel coordinates P i,j in the dataset i,j The difference I 差 between them. If the difference I 差 is below a predetermined threshold T, then in these embodiments, the processing circuit 210 is configured to classify the flash 105 as a static flash 107.

[0125] The number of previously captured images of the first eye 100 illuminated by one or more illuminators 112, 113 that form the basis of the dataset can vary, depending on whether it is important to keep the computational load as small as possible and thus use a smaller number of previously captured images, or whether having the highest possible quality and accuracy is important and thus a larger number of previously captured images are to be used. In some embodiments, the previously captured images used to generate the dataset are cached in a first - in - first - out (FIFO) manner to be optimized both in terms of obtaining the most recent flash detection information and the storage capacity of the memory (e.g., memory 240) storing the dataset.

[0126] It is well known that the intensity of an image pixel or sub - pixel can be represented in many different ways, depending on, for example, the image format, the selected color space, and how many information channels or bits of information are used to describe the pixel value of each pixel or sub - pixel. The embodiments presented herein do not depend on how the luminance value is represented or obtained.

[0127] In embodiments where the dataset is a list, determining whether the position of the flash 105 in the current image (image t+1 ) corresponds to the position of any of the detected flashes 105 in the dataset using the processing circuit 210 includes: for each flash among the corresponding detected flashes 105 in the current image (image t+1 ), determining, based on the list, one or more pixel or sub - pixel coordinates P t+1 associated with the flash 105 in the current image (image i,j ) and determining whether at least one of the pixel or sub - pixel coordinates is also associated with the flash 105 in each of one or more previously captured images of the first eye 100.

[0128] If at least one of the one or more pixel or sub - pixel coordinates P t+1 associated with the flash 105 in the current image (image i,j ) is also associated with the flash 105 in each of one or more previously captured images of the first eye 100, then the method includes using the processing circuit 210 to determine the current image (imaget+1 ) The position of the flash 105 in [ ] corresponds to the position of the detected flash 105 in the dataset. In other words, if the flash 105 appears at the same position or pixel / sub-pixel coordinates P in all previous frames i,j and also appears in the current image t+1 at the same position or pixel / sub-pixel coordinates P i,j then it is determined to be the same flash 105. Otherwise, if one or more of the pixel or sub-pixel coordinates P t+1 associated with the flash 105 in the current image (image i,j ) have none of their pixel or sub-pixel coordinates associated with the flash 105 in each of one or more previously captured images of the first eye 100, then the method includes using the processing circuit 210 to determine that the position of the flash 105 in the current image (image t+1 ) does not correspond to the position of any of the detected flashes 105 in the dataset. The processing circuit 210 can be correspondingly configured to determine whether the position of the flash 105 in the current image (image t+1 ) corresponds to the position of any of the detected flashes 105 in the dataset by performing any or all of these method steps. Of course, each of the one or more previously captured images of the first eye 100 does not need to have the same size and / or resolution as the current image (image t+1 ), but can alternatively have different sizes and / or resolutions. In such embodiments, on the one hand, a mapping function, mapping relationship, etc. between the pixel or sub-pixel coordinates P t+1 of the current image (image i,j ) and, on the other hand, the pixel or sub-pixel coordinates of the previously captured images of the first eye 100 can be predefined or calculated. In some embodiments, the method can include determining which pixel or sub-pixel coordinates of the previously captured images of the first eye 100 are associated with a specified pixel or sub-pixel coordinate P t+1 of the current image (image i,j ) based on such a mapping function, mapping relationship, etc.

[0129] If it is determined that the position of the flash in the current image (image t+1 ) corresponds to the position of any of the detected flashes 105 in the dataset, then the method continues at step 350.

[0130] In some embodiments, if it is determined that the position of the flash in the current image (image t+1 ) does not correspond to the position of any of the detected flashes 105 in the dataset, then the method continues with optional step 360.

[0131] In step 350: Classify flash 105 as static flash 107.

[0132] In optional step 360: Classify flash 105 as non-static flash.

[0133] In Figure 3 it is further shown a decision step 370 for more easily illustrating the fact that the method includes classifying each of the at least one obtained flash 105 present in the current image (image t+1 ). However, the inclusion of step 370 can clearly be considered to be implicitly disclosed by the above method description of the specification, that is, the method includes classifying each of the at least one obtained flash 105 present in the current image (image t+1 ).

[0134] Step 370 (as an optional independent step or implicitly disclosed by previous method steps) includes checking whether there are any more flashes 105 to be processed or classified among the at least one obtained flash 105 present in the current image (image t+1 ), and if there are any more flashes 105 to be processed or classified among the at least one obtained flash 105 present in the current image (image t+1 ): then return to step 340 for processing or classifying the next flash 105; or if there are no more flashes 105 to be processed or classified among the at least one obtained flash 105 present in the current image (image t+1 ): then end the method.

[0135] In combination with any of the embodiments given herein for classifying flashes, the method may further include excluding any static flash 107 from further processing in the eye tracking system 200, that is, a flash classified as static according to the method embodiments described herein. In other words, false flashes in the form of static flashes will not affect the results of any subsequent eye tracking or gaze tracking performed by the system 200.

[0136] If during use the head-mounted device 260 moves relative to the eyes or glasses of the user 270 (e.g., by sliding of the head-mounted device 260 when the user 270 moves, or if the user adjusts the position of the glasses), this may temporarily and negatively affect the accuracy of the eye tracking function because the information about the previously classified static flash 107 may no longer be applicable. For example, in an embodiment where the data set is a static flash image (image 静态 ), the information about all the static flashes 107 in the static flash image (image 静态 ) may be lost because the pixel or sub-pixel coordinates P i,jIt is likely that it will no longer be true that it still includes a relatively high brightness value / is bright. Even if it is still bright, it does not represent the same flash, which is also a loss of information. And since according to these embodiments, the pixel value of each pixel or sub-pixel coordinate P i,j is set to the minimum value of the brightness value I 静态 of the pixel or sub-pixel coordinate P t+1 in the static flash image (image i,j ) and in the current image (image i,j ), that is, its historical lowest value based on the available pixel value information. Therefore, pixels that have erroneously received a lower brightness value when the pixel actually represents a static flash 107 will not be able to restore their correct, brighter brightness value.

[0137] In embodiments where the data set is a static flash image (image 静态 ), to improve this problem, the method may further include: using the processing circuit 210, by adjusting the brightness value of each pixel or sub-pixel coordinate P 静态 in the current static flash image (image' i,j ) to compensate for possible information loss caused by sliding, etc., thereby obtaining an adjusted static flash image (image' 静态_经调整的 静态_经调整的 ).

[0138] In one or more embodiments, the method includes: using the processing circuit 210, by setting the brightness value I' 静态 of each pixel or sub-pixel coordinate P i,j in the current static flash image (image' i,j ) according to the following equation to adjust the current static flash image (image' 静态 ):

[0139]

[0140] where is the brightness value of the pixel or sub-pixel having the pixel or sub-pixel coordinate P 静态_经调整的 in the adjusted static flash image (image' i,j ); is the brightness value of the pixel or sub-pixel having the pixel or sub-pixel coordinate P 静态 in the current static flash image (image' i,j ); I max is the maximum brightness value that can be assigned to a pixel or sub-pixel; and α is a weighting value. In these embodiments, the processing circuit 210 may be configured to adjust each pixel or sub-pixel coordinate P 静态 in the current static flash image (image' i,jThe brightness value I' i,j to adjust the current static flash image (image' 静态 ).

[0141] In Figure 8 is illustrated the adjustment to the current static flash image (image' 静态 ) to compensate for possible loss of static flash information due to movement of the head-mounted device 260 relative to the user 270's eyes or glasses during use. Figure 8 Shows the Figure 7 in the example current static flash image before adjustment (image 静态 ') and the brighter adjusted static flash image obtained after adjustment (image 静态_经调整的 '). Figure 8 The current static flash image of (image' 静态 ) and the adjusted static flash image (image' 静态_经调整的 ) each include two pixel coordinates P 2,3 , P 2,3 , which pixel coordinates are associated with the respective flashes classified as static flash 107 according to the embodiments presented herein.

[0142] The weighting value α can be characterized in the form of an integer value or a floating-point value. In two non-limiting examples, the weighting value α can be a value within the interval 0 < α < 1 or the interval 0 < α < 255. Of course, the form in which the weighting value α is characterized depends on how many information bits are dedicated to its characterization and also on the sensitivity of the camera sensor pixels or sub-pixels. The adjustment described in connection with Equation 2 can be performed before, after, or in parallel with step 340 of determining whether the position of the flash 105 in the current image (image t+1 ) corresponds to any of the detected flashes 105 in the dataset.

[0143] The adjustment described above in connection with Equation 2 will advantageously make the flash classification solution more robust because compensating causes all pixels or sub-pixels of the static flash image (image 静态 ) to become brighter over time and because this static flash information lost due to movement of the head-mounted device 260 relative to the user's glasses will not result in irreparable loss of static flash information. Of course, any pixel or sub-pixel coordinates P 静态 in the static flash image (image i,j ) that do not represent static flash 107 will become darker again for each captured and processed image frame in which the brightness value of the pixel or sub-pixel coordinates P i,j is lower than that of the pixel or sub-pixel coordinates P in the static flash image (image 静态 )i,j The current brightness value, thereby displaying a still image of the still flash 107 (image 静态 ) is restored. In different embodiments, according to the system settings, for every x frames, this adjustment of the brightness value can be performed whenever an image frame is captured or at a set time interval.

[0144] After classification according to any of the embodiments given herein, the method may further include excluding any still flash 107 from further processing in the eye tracking system 200.

[0145] Additional embodiments

[0146] In one or more embodiments, a non - transitory computer - readable storage medium storing instructions is provided, which when executed by the processing circuit 210 of the system 200, cause the system 200 to perform the methods defined in any of the methods disclosed herein (in other words, in the claims, the summary of the invention, or the detailed description).

[0147] The non - transitory computer - readable storage medium may store instructions which, when executed by the processing circuit 210 of the system 200, cause the system 200 to: obtain, using the processing circuit 210, the corresponding positions of any flashes 105 present in a current image (image t+1 ) of a first eye 100 of a user 270 of the head - mounted device 260 being illuminated by one or more illuminators 112, 113, where the current image (image t+1 ) is captured at a subsequent time t + 1; obtain, using the processing circuit 210, a data set that indicates the corresponding positions of at least one flash 105 detected in one or more previously captured images of the first eye 100 being illuminated by one or more illuminators 112, 113; for each obtained flash 105 present in the current image (image t+1 ), use the processing circuit 210 to determine whether the position of the flash 105 in the current image (image t+1 ) corresponds to the position of any of the detected flashes 105 in the data set; and if the position of the flash in the current image (image t+1 ) corresponds to the position of any of the detected flashes 105 in the data set, use the processing circuit 210 to classify the flash 105 as a still flash 107.

[0148] In one or more embodiments, the non-transitory computer-readable storage medium may further store instructions that, when executed by the processing circuitry 210 of the system 200, cause the system 200 to: perform eye tracking using the processing circuitry 210 to determine whether a first eye 100 of a user 270 of the head-mounted device 260 has moved between a previous time t and a successor time t+1; and if it is determined that the first eye 100 has moved between the previous time t and the successor time t+1, update a data set using the processing circuitry (210) based on the respective positions of any flashes (105) present in the current image (image t+1 ).

[0149] The non-transitory computer-readable storage medium may be provided, for example, in a computer program product. In other words, the computer program product may include, for example, a non-transitory computer-readable storage medium storing instructions that, when executed by the processing circuitry 210 of the system 200, cause the system 200 to perform the methods as defined in any of the method embodiments.

[0150] As described above with reference to Figure 2 the storage medium need not be included in the system 200.

[0151] Those skilled in the art will appreciate that the present invention is in no way limited to the preferred embodiments described above. On the contrary, many modifications and variations are possible within the scope of the appended claims. For example, as explained herein, the embodiments described above with reference to Figure 3 may be combined to form additional embodiments. Further, it should be understood that Figure 2 the system 200 shown in Figure 3 is only intended as an example, and other systems may also perform the methods described above with reference to

[0152] It should be understood that the processing circuitry 210 (or processor) may include a combination of one or more of the following: a microprocessor, a controller, a microcontroller, a central processing unit, a digital signal processor, an application specific integrated circuit, a field programmable gate array, or any other suitable computing device, resource, or combination of hardware, software, and / or coded logic operative to provide computer functionality either alone or in combination with other computer components such as memory or storage media.

[0153] It should also be understood that a memory or storage medium (or computer-readable medium) can include any form of volatile or non-volatile computer-readable memory, including but not limited to persistent memory, solid-state memory, remotely installed memory, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), mass storage media (e.g., hard disk), removable storage media (e.g., flash drive, compact disc (CD) or digital video disc (DVD)), and / or any other volatile or non-volatile, non-transitory device-readable and / or computer-executable storage device that stores information, data, and / or instructions that can be used by a processor or processing circuitry.

[0154] Herein, from a study of the drawings, the present disclosure, and the appended claims, those skilled in the art can understand and realize other variations of the disclosed embodiments when implementing the claimed invention. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. In the claims, the word "or" is not to be interpreted as an exclusive or (sometimes referred to as "xor"). Instead, unless otherwise stated, expressions such as "A or B" cover all cases "A and not B", "B and not A", and "A and B". The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. A method for classifying a flash (105) using an eye tracking system (200) in a head-mounted device (260), the method comprising: - Obtain, using the processing circuitry (210) of the head-mounted device (260), the position of at least one flash (105) present in a current image (image t+1 ) of a first eye (100) of a user (270) of the head-mounted device (260) that is being illuminated by one or more illuminators (112, 113), wherein the current image (image t+1 ) is captured at a first moment (t+1); - Obtain a data set using the processing circuit (210), the data set indicating the position of at least one detected flash (105) in one or more previously captured images of the first eye (100) captured at corresponding times before the first time (t + 1) when illuminated by one or more illuminators (112, 113); - Perform eye tracking using the processing circuit (210) to determine whether the first eye (100) of the user (270) of the head-mounted device (260) has moved between the first time (t + 1) and the second time (t), where the second time (t) is before the first time (t + 1), and - If it is determined that the first eye (100) has moved between the first moment (t + 1) and the second moment (t), the processing circuit (210) is used to update the data set based on the corresponding positions of any flashes (105) present in the current image (image t+1 ) Among them, for each flash (105) existing in the current image (image t+1 ), the method further includes: - Use the processing circuit (210) to determine whether the position of the flash (105) in the current image (image t+1 ) corresponds to the position of any of the flashes (105) in the one or more previously captured images of the first eye (100) captured at a corresponding time before the first moment (t + 1) indicated by the data set; and - If the position of the flash in the current image (image t+1 ) corresponds to the position of any of the flashes (105) in the one or more previously captured images of the first eye (100) captured at corresponding times before the first moment (t + 1) indicated by the data set, then the processing circuit (210) is used to classify the flash (105) as a static flash (107).

2. The method according to claim 1, wherein, The current image (image t+1 ) includes i*j pixels, wherein the position of each flash in the obtained flash (105) present in the current image (image t+1 ) is defined as one or more pixel or sub-pixel coordinates (P i,j ), wherein the current image (image t+1 ) further includes a brightness value (I i,j ) associated with each pixel or sub-pixel coordinate in the one or more pixel or sub-pixel coordinates (P i,j ) of the corresponding at least one flash (105) present in the current image (image t+1 ), wherein the data set includes a brightness value (I' i,j ) of each pixel or sub-pixel coordinate in the one or more pixel or sub-pixel coordinates (P i,j ) associated with each flash in the corresponding at least one flash (105) present in the current image (image t+1 ), Among them, for each obtained flash (105) present in the current image (image t+1 ), using the processing circuit (210) to determine whether the position of any obtained flash (105) present in the current image (image t+1 ) corresponds to the position of any flash among the detected flashes (105) in the dataset, the determination includes: - For each pixel or sub - pixel coordinate (P i,j ) associated with the flash (105), determine the difference (I 差 ) between the luminance value (I i,j ) of the pixel or sub - pixel in the current image (image t+1 ) having the pixel or sub - pixel coordinate (P i,j ) and the luminance value (I’ i,j ) associated with the pixel or sub - pixel coordinate (P i,j ) in the dataset; and - If the difference (I 差 ) is below a predetermined threshold (T), then classify the flash (105) as a static flash (107).

3. The method according to claim 2, wherein, The data set indicating at least one flash (105) detected in one or more previously captured images of the first eye (100) is a static flash image (image 静态 ), wherein, for each corresponding pixel or sub-pixel coordinate (P 静态 ) in the static flash image (image i,j ), the associated pixel or sub-pixel brightness value (I' i,j ) corresponds to the minimum brightness value among the brightness values of the pixel or sub-pixel coordinates (P i,j ) in the one or more previously captured images, and wherein the method includes, after the method step of obtaining the data set: - Use the processing circuit (210) to generate a current static flash image (image 静态 ) based on the static flash image (image 静态 ), which is achieved by the following steps: setting the corresponding luminance value (I' 静态 ) associated with each pixel or sub-pixel coordinate (P i,j ) in the current static flash image (image' i,j ) according to the following equation: wherein, is the luminance value of the pixel or sub-pixel having the pixel or sub-pixel coordinates (P 静态 ) in the current static flash image (image ' i,j ); is the luminance value of the pixel or sub-pixel having the pixel or sub-pixel coordinates (P 静态 ) in the static flash image (image i,j ) at time t; and is the luminance value of the pixel or sub-pixel having the pixel or sub-pixel coordinates (P t+1 ) in the current image (image i,j ).

4. The method according to claim 3, further comprising using the processing circuit (210) to adjust the current static flash image (image' 静态 ) by setting the luminance value (I' i,j ) of each pixel or sub-pixel coordinate (P i,j ) in the current static flash image (image' 静态 ) according to the following equation: Where is the luminance value of the pixel or sub-pixel having the pixel or sub-pixel coordinate (P 静态_经调整的 ) in the adjusted static flash image (image' i,j ); is the luminance value of the pixel or sub-pixel having the pixel or sub-pixel coordinate (P 静态 ) in the current static flash image (image' i,j ); I max is the maximum luminance value that a pixel or sub - pixel can be assigned; and α is a weighting value.

5. The method according to claim 1, wherein, The data set indicating at least one flash (105) detected in one or more previously captured images of the first eye (100) is a list that includes one or more pixel or sub-pixel coordinates (P i,j ) associated with each flash in the respective detected flash (105) of each of the one or more previously captured images of the first eye (100), wherein, for each flash in the respective detected flash (105) in the current image (image t+1 ), the processing circuit (210) determines whether the position of the flash (105) in the current image (image t+1 ) corresponds to the position of any flash in the detected flash (105) in the data set, the determination including: - Determine, based on the list, whether at least one of the one or more pixel or sub-pixel coordinates (P t+1 ) associated with the flash (105) in the current image (image i,j ) is also associated with the flash (105) in each of the one or more previously captured images of the first eye (100); and - If at least one of the pixel or sub-pixel coordinates (P t+1 ) associated with the flash (105) in the current image (Image i,j ) is also associated with the flash (105) in each of one or more previously captured images of the first eye (100), then determine that the position of the flash (105) in the current image (Image t+1 ) corresponds to the position of the detected flash (105) in the dataset.

6. The method according to claim 1, wherein,Determining whether the first eye (100) has moved between the first time (t + 1) and the second time (t) using the processing circuit (210) includes: - Obtain a first image (Image 1) of the first eye (100) of the user (270) of the head-mounted device (260) using the processing circuit (210), where the first image (Image 1) is captured at the second time (t); - Obtain a second image (Image 2) of the first eye (100) using the processing circuit (210), where the second image (Image 2) is captured at the first time (t + 1); and - Use the processing circuit (210) to determine whether the first eye (100) has moved between the first time (t + 1) and the second time (t) based on the first image (Image 1) and the second image (Image 2).

7. The method according to claim 1, further comprising excluding any static flash (107) from further processing in the eye tracking system (200).

8. An eye tracking system (200) for a head-mounted device (260) configured to classify a flash (105), the system comprising processing circuitry (210) configured to: - obtain the position of at least one flash (105) present in a current image (image t+1 ) of a first eye (100) of a user (270) of the head-mounted device (260) being illuminated by one or more illuminators (112, 113), wherein, The current image (image t+1 ) is captured at a first moment (t + 1); - Obtain a data set, the data set indicating the position of at least one detected flash (105) in one or more previously captured images of the first eye (100) captured at corresponding times before the first time (t + 1) when illuminated by the one or more illuminators (112, 113); - Perform eye tracking to determine whether the first eye (100) of the user (270) of the head-mounted device (260) has moved between the first time (t + 1) and the second time (t), where the second time (t) is before the first time (t + 1), and - If it is determined that the first eye (100) has moved between the first moment (t+1) and the second moment (t), update the data set based on the corresponding positions of any flashes (105) present in the current image (image t+1 ); Among them, for each flash existing in the current image (image t+1 ), the processing circuit (210) is further configured to: - Determine whether the position of the flash (105) in the current image (Image t+1 ) corresponds to the position of any of the flashes (105) in the one or more previously captured images of the first eye (100) captured at corresponding times before the first moment (t + 1) indicated by the data set; and - If the position of the flash in the current image (image t+1 ) corresponds to the position of any of the detected flashes (105) in the one or more previously captured images of the first eye (100) captured at corresponding times before the first moment (t + 1) indicated by the data set, then classify the flash (105) as a static flash (107).

9. The eye tracking system (200) according to claim 8, wherein, The current image (image t+1 ) includes i*j pixels, wherein the position of each flash in the obtained flash (105) present in the current image (image t+1 ) is defined as one or more pixel or sub-pixel coordinates (P i,j ), wherein the current image (image t+1 ) further includes the one or more pixel or sub-pixel coordinates (P t+1 ) corresponding to at least one flash (105) obtained and present in the current image (image i,j ), and a luminance value (I i,j ) associated with each pixel or sub-pixel coordinate among the one or more pixel or sub-pixel coordinates (P t+1 ), wherein the data set further includes the luminance value (I’ i,j ) of each pixel or sub-pixel coordinate among the one or more pixel or sub-pixel coordinates (P i,j ) associated with each flash among at least one flash (105) obtained and present in the current image (image ) Wherein, the processing circuit (210) is further configured to, for each acquired flash (105) present in the current image (image t+1 ), determine whether the position of any acquired flash (105) present in the current image (image t+1 ) corresponds to the position of any flash among the detected flashes (105) in the dataset: - For each pixel or sub-pixel coordinate (P i,j ) associated with the flash (105), determine the difference (I 差 ) between the luminance value (I i,j ) of the pixel or sub-pixel in the current image (image t+1 ) having the pixel or sub-pixel coordinate (P i,j ) and the luminance value (I') associated with the pixel or sub-pixel coordinate (P i,j i,j ) in the dataset; and - If the difference (I 差 ) is below a predetermined threshold (T), then classify the flash (105) as a static flash (107).

10. The eye tracking system (200) according to claim 9, wherein, The data set is a static flash image (image 静态 ), wherein the pixel or sub-pixel brightness value (I' 静态 ) associated with each corresponding pixel or sub-pixel coordinate (P i,j ) in the static flash image (image i,j ) corresponds to the minimum brightness value among the brightness values of the pixel or sub-pixel coordinates (P i,j ) in the one or more previously captured images, wherein the processing circuit (210) is further configured to, after obtaining the data set: - Generate a current static flash image (image 静态 ) from the static flash image (image 静态 ), which is achieved by the following steps: Set the corresponding luminance value (I' 静态 ) associated with each pixel or sub-pixel coordinate (P i,j ) in the current static flash image (image' i,j ) according to the following equation: where is the luminance value of the pixel or sub-pixel having pixel or sub-pixel coordinates (P 静态 ) in the current static flash image (image ' i,j ); is the luminance value of the pixel or sub-pixel having pixel or sub-pixel coordinates (P 静态 ) in the static flash image (image i,j ) at time t; and is the luminance value of the pixel or sub-pixel having pixel or sub-pixel coordinates (P t+1 ) in the current image (image i,j ).

11. The eye tracking system (200) according to claim 10, wherein, The processing circuit (210) is further configured to adjust the current static flash image (image' 静态 ) by setting the luminance value (I' i,j ) of each pixel or sub-pixel coordinate (P i,j ) in the current static flash image (image' 静态 ) according to the following equation: wherein is the luminance value of a pixel or sub-pixel having pixel or sub-pixel coordinates P 静态_经调整的 in the adjusted static flash image (image ' i,j ); is the luminance value of a pixel or sub-pixel having pixel or sub-pixel coordinates P 静态 in the current static flash image (image ' i,j ); I max is the maximum luminance value that a pixel or sub-pixel can be assigned; and α is a weighting value.

12. The eye tracking system (200) according to claim 8, wherein, The data set indicating at least one flash (105) detected in one or more previously captured images of the first eye (100) is a list that includes one or more pixel or sub - pixel coordinates (P i,j ) associated with each flash in each of the respective detected flashes (105) in one or more previously captured images of the first eye (100), where, for each flash in the respective detected flashes (105) in the current image (image t+1 ), the processing circuit (210) is configured to determine whether the position of the flash (105) in the current image (image t+1 ) corresponds to the position of any of the detected flashes (105) in the data set by the following steps: - Determine, based on the list, whether at least one pixel or sub - pixel coordinate (P t+1 ) among the one or more pixel or sub - pixel coordinates associated with the flash (105) in the current image (image i,j ) is also associated with the flash (105) in each of the one or more previously captured images of the first eye (100); and - If at least one of the one or more pixel or sub - pixel coordinates (P t+1 ) associated with the flash (105) in the current image (image i,j ) is also associated with the flash (105) in each of the one or more previously captured images of the first eye (100), then determine that the position of the flash (105) in the current image (image t+1 ) corresponds to the position of the detected flash (105) in the dataset; or - If none of the one or more pixel or sub-pixel coordinates (P t+1 ) associated with the flash (105) in the current image (image i,j ) are associated with the flash (105) in each of the one or more previously captured images of the first eye (100), then determine that the position of the flash (105) in the current image (image t+1 ) does not correspond to the position of any of the detected flashes (105) in the dataset.

13. The eye tracking system (200) according to claim 8, wherein, The processing circuit (210) is configured to determine whether the first eye (100) has moved between the first time (t + 1) and the second time (t) by: - Obtain a first image (Image 1) of the first eye (100) of the user (270) of the head-mounted device (260) using the processing circuit (210), where the first image (Image 1) is captured at the second time (t); - Obtain a second image (Image 2) of the first eye (100) using the processing circuit (210), where the second image (Image 2) is captured at the first time (t + 1); and - Determine whether the first eye (100) has moved between the first time (t+1) and the second time (t) based on the first image (Image 1) and the second image (Image 2).

14. The eye tracking system (200) according to claim 8, wherein, The processing circuit (210) is further configured to exclude any flash (105) classified as a static flash (107) from further processing in the eye tracking system (200).

15. A head-mounted device (260), the head-mounted device (260) comprising the eye tracking system (200) according to any one of claims 8 to 14.

16. The head-mounted device (260) according to claim 15, wherein, The head-mounted device (260) is configured to prevent light from any surrounding light source from entering.

17. A non-transitory computer-readable storage medium storing instructions that, when executed by a processing circuit (210) of a system (200), cause the system to: - obtain, using the processing circuit (210), the position of at least one flash (105) present in a current image (image t+1 ) of a first eye (100) of a user (270) of a head-mounted device (260) being illuminated by one or more illuminators (112, 113), wherein, The current image (image t+1 ) is captured at a first moment (t + 1); - Obtain a data set using the processing circuit (210), the data set indicating the position of at least one flash (105) detected in one or more previously captured images of the first eye (100) captured at corresponding times before the first time (t+1) when illuminated by the one or more illuminators (112, 113); - Perform eye tracking to determine whether the first eye (100) of the user (270) of the head-mounted device (260) has moved between the first time (t+1) and the second time (t), where the second time (t) is before the first time (t+1), and - If it is determined that the first eye (100) has moved between the first moment (t+1) and the second moment (t), update the data set based on the corresponding positions of any flashes (105) present in the current image (image t+1 ); Wherein, the non-transitory computer-readable storage medium further stores instructions that, when executed by a processing circuit (210) of the system (200), cause the system to perform the following operations for each of the at least one flash (105) obtained in the current image (image t+1 ) present therein: - Use the processing circuit (210) to determine whether the position of the flash (105) in the current image (image t+1 ) corresponds to the position of any of the flashes (105) in the one or more previously captured images of the first eye (100) captured at corresponding times before the first moment (t + 1) indicated by the data set; and - If the position of the flash in the current image (image t+1 ) corresponds to the position of any of the flashes (105) in the one or more previously captured images of the first eye (100) captured at a corresponding time before the first moment (t + 1) indicated by the data set, then the flash (105) is classified as a static flash (107) using the processing circuit (210).

18. The non-transitory computer-readable storage medium according to claim 17, further storing instructions that, when executed by a processing circuit (210) of the system (200), cause the system to perform the method steps according to any one of claims 3 to 7.

19. A method for classifying a flash (105) using an eye tracking system (200) in a head-mounted device (260), comprising any one of the technical features or any combination of the technical features in claims 1 - 7.

20. An eye tracking system comprising any one of the technical features or any combination of the technical features in claims 8 - 14.

21. A head-mounted device, comprising any one of the technical features in claims 15-16 or any combination of the technical features.

22. A non-transitory computer-readable storage medium, comprising any one of the technical features in claims 17-18 or any combination of the technical features.

Images