Event camera system for pupil detection and eye tracking

Abstract

To provide a system and method for eye tracking.SOLUTION: An eye tracking system includes a light source that emits a light pulse toward a user's eye over a predetermined period of time, a camera aligned along the optical path of a part of the light pulse that returns as reflected light from the surface of the eye, and a controller that controls the operation of the light source and the camera respectively. Each sensing pixel in a subset of the camera is configured to detect a current intensity value corresponding to a part of the reflected light and output a data value representing a difference between a previous intensity value and a current intensity value. The controller is configured to group the data values output from the sensing pixels of the subset into a two-dimensional matrix, identify a pupil of the eye on the basis of the grouped data values, and determine an orientation of the eye on the basis of the identified eye.SELECTED DRAWING: Figure 3A

Description

Technical Field

[0001] [Field of the Invention] The present disclosure generally relates to eye tracking, and more specifically, to methods of using an event camera system for pupil detection and eye tracking.

Background Art

[0002] [Cross - Reference to Related Applications] This application claims the benefit of U.S. Provisional Application No. 63 / 126,750, filed on December 17, 2020 and the entire content thereof is incorporated herein by reference.

[0003] [Background] A gaze tracking system captures images of the eyes to determine a user's 3D gaze, or a 2D projection of that gaze onto a surface or plane, such as a screen or a typical viewing distance. This can be done by segmenting the eye images through computer vision into various parts, namely the pupil, sclera, iris, eye lids, and canthus, and then exporting their features as parameters that can be used to calculate the user's gaze based on calibration data, or generate an eye model for the same purpose, or directly inputting the eye images into a neural network or other machine learning approach that segments and / or directly infers the user's gaze from the images based on a database of labeled eye images. Parameters extracted from traditional computer vision approaches can also be used in machine learning approaches, with or without eye images, although these can be tuned for accuracy at various low resolutions. In either case, the quality of the images with respect to contrast, lighting, sensitivity, etc., and the amount of computation required to extract eye features or directly infer gaze from the images are of utmost importance for robustness and the quality of the gaze estimate. This is particularly true for head-mounted mobile systems intended to operate under uncontrolled variable lighting conditions both indoors and outdoors.The complexity of extracting information, particularly the position of the important pupil, from an image of the eye requires a high level of complexity in the computer vision algorithms used for that task, and the robustness against the influence of the environment on these images becomes the main issue left for the eye-tracking system. SUMMARY OF THE INVENTION

[0004] Disclosed is an event camera system for pupil detection and gaze tracking. The event camera system can include a camera assembly and a controller, and can include one or more off-axis light sources. In some embodiments, the camera assembly can include a co-aligned light source camera assembly (a “co-aligned LSCA”). In some embodiments, the one or more off-axis light sources are not part of the event camera system; instead, one or more off-axis light sources generate some or all of the ambient light. The camera assembly includes one or more infrared (IR) light sources and an event camera. The one or more IR light sources are configured to emit pulses of IR light along an optical path toward an eyebox. The IR light is reflected from the eyeball in the eyebox, and the reflected light propagates along the optical path and returns toward the event camera for detection. Similarly, in embodiments including one or more off-axis light sources, at least a portion of the pulses of light emitted from the one or more off-axis light sources is reflected from the eyeball and surrounding facial regions and propagates along the optical path toward the event camera for detection. Light reflected by the retina via the pupil from one or more IR lights can increase the intensity on a pixel of the sensor of the event camera above a threshold level to detect an event. In contrast, light reflected by the eyeball or facial features surrounding the eyeball from one or more off-axis light sources can be adjusted to fall within the trigger threshold of the event camera so that the light does not generate an event. In some embodiments, multiple off-axis light sources are arranged in a ring shape centered on the axis of the event camera. The light reflected back to the sensor for each off-axis light source may correspond to a portion of the perimeter of the pupil.The event camera system can combine data regarding each off-axis camera to a bright ring corresponding to the outer periphery of the pupil. The controller is configured to determine the orientation of the eye using data output from the event camera.

[0005] In some embodiments, the gaze tracking system includes a first infrared (IR) light source, an event camera, and a controller. The first IR light source is configured to emit a first pulse of IR light over a first time period. The first pulse of IR light is directed along an optical path (e.g., via a coaxial LED or via a beam splitter) to an eye box that includes the user's eye. The eye reflects a portion of the first pulse of IR light and returns it along the optical path towards a target area at a first brightness. The eye reflects IR light generated from an off-axis IR light source (e.g., another IR light source in the gaze tracking system, IR light from the user's local area, etc.) and returns it along the optical path towards the target area at a second brightness. The light reflected from the off-axis IR light source outside the eye to the event camera may remain relatively constant during the first time period. The event camera is disposed in the target area. The event camera is configured to detect the IR light reflected from the eye box along the optical path. The event camera includes a plurality of photodiodes. Each photodiode is configured to detect an intensity value corresponding to a portion of the first pulse of reflected IR light and asynchronously output a data value that is at least partially based on the difference between the data value previously output by the photodiode and the intensity value detected by the photodiode with respect to an intensity threshold value. The controller is configured to identify the pupil of the eye from the data values output from the event camera resulting from the first pulse. The controller is configured to determine the gaze location of the user based at least in part on the identified pupil.

[0006] In some embodiments, the method may include receiving, by a sensor of an event camera, infrared light from a plurality of off-axis light sources reflected from a user's eyeball, and identifying a pupil of the eyeball from data values output from the event camera, the identifying including detecting a bright ring corresponding to an outer periphery of the pupil, and determining a line-of-sight position of the user based on the bright ring.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2A

Figure 2B

Figure 3A

Figure 3B

Figure 4A

Figure 4B

Figure 4C

Figure 4D

Figure 4E

Figure 5A

Figure 5B

Figure 6

Figure 7A

Figure 7B

[0008] The drawings are drawn for illustrative purposes only and depict various embodiments. Those skilled in the art will readily understand from the following description that alternative embodiments of the structures and methods illustrated herein may be used without departing from the principles described herein.

Best Mode for Carrying Out the Invention

[0009] [Detailed Description] This is a system for generating a high-contrast image of a pupil segmented from a background by using an event camera system to utilize the bright pupil reflex of the eye. In some embodiments, the event camera may include a co-aligned light source. When the light source hits the eye, the light entering the pupil reflects from the curved retina so that the light returns to the light source, and this light source may be coaxial with the camera sensor. This gives a characteristic bright pupil image where the pupil is seen as having a high intensity value. In contrast, the light from an off-axis light source reflects from the retina and does not reach the camera, instead generating a dark pupil image where the pupil is seen as having a low intensity value relative to the rest of the image of the eye. The light from the off-axis light source reflects from other parts of the eye and features of the surrounding face and reaches the sensor of the camera. The light reflected from other parts of the eye and the surrounding area may remain constant or increase by a minimal amount in response to the illumination of the pupil from the coaxial light source and may not result in events at these positions. The bright pupil image and the dark pupil image can be subtracted to generate a high-contrast image containing only the pupil. Using a conventional camera, this approach requires taking two separate images, but considering the high speed of eye movement, it is not actually possible to take images at a sufficient speed to generate a robust image of the pupil. By combining this technique with an event camera system, this system can capture and separate only the changes in the eye image at the moment when the illumination switches between various combinations of bright pupils (coaxial light sources) and dark pupils (off-axis light sources). This causes events that show only the high-contrast changes of the pupil itself, and it results in a significant simplification of the down-stream processing for the image output compared to the complex architectures that are usually required, especially for analyzing the feeds of the event camera system.

[0010] The disclosed system provides various advantages. In contrast to conventional video-based gaze tracking, the sensor can use significantly less power to detect the position of the pupil at equivalent frame rates. Further, the processor can utilize significantly less power to analyze a frame that includes only the pupil detected by the event camera, as opposed to analyzing the entire video frame. Additionally, the disclosed system can track the pupil even when it is partially occluded or when it is near other dark objects such as mascara, tattoos, or skin blemishes. In contrast, conventional systems can misidentify dark objects as the pupil. Further, conventional camera-based systems have difficulty tracking the pupil when the camera is out of focus, such as when the headset moves closer to or farther from the user's eye. In contrast, the present system can detect the position of the pupil using reflected light even when the camera assembly moves relative to the user's eye.

[0011] In some embodiments, the system can activate the coaxial light source while the off-axis light source is operating. The off-axis light source can create a high baseline of light reflected from surfaces other than the pupil. In response to the coaxial light source being activated, the event camera system can detect an event at a position within the pupil. The high baseline of light detected at positions other than the pupil can potentially reduce the number of triggering events at those positions.

[0012] In some embodiments, the system may include a plurality of off-axis light sources and may not include a coaxial light source. The off-axis light sources may be arranged in a ring shape centered on the axis of the event camera. An off-axis light source disposed close to the axis of the event camera may cause a partial reflection from the retina from the off-axis light source to the event camera. The sensor data captured by each off-axis light source may result in a crescent shape on a part of the edge of the pupil. Combining the data from all sensors may result in a bright ring on the outer periphery of the pupil. The outer periphery of the pupil may be used to determine the diameter of the pupil, the center of the pupil, and the position of the line of sight.

[0013] FIG. 1 is an event camera system 100 for differential pupil detection according to one or more embodiments. The event camera system 100 may be integrated within a headset (e.g., as described in detail below with respect to FIGS. 7A and 7B), a tablet, a computer, a car dashboard, a television, a mobile device, any other system using eye tracking, or some combination thereof. The event camera system 100 includes at least one camera assembly 105 (“camera assembly 105”) and a controller 110, and may optionally include one or more off-axis light sources (e.g., off-axis light source 115, etc.). In some embodiments, the camera assembly 105 includes a coaxial light source and may be referred to as a coaxial light source camera assembly (LSCA).

[0014] The light source of the camera assembly 105 emits and can receive light along the optical path 120. The camera assembly 105 may include one or more infrared (IR) light sources (sometimes referred to as coaxial IR light sources) and one or more event cameras. The one or more IR light sources may be, for example, light emitting diodes, vertical cavity surface-emitting lasers (VCSELs), any other IR light source or near-IR light source, or some combination thereof. The one or more IR light sources are generally configured to emit light in the IR band and / or the near-IR band. In some embodiments, there are multiple IR light sources, and at least two IR light sources emit in different optical bands. The one or more IR light sources are configured to emit pulses of IR light in accordance with instructions from the controller 110. The emitted light pulses are directed to propagate along the optical path 120 towards the eye box 125. The eye box 125 is a region within the space that will be occupied by the user's eyeball 130 and may also include surrounding facial features such as eyelashes and skin that surround the eyeball 130. In some embodiments, the camera assembly includes a plurality of IR light sources arranged in a ring shape centered on the optical path 120.

[0015] The camera assembly 105 can be co-aligned in the sense that one or more optical IR light sources are substantially aligned with the optical path(s) related to one or more event cameras. The angle between the optical path(s) of one or more event cameras and the optical path(s) of one or more IR light sources is kept small enough so that their optical paths essentially overlap and can form the optical path 120. The light emitted from one or more IR light sources travels towards the eyeball 130, reflects from the retina 140, and returns along the optical path 120 to one or more IR light sources and one or more event cameras. This optical path can be aligned to form the optical path 120 in various ways. For example, a beam splitter or other optical mixing device within the camera assembly 105 can be used to adjust the optical path, and one or more IR light sources and one or more event cameras can be mounted side-by-side within the camera assembly 105 with a sufficiently small center-center spacing. One or more IR light sources and one or more event cameras may share a common substrate (e.g., a substrate on which IR illumination pixels and sensing pixels are interleaved) within the camera assembly 105. One or more IR light sources may be coupled to optical elements such as lenses, or some combination thereof, along the optical path. Also, the camera assembly 105 may incorporate various optical filters, including a band-path filter specific to the wavelength of the light source, a spatially-varying bandpass filter in the interleaved illumination and event camera pixels, or some combination thereof.

[0016] The IR light is reflected from the retina 140, and the reflected light propagates along the optical path 120 and returns towards the event camera within the camera assembly 105 for detection. One or more event cameras (commonly also referred to as dynamic vision sensors) are configured to detect intensity values corresponding to the IR light reflected from the eyeball 130 along the optical path 120. The event camera among the one or more event cameras includes a plurality of photodiodes, and each photodiode is configured to asynchronously output a data value that is at least partially based on the difference between the data value previously output by the photodiode and the intensity value detected by the photodiode with respect to an intensity threshold.

[0017] One or more off-axis light sources radiate light in an off-axis direction from the optical path 120 in accordance with an instruction from the controller 110. The one or more off-axis light sources include the off-axis light source 115. The off-axis light source may be, for example, a light emitting diode, a vertical cavity surface emitting laser (VCSEL), some other IR light source or near-IR light source, or some combination thereof. The off-axis light source is configured to radiate light in the IR band and / or the near-IR band. In some embodiments, the one or more off-axis light sources radiate light at the same wavelength as one or more IR sources in the camera assembly 105. In some embodiments, there are a plurality of off-axis light sources, and at least two of the off-axis light sources radiate in different optical bands. The one or more off-axis light sources may be configured to radiate pulses of IR light in accordance with an instruction from the controller 110. The emitted light pulses are directed to propagate along an optical path separate from the optical path 120 towards the eye box 125. It should be noted that in some embodiments, there are no off-axis light sources that are part of the event camera system 100, and instead the off-axis light is ambient light, and the one or more off-axis light sources are sources that generate some or all of the ambient light.

[0018] Controller 110 controls the component group of the event camera system 100. The controller 110 can, for example, control the activation and intensity of one or more IR light sources, control the activation and intensity of the off-axis light source 115, and control the threshold settings and enablement of frame capture for one or more event cameras. One or more IR light sources and one or more event cameras may be part of a single coaxial LSAC 105 and / or part of multiple coaxial LSACs 105 (e.g., one coaxial LSAC 105 for each user's eyeball). The controller 110 can dynamically adjust the intensity settings of the light sources (i.e., one or more IR light sources and / or the off-axis light source 115) and the threshold settings of one or more event cameras according to data values from one or more event cameras, one or more separate external sensors (e.g., a photodiode that senses ambient light, or a conventional camera imaging device that also captures eye images), or some combination thereof. The controller 110 can enable one or more event cameras, synchronize the activation of each light source (i.e., one or more IR light sources and / or the off-axis light source 115) with the activation of one or more event cameras, and generate and output data values in time windows corresponding to specific configurations of the illumination settings.

[0019] In some embodiments, the controller 110 configures the light sources (i.e., one or more IR light sources and the off-axis light source 115) and one or more event cameras to activate only the off-axis light source 115 to capture data values. In these cases, the pupil 150 of the eyeball 130, and / or another eyeball of the user, appears relatively dark in the data values processed by the controller 110. In some embodiments, the controller 110 configures the light sources and one or more event cameras to activate only the coaxial light source 105 to capture data values. In these cases, the pupil 150 of the eyeball 130 and / or another eyeball of the user appears relatively bright in the data values processed by the controller 110. In some embodiments, the controller 110 may capture only one of these two configurations, or may capture multiple additional images using other light source configurations that include the off-axis light source 115, which is either always on or always off. In some embodiments, the controller 110 configures the light sources and one or more event cameras to capture data values while the off-axis light source 115 is active and the coaxial light source 105 is switched from an inactive state to an active state. In these cases, the pupil of the eyeball appears relatively bright in the data values processed by the controller 110.

[0020] The controller 110 reads data values from one or more event cameras. The controller 110 processes the data values to identify the pupil of the eyeball. Based on the identified pupil, the controller 110 may determine the orientation of the eyeball and / or the position of the line of sight in the eyeball.

[0021] In some embodiments, the event camera system 100 may be configured to detect the presence and position of an eyeball indoors. For example, the event camera system 100 may be installed at a fixed position indoors. In some embodiments, the event camera system 100 may include an integrated coaxial illuminator and / or an off-axis illuminator. In some embodiments, the event camera system 100 may include off-axis illuminators at various positions indoors. The event camera system 100 may be integrated, for example, within a television, a phone, a sign, or a computer. The controller 110 may be configured to identify the pupil of any eyeball within the field of view of the event camera system. The controller 110 may determine general line-of-sight directions at the pupil, such as the number of pupils detected indoors and whether the pupils are looking in the direction of the event camera system 100. Detecting bright pupils using the event camera system 100 may utilize much less processing power than processing a full image to detect the pupils. Room-scale tracking may be used for many applications, such as counting the number of people, tracking attention, privacy protection measures, or any other suitable scenario where detecting eyeballs may be beneficial.

[0022] Figure 2A shows the event camera system of FIG. 1 in a first time period according to one or more embodiments. In this first time period, the controller 110 activates the off-axis light source 115 to enable the event camera to output image data. The light from the off-axis light source 115 that reaches the pupil 150 and strikes the curved retina 140 at the back of the eyeball 130 will be reflected through the pupil 150 back to the light source, in this case the position of the off-axis light source 115. Some or all of this light does not reach the event camera of the camera assembly 105, but the remaining light that reaches the eye box 125 will be reflected in all directions, including reaching the camera assembly 105. In a conventional imager, this results in most of the features within the eye box 125 being more brightly illuminated, but since the curved retina 140 functions as a retroreflector, a dark pupil image is generated where the pupil 150 itself has a relatively low intensity value. In some embodiments, the output of the event camera is discarded during this time period, but in other embodiments, the off-axis light source 115 can be turned on for a sufficiently long time while the output of the event camera remains unchanged. It should be noted that the pixels of the event camera in response to a certain input either do not output a data value or output a no-change value. In some embodiments, the activation of the off-axis light source 115 during this time period may be completely ignored.

[0023] In some embodiments, while the off-axis light source 115 is activated, the controller 110 can switch the camera assembly 105 from a stopped state to an operating state. The pixels of the event camera can detect changes in the light reflected by the pupil 150 and returned to the camera assembly 105. Since the light from the off-axis light source 115 is reflected by areas other than the pupil, the pixels of the event camera may not detect events at positions other than the pupil. Therefore, the event camera may output a bright pupil image.

[0024] Figure 2B shows the event camera system of FIG. 1 in a second time period according to one or more embodiments. In this second time period, the off-axis light source 115 is stopped by the controller 110, and one or more IR light sources in the camera assembly 105 are activated by the controller 110, and the event camera is capable of outputting data values. Light from one or more IR light sources reaching the pupil 150 is reflected from the curved retina 140 at the rear of the eyeball 130, passes through the pupil 150 along the optical path 120, and returns to one or more event cameras in the camera assembly 105. The rest of the light reaching the eye box 125 is reflected in all directions, including along the optical path 120 of the camera assembly 105. Since most of the light entering the pupil 150 is retroreflected and returns to the camera assembly 105, the pupil 150 appears relatively bright compared to the rest of the eye image in a conventional imaging device, creating a "bright pupil image". However, in an event camera, the transition from the "dark pupil" from a previous (first) time period, characterized by the operation of the off-axis light source 115, to the "bright pupil" in this current (second) time period, characterized by the operation of the coaxial light source, generates a significant difference image described by the data values output by one or more event cameras. The difference image is localized in the region including the pupil 150.

[0025] In some embodiments, camera assembly 105 may detect glints from one or more light sources that reflect off the surface of eye 130. This light source may be an off-axis light source 115, a coaxial light source of camera assembly 105, or any other suitable light source. Controller 110 may analyze the glints in combination with the bright pupil image to determine the position of eye 130. Different light sources may be strobed at different frequencies. The controller may determine that the glint detected at the first position corresponds to the first light source based on the frequency of the detected glint. Similarly, the controller may determine that the glint detected at the second position corresponds to the second light source based on the frequency of the glint detected at the second position.

[0026] FIG. 3A is an example showing the optical path of coaxial LSCA 310 including an off-axis light source and beam splitter 320 according to one or more embodiments. Beam splitter 320 includes a first port 360, a second port 370, and a third port 380. In this embodiment, beam splitter 320 receives light from IR light source 330 at first port 360 and redirects at least a portion of the received light from beam splitter 320 along optical path 120 toward eye 130 in the forward direction at third port 380. A portion of the light returning from eye 130 is received by beam splitter 320 at third port 380, and the beam splitter directs a portion of the received light from second port 370 toward event camera 340 of coaxial LSCA 310. This configuration enables the optical path 120 of IR light source 330 at the first port 360 of beam splitter 320 to be exactly coaxial with the optical path 120 of event camera 340 at the second input port 370 of beam splitter 320. Thereby, it is ensured that the light from IR light source 330 is retroreflected from the retina of eye 130 and the retroreflected light reaches event camera 340. The light from off-axis IR light source 320 is reflected from eye 130 in all directions and can still reach event camera 340.

[0027] As shown, the coaxial LSCA 310 includes a filter 350 configured to transmit light in a narrow band that includes the wavelengths of the light emitted by the IR light source 330 and the light emitted by the off-axis IR light source 320, and to attenuate light of other wavelengths. In this way, the filter 350 attenuates ambient light received at the event camera 340. Although shown as separate from the event camera 340, in an alternative embodiment, the filter 350 may be integrated into the event camera 340.

[0028] In an alternative embodiment, the beam splitter 320 is omitted, and the alignment of the optical paths 120 of the IR light source 330 and the event camera 340 is approximately matched by arranging the IR light source 330 and the event camera 340 side by side with a sufficiently small center-to-center spacing such that the light from the IR light source 330 that passes through the pupil of the eyeball 130 and is retroreflected from the retina and returns to the coaxial LSCA 310 is also received at the event camera 340.

[0029] Similarly, as described above with respect to FIG. 1, in some embodiments, the beam splitter 320 is omitted, the IR light source 330 is replaced with a plurality of IR illumination pixels arranged alternately with sensing pixels, and the IR illumination pixels and the sensing pixels share a common substrate. The IR illumination pixels function similarly to the IR light source 330 in that they illuminate the eyeball 130 via the optical path 120. Also, the sensing pixels are pixels of the event camera. In this way, the functions of the event camera 340 and the IR light source 330 are combined and integrated into a single device that includes both sensing pixels and IR illumination pixels on a common substrate.

[0030] FIG. 3B is an example showing an optical path related to an off-axis light source and a coaxial LSCA including a miniaturized light source in the optical path of an event camera according to one or more embodiments. The IR light source 330 can be coupled to the optical element 390. The optical element 390 can include a lens, a window, a mirror, a grating, or any other suitable optical element. In some embodiments, the optical element 390 is a lens. The IR light source 330 may be along the central axis of the optical path 120. The IR light source 330 may be small relative to the lens 390, such as occupying less than 1% or less than 5% of the area of the lens 390. In some embodiments, the IR light source 330 may have a diameter of less than 3 mm, less than 1 mm, less than 100 microns, or less than 10 microns. Therefore, the IR light source 330 may minimally shield the sensor of the event camera 340.

[0031] Figure 4A is an example of a timing diagram 400 for differential pupil detection for a single frame by an event camera system (e.g., event camera system 100, etc.) according to one or more embodiments. Here, the scene starts only with ambient illumination from the environment. In this case, an off-axis light source can be considered as a light source that generates some or all of the ambient light without being controlled by the event camera system. Next, the controller (e.g., controller 110, etc.) enables the data values output from the event camera of the camera assembly (e.g., camera assembly 105, etc.), then enables (turns on) the coaxial IR light source of the camera assembly, and later disables the data values output from the event camera. The pulse enabled for the IR light source may have a pulse width of 1 second or less, and may be even shorter. For example, the overall "frame rate" or period of the entire system may be 30 Hz (a period of approximately 33 milliseconds). In this case, the pulse width is 10 ms, and the corresponding pulse rate is 50 Hz or 100 Hz at a duty cycle of less than 50%. In some embodiments, the duty cycle can be greater than 0% and less than 100%. Therefore, the event camera outputs data values while the coaxial IR light source transitions from off to on, which shows a large change in the intensity value at the pixels including the pupil of the eye due to the bright pupil effect. Next, the controller sets the threshold of the event camera so that small background changes in the output data values are not reported by the event camera. Further, the controller can group all events from the time period when the enabling of the event camera is active into a single "frame" similar to a conventional image frame. For example, the data values will include a timestamp of the magnitude of the change at a specific pixel at a specific time, and the controller will group the data values as a single 2D matrix covering all pixels that reported changes during this entire time period, similar to a conventional camera.In some embodiments, the activation of the event camera is configured to capture the falling edge of the illumination signal instead of the rising edge.

[0032] Note in FIG. 4A that the event camera is activated in the same time period that includes the activation of the IR light source. In other embodiments, the timing can be changed so that the event camera is activated in the time period spanning the deactivation of the IR light source (i.e., the transition from light emission to non-light emission).

[0033] FIG. 4B is an example of a timing diagram 440 for an event camera system (e.g., event camera system 100) for differential pupil detection including an off-axis light source (e.g., off-axis light source 115, etc.) for a single frame according to one or more embodiments. In this case, the controller (e.g., controller 110, etc.) first enables (turns on) the off-axis light source and then enables the data values output from the event camera of the camera assembly (e.g., camera assembly 105, etc.). While the event camera is enabled, the controller disables (turns off) the off-axis light source and enables (turns on) the IR light source of the camera assembly. This induces a data value output that shows a large transition in the intensity value from dark pupil illumination to bright pupil illumination in the pupil (e.g., pupil 150, etc.). The controller can then disable the data values output from the event camera and then disable (turn off) the IR light source of the camera assembly. The data values output from the event camera during this time period may be grouped into a single "frame" that represents a change in the image data during the synchronized switching of the light sources.

[0034] Figure 4C is an example of timing diagram 470 for differential pupil detection for multiple frames by an event camera system (e.g., event camera system 100, etc.) according to one or more embodiments. The controller (e.g., controller 110, etc.) may capture multiple frames by repeating a sequence of an IR light source activation signal and an event camera activation signal according to which embodiment is implemented. These frames may be required at a fixed frame rate by periodically strobing the illumination and enabling the event camera output, or the frames may be required at a variable timing dynamically selected by controller 110 by strobing the illumination at desired times. As shown, it should be noted that the event camera activation time period is, for example, longer than the event camera activation time period shown in Figure 4B. In other embodiments, the event camera activation time period may be the same as that shown in Figure 4B.

[0035] The controller may instruct the coaxial LSAC IR light source and, optionally, one or more off-axis IR light sources to emit multiple pulses each with their respective pulse lengths and duty cycles, enabling the event camera of the coaxial LSAC to periodically capture data values. The controller may use the data values output as a result of each of the respective pulses of the IR light source to generate an image frame and track the orientation of the eye based, in part, on the generated image frame.

[0036] FIG. 4D is an example of a timing diagram 480 for an event camera system (e.g., event camera system 100) for differential pupil detection that includes an off-axis light source (e.g., off-axis light source 115, etc.) for a single frame according to one or more embodiments. In this case, the controller (e.g., controller 110, etc.) first enables (turns on) the off-axis light source and then enables the data values output from the event camera of the camera assembly (e.g., camera assembly 105). While the event camera is enabled, the controller enables (turns on) the IR light source on the axis of the camera assembly. This induces a data value output that shows a large transition in the intensity value at the pupil (e.g., pupil 150, etc.) without inducing data values output at positions other than the pupil due to the reflected light from the off-axis light source. For example, as shown in FIG. 4E, when there is an off-axis light source, when the on-axis light source becomes effective at t = 1, the pixel illuminance P corresponding to the pupil position exceeds the event threshold (indicated by the dashed line) at point A. However, due to the bias from off-axis illumination, the amount of light reflected back from the on-axis light source is not sufficient to raise the pixel illuminance beyond the event threshold, so the pixel illuminance P corresponding to the skin position does not exceed the event threshold. In some embodiments, the controller may modulate the intensity of the camera assembly multiple times within a single frame to induce multiple events. For example, the intensity of the IR source of the camera assembly may be modulated in a square wave pattern, a triangle wave pattern, or some combination thereof. Regardless of the shape of the pulse, the amplitude of the pulse may be modulated one or more times within the frame such that the amplitude of the signal is sufficient to exceed the event threshold one or more times within the frame. In some embodiments, the intensity of the camera assembly may increase gradually in a ramp within a single frame. Then, the controller may disable the data values output from the event camera and then disable (turn off) the IR light source of the camera assembly.Next, the controller can disable (turn off) the off-axis light sources. The data values output from the event camera during this time period may be grouped into a single "frame" that represents the changes in the image data during the synchronized switching of the light sources.

[0037] FIG. 5A is an event camera system 500 for differential pupil detection including a plurality of off-axis light sources, according to one or more embodiments. The event camera system 500 may be one embodiment of the event camera system 100. The camera system 500 includes a plurality of off-axis light sources 520, a coaxial LSCA 510, and a controller (not shown). The coaxial LSAC 510 is substantially the same as the camera assembly 105. The off-axis light sources 520 are one embodiment of the off-axis light sources in FIG. 1, arranged at positions that increase the distance of the off-axis light sources from the coaxial LSCA 510. The off-axis light sources 520 may be arranged at a plurality of positions to adapt to the line-of-sight direction and the corresponding pupil and eye orientations.

[0038] In some embodiments, the controller may instruct the coaxial LSCA 510 and the off-axis light source 520 to sequentially emit light while collecting data values from the event camera of the coaxial LSCA 510. In this way, data values resulting from changes in the illuminance of the pupil of the eye caused by different off-axis light sources that increase the distance from the coaxial LSCA 510 are captured. It should be noted that the coaxial IR light source of the coaxial LSCA 510 still provides a bright pupil image of the eye 130 because the light is directly reflected from the eye 130 and returns to the event camera of the coaxial LSAC 510. Similarly, the light from the farthest off-axis light source 520c still provides a dark pupil image because the light source is sufficiently off-axis so that the light entering the pupil does not reach the coaxial LSCA 510. Off-axis light sources placed close to the coaxial LSCA 510 (e.g., off-axis light source 520a, etc.) can still be detected to produce an image that closely resembles a properly aligned bright pupil image because they are sufficiently aligned with the event sensor. However, off-axis light sources between these two ends (e.g., off-axis light source 520b, etc.) present a mixed bright pupil response, where only a portion of the pupil reflects light from the position of a given off-axis light source and returns it to the event camera.

[0039] The controller can measure the bright pupil response to a plurality of off-axis light sources 520 and analyze the case where the resulting image is a bright pupil image, a mixed image, or a dark pupil image. Based on known angular tolerances regarding the bright pupil effect and known relative locations of the off-axis light sources 520 and the coaxial LSCA 510, the controller 110 can estimate the distance from the coaxial LSCA 510 to the eye 130. The distance from the coaxial LSCA 510 to the eye 130, and thus the distance from all other fixed parts in the system to the eye 130, can be used to extract the relative position of the sensor on the event camera system and geometric information regarding the user's eye. This can be incorporated into an eye model to improve the accuracy of the measurement or used as calibration information. This distance, along with the intrinsics of the event camera, can be used to calculate quantitative features of the eye 130, such as size or the user's interpupillary distance (IPD), which in turn is an important metric for calculating the gaze distance from the vergence state of the eye. Further, in some embodiments, the plurality of off-axis light sources can be used to select an ideal individual light source or group of light sources for properly illuminating a given user's eye, regardless of changes in the shape of the face and variations in the relative fit of the gaze tracking system.

[0040] FIG. 5B is an event camera system 550 for differential pupil detection including a plurality of off-axis light sources according to one or more embodiments. The event camera system 550 can be an embodiment of the event camera system 100. The camera system 550 includes a plurality of off-axis light sources 560, a camera assembly 570, and a controller (not shown). The camera assembly 570 can be an embodiment of the camera assembly 105. However, in some embodiments, the camera assembly 570 does not include a coaxial light source. The off-axis light sources 560 can be arranged at a plurality of positions that are slightly off-axis from the camera assembly 570. Light from each off-axis light source 560 is partially reflected by the retina and reflected back to the camera assembly 570, which may result in a crescent pattern on the outer periphery of the pupil. In some embodiments, there can be 2, 3, 4, 6, 8, or any other suitable arbitrary number of off-axis light sources 560. The off-axis light sources can be arranged equidistantly in a ring shape centered on the axis of the camera assembly 570.

[0041] In some embodiments, the controller may instruct the off-axis light source 560 to sequentially emit light while collecting data values from the event camera of the camera assembly 570. In this way, data values resulting from changes in the illumination of the pupil of the eye caused by different off-axis light sources at various angular positions around the axis of the camera assembly 570 are captured. The off-axis light sources 560a, 560b, 560c, 560d present a mixed bright pupil response, where only a portion of the pupil reflects light back to the event camera from a given off-axis light source position. By combining the responses from each of the off-axis light sources, the controller may detect a bright ring at the edge of the pupil. In some embodiments, the controller may strobe the off-axis light sources 560 one at a time in a sequence and combine the responses of the sensors. In some embodiments, the controller 560 may activate all of the off-axis light sources 560 simultaneously. By detecting only the positions at the edge of the pupil, the controller may reduce the amount of processing power used to estimate the line of sight direction of the eye 130. The positions at the edge of the pupil may be sufficient to calculate the diameter, shape, and center position of the pupil. By detecting events at the edge of the pupil, the event camera system 550 may use less processing power by analyzing only the pixels corresponding to the edge of the pupil for the entire pupil.

[0042] FIG. 6 is a flowchart illustrating a process for determining the orientation of an eye using an event camera system 600 for differential pupil detection, according to one or more embodiments. Embodiments may include different steps and / or additional steps, and may perform the steps in a different order. The event camera system 600 may be an embodiment of the event camera system 100.

[0043] In step 610, the event camera system 600 receives IR light from at least one off-axis light source that is reflected from the user's eyeball in an event camera of a camera assembly (e.g., camera assembly 105). The eyeball is within the eyebox of the event camera system 600. In some embodiments, the off-axis light source is ambient light. In other embodiments, the off-axis light source is part of the event camera system 600, and the event camera system has pre-instructed the off-axis light source to emit IR light (e.g., as one or more IR light pulses) over a time period that at least partially overlaps with a second time period when the event camera is activated. In some embodiments, the off-axis light source includes a plurality of light sources arranged in a ring shape centered on the axis of the camera assembly.

[0044] In step 620, the event camera system 600 optionally emits a first pulse of IR light along the optical path from a coaxial IR source over a first time period. The first time period overlaps with the time period when the event camera is activated, and in embodiments where the off-axis light source is part of the event camera system, it should be noted that the first time period slightly overlaps with the time period when the off-axis light source is activated (e.g., as shown in FIGS. 4B and 4C above). The light is emitted from a coaxial IR light source within the camera assembly. The first pulse of IR light is directed along the optical path towards the eyebox. The retina of the eyeball reflects a portion of the first pulse of IR light and returns it along the optical path towards the target region at a first brightness. In this case, it should be noted that the IR light is essentially retroreflected from the retina of the eyeball, and the brightness of such light is brighter than the light reflected from the off-axis light source by the eyeball.

[0045] In step 630, the event camera system 600 optionally detects IR light reflected from the eye box along the optical path. The event camera system 600 uses an event camera to detect the IR light. The event camera includes a plurality of photodiodes, each photodiode detecting an intensity value corresponding to a portion of the reflected IR light and asynchronously outputting a data value based at least in part on the difference between the intensity value detected by the photodiode and the intensity value previously output by the photodiode compared to an intensity threshold. Since the event camera measures the difference in the detected light, the event camera outputs data values corresponding to the transition from the relatively low luminance in the IR light reflected by the eye from an off-axis light source to the relatively high luminance in the IR light retroreflected by the retina from an IR source within the camera assembly.

[0046] In step 640, the event camera system 600 identifies the pupil of the eye from the data values output from the event camera. As described above, the event camera outputs a differential image. In some embodiments, the light reflected from each off-axis light source and detected by the sensor creates a crescent shape corresponding to a portion of the outer periphery of the pupil. By combining the sensor data for each off-axis light source, whether sequentially or simultaneously, the event camera system detects a bright ring shape corresponding to the outer periphery of the pupil.

[0047] In some embodiments, the differential image is associated with changes in brightness caused by the retro-reflection of light from a coaxial IR light source. Thus, the data values output by the event sensor describe, among other things, the shape of the pupil. It should be noted that the shape of the pupil changes as a function of the orientation of the eye. In some embodiments, the controller may perform a series of low-level image operations including subtraction, dilation, spatial filtering, or ellipse fitting to determine the position of the pupil. In some embodiments, some or all of these low-level image operations are implemented in the controller 110 within the accelerated electronics, although some operations may be implemented in software. In some embodiments, the controller uses the data values to generate an image and uses shape recognition to identify the pupil within the generated image.

[0048] In step 650, the event camera system 600 determines the user's line-of-sight position based at least in part on the identified pupil. In some embodiments, the event camera system 600 uses an eye model that associates different shapes in the pupils of one or both eyes with different line-of-sight positions. The line-of-sight position in this context may be, for example, the position in space where the lines of sight of both eyes intersect (i.e., the vergence point).

[0049] In step 660, the event camera system 600 dynamically adjusts the lighting and / or camera settings to optimize the image quality and detection performance. The controller can, for example, increase or decrease the luminance of the light emitted by a light source (i.e., a coaxial IR light source and / or one or more off-axis light sources), adjust the time period during which the light source is active, adjust which of the one or more off-axis light sources are active, adjust the threshold for the pixels of the event camera to report changes, or perform some combination thereof. In this way, the controller can use the above to reduce and, in some cases, remove background noise from the image to optimize pupil position detection.

[0050] FIG. 7A is a perspective view of a headset 700 including an event camera system for differential pupil detection according to one or more embodiments. FIG. 7B is a cross-sectional view of the headset in FIG. 7A. The headset 700 can perform eye tracking for various purposes. In some embodiments, the eye tracking can be used to determine one or more health metrics in a user.

[0051] In some embodiments, the headset 700 includes a varifocal optical system. The varifocal optical system can dynamically adjust its focal length according to the estimated gaze location of the user in the headset. The varifocal optical system may include a varifocal lens assembly 720 for each eyeball. The varifocal lens assembly 720 dynamically adjusts its focal length based on the user's line-of-sight position. The varifocal lens assembly 720 includes one or more optical elements with variable focal length that operate alone or together such that the varifocal lens assembly has a range of focal lengths. The range of focal lengths enables the varifocal lens assembly 720 to provide a variable optical power. The range of optical power may include negative optical power, zero optical power, positive optical power, or some combination thereof. In some embodiments, the range of optical power is continuous (e.g., 0 to 3 Diopters, etc.). In some embodiments, the range of optical power is discrete (e.g., 0 to 3 Diopters in 0.1 Diopter increments, etc.). Also, in some cases, the discrete range of optical power may be set to correspond to a specific distance from the user (e.g., reading distance, computer distance, a distance of 20 feet or more, etc.). The optical elements with variable focal length may be, for example, an Alvarez lens, a liquid lens, a liquid crystal lens, any other lens with a dynamically adjustable focal length, or some combination thereof.In some embodiments, the varifocal lens assembly may include one or more optical elements of fixed focal length and / or prisms.

[0052] The event camera system is an embodiment of the event camera system 100. The event camera system includes an off-axis light source 760, a camera assembly 770, and a controller 730. The off-axis light source 760, the camera assembly 770, and the controller 730 are embodiments of the off-axis light source 115, the camera assembly 105, and the controller 110.

[0053] In some embodiments, the lens 720 may be configured to be autofocused based on the position of the eyeball 130. The user's eyeballs 130 are not only rotating so that each of the two eyeballs aligns the foveal axis with the object, but also viewing a 3D scene in which the object needs to be focused on by the biological adjustment of the lens in the eyeball. The rotation of the eyeballs 130 is known as vergence, which indicates the distance to the object. For example, as described above with respect to FIGS. 1, 2A, 2B, 4A to 4C, 5A, 5B, and 6, the off-axis light source 760 and the camera assembly 770 can be used to collect image data from the eyeball 130 including the pupil position, which is calculated by the controller 730 from the data values output by the event camera of the camera assembly 770. The pupil position of each eyeball can be used to calculate the user's line-of-sight position (i.e., vergence) and thus the distance the user is looking at. The controller 730 uses this estimated value to drive the focus state of the lens 720 to match the refractive power of the lens to the estimated line-of-sight distance.

[0054] In some embodiments, the event camera system 700 is configured to obtain a user's health metrics based on the eye position. For example, as described above with respect to FIGS. 1, 2A, 2B, 4A through 4C, 5A, 5B, and 6, the off-axis light source 760 and the camera assembly 770 can be used to collect image data from the eye 130 including the pupil position, which the controller 730 calculates from the data values output by the event camera of the camera assembly 770. The pupil position for each eye can be used to calculate the user's line of sight position. The controller 730 can use this estimate to track the user's line of sight position for any suitable purpose.

[0055] [Additional configuration information] The description of the above embodiments has been presented for purposes of illustration, i.e., it is not intended to be exhaustive or to limit the patent rights to the exact forms disclosed. Those skilled in the art can understand that many changes and modifications are possible in light of the above disclosure.

[0056] In some parts of this description, the embodiments are described from the perspective of algorithms for operations on information and symbolic representations. These descriptions and representations of algorithms are commonly used by those skilled in the data processing arts to effectively convey the essence of their work to other skilled artisans. These operations are described functionally, computationally, or logically, but are understood to be implemented by a computer program or equivalent electrical circuit, or microcode, etc. Further, it has also been proven convenient at times to refer to the arrangement of these operations as modules without loss of generality. The described operations and the modules associated therewith can be embodied in software, firmware, hardware, or any combination thereof.

[0057] Any of the steps, operations, or processes described in this specification may be performed or implemented by one or more hardware or software modules alone or in combination with other devices. In one embodiment, the software module is implemented using a computer program product that includes a computer-readable medium containing computer program code, where the computer program code may be executed by a computer processor to perform any or all of the described steps, operations, or processes.

[0058] Embodiments may also relate to an apparatus for performing the operations of this specification. The apparatus may be specially constructed for the required purposes and / or may include a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory tangible computer-readable storage medium or any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. Further, any computing system referred to in this specification may include a single processor or may be an architecture that employs multiple processor designs to enhance computing capability.

[0059] Embodiments can also relate to products produced by a computing process described herein. Such products can include information obtained from the computing process, which information is stored on a non-transitory tangible computer-readable storage medium and can include any embodiment related to a computer program product or other combination of data described herein.

[0060] Finally, the language used in the specification has been selected primarily for readability and instructional purposes and may not have been selected to circumscribe or delimit patent rights. Accordingly, the scope of the patent rights is intended to be limited not by this detailed description but rather by any claims issued on an application based on this specification. Therefore, the disclosure of embodiments is intended to be illustrative rather than limiting of the scope of the patent rights, which scope is defined in the following claims.

Claims

1. A gaze tracking system, a first light source that emits a first light pulse over a first time period, the first light pulse being directed along an optical path towards a user's eyeball, and a surface of the eyeball reflecting a part of the first light pulse returning as reflected light towards a target area along the optical path with a first luminance, the first light source; a camera disposed in the target area, the camera and the first light source being aligned along the optical path, the camera including a plurality of sensing pixels, and each sensing pixel of a subset of the plurality of sensing pixels detecting an intensity value corresponding to a part of the reflected light; outputting a data value representing a difference between a previously detected intensity value detected by each sensing pixel in the subset of sensing pixels and the detected intensity value in response to the detected intensity value exceeding a threshold level; a camera configured to perform the above; a controller, grouping the data values output by the subset of sensing pixels into a two-dimensional matrix; identifying the pupil of the eyeball based on the data values grouped in the two-dimensional matrix; determining the gaze position of the eyeball based at least in part on the identified pupil; a controller configured to perform the above; A gaze tracking system comprising the above.

2. A second light source disposed off-axis with respect to the optical path and configured to emit a second light pulse, further comprising, and the controller instructing the second light source to emit the second light pulse over a second time period that partially overlaps the first time period. The gaze tracking system according to claim 1, further configured to perform the above.

3. A part of the second light pulse is reflected from the surface of the eyeball along the optical path towards the target area with a second luminance that is less than the first luminance. The gaze tracking system according to claim 2.

4. The first light source emits the first light pulse off-axis with respect to the optical path, and the gaze tracking system is disposed within the optical path, redirecting at least a part of the first light pulse to propagate substantially along the optical path; transmitting back the part of the first light pulse reflected from the surface of the eyeball towards the camera along the optical path; A beam splitter configured to perform the above. The gaze tracking system according to claim 1, further comprising

5. The gaze tracking system according to claim 1, wherein the threshold level is set such that the data value at least partially identifies the pupil.

6. The gaze tracking system according to claim 1, wherein the controller is configured to instruct the first light source to emit the first light pulse.

7. The gaze tracking system according to claim 1, wherein the controller is configured to instruct the first light source to emit a plurality of light pulses having a predetermined pulse rate in a predetermined duty cycle, and the plurality of light pulses includes the first light pulse.

8. The gaze tracking system according to claim 1, wherein the controller is further configured to instruct the first light source to emit a plurality of light pulses at a plurality of time points dynamically selected by the controller.

9. The controller instructs the first light source to emit a plurality of light pulses including the first light pulse; emits each of the plurality of light pulses and uses data values generated by the camera as a result of reflection from the surface of the eyeball to generate each image frame of a plurality of image frames; further configured to track the orientation of the eyeball based at least in part on the plurality of generated image frames The gaze tracking system according to claim 1.

10. The camera is configured to detect the glow from the first light source reflected from the surface of the eyeball, The controller is further configured to determine the gaze position based on the detected glow. The gaze tracking system according to claim 1.

11. further comprising a second light source disposed off-axis with respect to the optical path and configured to emit a second light pulse, The camera is further configured to detect the glow from the first light source and the second light source reflected from the surface of the eyeball, The controller is further configured to determine the gaze position based on the detected glow. The gaze tracking system according to claim 1.

12. A method comprising A step of emitting a first light pulse over a first time period by a first light source, wherein the first light pulse is guided along an optical path towards a user's eyeball, and a surface of the eyeball reflects a part of the first light pulse returning as reflected light towards a target region along the optical path with a first luminance. A step of detecting an intensity value corresponding to a part of the reflected light by each of a subset of sensing pixels including a plurality of sensing pixels in a camera disposed in the target region, wherein the camera and the first light source are aligned along the optical path. A step of outputting, in response to the detected intensity value exceeding a threshold level, a data value representing a difference between a previous intensity value detected by each of the sensing pixels in the subset of sensing pixels and the detected intensity value, by each of the sensing pixels in the subset of sensing pixels. A step of grouping the data values output by the subset of sensing pixels into a two-dimensional matrix. A step of identifying a pupil of the eyeball based on the data values grouped into the two-dimensional matrix. A step of determining a gaze position of the eyeball based at least in part on the identified pupil A method comprising.

13. A step of instructing a second light source disposed off-axis with respect to the optical path to emit a second light pulse over a second time period that partially overlaps with the first time period. A step of reflecting a part of the second light pulse from the surface of the eyeball towards the target region along the optical path with a second luminance that is less than the first luminance. The method according to claim 12, further comprising.

14. A step of emitting the first off-axis light pulse with respect to the optical path by the first light source. A step of redirecting at least a part of the first light pulse by a beam splitter disposed in the optical path so as to substantially propagate along the optical path. A step of transmitting back, by the beam splitter, a part of the first light pulse reflected from the surface of the eyeball towards the camera along the optical path. The method according to claim 12, further comprising.

15. A step of instructing the first light source to emit the first light pulse having a pulse width of 1 second or less. The method according to claim 12, further comprising.

16. Instructing the first light source to emit a plurality of optical pulses having a predetermined pulse rate in a predetermined load cycle, wherein the plurality of optical pulses includes the first optical pulse, step The method according to claim 12, further comprising.

17. Instructing the first light source to emit a plurality of optical pulses at a plurality of dynamically selected times The method according to claim 12, further comprising.

18. Instructing the first light source to emit a plurality of optical pulses including the first optical pulse; Emitting each of the plurality of optical pulses and using data values generated by the camera as a result of reflection from the surface of the eyeball to generate each image frame of a plurality of image frames; Tracking the orientation of the eyeball based at least in part on the plurality of generated image frames The method according to claim 12, further comprising.

19. Detecting, by the camera, the glow from the first light source reflected from the surface of the eyeball; Determining the line-of-sight position further based on the detected glow The method according to claim 12, further comprising.

20. Instructing a second light source disposed off-axis with respect to the optical path to emit a second optical pulse; Detecting, by the camera, the glow from the first light source and the second light source reflected from the surface of the eyeball; Determining the line-of-sight position further based on the detected glow The method according to claim 12, further comprising.

Images