Pupil detection using event-based sensor
The two-phase illumination method for eye tracking enhances contrast and reduces power consumption, addressing high processing demands in event-based sensors for dark pupil detection, achieving efficient and faster pupil detection.
Patent Information
- Application Number
- JP2024074535
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-09
- Filing Date
- 2024-05-01
- Publication Date
- 2024-11-21
AI Technical Summary
Existing eye tracking technologies using event-based sensors for dark pupil detection face high power consumption and processing demands due to significant pixel events triggered by brightness changes in the eye area, limiting detection speed and efficiency.
A method involving two-phase illumination with a first light source offset from the optical axis and a background signal to enhance contrast differences between the pupil and the rest of the eye, using event-based sensors to reduce power consumption and processing load.
The method reduces power consumption and processing requirements by optimizing pixel event triggers, enabling faster and more efficient pupil detection with improved contrast differentiation.
Smart Images

Figure 2024163050000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates generally to eye tracking technology using pupil detection methods. [Background technology]
[0002] Eye tracking can be implemented in a variety of devices, for example, virtual or augmented reality glasses or goggles, camera viewfinders, etc. Eye tracking is often based on pupil detection using a visual sensor and specific lighting conditions, usually with an infrared source, that do not interfere with normal vision.
[0003] "Bright pupil" detection techniques generally involve illuminating the retina through the pupil with a light source placed near or far enough in front of the eye that it appears to be on the optical axis of the camera. Under such conditions, the pupil appears very bright relative to its surroundings and is relatively easy to detect by contrast differences. Such techniques are not suitable for devices placed close to the eye (goggles, glasses, viewfinders) because infrared sources cannot be placed close enough to the optical axis in terms of distance to the pupil.
[0004] "Dark pupil" detection technology is more suited to devices closer to the eye: it involves illuminating the eye from the side, so that the pupil absorbs almost all light and appears much darker than its surroundings.
[0005] Detection is typically performed with frame-based cameras, where image analysis is performed on frames generated by the camera. To conserve power, the eye is intermittently illuminated or flashed at the rate of the frames. However, this technique is still power-intensive, since it involves performing analysis on a significant amount of frame data. Furthermore, frame-based sensors cannot detect motion faster than the frame rate, and increasing the frame rate to capture faster motion increases the computational power required.
[0006] The use of event-based sensors in pupil detection applications overcomes the lack of speed of frame-based sensors, but as will be shown below, the amount of data to process remains large.
[0007] 1A and 1B illustrate two phases of a dark pupil detection method that conventionally uses an event-based sensor and intermittent illumination of the eye. Since there is no frame rate limitation, the illumination flash frequency can approach the reaction speed limit of the event-based sensor.
[0008] Fig. 1A illustrates the appearance of the eye as perceived by the sensor in two illumination phases, and Fig. 1B illustrates the corresponding luminance variations of the pupil and the rest of the eye. On the left, there is no illumination and the eye and pupil are uniformly dark (black). On the right, the illumination is turned on. The pupil remains dark or barely bright, while the rest of the eye becomes significantly brighter (white).
[0009] Thus, it is clear that significant contrast changes are visible in many pixels around the pupil, and in any case more pixels within the pupil area. As a result, event-based sensors trigger a significant number of events that require a corresponding amount of processing power. In essence, the use of event-based sensors for dark pupil detection improves the speed of detection, but the amount of processing power remains significant. Summary of the Invention
[0010] A general method is provided for detecting an eye pupil using an event-based visual sensor, comprising the steps of directing a first light source toward the eye, observing the eye with the event-based sensor, and intermittently subjecting the sensor to a uniform background signal.
[0011] The method may further include setting a background signal level such that the relative luminance change sensed for the pupil is greater than the relative luminance change sensed for the remainder of the eye, and setting a detection threshold for the pixels of the sensor such that the relative luminance change of the pupil triggers an event and the relative luminance change of the remainder of the eye does not trigger an event.
[0012] The method may further comprise the step of subjecting the sensor to a background signal during a phase in which the eye is illuminated with the first light source.
[0013] Alternatively, the method may include illuminating the eye in opposite phase to the background signal.
[0014] The background signal may be gradually increased until an event is triggered by the sensor.
[0015] A background signal can be obtained by directing a second light source towards the sensor.
[0016] Alternatively, the background signal may be an electrical signal applied within the pixel circuits of the sensor that has the effect of uniform illumination of the sensor.
[0017] The electrical signal may be a constant current that is added to the photocurrent generated in the pixel.
[0018] A lens may be attached to the sensor and the second light source is positioned such that light partially reflected by the lens passes through the pupil of the eye.
[0019] The first light source may be a display device viewed by the eye.
[0020] A general method for detecting the pupil of an eye using a lens-mounted visual sensor is also provided, comprising the steps of observing the eye with the sensor, i) the sensor is subjected to a uniform background signal, and ii) directing a light source towards the lens of the sensor such that light partially reflected by the lens passes through the pupil of the eye, and performing light pupil detection on the image perceived by the sensor. [Brief description of the drawings]
[0021] Embodiments are set forth in the following description, given by way of example only, in connection with the accompanying drawings, in which:
[0022] [Figure 1] 1A and 1B illustrate two phases of a dark pupil detection method that conventionally uses an event-based sensor and intermittent illumination of the eye.
[0023] [Diagram 2] 1A, B and C illustrate two phases of a dark pupil detection method using an event-based sensor according to a first embodiment.
[0024] [Diagram 3] 2A and 2B illustrate absolute and relative log contrast variation graphs used in the embodiment of FIGS. 2A-2C.
[0025] [Figure 4] 13A and 13B illustrate two phases of a dark pupil detection method using an event-based sensor according to a second embodiment.
[0026] [Diagram 5] 1 illustrates a partial pixel circuit modified in accordance with one embodiment to add a uniform background signal. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0027] As mentioned above, when performing eye tracking with an event-based sensor using the dark pupil method and intermittent illumination of the eye, the number of events triggered in each illumination flash, i.e. the amount of data to process, is significant because a significant portion of the eye sees luminance changes that trigger events, whereas the region of interest, the pupil, sees very little change and does not trigger an event.
[0028] It is proposed herein to reverse the area that sees significant luminance changes, i.e. to illuminate the eye under certain conditions such that the pupil sees the most significant changes while the rest of the eye sees insufficient changes to trigger an event.
[0029] 2A-2C respectively illustrate a first embodiment of a two-phase illumination configuration, the corresponding eye appearance variation as perceived by a sensor, and the corresponding luminance variation in the eye and pupil area.
[0030] In the first phase, on the left, the eye 10 is illuminated by a light source 12, preferably an infrared light source, in a dark pupil configuration, for example placed between the eye and a sensor 14 and offset with respect to the optical axis between the eye and the sensor. The sensor includes an array of event-based pixels and is typically fitted with a lens (not shown). Both the sensor and the light source 12 may be located on the rim of glasses or goggles worn by the user. In other configurations, the light source 12 may be located in a plane behind the sensor, or may be a display device typically included in the goggles.
[0031] The illuminance of the light source 12 does not need to be as bright as would be required in a conventional situation. As shown on the left side of Fig. 2B, the pupil remains dark and the surrounding area is gray instead of white. Exemplary luminance levels are shown on the left side of Fig. 2C on a linear amplitude scale, with the pupil luminance slightly above 0, while the rest of the eye area has a given level of luminance, which is shown as being lower than that of the conventional illumination phase of Fig. 1B, as an example, to save power.
[0032] In the second illumination phase on the right, the sensor 14 is subjected to a uniform background signal. This background signal can be generated by a second illumination source 16 directed towards the sensor, as shown by way of example. The light source 16 can be located outside the sensor housing incorporating the lens and directed towards the lens. The light source 16 can also be located inside the sensor housing and therefore be in the visible spectrum, as it does not interfere with the user's vision. Alternatively, the background signal can be an electrical signal applied in the pixel circuit that produces the same effect as a uniform illumination of the sensor, for example a constant bias current applied to the photocurrent of each pixel in the sensor array, as described below in connection with FIG. 5.
[0033] This background signal causes all areas viewed by the sensor to appear to have their brightness increased in the same steps, as illustrated on the right side of Figure 2C. For illustrative purposes, the pupil changes from black to gray while the rest of the eye changes from light gray to white, as shown on the right side of Figure 2B.
[0034] This mechanism finds its relevance in the way event-based pixels are typically designed, i.e., luminance is measured on a logarithmic scale and changes on such a logarithmic scale correspond to a rate of change that is much larger for the pupil than for the rest of the eye using the illumination method shown.
[0035] Figure 3A shows the luminance evolution of Figure 2C on a logarithmic amplitude scale, where the step seen by the pupil is significantly larger than the step seen by the rest of the eye, since the pupil luminance increases from nearly zero, whereas the luminance of the rest of the eye increases between two values relatively far from zero.
[0036] FIG. 3B illustrates the comparison performed by the event-based pixels corresponding to the pupil region and the rest of the eye region. Each pixel is typically designed to compare the logarithmic amplitude change, i.e., the difference between the current logarithmic amplitude and the previous logarithmic amplitude, which reflects the rate of change, with a threshold. The threshold Th can be set as shown such that the change in the pupil region exceeds the threshold while the change in the rest of the eye region remains below the threshold. This configuration provides a certain tolerance for the selection of the intensities of the light sources 12 and 16 to reduce power consumption.
[0037] In one example, the light source 12 that illuminates the eye may be on continuously, but at an intensity significantly lower than that required in the conventional setup of Figures 1A, 1 B. The intensity of the intermittent light source 16 may also be set to a relatively low level.
[0038] Alternatively, the two illustrated illumination phases may be interleaved with a standby phase in which both light sources are off and the sensors are disabled.
[0039] To optimize power consumption, the intensity of the background illumination source 16 may be increased gradually, instead of in a single large step, until an event corresponding to the pupil region is triggered.
[0040] Figures 4A and 4B illustrate two phases of a dark pupil detection method using an event-based sensor according to a second embodiment. In Figure 4A, a layout similar to that of Figure 2A is used for the sensor and illumination source. However, instead of continuously illuminating the eye with light source 12 and flashing light source 16 and intermittently adding a background illumination signal, light sources 12 and 16 are flashed in opposite phases. On the left side, light source 12 illuminates the eye while light source 16 is off, creating an image similar to that of the left side of Figure 2B, revealing a dark pupil and lighter surrounding areas shown in grey. On the right side, light source 16 illuminates the sensor while light source 12 is off, creating an image with a uniform brightness level shown as grey.
[0041] Theoretically, the luminance level of the remainder of the eye area in both phases can be made substantially equal by adjusting the intensity of the light sources 12, 16, so that there is no measurable difference for this area, and therefore the discrimination tolerance for detecting the pupil is increased. However, this can be difficult in practice, since the two phases correspond to different lighting conditions (eye on one side and sensor on the other). In both cases, the rate of change of the pupil area remains substantially greater than the rate of change of the remainder of the eye area, as in the embodiment of Figures 2A-2C.
[0042] FIG. 5 illustrates a pixel circuit of an array modified according to one embodiment to add a uniform background signal in the sensor array. Three pixels Px1-Px3 are partially illustrated, with pixel Px3 being shown in more detail. Each pixel of the event-based sensor generally includes a photodiode PD with its anode grounded. Photocurrent generated by the diode flows from a high power supply line through a transimpedance amplifier TIA, typically with a logarithmic response, to the cathode of the photodiode. The output of the TIA is processed by a circuit (not shown) to generate events in any known manner.
[0043] A respective N-MOS transistor M1-M3 is connected between the cathode of each photodiode and ground. The transistors M1-M3 form the output of a current mirror sharing a common diode-connected input N-MOS transistor M0. A current source Ibgr is connected to inject a background reference current Ibgr into the diode-connected transistor M0, which is copied into each of the transistors M1-M3 and added to the photocurrent measured by the TIA. The current Ibgr is adjusted to produce the desired background signal strength.
[0044] N-MOS transistor Mb connects the gates of transistors M0-M3 to ground and is controlled by an active-low background enable signal / BGR, which can thus switch the sensor between a normal mode and a simulated background illumination mode.
[0045] As mentioned above, one embodiment of background signal source 16 is an illumination source directed towards the lens of the sensor, according to the configuration shown on the right side of Figure 2A. The lens partially reflects the light of source 16 along a cone. If source 16 is close enough to the optical axis, or if the lens has sufficient curvature, the reflected cone can intersect the pupil, causing light to pass through the pupil along the optical axis and placing the pupil in a bright pupil detection condition.
[0046] Such a configuration further increases the luminance variation of the pupil. In other words, the linear amplitude step of the pupil in Fig. 2C is larger than that of the rest of the eye. As an example, on the right side of Fig. 2B and Fig. 4B, the light source 16 is turned on and the pupil becomes a lighter shade of grey.
[0047] This particular configuration can be simplified by operating the bright pupil detection technique using only the light source 16. The light source 16 is then permanently turned on at a relatively low intensity to generate a uniform grey background on the sensor. The light source 12 is either not present or permanently off. The light partially reflected by the lens illuminates the eye and pupil, which creates an image with a bright pupil superimposed on the background grey level of the sensor. The pupil area may not be as bright as a properly exposed eye situation, but is still noticeably brighter than the rest of the perceived image. Thus, the pupil is relatively easy to track with any type of sensor (event-based or frame-based) and offers the advantages of bright pupil sensing techniques, i.e. avoiding the false positives that a dark iris can cause with a dark pupil sensing technique.
Claims
1. 1. A method for detecting an eye pupil using an event-based vision sensor, comprising: directing a first light source (12) towards the eye (10); observing the eye with the event-based sensor (14); and intermittently subjecting said sensor to a uniform background signal (16) so that all areas viewed by said sensor appear to have their brightness increased in the same step.
2. setting the level of the background signal (16) such that the relative luminance change sensed for the pupil is greater than the relative luminance change sensed for the remainder of the eye; and setting a detection threshold for pixels of the sensor such that the relative luminance change of the pupil triggers an event and the relative luminance change of the remainder of the eye does not trigger an event.
3. 3. The method of claim 2, comprising subjecting the sensor to the background signal during a phase in which the eye is illuminated with the first light source.
4. The method of claim 2 , comprising illuminating the eye in phase opposition to the background signal.
5. The method of claim 1 , wherein the background signal increases gradually until an event is triggered by the sensor.
6. The method of claim 1 , wherein the background signal is obtained by directing a second light source (16) towards the sensor (14).
7. 2. The method of claim 1, wherein the background signal is an electrical signal applied within a pixel circuit of the sensor that has the effect of uniform illumination of the sensor.
8. The method of claim 7 , wherein the electrical signal is a constant current that is added to a photocurrent generated in the pixel.
9. 7. The method of claim 6, wherein the sensor is fitted with a lens and the second light source (16) is positioned such that light partially reflected by the lens passes through the pupil of the eye.
10. The method of claim 1 , wherein the first light source is a display device viewed by the eye.
11. 1. A method for detecting a pupil of an eye using a lens-mounted visual sensor, comprising: observing the eye with the sensor (14); - directing a light source (16) towards the lens of the sensor such that i) the sensor is subjected to a uniform background signal and ii) light partially reflected by the lens passes through the pupil of the eye; and performing bright pupil detection on the image perceived by the sensor.
