Pupil detection using event-based sensors
By using event-based sensors in eye tracking technology, setting background signal and pixel detection thresholds, combining uniform background signal and inverse lighting, the problems of large power consumption and insufficient processing capabilities in traditional technology are solved, and efficient pupil detection and Identification.
Patent Information
- Application Number
- CN202410540161.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-09
- Filing Date
- 2024-04-30
- Publication Date
- 2024-11-12
AI Technical Summary
In the existing eye tracking technology, frame-based sensors consume a large power consumption and cannot detect faster motion than frame rate. Although event-based sensors improve speed, their processing capabilities are still significant, especially in dark pupil detection, which requires processing. The problem of large amount of data.
Using an event-based vision sensor, by setting background signals and pixel detection thresholds, the relative brightness changes of the pupil trigger the event, while the relative brightness changes of the rest of the eye do not trigger the event, using uniform background signals and inverse lighting to reduce power consumption. And partially reflect light through the lens to penetrate the pupil to increase the brightness change ratio.
It effectively reduces power consumption, improves detection speed and identification tolerance, reduces the amount of data processed, makes pupil detection more efficient, and is suitable for various eye tracking devices.
Smart Images

Figure CN118942147A_ABST
Abstract
Description
Technical Field
[0001] The present invention generally relates to eye tracking techniques using pupil detection methods. Background Art
[0002] Eye tracking can be implemented in various devices such as virtual or augmented reality glasses or goggles, camera viewfinders, etc. Eye tracking is typically based on pupil detection with a vision sensor and specific lighting conditions (usually with an infrared source that does not interfere with normal vision).
[0003] "Bright pupil" detection techniques typically involve illuminating the retina through the pupil using a source placed near the optical axis of the camera or far enough in front of the eye that it appears to be on the optical axis. Under these conditions, the pupil appears very bright relative to its surroundings and is relatively easy to detect by contrast differences. This technique is not suitable for devices placed close to the eye (goggles, glasses, viewfinders) because the infrared source cannot be placed close enough to the optical axis relative to its distance to the pupil.
[0004] The "dark pupil" detection technique is more suitable for devices close to the eye. It involves lighting the eye from the side, whereby the pupil absorbs almost all the light and appears much darker than its surroundings.
[0005] Detection is typically performed using a frame-based camera and image analysis performed on the frames produced by the camera. To save power, the eye is illuminated or flashed intermittently at the rate of the frame. However, this technique is still power-intensive because it involves performing analysis on a large number of frames of data. In addition, frame-based sensors cannot detect motion faster than the frame rate, and increasing the frame rate to capture faster motion increases the required computing power.
[0006] The use of event-based sensors in pupil detection applications overcomes the lack of speed of frame-based sensors, but the amount of data to be processed is still large, as shown below.
[0007] Figure 1A and Figure 1B The two stages of a traditional dark pupil detection method using an event-based sensor and intermittent illumination of the eye are illustrated. Since there is no frame rate limit, the illumination flash frequency can approach the reaction speed limit of the event-based sensor.
[0008] Figure 1A illustrates the appearance of an eye as perceived by the sensor in two illumination stages, and Figure 1B The corresponding brightness changes of the pupil and the rest of the eye are illustrated. On the left, there is no illumination, and the eye and pupil are uniformly dark (black). On the right, the illumination is turned on. The pupil remains dark or barely bright, while the rest of the eye becomes significantly brighter (white).
[0009] It is therefore apparent that a large number of pixels around the pupil (in any case more pixels than in the pupil region) see significant contrast changes. Consequently, an event-based sensor will trigger a large number of events requiring a corresponding amount of processing power. In essence, although the use of an event-based sensor for dark pupil detection increases the speed of detection, the amount of processing power is still significant. Summary of the invention
[0010] A general method for detecting the pupil of an eye using an event-based vision sensor is provided, the method comprising the steps of directing a first light source toward the eye; observing the eye using the event-based sensor; and intermittently subjecting the sensor to a uniform background signal.
[0011] The method may also include the following steps: setting the level of the background signal so that the relative brightness change sensed for the pupil is greater than the relative brightness change sensed for the rest of the eye; and setting the detection threshold of the sensor's pixels so that the relative brightness change of the pupil triggers an event and the relative brightness change of the rest of the eye does not trigger an event.
[0012] The method may further comprise the step of subjecting the sensor to a background signal during the phase of illuminating the eye with the first light source.
[0013] The method may alternatively include the step of illuminating the eye in phase opposite to the background signal.
[0014] The background signal can gradually increase until the sensor triggers an event.
[0015] A background signal may be acquired by directing a second light source toward the sensor.
[0016] Alternatively, the background signal may be an electrical signal added within the pixel circuitry of the sensor which has the effect of a uniform illumination of the sensor.
[0017] The electrical signal may be a constant current added to the photocurrent generated in the pixel.
[0018] The sensor may be fitted with a lens, and the second light source arranged such that light partially reflected by the lens penetrates the pupil of the eye.
[0019] The first light source may be a display device viewed by the eye.
[0020] A general method for detecting the pupil of an eye using a visual sensor equipped with a lens is also provided, the method comprising the steps of observing the eye using the sensor; directing a light source toward the lens of the sensor so that i) the sensor is subjected to a uniform background signal and ii) light partially reflected by the lens penetrates the pupil of the eye; and performing bright pupil detection on the image perceived by the sensor. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Embodiments will be presented in the following description provided for exemplary purposes only, in conjunction with the accompanying drawings, in which:
[0022] Figure 1A and Figure 1B Two stages of a traditional dark pupil detection approach using an event-based sensor and intermittent illumination of the eye are illustrated;
[0023] Figure 2 A to Figure 2C illustrates two stages of a dark pupil detection method using an event-based sensor according to a first embodiment;
[0024] Figure 3A and Figure 3B Example Figure 2 A to Figure 2C Absolute and relative logarithmic contrast ratio change curves used in embodiments of the present invention;
[0025] Figure 4A and Figure 4B illustrates two stages of a dark pupil detection method using an event-based sensor according to a second embodiment; and
[0026] Figure 5 A portion of a pixel circuit modified according to an embodiment to add a uniform background signal is illustrated. DETAILED DESCRIPTION
[0027] As mentioned previously, when performing eye tracking with an event-based sensor using a dark pupil approach and intermittent illumination of the eye, the number of events triggered at each illumination flash (i.e., the amount of data to be processed) is significant. This is because the majority of the eye sees a change in brightness that triggers an event, while the region of interest, the pupil, sees hardly any change and does not trigger an event.
[0028] This paper proposes illuminating the eye under specific conditions that invert the areas that see significant brightness changes (ie, so that the pupil sees the most significant changes, while the rest of the eye sees changes that are not sufficient to trigger an event).
[0029] Figure 2 A to Figure 2C A first implementation of a two-stage lighting configuration, the corresponding changes in eye appearance as sensed by the sensor, and the corresponding changes in brightness of the eye and pupil regions are illustrated, respectively.
[0030] In the first stage on the left, in a dark pupil configuration, the eye 10 is illuminated by a source 12 (preferably an infrared source) placed, for example, between the eye and a sensor 14 and offset relative to the optical axis between the eye and the sensor. The sensor comprises an array of event-based pixels and is typically fitted with a lens (not shown). Both the sensor and the source 12 may be positioned on the edge of glasses or goggles worn by the user. In other configurations, the source 12 may be positioned in a plane behind the sensor and may even be a display device typically contained in goggles.
[0031] The illumination intensity of source 12 does not need to be as bright as required in conventional situations. Figure 2 As shown on the left side of B, the pupil is still dark and the surrounding area is gray instead of white. Figure 2C The left side of FIG. 1 shows on a linear magnitude scale that the pupil brightness is just above zero, while the rest of the eye region has a given level of brightness, illustrated as an example below Figure 1B The brightness of the regular lighting stage is reduced to save power.
[0032] In the second illumination phase on the right, the sensor 14 is subjected to a uniform background signal. As shown in the example, this background signal can be generated by a second illumination source 16 directed toward the sensor. The source 16 can be placed outside the sensor housing containing the lens and directed toward the lens. The source 16 can also be placed inside the sensor housing and thus can be in the visible spectrum since it will not interfere with the user's vision. Alternatively, the background signal can be an electrical signal added within the pixel circuit that will produce the same effect as uniform illumination of the sensor, for example, a constant bias current added to the photocurrent of each pixel in the sensor array, as described below with respect to Figure 5 described.
[0033] like Figure 2C As illustrated on the right side of , this background signal causes all areas viewed by the sensor to see their brightness increase in the same step. Figure 2 As shown on the right side of B, for illustration purposes, the pupil changes from black to gray, while the rest of the eye changes from light gray to white.
[0034] The mechanism finds its correlation in the way event-based pixels are usually designed, i.e., brightness is measured on a logarithmic scale, and changes on such a logarithmic scale correspond to ratios of change. With the illumination method shown, the ratio of brightness change of the pupil happens to be much larger than that of the rest of the eye.
[0035] Figure 3A Shown on a logarithmic amplitude scale Figure 2CHere, the step seen by the pupil is significantly larger than the step seen by the rest of the eye, because the brightness of the pupil increases from close to zero, while the brightness of the rest of the eye increases between two values relatively far from zero.
[0036] Figure 3B The comparisons made by event-based pixels corresponding to the pupil region and the rest of the eye region are illustrated. Each pixel is typically designed to compare the logarithmic amplitude change to a threshold, i.e., the difference between the current logarithmic amplitude and the previous logarithmic amplitude, which reflects the rate of change. As shown, the threshold Th can be set so that the change in the pupil region exceeds the threshold, while the change in the rest of the eye region remains below the threshold. This configuration provides a certain range for the selection of the intensity of the sources 12 and 16 to reduce power consumption.
[0037] In one example, the source 12 illuminating the eye may be continuously on, but its intensity may be significantly less than Figure 1A , Figure 1B The intensity of the intermittent source 16 can also be set at a relatively low level.
[0038] Alternatively, the two illustrated lighting phases may be interleaved with a standby phase in which both sources are turned off and the sensor is disabled.
[0039] To optimize power consumption, the intensity of the background illumination source 16 may be increased gradually rather than in a single large step until an event corresponding to the pupil region is triggered.
[0040] Figure 4A and Figure 4B Two stages of a dark pupil detection method using an event-based sensor according to a second embodiment are illustrated. Figure 4A In, similar to Figure 2 The layout of A is used for sensors and illumination sources. However, instead of illuminating the eye continuously with source 12 and flashing source 16 to intermittently add a background illumination signal, sources 12 and 16 are flashed in anti-phase. On the left, source 12 illuminates the eye while source 16 is off, producing a pattern similar to Figure 2 B The image on the left shows a dark pupil and a lighter surrounding area, shown in gray. On the right, source 16 illuminates the sensor while source 12 is off, producing an image with a uniform brightness level, shown in gray.
[0041] In theory, by adjusting the intensity of the sources 12, 16, the brightness level of the rest of the eye region in the two stages can be made substantially equal, so that no difference will be measurable for that region, thus increasing the discrimination tolerance for detecting the pupil. However, this may be difficult in practice because the two stages correspond to different lighting conditions (the eye on one side and the sensor on the other side). In any case, similar to Figure 2 A to Figure 2C In an embodiment, the rate of change in the pupil region is maintained substantially greater than the rate of change in the rest of the eye region.
[0042] Figure 5 Pixel circuits of an array modified according to an embodiment are illustrated in order to add a uniform background signal within the sensor array. Three pixels Px1 to Px3 are partially illustrated, with pixel Px3 being illustrated in more detail. Each pixel of an event-based sensor typically includes a photodiode PD with an anode connected to ground potential. The photocurrent generated by the diode flows from a high power line through a transimpedance amplifier TIA, which typically has a logarithmic response, into the cathode of the photodiode. The output of the TIA is processed by a circuit (not shown) to generate an event in any known manner.
[0043] A corresponding N-MOS transistor M1 to M3 is connected between the cathode of each photodiode and ground potential. Transistors M1 to M3 form the output of a current mirror that shares a common diode-connected input N-MOS transistor M0. A current source Ibgr is connected to inject a background reference current Ibgr into the diode-connected transistor M0, which is copied into each of transistors M1 to M3 and added to the photocurrent measured by the TIA. The current Ibgr is adjusted to produce the desired background signal strength.
[0044] N-MOS transistor Mb connects the gates of transistors M0 to M3 to ground and is controlled by an active low background enable signal / BGR. Signal / BGR can thus switch the sensor between a normal mode and a simulated background illumination mode.
[0045] As mentioned above, according to Figure 2 In the configuration shown at right side of A, one embodiment of background signal source 16 is an illumination source directed toward the lens of the sensor. The lens will partially reflect light from source 16 along a cone. If source 16 is close enough to the optical axis or the lens has sufficient curvature, the reflected cone may intersect the pupil, whereby the light penetrates the pupil along the optical axis, placing the pupil in a bright pupil detection condition.
[0046] This configuration further increases the brightness variation of the pupil. In other words, Figure 2C The linear amplitude step in the pupil is larger than that in the rest of the eye. Figure 2 B and Figure 4B To the right of FIG. 1 , when source 16 is switched on, the pupil will be a lighter shade of grey.
[0047] This particular configuration can be simplified to operate a bright pupil detection technique by using only source 16. Source 16 is then permanently turned on at a relatively low intensity, generating a uniform gray background on the sensor. Source 12 is not present or is permanently turned off. The light partially reflected by the lens illuminates the eye and pupil, which produces an image with a bright pupil superimposed on the background gray level of the sensor. The pupil area, although perhaps not as bright as in the case of a properly exposed eye, will still be significantly brighter than the rest of the perceived image. Therefore, the pupil will be relatively easy to track with any type of sensor (event-based or frame-based), thereby providing the advantages of the bright pupil sensing technique, namely, avoiding false positives that may result from dark irises with dark pupil sensing techniques.
Claims
1. A method for detecting the pupil of an eye using an event-based vision sensor, the method comprising the following steps: directing a first light source (12) toward the eye (10); observing the eye using the event-based sensor (14); as well as The sensor is intermittently subjected to a uniform background signal (16) so that all areas viewed by the sensor see their brightness increase in the same step.
2. The method according to claim 1, comprising the steps of: Setting the level of the background signal (16) so that the relative brightness change sensed for the pupil is greater than the relative brightness change sensed for the rest of the eye; as well as The detection thresholds of the pixels of the sensor are set so that the relative brightness change of the pupil triggers an event and the relative brightness change of the rest of the eye does not trigger an event.
3. A method according to claim 2, comprising the step of subjecting the sensor to the background signal during a phase of illuminating the eye with the first light source.
4. The method of claim 2, comprising the step of illuminating the eye in an opposite phase to the background signal. The method of claim 1 , wherein the background signal is gradually increased until the sensor triggers an event.
6. The method of claim 1, wherein the background signal is obtained by directing a second light source (16) toward the sensor (14).
7. The method of claim 1, wherein the background signal is an electrical signal added within a pixel circuit of the sensor, the electrical signal having the effect of uniform illumination of the sensor.
8. The method of claim 7, wherein the electrical signal is a constant current added to a photocurrent generated in the pixel.
9. The method according to claim 6, wherein the sensor is equipped with a lens and the second light source (16) is arranged so that light partially reflected by the lens penetrates the pupil of the eye.
10. The method of claim 1, wherein the first light source is a display device viewed by the eye.
11. A method for detecting the pupil of an eye using a visual sensor equipped with a lens, the method comprising the following steps: observing the eye using the sensor (14); directing a light source (16) toward the lens of the sensor so that i) the sensor is subjected to a uniform background signal and ii) light partially reflected by the lens penetrates the pupil of the eye; as well as Bright pupil detection is performed on the image perceived by the sensor.
