Pupil assessment using modulated coaxial illumination

Abstract

Various implementations use coaxial illumination to determine pupil features (e.g., perimeter location, pupil shape, pupil diameter, pupil center, etc.). Coaxial illumination involves producing light from a light source that is approximately coaxial with an image sensor that is configured to capture reflections of the light from the light source from an eye. The light from the light source can pass through the pupil into the eye and reflect off the retina to produce a bright-pupil type light pattern in data obtained by the image sensor. The light can be pulsed at a certain frequency so that frequency division can be used to distinguish reflections of the light that pass through the pupil off the retina from reflections of light from other light sources. In some implementations, the image sensor is an event camera that detects events. Pupil features can be assessed by evaluating events that recur in a given region at a given frequency.

Description

Technical Field

[0001] This disclosure generally relates to assessing the characteristics of the pupil of the eye, and more particularly to systems, methods, and apparatus for assessing pupil characteristics using light reflected from the eye. Background Technology

[0002] Some existing systems use light reflected from the surface of the eye to estimate eye features. For example, such techniques can use multiple flashes to identify the shape, position, and orientation of a user's eyes to estimate the user's gaze direction. Existing pupil detection techniques use an image sensor and integrate the light intensity level over the exposure cycle to produce a grayscale image, then attempt to detect the pupil using this grayscale image. Pupil detection is based on the grayscale contrast between the pupil region and the surrounding iris region, thus relying on a significant contrast between the pupil and the iris. Such pupil detection techniques may not be accurate or efficient, especially when the contrast between the pupil and the iris is not significant. Summary of the Invention

[0003] Various implementations use coaxial illumination to determine pupil features (e.g., peripheral position, pupil shape, pupil diameter, pupil center, etc.). Coaxial illumination involves generating light from a light source that is substantially coaxial with an image sensor configured to capture the reflection of light from that source from the eye. Light from the light source can pass through the pupil into the eye and reflect from the retina to produce a bright pupil-shaped light pattern in the data acquired by the image sensor. The light can be modulated at a certain frequency or otherwise pulsed, and frequency segmentation can be used to distinguish the reflection from the retina through the pupil from light from other light sources. In some implementations, the image sensor is an event camera that detects events. The pupil features can be evaluated by assessing events that recur at a given frequency in a given area. The pupil features can be used to determine the direction of gaze or for other purposes.

[0004] Some implementations involve a method for determining pupil features at an electronic device with a processor. For example, the processor may execute instructions stored in a non-transitory computer-readable medium to determine pupil features based on the reflection of pulsed light leaving the retina and passing through the pupil of the eye. This method involves generating pulsed light of a certain frequency via a light source. The light source may be substantially “coaxial” with the optical axis of the image sensor. In some implementations, “substantially” coaxial illumination is achieved using waveguides or beam splitters. In some implementations, “substantially” coaxial illumination is achieved using a light source sufficiently close to the image sensor optics, such that, under the intended use conditions (e.g., for the intended distances of the light / source and the image sensor from the eye), the axes are sufficiently aligned so that light from the source is reflected from the retina of the eye and sensed by the image sensor. In one example, the light source is positioned with a ring-shaped configuration around (and sufficiently close to) the lens / optics of the image sensor. Generally, a bright pupil effect can be observed with an angle of approximately 15 degrees between the axis of the light source and the camera, depending on pupil dilation.

[0005] This method receives sensor data at a sensor and identifies a subset of the sensor data corresponding to the reflection of pulsed light leaving the retina and passing through the pupil of the eye, based on the frequency. In some implementations, the method distinguishes data corresponding to the reflection of pulsed light from reflections of light from another light source based on the frequency. In some implementations, the image sensor is an event camera, and the method determines the amount of time between events corresponding to the rising or falling edge of light reflected back through the pupil to determine events occurring at that frequency. In some implementations, the image sensor is a frame-based camera, and the method subtracts one image from the next along an image sequence to identify light pulses appearing at that frequency in the images.

[0006] This method determines pupil features (e.g., perimeter position, pupil outline, pupil shape, pupil center, etc.) based on a subset of sensor data corresponding to the reflections of pulsed light leaving the retina and passing through the pupil. For example, the positions of multiple events corresponding to pupil position can provide information about the pupil's position, shape, center, size, and orientation. Pupil features can be used to determine gaze direction or for other purposes.

[0007] Some specific embodiments provide an apparatus configured to substantially align a light source and an image sensor to determine pupil features using frequency segmentation. Such an apparatus may include: a light source configured to generate pulsed light; a sensor configured to provide sensor data and having an optical axis, wherein the light source and the optical axis of the sensor are substantially coaxially aligned; a processor; and a computer-readable storage medium. The computer-readable medium may include instructions that, when executed by the processor, cause the system to perform frequency segmentation on the sensor data to distinguish data corresponding to reflections of pulsed light from data corresponding to reflections of light from another light source and to determine pupil features based on the data corresponding to reflections of pulsed light. The light source may use a waveguide or beamsplitter configured to align the light from the light source with the optical axis of the sensor. Alternatively, the light source may include a light source ring surrounding optics of the sensor.

[0008] According to some embodiments, an apparatus includes one or more processors, non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing to perform any of the methods described herein. According to some embodiments, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors of the apparatus, cause the apparatus to perform or cause to perform any of the methods described herein. According to some embodiments, an apparatus includes: one or more processors, non-transitory memory, and means for performing or causing to perform any of the methods described herein. Attached Figure Description

[0009] Therefore, this disclosure will be understood by those skilled in the art, and a more detailed description can be made with reference to some exemplary embodiments, some of which are shown in the accompanying drawings.

[0010] Figure 1 It is a block diagram based on some specific implementations of exemplary operating environments.

[0011] Figure 2 It is a block diagram of an exemplary controller based on some specific implementations.

[0012] Figure 3 It is a block diagram based on some specific implementations of exemplary devices.

[0013] Figure 4 It is a block diagram of an exemplary head-mounted device (HMD) based on some specific implementations.

[0014] Figure 5 A block diagram of an event camera according to some specific implementations is shown.

[0015] Figure 6It is a flowchart representation of a method for assessing pupil characteristics based on some specific implementations.

[0016] Figure 7 A functional block diagram illustrating the difference between the bright pupil effect and the dark pupil effect according to some specific implementations is shown.

[0017] Figure 8 A functional block diagram is shown illustrating the use of a beam splitter to provide approximately coaxial illumination, according to some specific implementations.

[0018] Figure 9 A functional block diagram is shown illustrating the use of a light source ring near the optics of a light sensor to provide approximately coaxial illumination, according to some specific implementations.

[0019] Figure 10 Explanation is shown Figure 9 Functional block diagram of the light source ring.

[0020] Figure 11 A functional block diagram is shown that combines pupil detection based on coaxial illumination with eye feature detection based on off-axis illumination, according to certain specific implementations.

[0021] Figure 12 A collection of event camera data acquired during coaxial illumination is shown, according to certain specific implementations.

[0022] Figure 13 It shows Figure 12 A close-up view of a portion of the event camera data.

[0023] As is customary, the various features shown in the accompanying drawings may not be drawn to scale. Therefore, for clarity, the dimensions of various features may be arbitrarily expanded or reduced. Additionally, some drawings may not depict all components of a given system, method, or apparatus. Finally, similar reference numerals may be used throughout the specification and drawings to denote similar features. Detailed Implementation

[0024] Numerous details have been described to provide a thorough understanding of the exemplary embodiments illustrated in the accompanying drawings. However, the drawings illustrate only some exemplary aspects of this disclosure and should not be considered limiting. Those skilled in the art will understand that other effective aspects and / or variations do not include all the specific details described herein. Furthermore, well-known systems, methods, components, devices, and circuits have not been described exhaustively so as not to obscure further relevant aspects of the exemplary embodiments described herein.

[0025] A pupil feature assessment system includes a light source, an image sensor, and a processor that performs pupil feature assessment based on data received from a light sensor regarding light reflected from the light source and exiting the user's eye. In various implementations, coaxial illumination from the light source is used to determine pupil features, such that light from the light source is reflected from the retina of the eye to produce a bright pupil-shaped light pattern in the data obtained by the image sensor. The light can be modulated at a certain frequency or otherwise pulsed, and frequency segmentation can be used to distinguish reflections through the pupil from the retina from light reflected from other light sources. In some implementations, the image sensor is a frame-based camera, and the method subtracts one image from the next along an image sequence to identify light pulses appearing at that frequency in the images.

[0026] In some implementations, the image sensor is an event camera, and the amount of time between events corresponding to light reflecting from the retina and passing through the pupil is used to determine events occurring at that frequency. The event camera has light sensors at multiple corresponding locations. In response to a particular light sensor detecting a change in light intensity, the light sensor generates an event message indicating the specific location of that particular light sensor. The event camera may include, or be referred to as, a dynamic vision sensor (DVS), a silicon retina, an event-based camera, or a frameless camera. Therefore, the event camera generates (and transmits) data about changes in light intensity, rather than a larger amount of data about the absolute intensity at each light sensor.

[0027] Figure 1 This is a block diagram of an exemplary operating environment 100 according to some specific implementations. Although relevant features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure further relevant aspects of the exemplary implementations disclosed herein. Therefore, as a non-limiting example, operating environment 100 includes a controller 110 and a device 120.

[0028] In some implementations, controller 110 is configured to manage and coordinate the user experience. In some implementations, controller 110 includes a suitable combination of software, firmware, and / or hardware. See below for reference. Figure 2 The controller 110 is described in more detail. In some embodiments, the controller 110 is a computing device located locally or remotely relative to physical set 105. In one example, the controller 110 is a local server located within physical set 105. In another example, the controller 110 is a remote server (e.g., a cloud server, central server, etc.) located outside physical set 105. In some embodiments, the controller 110 is communicatively coupled to the device 120 via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.).

[0029] In some implementations, device 120 is configured to present an environment to a user. In some implementations, device 120 includes a suitable combination of software, firmware, and / or hardware. See below for reference. Figure 3 Device 120 is described in more detail. In some specific implementations, the functionality of controller 110 is provided by and / or combined with device 120.

[0030] According to some specific implementations, when a user is virtually and / or physically present within physical environment 105, device 120 presents a simulated reality (SR) environment (e.g., an augmented reality / virtual reality (AR / VR) environment) to the user. In some specific implementations, when presenting an experience, device 120 is configured to present content and provide optical perspective of physical environment 105. In some specific implementations, when presenting an environment, device 120 is configured to present VR content and provide video perspective of physical environment 105.

[0031] In some embodiments, the user wears device 120 as a head-mounted device (HMD) on his or her head. Thus, device 120 includes one or more displays provided for displaying content. For example, device 120 surrounds the user's field of vision. In some embodiments, device 120 is a handheld electronic device (e.g., a smartphone or tablet) configured to present content to the user. In some embodiments, device 120 is replaced by a cavity, housing, or chamber configured to present content, wherein the user does not wear or hold device 120.

[0032] Figure 2This is a block diagram of an example controller 110 according to some specific implementations. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and in order not to obscure further relevant aspects of the specific implementations disclosed herein. Therefore, as a non-limiting example, in some specific implementations, controller 110 includes one or more processing units 202 (e.g., microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), graphics processing units (GPUs), central processing units (CPUs), processing cores, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., Universal Serial Bus (USB), FireWire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), Infrared (IR), Bluetooth, ZigBee, and / or similar type interfaces), one or more programming (e.g., I / O) interfaces 210, memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

[0033] In some embodiments, the one or more communication buses 204 include circuitry for communication between interconnecting system components and control system components. In some embodiments, one or more I / O devices 206 include at least one of a keyboard, mouse, touchpad, joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0034] Memory 220 includes high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some embodiments, memory 220 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 220 optionally includes one or more storage devices located remotely from the one or more processing units 202. Memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, memory 220 or the non-transitory computer-readable storage medium of memory 220 stores programs, modules, and data structures, or subsets thereof, including optional operating system 230 and experience module 240.

[0035] Operating system 230 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, experience module 240 is configured to manage and coordinate one or more experiences for one or more users (e.g., a single experience for one or more users, or multiple experiences for a corresponding group of one or more users). To this end, in various implementations, experience module 240 includes a data acquirer 242, a tracker 244, a coordinator 246, and a renderer 248.

[0036] In some implementations, the data acquirer 242 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least device 120. To this end, in various implementations, the data acquirer 242 includes instructions and / or logic components for those instructions, as well as heuristics and metadata for those heuristics.

[0037] In some embodiments, tracker 244 is configured to map physical environment 105 and track the orientation / position of at least device 120 relative to physical environment 105. In some embodiments, tracker 244 is configured to perform pupil assessment via one or more techniques disclosed herein. To this end, in various embodiments, tracker 244 includes instructions and / or logic for instructions, as well as heuristics and metadata for heuristics.

[0038] In some implementations, coordinator 246 is configured to manage and coordinate the experience presented to the user by device 120. To this end, in various implementations, coordinator 246 includes instructions and / or logic components for those instructions, as well as heuristics and metadata for those heuristics.

[0039] In some implementations, renderer 248 is configured to render content for display on device 120. To this end, in various implementations, renderer 248 includes instructions and / or logic components for the instructions, as well as heuristics and metadata for the heuristics.

[0040] Although the data acquirer 242, tracker 244, coordinator 246 and renderer 248 are shown residing on a single device (e.g., controller 110), it should be understood that in other specific implementations, any combination of these elements may reside in a separate computing device.

[0041] also, Figure 2 This is used more as a functional description of various features present in a specific implementation, and differs from the structural diagrams of the specific implementations described herein. As those skilled in the art will recognize, items shown individually can be combined, and some items can be separated. For example, Figure 2Some functional modules shown individually can be implemented in a single module, and the various functions of a single functional block can be implemented in various specific implementations through one or more functional blocks. The actual number of modules and the division of specific functions, as well as how features are allocated therein, will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software, and / or firmware selected for a particular implementation.

[0042] Figure 3 This is a block diagram of an example of a device 120 according to some specific implementations. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and in order not to obscure further relevant aspects of the specific implementations disclosed herein. Therefore, as a non-limiting example, in some specific implementations, device 120 includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, Firewire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BlueTooth, ZigBee, SPI, I2C, or similar interfaces), one or more programming (e.g., I / O) interfaces 310, one or more displays 312, one or more internal or external image sensors 314, memory 320, and one or more communication buses 304 for interconnecting these components and various other components.

[0043] In some embodiments, one or more communication buses 304 include circuitry for interconnecting and communicating between system components. In some embodiments, the one or more I / O devices and sensors 306 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, and / or one or more depth sensors (e.g., structured light, time-of-flight, etc.).

[0044] In some embodiments, one or more displays 312 are configured to present an experience to a user. In some embodiments, one or more displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conducting electron emitter display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical systems (MEMS), and / or similar display types. In some embodiments, one or more displays 312 correspond to waveguide displays such as diffraction, reflection, polarization, and holography. For example, device 120 includes a single display. As another example, device 120 includes displays for each of the user's eyes. In some embodiments, one or more displays 312 are capable of displaying SR content.

[0045] In some embodiments, the one or more image sensor systems 314 are configured to acquire image data corresponding to at least a portion of a user's face, including the user's eyes. For example, the one or more image sensor systems 314 may include one or more RGB cameras (e.g., having a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), monochrome cameras, IR cameras, event-based cameras, etc. In various embodiments, the one or more image sensor systems 314 may also include an illumination source that emits light onto a portion of the user's face, such as a flash, a flash source, or coaxial illumination. In some embodiments, the image sensor is configured to be substantially coaxial with the optical axis of the light source.

[0046] Memory 320 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, memory 320 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 320 optionally includes one or more storage devices located remotely from the one or more processing units 302. Memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, memory 320 or the non-transitory computer-readable storage medium of memory 320 stores programs, modules, and data structures, or subsets thereof, including optional operating system 330 and experience module 340.

[0047] Operating system 330 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, experience module 340 is configured to present content to a user via one or more displays 312. For this purpose, in various implementations, presentation module 340 includes data acquisition unit 342, presenter 344, pupil estimator 346, and data transmission unit 348.

[0048] In some implementations, the data acquisition unit 342 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the controller 110 and / or I / O devices and sensors 306. To this end, in various implementations, the data acquisition unit 342 includes instructions and / or logic components for those instructions, as well as heuristics and metadata for those heuristics.

[0049] In some implementations, the renderer 344 is configured to render content via one or more displays 312. To this end, in various implementations, the renderer 344 includes instructions and / or logic components for the instructions, as well as heuristics and metadata for the heuristics.

[0050] In some embodiments, the pupil evaluator 346 is configured to evaluate pupil features via one or more techniques disclosed herein. To this end, in various embodiments, the pupil evaluator 346 includes instructions and / or logic components for the instructions, a configured neural network, and heuristics and metadata for the heuristics.

[0051] In some implementations, the data transmitter 348 is configured to transmit at least data (e.g., presentation data, location data, etc.) to the controller 110. To this end, in various implementations, the data transmitter 348 includes instructions and / or logic components for instructions, as well as heuristics and metadata for heuristics.

[0052] Although these components are shown residing on a single device (e.g., device 120), it should be understood that in other specific implementations, any combination of components may reside in a separate computing device. Furthermore, Figure 3 This is used more as a functional description of various features present in a specific implementation, and differs from the structural diagrams of the specific implementations described herein. As those skilled in the art will recognize, items shown individually can be combined, and some items can be separated. For example, Figure 3Some functional modules shown individually can be implemented in a single module, and the various functions of a single functional block can be implemented in various specific implementations through one or more functional blocks. The actual number of modules and the division of specific functions, as well as how features are allocated therein, will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software, and / or firmware selected for a particular implementation.

[0053] Figure 4 A block diagram of a head-mounted device 400 according to some specific embodiments is shown. The head-mounted device 400 includes a housing 401 (or encapsulation) housing various components of the head-mounted device 400. The housing 401 includes (or is coupled to) eye pads 405 disposed at a proximal (relative to user 10) end of the housing 401. In various specific embodiments, the eye pads 405 are plastic or rubber parts that comfortably and snugly hold the head-mounted device 400 in a proper position on the face of user 10 (e.g., around the eyes of user 10).

[0054] Housing 401 houses display 410, which displays images, emits light toward or onto the eyes of user 10. In various embodiments, display 410 emits light through an eyepiece (not shown) that refracts the light emitted by display 410, making the display appear to user 10 at a virtual distance greater than the actual distance from the eyes to display 410. In order for the user to focus on display 410, in various embodiments, the virtual distance is at least greater than the minimum focal length of the eye (e.g., 7 cm). Furthermore, to provide a better user experience, in various embodiments, the virtual distance is greater than 1 meter.

[0055] Although Figure 4 A head-mounted device 400 is shown, including a display 410 and an eye pad 405. However, in various embodiments, the head-mounted device 400 may not include the display 410 or may include an optically transparent display but not the eye pad 405.

[0056] The housing 401 also houses a pupil assessment system comprising one or more light sources 422, an image sensor 424, and a controller 480. One or more light sources 422 emit light toward the eye of user 10, which reflects the light (e.g., a directional beam), and this light can be detected by the sensor 422. Based on the reflection, the controller 480 can determine the pupil characteristics of user 10. For example, the controller 480 can determine the pupil center, pupil size, gaze direction, or point of interest. Thus, in various embodiments, light is emitted by the one or more light sources 422, reflected from the eye of user 10, and detected by the sensor 424. In various embodiments, light from the eye of user 10 is reflected from a thermal mirror or passes through an eyepiece before reaching the sensor 424.

[0057] Display 410 can emit light within a first wavelength range, and the one or more light sources 422 can emit light within a second wavelength range. Similarly, sensor 424 can detect light within the second wavelength range. In various specific embodiments, the first wavelength range is the visible wavelength range (e.g., a wavelength range of approximately 400 nm to 700 nm within the visible spectrum), and the second wavelength range is the near-infrared wavelength range (e.g., a wavelength range of approximately 700 nm to 1400 nm within the near-infrared spectrum).

[0058] In various embodiments, one or more light sources 422 modulate or otherwise pulse the emitted light. For example, in various embodiments, the light sources in one or more light sources 422 are modulated at a certain frequency (e.g., 600 Hz). In various embodiments, one or more light sources 422 modulate the emitted light according to orthogonal codes (such as those that can be used in CDMA (Code Division Multiple Access) communications). For example, rows or columns of a Walsh matrix can be used as orthogonal codes.

[0059] In various embodiments, one or more light sources 422 modulate the emitted light between high and low intensity values. Therefore, at any given time, the intensity of the light emitted by the light source is either a high intensity value or a low intensity value. In various embodiments, the low intensity value is zero. Therefore, in various embodiments, the one or more light sources 422 modulate the intensity of the emitted light between an on state (at a high intensity value) and an off state (at a low intensity value).

[0060] In various specific implementations, one or more light sources 422 modulate the emitted light based on the user's biometrics. For example, if the user blinks more than normal, has an elevated heart rate, or is registered as a child, one or more light sources 422 attenuate the emitted light (or the sum of all light emitted by the light sources) to reduce eye strain. Alternatively, one or more light sources 422 may modulate the emitted light based on the user's eye color, since the spectral reflectance may differ for blue eyes compared to brown eyes.

[0061] In various specific implementations, one or more light sources 422 modulate the emitted light according to the presented user interface (e.g., the content displayed on the display 410). For example, if the display 410 is abnormally bright (e.g., displaying a video of an explosion), the one or more light sources 422 increase the intensity of the emitted light to compensate for potential interference from the display 410.

[0062] In various specific implementations, one or more other light sources (not shown) emit light toward the user's eyes, which is reflected from the surface of the eye in the form of one or more flashes.

[0063] In various specific implementations, sensor 424 is a frame / shutter-based camera that generates images of the user 10's eye at a frame rate at one or more time points. Each image includes a matrix of pixel values ​​corresponding to the pixels in the image, the pixels corresponding to the positions of the camera's light sensor matrix.

[0064] In various specific implementations, camera 424 is an event camera that includes multiple light sensors (e.g., a light sensor matrix) at multiple corresponding locations, which generates an event message indicating a specific location of a particular light sensor in response to a particular light sensor detecting a change in light intensity.

[0065] In various specific implementations, pupil feature assessment is used to facilitate gaze tracking, which can be used to enable user interaction (e.g., user 10 selects an option by looking at it on display 410), provide foveal rendering (e.g., rendering a higher resolution in the area of ​​display 410 that user 10 is looking at and a lower resolution elsewhere on display 410), or reduce geometric distortion (e.g., in 3D rendering of objects on display 410).

[0066] Figure 5 A functional block diagram of an event camera 500 according to some specific embodiments is shown. The event camera 500 includes a plurality of light sensors 515, each coupled to a message generator 532. In various specific embodiments, the plurality of light sensors 515 are arranged in a matrix 510 of rows and columns, and thus each of the plurality of light sensors 515 is associated with row values ​​and column values.

[0067] Each of the multiple light sensors 515 includes Figure 5 The light sensor 520 is shown in detail below. The light sensor 520 includes a photodiode 521 connected in series with a resistor 523 between a source voltage and a ground voltage. The voltage across the photodiode 521 is proportional to the intensity of light incident on the light sensor 520. The light sensor 520 includes a first capacitor 525 connected in parallel with the photodiode 521. Therefore, the voltage across the first capacitor 525 is the same as the voltage across the photodiode 521 (e.g., proportional to the intensity of light detected by the light sensor 520).

[0068] The light sensor 520 includes a switch 529 coupled between a first capacitor 525 and a second capacitor 527. The second capacitor 527 is coupled between the switch and ground. Therefore, when the switch 529 is closed, the voltage on the second capacitor 527 is the same as the voltage on the first capacitor 525 (e.g., proportional to the intensity of light detected by the light sensor 520). When the switch 529 is open, the voltage on the second capacitor 527 is fixed at the voltage on the second capacitor 527 when the switch 529 was last closed.

[0069] The voltage across the first capacitor 525 and the voltage across the second capacitor 527 are fed to comparator 531. When the difference 552 between the voltage across the first capacitor 525 and the voltage across the second capacitor 527 is less than a threshold value, comparator 531 outputs a "0" voltage. When the voltage across the first capacitor 525 is at least as high as the voltage across the second capacitor 527, comparator 531 outputs a "1" voltage. When the voltage across the first capacitor 525 is at least as low as the voltage across the second capacitor 527, comparator 531 outputs a "-1" voltage.

[0070] When comparator 531 outputs a voltage of "1" or "-1", switch 529 closes, and message generator 532 receives the digital signal and generates a pixel event message.

[0071] For example, at the first moment, the intensity of the light incident on the light sensor 520 is a first light value. Therefore, the voltage across the photodiode 521 is a first voltage value. Similarly, the voltage across the first capacitor 525 is a first voltage value. In this example, the voltage across the second capacitor 527 is also a first voltage value. Therefore, the comparator 531 outputs a "0" voltage, the switch 529 remains closed, and the message generator 532 does not perform any operation.

[0072] At the second time, the intensity of the light incident on the light sensor 520 increases to a second light value. Therefore, the voltage across the photodiode 521 is the second voltage value (higher than the first voltage value). Similarly, the voltage across the first capacitor 525 is the second voltage value. Because the switch 529 is open, the voltage across the second capacitor 527 remains the first voltage value. Assuming the second voltage value is at least this threshold higher than the first voltage value, the comparator 531 outputs a "1" voltage, closing the switch 529, and the message generator 532 generates an event message based on the received digital signal.

[0073] When switch 529 is closed due to a "1" voltage from comparator 531, the voltage across the second capacitor 527 changes from a first voltage value to a second voltage value. Therefore, comparator 531 outputs a "0" voltage, opening switch 529.

[0074] At the third time, the intensity of the light incident on the light sensor 520 increases again to a third light value. Therefore, the voltage across the photodiode 521 is the third voltage value (higher than the second voltage value). Similarly, the voltage across the first capacitor 525 is the third voltage value. Because the switch 529 is open, the voltage across the second capacitor 527 remains the second voltage value. Assuming the third voltage value is at least this threshold higher than the second voltage value, the comparator 531 outputs a "1" voltage, closing the switch 529, and the message generator 532 generates an event message based on the received digital signal.

[0075] When switch 529 is closed due to a "1" voltage from comparator 531, the voltage across the second capacitor 527 changes from a second voltage value to a third voltage value. Therefore, comparator 531 outputs a "0" voltage, opening switch 529.

[0076] At the fourth time, the intensity of the light incident on the light sensor 520 decreases back to the second light value. Therefore, the voltage on the photodiode 521 is the second voltage value (less than the third voltage value). Similarly, the voltage on the first capacitor 525 is the second voltage value. Because the switch 529 is open, the voltage on the second capacitor 527 remains the third voltage value. Therefore, the comparator 531 outputs a "-1" voltage, closes the switch 529, and the message generator 532 generates an event message based on the received digital signal.

[0077] When switch 529 is closed due to the -1 voltage from comparator 531, the voltage across the second capacitor 527 changes from the third voltage value to the second voltage value. Therefore, comparator 531 outputs a "0" voltage, opening switch 529.

[0078] The message generator 532 receives digital signals from each of the plurality of light sensors 510 at different times, the digital signals indicating an increase ("1" voltage) or a decrease ("-1" voltage) in light intensity. In response to receiving a digital signal from a specific light sensor among the plurality of light sensors 510, the message generator 532 generates a pixel event message.

[0079] In various embodiments, each pixel event message indicates a specific location of a particular light sensor in the location field. In various embodiments, the event message indicates a specific location in pixel coordinates, such as row values ​​(e.g., in the row field) and column values ​​(e.g., in the column field). In various embodiments, the event message further indicates the polarity of the light intensity change in the polarity field. For example, the event message may include a "1" in the polarity field to indicate an increase in light intensity, and may include a "0" in the polarity field to indicate a decrease in light intensity. In various embodiments, the event message further indicates the time at which the light intensity change was detected (e.g., the time of receiving a digital signal) in the time field. In various embodiments, the event message indicates a value representing the intensity of the detected light in the absolute intensity field (not shown), as an alternative to or in addition to polarity.

[0080] Figure 6 This is a flowchart representation of a method 600 for assessing pupil characteristics based on some specific implementations.

[0081] In some specific implementations, method 600 is performed by a device (e.g., Figure 1 and Figure 2The method 600 can be performed by a controller 110, such as a mobile device, desktop computer, laptop computer, or server device. The method 600 can also be performed on a device having a screen for displaying 2D images and / or a screen for viewing stereoscopic images (such as a head-mounted display (HMD)). Figure 1 and Figure 3 Execute on device 120).

[0082] In some embodiments, method 600 is performed by processing logic components (including hardware, firmware, software, or a combination thereof). In some embodiments, method 600 is performed by a processor that executes code stored in a non-transitory computer-readable medium (e.g., memory).

[0083] At box 610, method 600 generates pulsed light of a certain frequency via a light source. This frequency can be greater than the motion frequency of physical objects in the eye's physical environment. This frequency can be greater than the motion frequency of the content the user is viewing. In some embodiments, the light is infrared (IR) light. In some embodiments, a light source aligned substantially "coaxially" with the axis of the light sensor is used to pulse the light. In some embodiments, substantially coaxial illumination is generated via a waveguide or beam splitter. In some embodiments, substantially coaxial illumination is generated via a light source with optics sufficiently close to the sensor. In some embodiments, substantially coaxial illumination is generated via a ring of light sources surrounding the optics of the sensor.

[0084] At block 620, method 600 receives sensor data at the sensor. In some embodiments, the sensor is an event camera. In some embodiments, the sensor is a frame-based camera capable of capturing frames at a sufficient frequency to identify the pulse frequency of the pulsed light (e.g., 1000 fps).

[0085] At box 630, method 600 identifies a subset of sensor data corresponding to the reflection of pulsed light leaving the retina and passing through the pupil of the eye, based on this frequency. In some specific implementations, this involves distinguishing sensor data corresponding to the reflection of pulsed light from reflections of light from another light source based on this frequency. The sensor can be an event camera or a frame-based camera.

[0086] In a specific implementation where the sensor is an event camera, a subset of sensor data corresponding to the reflection of pulsed light can be identified by determining the amount of time between events and, based on that amount of time and frequency, determining whether the event corresponds to the reflection of pulsed light. Using an event camera can provide advantages over techniques that rely solely on shutter-based (e.g., frame-based) camera data. Event cameras can efficiently capture data at very high sampling rates and are therefore well-suited for effectively and accurately identifying reflected light pulses at specific frequencies.

[0087] In a specific implementation where the sensor is a frame-based camera that captures images, a subset of sensor data corresponding to the reflection of pulsed light can be identified by subtracting the images from each other to identify the light pulses of that frequency.

[0088] At box 640, method 600 involves determining pupil features (e.g., perimeter location, pupil shape, pupil center, etc.) based on a subset of reflections of pulsed light corresponding to the pulses of light leaving the retina and passing through the pupil of the eye from sensor data.

[0089] In various specific implementations, Figure 6 Pupil feature assessment is used to facilitate gaze tracking. For example, the pupil center can be determined based on pupil feature assessment and used as part of gaze direction determination, which is based on the pupil center, corneal center, eyeball center, etc. In some implementations, gaze direction is determined based on an ellipse corresponding to the pupil shape. In some implementations, additional information about the eye is used to determine gaze direction. For example, an additional light source can be used to generate flashes on the cornea to provide information about the corneal position and orientation or other eye features. Thus, in some implementations, gaze direction is determined based on pupil assessment and one or more flashes of light reflected from a second light source and exiting the corneal surface of the eye.

[0090] Gaze direction can be used for a variety of purposes. In one example, a determined or updated gaze direction is used to identify items displayed on a monitor, such as which button, image, text, or other user interface item the user is looking at. In another example, determined or updated gaze characteristics are used to observe the movement of graphical indicators (e.g., a cursor or other user-controlled icon) displayed on the monitor. In yet another example, determined or updated gaze characteristics are used to select items displayed on the monitor (e.g., selecting a command via the cursor). For example, specific gaze movement patterns can be identified and interpreted as specific commands.

[0091] In some implementations, gaze tracking is performed simultaneously on both eyes of the same individual. In implementations where images of both eyes are captured or exported, the system can determine or generate an output of a convergence point that can be used to determine the gaze direction of the two eyes. The system can be additionally or alternatively configured to account for special cases, such as misaligned optical axes.

[0092] In some implementations, post-processing of the gaze direction is employed. Filtering and prediction methods (e.g., using a Kalman filter) can be used to reduce noise in the tracked gaze direction. These methods can also be used to interpolate / extrapolate the gaze direction over time. For example, these methods can be used if the gaze direction state at a different timestamp than the recording state is required.

[0093] Some of the specific implementations disclosed in this paper apply pulsed light and frequency-based segmentation to evaluate pupils. Segmentation in the frequency space is robust to static noise sources and can be implemented with relatively low computational cost and power. In some implementations, an event camera can facilitate frequency segmentation by responding to flickering / pulsed light. In some implementations, the time interval between two events is counted to provide a “frequency response” image, which can be automatically interpreted to identify pupil features.

[0094] Some implementations utilize the retroreflective properties of the retina at the back of the eye (e.g., properties previously thought to cause the "red-eye" and "bright pupil" effects). In some implementations, light is pulsed at a frequency substantially different from the frequency spectrum of motion in the physical environment (e.g., higher than the highest frequency in said motion), making it easier to apply frequency-based segmentation to distinguish reflections passing through the pupil from other reflections.

[0095] When the light source is nearly coaxial with the image sensor (approximately coaxial illumination), it "illuminates" the pupil due to the retroreflection effect of the retina. This can be observed as the "red-eye" effect in photography and the "bright pupil" effect in IR light. Approximately coaxial illumination means that the light rays projected from the light source and the reflected light rays from the retina are almost parallel to the camera's optical axis.

[0096] Figure 7 The differences between the bright pupil effect and the dark pupil effect are shown according to some specific implementations. Figure 7 In this configuration, the light source 705 is positioned far enough from the image sensor 705 that the light 728 does not provide a reflection 726 from the retina 724 of the eye 720 captured by the image sensor 710. This off-axis illumination produces a normal occultation effect.

[0097] Conversely, when the light source 705 is sufficiently close to the image sensor 705, the light source 705 generates light 738, which provides a reflection 736 of the light captured by the image sensor 710 as it exits the retina 724 of the eye 720. This coaxial illumination produces a bright pupil effect, where the reflected light passes through the pupil 722 of the eye. The pattern of the light reflected through the pupil 722 corresponds to the size, shape, and position of the pupil.

[0098] Approximately coaxial illumination can be achieved using waveguides (e.g., beam splitters). Figure 8The illustration shows the use of a beam splitter 815 to provide approximately coaxial illumination. In this example, a light source 705 generates light reflected at an angle from the beam splitter 815 to provide a ray 836 parallel to a reflected ray 838 from the retina of the eye 720. The reflected ray 838 passes through the beam splitter and is captured by the image sensor 710. A beam splitter can be used to provide perfectly coaxial light, but this can result in light loss (e.g., only 25% of the light reaches the image sensor because it passes through the beam splitter twice). The same effect of making the virtual position of the light source substantially coaxial with the camera axis can be achieved using different types of waveguides, including mirrors, optical fibers, diffractive optics, holographic elements, etc.

[0099] An alternative is to illuminate the image sensor optics near or on top of them, or as a surrounding ring. Figure 9 A light source ring 905 is shown used near the optics 940 of the light sensor to provide approximately coaxial illumination. The light source ring 905 is positioned close enough to the image sensor optics 940 that, under intended use conditions (e.g., for the intended distances of the light source and the image sensor from the eye), the axes are sufficiently aligned so that light from the light source ring 905 is reflected from the retina of the eye 720 and sensed by the image sensor 710. Figure 10 A light source ring 905 surrounding the image sensor optics 940 is shown.

[0100] Some implementations combine eye assessment based on coaxial and off-axis illumination. In some implementations, coaxial illumination is used to generate a bright pupil effect for determining pupil characteristics, and off-axis illumination is used to generate flashes on the eye surface to determine other eye characteristics. By positioning the flashes sufficiently far from the image sensor axis, the flashes can be generated without interfering with the bright pupil frequency modulation. Pulsing the coaxial pupil illumination at a different frequency than the off-axis flash illumination can be used to distinguish them in the sensor data.

[0101] Figure 11 A functional block diagram illustrating the combination of coaxial illumination-based pupil detection with off-axis illumination-based eye feature detection, according to certain specific embodiments, is shown. In this example, a coaxial light source (e.g., ring light source 905) is used in conjunction with an off-axis light source 1105. The coaxial light source (e.g., ring light source 905) is used to generate a bright pupil effect for determining pupil features, and the off-axis illumination is used to generate flashes (e.g., flash 1110) on the eye surface to determine other eye characteristics.

[0102] Figure 12 This illustrates a collection of event camera data acquired during coaxial illumination. For example, events occurring within a given time period can be compiled and used to generate a frequency response image. Figure 13 It shows Figure 12This is a close-up view of a portion of the event camera data. In this example, events in event group 1300 can be distinguished from events in event group 1310 based on their frequency. The system can determine, based on frequency, that events in event group 1300 are associated with coaxial illumination and thus indicate the position, size, and shape of the pupil. Conversely, the system can determine that events in event group 1310 are not associated with coaxial illumination and therefore not with the pupil. Event group 1310 may be caused by the sun or other internal or external light sources.

[0103] This document sets forth numerous specific details to provide a comprehensive understanding of the claimed subject matter. However, those skilled in the art will understand that the claimed subject matter can be practiced without these specific details. In other instances, methods, apparatus, or systems known to a person of ordinary skill have not been described in detail so as not to obscure the claimed subject matter.

[0104] Unless otherwise specifically stated, it should be understood that throughout this specification, discussions using terms such as “processing,” “calculating,” “computing,” “determining,” and “identifying” refer to the actions or processes of computing devices, such as one or more computers or similar electronic computing devices, which manipulate or convert data representing physical electronic or magnetic quantities within the memory, registers, or other information storage, transmission, or display devices of a computing platform.

[0105] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include computer systems based on multi-purpose microprocessors that access stored software that programs or configures the computing system from a general-purpose computing device to a special-purpose computing device that implements one or more specific embodiments of the subject matter of this invention. The teachings contained herein can be implemented in the software used for programming or configuring the computing device using any suitable programming, scripting, or other type of language or combination of languages.

[0106] Specific implementations of the methods disclosed herein can be performed in the operation of such a computing device. The order of the boxes presented in the above examples can be varied; for example, the boxes can be reordered, combined, and / or divided into sub-blocks. Some boxes or processes can be executed in parallel.

[0107] The use of “applies to” or “configured to” in this document implies open and inclusive language, which does not exclude applicability to or configuration to devices performing additional tasks or steps. Similarly, the use of “based on” implies openness and inclusivity, as processes, steps, calculations, or other actions “based on” one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbering included in this document are for illustrative purposes only and are not intended to be restrictive.

[0108] It will also be understood that while terms such as "first," "second," etc., may be used in this document to describe various elements, these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, a first node can be called a second node, and similarly, a second node can be called a first node, changing the meaning of the description, provided that all occurrences of "first node" are consistently renamed and all occurrences of "second node" are consistently renamed. First nodes and second nodes are both nodes, but they are not the same node.

[0109] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the claims. As used in the description of these embodiments and the appended claims, the singular forms “a” and “the” are intended to also cover the plural forms unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will also be understood that the term “comprising” as used in this specification specifies the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0110] As used herein, the term "if" can be interpreted as meaning "when the prerequisite is true" or "when the prerequisite is true" or "in response to determination" or "according to determination" or "in response to detection" that the prerequisite is true, depending on the context. Similarly, the phrases "if it is determined [the prerequisite is true]" or "if [the prerequisite is true]" or "when [the prerequisite is true]" are interpreted as meaning "when it is determined that the prerequisite is true" or "in response to determination" or "according to determination" that the prerequisite is true or "when the prerequisite is detected" or "in response to detection" that the prerequisite is true, depending on the context.

[0111] The foregoing description and summary of the present invention should be understood as illustrative and exemplary in every respect, and not restrictive, and the scope of the invention disclosed herein is determined not only by the detailed description of the illustrative specific embodiments, but also by the full extent permitted by patent law. It should be understood that the specific embodiments shown and described herein are merely illustrative of the principles of the invention, and various modifications can be made by those skilled in the art without departing from the scope and spirit of the invention.

Claims

1. A method for pupil assessment, comprising: In electronic devices with processors: A pulsed light of a first frequency is generated by a light source, the first frequency corresponding to the pulse of the pulsed light; Receive sensor data at the sensor location; Identifying a subset of the sensor data corresponding to the reflection of the pulsed light leaving the retina and passing through the pupil of the eye based on the first frequency includes: distinguishing the sensor data corresponding to the reflection of the pulsed light from the sensor data reflecting light from another light source different from the light source based on the first frequency; as well as Pupil features are determined based on the subset of reflections of the pulsed light corresponding to the pulsed light leaving the retina and passing through the pupil of the eye, according to the sensor data.

2. The method of claim 1, wherein the sensor is an event camera, and the subset of sensor data corresponding to the reflection of the pulsed light is identified by determining the amount of time between events and determining, based on the amount of time and the first frequency, whether the event corresponds to the reflection of the pulsed light.

3. The method of claim 1, wherein the sensor is a frame-based camera that captures images, and the subset of sensor data corresponding to the reflection of the pulsed light is identified by subtracting the images from each other to identify light pulses of the first frequency.

4. The method according to claim 1, wherein the pulsed light is infrared (IR) light.

5. The method according to claim 1, wherein the light source is substantially coaxially aligned with the optical axis of the sensor.

6. The method according to claim 1, wherein the angle between the light source and the optical axis of the sensor is less than 15 degrees.

7. The method of claim 6, wherein the light source has a virtual image generated by a waveguide and produces substantially coaxial illumination.

8. The method of claim 1, wherein substantially coaxial illumination is generated via a light source sufficiently close to the optics of the sensor, the light source being sufficiently close to the optics of the sensor such that the light source generates reflected light that is captured by the optics of the sensor and exits the retina of the eye.

9. The method of claim 1, wherein substantially coaxial illumination is generated via a light source ring surrounding the optics of the sensor.

10. The method of claim 1, wherein the first frequency is substantially different from the spectrum of the motion frequency of the visible object.

11. The method according to claim 1, further comprising determining the gaze direction based on the pupil features.

12. The method of claim 11, wherein the gaze direction is determined based on an ellipse corresponding to an pupil shape determined based on the sensor data.

13. The method of claim 11, wherein the gaze direction is determined based on one or more flashes of light reflected from the second light source and exiting the corneal surface of the eye.

14. The method according to claim 1, wherein the pupil feature is the pupil perimeter, pupil outline, pupil shape, or pupil center.

15. The method of claim 1, wherein the electronic device is a mobile device.

16. The method of claim 1, wherein the electronic device is a head-mounted device (HMD).

17. A device for pupil assessment, comprising: A light source configured to generate pulsed light at a first frequency, the first frequency corresponding to a pulse of the pulsed light; A sensor configured to provide sensor data and having an optical axis, wherein the light source is substantially coaxially aligned with the optical axis of the sensor; processor; as well as A computer-readable storage medium including instructions that, when executed by the processor, cause the device to perform operations, the operations including: The sensor data is frequency-segmented to distinguish data corresponding to the reflection of the pulsed light from data corresponding to the reflection of light from another light source different from the light source. as well as Pupil features are determined based on the data corresponding to the reflection of the pulsed light.

18. The device of claim 17, wherein the angle between the light source and the optical axis of the sensor is less than 15 degrees.

19. A non-transitory computer-readable storage medium storing computer-executable program instructions on a computer to perform operations, said operations including: A pulsed light of a first frequency is generated by a light source, the first frequency corresponding to the pulse of the pulsed light; Receive sensor data at the sensor location; Identifying a subset of the sensor data corresponding to the reflection of the pulsed light leaving the retina and passing through the pupil of the eye based on the first frequency includes: distinguishing the sensor data corresponding to the reflection of the pulsed light from the sensor data reflecting light from another light source different from the light source based on the first frequency; as well as Pupil features are determined based on the subset of reflections of the pulsed light corresponding to the pulsed light leaving the retina and passing through the pupil of the eye, according to the sensor data.

Images