Method, system and apparatus for simultaneous eye feature tracking and eye model updating

Abstract

Various implementations disclosed include inconspicuous eye registration of a user device through simultaneous or nearly simultaneous eye feature tracking and eye model updating. The disclosed implementations direct light towards a user's eye to produce scintillation reflections, and obtain image frames comprising depictions of these scintillation reflections via one or more sensors. Based on the image frames, the disclosed implementations track eye features and update an eye model of the eye during a time period between obtaining a first one of the image frames and obtaining a second one of the image frames.

Description

Technical Field

[0001] The present disclosure generally relates to methods, systems, and apparatus for simultaneously or nearly simultaneously tracking eye features and updating an eye model. Background Art

[0002] Various techniques are used to track the eyes. Separately, various techniques are used to generate a three-dimensional model of the eyes. Summary of the invention

[0003] Various embodiments disclosed herein include methods, systems, and devices for simultaneously tracking eye features and updating an eye model. For example, by simultaneously tracking eye features and updating an eye model, the methods, systems, and devices described herein illustrate eye model registration without requiring the user to follow multiple / extensive "look here" instructions. Instead, during a normal (e.g., minimally guided) experience, the methods, systems, and devices described herein generate and update a model of the user's eyes and simultaneously track eye features (e.g., gaze direction). In some examples, an illuminator (e.g., an illuminator of an infrared (IR) pattern) produces a flicker on the eye, and an image of the eye with the flicker is captured by one or more sensors (e.g., a camera). Some (e.g., key frames) or all of the images are used in real time to simultaneously (a) determine eye features (e.g., gaze direction) and (b) update the model.

[0004] In some examples, a computing device performs a method by using one or more light (e.g., IR) sources to direct light toward a user's eye to produce a flickering reflection. The method obtains an image frame via one or more sensors (e.g., at least one image sensor) that includes a depiction of the reflection of light via the eye. Based on the image frames, the method tracks eye features and updates at least a three-dimensional eye model of the eye during a time period between obtaining a first image frame in the image frames and obtaining a second image frame in the image frames. Thus, tracking and updating can occur when new individual frames are captured and selected. In some examples, the frames can be selected by selecting a subset of image frames based on comparing occupancy data (e.g., identifying a portion of an eye depicted in a given frame) to an overlay (e.g., identifying a portion of an eye already represented in a two-dimensional representation of the eye).

[0005] In some examples, a system includes a non-transitory computer-readable storage medium, one or more processors communicating with the non-transitory computer-readable storage medium, and a program instruction configured to be executed by one or more processors to perform or cause the execution of any of the methods described herein. In some examples, the program instruction is stored in a non-transitory computer-readable storage medium. In some examples, the program instruction, when executed, causes the system to perform or cause the execution of any of the methods described herein. In some examples, a system includes: one or more processors, a non-transitory memory, and a component for performing or causing the execution of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] So that the present disclosure may be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0007] Figure 1 An exemplary user device associated with the examples disclosed herein is shown.

[0008] Figure 2 An example block diagram associated with the examples disclosed herein is shown.

[0009] Figure 3 An exemplary overlay associated with the examples disclosed herein is shown.

[0010] Figure 4 A flow chart representing a method of tracking eye features and updating an eye model associated with examples disclosed herein is shown.

[0011] Figure 5 A block diagram of an electronic device according to some examples disclosed herein is shown.

[0012] As a matter of practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some of the drawings may not depict all of the components of a given system, method, or device. Finally, throughout the specification and drawings, the same reference numerals may be used to represent the same features. DETAILED DESCRIPTION

[0013] In order to provide a thorough understanding of the example implementations shown in the drawings, many details are described. However, the drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. It will be understood by those of ordinary skill in the art that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail to avoid obscuring more relevant aspects of the example implementations described herein.

[0014] A user of a head mounted display (HMD) may be required to participate in a registration process or a process in which the user must perform a series of tasks in response to visible or audible prompts. Such a registration process enables the HMD to accurately assess certain eye characteristics to enable the user to have a beneficial extended reality (XR) experience. In some examples, the registration process is inconvenient or burdensome for a user who wants to immediately dive into an XR experience, but is prevented from doing so unless the required series of tasks are first performed. The methods, systems, and apparatus described herein provide a registration experience that occurs in the background with minimal user interaction while the user performs other tasks (e.g., those tasks for which the user wishes to use the HMD).

[0015] Figure 1 A device 100 for tracking one or more eyes 102a, 102b of a user is shown. Although relevant features are shown, a person of ordinary skill in the art will recognize from the present disclosure that various other features are not shown for the sake of brevity and in order not to obscure more relevant aspects of the example embodiments disclosed herein. The device 100 may be an eyewear device, an HMD, a wearable device, a handheld device, or a device designed for other purposes suitable for eye tracking. In some examples, the device 100 may have one or more sensors 104a, 104b. For example, the device 100 may have one or more cameras (e.g., infrared cameras). In some examples, the device has a camera for each eye. In some examples, the device has multiple cameras for each eye. In some examples, the device has only a first camera, but the device also has one or more mirrors arranged to provide a reflected image to the first camera, which was originally provided by one or more second cameras.

[0016] The device 100 may also include one or more illuminators 106a, 106b for directing light to one or more eyes 102a detectable by the one or more sensors 104a, 104b. The device 100 may also include one or more content viewing portions 108a, 108b. These components may be embedded in or attached to a housing or other portion of the device 100. For example, if the device 100 is an eyeglass device, the one or more sensors 104a, 104b and the one or more illuminators 106a, 106b may be embedded in a frame portion of the eyeglass device that surrounds or supports the one or more content viewing portions 108a, 108b.

[0017] The illuminators 106a, 106b may be configured and positioned to illuminate the corresponding eyes 102a, 102b and / or surrounding areas with light to facilitate eye tracking. The illuminators 106a, 106b may emit light to one or more eyes 102a, 102b of the user, which is reflected from the surface of the eyes 102a, 102b in the form of one or more flashes. One or more illuminators 106a, 106b may be configured to emit light within a near-infrared wavelength range (e.g., a wavelength range within the near-infrared spectrum of approximately 700nm to 1400nm). However, any wavelength may be used as long as the light produces a flash from the surface of the eyes 102a, 102b. Each of the one or more illuminators 106a, 106b may include multiple illuminators that can be operated synchronously or asynchronously. In some examples, other types of sensors (in addition to illuminators) may be used, such as, for example, depth sensors, ultrasonic sensors, etc.

[0018] The one or more sensors 104a, 104b may each include one or more light sensors, other sensors, and / or processing components that use the received light to track eye features of the eyes 102a, 102b. For example, the illuminator 106a may emit light onto the eye 102a, which is reflected as a light pattern (e.g., a flickering circle) that can be detected by the sensor 104a. Based on the image data (e.g., of the eye 102a and / or the reflected flickering pattern) captured by the sensor 104a, the device 100 may determine the eye tracking features. For example, the device 100 may use the image data to determine the eye shape, eye / pupil position, eye orientation, and / or gaze direction of the eye 102a. Thus, in various examples, light may be emitted by one or more illuminators 106a, 106b, may be reflected from the eyes 102a, 102b, and may be detected by one or more sensors 104a, 104b and used to determine eye features of the eyes 102a, 102b. In some examples, such data can also be used to generate a multi-dimensional (e.g., three-dimensional) model of the eye.

[0019] One or more sensors 104a, 104b may be configured to detect light (e.g., reflected light) corresponding to light emitted from one or more illuminators 106a, 106b. For example, one or more illuminators 106a, 106b may be configured to generate light in a near-infrared wavelength range, and sensors 104a, 104b may be configured to capture light in the same (or similar) near-infrared wavelength range. In other examples, no illuminators are used and sensors 104a, 104b may capture images based on ambient light reflected from eyes 102a, 102b.

[0020] One or more content viewing portions 108a, 108b may display content or make content visible to a user. For example, one or more content viewing portions 108a, 108b may be a transparent lens through which a user can view the surrounding physical environment. In some examples, content is displayed via such otherwise transparent lenses, for example, using one or more waveguides to form images visible to one or more eyes 102a, 102b, respectively, on the lenses. In some examples, one or more content viewing portions 108a, 108b are opaque (e.g., non-transparent) displays that display images, emitting light to or onto one or more eyes 102a, 102b. The device 100 may display content (e.g., a stimulus or one or more stimuli) to one or more eyes 102a, 102b and track one or more eyes 102a, 102b while one or more eyes 102a, 102b view content through or on one or more viewing portions 108a, 108b, for example, tracking gaze direction, pupil position and / or occupancy of one or more eyes 102a, 102b (i.e., the portion of the eye depicted in a given frame), determining the portion of the content / physical environment being gazed at based on the gaze direction, and / or estimating the visual axis of one or more eyes 102a, 102b.

[0021] Figure 2 A system configured to simultaneously or nearly simultaneously track and update a user's eye model (e.g., Figure 1 The eye modeler 200 may be a block diagram of an eye modeler 200 for one or more eyes 102a, 102b of the embodiment of the present invention. Figure 1 The eye modeler 200 may obtain the Figure 1 The eye image or frame sequence 202 is captured by one or more sensors 104a, 104b, and the eye image or frame sequence 202 is input into the eye tracker 204 and the key frame selector 206. Figure 2 As shown, the eye image or frame sequence 202 may include a plurality (n) of images or frames 202a1-202a n In some examples, multiple (n) images or frames 202a1 to 202a n In some examples, multiple (n) images or frames 202a1-202a n But it's not continuous.

[0022] The eye tracker 204 may use a plurality (n) of images or frames 202a1-202a nThe eye tracker 204 may track eye blinks / pupil occupancy, for example, by determining one or more portions of an eye depicted in a given frame based on the frame content and / or blinks depicted in the frame content. In some examples, multiple (n) images or frames 202a1-202a2 may be used. n Each image or frame (e.g., all images or frames) in the image or frame can be used to track eye features in real time or near real time. In some examples, multiple (n) images or frames 202a1-202a n In some examples, eye tracker 204 may send data corresponding to current glint / pupil occupancy (e.g., the portion of the eye depicted in a given frame) to keyframe selector 206.

[0023] The key frame selector 206 may be based on a plurality (n) of images or frames 202a1-202a n and selects images or a subset of frames as key frames 202b1-202b based on eye blink / pupil occupancy received from the eye tracker 204 n In some examples, the key frame selector 206 may analyze the key frame associated with a plurality (n) of images or frames 202a1-202a n 202a1-202a2 , and compares the eye glint / pupil occupancy to a threshold. In some examples, the threshold may be an indication that the current eye glint / pupil occupancy from the image or frame is significantly different from the previous eye glint / pupil occupancy. In some examples, the threshold may be associated with a cornea / pupil overlay generated by the eye (re)constructor 208, as discussed further below. If the multiple (n) images or frames 202a1-202a2 are associated with the eye (re)constructor 208, the eye glint / pupil occupancy may be significantly different from the previous eye glint / pupil occupancy. n If the eye blink / pupil occupancy associated with a particular image or frame in the image satisfies a threshold, the key frame selector 206 may select the particular image or frame as a key frame.

[0024] Each selected key frame 202b1-202b n may be input to the eye (re)constructor 208. The eye (re)constructor 208 may be based on the selected keyframes 202b1-202b n To generate an eye model and cornea / pupil overlay. In some examples, the eye (re)constructor 208 may be based on the selected keyframes 202b1-202b n In some examples, the eye model and / or cornea / pupil overlay may be updated each time a new keyframe is selected by keyframe selector 206. In some examples, the eye model and / or cornea / pupil overlay may be updated each time a new keyframe is selected by keyframe selector 206. nThe eye model and / or cornea / pupil overlay may be updated during a period of time between obtaining a first image or frame in (i) and obtaining a second image or frame. The eye (re)constructor 208 may generate each new eye model based on the new and old keyframes. Thus, over time, the eye (re)constructor 208 may update and refine the model of the user's eye. Thus, in some examples, the eye modeler 200 may run continuously, and Figure 2 The data represented in may be processed in one or more software loops. In some examples, the eye modeler 200 may involve an unsupervised machine learning algorithm.

[0025] In some examples, the eye (re)constructor 208 may update the eye model and / or cornea / pupil overlay simultaneously (or nearly simultaneously) with the tracking of eye features by the eye tracker 204. In some examples, the eye (re)constructor 208 may update the eye model and / or cornea / pupil overlay, and the eye tracker 204 may track multiple (n) images or frames 202a1-202a n In some examples, the eye (re)constructor 208 and the eye tracker 204 use some of the same images or frames, so that the eye (re)constructor 208 and the eye tracker 204 share at least some common data.

[0026] The eye model may be a multi-dimensional model of the eye. For example, the eye model may be a model that represents the user's eye in three-dimensional space. The eye model may be a point cloud, a triangle mesh, or any other type of multi-dimensional model. In some examples, the eye model may be a four-dimensional model in terms of both space and time (e.g., a three-dimensional model of an eye that moves over time). In some examples, the eye model may include multiple dimensions corresponding to various eye features rather than space and time. The cornea / pupil overlay may be a binary two-dimensional array or grid (e.g., 5×5, 15×15, 25×25) representation of the cornea / pupil. If a glint is identified within a cell of the grid, the cell is filled with coverage at that location (e.g., binary yes).

[0027] In some examples, each iteration of the cornea / pupil overlay and the eye model is stored in the database 210 before being sent to the keyframe selector 206 and the eye tracker 204, respectively. In some examples, each iteration of the cornea / pupil overlay overwrites the previous iteration of the cornea / pupil overlay stored in the database 210, and each iteration of the eye model overwrites the previous iteration of the eye model stored in the database 210. In some examples, the database 210 stores multiple iterations of each of the cornea / pupil overlay and the eye model.

[0028] The database 210 can be a database local to the eye modeler 200 and / or the device 100, so that the data used to generate the eye model of the user's eye and other user information (e.g., cornea / pupil overlay) are only used to help improve the user experience (e.g., seamless user eye registration). In the process described herein, as an example, the eye model and other aspects of user privacy should be respected by ensuring that the user understands and agrees to the use of user data, understands what type of user data is used, controls the collection and use of user data, and limits the distribution of user data (e.g., by ensuring that user data is processed locally on the user device). The user should have the option to opt in or opt out of whether to obtain or use their user data or otherwise turn on and off any feature that obtains or uses user information. In addition, each user will have the ability to access and otherwise find out anything about him or her that the system has collected or determined. Such user data described herein can be securely stored on the user's device in the database 210, for example, as a secure partition to protect certain user information, such as data from images and other sensors for eye or other biometric identification. User data associated with the user's eye and / or attention state may be stored in such a secure enclave, thereby limiting access to the user data and limiting transmission of the user data to other devices to ensure that the user data remains securely on the user's device. The user data may be prohibited from leaving the user's device and may only be used for the described systems and apparatus described herein and corresponding methods performed locally on the user's device.

[0029] Figure 3 is an illustration of an example cornea / pupil overlay map 300. As described above, the cornea / pupil overlay map 300 can be a binary two-dimensional grid representing the cornea / pupil of a user. The cornea / pupil overlay map 300 may include a plurality of cells 302 (e.g., 5×5), which may be empty (e.g., binary 0) or shaded (e.g., binary 1) to show whether there is coverage in the corresponding cells 302 of the cornea / pupil overlay map 300. Based on determining that one of the previously used keyframes includes content depicting a given portion of the eye, the associated cell within the cornea / pupil overlay map 300 is filled to indicate that the portion of the eye has been covered. Thus, generating and / or updating the cornea / pupil overlay map 300 may involve filling one or more cells of the two-dimensional grid representing the cornea / pupil of the user. In some examples, the eye (re)constructor 208 assigns a binary value (e.g., 1) to the corresponding cell. In some examples, the eye (re)constructor 208 visually changes the corresponding cell (e.g., by shading the cell in a manner that indicates cell coverage). As Figure 3As shown, the cells 304 are visually changed (eg, shaded) to show the location of the flashes detected by one or more sensors 104a, 104b. Alternatively, the same data may be represented by binary ones and zeros, as shown in Table 1 below:

[0030] 0 0 0 0 0 0 1 0 0 0 1 1 1 0 0 0 1 0 0 0 0 0 0 0 0

[0031] Table 1

[0032] Although Figure 3 The coverage of cell 304 is shown in black shading, while the remaining cells 302 are shaded white, but other visual representations are possible (inverted Figure 3 In some implementations, the cell value quantifies the level, quality, or amount of coverage, e.g., 0 indicates no coverage, 1 indicates limited or low-quality coverage, and 2 indicates high-quality coverage. Differences in coverage quality can be evaluated using one or more criteria, such as the location of the corresponding portion of the eye in the keyframe providing the coverage (e.g., image center or image perimeter) or the angle of the portion of the eye relative to the image capture direction (e.g., orthogonal, offset at a 45-degree angle, etc.). As another example, differences in coverage quality of different portions of the eye can be evaluated using statistical techniques (e.g., based on confidence values ​​associated with portions of the image data corresponding to respective portions of the eye).

[0033] As described herein, the cornea / pupil overlay 300 can be updated by changing a binary value or changing a visual representation of a cell. In some examples, the cornea / pupil overlay 300 is updated based on detecting that a new image / frame provides information that is different or inconsistent by a threshold amount from information generated from one or more previous keyframes.

[0034] In some implementations, the overlay map is updated over time as additional keyframes are selected and used to generate a multidimensional model of the eye. Each additional keyframe identified can be used to update the model, and over time, the model can improve with respect to the extent to which it "covers" the eye (i.e., represents the entirety of the eye). The overlay map provides a way to represent or quantify the extent to which various parts of the eye are currently represented in the model. Therefore, as the model is updated and its coverage improves as more and more keyframes are selected and used, the associated overlay map can also be updated to reflect the improvement in the coverage of the 3D model. The overlay map can therefore provide a representation / quantification of the extent to which various parts of the eye are currently represented in the 3D model, which can be used to select between keyframes from new candidate keyframes. New frames that depict portions of the eye that are not covered (or poorly covered) as shown in the overlay map may be more suitable for use as keyframes because those frames provide more "new" or otherwise more beneficial information in improving the model.

[0035] In some implementations, updating of the multidimensional model of the eye ends when one or more criteria are met, for example, once the coverage map is substantially complete (e.g., coverage is at least complete beyond a given threshold, such as 97%, 98%, 99%, 100%, etc.) or is complete with a coverage value indicating that the existing keyframes depict substantially all portions of the eye with at least a threshold quality level (e.g., a coverage value of at least X in the coverage map using quality values).

[0036] Figure 4 4 is a flowchart illustrating an exemplary method 400. In some examples, such as device 100 ( Figure 1 ) and / or eye modeler 200 ( Figure 2 ) performs the techniques of method 400 to track eye features and generate an eye model for the user. In some examples, method 400 is performed on processing logic (including hardware, firmware, software, or a combination thereof). In some specific implementations, method 400 is performed on a processor that executes code stored in a non-transitory computer-readable medium (e.g., memory).

[0037] At block 402, one or more illuminators (e.g., Figure 1 One or more illuminators 106a, 106b) direct light toward the user's eyes. The light may correspond to light of a particular wavelength, such as near IR light. At block 404, one or more sensors (e.g., Figure 1 One or more sensors 104a, 104b of the camera may obtain images or frames including depictions of reflections of light through the eye. The images or frames of the eye may each include a matrix of pixel values ​​representing light detected by the pixels of the image sensor of the camera. The images may correspond to light reflected from the surface of the eye. The images may correspond to light generated by an illuminator that is reflected from the surface of the eye. The images may correspond to light of a particular wavelength, for example, near IR light.

[0038] At block 406, an eye modeler (e.g., eye modeler 200) may track eye features based on the images or frames (e.g., via eye tracker 204) and update at least a three-dimensional model of the eye (e.g., via eye (re)constructor 208). Eye feature tracking may involve determining a gaze direction of the eye based on the images or frames of the eye at a capture time, and / or may involve determining a pupil position of a first eye based on an image of the first eye at a capture time. The pupil position may be determined in a 3D coordinate system or relative to a second eye. In some examples, eye feature tracking may be based on a current iteration of the eye model. As described in reference Figure 2 and Figure 3As discussed, the eye (re)constructor 208 may create a cornea / pupil overlay 300 based on pixels of an image or frame of the eye containing glints. Similarly, the eye (re)constructor 208 may create and update at least a three-dimensional model of the eye using various data received via the images or frames of the eye. In some examples, the tracking and updating is based on at least some common data from the images or frames. In some examples, the tracking and updating occurs after obtaining (e.g., multiple (n) images or frames 202a1-202a n During a time period between a first image or frame (in) and obtaining a second image or frame. In some examples, eye feature tracking and eye model updating occur with minimal user input, such that tracking and updating occur in the background and are unobtrusive to the user. In this way, the user can benefit from eye registration without having to perform dedicated processing prior to using the device.

[0039] Figure 5 is a block diagram of an electronic device 500. Device 500 illustrates an exemplary device configuration for device 100 and / or eye modeler 200 to track eye features and update at least a three-dimensional model of the eye. Although certain specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the specific implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, the device 500 includes one or more processors 502 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 504, one or more communication interfaces 506 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 508, one or more output devices 510, one or more internal-facing and / or external-facing image sensor systems 512, memory 514, and one or more communication buses 416 for interconnecting these components and various other components.

[0040] In some implementations, the one or more communication buses 516 include circuits that interconnect and control communication between system components. In some implementations, the one or more I / O devices and sensors 504 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0041] In some implementations, the one or more output devices 510 include one or more displays configured to present a view of a 3D environment to a user. In some implementations, the one or more output devices 510 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS) and / or similar display types. In some implementations, the one or more output devices 510 correspond to diffraction, reflection, polarization, holographic, etc. waveguide displays. In one example, the device 500 includes a single display. As another example, the device 500 includes a display for each eye of the user.

[0042] In some embodiments, one or more output devices 510 include one or more audio generating devices. In some embodiments, one or more output devices 510 include one or more speakers, surround sound speakers, speaker arrays, or headphones for generating spatialized sound (e.g., 3D audio effects). Such devices can virtually place sound sources in a 3D environment, including behind, above, or below one or more listeners. Generating spatialized sound can involve converting sound waves (e.g., using head-related transfer functions (HRTFs), reverberation, or cancellation techniques) to simulate natural sound waves (including reflections from walls and floors) that are emitted from one or more points in a 3D environment. Spatialized sound can induce the listener's brain to interpret the sound as if it occurs at one or more points in a 3D environment (e.g., from one or more specific sound sources), even though the actual sound may be generated by speakers in other locations. One or more output devices 510 may be configured to generate tactile sensations in addition or alternatively.

[0043] In some implementations, the one or more image sensor systems 512 are configured to obtain image data corresponding to at least a portion of the physical environment. For example, the one or more image sensor systems 512 may include one or more RGB cameras (e.g., with complementary metal oxide semiconductor (CMOS) image sensors or charge coupled device (CCD) image sensors), monochrome cameras, IR cameras, depth cameras, event-based cameras, etc. In various implementations, the one or more image sensor systems 512 also include an illumination source that emits light, such as a flash. In various implementations, the one or more image sensor systems 512 also include an on-camera image signal processor (ISP) that is configured to perform multiple processing operations on the image data.

[0044] The memory 514 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices. In some specific implementations, the memory 514 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 514 optionally includes one or more storage devices remotely located from the one or more processors 502. The memory 514 includes a non-transitory computer-readable storage medium.

[0045] In some implementations, the memory 514 or a non-transitory computer-readable storage medium of the memory 514 stores an optional operating system 518 and one or more instruction sets 520. The operating system 518 includes procedures for handling various basic system services and for performing hardware-related tasks. In some implementations, the instruction set 520 includes executable software defined by binary information stored in the form of an electric charge. In some implementations, the instruction set 520 is software that can be executed by one or more processors 502 to implement one or more of the techniques described herein. The instruction set 520 includes a key frame selector 522, which is configured to select an image or frame subset including eye glint or pupil occupancy that meets a threshold from a plurality of images or frames when executed, as described herein. The instruction set 520 includes an eye (re)constructor 524, which is configured to generate and / or update an eye model and a cornea / pupil overlay map when executed, as described herein. The set of instructions 520 includes an eye tracker 526 configured to track eye features in real time or near real time when executed, as described herein. The memory 514 may also include a database 528 configured to securely store eye models and / or cornea / pupil overlays as described herein.

[0046] Although the instruction set 520 is shown as residing on a single device, it should be understood that in other implementations, any combination of elements may be located in separate computing devices. Figure 5 More so as to serve as a functional description of various features present in a particular implementation that are different from the structural schematics of the implementations described herein. As will be appreciated by one of ordinary skill in the art, items shown separately may be combined, and some items may be separated. The actual number of instruction sets and how the features are distributed among them will vary depending on the implementation, and may depend in part on the specific combination of hardware, software, and / or firmware selected for a particular implementation.

[0047] It should be understood that the specific implementations described above are cited by way of example, and the present disclosure is not limited to what has been particularly shown and described above. On the contrary, the scope includes both combinations and sub-combinations of the various features described above, as well as variations and modifications of the various features that would occur to a person skilled in the art upon reading the foregoing description and that are not disclosed in the prior art.

[0048] As described above, one aspect of the present technology is a method that involves using one or more light sources to direct light toward a user's eye. In some examples, the method involves obtaining an image frame via one or more sensors, the image frame including a depiction of reflection of light via the eye. In some examples, the method involves tracking eye features and updating at least a three-dimensional eye model of the eye, wherein the tracking and updating are based on at least some common data from the image frames and occur during a time period between obtaining a first one of the image frames and obtaining a second one of the image frames.

[0049] Another aspect of the present technology is a system that includes a non-transitory computer-readable storage medium and one or more processors that communicate with the non-transitory computer-readable storage medium and store program instructions that, when executed on the one or more processors, cause the system to use one or more light sources to direct light toward a user's eye. In some examples, the system involves obtaining an image frame via one or more sensors, the image frame including a depiction of reflection of light via the eye. In some examples, the system involves tracking eye features and updating at least a three-dimensional eye model of the eye, wherein the tracking and updating are based on at least some common data from the image frames and occur during a time period between obtaining a first one of the image frames and obtaining a second one of the image frames.

[0050] Another aspect of the present technology is a non-transitory computer-readable storage medium storing program instructions executable via one or more processors to perform operations of directing light toward an eye of a user using one or more light sources. In some examples, the non-transitory computer-readable storage medium involves obtaining an image frame via one or more sensors, the image frame including a depiction of reflection of light via an eye. In some examples, the non-transitory computer-readable storage medium involves tracking eye features and updating at least a three-dimensional eye model of the eye, wherein the tracking and updating are based on at least some common data from the image frames and occur during a time period between obtaining a first one of the image frames and obtaining a second one of the image frames.

[0051] In some examples, the methods, systems, and / or non-transitory computer-readable storage media involve causing display of a stimulus to attract a user's eyes toward the stimulus.

[0052] In some examples, the methods, systems, and / or non-transitory computer-readable storage media involve estimating a visual axis of an eye based on display of a stimulus.

[0053] In some examples, the methods, systems, and / or non-transitory computer-readable storage media involve determining an eye overlay based on at least a three-dimensional eye model.

[0054] In some examples, the method, system and / or non-transitory computer-readable storage medium involve comparing the eye overlay map to corresponding eye glints or corresponding pupil occupancies, and determining a second subset of image frames associated with eye glints or pupil occupancies that satisfy a threshold associated with the eye overlay map.

[0055] In some examples, there are one or more other image frames between the first image frame and the second image frame.

[0056] In some examples, the common data includes a first subset of image frames having data that satisfies a threshold eye data amount.

[0057] In some examples, the image frame includes a sequence of image frames.

[0058] In some examples, tracking eye features includes tracking corresponding eye blinks or corresponding pupil occupancy from the image frames.

[0059] In some examples, tracking eye features and updating at least a three-dimensional eye model occurs with minimal user input.

[0060] In some examples, obtaining the image frame includes obtaining images from a plurality of infrared cameras.

[0061] In some examples, obtaining the image frame includes obtaining an image from an infrared camera and an image reflected from a mirror.

[0062] Numerous specific details are set forth herein to provide a comprehensive understanding of the claimed subject matter. However, one skilled in the art will appreciate that the claimed subject matter may be practiced without these specific details. In other instances, methods, devices, or systems known to those of ordinary skill are not described in detail so as not to obscure the claimed subject matter.

[0063] Unless otherwise specifically noted, it should be understood that throughout the specification, discussions utilizing terms such as "process," "compute," "calculate," "determine," and "identify" refer to the actions or processes of a computing device, such as one or more computers or similar electronic computing devices, that manipulate or transform data represented as physical electronic or magnetic quantities within a memory, register, or other information storage device, transmission device, or display device of a computing platform.

[0064] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems that access stored software that programs or configures the computing system from a general-purpose computing device to a dedicated computing device that implements one or more specific implementations of the subject matter of the present invention. Any suitable programming, scripting, or other type of language or combination of languages ​​may be used to implement the teachings contained herein in software for programming or configuring a computing device.

[0065] The specific implementation of the method disclosed herein can be performed in the operation of such a computing device. The order of the boxes presented in the above examples can be changed, for example, the boxes can be reordered, combined and / or divided into sub-blocks. Some boxes or processes can be executed in parallel.

[0066] The use of "suitable for" or "configured to" herein is meant to be open and inclusive language that does not exclude devices that are suitable for or configured to perform additional tasks or steps. Additionally, the use of "based on" is meant to be open and inclusive, as a process, step, calculation, or other action "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbering included herein are for ease of explanation only and are not intended to be limiting.

[0067] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements herein, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0068] The terms used herein are only for the purpose of describing a specific implementation and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "one", "an" and "the" are intended to also cover the plural forms unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their groupings.

[0069] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

[0070] The foregoing description and summary of the present invention should be understood to be illustrative and exemplary in every aspect, rather than restrictive, and the scope of the present invention disclosed herein is determined not only by the detailed description of the exemplary specific implementations, but by the full breadth allowed by the patent law. It should be understood that the specific implementations shown and described herein are only illustrative of the principles of the present invention, and that various modifications can be implemented by those skilled in the art without departing from the scope and essence of the present invention.

Claims

1. A method comprising: Using one or more light sources to direct light toward the eyes; obtaining, via one or more sensors, an image frame, the image frame comprising a depiction of reflection of the light via the eye; as well as Based on the image frames, eye features are tracked and at least a three-dimensional eye model of the eye is updated, wherein the tracking and the updating are based on at least some common data from the image frames and occur during a time period between obtaining a first one of the image frames and obtaining a second one of the image frames. 2 . The method of claim 1 , wherein there is an image frame between the first image frame and the second image frame. 3 . The method of claim 1 , wherein the common data comprises a first subset of the image frames having data that satisfies a threshold eye data amount.

4. The method according to any one of claims 1 to 3, wherein the image frame comprises a sequence of image frames.

5. A method according to any one of claims 1 to 4, further comprising causing display of a stimulus to attract the eye towards the stimulus.

6. The method of claim 5, further comprising estimating a visual axis of the eye based on the display of the stimulus.

7. The method according to any one of claims 1 to 6, wherein tracking the eye features comprises tracking corresponding eye glints or corresponding pupil occupancy from the image frames.

8. The method of claim 7, further comprising determining an eye coverage map based on the at least three-dimensional eye model.

9. The method according to claim 8, further comprising: comparing the eye coverage map to the corresponding eye glint or the corresponding pupil occupancy; as well as A second subset of image frames associated with eye glint or pupil occupancy that satisfies a threshold associated with the eye overlay map is determined.

10. A method according to any one of claims 1 to 9, wherein tracking the eye features and updating the at least three-dimensional eye model occurs with minimal user input.

11. The method of any one of claims 1 to 10, wherein obtaining the image frame comprises obtaining images from a plurality of infrared cameras.

12. The method of any one of claims 1 to 10, wherein obtaining the image frame comprises obtaining an image from an infrared camera and an image reflected from a mirror.

13. A system comprising: a non-transitory computer-readable storage medium; and one or more processors in communication with the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the system to perform operations including: Using one or more light sources to direct light toward the eyes; obtaining, via one or more sensors, an image frame, the image frame comprising a depiction of reflection of the light via the eye; as well as Based on the image frames, eye features are tracked and at least a three-dimensional eye model of the eye is updated, wherein the tracking and the updating are based on at least some common data from the image frames and occur during a time period between obtaining a first one of the image frames and obtaining a second one of the image frames.

14. The system of claim 13, wherein there is an image frame between the first image frame and the second image frame.

15. The system of claim 13, wherein the common data comprises a first subset of the image frames having data that satisfies a threshold eye data amount.

16. The system of any one of claims 13 to 15, wherein the image frame comprises a sequence of image frames.

17. The system of any one of claims 13 to 16, wherein the instructions, when executed on the one or more processors, cause the system to perform operations comprising: Display of a stimulus is caused to attract the eye toward the stimulus.

18. The system of claim 17, wherein the instructions, when executed on the one or more processors, cause the system to perform operations comprising: A visual axis of the eye is estimated based on the display of the stimulus.

19. The system of any one of claims 13 to 18, wherein tracking the eye features comprises tracking corresponding eye glints or corresponding pupil occupancy from the image frames.

20. A non-transitory computer-readable storage medium storing program instructions executable via one or more processors to perform operations comprising: Using one or more light sources to direct light toward the eyes; obtaining, via one or more sensors, an image frame, the image frame comprising a depiction of reflection of the light via the eye; as well as Based on the image frames, eye features are tracked and at least a three-dimensional eye model of the eye is updated, wherein the tracking and the updating are based on at least some common data from the image frames and occur during a time period between obtaining a first one of the image frames and obtaining a second one of the image frames.

Images