Retinal reflection tracking system

Abstract

Various implementations disclosed herein include electronic devices, systems, and methods that determine a characteristic of an eye based on detecting reflections of light. An example electronic device may include a light source, a set of one or more sensors, a transparent substrate, and a display area. The light source and sensors maybe distributed within the display area or distributed within an area proximate to a corner of the display area. When the electronic device is worn, the near-eye side is proximate an eye of the user and the far-eye side is an opposite side of the transparent substrate such that light projected by the light source that is reflected by the eye passes through the display area of the transparent substrate before capture by the sensors. A processor may be configured to receive sensor data and determine a characteristic of the eye based on the sensor data.

Description

Attorney Docket No. 097425-01473(P67428WO1 )RETINAL REFLECTION TRACKING SYSTEMTECHNICAL FIELD

[0001] The present disclosure generally relates to electronic devices, and in particular, to systems, methods, and devices for determining eye characteristics of users of electronic devices.BACKGROUND

[0002] Existing eye-tracking techniques analyze glints that are reflected off of a user’s eye and captured via an image sensor (e.g., a camera). Some head mounted systems may include eye-tracking techniques that project light via a light source and capture reflections of the light off an eye via a set of sensors that are located in front of the lens. The eyetracking system may lack accuracy, require more than one camera to capture a sufficient number of glints, and require eye camera placement that is suboptimal for capturing a sufficient number of glints. Thus, it may be desirable to provide a means of efficiently positioning of light sources to produce glints for assessing an eye characteristic (e.g., gaze direction, eye orientation, identifying an iris of the eye, etc.) for head mountable systems and not using one or more cameras.SUMMARY

[0003] Various implementations disclosed herein include devices, systems, and methods that focuses on a device that tracks an eye by projecting light via a light source and capturing reflections of the light off an eye via a set of one or more sensors (e.g., photodiodes), thus does not require a dedicated camera to capture the reflections. The camera- less eye tracking system may be used for inside refractive or diffractive media, for example behind the cover glass or lens, behind an optical module, or any other refractive and / or diffractive material. The device may be a head mounted device (HMD) such as a device that is worn around the head of a user. In some implementations, the HMD may be smart glasses, or may be a pair of augmented reality glasses. Tracking the eye may include identifying and tracking a position and / or orientation of an eye, a gaze direction, the cornea shape, and the like.

[0004] In some implementations, different light source and sensor configurations for the device (e.g., HMD, AR glasses) may be implemented for tracking an eye (e.g., retina, position / orientation, gaze direction, cornea shape, etc.). An eye may be tracked byAttorney Docket No. 097425-01473(P67428WO1 ) projecting light via a light source (e.g., a light emitting diode (LED)) and capturing reflections of the light off an eye via a sensor (e.g. , a set of photodiodes) that are positioned in different configurations with respect to a display area and behind a transparent substrate (e.g., on the display side of a lens that is positioned between an eye and a display) and accounting for the effect(s) of the transparent substrate on the reflections.

[0005] In some implementations, a sensor and a corresponding light source may be positioned within, or with respect to, the display area in different configurations. In some implementations, one or more sensors and / or one or more light sources may be uniformly distributed on a display area mapping to a display field of view (FoV). For example, use an existing pancake shape, but modify films and coating to be functional in both visible and infrared range. Additionally, or alternatively, in some implementations, in addition to or in lieu of modifying films, a diffractive layer (e.g., a meta optical elements (MOE) layer, a surface relief grating (SRG) layer, a volume phase holographic grating (VPH) layer, and the like) maybe used to correct focus and / or beam split through a lens (e.g., a pancake lens). For example, modifying the films may be avoided by applying a layer that reflects or transmits the light in a manner that focuses it on the panel, e.g., performs the beamsplitter function for near infrared (NIR) or adds power to transmit the image through the pancake lens.

[0006] Additionally, or alternatively, in some implementations, one or more sensors and / or one or more light sources may be positioned on or nearthe four corners of the display area (e.g., outside a display FoV). The four-corner configuration of the light sources and sensors may be configured as either i) using an IR mirror to keep similar pancake outline dimension, or ii) using four corner sensors image stitching to get a wider eye box. In some implementations, the IR mirror, a diffractive optical element (DOE), and / or grating may be powered or unpowered (e.g., correcting focus through the pancake lens or just reflecting I deflecting), and may be reflection or transmission (e.g., taking place of beamsplitter for NIR or as a standalone layer that deflects the beam to the corners of the lens).

[0007] In some implementations, the one or more sensors and / or one or more light sources are placed on a top side of the transparent substrate, may be placed on a bottom side of the cover glass, integrated in the display panel or cover glass, or a combination thereof. Some implementations focus on an enrollment-to-live correlation process that matches features of an undistorted retina map (e.g., at enrollment) and a distorted retina map (e.g., real time capture) to estimate current pupil coordinates.Attorney Docket No. 097425-01473(P67428WO1 )

[0008] In some implementations, additional vision correction optics may be worn by the user while wearing an HMD, and the corrected vision from the additional vision correction optics is accounted for by the systems described herein. These additional vision correction optics are not only necessary so that the user will sharply seethe display on the HMD and / or the external environment but are also necessary to obtain a sharp retinal image. For example, the systems described herein may compensate for the optical aberrations that the user’s eye may have. In some implementations, the optics are placed between the HMD and the user’s eyes, and may be either passive lenses (e.g., designated clip-on lenses, personal vision glasses, and the like), active lenses (e.g., a tunable lens), or contact lenses.

[0009] In general, one innovative aspect of the subject matter described in this specification can be embodied in an electronic device that includes a light source capable of projecting light, a set of one or more sensors, a transparent substrate having a near-eye side, a far-eye side, and a display area, wherein the light source and each sensor are distributed within the display area, and a processor configured to perform operations. When the electronic device is worn, the near-eye side is proximate an eye of the user and the far- eye side is an opposite side of the transparent substrate such that light projected by the light source that is reflected by the eye passes through at least a portion of the display area of the transparent substrate before capture by the set of one or more sensors. The operations include receiving sensor data from the set of one or more sensors, the sensor data corresponding to a plurality of reflections of light produced by the light source and reflected from the eye, determining a characteristic of the eye based on the sensor data.

[0010] In general, one innovative aspect of the subject matter described in this specification can be embodied in methods, at an electronic device having a processor, that include the actions of producing light from a light source capable of projecting light in a plurality of different directions over time toward an eye, wherein the light source and each sensor of a set of one or more sensors are distributed within a display area associated with a transparent substrate, and wherein the transparent substrate comprises a near-eye side and a far-eye side such that when the electronic device is worn, the near-eye side is proximate an eye of the user and the far-eye side is an opposite side of the transparent substrate. The actions may further include receiving sensor data from the set of one or more sensors, the sensor data corresponding to a plurality of reflections of light produced by the light source and reflected from the eye, wherein the light that is reflected by the eye passes through at least a portion of the display area of the transparent substrate before capture by the set of one or more sensors. The actions may further include determining a characteristic of the eye based on the sensor data.Attorney Docket No. 097425-01473(P67428WO1 )

[0011] These and other embodiments may each optionally include one or more of the following features.

[0012] In some aspects, the light source is configured to project light towards the eye through the transparent substrate. In some aspects, the light source and the set of the one or more sensors are positioned proximate the far-eye side of the transparent substrate. In some aspects, the light source and the set of the one or more sensors are positioned proximate to the near-eye side of the transparent substrate. In some aspects, the light source and the set of the one or more sensors are positioned within the transparent substrate.

[0013] In some aspects, the light source is one of a plurality of light sources coupled to the electronic device. In some aspects, the light source comprises a micro light emitting diode (LED). In some aspects, the light source comprises a near infrared (NIR) emitter.

[0014] In some aspects, the light source is a first light source of a set of one or more light sources, wherein the first light source and a first sensor of the set of the one or more sensors are positioned within a pixel array of the display area.

[0015] In some aspects, the display area comprises a plurality of pixel arrays, each pixel array comprising a light source of the set of one or more light sources and a sensor of the set of the one or more sensors. In some aspects, each pixel array that includes a corresponding pair of a light source and a sensor are approximately evenly distributed within a field of view of the display area. In some aspects, each corresponding pair of a light source and a sensor are approximately equidistant from an adjacent light source and corresponding sensor.

[0016] In some aspects, the processor is further configured to perform the actions of determining a distorted retina map of the eye based on the sensor data, and adjusting at least a portion of the display area of the transparent substrate based on the distorted retina map. In some aspects, the processor is further configured to perform the actions of determining a distorted retina map of the eye based on the sensor data, matching features of the distorted retina map with features of an undistorted retina map of the eye obtained during an enrollment process, and adjusting at least a portion of the display area of the transparent substrate based on the matched features of the distorted retina map and the undistorted retina map.

[0017] In some aspects, determining the characteristic of the eye based on the sensor data comprises determining a position of a pupil of the eye. In some aspects, determiningAttorney Docket No. 097425-01473(P67428WO1 ) the position of the pupil of the eye includes obtaining an undistorted retina map of the eye obtained during an enrollment process, determining a distorted retina map of the eye based on the sensor data, and matching features of the undistorted retina map with features of the distorted retina map. In some aspects, determining the characteristic of the eye based on the sensor data comprises determining a gaze direction based on a detected reflection angle.

[0018] In some aspects, the processor is further configured to perform the actions of initiating an action based on detecting that the gaze direction is approximately oriented towards a target area.

[0019] In general, one innovative aspect of the subject matter described in this specification can be embodied in an electronic device that includes a set of light sources capable of projecting light, a set of one or more sensors, where each light source is associated with a corresponding sensor, a transparent substrate having a near-eye side, a far-eye side, and a display area, and a processor configured to perform operations. Each light source and a corresponding sensor are distributed within an area proximate to a corresponding corner of the display area. When the electronic device is worn, the near-eye side is proximate an eye of the user and the far-eye side is an opposite side of the transparent substrate such that light projected by each light source that is reflected by the eye passes through at least a portion of the display area of the transparent substrate before capture by the set of one or more sensors. The operations include receiving sensor data from the set of one or more sensors, the sensor data corresponding to a plurality of reflections of light produced by the light source and reflected from the eye, and determining a characteristic of the eye based on the sensor data.

[0020] In general, one innovative aspect of the subject matter described in this specification can be embodied in methods, at an electronic device having a processor, that include the actions of producing light from a set of light sources, wherein each light source is capable of projecting light in a plurality of different directions over time toward an eye, wherein each light source of the set of light sources and each sensor of a set of one or more sensors are distributed within an area proximate to a corresponding corner of a display area associated with a transparent substrate, and wherein the transparent substrate comprises a near-eye side and a far-eye side such that when the electronic device is worn, the near- eye side is proximate an eye of the user and the far-eye side is an opposite side of the transparent substrate. The actions may further include receiving sensor data from the set of one or more sensors, the sensor data corresponding to a plurality of reflections of light produced by each light source and reflected from the eye, wherein the light that is reflectedAttorney Docket No. 097425-01473(P67428WO1 ) by the eye passes through at least a portion of the display area of the transparent substrate before capture by the set of one or more sensors. The actions may further include determining a characteristic of the eye based on the sensor data.

[0021] These and other embodiments may each optionally include one or more of the following features.

[0022] In some aspects, each light source is configured to project light towards the eye through the transparent substrate. In some aspects, each light source and a corresponding sensor are positioned proximate the far-eye side of the transparent substrate. In some aspects, each light source and a corresponding sensor are positioned proximate to the neareye side of the transparent substrate. In some aspects, each light source and a corresponding sensor are positioned within the transparent substrate.

[0023] In some aspects, each light source is one of a plurality of light sources coupled to the electronic device. In some aspects, each light source comprises a micro light emitting diode (LED). In some aspects, each light source comprises a near infrared (NIR) emitter.

[0024] In some aspects, the display area comprises four corners, and wherein each light source and the corresponding sensor are positioned at the area proximate to a corresponding corner of the four corners of the display area.

[0025] In some aspects, the light projected by each light source and the light that is reflected by the eye before capture by each sensor extends outside a field of view of the display area. In some aspects, determining a characteristic of the eye based on the sensor data is based on image stitching.

[0026] In some aspects, the device further includes one or more mirrors, wherein the light projected by each light source and the light that is reflected by the eye before capture by each sensor is reflected by at least one of the one or more mirrors.

[0027] In some aspects, the processor is further configured to perform the actions of including determining a distorted retina map of the eye based on the sensor data, and adjusting at least a portion of the display area of the transparent substrate based on the distorted retina map. In some aspects, the processor is further configured to perform the actions of determining a distorted retina map of the eye based on the sensor data, matching features of the distorted retina map with features of an undistorted retina map of the eye obtained during an enrollment process, and adjusting at least a portion of the display area of the transparent substrate based on the matched features of the distorted retina map and the undistorted retina map.Attorney Docket No. 097425-01473(P67428WO1 )

[0028] In some aspects, determining the characteristic of the eye based on the sensor data comprises determining a position of a pupil of the eye. In some aspects, determining the position of the pupil of the eye includes obtaining an undistorted retina map of the eye obtained during an enrollment process, determining a distorted retina map of the eye based on the sensor data, and matching features of the undistorted retina map with features of the distorted retina map.

[0029] In some aspects, determining the characteristic of the eye based on the sensor data comprises determining a gaze direction based on a detected reflection angle.

[0030] In some aspects, the processor is further configured to perform the actions of initiating an action based on detecting that the gaze direction is approximately oriented towards a target area. In some aspects, determining the characteristic of the eye based on the sensor data comprises determining a shape of the eye.

[0031] In some aspects, the electronic device is a head-mounted device (HMD).

[0032] In accordance with some implementations, a non-transitory computer readable storage medium has stored therein instructions that are computer-executable to perform or cause performance of any of the methods described herein. In accordance with some implementations, a device includes one or more processors, a non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors and the one or more programs include instructions for performing or causing performance of any of the methods described herein.BRIEF DESCRIPTION OF THE DRAWINGS

[0033] So that the present disclosure can be understood by those of ordinary skill in the art, a more detailed description may be had by reference to aspects of some illustrative implementations, some of which are shown in the accompanying drawings.

[0034] Figure 1 illustrates an environment in which extended reality (XR) content is provided to one or more users wearing head mounted displays (HMDs) in accordance with some implementations.

[0035] Figure 2 illustrates an example eye-tracking system using a set of light sources and a corresponding set of sensors that are distributed within a display area in accordance with some implementations.Attorney Docket No. 097425-01473(P67428WO1 )

[0036] Figures 3A and 3B illustrate an example eye-tracking system using a set of light sources and a corresponding set of sensors that are distributed within an area proximate to a corresponding corner of a display area in accordance with some implementations.

[0037] Figure 4A illustrates an example of a user wearing an HMD in accordance with some implementations.

[0038] Figures 4B and 4C illustrate an example view of a refractive / diffractive medium of the HMD of Figure 4A that include different spatial arrangements of light sources and corresponding sensors in accordance with some implementations.

[0039] Figures 5A-5D illustrate different spatial arrangements of light sources and corresponding sensors with respect to the refractive / diffractive medium of an HMD in accordance with some implementations.

[0040] Figure 6 illustrates an example of image stitching of sensor data for an eye box of an HMD in accordance with some implementations.

[0041] Figure 7 illustrates an example of estimating pupil coordinates of a user wearing an HMD in accordance with some implementations.

[0042] Figure 8 is a flowchart representation of a method for assessing an eye characteristic based on reflected light from a light source received at one or more sensors that are distributed within a display area in accordance with some implementations.

[0043] Figure 9 is a flowchart representation of a method for assessing an eye characteristic based on reflected light from a set of light sources received at one or more sensors that are distributed within an area proximate to a corresponding corner of a display area in accordance with some implementations.

[0044] Figure 10 illustrates device components of an exemplary device in accordance with some implementations.

[0045] Figure 11 illustrates an example HMD in accordance with some implementations.

[0046] In accordance with common practice the various features illustrated in the drawings may not be drawn to scale. Accordingly, the dimensions of the various features may be arbitrarily expanded or reduced for clarity. In addition, some of the drawings may not depict all of the components of a given system, method or device. Finally, like reference numerals may be used to denote like features throughout the specification and figures.Attorney Docket No. 097425-01473(P67428WO1 )DESCRIPTION

[0047] Numerous details are described in order to provide a thorough understanding of the example implementations shown in the drawings. However, the drawings merely show some example aspects of the present disclosure and are therefore not to be considered limiting. Those of ordinary skill in the art will appreciate that other effective aspects or variants do not include all of the specific details described herein. Moreover, well-known systems, methods, components, devices and circuits have not been described in exhaustive detail so as not to obscure more pertinent aspects of the example implementations described herein.

[0048] Figure 1 illustrates a real-world physical environment 100 including a first user 110 wearing a first device 105, a second user 130 wearing a second device 125, a third user 160 wearing a third device 165, a wall-hung picture 185, a plant 175, and a door 150. In some implementations, one or more of the devices 105, 125, 165 is configured to provide content based on one or more sensors on the respective devices or to share information and / or sensor data with one another. In some implementations, one or more of the devices 105, 125, 165 provide content that provides augmentations in XR using sensor data. The sensor data may be used to understand that a user’s state is associated with providing user assistance, e.g., a user’s appearance or behavior or an understanding of the environment may be used to recognize a need or desire for assistance.

[0049] In the example of Figure 1 , the first device 105 includes one or more sensors 116 that capture light-intensity images, depth sensor images, audio data or other information about the user 1 10 and the physical environment 100. For example, the one or more sensors 116 may capture images of the user’s forehead, eyebrows, eyes, eye lids, cheeks, nose, lips, chin, face, head, hands, wrists, arms, shoulders, torso, legs, or other body portion. Sensor data about a user’s eye 1 11 , as one example, may be indicative of various user characteristics, e.g., the user’s gaze direction 1 19 over time, user saccadic behavior over time, user eye dilation behavior over time, etc. The one or more sensors 1 16 may capture audio information including the user’s speech and other user-made sounds as well as sounds within the physical environment 100.

[0050] One or more sensors, such as one or more sensors 1 15 on device 105, may identify user information based on proximity or contact with a portion of the user 1 10. As example, the one or more sensors 115 may capture sensor data that may provide biological information relating to a user’s cardiovascular state (e.g., pulse), body temperature, breathing rate, etc.Attorney Docket No. 097425-01473(P67428WO1 )

[0051] The one or more sensors 116 or the one or more sensors 115 may capture data from which a user orientation 121 within the physical environment can be determined. In this example, the user orientation 121 corresponds to a direction that a torso of the user 110 is facing.

[0052] In some implementations, the content provided by the device 105 and sensor features of device 105 may be provided using components, sensors, or software modules that are sufficiently small in size and efficient with respect to power consumption and usage to fit and otherwise be used in lightweight, battery-powered, wearable products such as wireless ear buds or other ear-mounted devices or head mounted devices (HMDs) such as smart / augmented reality (AR) glasses. Features can be facilitated using a combination of multiple devices. For example, a smart phone (connected wirelessly and interoperating with wearable device(s)) may provide computational resources, connections to cloud or internet services, location services, etc.

[0053] In some implementations, data is shared amongst a group of devices to improve user state or environment understanding. For example, device 125 may share information (e.g., images, audio, or other sensor data) corresponding to user 110 or the physical environment 100 (including information about user 130 or user 160) with device 105 so that device 105 can better understand user 110 and physical environment 100.

[0054] In some implementations, devices 105, 125, 165 are head mounted devices (HMDs) that present visual or audio content (e.g., extended reality XR content) or have sensors that obtain sensor data (e.g., visual data, sound data, depth data, ambient lighting data, etc.) about the environment 100 or sensor data (e.g., visual data, sound data, depth data, physiological data, etc.) about the users 110, 130, 160. Such information may, subject to user authorizations, permissions, and preferences, be shared amongst the device 105, 125, 165 to enhance the user’s experiences on such devices.

[0055] In some implementations, the devices 105, 125, 165 obtain physiological data (e.g., EEG amplitude / frequency, pupil modulation, eye gaze saccades, etc.) from the users 110, 130, 160 via one or more sensors that are proximate or in contact with the respective user 110, 130, 160. For example, the device 105 may obtain pupillary data (e.g., eye gaze characteristic data) from an inward facing eye tracking sensor. In some implementations, the devices 105, 125, 165 include additional sensors for obtaining image or other sensor data of the physical environment 100.Attorney Docket No. 097425-01473(P67428WO1 )

[0056] In some implementations, the devices 105, 125, 165 are wearable devices such as ear-mounted speaker / microphone devices (e.g., headphones, ear pods, etc.), smart watches, smart bracelets, smart rings, smart / AR glasses, or other head-mounted devices (HMDs). In some implementations, the devices 105, 125, 165 are handheld electronic devices (e.g., smartphones or tablets). In some implementations, the devices 105, 125, 165 are laptop computers or desktop computers. In some implementations, the devices 105, 125, 165 have input devices such as audio command input systems, gesture recognitionbased input systems, touchpads or touch-sensitive displays (also known as a “touch screen” or “touch screen display”). In some implementations, multiple devices are used together to provide various features. For example, a smart phone (connected wirelessly and interoperating with wearable device(s)) may provide computational resources, connections to cloud or internet services, location services, etc.

[0057] Figure 1 illustrates an example in which the devices within the physical environment 100 include HMD devices 105, 125, 165. Numerous other types of devices may be used including mobile devices, tablet devices, wearable devices, hand-held devices, personal assistant devices, Al-assistant-based devices, smart speakers, desktop computing devices, menu devices, cash register devices, vending machine devices, juke box devices, or numerous other devices capable of presenting content, capturing sensor data, or communicating with other devices within a system, e.g., via wireless communication. For example, assistance may be provided to a vision impaired person to help the person understand a menu by providing data from the menu to a device being worn by the vision impaired person, e.g., enabling that device to enhance the user’s understanding of the menu by providing visual annotations, audible cues, etc.

[0058] In some implementations, the devices 105, 125, 165 include eye tracking systems for detecting eye position and eye movements. For example, an eye tracking system may include one or more infrared (IR) light-emitting diodes (LEDs), an eye tracking camera (e.g., near-IR (NIR) camera), and an illumination source (e.g., an NIR light source) that emits light (e.g., NIR light) towards the eyes of the user. Moreover, an illumination source on a device may emit NIR light to illuminate the eyes of the user and the NIR camera may capture images of the eyes of the user. In some implementations, images captured by the eye tracking system may be analyzed to detect position and movements of the eyes of the user, or to detect other information about the eyes such as pupil dilation or pupil diameter. Moreover, the point of gaze estimated from the eye tracking images may enable gaze-based interaction with content shown the device. Additional cameras may be included to capture other areas of the user (e.g., an HMD with a jaw camera to view the user’s mouth,Attorney Docket No. 097425-01473(P67428WO1 ) a down camera to view the body, an eye cam for tissue around the eye, and the like). These cameras and other sensors can detect motion of the body, or signals of the face modulated by the breathing of the user (e.g., remote PPG).

[0059] In some implementations, the devices 105, 125, 165 have graphical user interfaces (GUIs), one or more processors, memory and one or more modules, programs or sets of instructions stored in the memory for performing multiple functions. In some implementations, the users 1 10, 130, 160 may interact with a GUI through voice commands, finger contacts on a touch-sensitive surface, hand / body gestures, remote control devices, or other user input mechanisms. In some implementations, the functions include viewing / listening to content, image editing, drawing, presenting, word processing, website creating, disk authoring, spreadsheet making, game playing, telephoning, video conferencing, e-mailing, instant messaging, workout support, digital photographing, digital videoing, web browsing, digital music playing, or digital video playing. Executable instructions for performing these functions may be included in a computer readable storage medium or other computer program product configured for execution by one or more processors.

[0060] In some implementations, the devices 105, 125, 165 employ various physiological or behavioral sensor, detection, or measurement systems. Detected physiological data may include, but is not limited to, EEG, electrocardiography (ECG), electromyography (EMG), functional near infrared spectroscopy signal (fNIRS), blood pressure, skin conductance, or pupillary response. Detected behavioral data may include, but is not limited to, facial gestures, facial expressions, body gestures, or body language based on image data, voice recognition based on acquired audio signals, etc.

[0061] In some implementations, the devices 105, 125, 165 (or other devices) may be communicatively coupled to one or more additional sensors. For example, a sensor (e.g., an EDA sensor) may be communicatively coupled to a device 105, 125, 165 via a wired or wireless connection, and such a sensor may be located on the skin of a user (e.g., on the arm, placed on the hand / fingers of the user, etc.). For example, such a sensor can be utilized for detecting EDA (e.g., skin conductance), heart rate, or other physiological data that utilizes contact with the skin of a user. Moreover, a device 105, 125, 165 (using one or more sensors) may concurrently detect multiple forms of physiological data in order to benefit from synchronous acquisition of physiological data or behavioral data. Moreover, in some implementations, the physiological data or behavioral data represents involuntary data, e.g., responses that are not under conscious control. For example, a pupillary response mayAttorney Docket No. 097425-01473(P67428WO1 ) represent an involuntary movement. In some implementations, a sensor is placed on the skin as part of a watch device, such as a smart watch.

[0062] In some implementations, one or both eyes of a user, including one or both pupils of the user present physiological data in the form of a pupillary response (e.g., eye gaze characteristic data). The pupillary response of the user may result in a varying of the size or diameter of the pupil, via the optic and oculomotor cranial nerve. For example, the pupillary response may include a constriction response (miosis), e.g., a narrowing of the pupil, or a dilation response (mydriasis), e.g., a widening of the pupil. In some implementations, a device may detect patterns of physiological data representing a time- varying pupil diameter. In some implementations, the device may further determine the interpupillary distance (IPD) between a right eye and a left eye of the user.

[0063] The user data (e.g., upper facial feature characteristic data, lower facial feature characteristic data, and eye gaze characteristic data, etc.), including information about the position, location, motion, pose, etc., of the head or body of the user, may vary in time and a device 105, 125, 165 (or other devices) may use the user’s data to track a user state. In some implementations, the user data includes texture data of the facial features such as eyebrow movement, chin movement, nose movement, cheek movement, etc. For example, when a person (e.g., user 110, 130, 160) performs a facial expression or micro expression associated with lack of familiarity or confusion, the upper and lower facial features can include a plethora of muscle movements that are used to assess the state of the user based on the captured data from sensors.

[0064] The physiological data (e.g., eye data, head / body data, etc.) and behavioral data (e.g., voice, facial recognition, etc.) may vary in time and the device may use the physiological data or behavioral data to measure a physiological / behavioral response or the user’s attention to object or intention to perform an action. Such information may be used to identify a state of the user with respect to whether the user needs or desires assistance.

[0065] Information about such assistance predictions and how a user’s own data is used may be provided to a user and the user given the option to opt out of automatic predictions / use of their own data and given the option to manually override assistance features. In some implementations, the system is configured to ensure that users’ privacy is protected by requiring permissions to be granted before user state is assessed or assistance is enabled.

[0066] Figure 2 illustrates an example environment 200 of an example eye-tracking system using a set of light sources and a corresponding set of sensors that are distributed within a display area in accordance with some implementations. In particular, Figure 2Attorney Docket No. 097425-01473(P67428WO1 ) illustrates an imaging system used to image patterns reflected from a retina / fundus that are then imaged by a display lens on to a set of one or more detectors and used to form a retina image. The retina image may then be used to estimate a gaze angle.

[0067] Figure 2 provides a cross sectional view of an eye-tracking system. The eyetracking system of example environment 200 illustrates tracking an eye characteristic of eye 11 1 via a set of light sources 220, 222, 224, and 226 (e.g., a light projection component such as an LED, or the like) and corresponding set of sensors 230, 232, 234, 236 of a device 202 (e.g., an HMD, such as device 105 of Figure 1). The device 202 further includes a lens 210, and a display portion 240. As illustrated in Figure 2, the light sources 220, 222, 224, and 226 and corresponding sensors 230, 232, 234, and 236 are distributed in an area within the display 240.

[0068] In some implementations, the display 240 includes multiple pixel arrays, including, for example, pixel array 250 (e.g., a 5x5 pixel array). As illustrated in the expanded view of the pixel array 250, the pixel arrays throughout the display 240 may include a light source (e.g., light source 220) and a sensor (e.g., sensor 230) embedded in the middle of the array. For example, a light source, such as an NIR emitter, a micro-OLED, and the like, and a sub micro sensor, such as a photodiode, may be embedded in each 5x5 array of pixels. In some implementations, the sensor and light source may be embedded in a larger pixel array (e.g., 10x10, and the like) for covering different (e.g., larger) field of views. For example, eye focal length is approximately 20mm, and some of the smallest feature sizes of a vessel of a retina may be approximately 50pm (e.g., -0.14°), which requires a resolution of approximately 7 pixels per degree (PPD). For example, as illustrated in Figure 2, sensor 236 may obtain an image of the retina (e.g., image 260). The resolution can go down as FoV goes up (e.g., rely on larger blood vessels). The goal is to obtain a retina map of an eye in a user’s angle space (e.g., [0,0] coordinate is a fovea center and each coordinate is relative to the fovea center in degrees). In some implementations, each pair of a light source and a corresponding sensor may be approximately equidistant from an adjacent light source and corresponding sensor (e.g., approximately evenly distributed within the display area).

[0069] In some implementations, as further discussed herein, the light sources 220, 222, 224, and 226 and the sensors 230, 232, 234, and 236 may each be positioned in alternative spatial arrangements. For example, one or more of the light sources 220, 222, 224, and 226 and corresponding sensors 230, 232, 234, 236 may be positioned in front of the lens 210, embedded within the lens 210, around the edge of the lens 210 (e.g., on an edge of a frame), embedded within or around a display portion 240 of the device, embedded on the cover glass of the display panel, or a combination thereof. In some implementations,Attorney Docket No. 097425-01473(P67428WO1 ) the sensors may be transparent photodetectors that are imperceptible to the human eye. For example, the light source and the set of the one or more sensors may be positioned within the transparent substrate. In some implementations, this may be accomplished via a waveguide and may include transparent multispectral photodetectors that imperceptible to the human eye and directly placed onto transparent substrate. In some implementations, an illumination source as a diffractive waveguide may be integrated as a standalone layer or laminated to the coverglass to diffracts light out over the eyebox.

[0070] The eye-tracking system of example environment 200 illustrates a light system to observe light reflections from the eye 1 11 , and in particular, to detect patterns reflected from the retina / fundus. The reflected light may be imaged by the display lens and captured by one or more detectors or photo sensors (e.g., sensors 230, 232, 234, 236, etc.) and used to form a retina image, which may then be used to estimate a gaze angle. For example, as illustrated in Figure 2, the light source 226 produces light that is flashed at an eye 11 1 that is refracted within the lens 210 as illustrated. The light waves are then reflected off of the retina of the eye 1 1 1 and detected by a detector (e.g., sensor 236) which detects the reflected light rays (e.g., reflected light rays 254 from the emitted light rays 252). In one aspect, each light source (e.g., light sources 220, 222, 224, and 226) may be used both for illuminating specular and diffusive parts of an object (e.g., eye 1 11) and thus may provide at least a threshold level of illumination. Providing at least such a threshold level of illumination may result in a light reflection pattern that would be detected in images captured by the detectors (e.g., sensors 230, 232, 234, 236). For example, light rays 252 from a light source would produce the light rays 254 that are imaged by the display lens on the detectors and used to form a retina image.

[0071] In some implementations, a single light source may be used. However, an implementation of multiple light sources (e.g., light sources 220, 222, 224, and 226) and corresponding sensors (e.g., sensors 230, 232, 234, 236) may increase a field of view.

[0072] As illustrated in Figure 2, one or more sensors and / or one or more light sources may be uniformly distributed on a display area mapping to a display FoV. For example, use an existing pancake shape of the lens, but modify films and coating to be functional in both visible and infrared range. Additionally, or alternatively, in some implementations, in addition to or in lieu of modifying films, a diffractive layer (e.g., a meta optical elements (MOE) layer, a surface relief grating (SRG) layer, a volume phase holographic grating (VPH) layer, and the like) maybe used to correct focus and / or beam split through a lens (e.g., a pancake lens). For example, modifying the films may be avoided by applying a layer that reflects orAttorney Docket No. 097425-01473(P67428WO1 ) transmits the light in a manner that focuses it on the panel, e.g., performs the beamsplitter function for NIR or adds power to transmit the image through the pancake lens.

[0073] In some implementations, the eye-tracking system of example environment 200 does not include additional vision correction optics, such as contact lenses or other vision glasses. Alternatively, in some implementations, the eye-tracking system of example environment 200 includes vision correction optics that are positioned between the device (e.g., an HMD) and the user’s eyes. In some implementations, one option for vision correction optics may include lens 204, such as a passive lens (e.g., a designated clip-on lenses, personal vision glasses, and the like) or an active lens (e.g., a tunable lens). Additionally, or alternatively, another option for vision correction optics may include a contact lens 206.

[0074] Figures 3A and 3B illustrate an example eye-tracking system using a set of light sources and a corresponding set of sensors that are distributed within an area proximate to a corresponding corner of a display area in accordance with some implementations. In particular, Figures 3A and 3B illustrates an imaging system used to image patterns reflected from a retina / fundus that are then imaged by a display lens on to a set of one or more detectors and used to form a retina image. The retina image may then be used to estimate a gaze angle.

[0075] Figures 3A and 3B provide a cross sectional view of an eye-tracking system. Similar to example environment 200 of Figure 2, the eye-tracking system of example environment 300A and 300B illustrates tracking an eye characteristic of eye 1 1 1 via a set of light sources 320, 322, 324, and 326 (e.g., a light projection component such as an LED, or the like) and corresponding set of sensors 330, 332, 334, 336 of a device 302 (e.g., an HMD, such as device 105 of Figure 1). The device 302 further includes a lens 310, and a display portion 340. As illustrated, the light sources 320, 322, 324, and 326 and corresponding sensors 330, 332, 334, and 336 are distributed in an area near the corner of the display 340. The difference between environment 300A and 300B is that the device 302 in environment 300B includes a reflective component 360 (e.g., an IR mirror).

[0076] In some implementations, as further discussed herein, the light sources 320, 322, 324, and 326 and the sensors 330, 332, 334, and 336 may each be positioned in alternative spatial arrangements. For example, one or more of the light sources 320, 322, 324, and 326 and corresponding sensors 330, 332, 334, 336 may be positioned in front of the lens 310, embedded within the lens 310, around the edge of the lens 310 (e.g., on an edge of a frame), embedded within or around a display portion 340 of the device, embedded on the cover glass of the display panel, or a combination thereof. In some implementations,Attorney Docket No. 097425-01473(P67428WO1 ) the sensors may be transparent photodetectors that are imperceptible to the human eye. For example, the light source and the set of the one or more sensors may be positioned within the transparent substrate. In some implementations, this may be accomplished via a waveguide and may include transparent multispectral photodetectors that imperceptible to the human eye and directly placed onto transparent substrate. In some implementations, an illumination source as a diffractive waveguide may be integrated as a standalone layer or laminated to the coverglass to diffracts light out over the eyebox.

[0077] The eye-tracking system of example environments 300A and 300B illustrate a light system to observe light reflections from the eye 11 1 , and in particular, to detect patterns reflected from the retina / fundus. The reflected light may be imaged by the display lens and captured by one or more detectors or photo sensors (e.g., sensors 330, 332, 334, 336, etc.) and used to form a retina image, which may then be used to estimate a gaze angle. The photo sensors (e.g., sensors 330, 332, 334, 336, etc.) may be distributed at or near the corners of the display 340. For example, as illustrated in Figure 3A, the light source 326 produces light that is flashed at an eye 11 1 that is refracted outwardly with respect to the lens 310. In other words, a practical design is to extend the lens 310 as the imaging optics for the sensors on the corners such that this extended portion may be independent to an original display lens that will not affect a display FoV of the device (e.g., an HMD). Moreover, as illustrated in Figure 3B, the light source 326 produces light that is flashed at an eye 1 11 that is reflected within the lens 310 and reflected by the reflective component 360. The light waves are then reflected off of the retina of the eye 1 1 1 and detected by a detector (e.g., sensor 336) which detects the reflected light rays (e.g., reflected light rays 354 from the emitted light rays 352). However, in the environment 300A the reflected light rays 354 are deflected outside of the lens 310 and in the environment 300B the reflected light rays 354 may be reflected by the reflective component 360 before being detected by sensor 336. In some implementations, the reflective component 360 (e.g., a half mirror within a pancake lens), is transparent to NIR LED wavelengths.

[0078] In one aspect, each light source (e.g., light sources 320, 322, 324, and 326) may be used both for illuminating specular and diffusive parts of an object (e.g., eye 1 1 1) and thus may provide at least a threshold level of illumination. Providing at least such a threshold level of illumination may result in a light reflection pattern that would be detected in images captured by the detectors (e.g., sensors 330, 332, 334, 336). For example, light rays 352 from a light source would produce the light rays 354 that are imaged by the display lens on the detectors and used to form a retina image.Attorney Docket No. 097425-01473(P67428WO1 )

[0079] The eye-tracking systems of example environments 200, 300A, and 300B of Figures 2, 3A, and 3B, respectively, are based on identifying image patterns reflected from a retina / fundus that are then imaged by a display lens on to a set of one or more detectors and used to form a retina image, and the retina image may then be used to estimate a gaze angle. Alternatively, in some implementations, the eye-tracking systems of example environments 200, 300A, and 300B may observe glints that the eye 111 is reflecting into one or more photo sensors (e.g., sensors 330, 332, 334, 336, etc.).

[0080] As illustrated in Figures 3A and 3B, one or more sensors and / or one or more light sources may be positioned on or near the four corners of a display area (e.g., outside a display FoV). The four-corner configuration of the light sources and sensors may be configured as either i) using an IR mirror to keep similar pancake outline dimension, or ii) using four corner sensors image stitching to get a wider eye box. In some implementations, the IR mirror (e.g., reflective component 360), a diffractive optical element (DOE), and / or grating may be powered or unpowered (e.g., correcting focus through the pancake lens or just reflecting I deflecting), and may be reflection or transmission (e.g., taking place of beamsplitter for NIR or as a standalone layer that deflects the beam to the corners of the lens).

[0081] In some implementations, the eye-tracking system of example environments 300A, 300B does not include additional vision correction optics, such as contact lenses or other vision glasses. Alternatively, in some implementations, the eye-tracking system of example environments 300A, 300B include vision correction optics that are positioned between the device (e.g., an HMD) and the user’s eyes. In some implementations, one option for vision correction optics may include lens 304, such as a passive lens (e.g., a designated clip-on lenses, personal vision glasses, and the like) or an active lens (e.g., a tunable lens). Additionally, or alternatively, another option for vision correction optics may include a contact lens 306.

[0082] Figure 4A illustrates an example of a user wearing an HMD in accordance with some implementations. In particular, Figure 4A illustrates an example operating environment of the real-world environment 100 (e.g., a room) from Figure 1 , including the user 160 wearing device 165 (e.g., an HMD). In this example, the device 165 is an HMD that includes a transparent or a translucent display that includes a medium through which light representative of images is directed to the eyes of user 160. In particular, device 165 is an HMD that may also be referred to herein as “AR glasses” or “XR glasses.” Such XR glasses may include a transparent display to view the physical environment and be providedAttorney Docket No. 097425-01473(P67428WO1 ) a display to view other content via retinal projection technology that projects graphical images within a view of a person’s retina or onto a person’s retina.

[0083] As illustrated, device 165 includes a frame 412 that can be worn on the user’s head and may include additional extensions (e.g., arms) that are placed over ears of the user 160 to hold the frame in place on the user’s head. The device 165 includes two displays for a left eye and a right eye of the user 160. The frame 412 supports a first lens 415a, and a second lens 415b. Each lens 415 includes a refractive / diffractive medium (also referred to as a transparent substrate). Each lens 415 may be configured as a stack that includes a bias (+ / -) for prescription lenses, a waveguide for housing or embedding a plurality of IR light sources and transparent conductors, and the like.

[0084] The device 165 further includes one or more light sources and one or more sensors that may be configured in different spatial arrangements. Two of those spatial arrangements are illustrated in Figure 4B (e.g., distributed within the display area) and Figure 4C (e.g., distributed within an area proximate to a corresponding corner of a display area), but are not meant to be limiting examples. A light source may be a light projection component, such as an LED, microLED, NIR emitter, and the like, that projects light rays to an eye of the user in order to produce a reflection, such as a glint, that can be detected by one or more sensors.

[0085] In some implementations, the device 165 further includes projector 440a, 440b, for each lens 415a, 415b, respectively. A projector 440 may be used to display XR content to the user (e.g., virtual content that appears to the user at some focal point distance away from the device 165 based on the configuration of the lens). A waveguide stacked within the lens 415 may be configured to bend and / or combine light that is directed toward the eye of the user 160 to provide the appearance of virtual content within the real physical environment 100, as further illustrated herein with reference to Figures 5A-5D. In some implementations, the device 165 may only include one projector 440. For example, a pair of XR glasses for a user that only displays XR content on one side of the device 165 so the user 160 is less distracted and can have a greater view of the physical environment 100.

[0086] In some implementations, the device 165 further includes a controller 450. For example, the controller 450 may include a processor and a power source that controls the light being emitted from a light source. In some implementations, the controller 450 may be a microcontroller that can control the processes described herein for assessing characteristics of the eye (e.g., gaze direction, eye orientation, identifying an iris of the eye) based on the sensor data obtained from the photodiodes. Alternatively, the controller 450 may be communicatively coupled (e.g., wireless communication) with another device, suchAttorney Docket No. 097425-01473(P67428WO1 ) as a mobile phone, tablet, and the like, and the controller may send data collected from the photodiodes to be analyzed by the other device. In the exemplary implementation, the device 165 (with the controller 450) is a stand-alone unit that can project the virtual content via projector 440 and assess characteristics of the eye via light sources for eye tracking purposes without communicating with another device. In some implementations, the light source 520 and the plurality of photodiodes are individually addressable. For example, a processor within the controller 450 can manage a light source and assess each sensor (e.g., a photodiode, or the like) individually. A pattern of reflections can be created based on the spatial arrangement of the light sources and / or the sensors, and the controller 450 can control each light source 420 and asses the sensor data from each sensor.

[0087] Figures 4B and 4C illustrate an example view of a refractive / diffractive medium (e.g., lens 515) of the HMD 165 of Figure 4A in accordance with some implementations. In particular, Figures 4B and 4C illustrate a refractive / diffractive medium with components (some transparent / translucent) for an eye tracking system and XR display for the device 165. Figure 4B illustrates a set of light sources and a corresponding set of sensors that are distributed within a display area 445. Figure 4C illustrates a set of light sources and a corresponding set of sensors that are distributed within an area proximate to a corresponding corner of a display area 445.

[0088] Additionally, or alternatively, in some embodiments, the detectors and / or emitters may be integrated with the projector 440 instead of the lens area 445. For example, the projector optics together with the waveguide that are used to project light on the user retina are also used at the same time to focus light from to user retina to the detectors that are integrated with the projector.

[0089] In the example of Figure 4B, the lens 415 includes a plurality of light sources 420, 422, 424, 426, etc., a plurality of sensors 430, 432, 434, 436, and a controller 450. Although only four light sources and four sensors are labeled, there may be any number of pairs of light sources and sensors distributed within the display area 445. As illustrated in Figure 4B, there are fifteen pairs that are approximately evenly distributed (e.g., approximately equidistant from an adjacent pair) within the display area 445 (e.g., a light source and a corresponding sensor embedded within a pixel array, as illustrated by pixel array 250 of Figure 2).

[0090] In the example of Figure 4C, the lens 415 includes a plurality of light sources 460, 462, 464, 466, a corresponding plurality of sensors 470, 472, 474, 476. As illustrated, each pair of a light source and a sensor are positioned at each corner of the display area 445.Attorney Docket No. 097425-01473(P67428WO1 )

[0091] The controller 450 may control and provide power to each component via transparent conductors. The transparent conductors may be configured to have a size that is small enough and / or are made of one or more transparent materials (e.g., transparent conducting films (TCFs)) so as to not be detectable by a human eye, and thus would be considered transparent and / or translucent when viewing content through the lens 515. Transparent conductors (e.g., connections between each component) may include an optically transparent and electrically conductive material including, but not limited to, indium tin oxide (ITO), wider-spectrum transparent conductive oxides (TCOs), conductive polymers, metal grids and random metallic networks, carbon nanotubes (CNT), graphene, nanowire meshes, and / or ultra thin films. In some implementations, transparent conductors may include semi-transparent conductor materials such as silver nano traces or the like. For example, semi-transparent material may refer to a material that is not necessarily transparent but thin enough that the material is not perceptible to a human eye.

[0092] In some implementations, the plurality of sensors 430, 432, 434, 436, 470, 472, 474, 476 are a size that is small enough that is not detectable by a human eye, thus would be considered transparent and / or translucent when viewing content through the lens 415, such as pass-through content of the physical environment 100, or XR content via display 445. For example, the photodiodes may be 200|jm, 100pm, 75pm, 50pm, 25pm, 10pm, 5pm, 1 pm, and / or another size that is not detectable by a human eye in ordinary use conditions.

[0093] The XR display system of the device 165 through lens 415 includes a projector 440 and a display 445 that may appear to the user as illustrated at the location of display 445. However, the light projected from the projector 440, as powered and controlled by the controller 450, is not directly projected as illustrated. Instead, the light from projector 440 may be bent, via a waveguide, such that the XR content being displayed at display 445 appears to the user 160 at some focal point distance away from the device 165 based on the configuration of the waveguide.

[0094] In some implementations, the device 165 may only have one of the lens’ 415 display XR content (e.g., the left eye lens 415b would be a normal lens without a light source, without sensors, etc., without a projector 440, and thus without a display 445). For example, a left eye view would only present pass-through content of the physical environment 100 (e.g., such as a normal pair of glasses), and the right eye view would have both pass through content of the physical environment 100, and have the capability to present XR content to the right lens 415a only. For example, only the right lens 415a would include light sources, sensors, a projector 440, and a display 445 to present XR content.Attorney Docket No. 097425-01473(P67428WO1 )

[0095] Figures 5A, 5B, 5C, and 5D illustrate different spatial arrangements of sensors with respect to the refractive / diffractive medium of an HMD in accordance with some implementations. Figures 5A, 5B, 5C, and 5D each illustrate a lens 515 in an exemplary stacked configuration. Lens 515 illustrates an example lens used with within device 165 (e.g., an HMD). Lens 515, a refractive / diffractive medium, is stacked from a user’s side (e.g., the side that faces an eye of a user), with layers in order from the user’s side to a world side: Bias (-) 542, air gap 544, waveguide 546, air gap 547, and Bias (+) 548. Each bias 542, 548 (also referred to herein as a “bias layer”) may be used for prescription glasses, e.g., changing a level of prescription based on the size and shape of each bias. In some implementations, a prescription level is changed by only modifying one bias layer, e.g., Bias (+) 548. Figures 5A and 5B illustrate the spatial configuration of distributing pairs of lights sources and sensors within a display area. Figures 5C and 5D illustrate the spatial configuration of pairs of lights sources and sensors distributed within an area proximate to a corresponding corner of a display area.

[0096] Figure 5A illustrates the light sources and sensors of Figure 4B (e.g., light source 420, sensor 430, etc.) and Figure 5C illustrates the light sources and sensors of Figure 4C (e.g., light source 460, sensor 470, etc.) embedded within the lens 515 (e.g., within the air gap 547 between the waveguide 546 and the bias (+) 548). Figure 5B illustrates the light sources and sensors of Figure 4B (e.g., light source 420, sensor 430, etc.) and Figure 5D illustrates the light sources and sensors of Figure 4C (e.g., light source 460, sensor 470, etc.) positioned behind the lens 515. For example, the sensors (e.g., photodiodes) may be coupled directly to the back of the lens 515, or may be coupled to another component behind the lens 515, such as a frame 512. Other HMDs (e.g., headsets that cover a portion of a face around the eyes of a user) may include other components behind the lens to couple the photodiodes to, as discussed herein.

[0097] Figure 6 illustrates an example of image stitching of sensor data for an eye box of an HMD in accordance with some implementations. In particular, Figure 6 illustrates an example process 600 to generate a larger FoV using an image stitching technique based on obtaining sensor data received by sensors distributed within an area proximate to a corresponding corner of a display area (e.g., four corner sensors as illustrated in Figures 3A, 3B, and 4C). For example, at any given time the features of the eye that are being imaged (e.g., glints or a retina image) may exist in one (or more) of four separate images depending on where the eye is within the eyebox, and the image where the features exist will be used fortracking (while other images are discarded). In some implementations, thereAttorney Docket No. 097425-01473(P67428WO1 ) may be some overlap of eyebox regions between the four images to make sure the required features always exist within one of the images for any location of the eye in the eyebox.

[0098] For example, segment 610 illustrates a representation of an example eye box 615. For example, an eye box of an HMD may refer to a volume in which both the requirements for full FoV and full resolution are maintained. The eye box 615 may include four zones (e.g. , zone 1 , zone 2, zone 3, and zone 4, as illustrated) with a center focal point 614. The point 612 represents an example of sensor data obtained at an outside edge of the eye box 615.

[0099] At segment 620, the process 600 provides an exemplary illustration of light rays and reflected light rays 624 from an eye box region 622. The lights rays originating from a light source at the area 626, and the reflected light rays (e.g., from an eye) are sensed from a sensor at the area 626 (e.g., a paired light source and sensor at a corner of the display 240). Moreover, the light rays and reflected lights may be reflected by a reflective component 360 (e.g., an IR mirror).

[0100] At segment 630, the process 600 provides an exemplary illustration for stitching for the eye box diagram 645. In this example of process illustrated in Figure 6, the dot 612 corresponds to dot 642 via a corresponding optical path, and similarly, dot 614 corresponds to dot 644 via a corresponding optical path. For example, corner 2 in the eye box diagram 645 (e.g., dot 642, dot 644), the LED and sensor on dot 644 will pass through a corresponding optical path and reach the eye box center point (e.g., point 614) in the eye box 615 of segment 610. In other words, when an eye position is in the section 2 of eye box 615 of segment 610, the retina image may be captured by the sensors on the corner 2 in the eye box diagram 645. By stitching the captured images from sensors on four corners, processes described herein may determine a wider eye box with sections 1 +2+3+4, such that an eye box stitching process may capture a retina image when a pupil is in this eye box area.

[0101] Figure 7 illustrates an example of estimating pupil coordinates of a user wearing an HMD in accordance with some implementations. In particular, Figure 7 illustrates a process for acquiring an undistorted retina map (e.g., image 710) obtained during an enrollment process, acquiring distorted retina maps (e.g., image 720) during a live (e.g., real-time) process, and determining more accurate pupil coordinates (e.g., pupil coordinate data 760) based on feature matching in order to provide improved gaze accuracy (e.g., direct, accurate alignment between display and gaze subsystems to lower overall gaze error that may occur).Attorney Docket No. 097425-01473(P67428WO1 )

[0102] In some implementations, as illustrated in Figure 7, a light source (e.g., an LED or the like), illuminates a surface of the retina of the eye of the user as the user is accommodating his or her sight. During an enrollment process, an image sensor (e.g., a photodiode) then acquires retina-based gaze tracking images of the retina as the light is reflected off of a surface of the retina. Image 710 represents an undistorted retina map from user enrollment, where the marked portion 712 represents a coordinate in the retina map is mapped to an angle in a user’s FoV. For example, in some implementations, during an enrollment process, a user may be instructed to focus his or her gaze to a particular location that is off in the distance of the display (e.g., focus at a location with that is 1.5m away). For example, the particular location may be on the display of a device. If the user is wearing the device on his or her head (e.g., an HMD), than the location may appear on the display at a very far away distance (e.g., stare off into a small point such that the gaze may be looking out into infinity). The light waves from the light source are then reflected off of the retina of the eye and detected by a detector (e.g., image sensor 230, and the like described herein) to acquire image data of the retinal surface. The lens of the image sensor may be focused to infinity such that when combined with the eye's optics the retinal surface is sharply imaged onto the image sensor (e.g., when the eye is focused to infinity, which is the default for relaxed, healthy eye). Similarly, image 720 represents a distorted retina map that is captured in real-time, where the marked portion 722, which represents a similar coordinate as the marked portion 712, in the distorted retina map that is mapped to a pixel on a display panel (e.g., gaze tracking).

[0103] The feature matching instruction set 730 will then compare the undistorted retina map (e.g., image 710) from the enrollment process and compare features (e.g., portion 712, portion 722, etc.) from the distorted retina map (e.g., image 720) captured in real time, and generate a distortion map 735. The distortion map 735 represents matched features in a retina map and mapped display pixels to angles in user’s FoV. The distortion map 735 may be matched to factory calibration data 740 at block 750 to determine pupil coordinate(s) (e.g., pupil coordinate data 760). The pupil coordinate data 760 may then be used by a device (e.g., an HMD) for accurate gaze tracking. In some implementations, the distortion map (e.g., the distorted retina map captured in real time, e.g., distortion map 735) may also be an output of the algorithm and may be used to correct display distortion of a device (e.g., an HMD) in real-time.

[0104] In some implementations, when additional vision correction optics are worn by the user, the corrected vision from the additional vision correction optics may be accounted for by the systems described herein, including the feature matching instruction set 730.Attorney Docket No. 097425-01473(P67428WO1 )These additional vision correction optics are not only necessary so that the user will sharply see the display on the HMD and / or the external environment, but are also necessary to obtain a sharp retinal image. For example, the systems described herein may compensate forthe optical aberrations that the user’s eye may have. In some implementations, the optics are placed between the HMD and the user’s eyes, and may be either passive lenses (e.g., designated clip-on lenses, personal vision glasses, and the like) or active lenses (e.g., a tunable lens), such as lens 204 of Figure 2, or the vision correction optics may include a contact lens, such as lens 206 of Figure 2.

[0105] Figure 8 is a flowchart illustrating an exemplary method 800. In some implementations, a device (e.g., device 105 or device 165 of Figure 1) performs the techniques of method 800 to assess an eye characteristic of a user based on reflected light from a light source on a refractive / diffractive medium and received at one or more sensors that are distributed within a display area. In some implementations, the techniques of method 800 are performed on a mobile device, desktop, laptop, HMD, or server device. In some implementations, the method 800 is performed on processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method 800 is performed on a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory). In some implementations, the method 800 is performed in combination of one or more devices as described herein. For example, sensor data from a plurality of light sensors may be acquired at an HMD (e.g., device 165), but the processing of the data (e.g., assess an eye characteristic) may be performed at a separate device (e.g., a mobile device, such as device 105).

[0106] At block 802, the method 800 produces light from a light source capable of projecting light in a plurality of different directions over time toward an eye, and the light source and each sensor of a set of one or more sensors are distributed within a display area associated with a transparent substrate. For example, as illustrated in Figure 2, the example eye-tracking system uses a set of light sources (e.g., light sources 220, 222, 224, 226) and a corresponding set of sensors (e.g., sensors 230, 232, 234, 236) that are distributed within a display area (e.g., display 240). For example, light source 220 (e.g., an NIR emitter, or the like) and the sensor 230, are embedded within the pixel array 250 (e.g., a 5x5 pixel array), and the display portion 240 includes multiple pixel arrays to provide a display of content to a user.

[0107] In some implementations, the transparent substrate (e.g., refractive / diffractive medium, lens, etc.) includes a near-eye side and a far-eye side such that when the electronic device is worn, the near-eye side is proximate an eye of the user and the far-eye side is anAttorney Docket No. 097425-01473(P67428WO1 ) opposite side of the refractive / diffractive medium. In some implementations, the light source and the set of one or more sensors is configured in a spatial arrangement associated with a refractive / diffractive medium (e.g., uniformly distributed about the display area). In some implementations, the light source is positioned behind the far-eye side of the refractive / diffractive medium and configured to project light towards the eye through the refractive / diffractive medium. For example, the light source (e.g., an LED, NIR emitter, etc.) may be positioned outside of the refractive / diffractive medium, in an area behind the refractive / diffractive medium and / or in front of a display portion of the device, embedded within the refractive / diffractive medium, or positioned at another location on the device but within a line of sight of the reflections from the projected light from the light source. In some implementations, the light source and the set of the one or more sensors are positioned proximate the far-eye side of the transparent substrate (e.g., NIR and a corresponding photodiode are on the display side of a lens that is positioned between an eye and a display). In some implementations, the light source and the set of the one or more sensors are positioned proximate to the near-eye side of the transparent substrate, (e.g., NIR and a corresponding photodiode are on the near side of a lens that is positioned between an eye and the lens).

[0108] In some implementations, the light source (e.g., light sources 220, 222, 224, 226) produces glints (e.g., a specular reflection) by producing light that reflects off a portion of an eye. In some implementations, a glint may be a specular glint. In some implementations, if a light source is used both for illuminating specular and diffusive parts of the object (e.g., eye 1 11 of the user 1 10), the specular “glints” must be in saturation in order to detect the diffusive area of the object. For example, as illustrated in Figure 2, a light source 220 (e.g., a projection component such as an LED, an NIR emitter, a MEMS laser scanner, and the like) is flashed at an eye 1 11 , and the sensors (e.g., sensors 230, 232, 234, 236, or other light detectors, such as photodiodes, and the like) detect the glints such as the reflected light rays (e.g., reflected light rays 254 from the light rays 252) from the eye 1 11. In some implementations, one or more of the set of sensors are low power light sensors (e.g., a 1-2 pixel receiver, such as a photodiode) that can measure reflectance of light, which includes a time-amplitude-angle relationship, as the light is reflected off of a surface of an eye 45.

[0109] In some implementations, the light source is a micro-LED, a micro IR LED, a mini-LED, and the like. In some implementations, the light source is near infrared (NIR) emitter (e.g., an NIR illuminator) that produces IR light. In some implementations, the light source is one of a plurality of light sources coupled to the electronic device. For example, multiple LEDs, light projectors, laser scanners, NIR emitters, and the like, may be used toAttorney Docket No. 097425-01473(P67428WO1 ) increase a field of view (FoV) and could provide a stereoscopic view to allow the creation of 3D eye model for efficiency.

[0110] In some implementations, the light source is a first light source of a set of one or more light sources, and the first light source and a first sensor of the set of the one or more sensors are positioned within a pixel array of the display area. For example, as illustrated in Figure 2, a light source 220 and a sensor 230 are positioned with a 5x5 pixel array 250 (e.g., an LED or an NIR emitter in conjunction with a sensor / photodiode for each ~5x5 pixel array). In some implementations, the display area includes a plurality of pixel arrays, each pixel array including a light source of the set of one or more light sources anda sensor of the set of the one or more sensors. In some implementations, each pixel array that includes a corresponding pair of a light source and a sensor are approximately evenly distributed within a field of view of the display area (e.g., sensor & LED are uniformly distributed on display area mapping to a display FoV). In some implementations, the pixel array may be different sizes for each light source and corresponding sensor (e.g., ~10x10 array for covering a larger FoV). In some implementations, each corresponding pair of a light source and a sensor are approximately equidistant from an adjacent light source and corresponding sensor. For example, as illustrated in Figure 4B, the pairs of light sources and corresponding sensors (e.g., light sources 420, 422, 424, etc. and sensors 430, 432, 434, etc.) are positioned approximately the same distance apart (e.g., uniform distribution within the display area 445).

[0111] At block 804, the method 800 receives sensor data from a set of one or more sensors, the sensor data corresponding to a plurality of reflections of light produced by the light source and reflected from the eye, and the light that is reflected by the eye being passed through at least a portion of the display area of the transparent substrate before capture by the set of one or more sensors. For example, the set of one or more sensors (e.g., sensors 230, 232, 234, 236) may be a sensor / detector that receives the reflections of light off of the eye (e.g., glints), such as reflected light rays 254 from the light rays 252, as illustrated in Figure 2. In some implementations, the sensors (e.g., photodiodes) may be positioned in an area in front of the lens, outside of the lens (e.g., on a frame of a device), in an area behind the lens, or embedded within the lens.

[0112] In some implementations, the set of the one or more sensors are positioned proximate the far-eye side of the refractive / diffractive medium (e.g., on the display side of a lens that is positioned between an eye and a display). In some implementations, the set of the one or more sensors are positioned proximate to the near-eye side of the refractive / diffractive medium, (e.g., on the near side of a lens that is positioned between anAttorney Docket No. 097425-01473(P67428WO1 ) eye and the lens). In some implementations, the set of the one or more sensors are positioned between the far-eye side of the refractive / diffractive medium and a display portion of the device. In some implementations, the set of the one or more sensors are positioned within a display portion of the device. In some implementations, the set of the one or more sensors are positioned within the refractive / diffractive medium. For example, the sensors may be transparent multispectral photodetectors that are imperceptible to the human eye and directly placed onto the refractive / diffractive medium or onto a display of the device.

[0113] At block 806, the method 800 determines a characteristic of the eye based on the sensor data. For example, based on the sensor data obtained from the set of photodiodes, the eye tracking system described herein may be able to identify and / or track a position and / or orientation of an eye, a gaze direction, the cornea shape, and the like. Various embodiments regarding determining characteristics of the eye, and other embodiments, are described herein for both method 800 of Figure 8 and method 900 of Figure 9 following the description of Figure 9 below.

[0114] Figure 9 is a flowchart illustrating an exemplary method 900. In some implementations, a device (e.g., device 105 of Figure 1 or device 165 of Figure 2) performs the techniques of method 900 to assess an eye characteristic of a user based on reflected light from a light source on a refractive / diffractive medium and received at one or more sensors that are distributed within an area proximate to a corresponding corner of a display area. In some implementations, the techniques of method 900 are performed on a mobile device, desktop, laptop, HMD, or server device. In some implementations, the method 900 is performed on processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method 900 is performed on a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory). In some implementations, the method 900 is performed in combination of one or more devices as described herein. For example, sensor data from a plurality of light sensors may be acquired at an HMD (e.g., device 165), but the processing of the data (e.g., assess an eye characteristic) may be performed at a separate device (e.g., a mobile device, such as device 105).

[0115] At block 902, the method 900 produces light from a set of light sources capable of projecting light in a plurality of different directions over time toward an eye, and each light source and each sensor of a set of one or more sensors are distributed within an area proximate to a corresponding corner of a display area associated with a transparent substrate. For example, as illustrated in Figures 3A, 3B, the example eye-tracking system uses a set of light sources (e.g., light sources 320, 322, 324, 326) and a corresponding setAttorney Docket No. 097425-01473(P67428WO1 ) of sensors (e.g., sensors 330, 332, 334, 336) that are distributed within an area proximate to a corresponding corner of a display area (e.g., display portion 240). For example, as illustrated in Figure 3A, the light source 326 (e.g., an NIR emitter, or the like), which emits a light ray 352 and the sensor 336, which receives a reflected light ray 354, are positioned outside of and proximate to a corner of the display portion 240.

[0116] In some implementations, the transparent substrate (e.g., refractive / diffractive medium, lens, etc.) includes a near-eye side and a far-eye side such that when the electronic device is worn, the near-eye side is proximate an eye of the user and the far-eye side is an opposite side of the refractive / diffractive medium. In some implementations, each light source and a corresponding sensor is configured in a spatial arrangement associated with a refractive / diffractive medium (e.g., distributed at a corner of the display area). In some implementations, each light source and a corresponding sensor is positioned behind the far- eye side of the refractive / diffractive medium, and the light source is configured to project light towards the eye through the refractive / diffractive medium. For example, a light source (e.g., an LED) may be positioned outside of the refractive / diffractive medium, in an area behind the refractive / diffractive medium and / or in front of a display portion of the device, embedded within the refractive / diffractive medium, or positioned at another location on the device but proximate to a corner of the display area and within a line of sight of the reflections from the projected light from the light source. In some implementations, the light source and the set of the one or more sensors are positioned proximate the far-eye side of the transparent substrate (e.g., NIR and a corresponding photodiode are on the display side of a lens that is positioned between an eye and a display). In some implementations, the light source and the set of the one or more sensors are positioned proximate to the near-eye side of the transparent substrate, (e.g., NIR and a corresponding photodiode are on the near side of a lens that is positioned between an eye and the lens).

[0117] In some implementations, the light source (e.g., light sources 320, 322, 324, 326) produces glints (e.g., a specular reflection) by producing light that reflects off a portion of an eye. In some implementations, a glint may be a specular glint. In some implementations, if a light source is used both for illuminating specular and diffusive parts of the object (e.g., eye 1 11 of the user 1 10), the specular “glints” must be in saturation in order to detect the diffusive area of the object. For example, as illustrated in Figures 3A and 3B (e.g., a four corner configuration), light sources 320, 322, 324, 326 (e.g., a projection component such as an LED, an NIR emitter, a MEMS laser scanner, and the like) emits light ray 352 that is flashed at an eye 11 1 , and the sensors (e.g., sensors 330, 332, 334, 336, or other lightAttorney Docket No. 097425-01473(P67428WO1 ) detectors, such as photodiodes, and the like) detect the glints such as the reflected light rays (e.g., reflected light rays 254 from the light rays 252) from the eye 1 11.

[0118] In some implementations, the light source is a micro LED, a micro IR LED, a mini-LED, and the like. In some implementations, the light source is near infrared (NIR) emitter. In some implementations, the light source is one of a plurality of light sources coupled to the electronic device. For example, multiple LEDs, light projectors, laser scanners, NIR emitters, and the like, may be used to increase a field of view (FoV) and could provide a stereoscopic view to allow the creation of 3D eye model for efficiency.

[0119] In some implementations, the display area includes four corners, and each light source and the corresponding sensor are positioned at the area proximate to a corresponding corner of the four corners of the display area. For example, as illustrated in Figure 4C, the light sources 460, 462, 464, and 466, and corresponding sensors 470, 472, 474, and 476, respectively, are positioned near or proximate to the four corners of the display area 445. In some implementations, the light projected by each light source and the light that is reflected by the eye before capture by each sensor extends outside a field of view of the display area. For example, as illustrated by Figure 3A, a display outline FoV of the lens 210 (e.g., a pancake lens) may be enlarged based on having a light source and a sensor outside of the display area (e.g., at the four corners).

[0120] At block 904, the method 900 receives sensor data from a set of one or more sensors, the sensor data corresponding to a plurality of reflections of light produced by each light source and reflected from the eye, and the light that is reflected by the eye being passed through at least a portion of the display area of the transparent substrate before capture by the set of one or more sensors. For example, the set of one or more sensors (e.g., sensors 330, 332, 334, 336) may be a sensor / detector (e.g., a photodiode) that receives the reflections of light off of the eye (e.g., glints), such as reflected light rays 354 from the emitted light rays 352, as illustrated in Figure 3A and 3B for the four corner configuration implementation. In some implementations, the sensors (e.g., photodiodes) may be positioned in an area in front of the lens, outside of the lens, in an area behind the lens, or embedded within the lens.

[0121] In some implementations, the set of the one or more sensors are positioned proximate the far-eye side of the refractive / diffractive medium (e.g., on the display side of a lens that is positioned between an eye and a display). In some implementations, the set of the one or more sensors are positioned proximate to the near-eye side of the refractive / diffractive medium, (e.g., on the near side of a lens that is positioned between an eye and the lens). In some implementations, the set of the one or more sensors areAttorney Docket No. 097425-01473(P67428WO1 ) positioned between the far-eye side of the refractive / diffractive medium and a display portion of the device. In some implementations, the set of the one or more sensors are positioned within a display portion of the device. In some implementations, the set of the one or more sensors are positioned within the refractive / diffractive medium. For example, the sensors may be transparent multispectral photodetectors that are imperceptible to the human eye and directly placed onto the refractive / diffractive medium or onto a display of the device.

[0122] At block 906, the method 900 determines a characteristic of the eye based on the sensor data. For example, based on the sensor data obtained from the set of photodiodes, the eye tracking system described herein may be able to identify and / or track a position and / or orientation of an eye, a gaze direction, the cornea shape, and the like.

[0123] In some implementations, using the four-corner configuration described for method 900 (e.g., as illustrated in Figures 3A and 4C), determining a characteristic of the eye based on the sensor data may be based on image stitching (e.g., if not using the IR mirror, reflective component 360 illustrated in Figure 3B). For example, as illustrated in Figure 6, the data obtained from the four corner sensors may be used with an image stitching technique to generate a wider eye box.

[0124] In various implementations, method 800 and method 900 determine a characteristic of the eye based on receiving sensor data from a set of one or more sensors. The following implementations regarding determining characteristics of the eye, other similar implementations, may apply to either method 800 (e.g., using light sources and sensors that are uniformly distributed within a display area) or method 900 (e.g., using light sources and sensors that are distributed within an area proximate to a corresponding corner of a display area), and are not meant to be limited to any specific configuration described herein.

[0125] In some implementations, determining the characteristic of the eye based on the sensor data includes determining a position of a pupil of the eye. For example, an XYZ coordinate in a 3D space may be determined for the pupil position based on the sensor data of the set of photodiodes. In some implementations, determining the position of the pupil of the eye includes obtaining an undistorted retina map of the eye obtained during an enrollment process, determining a distorted retina map of the eye based on the sensor data, and matching features of the undistorted retina map with features of the distorted retina map. For example, as illustrated in Figure 7, determining the pupil position may be based on an enrollment-to-live correlation process that matches features (e.g., feature matchingAttorney Docket No. 097425-01473(P67428WO1 ) retina map (e.g., image 720) captured in real-time, to create a distortion map 735 in order to estimate current pupil coordinates.

[0126] In an exemplary implementation, a method may further include a process to directly measure and correct the distortion of the display lens based on the retina map (e.g., a retina map may be distorted by a catadioptric lens and pupil location may not be known at time of enrollment). In some implementations, method 800 and method 900 may further include determining a distorted retina map of the eye based on the sensor data and adjust at least a portion of the display area of the transparent substrate (display lens) based on the distorted retina map. Additionally, or alternatively, in some implementations, method 800 and method 900 may further include determining a distorted retina map of the eye based on the sensor data, matching features of the distorted retina map with features of an undistorted retina map of the eye obtained during an enrollment process, and adjust at least a portion of the display area of the transparent substrate (display lens) based on the matched features of the distorted retina map and the undistorted retina map.

[0127] In some implementations, determining the characteristic of the eye based on the sensor data includes determining at least one of a phase, an intensity, an angle, a timing, and a polarization of the light. In some implementations, determining the characteristic of the eye based on the sensor data includes determining a gaze direction based on a detected reflection angle. In some implementations, determining the characteristic of the eye based on the sensor data includes determining a shape of the eye. For example, the eye tracking system described herein can determine pupil position (e.g., X,Y,Z coordinates) as well as gaze and other features such as eyelid, eyebrows, etc., and their associated movements.

[0128] In some implementations, determining an eye characteristic may be based on a determined location of the glint. For example, the eye characteristic may include a gaze direction, eye orientation, identifying an iris of the eye, or the like, for an eye-tracking system. For example, if the electronic device is an HMD, the eye-tracking system for the HMD can track gaze direction, eye orientation, identification of the iris, etc., of a user.

[0129] In some implementations, the method 800 and / or method 900 determines a location of a glint based on the reflected light received at the sensor. For example, determining a location of the glint may include determining a centroid of the received light. In some implementations, multiple glints may be produced and located by a sensor (e.g., sensors 230, 232, 330, 332, etc.). For example, a centroid can be determined based on a non-saturated periphery (e.g., a halo).

[0130] In some implementations, determining an orientation of the eye is based on identifying a pattern of the glints / light reflections in an image. In one example, gaze directionAttorney Docket No. 097425-01473(P67428WO1 ) may be determined using the sensor data to identify two points on the eye, e.g., a cornea center and an eyeball center. In another example, gaze direction may be determined using the sensor data (e.g., a pattern of glints) to directly predict the gaze direction. For example, a machine learning model may be trained to directly predict the gaze direction based on the sensor data.

[0131] In some implementations, the processor may be further configured to initiate an action based on detecting that the gaze direction is approximately oriented towards a target area. In some implementations, sensor data may be used as input to a machine learning model that is trained to output a gaze direction. The machine learning model may include models that use neural networks, decision trees, linear regression, support vector machines, and the like. A machine learning model may be trained based on training data, e.g., ground truth data, that identifies the eye characteristic fora given set of training inputs. For example, gaze direction may be manually labelled by a training user following a displayed item on a display and deriving the gaze direction corresponding to the displayed location of the item over time. A machine learning model may utilize image and light sensor data corresponding to multiple states of a system (e.g., each of the last 5 images and all light sensor data received during that time period) to predict an eye characteristic for the current state and / for one or more future states of the system.

[0132] In some implementations, for iris identification, the user may be uniquely identified from a registration process or prior iris evaluation. For example, the method 800 and / or method 900 may include assessing the characteristic from the eye by performing an authentication process. The authentication process may include identifying an iris of an eye. For example, matching a pattern of glints / light reflections in an image with a unique pattern associated with the user. In some embodiments, the iris identification techniques (e.g., matching patterns), may be used for anti-spoofing. For example, there could be multiple enrolled patterns that may be changed and can be used to authenticate a user’s iris against a pre-enrolled biometric template, and confirm that the user is the right person, a real person, and is authenticating in real-time. Iris identification may be used as a primary authentication mode or as part of a multi-factor or step-up authentication. The matching patterns may be stored in a database located on the HMD (e.g., device 105), another device communicatively coupled to the HMD (e.g., a mobile device in electronic communication with the HMD), an external device or server (e.g., connected through a network), or a combination of these or other devices.

[0133] In some implementations, the device executing the techniques of method 800 and / or method 900 (e.g., device 165) includes a frame, a light source (e.g., a light projectionAttorney Docket No. 097425-01473(P67428WO1 ) component, such as a MEMS scanner) a refractive / diffractive medium coupled to the frame, and a plurality of photodiodes. In some implementations, the refractive / diffractive medium is configured to display content. The plurality of photodiodes may be configured in a spatial arrangement on a surface of the refractive / diffractive medium (in front of or behind, e.g., Figures 5B, 5D), or embedded within the refractive / diffractive medium (e.g., Figures 5A, 5C).

[0134] In some implementations, the refractive / diffractive medium includes a bias layer (e.g., bias(-), bias(+)). For example, for prescription lenses, the bias layer may be modified based on prescription. In some implementations, the plurality of sensors are configured in a spatial arrangement on a surface of the bias layer. For example, as illustrated in Figure 5C, the plurality of light sources 460, 462, 464, 466 and sensors 470, 472, 474, 476 are attached to the bias (+) 548 layer of the lens 515 within the air gap 547. Alternatively, as illustrated in Figure 5D, the plurality of light sources 460, 462, 464, 466 and sensors 470, 472, 474, 476 may be attached to the far-eye side of the lens 515. In some implementations, alternative configurations and placements of the light sources and sensors may be utilized provided that the size of the components are not perceptible to the eye when the HMD is worn by a user (e.g., less than 200 micrometers in diameter, or even smaller such as 100pm, 50pm, 25pm, 5pm, etc.). In other words, the sensors and light sources, and any connections (e.g., transparent conductors) that connect the photodiodes to another component, such as a controller, appear invisible to a human eye.

[0135] In some implementations, the plurality of light sources (e.g., light sources 460, 462, 464, 466) and the set of sensors (e.g., sensors 470, 472, 474, 476) are positioned behind the refractive / diffractive medium, as illustrated in Figure 5D. Alternatively, in some implementations, the set of light sources and sensors are embedded within the refractive / diffractive medium. For example, as illustrated in Figure 5C, the light sources 460, 462, 464, 466 and the sensors 470, 472, 474, 476, are embedded within the refractive / diffractive medium (e.g., lens 515) and positioned in the air gap 547 between the waveguide 546 and the bias (+) 548. In some implementations, the light sources, sensors, and the like, and transparent conductors connecting the sensors to a controller (e.g., a power source and a processor) are embedded within or on top of the waveguide 546.

[0136] Figure 10 is a block diagram of an example device 1000. Device 1000 illustrates an exemplary device system configuration for a device (e.g., devices 105, 125, 165, etc.). While certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein. ToAttorney Docket No. 097425-01473(P67428WO1 ) that end, as a non-limiting example, in some implementations the device 1000 includes one or more processing units 1002 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, and / or the like), one or more input / output (I / O) devices and sensors 1006, one or more communication interfaces 1008 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, and / or the like type interface), one or more programming (e.g., I / O) interfaces 1010, one or more displays 1012, one or more interior and / or exterior facing image sensor systems 1014, a memory 1020, and one or more communication buses 1004 for interconnecting these and various other components.

[0137] In some implementations, the one or more communication buses 1004 include circuitry that interconnects and controls communications between system components. In some implementations, the one or more I / O devices and sensors 1006 include at least one of an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., blood pressure monitor, heart rate monitor, blood oxygen sensor, blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptics engine, one or more depth sensors (e.g., a structured light, a time-of-flight, or the like), and / or the like.

[0138] In some implementations, the one or more displays 1012 are configured to present a view of a physical environment or a graphical environment to the user. In some implementations, the one or more displays 1012 correspond to holographic, digital light processing (DLP), liquid-crystal display (LCD), liquid-crystal on silicon (LCoS), organic lightemitting field-effect transitory (OLET), organic light-emitting diode (OLED), surfaceconduction electron-emitter display (SED), field-emission display (FED), quantum-dot lightemitting diode (QD-LED), micro-electromechanical system (MEMS), and / or the like display types. In some implementations, the one or more displays 1012 correspond to diffractive, reflective, polarized, holographic, etc. waveguide displays. In one example, the device 105 includes a single display. In another example, the device 105 includes a display for each eye of the user.

[0139] In some implementations, the one or more image sensor systems 1014 are configured to obtain image data that corresponds to at least a portion of the physical environment. For example, the one or more image sensor systems 1014 include one or more RGB cameras (e.g., with a complimentary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), monochrome cameras, IR cameras, depth cameras, event-based cameras, and / orthe like. In various implementations,Attorney Docket No. 097425-01473(P67428WO1 ) the one or more image sensor systems 1014 further include illumination sources that emit light, such as a flash. In various implementations, the one or more image sensor systems 1014 further include an on-camera image signal processor (ISP) configured to execute a plurality of processing operations on the image data.

[0140] The memory 1020 includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some implementations, the memory 1020 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 1020 optionally includes one or more storage devices remotely located from the one or more processing units 1002. The memory 1020 includes a non-transitory computer readable storage medium.

[0141] In some implementations, the memory 1020 or the non-transitory computer readable storage medium of the memory 1020 stores an optional operating system 1030 and one or more instruction set(s) 1040. The operating system 1030 includes procedures for handling various basic system services and for performing hardware dependent tasks. In some implementations, the instruction set(s) 1040 include executable software defined by binary information stored in the form of electrical charge. In some implementations, the instruction set(s) 1040 are software that is executable by the one or more processing units 1002 to carry out one or more of the techniques described herein.

[0142] The instruction set(s) 1040 include an illumination analysis instruction set 1042 and an eye characteristic instruction set 1044. The instruction set(s) 1040 may be embodied a single software executable or multiple software executables.

[0143] In some implementations, the illumination analysis instruction set 1042 is executable by the processing unit(s) 1002 to produce a reflection by directing light towards an eye using a light source (e.g., a MEMS scanner), receive sensor data from a sensor (e.g., a set of one or more photodiodes) and determine a reflective property (e.g., a spectral property) of the reflection based on the sensor data. To these ends, in various implementations, the instruction includes instructions and / or logic therefor, and heuristics and metadata therefor.

[0144] In some implementations, the eye characteristic instruction set 1044 is executable by the processing unit(s) 1002 to determine a characteristic of the eye based on the sensor data such as identifying and tracking a position and / or orientation of an eye, a gaze direction, the cornea shape, and the like, using one or more of the techniques discussed herein or as otherwise may be appropriate. To these ends, in variousAttorney Docket No. 097425-01473(P67428WO1 ) implementations, the instruction includes instructions and / or logic therefor, and heuristics and metadata therefor.

[0145] Although the instruction set(s) 1040 are shown as residing on a single device, it should be understood that in other implementations, any combination of the elements may be located in separate computing devices. Moreover, Figure 10 is intended more as functional description of the various features which are present in a particular implementation as opposed to a structural schematic of the implementations described herein. As recognized by those of ordinary skill in the art, items shown separately could be combined and some items could be separated. The actual number of instructions sets and how features are allocated among them may vary from one implementation to another and may depend in part on the particular combination of hardware, software, and / or firmware chosen for a particular implementation.

[0146] Figure 11 illustrates a block diagram of an exemplary head-mounted device 1100 in accordance with some implementations. The head-mounted device 1100 includes a housing 1 101 (or enclosure) that houses various components of the head-mounted device 1100. The housing 1 101 includes (or is coupled to) an eye pad (not shown) disposed at a proximal (to the user) end of the housing 1 101. In various implementations, the eye pad is a plastic or rubber piece that comfortably and snugly keeps the head-mounted device 1 100 in the proper position on the face of the user 1 10 (e.g., surrounding the eye of the user).

[0147] The housing 1 101 houses a display 11 10 that displays an image, emitting light towards or onto the pupil of an eye of a user. In various implementations, the display 1 1 10 emits the light through an eyepiece having one or more optical elements 1 105 that refracts the light emitted by the display 11 10, making the display appear to the user to be at a virtual distance farther than the actual distance from the eye to the display 11 10. For example, optical element(s) 1105 may include one or more lenses, a waveguide, other diffraction optical elements (DOE), and the like. For the user to be able to focus on the display 1 110, in various implementations, the virtual distance is at least greater than a minimum focal distance of the eye (e.g., 7 cm). Further, in order to provide a better user experience, in various implementations, the virtual distance is greater than 1 meter.

[0148] The housing 1101 also houses a tracking system including one or more light sources 1 122, camera 1124, camera 1132, camera 1134, camera 1 136, and a controller 1180. The one or more light sources 1 122 emit light onto the eye of the user that reflects as a light pattern (e.g., a circle of glints) that can be detected by the camera 1124. Based on the light pattern, the controller 1 180 can determine an eye tracking characteristic of the user.Attorney Docket No. 097425-01473(P67428WO1 )For example, the controller 1180 can determine a gaze direction or a blinking state (eyes open or eyes closed) of the user. As another example, the controller 1180 can determine a pupil center, a pupil size, or a point of regard associated with the pupil. Thus, in various implementations, the light is emitted by the one or more light sources 1122, reflects off the eye of the user, and is detected by the camera 1124. In various implementations, the light from the eye of the user is reflected off a hot mirror or passed through an eyepiece before reaching the camera 1124.

[0149] The display 1110 emits light in a first wavelength range and the one or more light sources 1122 emit light in a second wavelength range. Similarly, the camera 1124 detects light in the second wavelength range. In various implementations, the first wavelength range is a visible wavelength range (e.g., a wavelength range within the visible spectrum of approximately 400-700 nm) and the second wavelength range is a near-infrared wavelength range (e.g., a wavelength range within the near-infrared spectrum of approximately 700-1400 nm).

[0150] In various implementations, eye tracking (or, in particular, a determined gaze direction) is used to enable user interaction (e.g., the user selects an option on the display 1110 by looking at it), provide foveated rendering (e.g., present a higher resolution in an area of the display 1110 the user is looking at and a lower resolution elsewhere on the display 1110), or correct distortions (e.g., for images to be provided on the display 1110).

[0151] In various implementations, the one or more light sources 1122 emit light towards the eye of the user, which reflects in the form of a plurality of glints.

[0152] In various implementations, the camera 1124 is a frame / shutter-based camera that, at a particular point in time or multiple points in time at a frame rate, generates an image of the eye of the user. Each image includes a matrix of pixel values corresponding to pixels of the image which correspond to locations of a matrix of light sensors of the camera. In implementations, each image is used to measure or track pupil dilation by measuring a change of the pixel intensities associated with one or both of a user’s pupils.

[0153] In various implementations, the camera 1132, camera 1134, and camera 1136 are frame / shutter-based cameras that, at a particular point in time or multiple points in time at a frame rate, can generate an image of the face of the user 110 or capture an external physical environment. For example, camera 1132 captures images of the user’s face below the eyes, camera 1134 captures images of the user’s face above the eyes, and camera 1136 captures the external environment of the user (e.g., environment 100 of Figure 1). TheAttorney Docket No. 097425-01473(P67428WO1 ) images captured by camera 1132, camera 1134, and camera 1136 may include light intensity images (e.g., RGB) or depth image data (e.g., Time-of-Flight, infrared, etc.).

[0154] A physical environment refers to a physical world that people can sense or interact with without aid of electronic devices. The physical environment may include physical features such as a physical surface or a physical object. For example, the physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense or interact with the physical environment such as through sight, touch, hearing, taste, and smell. In contrast, an extended reality (XR) environment refers to a wholly or partially simulated environment that people sense or interact with via an electronic device. For example, the XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, or the like. With an XR system, a subset of a person’s physical motions, or representations thereof, are tracked, and, in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that comports with at least one law of physics. As one example, the XR system may detect head movement and, in response, adjust graphical content and an acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment. As another example, the XR system may detect movement of the electronic device presenting the XR environment (e.g., a mobile phone, a tablet, a laptop, or the like) and, in response, adjust graphical content and an acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment. In some situations (e.g., for accessibility reasons), the XR system may adjust characteristic(s) of graphical content in the XR environment in response to representations of physical motions (e.g., vocal commands).

[0155] There are many different types of electronic systems that enable a person to sense or interact with various XR environments. Examples include head mountable systems, projection-based systems, heads-up displays (HUDs), vehicle windshields having integrated display capability, windows having integrated display capability, displays formed as lenses designed to be placed on a person’s eyes (e.g., similar to contact lenses), headphones / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head mountable system may have one or more speaker(s) and an integrated opaque display. Alternatively, a head mountable system may be configured to accept an external opaque display (e.g., a smartphone). The head mountable system may incorporate one or more imaging sensors to capture images or video of the physical environment, orAttorney Docket No. 097425-01473(P67428WO1 ) one or more microphones to capture audio of the physical environment. Rather than an opaque display, a head mountable system may have a transparent or translucent display. The transparent or translucent display may have a medium through which light representative of images is directed to a person’s eyes. The display may utilize digital light projection, OLEDs, LEDs, uLEDs, liquid crystal on silicon, laser light source, or any combination of these technologies. The medium may be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In some implementations, the transparent or translucent display may be configured to become opaque selectively. Projection-based systems may employ retinal projection technology that projects graphical images onto a person’s retina. Projection systems also may be configured to project virtual objects into the physical environment, for example, as a hologram or on a physical surface.

[0156] It will be appreciated that the implementations described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope includes both combinations and sub combinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art.

[0157] As described above, one aspect of the present technology is the gathering and use of physiological data to improve a user’s experience of an electronic device with respect to interacting with electronic content. The present disclosure contemplates that in some instances, this gathered data may include personal information data that uniquely identifies a specific person or can be used to identify interests, traits, or tendencies of a specific person. Such personal information data can include physiological data, demographic data, location-based data, telephone numbers, email addresses, home addresses, device characteristics of personal devices, or any other personal information.

[0158] The present disclosure recognizes that the use of such personal information data, in the present technology, can be used to the benefit of users. For example, the personal information data can be used to improve interaction and control capabilities of an electronic device. Accordingly, use of such personal information data enables calculated control of the electronic device. Further, other uses for personal information data that benefit the user are also contemplated by the present disclosure.

[0159] The present disclosure further contemplates that the entities responsible for the collection, analysis, disclosure, transfer, storage, or other use of such personal informationAttorney Docket No. 097425-01473(P67428WO1 ) or physiological data will comply with well-established privacy policies or privacy practices. In particular, such entities should implement and consistently use privacy policies and practices that are generally recognized as meeting or exceeding industry or governmental requirements for maintaining personal information data private and secure. For example, personal information from users should be collected for legitimate and reasonable uses of the entity and not shared or sold outside of those legitimate uses. Further, such collection should occur only after receiving the informed consent of the users. Additionally, such entities would take any needed steps for safeguarding and securing access to such personal information data and ensuring that others with access to the personal information data adhere to their privacy policies and procedures. Further, such entities can subject themselves to evaluation by third parties to certify their adherence to widely accepted privacy policies and practices.

[0160] Despite the foregoing, the present disclosure also contemplates implementations in which users selectively block the use of, or access to, personal information data. That is, the present disclosure contemplates that hardware or software elements can be provided to prevent or block access to such personal information data. For example, in the case of user-tailored content delivery services, the present technology can be configured to allow users to select to “opt in’’ or “opt out’’ of participation in the collection of personal information data during registration for services. In another example, users can select not to provide personal information data for targeted content delivery services. In yet another example, users can select to not provide personal information, but permit the transfer of anonymous information for the purpose of improving the functioning of the device.

[0161] Therefore, although the present disclosure broadly covers use of personal information data to implement one or more various disclosed embodiments, the present disclosure also contemplates that the various embodiments can also be implemented without the need for accessing such personal information data. That is, the various embodiments of the present technology are not rendered inoperable due to the lack of all or a portion of such personal information data. For example, content can be selected and delivered to users by inferring preferences or settings based on non-personal information data or a bare minimum amount of personal information, such as the content being requested by the device associated with a user, other non-personal information available to the content delivery services, or publicly available information.

[0162] In some embodiments, data is stored using a public / private key system that only allows the owner of the data to decrypt the stored data. In some other implementations, theAttorney Docket No. 097425-01473(P67428WO1 ) data may be stored anonymously (e.g., without identifying or personal information about the user, such as a legal name, username, time and location data, orthe like). In this way, other users, hackers, or third parties cannot determine the identity of the user associated with the stored data. In some implementations, a user may access his or her stored data from a user device that is different than the one used to upload the stored data. In these instances, the user may be required to provide login credentials to access their stored data.

[0163] Numerous specific details are set forth herein to provide a thorough understanding of the claimed subject matter. However, those skilled in the art will understand that the claimed subject matter may be practiced without these specific details. In other instances, methods, apparatuses, or systems that would be known by one of ordinary skill have not been described in detail so as not to obscure claimed subject matter.

[0164] Unless specifically stated otherwise, it is appreciated that throughout this specification discussions utilizing the terms such as “processing,” “computing,” “calculating,” “determining,” and “identifying” or the like refer to actions or processes of a computing device, such as one or more computers or a similar electronic computing device or devices, that manipulate or transform data represented as physical electronic or magnetic quantities within memories, registers, or other information storage devices, transmission devices, or display devices of the computing platform.

[0165] The system or systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device can include any suitable arrangement of components that provides a result conditioned on one or more inputs. Suitable computing devices include multipurpose microprocessor-based computer systems accessing stored software that programs or configures the computing system from a general purpose computing apparatus to a specialized computing apparatus implementing one or more implementations of the present subject matter. Any suitable programming, scripting, or other type of language or combinations of languages may be used to implement the teachings contained herein in software to be used in programming or configuring a computing device.

[0166] Implementations of the methods disclosed herein may be performed in the operation of such computing devices. The order of the blocks presented in the examples above can be varied for example, blocks can be re-ordered, combined, or broken into subblocks. Certain blocks or processes can be performed in parallel.

[0167] The use of “adapted to” or “configured to” herein is meant as open and inclusive language that does not foreclose devices adapted to or configured to perform additionalAttorney Docket No. 097425-01473(P67428WO1 ) tasks or steps. Additionally, the use of “based on” is meant to be open and inclusive, in that a process, step, calculation, or other action “based on” one or more recited conditions or values may, in practice, be based on additional conditions or value beyond those recited. Headings, lists, and numbering included herein are for ease of explanation only and are not meant to be limiting.

[0168] It will also be understood that, although the terms “first,” “second,” etc. may be used herein to describe various objects, these objects should not be limited by these terms. These terms are only used to distinguish one object from another. For example, a first node could be termed a second node, and, similarly, a second node could be termed a first node, which changing the meaning of the description, so long as all occurrences of the “first node” are renamed consistently and all occurrences of the “second node” are renamed consistently. The first node and the second node are both nodes, but they are not the same node.

[0169] The terminology used herein is for the purpose of describing particular implementations only and is not intended to be limiting of the claims. As used in the description of the implementations and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “comprises” or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, objects, or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, objects, components, or groups thereof.

[0170] As used herein, the term “if may be construed to mean “when” or “upon” or “in response to determining” or “in accordance with a determination” or “in response to detecting,” that a stated condition precedent is true, depending on the context. Similarly, the phrase “if it is determined [that a stated condition precedent is true]” or “if [a stated condition precedent is true]” or “when [a stated condition precedent is true]" may be construed to mean “upon determining” or “in response to determining” or “in accordance with a determination” or “upon detecting” or “in response to detecting” that the stated condition precedent is true, depending on the context.

[0171] The foregoing description and summary of the invention are to be understood as being in every respect illustrative and exemplary, but not restrictive, and the scope of the invention disclosed herein is not to be determined only from the detailed description ofAttorney Docket No. 097425-01473(P67428WO1 ) illustrative implementations but according to the full breadth permitted by patent laws. It is to be understood that the implementations shown and described herein are only illustrative of the principles of the present invention and that various modification may be implemented by those skilled in the art without departing from the scope and spirit of the invention.

Claims

1. Attorney Docket No. 097425-01473(P67428WO1 )What is claimed is:1 . An electronic device comprising: a light source capable of projecting light; a set of one or more sensors; a transparent substrate having a near-eye side, a far-eye side, and a display area, wherein the light source and each sensor are distributed within the display area, wherein, when the electronic device is worn, the near-eye side is proximate an eye of the user and the far-eye side is an opposite side of the transparent substrate such that light projected by the light source that is reflected by the eye passes through at least a portion of the display area of the transparent substrate before capture by the set of one or more sensors; and a processor configured to perform operations comprising: receiving sensor data from the set of one or more sensors, the sensor data corresponding to a plurality of reflections of light produced by the light source and reflected from the eye; and determining a characteristic of the eye based on the sensor data.

2. The device of claim 1 , wherein the light source is configured to project light towards the eye through the transparent substrate.

3. The device of claims 1 or 2, wherein the light source and the set of the one or more sensors are positioned proximate the far-eye side of the transparent substrate.

4. The device of claims 1 or 2, wherein the light source and the set of the one or more sensors are positioned proximate to the near-eye side of the transparent substrate.

5. The device of claims 1 or 2, wherein the light source and the set of the one or more sensors are positioned within the transparent substrate.

6. The device of any of claims 1-5, wherein the light source is one of a plurality of light sources coupled to the electronic device.

7. The device of any of claims 1-6, wherein the light source comprises a micro light emitting diode (LED).Attorney Docket No. 097425-01473(P67428WO1 )8. The device of any of claims 1—7, wherein the light source comprises a near infrared (NIR) emitter.

9. The device of any of claims 1-8, wherein the light source is a first light source of a set of one or more light sources, wherein the first light source and a first sensor of the set of the one or more sensors are positioned within a pixel array of the display area.

10. The device of claim 9, wherein the display area comprises a plurality of pixel arrays, each pixel array comprising a light source of the set of one or more light sources and a sensor of the set of the one or more sensors.11 . The device of claim 10, wherein each pixel array that includes a corresponding pair of a light source and a sensor are approximately evenly distributed within a field of view of the display area.

12. The device of claims 10 or 11 , wherein each corresponding pair of a light source and a sensor are approximately equidistant from an adjacent light source and corresponding sensor.

13. The device of any of claims 1-12, wherein the processor is further configured to perform operations comprising: determining a distorted retina map of the eye based on the sensor data; and adjusting at least a portion of the display area of the transparent substrate based on the distorted retina map.

14. The device of any of claims 1-12, wherein the processor is further configured to perform operations comprising: determining a distorted retina map of the eye based on the sensor data; matching features of the distorted retina map with features of an undistorted retina map of the eye obtained during an enrollment process; and adjusting at least a portion of the display area of the transparent substrate based on the matched features of the distorted retina map and the undistorted retina map.Attorney Docket No. 097425-01473(P67428WO1 )15. The device of any of claims 1-13, wherein determining the characteristic of the eye based on the sensor data comprises determining a position of a pupil of the eye.

16. The device of claim 15, wherein determining the position of the pupil of the eye comprises: obtaining an undistorted retina map of the eye obtained during an enrollment process; determining a distorted retina map of the eye based on the sensor data; and matching features of the undistorted retina map with features of the distorted retina map.

17. The device of any of claims 1-15, wherein determining the characteristic of the eye based on the sensor data comprises determining a gaze direction based on a detected reflection angle.

18. The device of claim 17, wherein the processor is further configured to perform operations comprising: initiating an action based on detecting that the gaze direction is approximately oriented towards a target area.

19. A method comprising: at an electronic device having a processor: producing light from a light source capable of projecting light in a plurality of different directions overtime toward an eye, wherein the light source and each sensor of a set of one or more sensors are distributed within a display area associated with a transparent substrate, and wherein the transparent substrate comprises a near-eye side and a far-eye side such that when the electronic device is worn, the near-eye side is proximate an eye of the user and the far-eye side is an opposite side of the transparent substrate; receiving sensor data from the set of one or more sensors, the sensor data corresponding to a plurality of reflections of light produced by the light source and reflected from the eye, wherein the light that is reflected by the eye passes through at least a portion of the display area of the transparent substrate before capture by the set of one or more sensors; and determining a characteristic of the eye based on the sensor data.Attorney Docket No. 097425-01473(P67428WO1 )20. A non-transitory computer-readable storage medium, storing program instructions executable on a device including one or more processors to perform operations comprising: producing light from a light source capable of projecting light in a plurality of different directions overtime toward an eye, wherein the light source and each sensor of a set of one or more sensors are distributed within a display area associated with a transparent substrate, and wherein the transparent substrate comprises a near-eye side and a far-eye side such that when the electronic device is worn, the near-eye side is proximate an eye of the user and the far-eye side is an opposite side of the transparent substrate; receiving sensor data from the set of one or more sensors, the sensor data corresponding to a plurality of reflections of light produced by the light source and reflected from the eye, wherein the light that is reflected by the eye passes through at least a portion of the display area of the transparent substrate before capture by the set of one or more sensors; and determining a characteristic of the eye based on the sensor data.21 . An electronic device comprising: a set of light sources, wherein each light source is capable of projecting light; a set of one or more sensors, wherein each light source is associated with a corresponding sensor; a transparent substrate having a near-eye side, a far-eye side, and a display area, wherein each light source and a corresponding sensor are distributed within an area proximate to a corresponding corner of the display area, wherein, when the electronic device is worn, the near-eye side is proximate an eye of the user and the far-eye side is an opposite side of the transparent substrate such that light projected by each light source that is reflected by the eye passes through at least a portion of the display area of the transparent substrate before capture by the set of one or more sensors; and a processor configured to perform operations comprising: receiving sensor data from the set of one or more sensors, the sensor data corresponding to a plurality of reflections of light produced by the set of light sources and reflected from the eye; and determining a characteristic of the eye based on the sensor data.Attorney Docket No. 097425-01473(P67428WO1 )22. The device of claim 21 , wherein each light source is configured to project light towards the eye through the transparent substrate.

23. The device of claims 21 or 22, wherein each light source and a corresponding sensor are positioned proximate the far-eye side of the transparent substrate.

24. The device of claims 21 or 22, wherein each light source and a corresponding sensor are positioned proximate to the near-eye side of the transparent substrate.

25. The device of claims 21 or 22, wherein each light source and a corresponding sensor are positioned within the transparent substrate.

26. The device of any of claims 21-25, wherein each light source is one of a plurality of light sources coupled to the electronic device.

27. The device of any of claims 21-26, wherein each light source comprises a micro light emitting diode (LED).

28. The device of any of claims 21-27, wherein each light source comprises a near infrared (NIR) emitter.

29. The device of any of claims 21-28, wherein the display area comprises four corners, and wherein each light source and the corresponding sensor are positioned at the area proximate to a corresponding corner of the four corners of the display area.

30. The device of any of claims 21-29, wherein the light projected by each light source and the light that is reflected by the eye before capture by each sensor extends outside a field of view of the display area.31 . The device of claim 30, wherein determining a characteristic of the eye based on the sensor data is based on image stitching.

32. The device of any of claims 21-29, further comprising one or more mirrors, wherein the light projected by each light source and the light that is reflected by the eye before capture by each sensor is reflected by at least one of the one or more mirrors.Attorney Docket No. 097425-01473(P67428WO1 )33. The device of any of claims 21-32, wherein the processor is further configured to perform operations comprising: determining a distorted retina map of the eye based on the sensor data; and adjusting at least a portion of the display area of the transparent substrate based on the distorted retina map.

34. The device of any of claims 21-32, wherein the processor is further configured to perform operations comprising: determining a distorted retina map of the eye based on the sensor data; matching features of the distorted retina map with features of an undistorted retina map of the eye obtained during an enrollment process; and adjusting at least a portion of the display area of the transparent substrate based on the matched features of the distorted retina map and the undistorted retina map.

35. The device of any of claims 21-32, wherein determining the characteristic of the eye based on the sensor data comprises determining a position of a pupil of the eye.

36. The device of claim 35, wherein determining the position of the pupil of the eye comprises: obtaining an undistorted retina map of the eye obtained during an enrollment process; determining a distorted retina map of the eye based on the sensor data; and matching features of the undistorted retina map with features of the distorted retina map.

37. The device of any of claims 21-36, wherein determining the characteristic of the eye based on the sensor data comprises determining a gaze direction based on a detected reflection angle.

38. The device of claim 37, wherein the processor is further configured to perform operations comprising: initiating an action based on detecting that the gaze direction is approximately oriented towards a target area.Attorney Docket No. 097425-01473(P67428WO1 )39. The device of any of claims 21-38, wherein determining the characteristic of the eye based on the sensor data comprises determining a shape of the eye.

40. The device of any of claims 21-39, wherein the electronic device is a headmounted device (HMD).41 . A method comprising: at an electronic device having a processor: producing light from a set of light sources, wherein each light source is capable of projecting light in a plurality of different directions over time toward an eye, wherein each light source of the set of light sources and each sensor of a set of one or more sensors are distributed within an area proximate to a corresponding corner of a display area associated with a transparent substrate, and wherein the transparent substrate comprises a near-eye side and a far-eye side such that when the electronic device is worn, the neareye side is proximate an eye of the user and the far-eye side is an opposite side of the transparent substrate; receiving sensor data from the set of one or more sensors, the sensor data corresponding to a plurality of reflections of light produced by each light source and reflected from the eye, wherein the light that is reflected by the eye passes through at least a portion of the display area of the transparent substrate before capture by the set of one or more sensors; and determining a characteristic of the eye based on the sensor data.Attorney Docket No. 097425-01473(P67428WO1 )42. A non-transitory computer-readable storage medium, storing program instructions executable on a device including one or more processors to perform operations comprising: producing light from a set of light sources, wherein each light source is capable of projecting light in a plurality of different directions over time toward an eye, wherein each light source of the set of light sources and each sensor of a set of one or more sensors are distributed within an area proximate to a corresponding corner of a display area associated with a transparent substrate, and wherein the transparent substrate comprises a near-eye side and a far-eye side such that when the electronic device is worn, the neareye side is proximate an eye of the user and the far-eye side is an opposite side of the transparent substrate; receiving sensor data from the set of one or more sensors, the sensor data corresponding to a plurality of reflections of light produced by each light source and reflected from the eye, wherein the light that is reflected by the eye passes through at least a portion of the display area of the transparent substrate before capture by the set of one or more sensors; and determining a characteristic of the eye based on the sensor data.

Images