Retinal reflex tracking for gaze alignment

Abstract

Various implementations disclosed herein include devices, systems, and methods that initiate an action based on a detected gaze direction oriented toward a target area (e.g., a hot corner / hot zone). For example, an example process may include: generating a reflection by directing light toward an eye using an illuminator; receiving sensor data from a sensor wherein a direction sensed by the sensor (e.g., an optical axis of a camera) is substantially aligned with a direction from the eye to the target area; determining a reflective property (e.g., a spectral property) of the reflection based on the sensor data; detecting, based on the reflective property, that a gaze direction of the eye is substantially oriented toward the target area (e.g., a hot zone / hot spot); and initiating an action based on detecting that the gaze direction is generally oriented toward the target area.

Description

Technical Field

[0001] The present disclosure relates generally to electronic devices, and in particular to systems, methods, and devices for detecting an eye gaze direction of a user of an electronic device. Background Art

[0002] Existing eye tracking technology analyzes flashes reflected from the user's eyes and captured via an image sensor. Some head-mounted systems may include eye tracking technology using near infrared (nIR) illuminators and standard multi-pixel complementary metal oxide semiconductor (CMOS) cameras pointed at each eye. The combination of nIR illuminators and CMOS cameras can provide accurate and high frame rate (e.g., greater than 15 frames per second) tracking of each eye. However, this typical nIR+CMOS configuration has a baseline power that does not scale well with reduced frame rates, and it is not feasible to keep the CMOS imager and nIR LED on all the time in very low power applications. Therefore, it may be desirable to provide a means of efficiently providing an eye tracking system that does not need to be continuously tracked across the entire possible gaze space in order to provide power savings for head-mounted systems for evaluating eye characteristics (e.g., gaze direction, eye orientation, identifying the iris of the eye, etc.). Summary of the invention

[0003] Various embodiments disclosed herein include devices, systems, and methods for initiating actions based on detecting that the direction of a user's eye gaze is oriented approximately toward a target area (e.g., a hotspot / hot corner). Determining that the eye is gazing at a target area (e.g., a hotspot / hot corner) is based on detecting that the reflection of light leaving the eye has certain properties, such as spectral properties indicating a red eye type reflection that is approximately aligned toward the target area. The illuminator and / or detector may be optically aligned with the approximate direction from the eye to the target area, for example, by positioning the illuminator / detector in and / or behind the target area of ​​the display / lens or using an optical waveguide in the target area of ​​the lens. The detector may be a near infrared (nIR) transceiver sensitive to spectral reflections at narrow coaxial angles. The detector may use low-power hardware (e.g., a photodiode paired with a lens) that may be active when high-power eye tracking hardware (e.g., a multi-pixel complementary metal oxide semiconductor (CMOS)) is turned off. For example, a user may glance at a hot corner to trigger an action, such as initiating an extended reality (XR) experience using full eye tracking, providing a response to a notification, and the like. If the gaze meets criteria (eg, the glance lasts for more than a threshold amount of time), the desired action may be initiated.

[0004] Generally speaking, an innovative aspect of the subject matter described in this specification can be embodied in a method comprising the following actions: at an electronic device having a processor and a display, generating a reflection by directing light toward an eye using an illuminator; receiving sensor data from a sensor, wherein a sensing direction of the sensor is approximately aligned with a direction from the eye to a target area; determining a reflective property of the reflection based on the sensor data; detecting that a gaze direction of the eye is approximately oriented toward the target area based on the reflective property; and initiating an action based on detecting that the gaze direction is approximately oriented toward the target area.

[0005] These and other embodiments can optionally include one or more of the following features.

[0006] In some aspects, the reflective property indicates an amount of reflection of the light from the fundus of the eye. In some aspects, the amount of the reflection indicates a level of alignment between the sensing direction of the sensor and the gaze direction.

[0007] In some aspects, detecting that the gaze direction of the eye is oriented generally toward the target area includes determining that the reflective property is within an angular threshold, wherein the angular threshold indicates that the gaze direction is within a threshold of a target direction.

[0008] In some aspects, detecting that the gaze direction of the eye is oriented generally toward the target area includes generating a time-amplitude response based on the reflection to identify that the gaze direction is aligned generally toward the target area.

[0009] In some aspects, detecting that the gaze direction of the eye is oriented generally toward the target area includes generating a frequency-time response based on the reflection to identify when the gaze direction is aligned generally toward the target area.

[0010] In some aspects, detecting that the gaze direction of the eye is generally oriented toward the target area is based on using a parameterized model of the eye.

[0011] In some aspects, the sensor is a first sensor and the action triggers an eye tracking algorithm based on sensor data received from a second sensor, wherein the first sensor is different from the second sensor and uses less power than the second sensor.

[0012] In some aspects, the action is initiated based on whether the gaze direction satisfies a criterion. In some aspects, the action provides a user response to the notification.

[0013] In some aspects, the action includes launching an extended reality (XR) experience.

[0014] In some aspects, the target area is located at one of a plurality of zones on a display of the electronic device.

[0015] In some aspects, the sensor comprises a photodiode. In some aspects, the light is infrared (IR) light.

[0016] In some aspects, the method also includes detecting a gaze direction toward the multiple target areas using multiple illuminators and multiple sensors corresponding to the multiple target areas, wherein the multiple illuminators generate light having different wavelengths, and wherein the multiple sensors distinguish reflections of the light generated by the multiple illuminators based on the wavelength.

[0017] In some aspects, the electronic device is a head mounted device (HMD). In some aspects, the HMD comprises a waveguide, wherein the waveguide substantially aligns the sensing direction of the sensor or the direction of the light generated by the illuminator with the direction from the eye to the target area.

[0018] According to some specific implementations, a non-transitory computer-readable storage medium stores instructions, which are executable by a computer to perform or cause the performance of any of the methods described herein. According to some specific implementations, a device includes one or more processors, a non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and are configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing the performance of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] So that the present disclosure may be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0020] Figure 1 Devices for displaying content and obtaining physiological data from a user according to some implementations are illustrated.

[0021] Figure 2A An example head mounted display (HMD) according to some implementations is illustrated.

[0022] Figure 2B Examples according to some specific implementations Figure 2A area of ​​the transparent substrate of the HMD.

[0023] Figure 2C Examples according to some specific implementations Figure 2A Example hot corner detection in an area of ​​the HMD.

[0024] Figure 3A and Figure 3B Example eye tracking systems according to some implementations are illustrated.

[0025] Figure 4A and Figure 4B Optical effects for detecting eye gaze direction toward a target area are illustrated according to some implementations.

[0026] FIG. 5A to FIG. 5C An example user experience for detecting eye gaze direction according to some implementations is illustrated.

[0027] Figure 6 A parameterized model of an eye according to some specific implementations is illustrated.

[0028] Figure 7 Another parameterized model of an eye according to some implementations is illustrated.

[0029] Figure 8 A flowchart representation of a method for initiating an action based on detecting that a gaze direction is generally oriented toward a target area is illustrated according to some implementations.

[0030] Fig. 9 is a block diagram illustrating device components of an exemplary device according to some implementations.

[0031] Fig.10 is a block diagram of an example head mounted device (HMD) according to some specific implementations.

[0032] As a matter of practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily enlarged or reduced for clarity. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, the same reference numerals may be used to represent the same features. DETAILED DESCRIPTION

[0033] In order to provide a thorough understanding of the example implementations shown in the drawings, many details are described. However, the drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. Those of ordinary skill in the art will appreciate that other effective aspects or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail to avoid obscuring more relevant aspects of the example implementations described herein.

[0034] Figure 1 An example environment 100 of a real-world environment 5 (e.g., a room) is illustrated that includes a device 10 having a display 15. In some implementations, the device 10 displays content 20 to a user 25. For example, the content 20 may be a button, a user interface icon, a text box, a graphic, an avatar of the user or another user, etc. In some implementations, the content 20 may occupy the entire display area of ​​the display 15.

[0035] The device 10 obtains image data, motion data, and / or physiological data (e.g., pupil data, facial feature data, etc.) from the user 25 via one or more sensors (e.g., sensor 34, sensor 36). For example, the device 10 includes a light source 32 (e.g., a light emitting diode (LED)) that can be used to illuminate a specular portion and a diffuse portion of an eye 45 of the user 25 via light 40 (e.g., infrared (IR) light). Based on the specular illumination of the eye 45, the device 10 can obtain eye gaze characteristic data 42 via a high-power sensor 34 (such as a complementary metal oxide semiconductor (CMOS)). Additionally or alternatively, the device 10 can obtain eye gaze characteristic data 44 via a low-power sensor 36 (such as a photodiode).

[0036] Although this example and other examples discussed herein illustrate a single device 10 in a real-world environment 5, the techniques disclosed herein are applicable to multiple devices and other real-world environments. For example, the functionality of device 10 may be performed by multiple devices, with high-power sensor 34, low-power sensor 36, and light source 32 located on each respective device or divided among them in any combination.

[0037] In some specific implementations, such as Figure 1 As shown, device 10 is a handheld electronic device (e.g., a smartphone or a tablet computer). In some implementations, device 10 is a laptop or desktop computer. In some implementations, device 10 has a trackpad, and in some implementations, device 10 has a touch-sensitive display (also referred to as a "touch screen" or "touch screen display"). In some implementations, device 10 is a wearable device such as a head mounted device (HMD).

[0038] In some implementations, device 10 includes an eye tracking system for detecting eye positioning and eye movement via eye gaze characteristic data 42, 44. For example, the eye tracking system may include one or more IR LEDs (e.g., light source 32), an eye tracking camera (e.g., a near IR (NIR) camera), and an illumination source (e.g., a NIR light source) that emits light (e.g., NIR light) toward the eyes of user 25. In addition, the illumination source of device 10 may emit NIR light to illuminate the eyes of user 25, and the NIR camera may capture images of the eyes of user 25. In some implementations, the images captured by the eye tracking system may be analyzed to detect the positioning and movement of the eyes of user 25, or to detect other information about the eyes such as color, shape, state (e.g., fully open, squinting, etc.), pupil dilation, or pupil diameter. In addition, the gaze point estimated from the eye tracking image may enable gaze-based interaction with content shown on a near-eye display of device 10.

[0039] In some implementations, the device 10 has a graphical user interface (GUI), one or more processors, a memory, and one or more modules, programs, or instruction sets for performing multiple functions stored in the memory. In some implementations, the user 25 interacts with the GUI through finger contacts and gestures on the touch-sensitive surface. In some implementations, these functions include image editing, drawing, presentation, word processing, web page creation, disk editing, spreadsheet production, playing games, making and receiving calls, video conferencing, sending and receiving emails, instant messaging, fitness support, digital photography, digital video recording, web browsing, digital music playback, and / or digital video playback. Executable instructions for performing these functions may be included in a computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0040] In some implementations, one or both eyes 45 of user 25 (including one or both pupils 50 of user 25) present physiological data (e.g., eye gaze characteristic data 42, 44) in the form of pupil responses detected according to the flicker analysis. The pupil responses of user 25 may result in changes in the size or diameter of pupil 50 via the optic nerve and the oculomotor cranial nerve. For example, the pupil response may include a constriction response (miosis) (e.g., a narrowing of the pupil) or a dilation response (mydriasis) (e.g., a widening of the pupil). In some implementations, device 10 may detect a pattern of physiological data representing a time-varying pupil diameter.

[0041] Figure 2A Examples include a real-world environment 5 (e.g., from Figure 1 An example operating environment 200 of a room of a user 25 that includes a real-world environment including a user 25 wearing a device 210 (HMD). In this example, the device 210 is an HMD including a transparent or translucent display that includes a medium through which light representing an image is directed to the eyes of the user 25. Specifically, the device 210 is an HMD, which may also be referred to as "AR glasses" or "XR glasses" herein. Such XR glasses may include a transparent display to view the physical environment and be provided with a display to view other content via retinal projection technology that projects a graphic image within the field of view of the person's retina or onto the person's retina.

[0042] As shown, the device 210 includes a frame 212 that can be worn on the user's head and may include additional extensions (e.g., arms) placed above the ears of the user 25 to keep the frame in place on the user's head. The device 210 includes two displays for the left eye and the right eye of the user 25. The frame 212 supports a first lens 215a and a second lens 215b. Each lens 215 includes a transparent substrate. Each lens 215 can be configured as a stack that includes a deviation (+ / -) for prescription lenses, a waveguide and a transparent conductor for accommodating or embedding multiple IR light sources, etc. The device 210 also includes detectors 220a, 220b for each lens 215a, 215b, respectively. The detector 220 can be an image sensor, such as an IR camera, that detects light reflected from the user's eyes, such as a flash.

[0043] In some implementations, the device 210 also includes a projector 240a, 240b for each lens 215a, 215b, respectively. The projector 240 can be used to display XR content to the user (e.g., virtual content presented to the user at a certain focal distance from the device 210 based on the configuration of the lens). The waveguides stacked within the lens 215 can be configured to bend and / or combine light directed toward the eyes of the user 25 to provide the appearance of virtual content within the real physical environment 5. In some implementations, the device 210 may include only one projector 240. For example, a pair of XR glasses for a user displays XR content on only one side of the device 210, so the user 25 is less distracted and can have a larger view of the physical environment 5.

[0044] In some implementations, the device 210 also includes a controller 250. For example, the controller 250 may include a processor and a power source that controls the emission of light from the light source. In some implementations, the controller 250 is a microcontroller that can control the process described herein for evaluating the characteristics of the eye (e.g., gaze direction, eye orientation, iris of the eye) based on the sensor data obtained from the detector 220. Alternatively, the controller 250 can be communicatively coupled (e.g., wireless communication) with another device (such as a mobile phone, a tablet computer, etc.), and the controller 250 can send the data collected from the detector 220 for analysis by the other device. In an exemplary implementation, the device 210 (with the controller 250) is a stand-alone unit that does not communicate with another device, which can project virtual content via the projector 240 and evaluate the characteristics of the eye via the light source for eye tracking purposes.

[0045] Figure 2B An HMD (eg, Figure 2A Specifically, Figure 2BAn example of eight defined “hot spot zones” are illustrated for which the processes described herein may be used for hot spot detection regions (e.g., zones / regions around display FOV 230). Example zones include superior temporal (ST) zone 231, superior (S) zone 232, superior nasal (SN) zone 233, temporal zone 234, nasal zone 235, inferior temporal zone 236, inferior zone 237, and inferior nasal zone 238. However, more or fewer zones may be defined for each lens 215 of device 210.

[0046] In some implementations, each hotspot zone may include a zone tolerance. For example, each zone boundary may allow a 10° tolerance margin to account for sensor placement, occlusions, extreme angle constraints, etc. In some implementations, a hotspot update rate may be determined for each hotspot zone determined by the trigger-to-photon latency of a particular lens 215 of the device 210. For example, it may be desirable to have a trigger-to-photon latency of less than 100ms. In some implementations, detection of a hotspot trigger may require more than one frame to detect whether a hotspot is triggered based on a trigger threshold (e.g., at least three frames are required to generate a hotspot trigger).

[0047] Figure 2C An HMD (eg, Figure 2A An example of hot corner detection in an area of ​​a device 210). Specifically, Figure 2C The eye gaze position at the target area 262 of the user in the ST zone 231 of the lens 215 of the device 210 is illustrated, as illustrated by the hot zone area 260. In an exemplary implementation, the hot corners can be implemented using an array of ultrasonic sensors, photosensitive detectors, or electrooculogram sensors distributed around the edge of the frame 212 of the device 210. Eye postures and gestures that can be used by the subsystem can be defined by gaze, rapid glances, or a combination thereof. Therefore, the sensor (e.g., detector 220) of the device 210 can provide an estimate of the eye gaze speed and the gaze angle (e.g., angle θ).

[0048] like Figure 2B and Figure 2C As illustrated, spatial regions corresponding to hot spots are defined relative to the relative positions of the display FOV 230 and the eye. The hot spot region can extend outward from the corners and edges of the display FOV 230 until it reaches and is bounded by the frame 212. In some implementations, instead of providing a direct gaze angle (e.g., angle θ), a binary output corresponding to whether the gaze is in one of those regions (e.g., ST region 231) can be provided. Additionally, instead of providing a direct gaze velocity, an estimate of the time that the gaze is in one of those regions can be provided.

[0049] In some implementations, since the edges of the hot spots are defined relative to the display FOV 230 and the display FOV 230 is defined by the positioning of the eyes within the "eye-friendly zone", the user may be required to go through a registration process after wearing the device 210. However, in an exemplary embodiment, seamless and automatic registration can occur when the camera-based gaze tracking automatically locates the left and right pupils of the user 25 when worn and during normal use of the device 210 (e.g., when the user 25 is wearing the device 210 (such as an HMD)). Then, periodically (e.g., once a minute), the camera-based gaze / pupil tracking system may wake up or may be triggered to wake up to verify whether the calibration is within a specific calibration threshold. Alternatively, in some embodiments, it may be necessary to instruct the user to focus on a specific target. For example, the user 25 may be instructed to focus his or her gaze on a specific location a certain distance away from the display (e.g., focus on a location 1.5m away). For example, the specific location may be on the display of the device 10. If user 25 is wearing device 10 (e.g., an HMD, such as device 210) on his or her head, this location may appear on the display at a very far distance (e.g., staring at a small dot so that the gaze may be looking out to infinity). Light waves from the light source are then reflected from the retina of eye 45 and detected by the detector to collect image data of the retinal surface. The lens of the image sensor may be focused to infinity so that, when combined with the optics of the eye, the retinal surface is clearly imaged onto the image sensor (e.g., when the eye is focused to infinity, which is the default condition for a relaxed, healthy eye).

[0050] In some implementations, due to the binocular nature of gaze and variable eye dominance (dominant eye behavior), each eye may be provided with a separate hotspot subsystem (e.g., illuminator and detector 220 for each lens 215). For example, a user may initiate rapid saccades outside of the normal gaze angle (approximately + / - 20°), which may result in imperfect vergence behavior, where both eyes may not appear to be gazing at the same area. Therefore, a binocular hotspot subsystem may be used to compensate for such imperfect vergence behavior.

[0051] In some implementations, the focal width for hot corner detection can be determined to distinguish vestibular ocular reflex (VOR) gaze, visual search rapid glances, overshoots and re-gazes (such as those when reading text in the real world), and other non-hot corner triggers by aggregating information from the posture subsystem and / or application subsystem. Additionally, the dwell and / or gaze duration of the user's eye gaze may be important based on the application being applied. Additionally, the hot corner detection technology can also help with blink detection of the eye tracking system.

[0052] Figure 3A and Figure 3BExample environments 300A and 300B, respectively, of hot spot eye tracking systems according to some implementations are illustrated. The hot spot eye tracking systems of example environments 300A and 300B use a light source 320 (e.g., light source 32 on device 10) such as a near infrared (nIR) illuminator that generates IR light 324 at an emission angle 322 and a light sensor 330 (e.g., a low power sensor 36 on device 10, such as a photodiode). Figure 3A An example off-axis response is illustrated in which the user is looking directly forward (e.g., toward Figure 2C 230 ) to adjust his or her line of sight (e.g., the fundus axis). Figure 3B An example on-axis response (e.g., fundus axis toward the hot corner) is illustrated in which the user is in the upper region (e.g., above the display FOV 230 and in Figure 2B S area 232) to adjust his or her line of sight upward.

[0053] In some specific implementations, such as Figure 3A and Figure 3B As illustrated, a light source 320 (e.g., a nIR illuminator such as an LED or the like) illuminates the surface of the retina of the user's eye 45 as the user is adjusting his or her gaze. IR light 324 generated by the light source 320 is irradiated at an emission angle 322. Subsequently, a low-power light sensor 330 (e.g., a 1-2 pixel receiver such as a photodiode) measures the light reflectivity as the light reflects off the surface of the eye 45, which includes a time-amplitude-angle relationship. In some implementations, the emission angle 322 can cover a large area of ​​the eye 45 to allow for any shifts in the positioning of the glasses and allow the user to change his or her eye 45. For example, the emission angle 322 can be controlled so that it is not too wide to waste energy, but wide enough to cover all eyeball positions of the user's set.

[0054] for Figure 3A For an off-axis response, the user is not viewing the hot corner (e.g., the user's viewing angle exceeds a viewing angle threshold from the hot corner, such as greater than 5°), then the light reflectivity at the light sensor 330 (e.g., the output from the photodiode) is below the threshold. Figure 3B In an on-axis response, when the user is looking toward a hot corner, the light reflectance at the light sensor 330 (e.g., the output from a photodiode) increases as the fundus axis of the eye 45 is aligned with the sensor.

[0055] Additionally, Figure 3A and Figure 3BThe hot spot eye tracking system of device 10 includes an image sensor 310 (e.g., sensor 34 on device 10, such as a CMOS camera) to observe light scattered from the retina of eye 45 in order to collect an image of eye 45. Image sensor 310 is part of the eye gaze tracking camera and can be turned off during an always-on "hot spot", but periodically awakened to calibrate positioning and eye distance.

[0056] Figure 4A and Figure 4B Example environments 400A and 400B are illustrated, respectively, for detecting an optical effect of an eye gaze direction toward a target area according to some implementations. Specifically, Figure 4A and Figure 4B An example hot corner eye tracking system utilizing a transparent substrate is illustrated, such as an optical waveguide 410 including an integrated sensor 420 capable of measuring on-axis reflections from a human fundus. In some implementations, sensor 420 may be a single sensor covering multiple zones, as illustrated. Additionally or alternatively, an individual sensor (e.g., sensor 420) may be included for each zone. For example, a waveguide (such as waveguide 410) may be designed to support discrete locations (e.g., in 4 corners) without having to integrate emitters / detectors into the display optical engine of the waveguide. In some implementations, individual emitters / detectors may be integrated at different locations of waveguide 410. Additionally, in some implementations, an individual sensor module may be integrated at each corner of a frame of a device including waveguide 410 (e.g., frame 212 of device 210). In some implementations, having an individual sensor module integrated at each corner of the frame of the eye tracking device will not affect the display of waveguide 410, but will take up space for the edge / frame.

[0057] FIG. 5A to FIG. 5C An example user experience for detecting eye gaze direction through or on lens 502 according to some implementations is illustrated. For example, FIG. 5A to FIG. 5C It is illustrated that a user views a display of a device (eg, device 10 , 210 , etc.) or its vicinity and launches an application on a user interface based on his or her gaze toward a particular area (eg, toward a hot spot). Figure 5A The user is illustrated looking at a particular area to launch the clock application. For example, the user's gaze 520 is illustrated as looking at the upper right corner of the lens 502 outside the FOV display 510 (e.g., Figure 2B After the user's gaze 520 views the area in the upper right corner within a threshold amount of time (e.g., within 500ms of viewing the hot corner), the clock application 512 appears and another target application icon 504 appears in the upper left corner. Figure 5BThe user is illustrated moving his or her gaze 520 toward the application icon 504 in the upper left corner. For example, the application icon 504 can be used to determine the location (e.g., confirmation of the user wanting to launch the clock application 512). In other words, confirm that the user wants to view a particular application. Alternatively, the application icon 504 can be used to close the clock application 512 (e.g., telling the user to look at the upper left corner whenever he or she wants to close or reject an application). Figure 5C An alternative example is illustrated. For example, Figure 5C The display of application icons 532, 534, 536, 538 is illustrated after the user has fixed his or her gaze 520 on application icon 504 for a threshold amount of time (e.g., greater than 500 ms). Figure 5C Illustrated in an application of the hot spot eye gaze detection system described herein, after a user has gazed toward a target area (e.g., a hot spot) for a threshold amount of time, one or more user interface applications (e.g., application icons 532, 534, 536, 538) may be displayed to the user.

[0058] FIG. 5A to FIG. 5C An example user experience is illustrated for detecting eye gaze direction through or on lens 502 while a user is viewing a display of a device (e.g., device 10, 210, etc.) at or near the user's eye and launching an application on a user interface based on his or her gaze toward a particular area (e.g., toward a hot spot). Alternatively, in some implementations, nothing has to be displayed for the system to work, and the hot spot gesture to launch the experience can be a learned behavior. For example, a user may be able to customize a specific location on lens 502 for a hot spot to be displayed, such as in a particular zone.

[0059] In some implementations, detecting that the gaze direction of an eye is oriented approximately toward a target region (e.g., a hotspot) is based on tracking the eye using a parameterized oculomotor model of the eye. Figure 6 to Figure 7 Parameterized oculomotor eye models 600, 700 are illustrated, respectively. For example, camera images and light sensor data may be used to set parameters (eg, coefficients) of such models, and thus predict eye characteristics represented thereby.

[0060] The eye moves in the eye socket using six muscles. The medial rectus (MR) and lateral rectus (LR) are primarily responsible for horizontal eye movement. The superior rectus (SR) and inferior rectus (IR) are primarily responsible for vertical eye movement. The superior oblique (SO) and inferior oblique (IO) muscles are primarily responsible for eye rotation around the primary visual axis of the eye and vertical eye movement.

[0061] Model 600 models the biomechanics of the lateral rectus muscle (LR) of the eye 125 by modeling the rotation of the eye 125 in a plane. The rotation θ is attached to a series of elastic (SE) elements 610 stimulated by a neuron activation function 620. The neuron activation function 620 models the impulses transmitted to the eye 125. θ SE_LR 640 and θ LT_LR 650 represents the possible deformation of the lateral rectus (LR) muscle modeled by a series of elastic elements and length tension elements (relative to the eye socket, which is considered fixed). Model 700 illustrates the dual muscle nature of the eye and permits tracking of the eye around the horizontal axis. Additional similar models can be used to enable tracking of the eye around the vertical axis for torsional movement of the eye and dilation movement of the pupil. It additionally shows viscous elements 710a to 710f for modeling the damping response of the eye muscles.

[0062] Neuron activation functions are used to model the impulse characteristics of control signals from the brain. These neuron control signals are used to calculate the dynamic behavior of the muscle group. The oculomotor device is able to model the passive elasticity, tension, length tension, series elasticity and force-velocity response of the oculomotor muscles. This therefore allows the estimation of the time-displacement behavior of the eye based on previous and current measurements of the eye state.

[0063] In some implementations, eye movements are tracked based on camera images and light sensor data and used to calculate coefficients of a parameterized oculomotor model of the eye 125. The model can be used to provide a state space description of the eye. For example, a transformation matrix can be developed and used to make predictions. For example, assuming a particular state, the transformation matrix can be used to predict the velocity and / or actuation that would be expected according to the oculomotor model of the eye. Based on the observed movement, future movement of the eye can be predicted. In some implementations, camera and / or light sensor data is used to determine the coefficients of the model, and the model is used to interpolate eye characteristics during the intermediate time between frames of camera images based on the light sensor data received during that time. For example, once the eye finishes blinking, the model can predict a natural gaze in a particular direction. The light sensor can detect blinks during the intermediate time periods, and the system can determine the expected eye movement based on the modeled eye behavior.

[0064] The eyes may additionally or alternatively be tracked using an eye-specific vergence or accommodation model. Such a model may be based on previously tracked behavior of the eyes. The model may be learned and thus used to predict what the eye 125 will typically do when attempting to examine a scene. In some implementations, information from one eye is used to predict what the other eye is doing based on a model that describes the correspondence between the behavior of the user's two eyes.

[0065] In some implementations, camera and / or sensor data is used as input to a machine learning model that is trained to output gaze directions. The machine learning model may include models using neural networks, decision trees, linear regression, support vector machines, and the like. The machine learning model may be trained based on training data (e.g., ground truth data) that identifies eye characteristics for a given set of training inputs. For example, gaze directions may be manually marked by a training user who follows an item displayed on a display and derives a gaze direction corresponding to the display location of the item over time. The machine learning model may utilize images and light sensor data corresponding to multiple states of the system (e.g., each of the last 5 images and all light sensor data received during that time period) to predict eye characteristics for the current state and / or one or more future states of the system.

[0066] Figure 8 is a flow chart illustrating an exemplary method 800. In some implementations, a device (e.g., Figure 1 The device 10 of the embodiment of the present invention performs the technique of method 800 for initiating an action based on detecting that the gaze direction is approximately oriented toward the target area according to some specific implementations. In some specific implementations, the technique of method 800 is performed on a mobile device, a desktop computer, a laptop computer, an HMD, or a server device. In some specific implementations, method 800 is performed on a processing logic component (including hardware, firmware, software, or a combination thereof). In some specific implementations, method 800 is performed on a processor that executes code stored in a non-transitory computer-readable medium (e.g., a memory). In an exemplary specific implementation, the technique of method 800 is performed at an electronic device (e.g., an HMD) having a processor.

[0067] In some implementations, the HMD includes a waveguide as at least part of a display. In some implementations, as used herein, a waveguide can refer to an optical element including geometric, holographic, and / or diffractive optics. Additionally, a waveguide can include a waveguide region that is designed to emit or collect light from one portion of a device (e.g., device 210) to another portion of the device.

[0068] At block 802, method 800 generates reflections by directing light toward an eye using an illuminator. The illuminator generates light that is reflected from the retina of the eye. For example, Figure 1 As illustrated, the eye tracking system of device 10 uses a light source 32 that generates IR light and an image sensor (e.g., sensor 34 and / or sensor 36) to observe light scattered from the surface of the retina of eye 45 in order to collect data of reflections from the retina.

[0069] In some implementations, the light is IR light. In some implementations, the light source is an LED. Alternatively, another type of light source may be used that is sufficient to provide a retinal-based image when light from the light source is projected onto the eye.

[0070] At block 804, method 800 receives sensor data from a sensor, wherein a sensing direction of the sensor is substantially aligned with a direction from an eye to a target area. In some implementations, the direction may be an optical axis of a camera. For example, the sensor may be a low power sensor, such as a photodiode.

[0071] In some implementations, the direction of the light produced by the illuminator can be directed, or the light can be diffuse (e.g., a camera flash). In some implementations, there can be two light sources, such as in the upper temporal region (e.g., ST region 231) and another light source in the upper nasal region (e.g., SN region 233).

[0072] In some implementations, the device (device 210) receives an image of a portion of the retina from an image sensor, the image corresponding to multiple reflections of light scattered from the retina of the eye. For example, the sensor (e.g., sensor 330) can be an IR image sensor / detector. The sensor can be a low power sensor (e.g., low power sensor 36), such as a photodiode. For example, Figure 3A and Figure 3B As shown, the eye tracking system of example environment 300 uses a low power sensor 330 (eg, a 1-2 pixel receiver such as a photodiode) to observe light scattered from the surface of the retina of eye 45 in order to collect sensor data.

[0073] At box 806, method 800 determines the reflective properties of the reflection based on the sensor data. For example, the reflective properties (e.g., spectral properties) may indicate the amount of reflection of light from the fundus of the eye. The fundus is the back surface of the eye opposite the opening / pupil and may include the retina, macula, optic disc, fovea, and blood vessels. In some implementations, the amount of reflection indicates the level of alignment between the sensing direction and the gaze direction of the sensor. In an exemplary implementation, when the amount of reflection is greater, the eye is more aligned with the approximate sensor direction and the direction of the target area (e.g., target area 262) (also referred to as a "hot spot"). In some implementations, the more the fundus is aligned with the sensor direction, the greater the amount of reflection detected.

[0074] At block 808, method 800 detects that the gaze direction of the eye is oriented approximately toward a target region (e.g., a hotspot / hotspot) based on the reflective property. For example, the greater the amount of reflection detected, the more the gaze direction is aligned toward the target region. In some implementations, if the reflective property is within a threshold indicating that the gaze direction is within a threshold of the target direction (e.g., within 5 degrees), the gaze direction can be considered to be approximately toward the target region. In some implementations, the reflection can be used to generate a time-amplitude or frequency-time response to identify when the eye is approximately aligned (e.g., within an angle threshold) toward a hotspot.

[0075] In some embodiments, detecting that the gaze direction of the eye is oriented generally toward the target area includes determining that the reflective property is within an angular threshold (e.g., within 5 degrees), where the angular threshold indicates that the gaze direction is within a threshold of the target direction.

[0076] In some implementations, detecting that a gaze direction of the eye is oriented generally toward the target area includes generating a time-amplitude response based on the reflection to identify that the gaze direction is aligned generally toward the target area.

[0077] In some implementations, detecting that a gaze direction of an eye is oriented generally toward the target area includes generating a frequency-time response based on the reflection to identify when the gaze direction is aligned generally toward the target area.

[0078] In some implementations, detecting that the gaze direction of the eye is approximately oriented toward the target area is based on tracking the eye using a parameterized model of the eye. Figure 6 and Figure 7 As illustrated, models 600 and 700 may be utilized by using camera images and light sensor data to set parameters (eg, coefficients) of such models and thereby predict eye characteristics represented thereby.

[0079] In some implementations, the target area is located at one of a plurality of areas on a display of the device. For example, the target area can be located at a corner point of the display of the device. Figure 2B As illustrated, the target area may be located in one of the corner regions, such as ST region 231, SN region 233, IT region 236, or IN region 238 of device 210 (e.g., an HMD including a transparent substrate and a waveguide system for a display).

[0080] At block 810, method 800 initiates an action based on detecting that the gaze direction is generally oriented toward the target area. For example, a user may glance at a hot spot to trigger an action, such as initiating an XR experience using full eye tracking, providing a response to a notification, etc., as discussed herein. FIG. 5A to FIG. 5C exemplified in the sorted events of .

[0081] In some implementations, the action provides a user response to the notification. For example, the application may ask the user a question, such as "Do you want to proceed?". Then, the action may be to have the user glance at the target area where the word "yes" is displayed. For example, the application icon 504 may be replaced with the word "yes" or "no" to confirm whether the user wants to display the clock application 512.

[0082] In some implementations, if the gaze meets the criteria, an action can be initiated. For example, if the user's glance at the hotspot lasts for more than a threshold amount of time (such as longer than 2 seconds), an action can be initiated (e.g., starting an application on the display, such as FIG. 5A to FIG. 5C exemplified in ).

[0083] In some implementations, the action triggers the eye tracking algorithm based on sensor data received from a second sensor, where the first sensor is different from the second sensor and uses less power than the second sensor. Figure 1 As illustrated, the second sensor may include a high power sensor 34 (eg, a CMOS detector), and the first lower power sensor may include a low power sensor 36 (such as a photodiode) to obtain the eye gaze characteristic data 44 .

[0084] In some implementations, method 800 also includes detecting gaze directions toward multiple target areas using multiple illuminators and multiple sensors corresponding to multiple target areas, wherein the multiple illuminators generate light having different wavelengths, and wherein the multiple sensors distinguish reflections of light generated by the multiple illuminators based on wavelength. For example, an application may include different hotspots that initiate different actions. For example, a glance toward ST region 231 (upper left corner) may initiate a first application, a glance toward SN region 233 (upper right corner) may initiate a second application, and a glance toward IN region 238 (lower right corner) may initiate an exit all application action (e.g., close each open application).

[0085] In some implementations, the device includes a waveguide (e.g., Figure 4A and Figure 4B In some implementations, the waveguide substantially aligns a sensing direction of a sensor or a direction of light generated by an illuminator with a direction from an eye to a target area.

[0086] In some implementations, method 800 also includes determining eye characteristics based on the determined position of the user's adjustment depth. For example, for an eye tracking system, the eye characteristics may include gaze direction, eye orientation, etc. For example, if the electronic device is an HMD (e.g., device 210), the eye tracking system for the HMD may track the user's gaze direction, eye orientation, etc.

[0087] Fig. 9 is a block diagram of an example device 900. Device 900 illustrates an exemplary device configuration for device 10. Although certain specific features are illustrated, those skilled in the art will appreciate from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some embodiments, the device 10 includes one or more processing units 902 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 906, one or more communication interfaces 908 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 910, one or more displays 912, one or more internally-facing and / or externally-facing image sensors 914, memory 920, and one or more communication buses 904 for interconnecting these and various other components.

[0088] In some implementations, one or more communication buses 904 include circuits that interconnect system components and control communications between system components. In some implementations, one or more I / O devices and sensors 906 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0089] In some implementations, one or more displays 912 are configured to present a view of a physical environment or a graphical environment to a user. In some implementations, one or more displays 912 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS), and / or similar display types. In some implementations, one or more displays 912 correspond to diffraction, reflection, polarization, holographic, etc. waveguide displays. For example, device 10 includes a single display. In another example, device 10 includes a display for each eye of the user.

[0090] In some implementations, the one or more image sensor systems 914 are configured to obtain image data corresponding to at least a portion of the physical environment. For example, the one or more image sensor systems 914 include one or more RGB cameras (e.g., with complementary metal oxide semiconductor (CMOS) image sensors or charge coupled device (CCD) image sensors), monochrome cameras, IR cameras, depth cameras, event-based cameras, etc. In various implementations, the one or more image sensor systems 914 also include an illumination source that emits light, such as a flash. In various implementations, the one or more image sensor systems 914 also include an on-camera image signal processor (ISP) that is configured to perform multiple processing operations on the image data.

[0091] The memory 920 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices. In some specific implementations, the memory 920 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 920 optionally includes one or more storage devices remotely located from the one or more processing units 902. The memory 920 includes a non-transitory computer-readable storage medium.

[0092] In some implementations, the memory 920 or a non-transitory computer-readable storage medium of the memory 920 stores an optional operating system 930 and one or more instruction sets 940. The operating system 930 includes procedures for handling various basic system services and for performing hardware-related tasks. In some implementations, the instruction set 940 includes executable software defined by binary information stored in the form of electrical charge. In some implementations, the instruction set 940 is software that can be executed by one or more processing units 902 to implement one or more of the techniques described herein.

[0093] The instruction set 940 includes an illumination analysis instruction set 942 and an eye characteristics tracking instruction set 944. The instruction set 940 may be embodied as a single software executable file or as multiple software executable files.

[0094] In some implementations, the lighting analysis instruction set 942 can be executed by the processing unit 902 to generate a reflection by directing light toward an eye using an illuminator (e.g., the light can be IR), receiving sensor data from a sensor (e.g., the sensor can be a photodiode), and determining a reflective property (e.g., a spectral property) of the reflection based on the sensor data. For these purposes, in various implementations, the instructions include instructions and / or logic for the instructions as well as heuristics and metadata for the heuristics.

[0095] In some implementations, the eye characteristics tracking instruction set 944 can be executed by the processing unit 902 to track the user's eye gaze characteristics, such as using one or more of the techniques discussed herein or detecting the eye's gaze direction to be oriented generally toward a target area (e.g., a hotspot) based on reflective properties when other means may be appropriate. For these purposes, in various implementations, the instructions include instructions and / or logic for the instructions as well as heuristics and metadata for the heuristics.

[0096] Although the instruction set 940 is shown as residing on a single device, it should be understood that in other implementations, any combination of elements may be located in separate computing devices. Fig. 9 The instructions set is intended to be more of a functional description of the various features present in a particular implementation, as distinct from the structural schematics of the implementations described herein. As will be appreciated by one of ordinary skill in the art, items shown separately may be combined, and some items may be separated. The actual number of instruction sets and how features are distributed among them will vary depending on the implementation, and may depend in part on the specific combination of hardware, software, and / or firmware selected for a particular implementation.

[0097] Fig.10 A block diagram of an exemplary head mounted device 1000 according to some implementations is illustrated. The head mounted device 1000 includes a housing 1001 (or shell) that houses various components of the head mounted device 1000. The housing 1001 includes (or is coupled to) eye pads (not shown) disposed at the proximal (of the user 25) end of the housing 1001. In various implementations, the eye pads are plastic or rubber pieces that comfortably and snugly hold the head mounted device 1000 in place on the face of the user 25 (e.g., around the eyes of the user 25).

[0098] Housing 1001 houses display 1010, which displays images, emits light toward or onto the eyes of user 25. In various implementations, display 1010 emits light through an eyepiece having one or more optical elements 1005, which refract the light emitted by display 1010, so that the display appears to user 25 as a virtual distance farther than the actual distance from the eye to display 1010. For example, optical element 1005 may include one or more lenses, waveguides, other diffractive optical elements (DOEs), etc. To enable user 25 to focus on display 1010, in various implementations, the virtual distance is at least greater than the minimum focal length of the eye (e.g., 6 cm). In addition, to provide a better user experience, in various implementations, the virtual distance is greater than 1 meter.

[0099] The housing 1001 also houses a tracking system that includes one or more light sources 1022, a camera 1024, and a controller 1080. The one or more light sources 1022 emit light onto the eyes of the user 25, which is reflected as a light pattern (e.g., a flashing circle) that can be detected by the camera 1024. Based on the light pattern, the controller 1080 can determine the eye tracking characteristics of the user 25. For example, the controller 1080 can determine the gaze direction and / or blink state (eyes open or closed) of the user 25. As another example, the controller 1080 can determine the pupil center, pupil size, or focus point. Therefore, in various specific implementations, light is emitted by the one or more light sources 1022, reflected from the eyes of the user 25, and detected by the camera 1024. In various specific implementations, the light from the eyes of the user 25 is reflected from the hot mirror or passes through the eyepiece before reaching the camera 1024.

[0100] The display 1010 emits light within a first wavelength range, and the one or more light sources 1022 emit light within a second wavelength range. Similarly, the camera 1024 detects light within the second wavelength range. In various implementations, the first wavelength range is a visible wavelength range (e.g., a wavelength range of approximately 400 nm to 800 nm within the visible spectrum), and the second wavelength range is a near infrared wavelength range (e.g., a wavelength range of approximately 700 nm to 1400 nm within the near infrared spectrum).

[0101] In various implementations, eye tracking (or specifically, determined gaze direction) is used to enable user interaction (e.g., user 25 selecting an option on display 1010 by looking at it), provide foveated rendering (e.g., presenting a higher resolution in the area of ​​display 1010 that user 25 is looking at and a lower resolution elsewhere on display 1010), or correct distortion (e.g., for images to be provided on display 1010).

[0102] In various implementations, one or more light sources 1022 emit light toward the eyes of user 25, which is reflected in the form of multiple flashes.

[0103] In various implementations, camera 1024 is a frame / shutter based camera that generates images of the user's 25 eyes at a particular point in time or multiple points in time at a frame rate. Each image includes a matrix of pixel values ​​corresponding to pixels of the image, the pixels corresponding to locations of the camera's light sensor matrix. In a specific implementation, each image is used to measure or track pupil dilation by measuring changes in pixel intensity associated with one or both of the user's pupils.

[0104] In various implementations, camera 1024 is an event camera comprising multiple light sensors (e.g., a light sensor matrix) at multiple corresponding positions, which generates an event message indicating a specific position of a particular light sensor in response to the particular light sensor detecting a change in light intensity.

[0105] Back to Figure 1 , the physical environment refers to the physical world that people can sense and / or interact with without the help of electronic devices. The physical environment may include physical features, such as physical surfaces or physical objects. For example, the physical environment corresponds to a physical park including physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell. In contrast, the extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, the XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and the like. In the case of an XR system, a subset of a person's physical movements or a representation thereof is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that complies with at least one physical law. As an example, the XR system may detect head movement, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in the physical environment. As another example, the XR system may detect movement of an electronic device (e.g., a mobile phone, tablet, laptop, etc.) presenting the XR environment, and in response, adjust the graphical content and sound fields presented to a person in a manner similar to how such views and sounds would change in a physical environment. In some cases (e.g., for accessibility reasons), the XR system may adjust characteristics of the graphical content in the XR environment in response to representations of physical movement (e.g., voice commands).

[0106] There are many different types of electronic systems that enable people to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablet computers, and desktop / laptop computers. A head-mounted system may have an integrated opaque display and one or more speakers. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of a physical environment and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system may have a transparent or translucent display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eyes. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium may be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In some implementations, a transparent or translucent display may be configured to selectively become opaque. A projection-based system may employ retinal projection technology that projects a graphic image onto a person's retina. The projection system may also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0107] It should be understood that the specific implementations described above are cited by way of example, and the present disclosure is not limited to what has been particularly shown and described above. On the contrary, the scope includes both combinations and sub-combinations of the various features described above, as well as variations and modifications of the various features that would occur to a person skilled in the art upon reading the foregoing description and that are not disclosed in the prior art.

[0108] As described above, one aspect of the present technology is to collect and use physiological data to improve the user's electronic device experience in interacting with electronic content. The present disclosure contemplates that, in some cases, the collected data may include personal information data that uniquely identifies a particular person or can be used to identify the interests, characteristics, or tendencies of a particular person. Such personal information data may include physiological data, demographic data, location-based data, telephone numbers, email addresses, home addresses, device characteristics of personal devices, or any other personal information.

[0109] The present disclosure recognizes that the use of such personal information data in the present invention technology can be used to benefit users. For example, personal information data can be used to improve the interaction and control capabilities of electronic devices. Therefore, the use of such personal information data enables planned control of electronic devices. In addition, the present disclosure also anticipates other uses of personal information data that benefit users.

[0110] The disclosure also contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage or other purposes of such personal information and / or physiological data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to the use of privacy policies and practices that are recognized as meeting or exceeding the industry or government requirements for the privacy and security of maintaining personal information data. For example, personal information from users should be collected for the legal and reasonable purposes of entities, and not shared or sold outside these legal purposes. In addition, such collection should only be carried out after the user's informed consent. Additionally, such entities should take any required steps to safeguard and protect access to such personal information data, and ensure that other people who can access personal information data comply with their privacy policies and procedures. In addition, such entities can subject themselves to third-party assessments to prove that they comply with widely accepted privacy policies and practices.

[0111] Regardless of the foregoing, the present disclosure also contemplates specific implementations in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates that hardware elements or software elements may be provided to prevent or block access to such personal information data. For example, with respect to a content delivery service customized for a user, the technology of the present invention may be configured to allow a user to choose to "opt in" or "opt out" of participating in the collection of personal information data during registration for the service. In another example, a user may choose not to provide personal information data for a target content delivery service. In yet another example, a user may choose not to provide personal information, but allow anonymous information to be passed for use in improving the functionality of the device.

[0112] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may also be implemented without access to such personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or a portion of such personal information data. For example, content may be selected and delivered to a user by inferring preferences or settings based on non-personal information data or an absolute minimum amount of personal information, such as content requested by a device associated with the user, other non-personal information available to a content delivery service, or publicly available information.

[0113] In some embodiments, data is stored using a public / private key system that only allows the owner of the data to decrypt the stored data. In some other implementations, data can be stored anonymously (e.g., without identification and / or personal information about the user, such as legal name, username, time and location data, etc.). In this way, other users, hackers, or third parties cannot determine the identity of the user associated with the stored data. In some implementations, a user can access his or her stored data from a user device that is different from the user device used to upload the stored data. In these cases, the user may need to provide login credentials to access the data they store.

[0114] Numerous specific details are set forth herein to provide a comprehensive understanding of the claimed subject matter. However, one skilled in the art will appreciate that the claimed subject matter may be practiced without these specific details. In other instances, methods, devices, or systems known to those of ordinary skill are not described in detail so as not to obscure the claimed subject matter.

[0115] Unless otherwise specifically noted, it should be understood that throughout the specification, discussions utilizing terms such as "process," "compute," "calculate," "determine," and "identify" refer to the actions or processes of a computing device, such as one or more computers or similar electronic computing devices, that manipulate or transform data represented as physical electronic or magnetic quantities within a memory, register, or other information storage device, transmission device, or display device of a computing platform.

[0116] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems that access stored software that programs or configures the computing system from a general-purpose computing device to a dedicated computing device that implements one or more specific implementations of the subject matter of the present invention. Any suitable programming, scripting, or other type of language or combination of languages ​​may be used to implement the teachings contained herein in software for programming or configuring a computing device.

[0117] The specific implementation of the method disclosed herein can be performed in the operation of such a computing device. The order of the boxes presented in the above examples can be changed, for example, the boxes can be reordered, combined or divided into sub-boxes. Some boxes or processes can be executed in parallel.

[0118] The use of "suitable for" or "configured to" herein is meant to be open and inclusive language that does not exclude devices that are suitable for or configured to perform additional tasks or steps. Additionally, the use of "based on" is meant to be open and inclusive, as a process, step, calculation, or other action "based on" one or more stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbers included herein are for ease of explanation only and are not intended to be limiting.

[0119] It will also be understood that, although the terms "first", "second", etc. may be used to describe various objects in this article, these objects should not be limited by these terms. These terms are only used to distinguish one object from another object. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0120] The terms used herein are only for the purpose of describing a specific implementation and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "one", "an" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the terms "including" or "comprising" when used in this specification specify the presence of stated features, integers, steps, operations, objects or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, objects, components or their groupings.

[0121] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

[0122] The foregoing description and summary of the present invention should be understood to be illustrative and exemplary in every aspect, rather than restrictive, and the scope of the present invention disclosed herein is determined not only by the detailed description of the exemplary specific implementations, but by the full breadth allowed by the patent law. It should be understood that the specific implementations shown and described herein are only illustrative of the principles of the present invention, and that various modifications can be implemented by those skilled in the art without departing from the scope and essence of the present invention.

Claims

1. A method, comprising: At an electronic device having a processor and a display: Reflections are created by directing light toward the eye using an illuminator; receiving sensor data from a sensor, wherein a sensing direction of the sensor is substantially aligned with a direction from the eye to a target area; determining a reflective property of the reflection based on the sensor data; as well as A gaze direction of the eye is detected based on the reflective property and is oriented generally toward the target area.

2. The method according to claim 1, further comprising: An action is initiated based on detecting that the gaze direction is oriented generally toward the target area.

3. The method according to any one of claims 1 to 2, wherein the reflective property indicates an amount of reflection of the light from the fundus of the eye. The method of claim 3 , wherein the amount of the reflection indicates a level of alignment between the sensing direction of the sensor and the gaze direction.

5. A method according to any one of claims 1 to 4, wherein detecting that the gaze direction of the eye is approximately oriented toward the target area includes determining that the reflective property is within an angular threshold, wherein the angular threshold indicates that the gaze direction is within a threshold of the target direction.

6. The method according to any one of claims 1 to 4, wherein detecting that the gaze direction of the eye is oriented substantially toward the target area comprises generating a time-amplitude response based on the reflection to identify that the gaze direction is aligned substantially toward the target area.

7. A method according to any one of claims 1 to 4, wherein detecting that the gaze direction of the eye is approximately oriented toward the target area includes generating a frequency-time response based on the reflection to identify when the gaze direction is approximately aligned toward the target area.

8. The method according to any one of claims 1 to 4, wherein detecting that the gaze direction of the eye is substantially oriented towards the target area is based on using a parameterized model of the eye.

9. The method of any one of claims 1 to 8, wherein the sensor is a first sensor and the action triggers an eye tracking algorithm based on sensor data received from a second sensor, wherein the first sensor is different from the second sensor and uses less power than the second sensor.

10. The method according to any one of claims 1 to 9, wherein the action is initiated based on whether the gaze direction satisfies a criterion.

11. A method according to any one of claims 1 to 10, wherein the action provides a user response to the notification.

12. The method of any one of claims 1 to 11, wherein the action comprises initiating an extended reality (XR) experience.

13. The method according to any one of claims 1 to 12, wherein the target area is located at one of a plurality of zones on a display of the electronic device.

14. The method according to any one of claims 1 to 13, wherein the sensor is a photodiode.

15. The method of any one of claims 1 to 14, wherein the light is infrared (IR) light.

16. The method according to any one of claims 1 to 15, further comprising detecting a gaze direction toward a plurality of target areas using a plurality of illuminators and a plurality of sensors corresponding to a plurality of target areas, wherein the plurality of illuminators generate light having different wavelengths, wherein the plurality of sensors distinguish reflections of the light generated by the plurality of illuminators based on the wavelength.

17. The method according to any one of claims 1 to 16, wherein the electronic device is a head mounted device (HMD).

18. The method of claim 17, wherein the HMD comprises a waveguide, wherein the waveguide substantially aligns the sensing direction of the sensor or the direction of the light generated by the illuminator with the direction from the eye to the target area.

19. A device, comprising: a non-transitory computer-readable storage medium; and One or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the device to perform the method of any one of claims 1 to 18.

20. A non-transitory computer-readable storage medium storing program instructions executable on a device to perform the method according to any one of claims 1 to 18.

Images