Multimodal eye tracking

Abstract

A method is disclosed, the method comprising the following steps: receiving first data from a first sensor at a first time interval, the first sensor being configured to output data indicating a first position of an eye; receiving second data from a second sensor at a second time interval, the second sensor being configured to output data indicating an incremental position of the eye; determining a first position of the eye based on the first data; determining an incremental position of the eye based on the second data; determining a second absolute position of the eye using the first position of the eye and the incremental position of the eye; and generating an output signal indicating the second position of the eye in response to determining the second position of the eye.

Description

[0001] This application is a divisional application of the Chinese patent application “Multi-modal eye tracking” with application number 201880024458.7 (filing date April 13, 2018).

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of U.S. Patent Application No. 62 / 485,820, filed April 14, 2017, at 35 U.S.C. 119(e), which is hereby incorporated by reference in its entirety for all purposes. Technical Field

[0004] Examples of the present disclosure relate to systems and methods for tracking human eyes and, more particularly, for determining and / or characterizing the position, movement, and / or behavior of human eyes. Background Art

[0005] In general, an eye tracking system generates one or more signals corresponding to the position and / or movement of a user's eyes. (Throughout this disclosure, "position" should be understood to include position and / or orientation, and "motion" should be understood to include movement and / or rotation of the eyes.) These signals can be used as input to various computer systems and in various applications such as gaming, navigation, sports training, communications, and medical research; or in other situations where it is helpful to know where the user is looking. In particular, eye tracking systems can be used in 3D virtual environments, such as those employed by some "augmented reality" (AR) systems, where knowledge of the user's eye movements can enhance the sense of immersion in the virtual environment. In some examples, the eye tracking system involves a mobile device, such as a head-mounted device, with sensors oriented toward the wearer's eyes.

[0006] It is desirable for an eye tracking system to accurately reflect eye position and movement, even in dynamic or unpredictable situations, such as changing weather and lighting conditions. In addition, as with most computer systems, it is desirable to reduce the power consumed by the eye tracking system, for example to preserve battery life in mobile systems. These goals are not always compatible: for example, high-resolution optical scanning may produce accurate eye tracking results, but at the expense of high power consumption. Updating sensor data at a low refresh rate can save power, but cannot accurately capture high-frequency eye movements. The present disclosure relates to multimodal systems and methods for combining sensors (e.g., optical sensors and electro-ocular voltage sensors) to enhance the accuracy and / or power consumption of eye tracking systems. The present disclosure further relates to systems and methods for using such sensors to characterize the behavior of the eye (which information can be used to further enhance eye tracking accuracy and / or power consumption). Summary of the invention

[0007] Examples of the present disclosure describe systems and methods for tracking human eyes. According to examples of the present disclosure, sensors such as optical sensors and electrooculographic voltage sensors can be combined to enhance eye tracking, for example, by improving accuracy and power consumption. For example, first data indicating a first position (e.g., an absolute position) of an eye can be received from a first sensor at a first time interval. Second data indicating an incremental (delta) position of the eye can be received from a second sensor at a second time interval. The first position and the incremental position can be used to determine a second position (e.g., an absolute position) of the eye. The sensor can be further used (e.g., through machine learning techniques) to characterize various behaviors of the eye, and this information can be used to further enhance eye tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A cross section of a human eye is shown.

[0009] Figure 2A An example of electrodes configured to detect eye movements according to an example of the present disclosure is shown.

[0010] Figure 2B An example head mounted device according to examples of the present disclosure is shown.

[0011] Figure 3A and Figure 3B An example eye tracking system is shown that uses multiple sensors to output signals corresponding to the position and / or movement of the eyes according to examples of the present disclosure.

[0012] Figure 4 An example of a system architecture that may be embodied within any portable or non-portable device according to examples of the present disclosure is shown. DETAILED DESCRIPTION

[0013] In the following description of the examples, reference is made to the accompanying drawings which form a part hereof, and in which is shown by way of illustration specific examples that may be practiced. It should be understood that other examples may be used and structural changes may be made without departing from the scope of the disclosed examples.

[0014] Eye tracking systems must address the challenge of reliably obtaining accurate data from one of the most dynamic and variable parts of the body. Furthermore, such systems can be made to perform tasks unobtrusively, under unpredictable lighting conditions, and in harsh physical environments with minimal power consumption. Examples of the present disclosure relate to multimodal systems and methods for combining sensors (e.g., optical sensors and electro-oculographic voltage sensors) to address these challenges, as described below.

[0015] Ophthalmology

[0016] Figure 1 1 is a cross-sectional view of a human eye (100), and is depicted as including a cornea (102), an iris (104), a lens (106), a sclera (108), a choroid layer (110), a macula (112), a retina (114), and an optic nerve (116) to the brain. Light enters the front of the eye (100) at the cornea (102). The iris (104) regulates the amount of light received on the retina (114), and the lens (106) can regulate and focus the image at the retina (114). In turn, the retina (114) converts the visual stimulus into an electrical signal (e.g., a receptor potential), which is sent to the visual cortex via the optic nerve (116). The macula (112) is the center of the retina (114), which is used to see moderate detail. At the center of the macula (112) is the relatively small fovea, which is used to see fine detail and contains more light-sensitive receptors (about 120 cones per diopter) than any other part of the retina (114). The eye (100) can be rotated to provide six degrees of freedom by two pairs of direct muscles and a pair of oblique muscles (not shown). The optic nerve (116) is surrounded by the muscles of the eye.

[0017] The human visual system is configured to actively scan the environment by the activity of muscles coupled to the eyes, which include photoreceptors that generate neural signals in response to the light incident on the retina. The eyes can use these muscles to make many different movements. These include at least small movements (e.g., tremors), faster tracking movements (e.g., smooth pursuit) and very fast movements (e.g., glances, twitches (ballistic)). Some movements may be autonomous, most movements are involuntary, and other movements may be voluntary. As discussed herein, eye movements can at least refer to the rotation of the eyes around the horizontal axis, the (initial) vertical axis that rotates around the horizontal axis with the sphere, and the torsion axis along the gaze angle.

[0018] Some eye movements can be described as saccadic eye movements. Saccades are rapid conjugate twitching movements of the eyes that suddenly change the point of fixation. Saccades can involve the movement of the eyes at a speed of 900 degrees per second. Typically, saccadic eye movements bring the object of interest into the field of vision. For example, when reading a book, the eyes will have rapid saccadic movements and stop several times, moving quickly between each point of fixation. In another example, a vehicle driver will perform saccadic eye movements to look at other cars on the road, traffic signs, car interiors, etc. Rapid eye movement can make the fovea image different parts of the object. Saccadic eye movements can be voluntary, reflected as a response to visual stimulation, and / or corrected such as in response to optokinetics or vestibular movements. For example, reflexive saccades can be triggered by the disappearance of external stimulation or fixation stimulation. Reverse saccades can automatically move the eyes away from visual stimulation. Scanning saccades can be voluntary and allow different parts of the visual environment to be checked. Memory saccades can move the eyes to a memory point. Predictive saccades can anticipate the movement of the object of interest. Saccades can also occur during the rapid eye movement stage of sleep.

[0019] Some eye movements can be described as smooth pursuit movements, which are conjugate eye movements that smoothly track a moving visual object of focus over a range of about 1 degree per second to about 100 degrees per second to keep the image of the object steady on the retina. Smooth pursuit movements are generally not under voluntary control.

[0020] Certain eye movements may be described as fixations, which are a stationary state of the eye in which the eye holds the image of an object of interest on the retina. Fixations may last from about 100 milliseconds to about 1000 milliseconds.

[0021] Some eye movements can be described as nystagmus. Nystagmus is a form of involuntary eye movement that includes alternating between slow and fast phases. Nystagmus eye movements can be described as optokinetic or vestibular movements. Optokinetic nystagmus refers to stationary visual receptors and rapidly moving objects of interest. Optokinetic nystagmus can have a characteristic sawtooth pattern of eye movements; it includes a slow phase and a fast phase (i.e., a return saccade (saccadic jump)), in which the eyes are fixed on a part of the moving field (field) and then make a pursuit movement, and in which the eyes are fixed on a new part of the field. Vestibular nystagmus can occur in response to movement of the head to stimulate the semicircular canals of the inner ear. Sensory information from the semicircular canals can guide the eyes to move in the direction opposite to the head movement, thereby roughly keeping the image of the object of interest on the retina.

[0022] Some eye movements can be described as vestibular eye movements. Vestibular eye movements refer to movements of the head and / or body (e.g., neck) combined with movements of the eyes. Vestibular eye movements can reduce strain on the eyes by allowing the large muscles of the head and neck to assist in large or rapid movements, for example, when viewing objects in the periphery of the visual field or when tracking objects that move rapidly across the visual field. For example, most people prefer to move their heads when their gaze needs to deviate more than about 20 degrees from center to focus on a particular object of interest. Head movement also benefits the human visual system from depth cues in the form of head motion parallax, which can help identify the relative depth of objects in the visual field. The vestibular-ocular reflex is used to coordinate head and eye movements to stabilize image information relative to the retina during head rotation.

[0023] The eyes can engage in a combination of various types of eye movements (e.g., the types of eye movements described above). For the purposes of the following disclosure, eye behavior is one or more eye movements (or a pattern of one or more eye movements), and may include head movements, such as in the case of vestibular eye movements. Which eye behavior a person's eyes engage in is influenced by several external factors, such as the following: the activity the person is engaged in (e.g., reading, driving); the person's location and surroundings (e.g., in a quiet library, in busy vehicle traffic); and environmental factors (e.g., ambient lighting conditions, temperature). Understanding these external factors can help predict human eye behavior. For example, knowing that someone is reading a book can indicate that the person's eyes are performing a saccadic motion. Conversely, understanding human eye behavior can indicate various external factors. For example, knowing that someone's eyes are performing a saccadic motion can indicate that the person is reading a book.

[0024] Eye Tracking

[0025] There are several techniques for obtaining measurements related to the position of a user's eyes.

[0026] Some such technologies include optical sensors, such as cameras. For example, light can be reflected from the eye and sensed by an optical sensor to detect eye position and movement. Other such technologies may involve measuring the electrical potential that may exist between two locations of the human eye. For example, referring to Figure 1 , the cornea (102) of the eye (100) is electrically positive relative to the retina (114) and forms a stable electric potential field, which can be described as a fixed dipole with a positive pole at the cornea (102) and a negative pole at the retina (114). When the eye rotates, the electrostatic dipole rotates with it. This corneal retinal potential typically has a range of 0.4 mV to 1.0 mV and is independent of light stimulation.

[0027] Electrooculography is a technique that uses an electrooculography (EOG) sensor to detect the electrical potential between two locations of the eye (e.g., the cornea and Bruch's membrane). The sensing circuit may include one or more electrodes and, in some examples, one or more electrical components configured to measure the potential difference between the electrodes. The sensing circuit may be placed on the head and / or face to record the potential difference. Figure 2A An example electrode configuration in an example sensing circuit is shown. The amplitude of the electric potential of the eye is typically in the range of μV / degree, and can be in the range of, for example, about 5μV / degree to about 20μV / degree. The sign of the electrode potential change can indicate the direction of eye movement (e.g., left, right, up, down). The relative position of the eye can be inferred from the measured potential. Facial muscles can affect the electric potential on the facial surface when they are active. The voltage seen on the electrode pairs on the face can be represented as a convolution of eye muscle movement and facial muscle movement corresponding to facial features (e.g., blinking, winking, or facial wrinkling). For example, blinking has a characteristic signal pattern and duration observed on the eye movement electrogram. The average blink rate at rest can vary between about 12 and 19 blinks per minute, with an average duration between about 100 milliseconds and about 400 milliseconds. The electric potential of the eye can further provide insights into the user's mentality, including, for example, emotional state. (Although reference is made to detecting an electrical potential in the examples of this disclosure, in some examples, other suitable electrical characteristics (e.g., capacitance, inductance, resistance, impedance) may be detected instead of or in addition to an electrical potential for the purpose of eye tracking.)

[0028] Multimodal eye tracking system

[0029] EOG sensors and optical sensors can each have certain advantages relative to each other. For example, EOG sensors are generally more power efficient than optical sensors. In addition, EOG sensors may not be as prominent as optical sensors, may not impair the user's vision, and may be more compatible with corrective lenses such as glasses and contact lenses. Signal measurements using electrodes can have high temporal resolution and allow continuous signals. Unlike optical sensors, EOG sensors are generally not affected by bright light or darkness and can operate without controlled lighting. In addition, unlike optical trackers, eye movements can be tracked even when the eyes are closed or in other cases where the eyes are blocked from view (e.g., by eyelids, eyelashes, etc.). In addition, EOG sensors that generate outputs representing electrical potentials may not be as bandwidth-intensive as optical sensors, which can output relatively large image data. Moreover, processing EOG data can be faster and more computationally efficient than optical sensor data. However, such optical sensor data (e.g., 2D images) can also provide eye tracking with enhanced resolution and / or accuracy. That is, the position of the eye can be measured more reliably using an optical sensor (i.e., by extracting the eye position from image data) than using an EOG sensor (i.e., by inferring the eye position from electrical potentials). Likewise, optical sensors can benefit from limited drift in calibration values ​​compared to EOG sensors.

[0030] Given the relative advantages of optical eye tracking and EOG eye tracking, it may be beneficial for an eye tracking system to incorporate multiple types of sensors (e.g., optical sensors and EOG sensors). As described in further detail below, such a multimodal eye tracking approach may be particularly beneficial, for example, in systems with eye tracking functionality that may routinely consume relatively large amounts of power and / or routinely perform computationally intensive operations. Such systems may include, for example, augmented reality systems, virtual reality systems, and the like.

[0031] Figure 2B An exemplary head mounted device 200 including a multi-modal eye tracking system is shown. More specifically, in Figure 2B In the example of , the exemplary head mounted device 200 includes sensing circuitry including electrodes configured to contact the head and / or face of the wearer (e.g., to provide EOG sensing functionality) and / or at least one optical sensor configured to monitor one or both eyes of the wearer. The example device 200 can be used in an augmented reality system and can, for example, incorporate a display configured to present an augmented reality display to the wearer. Additional details regarding systems and techniques for providing electro-oculography (EOG), optical eye tracking, and augmented reality functionality are provided in U.S. Patent Application No. 15 / 072,290, which is incorporated herein by reference in its entirety.

[0032] In some examples, the Figure 2A At least some of the electrodes of the sensing circuit of the exemplary head mounted device 200 are arranged according to the exemplary electrode configuration shown in FIG. Figure 2A In the example of the example device 200, a first pair of electrodes (210 and 212) are configured to detect horizontal axis movement of the eye, while a second pair of electrodes (220 and 222) are configured to contact the skin to detect vertical movement of the eye. In the example shown, electrodes 210 and 212 are configured to contact the skin at the inner and outer canthuses of the left eye, respectively, and electrodes 220 and 222 are configured to contact the skin above and below the left eye, respectively. The electrodes should be placed as close to the eyes as possible to reduce interference. In the example device 200, electrodes 210 and 212 are shown as being configured to contact the wearer's face around the outer canthuses of the right and left eyes, respectively, to detect horizontal eye movement as described above. In the example device 200, electrode 220 is shown as being configured to contact the wearer's face above the right eyebrow, and electrode 222 is shown as being configured to contact the wearer's face below the right eyelid to detect vertical eye movement as described above. Referring again to Figure 2A In some examples, electrodes 210 and 212 and 220 and 222 are coupled to differential amplifiers 232 and 234, respectively, to amplify the signals generated by the electrodes. Thus, in some examples, example device 200 or another device operably coupled to example device 200 may include differential amplifiers 232 and 234 and / or a device for adjusting the combination Figure 2B Other circuits for generating signals from two or more electrodes are shown.

[0033] In addition, the sensing circuit may include a ground electrode to provide a reference potential. It may be desirable to place the ground electrode on an area of ​​the head or face where the potential changes little in response to eye movement (if any). For example, in the example device 200, electrode 240 represents a ground electrode configured to contact the back of the wearer's head. In some examples, the ground electrode 240 or another ground electrode may be configured to contact an area of ​​the head or face including an earlobe, a forehead, or one or more anatomical regions adjacent to the earlobe or forehead. In the example device 200, a third electrode 250 and a fourth electrode 252 are shown, which are configured to contact the bridge of the nose and / or the inner canthus of the eye. The EOG data from electrodes 250 and 252 can supplement the data provided by electrodes 210, 212, 220, and / or 222, which can simplify the processing of data from these electrodes, can provide data redundancy, and / or can improve the robustness of the system. Other electrodes can also be incorporated to provide similar benefits. The electrodes can be wet electrodes and / or dry electrodes. In some examples, the electrodes can be made of silver-silver chloride and / or gold-plated. In some examples, shielding and / or noise cancellation techniques, such as incorporating a common-mode rejection preamplifier, may be used to reduce electromagnetic interference.

[0034] In some examples, such as Figure 2B In the example device 200 shown in FIG. 1 , an optical sensor (eg, Figure 2B The optical sensor 260 in FIG. 260 may be mounted to a head mounted device, such as may be used in an augmented reality system. This may provide the following advantages: the optical sensor is relatively fixed relative to the eye, which minimizes the need to calibrate the sensor and may simplify the process of analyzing the output of the optical sensor. In some examples, the optical sensor may be accompanied by a light source (e.g., Figure 2B The head mounted device may include a light source 270 in the optical sensor sample to illuminate the eye and provide controlled, consistent illumination on the optical sensor sample. In some examples, the head mounted device may include a display through which, for example, augmented reality content may be presented to the user. Figure 3A-3B As described in more detail, in a multimodal eye tracking system that includes optical and EOG sensors (e.g., the optical and EOG sensors of example device 200), the optical and EOG sensors can be utilized in a manner that exploits the above-described relative advantages of the optical and EOG sensors to improve the overall power efficiency and / or computational efficiency of the system.

[0035] Figure 3A and 3B An exemplary multimodal eye tracking system 300 is shown that uses multiple sensors to output a signal 390 corresponding to the position and / or movement of the eye 100 according to an example of the present disclosure. In some examples, the multimodal eye tracking system 300 may include or may be included as a head mounted device (such as a head mounted device) having a sensor (e.g., an EOG sensor and / or an optical sensor) configured to output data related to the eye 100. Figure 2B The multimodal eye tracking system 300 may be incorporated into an augmented reality system and may include a display configured to present augmented reality content to a user.

[0036] In the example system 300, power and computing resource efficiency can be achieved by controlling the rate at which sensor measurements of the eye 100 are taken. In general, increasing the sensor measurement rate can improve eye tracking accuracy, but at the expense of consuming more power; conversely, reducing the sensor measurement rate can use less power, but will compromise eye tracking accuracy. The degree to which the sensor measurement rate affects the tradeoff between accuracy and power consumption will change during system operation. In some examples (such as the example system 300), the sensor measurement rate can be continuously calculated and adjusted in real time during system operation to maintain the desired tradeoff between accuracy and power consumption. For example, the example system 300 includes an eye data analyzer 340 that can perform such calculations and adjustments. The exemplary multi-modal eye tracking system 300 includes an optical sensor and an EOG sensor. However, in some examples, other sensors can be used. In addition, in some examples, more than two sensors can be used. For example, an accelerometer can be used in conjunction with an optical sensor and an EOG sensor to detect head movement, as described below. The following examples can be expanded to accommodate additional sensors.

[0037] Figure 3A Shows Figure 3A and 3BA high-level overview of an example system 300 of the present disclosure is provided. In this example, an optical sensor loop 310A and an EOG sensor loop 310B obtain corresponding sensor data from an eye 100 and output an optical signal 320A and an EOG signal 320B, respectively. An eye signal processor 330 uses the optical signal 320A and the EOG signal 320B as inputs and outputs an output signal 390 based on an eye position determined from the signals 320A and 320B. In some examples of the present disclosure (e.g., the example system 300), an eye data analyzer 340 inputs eye position data from the signal processor 330 and uses the data to determine parameters that affect the operation of the optical sensor loop 310A and the EOG sensor loop 310B. In some examples of the present disclosure (e.g., the example system 300), the eye data analyzer 340 receives additional sensor data 350 and uses the data to determine parameters that affect the operation of the optical sensor loop 310A and the EOG sensor loop 310B. For example, the additional sensor data 350 may be data from an accelerometer (e.g., configured to sense movement of a user's head), a GPS sensor, or another device. Examples of devices that may generate the additional sensor data 350 are further described below. In some examples of the present disclosure (e.g., the example system 300), the eye data analyzer 340 receives the prediction data 360 and uses the data to determine parameters that affect the operation of the optical sensor loop 310A and the EOG sensor loop 310B. For example, the prediction data 360 may be the output of a neural network configured to help the eye data analyzer 340 determine eye behavior corresponding to one or more input signals.

[0038] One or more processes described herein with respect to the example system 300 (e.g., processes that may be performed by the eye signal processor 330 and / or the eye data analyzer 340) may be implemented in a computer system using any suitable logic circuitry. Suitable logic circuitry may include one or more computer processors (e.g., CPUs, GPUs, etc.) that perform such processes when executing instructions implemented in software programs. In addition, such processes may also be implemented by corresponding logic designs implemented in hardware logic circuits, such as programmable logic (e.g., PLDs, FPGAs, etc.) or custom logic (e.g., ASICs, etc.) that implement logic designs that provide such processes. In addition, such processes may be provided by combining the implementation of both one or more processors running software and hardware logic circuits. In some examples, components of the example system 300 (e.g., the eye signal processor 330 and / or the eye data analyzer 340) may correspond to dedicated hardware units, such as computer processors configured to perform the functions of the eye signal processor 330. In some examples, components of the example system 300 may correspond to logic units implemented across one or more hardware units. In some examples, a single hardware unit (e.g., a computer processor) may perform all of the functionality described herein with respect to multiple components (e.g., the eye signal processor 330 and the eye data analyzer 340) of the example system 300. In some examples, multiple hardware units (e.g., multiple computer processors) may collectively perform the functionality described herein with respect to a single component (e.g., the eye signal processor 330) of the example system 300. The present disclosure is not limited to any particular implementation.

[0039] Figure 3B Shown in more detail Figure 3A Example system 300 of FIG. 10A. In the example shown, optical sensor circuit 310A includes logic for controlling when the optical sensor obtains updated measurements (318A) from eye 100. As described above, since obtaining optical sensor measurements can be a power intensive operation, it can be desirable to conserve power by limiting the rate at which such updates are performed. In this example, in optical sensor circuit 310A, the interval value int can be at least partially determined by the interval value int. opt (312A) to control the rate at which optical sensor measurements are obtained. In some examples, int opt The value of may be fixed, for example, fixed at a specific time value or a specific number of iterations of the optical sensor circuit 310A. opt The value of can be, for example, based on the function f opt As described in more detail below, the function f opt Various parameters (e.g., parameters x and y) that identify when and how it is desirable to adjust the update rate of the optical sensor measurements may be reflected. optSome examples of time values ​​are int opt can decrease over time during system operation ( Figure 3B ), so that int opt Indicates the remaining time until the optical sensor measurement should be updated. In stage 314A of this example, the optical sensor circuit 310A may read int opt , and uses that value to determine whether the optical sensor measurement should be updated. For example, if int opt is equal to or below a threshold value (eg, zero), the optical sensor circuit 310A may determine at 314A that an updated optical sensor measurement should be obtained (318A), and may int opt reset to a value above the threshold. Also, if int opt is above the threshold, then at 314A the optical sensor circuit 310A may not continue to acquire optical sensor measurements, but may wait for int opt reaches a threshold. In some examples, if the int opt above the threshold, the optical sensor circuit 310A may enter a low power mode (316A) instead of acquiring optical sensor measurements until int opt reaches a threshold or falls below a threshold. The low power mode can result in energy saving behaviors, such as suspending a processor thread associated with the optical sensor. In some examples, optical sensor measurements can be obtained when an event (e.g., user input) occurs, regardless of the int opt This optical sensor measurement can be obtained to replace or replace once the int opt In an iteration of optical sensor loop 310A in which an optical sensor measurement is obtained, the measurement is output as optical signal 320A.

[0040] exist Figure 3BIn the example system 300 shown, the EOG sensor loop 310B includes logic that controls when the EOG sensor obtains updated measurements (318B) from the eye 100. In the example shown, the EOG sensor loop 310B largely mirrors the optical sensor loop 310A. As described above, the optical sensor loop 310A can intentionally limit the rate at which optical sensor measurements are obtained to reduce power consumption. During time periods when no optical sensor measurements are obtained, it can be desirable to obtain relatively more power-efficient EOG sensor measurements rather than optical measurements. In addition, some examples can detect measurement problems related to the optical sensor (e.g., if the measurement is compromised by occlusion of an object such as an eyelid or extreme lighting conditions) and compensate by adjusting the rate at which the EOG sensor measurements are updated. In this example, in the EOG sensor loop 310B, similar to the optical sensor loop 310A, the rate at which the EOG sensor measurements are updated can be at least partially determined by the interval value int eog (312B) to control the rate at which EOG sensor measurements are obtained. In some examples, int eog The value of may be fixed, for example, fixed at a specific time value or a specific number of iterations of the EOG sensor loop 310B. In some examples, int eog The value of can be, for example, based on the function f eog As described in more detail below, the function f eog Various parameters (e.g., parameters x and y) that identify when and how you wish to adjust the rate at which you update the EOG sensor measurements may be reflected. eog Some examples of time values ​​are int eog can decrease over time during system operation ( Figure 3B ), so that int eog Indicates the remaining time until the EOG sensor measurement should be updated. In stage 314B of this example, the EOG sensor circuit 310B can read the int eog , and uses that value to determine whether the EOG sensor measurement should be updated. For example, if int eog If the value is equal to or below a threshold value (eg, zero), the EOG sensor circuit 310B may determine at 314B that an updated EOG sensor measurement should be obtained (318B) and may set the int eog reset to a value above the threshold. Also, if int eog is above the threshold, the EOG sensor circuit 310B may not continue to acquire EOG sensor measurements at 314B, but may wait for int eog reaches a threshold. In some examples, if the int eogabove the threshold, the optical sensor circuit 310B may enter a low power mode (316B) instead of obtaining EOG sensor measurements until int eog The low power mode may result in power saving behaviors, such as suspending processor threads associated with the EOG sensor. In some examples, EOG sensor measurements may be obtained when an event (e.g., user input) occurs, regardless of whether the int eog Such EOG sensor measurements can be obtained instead of or in addition to once int eog Reaching the threshold is in addition to obtaining any EOG sensor measurements. In an iteration of the EOG sensor loop 310B in which an EOG sensor measurement is obtained, the measurement is output as an EOG signal 320B.

[0041] In some examples, int eog The maximum value can be less than int opt This reflects that the EOG sensor measurements can be updated more frequently than the optical sensor measurements. For example, in an exemplary system that includes an optical sensor that is less power efficient than the included EOG sensor, updating the EOG sensor more frequently and updating the optical sensor less frequently can help optimize the overall power consumption of the exemplary system. In some examples, int eog and / or f eog The EOG sensor is configured to update at a frequency of approximately 500 Hz and can be int opt and / or f opt Configured so that the optical sensor is updated at a frequency of approximately 60 Hz. In some examples, it may be necessary to int eog and int opt Other relations between signals. For example, to hold an int eog Signal and int opt Phase shifting between the signals so that the optical sensor and the EOG sensor are not updated simultaneously may be beneficial to avoid unpredictable sequencing (eg, due to race conditions), improve throughput of the eye signal processor 330, or facilitate load balancing.

[0042] In some examples, in addition to or instead of a timer (e.g., with an int eog or int opt In addition to a timer (e.g., a timer with a period) reaching a threshold, the occurrence of an event can also trigger the acquisition of an EOG sensor measurement and / or an optical sensor measurement. One such event can be the detection of a sufficiently large change in the EOG sensor or optical sensor measurement that can indicate a change in the state of the user's eye. As an example, an optical sensor such as an eye tracking camera can enter a low power mode (e.g., with respect to Figure 3BThe optical sensor may be configured to be in low power mode when the user's eye position remains stationary, a situation in which the need for updated optical sensor measurements for the eye may be reduced. The power savings may be particularly significant in situations where the eye is expected to remain substantially stationary for long periods of time, such as when the user is sleeping or the eye is fixed on a particular point (e.g., while looking at a television screen), as the need for optical sensor measurements in such situations may be very limited.

[0043] When the EOG sensor output indicates that the eye position remains stationary, power savings can be further achieved by forgoing computationally expensive operations (e.g., performing image processing on the output of the optical sensor). For example, if the output of the EOG sensor remains constant, it indicates that the eye is stationary and image processing of an image of the eye may not produce new information over previous image processing. Therefore, in an example system configured to apply image processing to the output of an optical sensor (e.g., an eye tracking camera), the system can be configured to perform image processing in response to determining that the output of the EOG sensor has changed by a sufficiently large amount (e.g., in stage 318B). In some examples, a system including an eye tracking camera can be configured to capture an image and perform image processing (e.g., determine eye gaze) on the image in response to determining that the output of the EOG sensor has changed by a sufficiently large amount.

[0044] exist Figure 3B In the example system 300 shown, the optical signal 320A and the EOG signal 320B are input to the eye signal processor 330. The eye signal processor may also input a time value ( ) corresponding to the time at which the sensor measurement below the signals 320A and 320B were obtained. Figure 3B In the example shown, the eye signal processor 330 is operative to determine an updated eye position based on the signals 320A and 320B.

[0045] In some examples (e.g., example system 300), optical signal 320A (representing the position of eye 100 relative to imaging device) corresponds to the base position of eye 100, i.e., the current position of eye 100 independent of the previous position of eye 100. At stage 332, eye signal processor 330 may determine the base position of eye 100 based on optical signal 320A. For example, eye signal processor 330 may identify a correlation between the value of optical signal 320A and the position of eye 100, for example, by correlating optical signal 320A with various system parameters (such as parameters related to the placement of optical sensor, environmental conditions, and / or the appearance of the user's eye), and determine the position of eye 100 based on the correlation. Because optical sensors are subject to erroneous measurements, for example, due to occlusion by foreign matter (e.g., eyelids or eyelashes) or problematic lighting conditions, eye signal processor 330 may include logic for correcting erroneous optical measurements. For example, eye signal processor 330 may reject outlier measurements that are inconsistent with neighboring measurements. As described below, the signal processor 330 may also utilize the output of the eye data analyzer 340 to more accurately determine basic eye position from the EOG signals.

[0046] Likewise, in some examples, the EOG signal 320B (representing the electrical signal generated by moving the eye muscles) corresponds to an increment in the position of the eye 100, i.e., the degree to which the position of the eye 100 has changed from the previous position of the eye 100. The eye signal processor 330 may determine the increment in the position of the eye 100 based on the EOG signal 320B at stage 334. For example, the eye signal processor 330 may use the known correlation between the EOG signal and the eye muscle activity to determine a matrix describing the muscle movement corresponding to the value of the EOG signal 320B. The eye signal processor 330 may then accumulate matrices corresponding to successive values ​​of the EOG signal 320B (each representing a respective eye movement) to determine the net increment in the position of the eye 100 represented by those successive values. Because the accumulator system (e.g., used to obtain the net displacement from the differential movement) may drift, the eye signal processor 330 may include logic for correcting for the drift, e.g., by comparing the gradual change in the increment in position to the gradual change in the base position calculated at stage 332 and eliminating the deviation. Additionally, because EOG sensor measurements may be affected by electronic noise and / or interference (e.g., crosstalk), the signal processor 330 may include mechanisms (e.g., crosstalk cancellation filters) for correcting for such noise and / or interference. As described below, the signal processor 330 may also utilize the output of the eye data analyzer 340 to more accurately determine eye position increments from the EOG signal.

[0047] In some examples, the eye signal processor 330 can then calculate (336) the eye position as the sum of the base position (determined at stage 332 based on the optical signal 320A, as described above) and the incremental position (determined at stage 334 based on the EOG signal 320B, as described above). The output of this calculation can produce an output signal 390 representing the eye position, which can be used in various applications as described above.

[0048] In some examples including displays (e.g., head-mounted displays), eye position can be used to cause the display to present information to the user in the user's line of sight or on an area of ​​the display that is otherwise visible. For example, a head-mounted display can be fixed to the user's head at a known distance from the user's eyes. The position of the eyes can be used in conjunction with the known distance from the eyes to the display to identify the area of ​​the display that the user is currently looking at. The display state of the area can then be changed knowing that the user may immediately notice the change in the display state based on his or her current eye position. For example, an important message can be displayed directly in the user's line of sight. In an example where, for example, a display presents a 3D environment in a virtual reality or augmented reality system, a virtual object can appear at the exact location in the 3D environment that the user is currently looking at, thereby enhancing the user's immersion or control.

[0049] Similarly, the position of the eye can be used together with the distance from the eye to the display to identify areas of the display that the user is not currently looking at. The display state of that area can then be changed knowing that the user is unlikely to immediately notice the change in display state based on his or her current eye position. In an example where the display presents a 3D environment, such as in a virtual reality or augmented reality system, it may be desirable for a virtual object to unobtrusively enter or exit the environment, or change the state of a virtual object (such as the resolution of an asset used to render an object) without the user noticing. Such behavior can enhance the user's sense of immersion in the 3D environment. This can be achieved by identifying an area of ​​the display that the user is not looking at and changing the display state in that area.

[0050] Some examples may include an eye data analyzer such as Figure 3B332 and / or the incremental position data determined at stage 334) may be input to the eye data processor 340, which may be collected and stored by the eye data processor. In some examples (e.g., the example system 300), additional sensor data 350 and / or predicted data 360 are also input to the eye data processor 340. At stage 342, the eye data processor 340 may analyze the eye position data from the eye signal processor 330, and in some examples the additional sensor data 350 and / or predicted data 360, to identify patterns and features of the movement and behavior of the eye 100 over time. Such patterns and features may reveal ways in which the example system 300 may operate more effectively.

[0051] In some examples, the eye signal processor 330 and / or the eye data analyzer 340 may determine the probability of an eye behavior occurring rather than the certainty. These probabilities may be determined using statistical methods. In some examples, the eye signal processor 330 and / or the eye data analyzer 340 may generate and / or apply a statistical model that predicts the output of the system (e.g., the type of eye behavior) given the state of various inputs (e.g., eye position measurements). In some examples, the eye signal processor 330 and / or the eye data analyzer 340 may determine the probability by identifying or adjusting the weight or influence of one or more factors that affect the probability. For example, the eye signal processor 330 and / or the eye data analyzer 340 may determine that, given the current values ​​of various weighting factors, among several possible eye behaviors, a particular eye behavior is most likely to occur (even if the behavior does not necessarily occur). Similarly, even if such future behavior cannot be determined with certainty, the eye signal processor 330 and / or the eye data analyzer 340 may also predict future behavior based on various weighting factors. This reflects that in many cases it is difficult or impossible to conclude with certainty that the eye is engaging in a particular behavior; furthermore, it is difficult or impossible to predict future eye behavior. However, for many applications, it is not necessary to be absolutely certain that an eye behavior occurred; advantages (e.g., power efficiency and computational efficiency) can be conveyed by determining the relative likelihood of certain eye behaviors, or by making educated guesses about the likelihood of future eye behaviors.

[0052] Statistical methods may be used to determine the probability of an eye behavior occurring. For example, a confidence score may be assigned to the likelihood of a particular behavior occurring. The confidence score may be compared to a threshold, and upon determining that the confidence score exceeds the threshold, the eye signal processor 330 and / or the eye data analyzer 340 may determine that the behavior associated with the confidence score is sufficiently likely to occur. Other statistical methods may also be used.

[0053] As one example of identifying eye behavior, at stage 342, the eye data analyzer 340 may determine that the eye 100 is fixated on a stationary focal object based on the eye position data from the eye signal processor 330. For example, the eye data processor may make such a determination based on data indicating that the eye is performing only small, high-frequency movements, characteristic of a situation where the eye is fixated on a stationary object. In the case of such fixation, it may be expected that the position of the eye 100 will change little, if at all. In such a situation, the rate at which the optical and / or EOG sensors obtain new measurements may be reduced (e.g., by increasing the int opt and / or int eog ) is acceptable because each new measurement can be expected to provide little new significant position information. Eye signal processor 330 can also adjust its calculation of eye position to reflect that signals 320A and 320B are unlikely to exhibit significant position changes when eye 100 is fixating, possibly resulting in higher eye tracking accuracy.

[0054] As another example, at stage 342, the eye position data from the eye signal processor 330 may indicate that the movement of the eye 100 is changing direction rapidly, a condition that indicates that the eye 100 is engaging in saccadic behavior. When the eye is engaging in saccadic behavior, the signal describing the eye position may contain a large amount of high frequency information. In response to identifying such saccadic behavior, the eye data processor 340 may increase the rate at which the EOG sensor is updated (such as by reducing the int eog ), so that the high-frequency information in the eye position signal can be accurately captured without aliasing.

[0055] In some examples, the eye data processor 340 can utilize additional sensor data 350, such as from sensors other than optical sensors and EOG sensors, to more effectively identify eye behavior than otherwise possible. In some examples, machine learning techniques can be used to improve the recognition of eye behavior. As an example, a neural network can be trained using additional sensor data 350 associated with a single user to identify the user's eye behavior based on the additional sensor data 350. As another example, a generalized neural network can be trained using additional sensor data 350 associated with a user group rather than for a single user. Such neural networks can be recursively trained using data from the example system 300 (e.g., output signal 390). As another example, a genetic algorithm can be used to identify the relationship between input data (including additional sensor data 350) and eye behavior. Other machine learning techniques, such as support vector machines, Bayesian networks, rule-based systems, and learning classifier systems, as well as deep learning techniques, can be similarly employed. In some examples, these techniques are implemented within the eye data processor 340. In some examples, these techniques are implemented in other components of the example system 300. In some examples, these techniques are implemented at least in part in a system external to the example system 300. For example, a remote server may train a neural network on a large data set and communicate parameters or outputs of the neural network to the example system 300 over a computer network.

[0056] As one example of utilizing additional sensor data 350, additional sensor data 350 may include data from an accelerometer configured to detect head movement. An accelerometer output corresponds to a value of the acceleration of the accelerometer relative to an inertial frame; thus, if the accelerometer is affixed to a person's head, the accelerometer output may correspond to the acceleration of the head. A high output value of such an accelerometer may indicate that there is significant head movement, such as because a person is observing a fast-moving object or craning his or her neck to observe an object in the periphery of vision. Conversely, a low output value of the accelerometer may indicate that the head is relatively motionless, such as when reading a book. If the accelerometer data indicates that there is significant head movement, the eye data processor 340 may conclude that the eye 100 is engaged in vestibular-ocular movements (such as where the eye moves with the head or neck muscles, as described above). Because vestibular-ocular movements may be associated with eye movements of relatively small magnitude (e.g., because head movement makes large eye movements unnecessary), the eye data processor 340 may reduce the rate at which the optical sensors are updated (e.g., by increasing the int opt, thereby correspondingly reducing the power consumed by the optical sensor) to reflect that the basic position of the eye may not be expected to suddenly shift significantly. In contrast, if the additional sensor data 350 includes accelerometer data indicating that there is no obvious head movement, the eye data processor can conclude that the eye 100 is more likely to engage in a saccadic movement (e.g., when reading, where head movement is minimal) than otherwise. In examples utilizing machine learning techniques, such techniques can be employed to identify associations between accelerometer data and eye behavior. For example, a neural network can be trained to associate specific patterns of accelerometer data (e.g., sinusoidal outputs corresponding to simple harmonic motion) with specific eye behaviors associated with those patterns. In addition, in some examples, a gyroscope, electric compass, magnetometer, inertial measurement unit, or other device can be used instead of or in addition to an accelerometer.

[0057] Other types of additional sensor data 350 may also be used to beneficial effect. In some examples, the additional sensor data 350 may include data from an ambient light sensor. The eye data processor 340 may use this data to identify eye behaviors associated with certain lighting conditions, or to help the eye signal processor 330 correct for changes in lighting conditions. For example, because the pupil of the eye 100 may contract in response to exposure to increased light levels, the additional sensor data 350 indicating increased light levels may indicate that contraction of the pupil size is to be expected; in response, the optical sensor may prepare to recalibrate for a smaller pupil. The eye signal processor 330 may also employ a different eye tracking algorithm to accommodate a smaller pupil size. As another example, the additional sensor data 350 may include ambient light data indicating that it is too dark for the optical sensor to function properly. The eye data processor 340 may use this information in this case to slow down or stop updating the optical sensor (e.g., by increasing the int opt ). In some examples, additional sensor data 350 may provide information that may also be determined from sensors, such as optical sensors. In such examples, efficiencies may be gained by using relatively power-efficient sensors, such as ambient light sensors, to perform work that would otherwise be performed by a camera or other less efficient sensor. In examples utilizing machine learning techniques, such techniques may be employed to identify associations between sensors, such as ambient light sensors, and eye behavior. For example, a neural network may be trained to associate changes in ambient light with specific eye behaviors associated with those changes, such as pupil dilation and constriction.

[0058] In some examples, additional sensor data 350 may include time and / or position data, for example, from a GPS sensor. Eye data processor 340 may use this data to identify eye behavior associated with a specific time and position. As an example, additional sensor data 350 may include position data, which indicates that the user is stationary in a building at night, and therefore may be more likely to engage in reading and panning eye movements than in other ways. As another example, additional sensor data 350 may include time data, which may be used by eye data processor 340 to identify routine behavior (such as daily commuting from 6:00 to 7:00), and predict eye movement based on this behavior. As another example, additional sensor data 350 may include position data, which indicates that the user is driving a vehicle and that the user's eyes may be more likely to engage in panning movements than in other situations. In the example utilizing machine learning technology, such technology may be used to identify the association between time and / or position data and eye behavior. For example, neural network may be trained to associate the time of day with the specific eye behavior associated with a specific time.

[0059] In some examples, the eye data processor 340 can be assisted by map data, such as commercial map data that associates geographic coordinates (e.g., from a GPS sensor) with specific buildings, businesses, or landmarks. For example, the additional sensor data 350 may include location data that can be used in conjunction with the map data to indicate that the user is in a gym and is therefore more likely to engage in exercise and therefore vestibular eye movements than otherwise. Similarly, the additional sensor data 350 may include location data that can be used in conjunction with the map data to indicate that the user is in a movie theater and that the user's eyes are likely to engage in gaze behavior (e.g., while watching a movie screen) for several hours. In an example utilizing machine learning techniques, such techniques can be employed to identify associations between map data and eye behavior. For example, a neural network can be trained to associate a user's location with specific eye behaviors that tend to occur at that location.

[0060] In some examples, additional sensor data 350 may include data related to a medical condition, which may be related to eye behavior associated with the condition. For example, if additional sensor data 350 indicates that the user suffers from amblyopia (amblyopia), a condition that can cause an abnormally large amount of noise to appear in an eye tracking system, the eye data processor 340 may use this information to predict and reduce high noise levels. In addition, the eye data processor may adjust the update rate of the optical sensor and / or EOG sensor to accommodate the fact that the user's eyes may not be able to engage in normal movement due to the medical condition. In an example utilizing machine learning techniques, such techniques may be employed to identify associations between medical information and eye behavior. For example, a neural network may be trained to associate certain medical conditions with specific eye behaviors that accompany those medical conditions.

[0061] In some examples, the additional sensor data 350 may include data related to the user's use of the computer system; in particular, in examples where the eye tracking system is integrated with the computer system (e.g., providing input to the computer system), the computer system may indicate that the user is using specific software that may indicate certain eye behaviors. For example, the additional sensor data 350 may indicate that the user is using an e-book reader program to read text; the eye data processor 340 may use this information to predict that the user is engaging in a saccadic motion associated with reading. As another example, the additional sensor data 350 may include data indicating where objects appear on the user's display. Because the user may be expected to look at these objects, the eye data processor 340 may use this information to predict what the user's eyes may focus on. For example, such information may be used to predict that the eye 100 will engage in object tracking behavior, with the display coordinates of the tracked object indicated by the additional sensor data 350. In examples utilizing machine learning techniques, such techniques may be employed to identify associations between computer system use and eye behavior. For example, a neural network may be trained to associate certain computer usage situations (e.g., the operation of a particular software application) with specific eye behaviors that accompany these usage situations.

[0062] In some examples, the eye data processor 340 can use the prediction data 360 in combination with the eye position data (e.g., from the eye signal processor 330) and / or the additional sensor data 350 to more accurately identify or predict eye behavior. The prediction data 360 can include information that relates input data (e.g., eye position data) to the likelihood of certain output eye behaviors (e.g., saccadic movements). Various machine learning techniques can be used to generate the prediction data 360. In some examples, a neural network can be trained using known eye position data and eye behavior from a single user to generate prediction data 360 that relates eye behavior to eye position data from the user. In some examples, a generalized neural network can be trained for a user group, rather than a single user. In some examples, the prediction data 360 generated from an unsupervised learning technique can be used to identify the relationship between input data and eye behavior, which can improve the accuracy of the eye data processor 340 and can make the example system 300 more useful for large and diverse user groups. In some examples, the prediction data 360 generated from a deep learning technique can be used to identify the relationship between input data and eye behavior, especially when there is little prior knowledge about the input data. In some examples, genetic algorithms can be used to identify relationships between input data (including additional sensor data 350) and eye behavior. Other machine learning techniques, such as support vector machines, Bayesian networks, rule-based systems, and learning classifier systems can be similarly employed.

[0063] In some examples, the prediction data 360 may be communicated to the example system 300 by an external source. For example, a neural network may be trained on a remote server, the parameters or outputs of which are communicated to the example system 300 as the prediction data 360. This configuration may be particularly beneficial in examples using large eye datasets (e.g., from a large number of users) for which locally applied machine learning techniques may be computationally limited. However, in some examples (e.g., those involving "lightweight" implementations of machine learning techniques), the prediction data may be generated locally at the example system 300.

[0064] In such Figure 3BIn some examples shown, after the eye data processor 340 characterizes the behavior of the eye 100 at stage 342, the eye data processor 340 may determine one or more optical sensor parameters and / or sensing circuit parameters at stage 344. The optical sensor parameters or sensing circuit parameters may include an interval parameter that defines the rate at which the optical sensor loop 318A and the EOG sensor loop 318B update their respective loops. For example, if the eye data processor 340 determines that power consumption can be safely reduced by reducing the optical sensor update rate based on current eye behavior (e.g., as detected by an image of the eye output by the optical sensor), the eye data processor 340 may directly or indirectly increase the interval parameter. opt to increase the time between optical sensor updates (thereby reducing the optical sensor update rate) as described above. The eye data processor 340 may similarly adjust int eog , to adjust the time between EOG sensor updates as described above. In some examples, the eye data processor 340 can adjust other system parameters based on the determination of the eye behavior, such as parameters for filtering noise from the optical signal 320A and / or the EOG signal 320B. In some examples, the eye data processor 340 can perform other operations based on the determination of the eye behavior, such as priming the optical sensor or calibrating the optical sensor or the EOG sensor.

[0065] exist Figure 3B In the example system 300 shown, int opt and int eog Set them as function f opt and f eog , these functions may receive one or more parameters (e.g., x, y) from the eye data processor 340 at stage 346. For example, such parameters may correspond to categories of high-level eye behavior, frequency components of eye movement, and other aspects of eye movement. Such parameters may also correspond to information not directly related to eye movement, such as the current battery life (which may require less frequent sensor measurements when the battery is low). In such an example system, f opt and f eog can be configured to allow the eye data processor 340 to opt and f eog The parameters of continuously adjust the sensor update rate to maintain the best trade-off between eye tracking accuracy and power consumption.

[0066] Figure 4An example system 400 is shown that can be used to implement any or all of the above examples. The example system 400 can be included in a portable device (including a wearable device) or a non-portable device, such as a communication device (e.g., a mobile phone, a smart phone), a multimedia device (e.g., an MP3 player, a television, a radio), a portable or handheld computer (e.g., a tablet, a netbook, a laptop), a desktop computer, an all-in-one desktop computer, a peripheral device, a head-mounted device (e.g., which can include an integrated display), or any other system or device suitable for including the example system 400, including combinations of two or more of these types of devices. The above examples can be embodied in two or more physically separate devices, such as two or more computers that communicate via a wireless network. The above examples can be embodied in two or more physically different devices, such as a waist pack that communicates data to and / or from a head-mounted display. The example system 400 includes one or more computer-readable media 401, a processing system 404, an I / O subsystem 406, wireless communication circuitry (e.g., RF circuitry) 408, an audio device (e.g., speaker, microphone) 410, and a sensor 411. These components may be coupled via one or more communication buses or signal lines 403.

[0067] Figure 4 The architecture shown is only one example architecture of the example system 400, and the example system 400 may have more or fewer components than shown or have a different configuration of components. Figure 4 The various components shown in the drawings may be implemented in hardware, software, firmware, or any combination thereof, including one or more digital signal processors (DSPs) and / or application specific integrated circuits (ASICs).

[0068] refer to Figure 4 In the example system 400 in FIG. 4 , the wireless communication circuit 408 can be used to send and receive information to one or more other devices via a wireless (e.g., RF) link or network, and can include circuits for performing this function. The wireless communication circuit 408 and the audio device 410 can be coupled to the processing system 404 via a peripheral device interface 416. The peripheral device interface 416 may include various known components for establishing and maintaining communication between peripheral devices (e.g., wireless communication circuit 408, audio device 410, and sensor 411) and the processing system 404. The audio device 410 may include circuits for processing voice signals received from the peripheral device interface 416 to enable a user to communicate with other users in real time. The audio device 410 may include, for example, one or more speakers and / or one or more microphones. In some examples, the audio device 410 may include a headphone jack (not shown).

[0069] Sensors 411 may include various sensors, including, but not limited to, one or more light emitting diodes (LEDs) or other light emitters, one or more photodiodes or other light sensors, one or more photothermal sensors, a magnetometer, an accelerometer, a gyroscope, a barometer, a compass, a proximity sensor, a camera, an ambient light sensor, a thermometer, a GPS sensor, an electro-oculography (EOG) sensor, and various system sensors that may sense remaining battery life, power consumption, processor speed, CPU load, etc. In examples involving, for example, a head-mounted device (which may include a display), one or more sensors may be employed in conjunction with functionality related to the user's eyes, such as tracking the user's eye movements, or identifying a user based on images of his or her eyes.

[0070] The peripheral device interface 416 can couple the input and output peripheral devices of the system 400 to one or more processors 418 and one or more computer-readable media 401. The one or more processors 418 can communicate with the one or more computer-readable media 401 via the controller 420. The computer-readable medium 401 can be any device or medium (except signals) that can store code and / or data for use by one or more processors 418. In some examples, the computer-readable medium 401 can be a non-transitory computer-readable storage medium. The computer-readable medium 401 may include a memory hierarchy, including but not limited to cache, main memory, and auxiliary memory. This memory hierarchy can be implemented using any combination of RAM (e.g., SRAM, DRAM, DDRAM), ROM, FLASH, magnetic and / or optical storage devices (e.g., disk drives, tapes, CDs (compact disks), and DVDs (digital video disks)). The computer-readable medium 401 may also include a transmission medium for carrying information-bearing signals indicating computer instructions or data (but not including signals and not including carriers on which signals are modulated). For example, the transmission medium may include communication networks including but not limited to the Internet (including the World Wide Web), an intranet, a local area network (LAN), a wide area network (WLAN), a storage area network (SAN), a metropolitan area network (MAN), etc.

[0071] One or more processors 418 can run various software components stored in the computer-readable medium 401 to perform various functions of the example system 400. In some examples, the software components may include an operating system 422, a communication module (or instruction set) 424, an I / O processing module (or instruction set) 426, a graphics module (or instruction set) 428, and one or more applications (or instruction sets) 430. Each of these modules and the above-mentioned applications can correspond to an instruction set for performing one or more of the above-mentioned functions and methods described in this application (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) do not need to be implemented as separate software programs, processes, or modules, and therefore in various examples, various subsets of these modules can be combined or otherwise rearranged. In some examples, the computer-readable medium 401 can store a subset of the modules and data structures identified above. In addition, the computer-readable medium 401 can store additional modules and data structures not described above.

[0072] Operating system 422 may include various processes, instruction sets, software components, and / or drivers for controlling and managing general system tasks (eg, memory management, storage device control, power management, etc.), and facilitating communications between various hardware and software components.

[0073] Communications module 424 may facilitate communications with other devices through one or more external ports 436 or via wireless communications circuitry 408 , and may include various software components for processing data received from wireless communications circuitry 408 and / or external ports 436 .

[0074] The graphics module 428 may include various known software components for rendering, animating, and displaying graphical objects on one or more display surfaces. The display surface may include a 2D or 3D display. The display surface may be directly or indirectly coupled to one or more components of the example system 400. In examples involving a touch sensing display (e.g., a touch screen), the graphics module 428 may include components for rendering, displaying, and animating objects on the touch sensing display. In some examples, the graphics module 428 may include components for rendering to a remote display. In some examples, such as those with a camera, the graphics module 428 may include components for creating and / or displaying an image formed by synthesizing camera data (e.g., captured from a head-mounted camera) or photographic data (e.g., satellite-captured images) with rendered graphical objects. In some examples, the graphics module 428 may include components for rendering images to a head-mounted display. In some examples, the image may include a view of an element of virtual content (e.g., an object in a three-dimensional virtual environment), and / or a view of the physical world (e.g., a camera input indicating the user's physical environment). In some examples, the display may present a combination of views of the virtual content and the physical world. In some examples, the view of the physical world may be a rendered image; in some examples, the view of the physical world may be an image from a camera.

[0075] The one or more applications 430 may include any application installed on the example system 400, including but not limited to browsers, address books, contact lists, email, instant messaging, word processing, keyboard emulation, widgets, JAVA-enabled applications, encryption, digital rights management, voice recognition, voice replication, location determination capabilities (such as those provided by a global positioning system (GPS)), music players, etc.

[0076] The I / O subsystem 406 may be coupled to one or more I / O devices 414 for controlling or performing various functions. In examples involving processing of eye data (e.g., examples including eye tracking or iris recognition functions), the I / O subsystem 406 may be coupled to one or more I / O devices 412 dedicated to processing eye-related input and output. One or more eye I / O devices 412 may communicate with the processing system 404 via an eye I / O device controller 432, and may include various components for processing eye input (e.g., a sensor for eye tracking) or user gesture input (e.g., an optical sensor). One or more other I / O controllers 434 may send electrical signals to other I / O devices 414, or may receive electrical signals from other I / O devices 414. Such I / O devices 414 may include physical buttons, dials, slider switches, joysticks, keyboards, touchpads, additional display screens, or any combination thereof.

[0077] The I / O processing module 426 may include various software components for performing various tasks associated with one or more eye I / O devices 412 and / or one or more other I / O devices 414, including but not limited to receiving and processing input received from the eye I / O device 412 through the eye I / O device controller 432 or from other I / O devices 414 through the I / O controller 434. In some examples, the I / O device 414 and / or the I / O processing module 426 may perform various tasks associated with gesture input that may be provided by tactile or non-tactile means. In some examples, the gesture input may be provided by a camera or another sensor that is used to detect, for example, movement of the user's eyes, arms, hands, and / or fingers. In some examples, the one or more I / O devices 414 and / or the I / O processing module 426 may be configured to identify an object on a display that the user wishes to interact with, such as a GUI element pointed to by the user. In some examples, one or more eye I / O devices 412 and / or I / O processing modules 426 may be configured to perform eye tracking tasks (e.g., with the aid of optical or EOG sensors), such as identifying an object or area on a display that a user is looking at. In some examples, a device (e.g., a hardware "beacon") may be worn or held by a user to assist one or more I / O devices 414 and / or I / O processing modules 426 in completing gesture-related tasks, such as identifying the position of a user's hand relative to a 2D or 3D environment. In some examples, one or more eye I / O devices 412 and / or I / O processing modules 426 may be configured to identify a user based on sensor input (e.g., data from a camera sensor relating to the user's eyes).

[0078] In some examples, graphics module 428 can display visual output to the user in a graphical user interface (GUI). Visual output can include text, graphics, video and any combination thereof. Some or all of the visual output can correspond to user interface objects. In some examples, one or more I / O devices 412 and / or 414 and / or controllers 432 and / or 434 (and any associated modules and / or instruction sets in medium 401) can detect and track gestures and / or eye movements, and detected gestures and / or eye movements can be converted into interactions with graphical objects (e.g., one or more user interface objects). In an example where one or more eye I / O devices 412 and / or eye I / O device controller 432 are configured to track the eye movements of the user, the user can interact directly with the graphical objects by watching them.

[0079] Feedback may be provided, for example, through the eye I / O device(s) 412 or the other I / O device(s) 414 based on one or more states of the display and / or example system 400. Feedback may be sent optically (e.g., a light signal or a displayed image), mechanically (e.g., haptic feedback, touch feedback, force feedback, etc.), electrically (e.g., electrical stimulation), olfactorily, auditorily (e.g., a beep, etc.), or any combination thereof and in a variable or invariant manner.

[0080] The example system 400 may also include a power system 444 for powering various hardware components, and may include a power management system, one or more power supplies, a charging system, a power fault detection circuit, a power converter or inverter, a power status indicator, and any other components typically associated with the generation, management, and distribution of power in a portable device.

[0081] In some examples, peripherals interface 416, one or more processors 418, and controller 420 can be implemented on a single chip, such as processing system 404. In some other examples, they can be implemented on separate chips.

[0082] In some examples, a method is disclosed. The method may include: receiving first data from a first sensor at a first time interval, the first sensor being configured to output data indicating a first position of an eye; receiving second data from a second sensor at a second time interval, the second sensor being configured to output data indicating an incremental position of the eye; determining the first position of the eye based on the first data; determining the incremental position of the eye based on the second data; using the first position of the eye and the incremental position of the eye, determining a second position of the eye; and in response to determining the second position of the eye, generating an output signal indicating the second position of the eye. In addition or in lieu of one or more of the above examples, the first sensor may include an optical sensor. In addition or in lieu of one or more of the above examples, the second sensor may include an electro-oculography sensor. In addition or in lieu of one or more of the above examples, the first time interval may be greater than the second time interval. In addition or in lieu of one or more of the above examples, the first sensor may operate in a low power mode during the first time interval. In addition or in lieu of one or more of the above examples, the second sensor may operate in a low power mode during the second time interval. In addition or in lieu of one or more of the above examples, the method may also include determining a first eye movement behavior using the second position of the eye. In addition or in lieu of one or more of the above examples, the first eye movement behavior may include saccadic movement, smooth pursuit, fixation, nystagmus, or vestibular eye movement. In addition or in lieu of one or more of the above examples, the method may further include: in response to determining the first eye movement behavior: determining a third time interval for receiving data from the first sensor, and determining a fourth time interval for receiving data from the second sensor. In addition or in lieu of one or more of the above examples, determining the first eye movement behavior may include: generating a confidence score corresponding to the likelihood of the first eye movement behavior; comparing the confidence score to a threshold; and determining that the confidence score exceeds the threshold. In addition or in lieu of one or more of the above examples, the method may further include: receiving third data from a third sensor, and the third data may be used to determine the first eye movement behavior. In addition or in lieu of one or more of the above examples, the third sensor may include an accelerometer, a gyroscope, an electronic compass, a magnetometer, or an inertial measurement unit. In addition or in lieu of one or more of the above examples, the third sensor may include a GPS sensor. In addition or in lieu of one or more of the above examples, the third sensor may include an ambient light sensor. In addition or in lieu of one or more of the above examples, a neural network may be used to determine the first eye movement behavior. In addition or alternatively to one or more of the above examples, the method may also include training the neural network using information including the first data, the second data, the third data, the second position of the eye, or the first eye movement behavior.In addition or alternatively to one or more of the above examples, the method may further include determining a second eye movement behavior using a neural network. In addition or alternatively to one or more of the above examples, the first sensor and the second sensor may be attached to a head mounted device including a display. In addition or alternatively to one or more of the above examples, the method may further include, in response to determining a second position of the eye: determining an area of ​​the display corresponding to the second position of the eye, the area having a display state equal to the first display state; and changing the display state of the area from the first display state to the second display state.

[0083] In some examples, a method is disclosed. The method may include: receiving first data from a sensor associated with a user of an augmented reality system including a head-mounted display at a first time interval, the first data indicating the position of the user's eyes; determining an eye movement behavior associated with the eyes based on the first data and the properties of the augmented reality system; and determining a second time interval for receiving data from the sensor in response to determining the eye movement behavior associated with the eyes. In addition or in lieu of one or more of the above examples, determining the eye movement behavior may include: generating a confidence score corresponding to the likelihood of the eye movement behavior; comparing the confidence score to a threshold; and determining that the confidence score exceeds the threshold. In addition or in lieu of one or more of the above examples, the augmented reality system may be configured to execute a software application, and the properties of the augmented reality system may indicate the state of the software application. In addition or in lieu of one or more of the above examples, the sensor may operate in a low power mode during the second time interval. In addition or in lieu of one or more of the above examples, the eye movement behavior may include saccadic movements, gentle pursuits, fixations, nystagmus, or vestibular eye movements. In addition or alternative to one or more of the above examples, the augmented reality system may include an accelerometer, a gyroscope, an electronic compass, a magnetometer, or an inertial measurement unit, and the properties of the augmented reality system include the output of the accelerometer, gyroscope, electronic compass, magnetometer, or inertial measurement unit. In addition or alternative to one or more of the above examples, the augmented reality system may include a GPS sensor, and the properties of the augmented reality system may include the output of the GPS sensor. In addition or alternative to one or more of the above examples, the augmented reality system may include an ambient light sensor, and the properties of the augmented reality system may include the output of the ambient light sensor. In addition or alternative to one or more of the above examples, a neural network may be used to determine eye movement behavior. In addition or alternative to one or more of the above examples, the sensor may include an optical sensor. In addition or alternative to one or more of the above examples, the sensor may include an electro-oculography sensor.

[0084] In some examples, a wearable computing system is disclosed. The wearable computing system may include: a frame configured to be worn around a user's head; a sensing circuit including at least one electrode attached to the frame, the sensing circuit configured to measure an electrical potential of an eye of the user; an optical sensor attached to the frame and configured to detect an image of the user's eye according to an optical sensor parameter; and a processor operably coupled to the sensing circuit and the optical sensor, wherein the processor is configured to: obtain first data from the sensing circuit, the first data indicating an electrical potential of the user's eye; and adjust the optical sensor parameter based on the first data. In addition or in lieu of one or more of the above examples, the optical sensor parameter may determine a rate at which the optical sensor detects an image of the eye. In addition or in lieu of one or more of the above examples, the optical sensor parameter may determine a power consumption mode of the optical sensor. In addition or in lieu of one or more of the above examples, the processor may be further configured to selectively activate and deactivate the optical sensor based on the first data. In addition or in lieu of one or more of the above examples, the processor may be further configured to determine a position of the eye based on an image detected by the optical sensor. In addition or in lieu of one or more of the above examples, the processor may be further configured to detect movement of the eye based on the first data. In addition or alternative to one or more of the above examples, the processor may be further configured to adjust the optical sensor parameters based on the detected movement. In addition or alternative to one or more of the above examples, the processor may be further configured to determine whether the eye is engaged in an eye movement behavior in a plurality of predetermined eye movement behaviors, the determination being based on at least the first data. In addition or alternative to one or more of the above examples, the processor may be further configured to adjust the optical sensor parameters based on the determination. In addition or alternative to one or more of the above examples, the sensing circuit may be configured to measure the electric potential of the user's eye according to the sensing circuit parameters, and the processor may be further configured to adjust the sensing circuit parameters based on the image of the eye output by the optical sensor. In addition or alternative to one or more of the above examples, the sensing circuit parameters may determine the rate at which the sensing circuit outputs data indicating the electric potential of the eye to the processor. In addition or alternative to one or more of the above examples, the sensing circuit may include two electrodes and at least one electrical component configured to measure the electric potential difference between the two electrodes.

[0085] While the disclosure has been particularly shown and described with reference to examples thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the disclosure.

Claims

1. A method comprising: receiving first data indicating a first position of an eye, the first data being received from a first sensor at a first rate; receiving second data indicative of a second position of the eye, the second data being received at a second rate from a second sensor different from the first sensor, wherein the first sensor is configured to detect the eye data at a first frequency, and wherein the second sensor is configured to detect the eye data at a second frequency higher than the first frequency; determining, based on optical sensor parameters, whether the first rate will increase relative to the second rate; Based on determining that the first rate will increase relative to the second rate, adjusting a system parameter to increase the first rate relative to the second rate; and Based on determining that the first rate will not increase relative to the second rate, adjusting the system parameter is abandoned.

2. The method according to claim 1, wherein: The first sensor includes an optical sensor.

3. The method according to claim 1, wherein: The second sensor includes an electro-oculogram sensor.

4. The method according to claim 1, wherein: The first sensor operates in a low power mode while receiving the first data at the first rate.

5. The method according to claim 1, wherein: The second sensor operates in a low power mode while receiving the second data at the second rate.

6. The method according to claim 1, further comprising: determining a third position of the eye based on the first position of the eye and the second position of the eye; as well as Based on the third position of the eye, a first eye movement behavior is determined.

7. The method according to claim 6, wherein: The optical sensor parameters are based on the first eye movement behavior.

8. The method according to claim 6, wherein: Determining the first eye movement behavior includes: generating a confidence score corresponding to the likelihood of the first eye movement behavior; comparing the confidence score to a threshold; and It is determined that the confidence score exceeds the threshold.

9. The method according to claim 6, wherein: The first sensor and the second sensor are associated with a head mounted device including a display, and The method further comprises: determining the third position of the eye according to the determining an area of ​​the display corresponding to the second position of the eye, the area having a display state equal to a first display state; and The display state of the area is changed from the first display state to a second display state.

10. The method according to claim 1, further comprising: receiving third data from a third sensor; as well as Based on the third data, a first eye movement behavior is determined.

11. The method according to claim 1, wherein: The optical sensor parameters are based on outputs from a neural network.

12. The method according to claim 1, further comprising: A neural network is trained using information including the first data, the second data, or both.

13. The method of claim 1, wherein: The second data indicates an incremental position of the eye, and The method further includes determining whether the incremental position of the eye is less than a threshold, wherein: Based on determining that the incremental position is greater than the threshold, the optical sensor parameter indicates that the first rate will increase relative to the second rate, and Based on determining that the incremental position is not greater than the threshold, the optical sensor parameter indicates that the system parameter is not to be adjusted.

14. The method according to claim 1, further comprising: determining, based on the optical sensor parameters, whether the first rate will decrease relative to the second rate; adjusting the system parameter to reduce the first rate relative to the second rate based on determining that the first rate will decrease relative to the second rate; as well as Based on determining that the first rate will not decrease relative to the second rate, adjusting the system parameter is abandoned.

15. The method according to claim 14, wherein: Determining that the first rate is to be reduced relative to the second rate includes determining that the optical sensor parameter includes an indication to reduce power consumption of the first sensor.

16. The method according to claim 14, wherein: Determining that the first rate is to increase relative to the second rate includes determining that the optical sensor parameter includes an indication of increasing accuracy of the first sensor.

17. The method according to claim 1, further comprising: determining, based on a second sensor parameter, whether the second rate will increase relative to the first rate; adjusting a second system parameter to increase the second rate relative to the first rate based on determining that the second rate will increase relative to the first rate; as well as Based on determining that the second rate will not increase relative to the first rate, adjusting the second system parameter is abandoned.

18. A system comprising: a first sensor configured to detect eye data at a first frequency; a second sensor configured to detect eye data at a second frequency higher than the first frequency; One or more processors operably coupled to the first sensor and the second sensor, the one or more processors configured to perform a method comprising the steps of: receiving first data indicating a first position of an eye, the first data being received from a first sensor at a first rate; receiving second data indicative of a second position of the eye, the second data being received at a second rate from a second sensor different from the first sensor; determining, based on optical sensor parameters, whether the first rate will increase relative to the second rate; adjusting a system parameter to increase the first rate relative to the second rate based on determining that the first rate will increase relative to the second rate; as well as Based on determining that the first rate will not increase relative to the second rate, adjusting the system parameter is abandoned.

19. A non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method comprising: receiving first data indicating a first position of an eye, the first data being received from a first sensor at a first rate; receiving second data indicative of a second position of the eye, the second data being received at a second rate from a second sensor different from the first sensor, wherein the first sensor is configured to detect the eye data at a first frequency, and wherein the second sensor is configured to detect the eye data at a second frequency higher than the first frequency; determining, based on optical sensor parameters, whether the first rate will increase relative to the second rate; Based on determining that the first rate will increase relative to the second rate, adjusting a system parameter to increase the first rate relative to the second rate; and Based on determining that the first rate will not increase relative to the second rate, adjusting the system parameter is abandoned.

20. A method comprising: receiving first data indicating a first position of an eye, the first data being received at a first time interval, the first data being output by a first sensor, the first sensor being configured to output data indicating the first position of the eye; receiving second data indicating a second position of the eye, the second data being received at a second time interval, the second data being output by a second sensor, the second sensor being configured to output data indicating the second position of the eye, wherein the second sensor is different from the first sensor, wherein the first sensor is configured to obtain the first data at a first frequency, and wherein the second sensor is configured to obtain the second data at a second frequency higher than the first frequency; determining the first position of the eye based on the first data; determining the second position of the eye based on the second data; determining a third position of the eye based on the first position of the eye and the second position of the eye; and An output signal is generated that is indicative of the third position of the eye.

21. The method according to claim 20, wherein: The first sensor includes an optical sensor.

22. The method according to claim 20, wherein: The second sensor includes an electro-oculogram sensor.

23. The method according to claim 20, wherein: The first sensor operates in a low power mode during the first time interval.

24. The method according to claim 20, wherein: The second sensor operates in a low power mode during the second time interval.

25. The method of claim 20, further comprising: Based on the third position of the eye, a first eye movement behavior is determined.

26. The method according to claim 25, wherein: The first eye movement behavior includes saccadic movement, smooth pursuit, fixation, nystagmus, or vestibular eye movement.

27. The method of claim 25, further comprising: determining a third time interval for receiving data from the first sensor based on the first eye movement behavior, and Based on the first eye movement behavior, a fourth time interval for receiving data from the second sensor is determined.

28. The method according to claim 25, wherein: Determining the first eye movement behavior includes: generating a confidence score corresponding to the likelihood of the first eye movement behavior; comparing the confidence score to a threshold; and It is determined that the confidence score exceeds the threshold.

29. The method of claim 25, further comprising: Third data is received from a third sensor, and wherein the first eye movement behavior is determined based on the third data.

30. The method of claim 29, wherein: The third sensor includes an accelerometer, a gyroscope, an electronic compass, a magnetometer, or an inertial measurement unit.

31. The method of claim 29, wherein: The third sensor includes a GPS sensor.

32. The method of claim 29, wherein: The third sensor includes an ambient light sensor.

33. The method of claim 29, wherein: The first eye movement behavior is determined using a neural network.

34. The method of claim 29, further comprising: A neural network is trained using information including the first data, the second data, the third data, the third position of the eye, or the first eye movement behavior.

35. The method of claim 34, further comprising: Based on the neural network, a second eye movement behavior is determined.

36. The method of claim 20, wherein: The first sensor and the second sensor are attached to a head mounted device including a display.

37. The method of claim 36, further comprising: In response to determining the third position of the eye: determining an area of ​​the display corresponding to the second position of the eye, the area having a display state equal to a first display state; as well as The display state of the area is changed from the first display state to a second display state.

38. The method of claim 20, further comprising: determining an incremental state where the incremental position of the eye is less than a threshold, Wherein the first sensor is configured to operate in a low power mode in response to the determined delta state.

39. A system comprising: a first sensor configured to output first data indicating a first position of an eye, wherein the first sensor is configured to obtain the first data at a first frequency; a second sensor configured to output second data indicative of a second position of the eye, wherein the second sensor is different from the first sensor, and wherein the second sensor is configured to obtain the second data at a second frequency higher than the first frequency; One or more processors operably coupled to the first sensor and the second sensor, the one or more processors configured to perform a method comprising: receiving the first data indicating the first position of the eye, the first data being received at a first time interval; receiving second data indicating the second position of the eye, the second data being received at a second time interval, determining the first position of the eye based on the first data; determining the second position of the eye based on the second data; determining a third position of the eye based on the first position of the eye and the second position of the eye; and An output signal is generated that is indicative of the third position of the eye.

40. The system of claim 39, wherein: The first sensor includes an optical sensor.

41. The system of claim 39, wherein: The second sensor includes an electro-oculogram sensor.

42. The system of claim 39, wherein: The first sensor is configured to operate in a low power mode during the first time interval.

43. The system of claim 39, wherein: The second sensor is configured to operate in a low power mode during the second time interval.

44. The system of claim 39, wherein: The method further comprises: Based on the third position of the eye, a first eye movement behavior is determined.

45. The system of claim 44, wherein: The method further comprises: determining a third time interval for receiving data from the first sensor based on the first eye movement behavior, and Based on the first eye movement behavior, a fourth time interval for receiving data from the second sensor is determined.

46. ​​The system of claim 44, further comprising a third sensor, wherein: The method further comprises receiving third data from the third sensor, and wherein the first eye movement behavior is determined based on the third data.

47. The system of claim 46, wherein: The third sensor includes an accelerometer, a gyroscope, an electronic compass, a magnetometer, an inertial measurement unit, a GPS sensor, or an ambient light sensor.

48. The system of claim 39, wherein: The first sensor and the second sensor are attached to a head mounted device including a display.

49. The system of claim 48, wherein: The method further comprises: determining an area of ​​the display corresponding to the third position of the eye, the area having a display state equal to a first display state; and The display state of the area is changed from the first display state to a second display state.

50. The system of claim 39, wherein: The method also includes determining a delta state based on the first position of the eye and the second position of the eye, and wherein the first sensor is configured to operate in a low power mode in response to the determined delta state.

51. A non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method comprising: receiving first data indicating a first position of an eye, the first data being received at a first time interval, the first data being output by a first sensor, the first sensor being configured to output data indicating the first position of the eye; receiving second data indicating a second position of the eye, the second data being received at a second time interval, the second data being output by a second sensor, the second sensor being configured to output data indicating the second position of the eye, wherein the second sensor is different from the first sensor, wherein the first sensor is configured to obtain the first data at a first frequency, and wherein the second sensor is configured to obtain the second data at a second frequency higher than the first frequency; determining the first position of the eye based on the first data; determining the second position of the eye based on the second data; determining a third position of the eye based on the first position of the eye and the second position of the eye; and An output signal is generated that is indicative of the third position of the eye.

52. The non-transitory computer readable medium of claim 51, wherein: The first sensor includes an optical sensor.

53. The non-transitory computer readable medium of claim 51, wherein: The second sensor includes an electro-oculogram sensor.

54. The non-transitory computer readable medium of claim 51, wherein: The first sensor is configured to operate in a low power mode during the first time interval.

55. The non-transitory computer readable medium of claim 51, wherein: The second sensor is configured to operate in a low power mode during the second time interval.

56. The non-transitory computer readable medium of claim 51, the method further comprising: Based on the second position of the eye, a first eye movement behavior is determined.

57. The non-transitory computer readable medium of claim 56, the method further comprising: determining a third time interval for receiving data from the first sensor based on the first eye movement behavior, and Based on the first eye movement behavior, a fourth time interval for receiving data from the second sensor is determined.

58. The non-transitory computer readable medium of claim 56, wherein: The method further comprises receiving third data from a third sensor, and wherein the first eye movement behavior is determined based on the third data.

59. The non-transitory computer readable medium of claim 58, wherein: The third sensor includes an accelerometer, a gyroscope, an electronic compass, a magnetometer, an inertial measurement unit, a GPS sensor, or an ambient light sensor.

60. The non-transitory computer readable medium of claim 51, wherein: The first sensor and the second sensor are attached to a head mounted device including a display.

61. The non-transitory computer readable medium of claim 60, wherein: The method further comprises: determining an area of ​​the display corresponding to the third position of the eye, the area having a display state equal to a first display state; and The display state of the area is changed from the first display state to a second display state.

62. The non-transitory computer readable medium of claim 51, wherein: The method further includes determining a delta state based on the first position of the eye and the second position of the eye, and Wherein the first sensor is configured to operate in a low power mode in response to the determined delta state.

Images