Eye Tracking System

Abstract

The present application provides an eye tracking system, comprising a machine, an eye tracking device and a signal processing unit. The machine comprises a display; the eye tracking device comprises a single or multiple optical emitters, one or multiple photodetectors and a circuit. The optical emitter is used to emit one or more specific wavelengths to illuminate the user's eyes; the photodetector is used to receive the light signal reflected by the user's eyes; the circuit is used to: obtain an electrical signal corresponding to the light signal measured by the detector; determine the depth map of the user's eyes based on the phase difference between the reference signal and the electrical signal; and determine the line of sight information representing the user's eyes based on the depth map. The signal processing unit receives output data representing the line of sight information from the eye tracking device, and determines the relationship between the display and the line of sight information.

Description

[0001] This application is a divisional application of the application with national application number 201680076422.4 (international application number PCT / US2016 / 066073, international application date December 12, 2016, invention name “Eye gesture tracking”).

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of U.S. Patent Application No. 15 / 359,460, filed on November 22, 2016, entitled “EYE GESTURE TRACKING,” which is a continuation-in-part of and claims the benefit of U.S. Patent Application No. 15 / 228,282, filed on August 4, 2016, entitled “GERMANIUM-SILICON LIGHT SENSING APPARATUS,” both of which claim the benefit of U.S. Provisional Application No. 62 / 363,179, filed on July 15, 2016. U.S. Patent Application No. 15 / 228,282 claims the benefit of U.S. Provisional Patent Nos. 62 / 200,652, filed on August 4, 2015; 62 / 209,349, filed on August 25, 2015; 62 / 210,946, filed on August 27, 2015; 62 / 210,991, filed on August 28, 2015; 62 / 211,004, filed on August 28, 2015; 62 / 217,031, filed on September 11, 2015; 62 / 251,691, filed on November 6, 2015; and 62 / 271,386, filed on December 28, 2015. Each of these prior applications is incorporated herein by reference in its entirety. Background Art

[0004] This specification generally relates to eye gesture tracking.

[0005] Light can be directed toward the eye and the reflected light can be observed. The reflected light can be processed to determine information about the eye. Summary of the invention

[0006] In some specific implementations, an eye gesture tracking method can be used to determine the gaze information of the eye. The eye gesture tracking method may include demodulating a modulated light signal reflected from the eye. The demodulated signal can be processed to generate a depth map of the eye and further determine the gaze information of the eye. The gaze information of the eye may include information representing, for example, the pupil or iris of the eye, which can then be used for various applications, such as determining user preference data, visually controlling a human-computer interaction device, providing cross-platform peripheral control, etc. In addition, by tracking the gesture of the eye, the corresponding eye gaze information can be used to refocus the tunable optical element in real time to change the light incident on the eye, thereby forming, for example, a viewing experience without nausea. The eye gesture tracking method can also be used on various platforms to provide an enhanced viewing experience through dynamically refocused optical elements, thereby providing, for example, three-dimensional (3D) foveated imaging.

[0007] One innovative aspect of the subject matter described in this specification is embodied in a method comprising the following actions: obtaining an electrical signal representing a measurement result of a light signal reflected from an eye by a photodetector, and determining a depth map of the eye based on a phase difference between the electrical signal generated by the photodetector and a reference signal. In addition, the method may include the following actions: determining information representing eye gaze based on the depth map, and providing output data representing the gaze information.

[0008] Other implementations of this and other aspects include corresponding systems, apparatus, and computer programs configured to perform the actions of the methods and encoded on computer storage devices.

[0009] Specific implementations may each optionally include one or more of the following features. For example, the method may include providing one or more filters for light signals reflected from the eye to remove non-target wavelength signals. In addition, the method may include providing one or more lenses for light signals reflected from the eye to focus the light signals onto a photodetector. The depth map may include one or more data sets of 3D information. Gaze information may include identifying one or more of the following items of the eye: a specific area, a pupil, an iris, or a physiological structure. In some aspects, providing output data representing gaze information includes providing the output data representing gaze information as input data to another device, machine, or system.

[0010] The method may include determining an eye gesture based on gaze information and providing output data representing the eye gesture. In this case, the eye gesture may include one or more of the following items of the eye: movement, rotation, stable state and duration thereof, closed eye state and duration thereof, open eye state and duration thereof, blinking state and duration thereof or frequency thereof. In addition, providing output data representing the eye gesture may include providing the output data representing the gaze information as input data to another device, machine or system.

[0011] In some aspects, the light signal reflected from the eye is generated by one or more light sources biased by a modulated signal that is synchronized with a reference signal. The method may include generating an iris vector perpendicular to a plane tangential to the eye and determining information representative of the eye's gaze based on the depth map and the iris vector. The method may also include generating a pupil position of the eye on a plane tangential to the eye and determining information representative of the eye's gaze based on the depth map and the pupil position.

[0012] Another innovative aspect of the present disclosure may be embodied in a system including a machine having a display, the display including a plurality of tunable optical elements. The system may also include a device having a circuit configured to obtain an electrical signal representing a measurement result of a photodetector on a light signal reflected from an eye. The circuit may be further configured to determine a depth map of the eye based on a phase difference between a reference signal and the electrical signal generated by the photodetector, and to determine information representing eye gaze based on the depth map. In addition, the system may include one or more processors in communication with the machine and the device, the one or more processors including one or more storage devices storing instructions, the instructions being operable when executed by the one or more processors so that the one or more processors perform operations including the following steps: receiving output data representing gaze information from the device; and determining information representing the gaze of the eye relative to the display of the machine.

[0013] In some aspects, the operations may also include determining a specific location on the display where the eye is focused based on information representing a gaze of the eye relative to the display and providing an indication at the specific location on the display. The operations may include determining a specific location on the display where the eye is focused based on information representing a gaze of the eye relative to the display and providing a fovea image at a specific area on the display. The plurality of tunable optical elements may include a tunable element or a tunable mirror. In this case, tuning of a subset of the plurality of tunable optical elements is activated based on the information representing a gaze of the eye relative to the display. Further, tuning of the subset of the plurality of tunable optical elements may include dynamically refocusing light incident on the subset of the plurality of tunable optical elements.

[0014] The system may include a wearable device coupled to a machine, an apparatus, and one or more processors to form an integrated hardware package, the display of the machine being opaque, wherein a visible image is displayed on the display by one or more light sources in an array of light sources. In certain aspects, the system may include a wearable device coupled to a machine and an apparatus to form an integrated hardware package, the display of the machine being opaque, wherein a visible image is displayed on the display by one or more light sources in an array of light sources, the one or more processors being located at a remote location and communicating with the integrated hardware package via a wireless or wired connection. In other aspects, the system may include a wearable device coupled to a machine, an apparatus, and one or more processors to form an integrated hardware package, the display of the machine being at least partially transparent to an image projected toward the display, whereby properties of the image projected toward the display are modified by one or more of a plurality of tunable optical elements of the display.

[0015] In addition, the system may include a wearable device coupled to a machine and a device to form an integrated hardware package, the display of the machine being at least partially transparent to an image projected toward the display, whereby properties of the image projected toward the display are modified by one or more of a plurality of tunable optical elements of the display, the one or more processors being located at a remote location and communicating with the integrated hardware package via a wireless or wired connection. The system may also include a pluggable device coupled to the device and the one or more processors to form an integrated hardware package, the machine being located at a remote location and communicating with the integrated hardware package via a wireless or wired connection, the display of the machine being opaque, wherein a visible image is displayed on the display by one or more light sources in an array of light sources.

[0016] In some aspects, the system may include a wearable device coupled to the device and one or more processors to form an integrated hardware package, the machine being located at a remote location and communicating with the integrated hardware package via a wireless or wired connection, the display of the machine being opaque, wherein a visible image is displayed on the display by one or more light sources in an array of light sources. In this case, the operations may also include determining a specific location on the display where the eye is focused, the specific location being based on information representing the gaze of the eye relative to the display and providing an indication at a specific location on the display. In some aspects, the light signal reflected from the eye is generated by a light source biased by a modulated signal that is synchronized with a reference signal.

[0017] Another innovative aspect of the present disclosure may be embodied in a device including multiple tunable optical elements for adjusting focal length. The wearable device may also include one or more processors, the one or more processors including one or more storage devices storing instructions, the instructions being operable when executed by the one or more processors so that the one or more processors perform operations including the following steps: obtaining an electrical signal representing a measurement result of a photodetector on a light signal reflected from an eye; determining a depth map of the eye based on a phase difference between a reference signal and the electrical signal generated by the photodetector. The operations may also include determining information representing an eye gaze based on the depth map, the gaze information representing a gaze of the eye relative to a display of a remote device and activating tuning of a subset of the multiple tunable optical elements based on the gaze information.

[0018] Advantageous implementations may include one or more of the following features. The eye gesture tracking method of the present disclosure may be used to provide cross-platform peripheral control. Cross-platform peripheral control may be used to exchange information between multiple devices. The exchanged information may include eye gesture information, commands corresponding to the eye gesture information, gaze position of the eye, etc. Compared with traditional eye tracking schemes, such cross-platform peripheral control can be used to expand the operating area. Thus, since traditional eye tracking schemes are limited to limited detection areas and positioning of specific devices, the eye gesture tracking method of the present disclosure provides a larger operating area that is not constrained like traditional eye tracking schemes. In addition, more than one user can apply cross-platform peripheral control to multiple devices at the same time, thereby effectively creating user-to-user interaction.

[0019] In addition, the eye gesture tracking method of the present disclosure can be used to provide a viewing experience without nausea. In some aspects, the eye gesture tracking information can be used in an optical system that refocuses the image using tunable optical elements based on the eye gesture tracking information and known distance information. The tunable optical element adjusts the angle of incident light to the eye to provide real-time focusing. Real-time focusing based on the eye gesture tracking method of the present disclosure can reduce the feeling of nausea by maintaining consistent depth perception between the user's eyes and brain. In addition, the eye gesture tracking information can be used to control a subset of tunable optical elements that form a foveal focus, where the focal length of each area in the image presented to the observer can be controlled to be different. Unlike traditional foveal rendering that provides an artificial 3D effect through complex algorithms, the foveal focus of the present disclosure provides a natural 3D effect through simple tunable optical devices.

[0020] The details of one or more embodiments of the present invention are described in the accompanying drawings and the detailed description below. Other features and advantages of the present invention will become apparent from the detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1A is an exemplary schematic diagram of an eye gesture tracking system.

[0022] Figure 1B is an exemplary schematic diagram of a time-of-flight device.

[0023] Figure 1C is an exemplary schematic diagram of a time-of-flight device.

[0024] Figure 1D and Figure 1E is an example technique for determining characteristics of a user's eyes.

[0025] Figure 1F is an exemplary schematic diagram of the phases of charge collection.

[0026] Figure 1G is an exemplary schematic diagram of light emission, detection, and charge collection.

[0027] Figure 1H is an exemplary schematic diagram of the signal voltage during charge collection.

[0028] Fig. 1I is an exemplary schematic diagram of the offset phase of charge collection.

[0029] Figure 1J is an exemplary schematic diagram of light emission, detection, and phase-shifted charge collection.

[0030] Figure 1K is an exemplary schematic diagram of the signal voltage during phase-shifted charge collection.

[0031] Figure 1L is an exemplary schematic diagram of a time-of-flight device.

[0032] Figure 2A is an exemplary schematic diagram of a cross-platform peripheral control system using eye gesture tracking.

[0033] Figure 2B is an exemplary schematic diagram of a cross-platform peripheral control system using eye gesture tracking.

[0034] Figure 3A is an exemplary schematic diagram of a wearable device using eye gesture tracking.

[0035] Figure 3B is an exemplary schematic diagram of an optical image refocusing system using a lens.

[0036] Figure 3C is an exemplary schematic diagram of an optical image refocusing system using a mirror.

[0037] Figure 4is an exemplary schematic diagram of a wearable device using eye gesture tracking.

[0038] Figure 5A is an exemplary schematic diagram of a stand-alone eye gesture tracking device attached to a machine.

[0039] Figure 5B is an exemplary schematic diagram of an embedded eye gesture tracking device packaged in a machine.

[0040] Figure 6 is a flow chart illustrating a process for eye gesture tracking.

[0041] Figure 7 is a flow chart illustrating a process for tuning optical elements based on eye gesture tracking.

[0042] Figure 8 is an exemplary schematic diagram of a computing device and a mobile computing device.

[0043] Like reference numbers and designations throughout the various drawings represent like elements. DETAILED DESCRIPTION

[0044] Eye gesture tracking methods can be used to determine gaze information related to tracked eyes. The method may include illuminating the eye and detecting reflected light signals from the eye to track the gaze direction and focus of the eye. Determining the gaze direction and focus of the eye may be useful when communicating with other devices. For example, gaze information of the eye can be used to provide one or more commands to another device. In some specific implementations, gaze information and / or other information (such as gestures) can be detected by the system described herein embedded in a mobile phone, and the mobile phone can be used as a remote control that receives commands from the user and connects to other devices such as tablets, televisions, etc. to execute commands. In some specific implementations, gaze information can include eye gestures. In this way, eye gestures such as movement, rotation, state, etc. can be used to indicate certain commands to be provided to another device. In some specific implementations, gaze information of the eye can be used to determine the focus position of the eye, such as the focus position of the eye on a specific display. In this case, the position of the focus of the eye relative to the display can be used to collect information indicating the user's interest. For example, if an advertisement is provided on the display, the focus of the user's eye relative to the position of the advertisement provided on the display can be used to determine the content of interest to the user. Thus, the location of an eye gaze and, for example, how long the eyes maintain that particular gaze may help determine a user's level of interest in content provided on a particular display.

[0045] In some embodiments of the present disclosure, eye posture tracking methods may be incorporated into wearable devices and / or peripheral devices. For example, a wearable device may be used to provide illumination at the eye and detect reflected light signals from the eye. The wearable device may include components such as an accelerometer, a gyroscope, or both to help track the eye and the focus of the eye on a particular display, so that the eye posture can be effectively and continuously tracked. In some embodiments, the wearable device may also include a tunable optical element for optical path adjustment. The tunable optical element may include a reflector and / or lens that is adjusted based on the movement of the tracked eye or its lack of movement. The tunable optical element may be used to provide real-time dynamic focusing and defocusing to help the eye view a specific object or display. For example, when viewing an image on a virtual reality (VR) or augmented reality (AR) display, a tunable optical element may be used to resolve inconsistencies between accommodation and convergence. In some embodiments, components of the wearable device may be implemented externally in a remote device separate from the wearable device. Eye tracking methods may be used to provide eye gaze specific data as an output and use that output to provide commands at a remote device and / or tunable optical elements to facilitate a variety of viewing experiences.

[0046] Figure 1A is an exemplary schematic diagram of an eye gesture tracking system 100. The eye gesture tracking system 100 may be used to process information of a user's eyes in response to generating a depth map of the eyes. The eye gesture tracking system 100 includes an eye gesture tracking device 110 for tracking the movement of a user's eyes 120, a graphics display 130, a signal processing unit 140 for processing eye data detected at the eye gesture tracking device 110, and optionally a console 170 for providing additional user input to the system depending on the nature of the application. The user's eyes 120 may include a single eye or both eyes of the user viewing the graphics display 130.

[0047] The graphic display 130 may be one or more graphic displays on a computer, laptop, desktop computer, television, smart phone, tablet computer, etc. The graphic display 130 may include a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a head mounted display (HMD), etc. In some implementations, the graphic display 130 may include tunable optical elements, such as a reflector and / or a tunable lens. In this case, the tunable optical elements of the graphic display 130 may be configured to adjust the focus and defocus in real time to help the user's eyes 120 view the graphic display 130.

[0048] The eye gesture tracking device 110 may include one or more eye gesture tracking devices in communication with the signal processing unit 140. The eye gesture tracking device 110 may provide illumination at the user's eye 120 and receive reflected light signals from the user's eye 120. The eye gesture tracking device 110 may include a modulated light source that illuminates the user's eye 120 at one or more selected wavelengths. The modulated light source may include a single light emitter or multiple light emitters modulated by a radio frequency (RF) or microwave frequency voltage source that provides the illumination. In some implementations, the light emitter may be used to illuminate an entire portion of the user's eye 120. In other implementations, the light emitter may be used to illuminate a selected portion of the user's eye 120. The one or more wavelengths used for the eye gesture tracking system 100 may be predetermined based on various criteria, such as non-permeability to the human eye, low solar irradiance at sea level, eye safety, etc.

[0049] In some implementations, the eye gesture tracking device 110 may include one or more photodetectors for receiving reflected light signals from the user's eye 120. The reflected light signals from the user's eye 120 may be reflections of modulated light signals provided by the eye gesture tracking device 110. In some implementations, the eye gesture tracking device 110 may detect the reflected modulated light signals through one or more photodetectors. The photodetectors may be implemented using the technology described in U.S. Patent Application No. 15 / 338,660, entitled “High-Speed ​​Light Sensing Apparatus,” filed on October 31, 2016, and U.S. Patent Application No. 15 / 228,282, entitled “GERMANIUM-SILICON LIGHT SENSING APPARATUS,” filed on August 4, 2016.

[0050] The signal processing unit 140 may include one or more signal processing units in communication with the graphic display 130 and the eye gesture tracking device 110. The signal processing unit 140 may be configured to determine gaze information 150 of the user's eye 120 via data corresponding to the eye gesture tracking device 110 and the graphic display 130. The eye gesture tracking device 110 may be configured to demodulate the reflected modulated light signal. In addition, the eye gesture tracking device 110 may be configured to create a depth map of the illuminated portion of the user's eye 120. The depth map may correspond to the reflected light signal detected by the photodetector of the eye gesture tracking device 110. Specifically, the depth map may provide two-dimensional (2D) and three-dimensional (3D) information related to the user's eye 120. The signal processing unit 140 may process the depth map based on data representing the time-of-flight information of the reflected light signal. In some specific implementations, the depth map may be based on a phase difference between the reflected light signal and a reference signal. For example, the eye gesture tracking device 110 may provide a comparison between the reflected light signal and the reference signal, and may be used to determine a depth map of the user's eye 120. The depth map may also include a 3D model representing the user's eye 120. In this way, a 3D eye model may be generated and constructed, thereby allowing the signal processing unit 140 to determine gaze information 150 of the user's eye 120.

[0051] The signal processing unit 140 may be located near the user's eyes 120. For example, the signal processing unit 140 and the eye gesture tracking device 110 may be implemented in a single wearable device located at a nearby location near the user's eyes 120. The signal processing unit 140 and the eye gesture tracking device 110 may also be implemented in a single peripheral device located at a remote location away from the user's eyes 120. In other specific implementations, the signal processing unit 140 may be located separately from the eye gesture tracking device 110. For example, the signal processing unit 140 may be located at the graphics display 130 and communicate with the eye gesture tracking device 110 implemented in a single wearable or peripheral device.

[0052] The gaze information 150 may include information such as the gaze direction and focus of the user's eyes. The gaze information 150 may be determined by the signal processing unit 140 relative to the light signal received by the eye gesture tracking device 110. The gaze information 150 may be used to analyze the user's eye behavior. In addition, the gaze information 150 may be used to identify the position of the focus of the user's eyes 120 relative to the display 130. In this case, the gaze information 150 may be used to determine the specific item displayed on the display 130 on which the user's eyes 120 are focused. Therefore, the user's interest may be determined without physically actuating a specific device. For example, an advertising provider may determine the user's interest based solely on the user's eyes 120 without the need for activation / detection via a computer mouse, a computer trackpad, a touch screen, etc. In other cases, physically actuating a specific device may be used to perform certain functions of the user and system interaction. As the complexity of the interaction between the system and the user increases, the use of such devices may improve efficiency. For example, a fighter pilot may use the eye gaze information 150 to identify / select a target of interest on the display 130 and use the console 170 to perform tasks on the target of interest, such as target acquisition, target priority assignment, weapon selection, etc.

[0053] In some implementations, gaze information 150 may be used to indicate a command to be provided to another device. In this case, gaze information 150 may include eye gestures such as movement, rotation, closed eye state, open eye state, any duration thereof, etc. The device receiving gaze information 150 may analyze gaze information 150 in real time to determine a command as eye gesture tracking device 110 dynamically tracks the user's eyes 120.

[0054] The eye gesture tracking device 110, the graphic display 130, and the signal processing unit 140 may be independent structures or connected together in an integrated hardware package. For example, the eye gesture tracking device 110, the graphic display 130, and the signal processing unit 140 may be integrated in a single hardware package, wherein the display of the graphic display 130 is opaque and displays a visible image on the display through an array of light-emitting diodes that generate visible light, a liquid crystal that filters white light, or any other array of light sources. In some specific implementations, the display of the graphic display 130 is at least partially transparent and projects a visible image onto the display through optical refraction, diffraction, reflection, guidance, or other optical means.

[0055] In another example, the eye gesture tracking device 110 and the signal processing unit 140 can be integrated into a single hardware package such as a wearable device. The wearable device can be a headset, a pair of glasses, or any other suitable wearable device. In this case, the wearable device communicates with a host or machine in which the graphics display 130 is embedded. In addition, the main frame or machine containing the graphics display 130 can communicate with the wearable device via a wireless or wired connection.

[0056] In another example, the eye gesture tracking device 110 and the signal processing unit 140 can be integrated into a single hardware package such as a pluggable device. The pluggable device can be a game console, a portable camera, or any other suitable pluggable device. In this case, the pluggable device communicates with the host or machine in which the graphics display 130 is embedded. In addition, the main frame or machine containing the graphics display 130 can communicate with the pluggable device via a wireless or wired connection.

[0057] Figure 1B is an exemplary schematic diagram of a time-of-flight device. The time-of-flight device may be integrated into the eye gesture tracking device 110 and may be used to determine a depth map of the user's eye 120 . Figure 1B The time-of-flight device includes a time-of-flight (TOF) pixel 160 and two sets of transistors. Figure 1B As shown, each group of transistors may include three switch transistors (3T), namely a reset transistor 162a or 162b, a source follower transistor 164a or 164b, and a select transistor 166a or 166b. In some other specific implementations, other arrangements of transistors may be used to achieve similar functions. The TOF pixel 160 may be one or more TOF pixels for detecting light. When the TOF pixel 160 detects light, the TOF pixel determines whether the charge should be processed by the first group of transistors or the second group of transistors. In some aspects, the received light signal may be out of phase with respect to the emitted light signal. In this case, the TOF pixel can be designed as a dual-switch TOF pixel so that one switch is modulated in phase and the other switch is modulated 180 degrees out of phase with respect to the emitted light signal to accommodate the received out-of-phase light signal. The dual-switch TOF pixel can be implemented using the technology described in U.S. patent application No. 15 / 338,660, entitled “High-Speed ​​Light Sensing Apparatus,” filed on October 31, 2016, and U.S. patent application No. 15 / 228,282, entitled “GERMANIUM-SILICON LIGHT SENSING APPARATUS,” filed on August 4, 2016.

[0058] In some aspects, these two groups of transistors can be manufactured together with TOF pixel 160 on a single wafer. In this case, the two groups of transistors can share and occupy the same illumination area as TOF pixel 160, thereby reducing the effective fill factor of the TOF device. The two groups of transistors can be implemented by NMOS gates. NMOS gates are used to reduce transistors and thus reduce the size of the TOF device. The two groups of transistors can also be implemented by PMOS gates. PMOS gates are used to increase certain operating parameters, such as providing a larger available voltage margin space. The PMOS and NMOS implementations of transistor groups will be further discussed in this article.

[0059] Figure 1C is an exemplary schematic diagram of a time-of-flight device. Figure 1C The TOF device includes a first wafer and a second wafer bonded together via a die or wafer bond 167. The first wafer may include a TOF pixel 165 manufactured on the first wafer. The TOF pixel 165 can be used to detect light pulse information. The second wafer can be a circuit wafer 169 including two sets of transistors. When light pulse information is detected on the TOF pixel 165, the circuit wafer 169 can be used to process charge. In some specific implementations, the transistors of the circuit wafer 169 do not occupy the illuminated area, thereby increasing the effective fill factor of the TOF device.

[0060] These two groups of transistors can be implemented by NMOS or PMOS gates. For example, each of the two groups of transistors can be implemented by an NMOS gate with a threshold voltage of 0.7 volts. In this case, when the gate voltage is provided at 3.3 volts, a maximum source voltage of about 2.6 volts can be obtained when the NMOS gate is turned on. Therefore, when NMOS is used as a reset transistor, the reset voltage applied to the TOF pixel can only be as high as 2.6 volts, resulting in a smaller voltage margin space. In contrast, another example may include each of the two groups of transistors implemented with a PMOS gate having a negative threshold voltage of -0.8 volts. In this case, when the gate voltage is provided at 0 volts, a maximum source voltage of about 3.3 volts can be obtained when the PMOS gate is turned on. Therefore, when PMOS is used as a reset transistor, the reset voltage applied to the TOF pixel can be as high as 3.3 volts, resulting in a larger voltage margin space.

[0061] Therefore, when implemented with a PMOS gate, these two sets of transistors can produce a larger available voltage headroom. This aspect of the PMOS implementation can be attributed in part to the negative threshold voltage. In addition, when the PMOS is turned on as a switch and passes a voltage whose value is close to the supply voltage, the PMOS implementation can produce a smaller impedance. In this way, the PMOS implementation of these two sets of transistors provides the operating benefits of the TOF device, however, the physical area of ​​the PMOS gate is larger than the physical area of ​​the NMOS gate, so the PMOS implementation requires a physically larger TOF device to provide such an implementation. Figure 1C As shown, the above problems can be solved when the TOF pixel and the PMOS circuit are implemented on two separate wafers, followed by wafer or die bonding to electrically connect the two separate wafers or dies. Figure 1B and 1C The TOF pixel shown in FIG. 1 may include a light absorbing layer including germanium. In some implementations, such as Figure 1B and Figure 1C The TOF pixel shown also includes a demodulation function implemented by a dual switch transistor or multiple PN junctions to implement the demodulation function. The dual switch TOF pixel can be implemented using the technology described in U.S. Patent Application No. 15 / 338,660, entitled "High-Speed ​​Light Sensing Apparatus" filed on October 31, 2016 and U.S. Patent Application No. 15 / 228,282, entitled "GERMANIUM-SILICON LIGHT SENSING APPARATUS" filed on August 4, 2016.

[0062] Figure 1D An exemplary technique for determining characteristics of a user's eye 120 is shown. The eye gesture tracking device 110 may emit light pulses at a frequency f m Modulation, duty cycle is 50%. The eye gesture tracking device 110 can receive reflected light pulses with a phase difference Φ. The photodiode can be controlled so that the reading circuit 1 reads the collected charge Q1 whose phase is synchronized with the emitted light pulse, and the reading circuit 2 reads the collected charge Q2 whose phase is opposite to the emitted light pulse. In some specific implementations, the distance D between the eye gesture tracking device 110 and a point of the user's eye 120 can be derived using the following formula:

[0063]

[0064] Where c is the speed of light. The eye gesture tracking device 110 may scan the user's eyes 120 to obtain a depth distribution of the user's eyes 120 .

[0065] Figure 1E Another exemplary technique for determining characteristics of the user's eye 120 is shown. The eye gesture tracking device 110 may emit light pulses at a frequency f m Modulation, with a duty cycle of less than 50%. By reducing the signal-to-noise ratio of the light pulse by a factor of N, but at the same time increasing the intensity of the light pulse by a factor of N, the signal-to-noise ratio of the received reflected light pulse can be improved while maintaining substantially the same power consumption for the eye gesture tracking device 110. This is achievable when the device bandwidth is increased so that the duty cycle of the light pulse can be shortened without deforming the pulse shape. The eye gesture tracking device 110 can receive reflected light pulses with a phase difference Φ. The photodiode can be controlled so that the reading circuit 1 reads the collected charge Q1' whose phase is synchronized with the emitted light pulse, and the reading circuit 2 reads the collected charge Q2' whose phase is delayed than the emitted light pulse. In some specific implementations, the distance D between the eye gesture tracking device 110 and a point on the user's eye 120 can be derived using the following formula:

[0066]

[0067] Figure 1F is an exemplary schematic diagram of the phase of charge collection. The phase of charge collection represents the phase in which the light pulse is emitted and the charge is collected by the eye gesture tracking device 110. The phase of charge collection includes a 0 degree phase, a 90 degree phase, a 180 degree phase, and a 270 degree phase, as well as a controllable phase shift A phase difference Φ may be observed between the light pulses emitted by the eye gesture tracking device 110 and the light pulses received by the eye gesture tracking device 110. In some implementations, the phase difference Φ occurs due to the distance between the user's eye 120 and the eye gesture reading device 110. A small phase difference may make it difficult for the eye gesture tracking device 110 to effectively detect gesture recognition of the user's eye 120, mapping of the user's eye 120, etc. Therefore, a phase shift is added to the collected charge. It may be beneficial so that eye gesture recognition may be performed efficiently.

[0068] Figure 1Gis an exemplary schematic diagram of light detection and charge collection. Light detection and charge collection include time steps of light emission, light detection, and charge collection at the eye gesture reading device 110. At each of the time steps, data is collected to represent the received light, the charge collected at a 0-degree phase, the charge collected at a 90-degree phase, the charge collected at a 180-degree phase, and the charge collected at a 270-degree phase. The charge collection for each phase may indicate the amount of charge collected at each received phase. In this case, the amount of charge collected at each time step in each phase may affect the accuracy of the eye gesture reading device 110 in measuring the user's eye 120.

[0069] For example, the eye gesture tracking device 110 may emit light pulses at a frequency f m Modulation, the duty cycle is 50%. The eye gesture tracking device 110 can receive a reflected light pulse with a phase difference Φ. The TOF pixel can be controlled so that the first readout circuit of the eye gesture tracking device 110 reads a collected charge Q0 that is in a phase synchronized with the emitted light pulse (thus corresponding to a 0 degree phase). The eye gesture tracking device 110 may also include a second readout circuit that reads a collected charge Q180 that is in a phase opposite to the emitted light pulse (such as a 180 degree phase). In another time step, the TOF pixel is controlled so that the first readout circuit reads a collected charge Q90 that is in an orthogonal phase (such as a 90 degree phase) relative to the emitted light pulse. In this case, the second readout circuit can read a collected charge Q270 that is in another orthogonal phase (such as a 270 degree phase) relative to the emitted light pulse. In some specific implementations, the distance D between the eye gesture tracking device 110 and the user's eye 120 can be derived using the following two formulas:

[0070] or

[0071]

[0072] See again Figure 1G , when the phase difference Φ between the light pulse emitted by the eye gesture tracking device 110 and the light pulse received by the eye gesture tracking device 110 is small, the charge collection at the 0 degree phase is the largest in the provided time step, and the charge collection at the 180 degree phase is the smallest in the provided time step. Such a large difference in charge collection may affect the accuracy of the entire charge collection. Therefore, by reducing the charge collection difference of each phase, a phase shift is introduced Eye gesture detection may be facilitated to achieve a more accurate depth map of the user's eyes 120 .

[0073] Figure 1His an exemplary schematic diagram of the signal voltage during charge collection. The signal voltage during charge collection shows the change of the signal voltage of multiple phases over time. Specifically, Figure 1H The change of the signal voltage at the 0 degree phase, 90 degree phase, 180 degree phase and 270 degree phase is shown. The decrease of the signal voltage of each phase over time indicates the amount of charge stored for a specific phase over a period of time. Figure 1H As shown, the signal voltage at the 180 degree phase is much higher than the signal voltage at the 0 degree phase. Therefore, the 180 degree phase includes a lower charge storage rate than the charge storage rate at the 0 degree phase. In this case, due to the difference between the charge storage rates of the different phases, the accuracy of the eye gesture tracking device 110 detecting the user's eye 120 may be negatively affected. Therefore, it may be beneficial to include a phase shift in the received light signal. To aid in charge collection so that a more accurate depth map of the user's eye 120 can be performed.

[0074] Fig. 1I is an exemplary schematic diagram of the offset phase of charge collection. The offset phase of charge collection includes a 45 degree phase, a 135 degree phase, a 225 degree phase, and a 315 degree phase. A phase difference Φ can be observed between the light pulse emitted by the eye gesture tracking device 110 and the light pulse received by the eye gesture tracking device 110. In some specific implementations, the phase difference Φ occurs due to the distance between the user's eye 120 and the eye gesture reading device 110. A small phase difference can make it difficult for the eye gesture tracking device 110 to effectively detect the user's eye 120 gesture recognition, the user's eye 120 mapping, etc. Fig. 1I The 45 degree phase shift of the collected charge is shown in This allows all phases to be offset by the same 45 degree phase shift

[0075] Figure 1J is an exemplary schematic diagram of light detection and phase-shifted charge collection. Light detection and phase-shifted charge collection include time steps of light emission, light detection, and charge collection at the eye gesture reading device 110. At each of the time steps, data is collected to represent the received light, the charge collected at a 45-degree phase, the charge collected at a 135-degree phase, the charge collected at a 225-degree phase, and the charge collected at a 315-degree phase. The charge collection for each phase may indicate the amount of charge collected at each received phase. In this case, the amount of charge collected at each time step in each phase may affect the accuracy of the eye gesture reading device 110 in measuring the user's eye 120.

[0076] For example, the eye gesture tracking device 110 may emit light pulses at a frequency f mModulation, the duty cycle is 50%. The eye gesture tracking device 110 can receive a reflected light pulse with a phase difference Φ. The TOF pixel can be controlled so that the first readout circuit of the eye gesture tracking device 110 reads a collected charge Q45 that is in an offset phase (such as a 45-degree phase) relative to the emitted light pulse. The eye gesture tracking device 110 may also include a second readout circuit that reads a collected charge Q225 that is in an offset phase (such as a 225-degree phase) relative to the emitted light pulse. In another time step, the TOF pixel is controlled so that the first readout circuit reads a collected charge Q135 that is 135 degrees phase shifted relative to the emitted light pulse. In this case, the second readout circuit can read a collected charge Q315 that is 315 degrees phase shifted relative to the emitted light pulse. In some specific implementations, the distance D between the eye gesture reading device 110 and the user's eye 120 can be derived using the following two formulas:

[0077] or

[0078]

[0079] See again Figure 1J , when the phase difference Φ between the light pulse emitted by the eye gesture tracking device 110 and the light pulse received by the eye gesture tracking device 110 is small, the charges collected at the 45 degree phase and the 225 degree phase are closer in the provided time step. Figure 1G In contrast, where charge collection is not phase-shifted The charges collected at 0 and 180 degrees are quite different. Figure 1J The phase-shifted charge collection provides higher eye mapping performance due to the smaller charge collection difference in each phase. Since the difference in charge collection can affect the accuracy of the entire charge collection, reducing the charge collection difference in each phase can help eye gesture detection to achieve a more accurate depth map of the user's eye 120.

[0080] Figure 1K is an exemplary schematic diagram of a signal voltage during a phase-shifted charge collection period. The signal voltage during a phase-shifted charge collection period shows changes in signal voltages of multiple phases over time. Specifically, Figure 1K The change in signal voltage for a 45 degree offset phase, a 135 degree offset phase, a 225 degree offset phase, and a 315 degree offset phase is shown. The decrease in signal voltage for each phase over time represents the amount of charge stored for a particular phase over a period of time. Figure 1K As shown in Figure 1HThe signal voltages of the offset phases include more similar average rates of signal voltage drop than the more different average rates of signal voltage drop shown. The similarity of the rate of drop of the signal voltages of the offset phases can enable higher accuracy of eye gesture detection and eye mapping of the user. Therefore, it may be beneficial to include phase shift in charge collection To aid in charge collection so that a more accurate reading of the user's eye 120 can be performed.

[0081] Figure 1L 1 is an exemplary schematic diagram of a TOF device. The TOF device includes a TOF pixel 190, two capacitors 192a and 192b, and two groups of transistors 194 and 196. Each group of transistors may include five switch transistors (5T). In some other specific implementations, other arrangements of transistors may be used to implement similar functions. The TOF pixel 190 may be one or more TOF pixels for detecting light. The charge generated by the TOF pixel 190 may be collected by two capacitors 192a and 192b. Transistors M1 to M4, which may be implemented by NMOS, PMOS, or any combination of NMOS and PMOS, are used to redistribute the collected charge by resetting the common mode charge and connecting the common mode voltage to VREF. The voltage VREF may be the operating voltage or a predetermined voltage of the TOF device 190, depending on the design constraints. Transistors M5 and M6, which may be implemented by NMOS, PMOS, or any combination of NMOS and PMOS, are used to reset the collected charge and connect them to VREF2. The voltage VREF2 may be the same voltage as VREF, the operating voltage or a predetermined voltage of the TOF device 190, depending on the design constraints.

[0082] Figure 2A2 is an exemplary schematic diagram of a cross-platform peripheral control system using eye gesture tracking. The cross-platform peripheral control system using eye gesture tracking may include a wearable device, such as a headset 201, and a connected device, such as a phone 220, a tablet 230, a computing device 240, and / or a television 250, that communicates with the headset 201. The headset 201 may be used by a pair of eyes 216A and / or 216B of a user to view a connected device, such as a phone 220, a tablet 230, a computing device 240, and / or a television 250. The headset 201 may include an eye tracking gesture device for tracking the gesture of one of a first eye 216A and a second eye 216B of the user and a signal processing unit implemented in an eye tracking module 213, an accelerometer 211 and a gyroscope 212 for determining the position of the user's head, a wireless communication unit 214 for communicating with a connected device, such as a phone 220 and / or a tablet 230 and / or a computing device 240 and / or a television 250, and a transparent lens 218. In some implementations, transparent lens 218 can include one or more tunable elements for adjusting based on tracking of the user's eyes 216A and / or 216B.

[0083] Here, the eye tracking module 213 can be used to illuminate the user's eyes 216A with light signals, and detect light signals reflected from the user's eyes 216A. The detected light signals can be used to determine gaze information related to the user's eyes 216A. The gaze information may include the gaze of the user relative to the display of the connected device. The gaze information may also include commands corresponding to the gestures of the user's eyes 216A. Eye gesture commands may be provided as input commands to the connected device. In some specific implementations, the eye tracking module 213 can illuminate the user's eyes 216A and 216B with light signals, and detect light signals reflected from the user's eyes 216A and 216B to determine gaze information of the binoculars 216A and 216B.

[0084] The accelerometer 211 and the gyroscope 212 may be used to detect the orientation of the user's head. The orientation of the user's head may be used to effectively determine gaze information. In addition, the accelerometer 211 and the gyroscope 212 may be used to track the movement of the user's head. Thus, any possible head movement of the user may be identified so that the gaze information is not incorrectly represented based on the user's head movement.

[0085] The wireless communication unit 214 may be used to establish a connection between the headset 201, the phone 220, the tablet 230, the computing device 240 and / or the TV 250 via a network. The network may include Wi-Fi, Bluetooth, Bluetooth Low Energy (BLE), a local area network (LAN), etc.

[0086] The transparent lens 218 can be used to help the user's eyes 216A and 216B view the display of the phone 220, tablet 230, computing device 240, and / or television 250. The transparent lens 218 can include a tunable optical element that can be tuned based on the determined gaze information representing the tracking of the user's eyes 216A and 216B. In some implementations, the entirety of the transparent lens 218 can be tuned based on the gaze information. In other implementations, a selected portion of the transparent lens 218 can be tuned based on the gaze information. For example, a selected portion of the tunable optical element of the transparent lens 218 can be tuned to provide a foveated image at a specific location on the display of the phone 220, where the specific location at the phone 220 is based on the location on the display to which the gaze of the user's eyes 216A and 216B is directed.

[0087] In some implementations, the phone 220 can include an accelerometer 221 and a gyroscope 222 for determining the orientation of the phone 220, and a wireless communication unit 224 for communicating with the headset 201. The accelerometer 221 and the gyroscope 222 of the phone 220 can help track the position and movement of the phone 220. By tracking the position and movement of the phone 220, the headset 201 can effectively determine the gaze information of the user's eyes 216A and 216B when comparing the user's focus to the position of the phone 220. The position and movement of the phone 220 can be transmitted from the phone 220 to the headset 201 via the wireless communication device 224.

[0088] In some implementations, the tablet computer 230 can include an accelerometer 231 and a gyroscope 232 for determining the orientation of the tablet computer 230, and a wireless communication unit 234 for communicating with the headset 201. The accelerometer 231 and the gyroscope 232 of the tablet computer 230 can help track the position and movement of the tablet computer 230. In tracking the position and movement of the tablet computer 230, the headset 201 can effectively determine a reference gaze point 236 of the user's eye 216A. The position and movement of the tablet computer 230 can be transmitted from the tablet computer 230 to the headset 201 via the wireless communication unit 234.

[0089] The computing device 240 may include a wireless communication unit 244 for communicating with the headset 201. Additionally, the television 250 may include a wireless communication unit 254 for communicating with the headset 201.

[0090] Figure 2Bis an exemplary schematic diagram of a cross-platform peripheral control system using eye gesture tracking. The cross-platform peripheral control system may include a wearable device, such as a headset 202, and a connected device, such as a phone 220, a tablet 230, a computing device 240, and / or a television 250, that communicates with the headset 202. The headset 202 may be used by the user's eyes 216A and 216B to view the connected device. The headset 202 may include two eye tracking gesture devices and signal processing unit pairs for tracking the gestures of the user's eyes 216A and 216B, the first pair being implemented in a first eye tracking module 213A and the second pair being implemented in a second eye tracking module 213B, an accelerometer 211 and a gyroscope 212 for determining the user's head position, a wireless communication unit 214 for communicating with the connected device, a first transparent lens 218A including one or more tunable elements, and a second transparent lens 218B including one or more tunable elements.

[0091] The first eye tracking module 213A can be used to illuminate the eye 216A of the first user with a light signal and detect the light signal reflected from the eye 216A of the first user. The detected light signal can be used to determine gaze information related to the eye 216A of the first user. The gaze information can include the gaze of the first eye of the user relative to the display such as 220, 230, 240 and 250 of the connected device. The gaze information can also include a command corresponding to the gesture of the eye 216A of the first user. The eye gesture command can be provided as an input command to the connected device.

[0092] The second eye tracking module 213B can be used to illuminate the eye 216B of the second user and detect light signals reflected from the eye 216B of the second user. The detected light signals can be used to determine gaze information related to the eye 216B of the second user. The gaze information can include the gaze of the eye 216B of the second user relative to the display of the connected device. The gaze information can also include commands corresponding to the gesture of the eye 216B of the second user. The eye gesture command can be provided as an input command to the connected device.

[0093] The first transparent lens 218A can be used to help the first user's eye 216A view the display of the connected device. The first transparent lens 218A may include a tunable element that can be tuned based on determined gaze information representing tracking of the first user's eye 216A. In some implementations, the entirety of the first transparent lens 218A can be tuned based on the gaze information. In other implementations, a selected portion of the first transparent lens 218A can be tuned based on the gaze information. For example, a selected portion of the tunable optical element of the first transparent lens 218A can be tuned to a small concave image at a specific location on the display of the computing device 240, where the specific location at the computing device 240 is based on the location on the display where the first user's eye 216A is focused.

[0094] The second transparent lens 218B can be used to help the second user's eye 216B view the connected device. The second transparent lens 218B may include a tunable optical element that is tuned based on determined gaze information representing tracking of the second user's eye 216B. In some implementations, the entirety of the second transparent lens 218B can be tuned based on the gaze information. In other implementations, a selected portion of the second transparent lens 218B can be tuned based on the gaze information. For example, a selected portion of the tunable optical element of the second transparent lens 218B can be tuned to provide enhanced focus of a specific location on the display of the computing device 240, where the specific location at the computing device 240 is based on the location where the second user's eye 216B is focused.

[0095] In some implementations, the first user's eye 216A and the second user's eye 216B can be focused at a single location. For example, the user's eyes 216A and 216B can include a reference gaze 246 located at a display of a computing device 240, such as a laptop or desktop computer. Although the reference gaze 246 can be directed to a single point on the display of the laptop or desktop computer, the tunable optical element of each of the first transparent lens 218A and the second transparent lens 218B can be independently tuned based on the determined gaze information of the first user's eye 216A and the second user's eye 216B.

[0096] Figure 3Ais an exemplary schematic diagram of a wearable device 300 using eye gesture tracking. The wearable device 300 may include a single vision wearable device that provides optical path adjustment based on eye gesture tracking. The wearable device 300 includes a transparent or opaque screen 310 that a user can see through or view, a tunable optical element 330 for adjusting the optical path at the transparent or opaque screen 310, a wireless communication unit 340 for communicating with a remote device, an image projector 350 for projecting 2D vision through or at the transparent or opaque screen 310, and an eye gesture tracking module 360 ​​for tracking the eye gestures of the user's eyes 320A and 320B and determining a depth map corresponding to each of the user's eyes 320A and 320B.

[0097] The gaze 325 of the user's eyes 320A and 320B can be determined by the eye gesture tracking module 360. In some specific implementations, only certain portions of the transparent or opaque screen 310 are tuned according to the gaze 325 of the user's eyes 320A and 320B. The eye gesture tracking module 360 ​​can use the gaze information corresponding to the gaze 325 to tune the selected portion of the transparent or opaque screen 310, such as a plurality of tunable optical elements 330. The tunable optical element 330 can be tuned to adjust the focus / defocus of a specific light path passing through a specific portion of the transparent or opaque screen 310. The tunable optical element 330 may include a tunable reflector, a tunable lens, a tunable grating, or any other suitable tunable optical element and any combination thereof. The tunable optical element 330 can be adjusted based on the gaze information corresponding to the gaze 325 so that real-time focus / defocus can be provided at the wearable device 300.

[0098] The real-time focus / defocus of the tunable optical element 330 can be used to resolve the inconsistency between accommodation and vergence when viewing a display. For example, a conventional VR experience can cause nausea due to inconsistent depth perception mechanisms. An inconsistent depth perception mechanism is created when the focus of the user's eyes (accommodation) perceives the image as being at the same distance from the display, while the convergence of the user's eyes (vergence) simultaneously perceives the image as being at different depths. These conflicting sensations felt by the user due to the inconsistency between accommodation and vergence can lead to nausea.

[0099] To resolve this inconsistent depth perception mechanism, among other inconsistencies, the eye tracking method of the present invention can be implemented in a wearable device such as wearable device 300. Wearable device 300 can refocus light based on eye gaze information to adjust the angle of eye-incident light passing through or at a selected portion of transparent or opaque screen 310. Therefore, tunable optical element 330 of transparent or opaque screen 310 can be configured to refocus light based on determined gaze information of the user's eyes 320A and 320B, thereby providing a solution to the inconsistency that may occur between accommodation and vergence during certain viewing experiences.

[0100] Figure 3B is an exemplary schematic diagram of an optical image refocusing system using a lens. The optical image refocusing system using a lens shows the use of a lens to refocus an object illusion according to gaze information of a user's eyes.

[0101] In Example 1 of the optical image refocusing system using a lens, the user's eye 320 is viewing the object 370 through a medium such as air, without using a screen such as a VR display. The user's eye 320 may not be viewing the object 370 through a transparent lens. In addition, the user's eye 320 is viewing the real object 370, not a virtual representation of the object.

[0102] In Example 2 of an optical image refocusing system using a lens, the user's eye 320 is viewing an object illusion 375 through a screen 380. In this case, the image projector may project a virtual representation of the object 370 through the screen 380 as the object illusion 375. In this case, the user's eye 320 may be experiencing a disparity between accommodation and vergence.

[0103] In Example 3 of an optical image refocusing system using a lens, the user's eye 320 is viewing an object illusion 375 through a lens 330 located between the user's eye 320 and a screen 380. The lens 330 may be a fixed lens for refocusing the object illusion 375. In other implementations, the lens 330 may be a tunable lens for dynamically refocusing the object illusion 375 through the screen 380 in real time. In this case, the lens 330 may be tuned based on the determined gaze information of the user's eye 320.

[0104] Figure 3C is an exemplary schematic diagram of an optical image refocusing system using a reflector. The optical image refocusing system using a reflector shows the use of a reflector to refocus an object illusion according to gaze information of a user's eyes.

[0105] In Example 1 of the optical image refocusing system using a reflector, the user's eye 320 is viewing the object 370 through a medium such as air, without using a screen such as a VR display. The user's eye 320 may not be viewing the object 370 through a transparent lens. In addition, the user's eye 320 is viewing the real object 370, not a virtual representation of the object.

[0106] In Example 2 of the optical image refocusing system using a mirror, the user's eye 320 is viewing the object illusion 376 through the screen 380. In this case, the image projector may project a virtual representation of the object 370 through the screen 380 as the object illusion 376. In this case, the user's eye 320 may be experiencing a disparity between accommodation and vergence.

[0107] In Example 4 of the optical image refocusing system using a mirror, the user's eye 320 is viewing an object illusion 376 through a screen 380 including a mirror 385. The mirror 385 may be a fixed mirror for refocusing the object illusion 376. In other implementations, the mirror 385 may be a tunable mirror for dynamically refocusing the object illusion 376 in real time through the screen 380 including the mirror 385. In this case, the mirror 385 may be tuned based on the determined gaze information of the user's eye 320.

[0108] Figure 4is an exemplary schematic diagram of a wearable device 400 using eye gesture tracking. The wearable device 400 using eye gesture tracking may include a stereoscopic vision wearable device that provides light path adjustment based on eye gesture tracking. The wearable device 400 includes a first transparent or opaque screen 410A and a second transparent or opaque screen 410B that a user can see through or view, a first set of tunable optical elements 430A positioned to adjust the light path at the first transparent or opaque screen 410A, and a second set of tunable optical elements 430B for adjusting the light path at the second transparent or opaque screen 410B. The wearable device 400 may also include a first wireless communication unit 440A for communicating with a remote device or a second wireless communication unit 440B for communicating with a remote device, a first image projector 450A for projecting 2D vision through the first transparent or opaque screen 410A or at the first transparent or opaque screen 410A, a second image projector 450B for projecting 2D vision through the second transparent or opaque screen 410B or at the second transparent or opaque screen 410B, a first eye gesture tracking module 460A for tracking the eye gesture of the first user's eye 420A and determining a depth map corresponding to the first user's eye 420A, and a second eye gesture tracking module 460B for tracking the eye gesture of the second user's eye 420B and determining a depth map corresponding to the second user's eye 420B.

[0109] The wearable device 400 may also include one continuous or two separate transparent or opaque screens 410A and 410B that enable determination of two different gaze points 425A and 425B. When each of the user's eyes 420A and 420B is tracked by each corresponding eye gesture tracking module 460A and 460B, respectively, the first optical element 430A and the second optical element 430B may be tuned independently of each other. In addition, each of the image projectors 450A and 450B may be operated independently. Thus, a portion may be selected on each of the transparent or opaque screens 410A and 410B to refocus the light incident on each of the user's eyes 420A and 420B. In this case, the 3D projection may be interpreted by the user's eyes 420A and 420B via simultaneous projection of multiple images through the transparent or opaque screens 410A and 410B or at the transparent or opaque screens 410A and 410B.

[0110] Figure 5Ais an exemplary schematic diagram of a stand-alone eye gesture tracking device attached to a machine. The stand-alone eye gesture tracking device is implemented as a stand-alone peripheral device 530 located near the machine, such as a display device 520. The stand-alone eye gesture tracking device attached to the machine includes a display device 520 communicating with the stand-alone peripheral device 530 located at a remote location away from the user's eyes 510A and 510B.

[0111] The independent peripheral device 530 includes a mechanical module 532 to control the direction of light emission and detection from the eye gesture tracking module 534 so that the user's eyes are always located by the independent peripheral device 530. The eye gesture tracking module 534 tracks the eye gestures of the user's eyes 510A and 510B and determines gaze information corresponding to the user's eyes 510A and 510B. The display device 520 may include a gaze reference point 515 corresponding to the focus of the user's eyes 510A and 510B relative to the display device 520. The gaze reference point 515 can be determined by the eye gesture tracking module 534 of the independent peripheral device 530. In some specific implementations, the display device 520 may include a tunable optical element that is tuned based on the gaze reference point 515. The tunable optical element may include a tunable reflector located at the display device 520. In other specific implementations, the display device 520 may include a fixed optical element, such as a fixed reflector for refocusing the optical path.

[0112] The eye gesture tracking module 530 may be configured to provide output data to the display device 520. The output data may include gaze information of the user's eyes 510A and 510B. The display device 520 may use the gaze information to present an image at a specific portion of the display corresponding to the user's gaze reference point 515. The presented image may be displayed on the display of the display device 520 by an array of light emitting diodes that produce visible light, a liquid crystal that filters white light, or any other array of light sources located at the display of the display device 520. In addition, the presented image may be displayed on the display of the display device 520 by optical refraction, diffraction, reflection, guidance, or any other optical technology.

[0113] Figure 5B 5 is an exemplary schematic diagram of an eye gesture tracking device embedded in a machine. The eye gesture tracking device embedded in the machine includes an embedded peripheral device 545 integrated into the machine such as a display device 540. The embedded peripheral device 545 may include a mechanical module 546 to control the direction of light emission and detection from an eye gesture tracking module 547 so that the user's eyes are always located by the embedded peripheral device 545. The eye gesture tracking module 547 tracks the eye gestures of the user's eyes 510A and 510B and determines gaze information corresponding to the user's eyes 510A and 510B.

[0114] The display device 540 may also include a gaze reference point 555, which indicates the location of the display device 540 where the user's eyes 510A and 510B are focused. In some implementations, the display device 540 may include a tunable optical element that is tuned based on the gaze reference point 555. The tunable optical element may include a tunable mirror located at the display device 540. In other implementations, the display device 540 may include a fixed optical element such as a fixed mirror.

[0115] In some implementations, the distance between the eye 510A or 510B and the eye tracking modules 534 and 547 can be determined based on the TOF concept or by other methods such as imaging processing or triangulation. The light emission power can be adjusted accordingly based on the distance between the eye 510A or 510B and the eye tracking modules 534 and 547. For example, considering the closer distance between the eye 510A or 510B and the eye tracking modules 534 and 547, the light emission power can be dynamically reduced to reduce the exposure of the eye 510A or 510B to the light emission.

[0116] Figure 6 6 is a flow chart illustrating a process 600 for eye gesture tracking according to certain exemplary implementations. The process 600 for eye gesture tracking describes a process for monitoring the movement of an eye based on a generated eye depth map. In step 610, an electrical signal representing a measurement result of a light signal reflected from the eye is obtained. The light signal may be provided by a light source. The light source may be biased by a modulated voltage signal synchronized with a predetermined reference signal. Thus, the light source may provide a light signal in the direction of the eye to be reflected back from the eye.

[0117] The reflected light signal may be received by one or more photodetectors. In some implementations, the received light signal may be filtered to remove certain wavelengths. For example, one or more filters may be provided to filter the light signal so that only the target wavelength remains in the filtered light signal. In some implementations, one or more lenses may be provided to focus the light signal before the light signal is received by the photodetector. The lens may be a transparent lens, a fixed lens, a tunable lens, a photonic grating-based lens, etc.

[0118] In step 620, a depth map is determined based on a phase difference between the received light signal and a reference signal. The received light signal may be compared to the reference signal as it is received. In other implementations, the received light signal may be filtered and then a comparison may be made between the filtered light signal and the reference signal. The depth map may include one or more data sets of 3D information corresponding to the eye. In some implementations, a 3D representation of the eye may be generated based on the 3D information of the depth map. The depth map may be continuously determined in real time. The depth map may also be determined and updated at predetermined points in time. For example, the depth map may be determined and updated every microsecond, every millisecond, every second, etc.

[0119] In step 630, gaze information is determined based on the depth map. The gaze information may represent the gaze of the eye based on the depth map. In some specific implementations, the gaze information may be determined based on a comparison between a provided reference signal and a reflected light signal. In addition, the gaze information may include identifying one or more of the following items of the eye: a specific area, a pupil, an iris, or a physiological structure. In some specific implementations, the posture of the eye may be determined from the gaze information. The eye posture information may include one or more of the following items of the eye: movement, rotation, a stable state and its duration, a closed eye state and its duration, an open eye state and its duration, a blinking state and its duration, or its frequency.

[0120] The depth map may be used to generate an iris vector perpendicular to a plane tangential to the eye. In this case, gaze information may be determined based on the iris vector and the depth map. The depth map may also be used to generate an eye pupil position on a plane tangential to the eye. In this case, gaze information may be determined based on the eye pupil position and the depth map.

[0121] In step 640, gaze information of the eyes is provided as output data. Output data representing gaze information can be transmitted to equipment, machine, system, etc. In this case, gaze information can be transmitted to equipment, machine or system as input data. In addition, eye gestures determined from gaze information can be provided as output data. Eye gestures can be used to provide commands to equipment, machine or system or to interact with equipment, machine or system. For example, if tracking eyes blinking three times in rapid succession, this may indicate that commands will be provided to remote devices such as television sets. Therefore, television sets can be configured to turn off when tracking eyes blinking several times in rapid succession.

[0122] Figure 77 is a flow chart illustrating a process 700 for tuning an optical element based on eye gesture tracking according to certain exemplary implementations. The process 700 for tuning an optical element based on eye gesture tracking describes a process of monitoring movement of an eye and tuning an optical element based on the movement of the eye. In step 710, an electrical signal representing a measurement of a light signal reflected from the eye is obtained. The light signal may be provided by a light source. The light source may be biased by a modulated voltage signal synchronized with a predetermined reference signal. Thus, the light source may provide a light signal in the direction of the eye to be reflected back from the eye.

[0123] The reflected light signal may be received by one or more photodetectors. In some implementations, the received light signal may be filtered to remove certain wavelengths. For example, one or more filters may be provided to filter the light signal so that only the target wavelength remains in the filtered light signal. In some implementations, one or more lenses may be provided to focus the light signal before the light signal is received by the photodetector. The lens may be a transparent lens, a fixed lens, a tunable lens, a photonic grating-based lens, etc.

[0124] In step 720, a depth map is determined based on a phase difference between the received light signal and a reference signal. The received light signal may be compared to the reference signal as it is received. In other implementations, the received light signal may be filtered and then a comparison may be made between the filtered light signal and the reference signal. The depth map may include one or more data sets of 3D information corresponding to the eye. In some implementations, a 3D representation of the eye may be generated based on the 3D information of the depth map. The depth map may be continuously determined in real time. The depth map may also be determined and updated at predetermined points in time. For example, the depth map may be determined and updated every microsecond, every millisecond, every second, etc.

[0125] In step 730, gaze information is determined based on the depth map. The gaze information may represent the gaze of the eye based on the depth map. The gaze information may include identifying one or more of the following items of the eye: a specific area, a pupil, an iris, or a physiological structure. In some specific implementations, the posture of the eye may be determined from the gaze information. The eye posture information may include one or more of the following items of the eye: movement, rotation, a stable state and its duration, a closed eye state and its duration, an open eye state and its duration, a blinking state and its duration, or its frequency.

[0126] The depth map can also be used to generate an iris vector perpendicular to a plane tangential to the eye. In this case, gaze information can be determined based on the iris vector and the depth map. The depth map can also be used to generate the eye pupil position on a plane tangential to the eye. In this case, gaze information can be determined based on the eye pupil position and the depth map.

[0127] In step 740, tuning of a tunable optical element is activated based on the determined gaze information. The gaze information can be used to tune a specific tunable optical element. For example, the gaze information can include the focus of an eye relative to a specific display. The focus of the eye can be directed through a tunable optical element such as a tunable lens or a reflector. Based on the gaze information, the tunable lens or reflector can be activated or tuned to refocus the light passing through the lens or reflector. In some specific implementations, the tunable optical element can be located at the display. In this case, the tunable optical element can be activated or tuned at the display in real time as the eye is tracked. The tunable optical element can include one or more lenses, reflectors, or any combination thereof.

[0128] Thus, the tunable optical element can be adjusted based on the movement or lack of movement of the tracked eye. The tunable optical element can be used to provide dynamic focus and defocus in real time. For example, the tunable optical element can be used to resolve inconsistencies between accommodation and vergence when viewing images on a VR or AR display.

[0129] Figure 8 Examples of a general purpose computing device 800 and a general purpose mobile computing device 850 that can be used with the techniques described herein are shown. Computing device 800 is intended to represent various forms of digital computers, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 850 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart computers, and other similar computing devices. The components shown here, their connections and relationships, and their functions are meant to be exemplary only and are not meant to limit the specific implementations of the invention described and / or claimed herein.

[0130] The computing device 800 includes a processor 802, a memory 804, a storage device 806, a high-speed interface 808 connected to the memory 804 and a high-speed expansion port 810, and a low-speed interface 812 connected to a low-speed bus 814 and the storage device 806. Each of the components 802, 804, 806, 808, 810 and 812 is interconnected using various buses and can be installed on a common motherboard or in other ways as appropriate. The processor 802 can process instructions for execution within the computing device 800, including instructions stored in the memory 804 or on the storage device 806, to display graphical information of a GUI on an external input / output device (such as a display 816 connected to the high-speed interface 808). In other specific implementations, multiple processors and / or multiple buses can be used together with multiple memories and multiple types of memories as appropriate. Moreover, multiple computing devices 800 can be connected, each device providing a portion of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system).

[0131] The memory 804 stores information within the computing device 800. In one implementation, the memory 804 is one or more volatile memory units. In another implementation, the memory 804 is one or more non-volatile memory units. The memory 804 may also be another form of computer-readable medium, such as a magnetic disk or an optical disk.

[0132] The storage device 806 can provide mass storage for the computing device 800. In one specific implementation, the storage device 806 can be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device or a tape device, a flash memory or other similar solid-state storage device, or a device array including devices in a storage area network or other configurations. A computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that, when executed, perform one or more methods (such as those described above). The information carrier is a computer-readable medium or a machine-readable medium, such as a memory 804, a storage device 806, or a memory on a processor 802.

[0133] The high-speed controller 808 manages bandwidth-intensive operations of the computing device 800, while the low-speed controller 812 manages less bandwidth-intensive operations. This allocation of functions is exemplary only. In one implementation, the high-speed controller 808 is coupled to the memory 804, the display 816 (e.g., through a graphics processor or accelerator), and to a high-speed expansion port 810 that can accept various expansion cards (not shown). In this implementation, the low-speed controller 812 is coupled to the storage device 806 and the low-speed expansion port 814. The low-speed expansion port, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device, such as a switch or a router, for example, coupled through a network adapter.

[0134] As shown, computing device 800 can be implemented in a variety of different forms. For example, it can be implemented as a standard server 820, or multiple implementations in a group of such servers. It can also be implemented as part of a rack server system 824. In addition, it can be implemented in a personal computer such as a laptop computer 822. Alternatively, components from computing device 800 can be combined with other components in a mobile device (not shown), such as computing device 850. Each of these devices can contain one or more of computing devices 800, 850, and the entire system can be composed of multiple computing devices 800, 850 that communicate with each other.

[0135] The computing device 850 includes, among other components, a processor 852, a memory 864, an input / output device such as a display 854, a communication interface and a transceiver. The computing device 850 may also be provided with a storage device such as a micro drive or other device to provide additional storage. Each of the components 850, 852, 864, 854 is interconnected using various buses, and several of these components may be mounted on a common motherboard or otherwise as appropriate.

[0136] Processor 852 may execute instructions within computing device 850, including instructions stored in memory 864. The processor may be implemented as a chipset including separate multiple analog and digital processor chips. The processor may, for example, provide coordination of other components of computing device 850, such as control of a user interface, applications run by computing device 850, and wireless communications performed by computing device 850.

[0137] The processor 852 can communicate with the user through the control interface 858 and the display interface 856 connected to the display 854. The display 854 can be, for example, a TFT LCD (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) display, or other appropriate display technology. The display interface 856 may include appropriate circuits for driving the display 854 to present graphics and other information to the user. The control interface 858 can receive commands from the user and convert them to submit to the processor 852. In addition, an external interface 862 that communicates with the processor 852 can also be provided to enable close range communication between the computing device 850 and other devices. The external interface 862 can, for example, provide wired communication in some specific implementations, or provide wireless communication in other specific implementations, and multiple interfaces can also be used.

[0138] The memory 864 stores information within the computing device 850. The memory 864 may be implemented as one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. An expansion memory 854 may also be provided and connected to the computing device 850 via an expansion interface 852, which may include, for example, a SIMM (single in-line memory module) card interface. Such an expansion memory 854 may provide additional storage space for the computing device 850, or may also store applications or other information for the computing device 850. Specifically, the expansion memory 854 may include instructions for executing or supplementing the above-mentioned processes, and may also include security information. Therefore, for example, the expansion memory 854 may be provided as a security module for the computing device 850, and may be programmed with instructions that allow the computing device 850 to be used securely. In addition, a security application may be provided through a SIMM card together with other information, such as placing identification information on the SIMM card in an unbreakable manner.

[0139] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one specific implementation, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods (such as those described above). The information carrier is a computer-readable medium or machine-readable medium, such as memory 864, expansion memory 854, memory on processor 852, or a propagated signal that can be received, for example, via a transceiver or external interface 862.

[0140] The computing device 850 can communicate wirelessly through a communication interface, which may include a digital signal processing circuit when necessary. The communication interface can also provide communication under various modes or protocols, such as GSM voice calls, SMS, EMS or MMS message transceivers, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS, among others. This communication can occur, for example, through a radio frequency transceiver. In addition, short-range communication can also occur, such as using Bluetooth, WiFi or other such transceivers (not shown). In addition, a GPS (global positioning system) receiver module can also provide additional navigation and location-related wireless data to the computing device 850, which can be used as appropriate by the application program running on the computing device 850.

[0141] The computing device 850 may also communicate audibly using an audio codec 860, which may receive verbal information from a user and convert it into usable digital information. The audio codec 860 may also generate audible sounds for the user, such as through a speaker, such as in a cell phone of the computing device 850. Such sounds may include sounds from voice phone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications operating on the computing device 850.

[0142] As shown, computing device 850 can be implemented in many different forms. For example, it can be implemented as a cellular phone 880. It can also be implemented as a part of a smart phone 882, a personal digital assistant or other similar mobile device.

[0143] Many applications can be realized based on the concepts described in this article. For example, Figure 1B and Figure 1CThe TOF pixels shown can also be used to detect facial features of the user, with the option of including eye gesture tracking for facial recognition or emotion detection. For another example, eye gesture tracking based on the specific implementation described herein can be used to replace or supplement the mouse to locate the user's interest or focus on the display. In some specific implementations, eye gesture tracking based on the specific implementation described herein can be used for more accurate advertising positioning or predicting user behavior. For example, machine learning using artificial neural networks can be used to learn and identify user behavior based on where the user is watching on the display. Different content can be given different weights based on different user behaviors, such as (1) not watching (2) watching but not selecting (3) watching and selecting, where the weight can be from small to large. In addition, the duration of a user's attention to a particular content can also be used to record the user's interest level, for example, the longer the duration, the higher the weight. In some specific implementations, displaying an advertisement on a website can result in different fees being charged to the payer of the advertisement based on the interest level received from the user rather than the traditional click or non-click behavior.

[0144] In some implementations, eye gesture tracking based on the implementations described herein can also be used for gaming. For example, see Figure 2A , a racing or aircraft flying game can be played by a user on a mobile phone 224 or a tablet computer 230. Changes in the user's eye gaze over time can be used to control the movement (e.g., direction) of the car, indicating where the user wants to go. As another example, data collected by the accelerometer 211, the gyroscope 212, and the eye tracking module can be used to track both the user's head movement and the eye gaze. In some specific implementations, a separate button can be included to control the speed, and another optional button can also be included to control additional actions, such as shooting or switching devices. Alone or in combination, the head movement information and the eye gaze information can be used by a game running on a mobile phone or tablet computer to determine the user's actions, and then the game can respond accordingly. In this example, a combination of a vector representing the rotational position of the user's head and a vector representing the gaze of the user's eyes can be used to determine the angle of the gaze relative to the head, which can be used by the game to interpret the user's actions or psychological state.

[0145] Several specific implementations have been introduced. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. For example, various forms of the processes shown above may be used, wherein certain steps may be rearranged, added, or deleted.

[0146] The embodiments of the present invention and all functional operations described in this specification can be implemented in digital electronic circuits or in computer software, firmware or hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more thereof. The embodiments of the present invention can be implemented as one or more computer program products, such as one or more modules of computer program instructions encoded on a computer-readable medium, for execution by a data processing device or control of the operation of the data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of a substance that affects a machine-readable propagation signal, or a combination of one or more thereof. The term "data processing device" covers all devices, equipment and machines for processing data, including, by way of example, a programmable processor, a computer or multiple processors or computers. In addition to hardware, the device may also include code that creates an execution environment for the computer program involved, such as code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more thereof. A propagation signal is an artificially generated signal, such as a machine-generated electrical signal, an optical signal, or an electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device.

[0147] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communications network.

[0148] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. These processes and logic flows can also be performed by special-purpose logic circuits such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits), and the apparatus can also be implemented as special-purpose logic circuits.

[0149] For example, processors suitable for executing computer programs include both general-purpose microprocessors and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include or be operably connected to receive data from one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data or to send data to a mass storage device or both. However, a computer does not necessarily require such devices. In addition, a computer can be embedded in another device, such as a tablet computer, a mobile phone, a personal digital assistant (PDA), a mobile audio player, a global positioning system (GPS) receiver, to name a few. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, by way of example, semiconductor storage devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0150] To provide interaction with a user, embodiments of the present invention may be implemented on a computer having a display device, a keyboard, and a pointing device, the display device being used to display information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and the user may provide input to the computer through a keyboard and a pointing device such as a mouse or trackball. Other types of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input.

[0151] Embodiments of the present invention may be implemented in a computing system including a back-end component (e.g., as a data server) or including a middleware component (e.g., an application server) or including a front-end component (e.g., a client computer) having a graphical user interface or web browser through which a user can interact with a specific implementation of the present invention, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), such as the Internet.

[0152] The computing system may include clients and servers. Clients and servers are generally remote from each other and generally interact through a communication network. The relationship between a client and a server is due to computer programs running on the respective computers, and they have a client-server relationship to each other.

[0153] Although this specification contains many details, these details should not be regarded as limitations on the scope of the present invention or the content that may be claimed, but should be regarded as descriptions of the features specifically for the specific embodiments of the present invention. Certain features described in this specification in the context of a separate embodiment may also be implemented in combination in a single embodiment. On the contrary, the individual features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Moreover, although the features may be described above as working in certain combinations, even initially claimed as such, in some cases, one or more features in the claimed combination may be deleted from this combination, and the claimed combination may be a variation for a certain sub-combination or sub-combination.

[0154] Similarly, although the drawings show the operations in a specific order, this should not be understood as requiring that such operations must be performed in the specific order shown or in a sequential order, or that all of the operations shown must be performed to obtain the desired results. In some cases, multitasking and parallel processing can be advantageous. Moreover, the separation of the various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, but rather that the program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0155] In each case where an HTML file is mentioned, other file types or formats may be substituted. For example, an HTML file may be replaced by XML, JSON, plain text, or other types of files. In addition, where a table or hash table is mentioned, other data structures (e.g., a spreadsheet, a relational database, or a structured file) may be used.

[0156] Specific embodiments of the invention have been described. Other embodiments are within the scope of the following claims. For example, the steps recited in the claims can be performed in a different order and still achieve the desired results.

Claims

1. An eye tracking system, characterized in that: The eye tracking system comprises: A machine comprising a display comprising one or more tunable optical elements; An eye movement tracking device, comprising: A single optical emitter or multiple optical emitters for emitting one or more specific wavelengths to illuminate a user's eyes; One or more light detectors for receiving light signals reflected from the user's eyes; as well as A circuit for: Acquire an electrical signal corresponding to the light signal reflected from the user's eyes measured by the one or more light detectors; Determining a depth map of the user's eyes according to a phase difference between a reference signal and the electrical signal generated by the light detector; as well as Determining gaze information representing eyes of the user according to the depth map; a signal processing unit, which is in communication with the eye tracking device and is disposed separately from the eye tracking device, the signal processing unit being configured to: receiving an output data representing the gaze information from the eye tracking device; and determining the gaze information of the user's eyes relative to the display of the machine, Wherein the one or more adjustable optical elements refocus the light based on the determined eye gaze information.

2. The eye tracking system of claim 1, wherein: The method further comprises one or more filters for removing a non-target wavelength signal from the optical signal.

3. The eye tracking system of claim 1, wherein: One or more lenses are also included to focus the optical signal onto the optical detector.

4. The eye tracking system of claim 1, wherein: The one or more adjustable optical elements include an adjustable lens or an adjustable mirror.

5. The eye tracking system of claim 1, wherein: Also included is a wearable device coupled to the machine and the eye tracking device to form an integrated hardware package, the display of the machine is opaque, and a visual image is displayed on the display through one or more light source arrays.

6. The eye tracking system of claim 1, wherein: Also included is a wearable device coupled to the machine and the eye tracking device to form an integrated hardware package, wherein the display of the machine is at least partially transparent to the image projected toward the display, and the image characteristics projected toward the display are adjusted by the one or more adjustable optical elements on the display.

7. The eye tracking system according to claim 5 or 6, wherein: The signal processing unit communicates with the integrated hardware package via a wireless or wired connection.

8. The eye tracking system of claim 1, wherein: The signal processing unit is located at the display and communicates with the separate eye tracking device.

Images