Low Power Eye Tracking System

Abstract

The present disclosure relates to low-power eye tracking systems. The present application discloses a low-power eye tracking for detecting the position and movement of a user's eyes in a head-mounted device (HMD). The eye tracking system for the HMD may include a head rangefinder, an eye rangefinder, or both a head rangefinder and an eye rangefinder to reduce the power consumption of the system. The head rangefinder can be used to detect the movement of the device relative to the user's eyes, which can allow 3D reconstruction to be performed only when the movement of the device relative to the user's eyes has been detected to generate a 3D model of the user's eyes for eye tracking. The eye rangefinder can be used as a low-power component to track the relative movement of the user's eyes relative to the device between frames captured by the eye tracking camera, which can allow the frame rate of the eye tracking camera to be reduced.

Description

[0001] Priority information

[0002] This patent application claims the benefit of priority to U.S. Provisional Patent Application Serial No. 62 / 902,329, filed on September 18, 2019, and entitled “LOW-POWEREYE TRACKING SYSTEM,” the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates to a low power eye tracking system. Background Art

[0004] Virtual reality (VR) allows users to experience and / or interact with an immersive artificial environment so that users feel as if they are in the environment. For example, a virtual reality system may display a stereoscopic scene to a user to create an illusion of depth, and a computer may adjust the scene content in real time to provide the illusion that the user is moving within the scene. When a user views an image through a virtual reality system, the user may therefore feel as if they are moving within the scene from a first-person perspective. Similarly, mixed reality (MR) combines computer-generated information (referred to as virtual content) with real-world images or real-world views to enhance or add content to the user's world view. Therefore, the simulated environment of VR and / or the mixed environment of MR can be used to provide an interactive user experience for multiple applications, such as an application that adds virtual content to a real-time view of the viewer's environment, an application that interacts with a virtual training environment, a game application, an application that remotely controls a drone or other mechanical system, an application that views digital media content, an application that interacts with the Internet, and the like.

[0005] An eye tracker is a device used to estimate eye position and eye movement. Eye tracking systems have been used in the study of visual systems, psychology, psycholinguistics, marketing, and as input devices for human-computer interaction. In the latter application, the intersection of a person's gaze point and a desktop monitor is usually considered. Summary of the invention

[0006] Various embodiments of methods and apparatus for low-power eye tracking in virtual and mixed or augmented reality (VR / AR) applications are described.

[0007] Embodiments of methods and apparatus for tracking relative movement of a device relative to a user's head are described, wherein sensors (referred to herein as head motion sensors or head rangefinders) are placed at one or more locations in or on the device. In some embodiments of an eye tracking system, in order to accurately determine the position of a user's eyes relative to an eye tracking camera, a controller may execute an algorithm that uses images captured by the eye tracking camera to perform a three-dimensional (3D) reconstruction to generate a 3D model of the user's eyes. Signals from the head rangefinder may be used to detect movement of the device relative to the user's eyes. This may allow 3D reconstruction to be performed only when movement of the device relative to the user's eyes is detected, thereby significantly reducing the power consumption of the eye tracking system. In some embodiments, instead of performing 3D reconstruction when movement of the device relative to the user's eyes has been detected, the magnitude and direction of the detected movement may be determined, and the 3D model of the user's eyes previously generated by the 3D reconstruction method may be adjusted according to the magnitude and direction of the detected movement of the HMD.

[0008] Embodiments of methods and apparatus for tracking relative movement of a user's eyes relative to an HMD using an eye rangefinder are also described. In some embodiments, a sensor (referred to herein as an eye movement sensor or eye rangefinder) is placed at one or more locations in the device to augment the eye tracking camera. The eye rangefinder can be used as a low power component to track relative movement of a user's eyes relative to the device in the intervals between processing of frames captured by the eye tracking camera at a frame rate.

[0009] Embodiments of methods and apparatus for tracking relative movement of a user's eyes relative to an HMD using low-resolution images are also described. In some embodiments, the eye-tracking camera itself may capture low-resolution frames (e.g., by binning pixels on the camera sensor or by capturing horizontal and vertical strips or lines of pixels on the camera sensor rather than entire frames). This low-resolution information may be used to track relative movement of the user's eyes relative to the device in the intervals between processing of full high-resolution frames captured by the eye-tracking camera.

[0010] These eye tracking methods and apparatus may allow the frame rate of the eye tracking camera to be reduced, for example from 120 frames / second to 10 frames / second or less, and may also allow 3D reconstruction to be performed less frequently, thereby significantly reducing the power consumption of the eye tracking system. The eye tracking methods may be used alone or in combination in various embodiments.

[0011] The present invention describes embodiments of an HMD that includes both a head rangefinder and an eye rangefinder, or alternatively includes both a head rangefinder and an eye tracking camera that captures low-resolution images to track the relative movement of the eyes. Embodiments of an eye tracking system for an HMD that includes both a head rangefinder and an eye rangefinder, or alternatively includes both a head rangefinder and an eye tracking camera that captures low-resolution images to track the relative movement of the eyes, can, for example, further reduce the frequency at which 3D reconstruction is performed, and can also reduce the frequency at which two-dimensional (2D) image processing of frames captured by the eye tracking camera is performed to further reduce power consumption of the eye tracking system. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 An exemplary VR / AR HMD implementing an eye tracking system according to some embodiments is shown.

[0013] Figure 2 An exemplary VR / AR HMD implementing an eye tracking system according to some embodiments is shown, which includes sensors for detecting movement of the HMD relative to the user's eyes.

[0014] Figure 3 is a flow chart of an eye tracking method according to some embodiments, wherein sensors are used to detect movement of an HMD relative to the user's eyes, and wherein 3D reconstruction is performed only when movement of the HMD is detected.

[0015] Figure 4 is a flow chart of an eye tracking method according to some embodiments, wherein sensors are used to detect movement of an HMD relative to a user's eyes, and wherein a 3D model of the eyes is adjusted when movement of the HMD is detected.

[0016] Figure 5 An exemplary VR / AR HMD implementing an eye tracking system according to some embodiments is shown, the eye tracking system comprising sensors for tracking movement of the eyes in intervals between processing of frames captured by an eye tracking camera.

[0017] Figure 6 is a flow chart of an eye tracking method according to some embodiments, in which a sensor is used to track the movement of the eye in intervals between processing of frames captured by an eye tracking camera.

[0018] Figure 7 An exemplary VR / AR HMD implementing an eye tracking system according to some embodiments is shown, wherein low-resolution frames are captured by an eye tracking camera and used to track eye movements in intervals between processing of high-resolution frames captured by the eye tracking camera.

[0019] Fig. 8A and Figure 8B An exemplary low-resolution frame captured by an eye-tracking camera is shown in accordance with some embodiments.

[0020] Fig. 9 is a flowchart of an eye tracking method according to some embodiments, wherein low-resolution frames are captured by an eye tracking camera and are used to track eye movements in intervals between processing of high-resolution frames captured by the eye tracking camera.

[0021] Fig.10 An exemplary VR / AR HMD implementing an eye tracking system according to some embodiments is shown, the eye tracking system comprising a head rangefinder that detects movement of the HMD relative to the user's eyes, and an eye rangefinder that tracks movement of the eyes in intervals between processing of frames captured by an eye tracking camera.

[0022] Fig.11 is a flowchart of an eye tracking method according to some embodiments, wherein a head rangefinder is used to detect movement of the HMD relative to the user's eyes, and an eye rangefinder is used to track the movement of the eyes in intervals between processing of frames captured by an eye tracking camera.

[0023] Fig.12 is a block diagram illustrating an exemplary VR / AR system according to some embodiments, the exemplary VR / AR system including: Figures 2 to 11 Components of the eye tracking system shown.

[0024] Fig.13 An alternative VR / AR device is shown according to some embodiments that includes an eye tracking system having a head rangefinder that detects movement of the device relative to the user's eyes, and an eye rangefinder that tracks movement of the eyes in intervals between processing of frames captured by an eye tracking camera.

[0025] This specification includes references to "one embodiment" or "an embodiment." The appearance of the phrase "in one embodiment" or "in an embodiment" does not necessarily refer to the same embodiment. The particular features, structures or characteristics may be combined in any suitable manner consistent with the present disclosure.

[0026] The term "comprising" is open ended. As used in the claims, the term does not exclude additional structures or steps. Consider the following cited claim: "an apparatus comprising one or more processor units..." Such a claim does not exclude the apparatus from including additional components (e.g., a network interface unit, a graphics circuit, etc.).

[0027] "Configured to", various units, circuits or other components may be described or described as "configured to" perform one or more tasks. In such contexts, "configured to" is used to imply a structure (e.g., a circuit) that performs the one or more tasks during operation by indicating that the unit / circuit / component includes the structure. In this way, the unit / circuit / component is said to be configured to perform the task even when the specified unit / circuit / component is currently inoperable (e.g., not turned on). The units / circuits / components used with the "configured to" language include hardware-such as circuits, memories storing executable program instructions to implement operations, etc. The reference to a unit / circuit / component "configured to" perform one or more tasks is explicitly intended not to invoke the sixth paragraph of 35 U.S.C. §112 for the unit / circuit / component. In addition, "configured to" may include a general structure (e.g., a general circuit) manipulated by software or firmware (e.g., an FPGA or a general processor executing software) to operate in a manner capable of performing one or more tasks to be solved. "Configured to" may also include adapting a manufacturing process (eg, a semiconductor fabrication facility) to produce a device (eg, an integrated circuit) suitable for implementing or performing one or more tasks.

[0028] "First," "second," etc. As used herein, these terms act as labels for the nouns that precede them, and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.). For example, a buffer circuit may be described herein as performing a write operation of a "first" value and a "second" value. The terms "first" and "second" do not necessarily imply that the first value must be written before the second value.

[0029] "Based on" or "depending on", as used herein, these terms are used to describe one or more factors that influence a determination. These terms do not exclude additional factors that may influence the determination. That is, the determination may be based solely on these factors or at least in part on these factors. Consider the phrase "A is determined based on B." In this case, B is a factor that influences the determination of A, and such phrases do not exclude that the determination of A may also be based on C. In other examples, A may be determined based solely on B.

[0030] "Or", when used in the claims, the term "or" is used as an inclusive or rather than an exclusive or. For example, the phrase "at least one of x, y or z" means any one of x, y and z and any combination thereof. DETAILED DESCRIPTION

[0031] Various embodiments of methods and apparatus for eye tracking in virtual and mixed or augmented reality (VR / AR) applications are described. A VR / AR system may include a device such as a headset, a helmet, goggles, or glasses (referred to herein as a head mounted device (HMD)), which includes a display (e.g., a left display and a right display) for displaying a frame including a left image and a right image in front of a user's eyes to thereby provide a three-dimensional (3D) virtual view to the user. The VR / MR system may also include a controller. The controller may be implemented in the HMD, or alternatively may be implemented at least in part by an external device (e.g., a computing system) communicatively coupled to the HMD via a wired or wireless interface. The controller may include one or more of various types of processors, image signal processors (ISPs), graphics processing units (GPUs), encoders / decoders (encoders), and / or other components for processing and rendering video and / or images. The controller may render frames (each frame including a left image and a right image) including virtual content based at least in part on inputs obtained from cameras and other sensors on the HMD, and may provide the frames to a projection system of the HMD for display.

[0032] The VR / AR system may include an eye tracking system (which may also be referred to as a gaze tracking system). An embodiment of an eye tracking system for a VR / AR system is described, which includes at least one eye tracking camera (e.g., an infrared (IR) camera) positioned at each side of a user's face and configured to capture an image of the user's eyes. The eye tracking system may also include a light source (e.g., an IR light source) that emits light (e.g., IR light) toward the user's eyes. A portion of the IR light is reflected from the user's eyes to the eye tracking camera. For example, an eye tracking camera located at or near the edge of an HMD display panel captures an image of the user's eyes from the IR light reflected from the eyes. The images captured by the eye tracking system may be analyzed by a controller to detect features (e.g., through-holes), position, and movement of the user's eyes, and / or detect other information about the eyes, such as pupil dilation. For example, a gaze point on a display estimated from an eye tracking image may enable gaze-based interaction with content shown on a near-eye display of the HMD. Other applications may include, but are not limited to, creating eye image animations for avatars in VR / AR environments.

[0033] An HMD may be a mobile device with an internal power supply. Therefore, reducing power consumption to increase power supply life is an issue. Eye tracking systems (both camera and processing components) consume power. Embodiments of methods and apparatus for providing a low-power eye tracking system are described that can reduce the amount of power consumed by the eye tracking hardware and software components of an HMD.

[0034] The key to providing accurate eye tracking is knowing the position of the user's eyes relative to the eye tracking camera. In some embodiments of the eye tracking system, in order to accurately determine the position of the user's eyes relative to the eye tracking camera, the controller may execute an algorithm that uses images captured by the eye tracking camera to perform a three-dimensional (3D) reconstruction to generate a 3D model of the user's eyes. The 3D model of the eye indicates the 3D position of the eye relative to the eye tracking camera, which allows the eye tracking algorithm executed by the controller to accurately track eye movements.

[0035] A key element of accurate eye tracking is the robustness of the device with respect to movement of the device on the user's head. During use, the HMD may move on the user's head. In addition, the user may remove the device and put it back. In either case, the initial calibration of the eye tracking system performed using 3D reconstruction may be invalidated. However, in an eye tracking system, there is an inherent ambiguity between whether the user's eyes move relative to the camera / HMD or whether the camera / HMD moves relative to the user's eyes. The solution to this ambiguity is to perform 3D reconstruction for each frame captured by the eye tracking camera. However, 3D reconstruction is computationally expensive and consumes a significant amount of power.

[0036] Embodiments of methods and apparatus for tracking relative movement of a device relative to a user's head are described, wherein sensors (referred to herein as head motion sensors or head rangefinders) are placed at one or more locations in or on the device, such as at or near the user's ears to primarily track pitch and at or near the bridge of the nose to primarily track y movement. Signals from the head rangefinders can be used to detect movement of the device relative to the user's eyes. This can allow 3D reconstruction to be performed only when movement of the device relative to the user's eyes is detected, thereby significantly reducing the power consumption of the eye tracking system. When movement of the device is not detected, 2D image processing of frames captured by the eye tracking camera (which is computationally much cheaper than 3D image processing) can be performed to track the user's eyes. In some embodiments, 3D reconstruction can be performed periodically (e.g., once per second, or once every N frames) even if movement of the device is not detected to prevent error / drift accumulation.

[0037] In some embodiments, instead of performing a 3D reconstruction when movement of the device relative to the user's eyes has been detected, the magnitude and direction of the detected motion may be determined, and the 3D model of the user's eyes previously generated by the 3D reconstruction method may be adjusted according to the magnitude and direction of the detected motion of the HMD. In these embodiments, the 3D reconstruction may be performed periodically (e.g., once per second, or once every N frames) to prevent error / drift accumulation.

[0038] In addition, embodiments of methods and apparatus for reducing power consumption of an eye tracking system are described, wherein a sensor (e.g., a photoelectric sensor or photodiode, referred to herein as an eye movement sensor or eye rangefinder) is placed at one or more locations in the device to enhance the eye tracking camera. The eye rangefinder can be used as a low-power component to track the relative movement of the user's eyes relative to the device in the intervals between the processing of frames captured by the eye tracking camera. A 3D model generated from images captured by the eye tracking camera provides absolute gaze information, while data captured by the eye rangefinder in the intervals between the processing of camera frames is processed to provide relative updates to a previously known and trusted 3D model. This can allow the frame rate of the eye tracking camera to be reduced, for example from 120 frames / second to 10 frames / second or less, and can also allow 3D reconstruction to be performed less frequently, thereby significantly reducing the power consumption of the eye tracking system. Reducing the frame rate of the eye tracking camera by enhancing eye tracking with an eye rangefinder can also significantly reduce bandwidth usage and latency of the eye tracking system.

[0039] In addition, embodiments of methods and apparatus for reducing the power consumption of an eye tracking system are described, wherein an eye tracking camera may capture a low-resolution frame, such as by merging pixels on a camera sensor or by capturing horizontal and vertical strips or lines of pixels on a camera sensor instead of an entire frame (a high-resolution frame). This low-resolution information may be used to track the relative movement of the user's eyes relative to the device in the intervals between the processing of the full high-resolution frame captured by the eye tracking camera. This may allow 3D reconstruction to be performed less frequently, thereby significantly reducing the power consumption of the eye tracking system. In these embodiments, the eye tracking camera itself may be considered an "eye rangefinder" when capturing and processing low-resolution frames. A high-resolution frame captured by a camera sensor contains more pixels than a low-resolution frame captured by the same camera sensor. A high-resolution frame may include most or all of the pixels that a camera sensor is capable of capturing, while a low-resolution frame may include significantly fewer pixels than the same camera sensor is capable of capturing. In some embodiments, a camera sensor may be switched from a high-resolution mode that captures a high-resolution frame to a low-resolution mode that captures a low-resolution frame, such as by merging pixels on a camera sensor or by capturing horizontal and vertical strips or lines on a camera sensor, for example, by a controller including one or more processors.

[0040] The head rangefinder method and the eye rangefinder method described above can be used alone or in combination. Embodiments of an HMD including a head rangefinder to reduce the power consumption of an eye tracking system are described. Embodiments of an HMD including an eye rangefinder (implemented by a sensor or by an eye tracking camera) to reduce the power consumption and bandwidth usage of an eye tracking system are also described. In addition, embodiments of an HMD including both a head rangefinder and an eye rangefinder are described. Embodiments of an eye tracking system of an HMD including both a head rangefinder and an eye rangefinder can, for example, further reduce the frequency at which 3D reconstruction is performed, and can also reduce the frequency at which 2D image processing of frames captured by an eye tracking camera is performed to further reduce the power consumption of the eye tracking system.

[0041] Although embodiments of eye tracking systems are generally described herein as including at least one eye tracking camera positioned at each side of a user's face to track the gaze of both of the user's eyes, eye tracking systems may also be implemented that include at least one eye tracking camera positioned at only one side of a user's face to track the gaze of only one of the user's eyes.

[0042] Figure 1 1 shows a side view of an exemplary VR / AR HMD 100 implementing an eye tracking system according to some embodiments. Figure 1 The illustrated HMD 100 is given by way of example and is not intended to be limiting. In various embodiments, the shape, size, and other features of the HMD 100 may vary, and the location, number, type, and other features of the components of the HMD 100 may vary. The VR / AR HMD 100 may include, but is not limited to, a display 110 and two optical lenses (eyepieces) 120 mounted in a wearable housing or frame. Figure 1 As shown, HMD 100 may be positioned on the head of a user 190 such that display 110 and eyepiece 120 are disposed in front of the user's eyes 192. The user looks toward display 110 through eyepiece 120.

[0043] The controller 160 for the VR / AR system may be implemented in the HMD 100, or alternatively may be implemented at least in part by an external device (e.g., a computing system) communicatively coupled to the HMD 100 via a wired or wireless interface. The controller 160 may include one or more of various types of processors, image signal processors (ISPs), graphics processing units (GPUs), encoders / decoders (codecs), and / or other components for processing and rendering video and / or images. The controller 160 may render frames (each including a left image and a right image) including virtual content based at least in part on input obtained from a sensor, and may provide the frames to a projection system of the HMD 100 for display to the display 110. Fig.12 Also shown are components of an HMD and VR / AR system according to some embodiments.

[0044] The eye tracking system may include, but is not limited to, one or more eye tracking cameras 140 and IR light sources 130. The IR light sources 130 (e.g., IR LEDs) may be positioned in the HMD 100 (e.g., around the eyepiece 120 or elsewhere in the HMD 100) to illuminate the user's eyes 192 with IR light. At least one eye tracking camera 140 (e.g., an IR camera, such as a 400x400 pixel digital camera or a 600x600 pixel digital camera that operates at 850nm or 940nm or at some other IR wavelength and captures frames at a rate of 60-120 frames per second (FPS)) is located on each side of the face of the user 190. In various embodiments, the eye tracking cameras 140 may be positioned in the HMD 100 on each side of the face of the user 190 to provide a direct view of the eye 192, a view of the eye 192 through the eyepiece 120, or a view of the eye 192 via reflection from a hot mirror or other reflective component. Note that the position and angle of eye tracking camera 140 are given by way of example and are not intended to be limiting. Figure 1 A single eye tracking camera 140 is shown located on each side of the user's 190 face, but in some embodiments, there may be two or more eye tracking cameras 140 on each side of the user's 190 face.

[0045] A portion of the IR light emitted by the one or more light sources 130 reflects from the eyes of the user 190 and is captured by the eye tracking camera 140 to image the user's eye 192. The images captured by the eye tracking camera 140 may be analyzed by the controller 160 to detect features (e.g., through-holes), positions, and movements of the user's eye 192, and / or to detect other information about the eye 192, such as pupil dilation. For example, the gaze point on the display 110 may be estimated from the eye tracking images to enable gaze-based interaction with the content shown on the display 110. As another example, in some embodiments, the information collected by the eye tracking system may be used to adjust the rendering of images to be projected, and / or adjust the projection of images by the projection system of the HMD 100 based on the direction and angle at which the eyes of the user 190 are looking.

[0046] With Figure 1 The illustrated embodiment of the HMD 100 of the eye tracking system may be used, for example, for augmented or mixed (AR) applications to provide an augmented or mixed reality view to the user 190. Although not shown, in some embodiments, the HMD 100 may include one or more sensors, for example, located on an external surface of the HMD 100, that collect information about the external environment of the user 190 (video, depth information, lighting information, etc.); the sensors may provide the collected information to the controller 160 of the VR / AR system. In some embodiments, the sensors may include one or more visible light cameras (e.g., RGB cameras) that capture video of the user's environment, which may be used to provide the user 190 with a virtual view of his or her real environment. In some embodiments, the video stream of the real environment captured by the visible light camera may be processed by the controller of the HMD 100 to render an augmented or mixed reality frame including virtual content overlaid on a view of the real environment, and the rendered frame may be provided to the projection system of the HMD 100 for display on the display 110. In some embodiments, display 110 emits light in the visible range and does not emit light in the IR range, and therefore does not introduce noise into the eye tracking system. Figure 1 Embodiments of the HMD 100 of the eye tracking system shown may also be used in virtual reality (VR) applications to provide a VR view to the user 190. In these embodiments, a controller of the HMD 100 may render or obtain a virtual reality (VR) frame including virtual content, and the rendered frame may be provided to a projection system of the HMD 100 for display on the display 110.

[0047] The key to providing accurate eye tracking is knowing the position of the user's eye 192 relative to the eye tracking camera 140. In some embodiments of the eye tracking system, in order to accurately determine the position of the user's eye relative to the eye tracking camera, the controller 160 can use the images captured by the eye tracking camera 140 to perform 3D reconstruction to generate a 3D model of the user's eye 192. The 3D model of the eye 192 indicates the 3D position of the eye 192 relative to the eye tracking camera 140, which allows the eye tracking algorithm executed by the controller 160 to accurately track eye movements. However, a key element of accurate eye tracking is the robustness of the device with respect to the movement of the user's head. The movement or removal of the HMD 190 may invalidate the initial calibration performed using 3D reconstruction. However, in an eye tracking system, there is an inherent ambiguity between whether the user's eye 192 moves relative to the camera 140 or whether the camera 140 moves relative to the user's eye 192. The conventional solution to this ambiguity is to perform 3D reconstruction for each frame captured by the eye tracking camera 140. However, 3D reconstruction is computationally expensive and consumes a significant amount of power.

[0048] Figure 2 2 shows an exemplary VR / AR HMD 200 implementing an eye tracking system that includes sensors (referred to as head motion sensors or head rangefinders) for detecting movement of the HMD 200 relative to the user's eyes 292 in accordance with some embodiments. Figure 2 The HMD 200 shown is given by way of example and is not intended to be limiting. In various embodiments, the shape, size, and other features of the HMD 200 may be different, and the location, number, type, and other features of the components of the HMD 200 may vary. The VR / AR HMD 200 may include, but is not limited to, a display 210 and two eyepieces 220 mounted in a wearable housing or frame. A controller 260 for a VR / AR system may be implemented in the HMD 200, or alternatively may be implemented at least in part by an external device communicatively coupled to the HMD 200 via a wired or wireless interface. The controller 260 may include one or more of various types of processors, image signal processors (ISPs), graphics processing units (GPUs), encoders / decoders (codecs), and / or other components for processing and rendering video and / or images. Fig.12 Also shown are components of an HMD and VR / AR system according to some embodiments.

[0049] The HMD 200 may include an eye tracking system including, but not limited to, one or more eye tracking cameras 240 and an IR light source 230. The IR light source 230 (e.g., an IR LED) may be positioned in the HMD 200 (e.g., around the eyepiece 220 or elsewhere in the HMD 200) to illuminate the user's eyes 292 with IR light. At least one eye tracking camera 240 (e.g., an IR camera, such as a 400×400 pixel digital camera or a 600×600 pixel digital camera that operates at 850 nm or 940 nm or at some other IR wavelength and captures frames at a rate of 60-120 frames per second (FPS)) is located at each side of the face of the user 290. In various embodiments, the eye tracking cameras 240 may be positioned in the HMD 200 on each side of the face of the user 290 to provide a direct view of the eye 292, a view of the eye 292 through the eyepiece 220, or a view of the eye 292 via reflection from a hot mirror or other reflective component. Note that the position and angle of eye tracking camera 240 are given by way of example and are not intended to be limiting. Figure 2 A single eye-tracking camera 240 is shown located on each side of the face of the user 290 , but in some embodiments, there may be two or more eye-tracking cameras 240 on each side of the face of the user 290 .

[0050] In order to track the relative movement of the HMD 200 relative to the user's eyes, the eye tracking system may also include sensors 242 (referred to herein as head motion sensors or head rangefinders) placed at one or more locations in or on the device, such as at or near the user's ears (242A) to primarily track pitch and at or near the bridge of the nose (242B) to primarily track y movement. Signals from the head rangefinder 242A and the head rangefinder 242B may be used to detect movement of the HMD 200 relative to the user's eyes. This may allow 3D reconstruction to be performed only when movement of the HMD 200 relative to the user's eyes is detected, thereby significantly reducing the power consumption of the eye tracking system. When movement of the HMD 200 is not detected, 2D image processing (which is computationally much cheaper than 3D image processing) of frames captured by the eye tracking camera 240 may be performed to track the user's eyes 292. In some implementations, 3D reconstruction may be performed periodically (eg, once per second, or once every N frames) to prevent error / drift accumulation, even if movement of the HMD 200 is not detected.

[0051] In some embodiments, instead of performing a 3D reconstruction when movement of the HMD 200 relative to the user's eye 292 has been detected, the magnitude and direction of the detected motion may be determined, and the 3D model of the user's eye 292 previously generated by the 3D reconstruction method may be adjusted according to the magnitude and direction of the detected motion of the HMD 200. In these embodiments, the 3D reconstruction may be performed periodically (e.g., once per second, or once every N frames) to prevent error / drift accumulation.

[0052] Examples of different technologies that may be used to implement the head rangefinder 242 in an eye tracking system are given below in the section entitled Exemplary Sensors.

[0053] Figure 3 is a flow chart of an eye tracking method according to some embodiments, wherein a sensor is used to detect movement of an HMD relative to a user's eyes, and wherein 3D reconstruction is performed only when movement of the HMD is detected. For example, Figure 3 The method can be found in Figure 2 The VR / AR system shown is executed.

[0054] As indicated at 310, a 3D reconstruction method may be performed to generate a 3D model of the user's eyes using frames captured by the eye tracking camera. For example, the 3D reconstruction may be performed at initialization / calibration of the HMD. The 3D model of the eye indicates the 3D position of the eye relative to the eye tracking camera, which allows the eye tracking algorithm executed by the controller to accurately track the movement of the eye relative to the HMD.

[0055] 2D image processing of frames captured by the eye tracking camera may be performed to track movement of the user's eyes relative to the HMD, as indicated at 320. Eye movements tracked by 2D image processing may be used, for example, to determine a gaze point on a display of the HMD.

[0056] As indicated at 330, sensors on the HMD may be used to track relative movement of the HMD relative to the user's head. A key element to accurate eye tracking is the robustness of the device with respect to movement of the device on the user's head. If the HMD moves on the user's head during use, the initial calibration of the eye tracking system performed using 3D reconstruction at element 310 may be invalidated. However, in an eye tracking system, there is an inherent ambiguity between whether the user's eyes move relative to the camera / HMD or whether the camera / HMD moves relative to the user's eyes. In order to track the movement of the HMD relative to the user's eyes, sensors (referred to as head motion sensors or head rangefinders) are placed at one or more locations in or on the HMD, such as Figure 2 Signals from the head rangefinder may be processed by the controller to track the movement of the HMD relative to the user's eyes.

[0057] At 340, as long as the controller does not detect movement of the HMD relative to the user's eyes from the head rangefinder signals, the eye tracking method may continue to perform 2D image processing to track movement of the user's eyes relative to the HMD, as indicated at 320. At 340, if the controller detects movement of the HMD relative to the user's eyes, the method returns to element 310 to again perform the 3D reconstruction method to generate a new 3D model of the user's eyes using frames captured by the eye tracking camera, which allows the eye tracking algorithm executed by the controller to continue to accurately track movement of the eyes relative to the HMD.

[0058] Figure 3 The method may allow 3D reconstruction to be performed only when movement of the HMD relative to the user's eyes is detected, thereby significantly reducing the power consumption of the eye tracking system. When movement of the HMD is not detected, 2D image processing (which is computationally much cheaper than 3D image processing) of frames captured by the eye tracking camera may be performed to track the user's eyes. In some embodiments, 3D reconstruction may be performed periodically (e.g., once per second, or once every N frames) to prevent error / drift accumulation even if movement of the HMD relative to the user's eyes is not detected.

[0059] Figure 4 is a flow chart of an eye tracking method according to some embodiments, wherein a sensor is used to detect movement of an HMD relative to a user's eyes, and wherein a 3D model of the eyes is adjusted when movement of the HMD is detected. For example, Figure 4 The method can be found in Figure 2 The VR / AR system shown is executed.

[0060] 2D image processing of frames captured by the eye tracking camera may be performed to track movement of the user's eyes relative to the HMD, as indicated at 420. Eye movements tracked by 2D image processing may be used, for example, to determine a gaze point on a display of the HMD.

[0061] As indicated at 430, sensors on the HMD may be used to track the relative movement of the HMD relative to the user's head. To track the movement of the HMD relative to the user's eyes, sensors (referred to as head motion sensors or head rangefinders) are placed at one or more locations in or on the HMD, such as Figure 2 Signals from the head rangefinder may be processed by the controller to track the movement of the HMD relative to the user's eyes.

[0062] At 440, as long as the controller does not detect movement of the HMD relative to the user's eyes from the head rangefinder signals, the eye tracking method may continue to perform 2D image processing to track movement of the user's eyes relative to the HMD, as indicated at 420. At 440, if the controller detects movement of the HMD relative to the user's eyes, the controller may adjust the 3D model generated at 410, as indicated at 450, and the method may continue to perform 2D image processing to track movement of the user's eyes relative to the HMD, as indicated at 420. The magnitude and direction of the movement may be determined from the head rangefinder signals. The controller may then use this information to adjust the 3D model, which allows the eye tracking algorithm executed by the controller to continue to use 2D image processing to accurately track movement of the eyes relative to the HMD without performing expensive 3D reconstruction. However, in these embodiments, 3D reconstruction may be performed periodically (e.g., once per second, or once every N frames) to recalibrate the eye tracking system.

[0063] Figure 4 The method allows Figure 3 3D reconstruction can be performed less frequently than in the previous method, thereby further reducing the power consumption of the eye tracking system. When no movement of the HMD is detected, 2D image processing (which is computationally much cheaper than 3D image processing) of the frames captured by the eye tracking camera can be performed to track the user's eyes. When movement is detected, the 3D model can be adjusted without the need for expensive 3D reconstruction. It may only be necessary to perform 3D reconstruction occasionally to prevent error / drift accumulation.

[0064] Figure 5 An exemplary VR / AR HMD implementing an eye tracking system including sensors for tracking eye movements in the intervals between processing of frames captured by an eye tracking camera is shown in accordance with some embodiments. Figure 5 The HMD 500 shown is given by way of example and is not intended to be limiting. In various embodiments, the shape, size, and other features of the HMD 500 may be different, and the location, number, type, and other features of the components of the HMD 500 may vary. The VR / AR HMD 500 may include, but is not limited to, a display 510 and two eyepieces 520 mounted in a wearable housing or frame. A controller 560 for a VR / AR system may be implemented in the HMD 500, or alternatively may be implemented at least in part by an external device communicatively coupled to the HMD 500 via a wired or wireless interface. The controller 560 may include one or more of various types of processors, image signal processors (ISPs), graphics processing units (GPUs), encoders / decoders (codecs), and / or other components for processing and rendering video and / or images. Fig.12Also shown are components of an HMD and VR / AR system according to some embodiments.

[0065] The HMD 500 may include an eye tracking system including, but not limited to, one or more eye tracking cameras 540 and an IR light source 530. The IR light source 530 (e.g., an IR LED) may be positioned in the HMD 500 (e.g., around the eyepiece 520 or elsewhere in the HMD 500) to illuminate the user's eyes 592 with IR light. At least one eye tracking camera 540 (e.g., an IR camera, such as a 400×400 pixel digital camera or a 600×600 pixel digital camera that operates at 850nm or 940nm or at some other IR wavelength and may be capable of capturing frames at a rate of 60-120 frames per second (FPS)) is located at each side of the face of the user 590. In various embodiments, the eye tracking camera 540 may be positioned in the HMD 500 on each side of the face of the user 590 to provide a direct view of the eye 592, a view of the eye 592 through the eyepiece 520, or a view of the eye 592 via reflection from a hot mirror or other reflective component. Note that the position and angle of eye tracking camera 540 are given by way of example and are not intended to be limiting. Figure 5 A single eye tracking camera 540 is shown located on each side of the user's 590 face, but in some embodiments, there may be two or more eye tracking cameras 540 on each side of the user's 590 face.

[0066] To track the relative movement of the user's eyes 592 relative to the HMD 500, the eye tracking system may also include a sensor 544 (e.g., a photosensor or photodiode, referred to herein as an eye movement sensor or eye rangefinder) placed at one or more locations in the HMD 500 to augment the eye tracking camera 540. The eye rangefinder 544 may be used as a low-power component to track the relative movement of the user's eyes 592 relative to the HMD 500 in the intervals between the processing of frames captured by the camera 540 to generate a 3D model of the user's eyes. The 3D model generated from the images captured by the eye tracking camera 540 provides absolute gaze information at the frame rate of the camera, while the data captured by the eye rangefinder 544 is processed in the intervals between the processing of the camera frames to provide relative updates to the 3D model generated from the captured images. This may allow the frame rate of eye tracking camera 540 to be reduced, e.g., from 120 frames / second to 10 frames / second or less, and may also allow 3D reconstruction based on captured frames to be performed less frequently (e.g., 10 times per second or less), thereby significantly reducing power consumption of the eye tracking system. Reducing the frame rate of eye tracking camera 540 by enhancing eye tracking with eye rangefinder 544 may also significantly reduce bandwidth usage and latency of the eye tracking system. Examples of different techniques that may be used to implement eye rangefinder 544 in an eye tracking system are given below in the section entitled Exemplary Sensors.

[0067] Figure 6 is a flow chart of an eye tracking method according to some embodiments, in which a sensor is used to track eye movements in the intervals between processing of frames captured by an eye tracking camera. For example, Figure 6 The method can be found in Figure 5 As indicated at 610, one or more frames may be captured by an eye tracking camera. As indicated at 620, a 3D reconstruction method may be performed to generate a 3D model of the user's eyes using the frames captured by the eye tracking camera. For example, the 3D reconstruction may be performed at initialization / calibration of the HMD. The 3D model of the eye indicates the 3D position of the eye relative to the eye tracking camera, which allows the eye tracking algorithm executed by the controller to accurately track the movement of the eye relative to the HMD.

[0068] As indicated at 630, after performing the initial 3D reconstruction to generate the 3D model of the eye, the relative movement of the user's eyes relative to the HMD may be tracked using data captured by and collected from an eye rangefinder (e.g., a photosensor or photodiode). At 640, the eye tracking system may continue to track the user's eyes using data from the eye rangefinder until movement of the HMD relative to the user's head is detected, a time limit has passed, or a confidence threshold for the eye rangefinder data is exceeded. If movement of the HMD relative to the user's head is detected, the time limit has passed, or the confidence threshold has been exceeded, the method returns to element 610 to capture one or more frames using the eye tracking camera, and performs a 3D reconstruction method 620 to generate a new 3D model of the user's eyes using the frames captured by the eye tracking camera. The method then resumes tracking the eyes using the eye rangefinder information at 630.

[0069] Figure 7 An exemplary VR / AR HMD implementing an eye tracking system is shown, wherein low-resolution frames are captured by an eye tracking camera and used to track eye movements in the intervals between processing of high-resolution frames captured by the eye tracking camera, according to some embodiments. Figure 7 The HMD 700 shown is given by way of example and is not intended to be limiting. In various embodiments, the shape, size, and other features of the HMD 700 may be different, and the location, number, type, and other features of the components of the HMD 700 may vary. The VR / AR HMD 700 may include, but is not limited to, a display 710 and two eyepieces 720 mounted in a wearable housing or frame. A controller 760 for a VR / AR system may be implemented in the HMD 700, or alternatively may be implemented at least in part by an external device communicatively coupled to the HMD 700 via a wired or wireless interface. The controller 760 may include various types of processors, image signal processors (ISPs), graphics processing units (GPUs), encoders / decoders (codecs), and / or one or more of other components for processing and rendering video and / or images. Fig.12 Also shown are components of an HMD and VR / AR system according to some embodiments.

[0070] The HMD 700 may include an eye tracking system including, but not limited to, one or more eye tracking cameras 740 and an IR light source 730. The IR light source 730 (e.g., an IR LED) may be positioned in the HMD 700 (e.g., around the eyepiece 720 or elsewhere in the HMD 700) to illuminate the user's eyes 792 with IR light. At least one eye tracking camera 740 (e.g., an IR camera, such as a 400×400 pixel digital camera or a 600×600 pixel digital camera that operates at 850nm or 940nm or at some other IR wavelength and captures frames at a rate of 60-120 frames per second (FPS)) is located at each side of the face of the user 790. In various embodiments, the eye tracking camera 740 may be positioned in the HMD 700 on each side of the face of the user 790 to provide a direct view of the eye 792, a view of the eye 792 through the eyepiece 720, or a view of the eye 792 via reflection from a hot mirror or other reflective component. Note that the position and angle of eye tracking camera 740 are given by way of example and are not intended to be limiting. Figure 7 A single eye-tracking camera 740 is shown located on each side of the face of the user 790 , but in some embodiments, there may be two or more eye-tracking cameras 740 on each side of the face of the user 790 .

[0071] To track the relative movement of the user's eye 792 relative to the HMD 700, instead of Figure 5 As shown in HMD 500, an eye range finder 544 is included, and an eye tracking camera 740 can capture low-resolution frames (e.g., by Fig. 8A , or by combining pixels on the camera sensor as shown Figure 8B792). In these embodiments, the camera 740 may be operated in a high-resolution mode or a low-resolution mode. The controller 760 may signal the camera 740 to switch to the low-resolution mode during or after processing one or more high-resolution frames to generate a 3D model of the eye 792, and may signal the camera 740 to switch to the high-resolution mode, for example, when movement of the HMD 700 relative to the user's head 790 is detected. The low-resolution images may be used to track the relative movement of the user's eye 792 relative to the HMD 700 in the intervals between the controller 760 processing the full high-resolution frames captured by the eye tracking camera. This may allow the high-resolution frames to be captured less frequently, and also allow 3D reconstruction to be performed less frequently, thereby significantly reducing the power consumption of the camera 740 and the controller 760 in the eye tracking system. Reducing the number of high-resolution frames captured and processed may also significantly reduce the bandwidth usage and latency of the eye tracking system. Therefore, when capturing and processing low-resolution frames in the intervals between the capture and processing of high-resolution frames, the eye tracking camera 740 itself may be considered an "eye rangefinder".

[0072] Fig. 8A and Figure 8B shows that according to some embodiments, Figure 7 Non-limiting example low resolution frames captured by eye tracking camera 740. Fig. 8A An example of a frame at 640x640 resolution (400k pixels) in high-resolution mode is shown. In low-resolution mode, the camera 740 combines 32x32 pixel blocks into one pixel, resulting in a 20x20 resolution (400 pixels) that can be processed in 2D to track the movement of the user's eyes in the interval between capturing and processing the full high-resolution frame in 3D. Figure 8B An example of capturing horizontal and vertical lines of pixels in low-resolution mode rather than a full frame is shown. The captured lines can be processed in 2D to track the movement of the user's eyes in the interval between capturing and processing the full high-resolution frame in 3D.

[0073] Fig. 9 is a flow chart of an eye tracking method according to some embodiments, wherein low-resolution frames are captured by an eye tracking camera and used to track eye movement in intervals between processing of high-resolution frames captured by the eye tracking camera. For example, Fig. 9 The method can be found in Figure 7As indicated at 910, one or more high-resolution frames may be captured by an eye-tracking camera. As indicated at 920, a 3D reconstruction method may be performed to generate a 3D model of the user's eyes using the high-resolution frames captured by the eye-tracking camera. For example, the 3D reconstruction may be performed at initialization / calibration of the HMD. The 3D model of the eye indicates the 3D position of the eye relative to the eye-tracking camera, which allows the eye-tracking algorithm executed by the controller to accurately track the movement of the eye relative to the HMD.

[0074] As indicated at 930, after performing an initial 3D reconstruction to generate a 3D model of the eye, the eye tracking camera may switch to a low resolution mode to capture low resolution frames, such as by Fig. 8A , or by combining pixels on the camera sensor as shown Figure 8B Horizontal and vertical lines of pixels on the capture camera sensor are shown. In some embodiments, the controller may signal the eye tracking camera to switch to low-resolution mode during or after processing the captured high-resolution frames to generate a 3D model of the eye. As indicated at 940, the low-resolution images captured in the low-resolution mode may be used to track the movement of the user's eyes relative to the HMD. At 950, the eye tracking system may continue to capture low-resolution images and track the user's eyes relative to the HMD using the low-resolution information until the movement of the HMD relative to the user's head is detected, the time limit is passed, or the confidence threshold is exceeded. At 950, if the movement of the HMD relative to the user's head is detected, the time limit has passed, or the confidence threshold has been exceeded, the method returns to element 910 to capture one or more high-resolution frames using the eye tracking camera, and perform a 3D reconstruction method 920 to generate a new 3D model of the user's eyes using the frames captured by the eye tracking camera. In some embodiments, when the movement of the HMD relative to the user's head is detected, the time limit is determined to have passed, or the confidence threshold is determined to have been exceeded, the controller may signal the eye tracking camera to switch to high-resolution mode. The eye tracking system then resumes capturing low resolution images at 930 and uses the low resolution images to track the eyes at 940 .

[0075] References Figures 2 to 9 The head rangefinder method and the eye rangefinder method can be used in combination. Fig.10 and Fig.11 Embodiments of an HMD that includes both a head rangefinder and an eye rangefinder are described.

[0076] Fig.10An exemplary VR / AR HMD implementing an eye tracking system according to some embodiments is shown, the eye tracking system comprising a head rangefinder that detects movement of the HMD relative to the user's eyes, and an eye rangefinder that tracks the movement of the eyes in intervals between processing of frames captured by an eye tracking camera. Note that Fig.10 The HMD 1000 shown is given by way of example and is not intended to be limiting. In various embodiments, the shape, size, and other features of the HMD 1000 may be different, and the location, number, type, and other features of the components of the HMD 1000 may vary. The VR / AR HMD 1000 may include, but is not limited to, a display 1010 and two eyepieces 1020 mounted in a wearable housing or frame. A controller 1060 for a VR / AR system may be implemented in the HMD 1000, or alternatively may be implemented at least in part by an external device communicatively coupled to the HMD 1000 via a wired or wireless interface. The controller 1060 may include one or more of various types of processors, image signal processors (ISPs), graphics processing units (GPUs), encoders / decoders (codecs), and / or other components for processing and rendering video and / or images. Fig.12 Also shown are components of an HMD and VR / AR system according to some embodiments.

[0077] The HMD 1000 may include an eye tracking system including, but not limited to, one or more eye tracking cameras 1040 and an IR light source 1030. The IR light source 1030 (e.g., an IR LED) may be positioned in the HMD 1000 (e.g., around the eyepiece 1020 or elsewhere in the HMD 1000) to illuminate the user's eyes 1092 with IR light. At least one eye tracking camera 1040 (e.g., an IR camera, such as a 400×400 pixel digital camera or a 600×600 pixel digital camera that operates at 850 nm or 940 nm or at some other IR wavelength and captures frames at a rate of 60-120 frames per second (FPS)) is located at each side of the user's 1090 face. In various embodiments, eye tracking cameras 1040 may be positioned in HMD 1000 on each side of the face of user 1090 to provide a direct view of eye 1092, a view of eye 1092 through eyepiece 1020, or a view of eye 1092 via reflection from a thermal mirror or other reflective component. Note that the positions and angles of eye tracking cameras 1040 are given by way of example and are not intended to be limiting. Fig.10 A single eye tracking camera 1040 is shown located on each side of the user's 1090 face, but in some embodiments, there may be two or more eye tracking cameras 1040 on each side of the user's 1090 face.

[0078] To track the relative movement of the HMD 1000 relative to the user's eyes, the eye tracking system may also include a head rangefinder 1042 placed at one or more locations in or on the device, such as at or near the user's ears (1042A) to primarily track pitch and at or near the bridge of the nose (1042B) to primarily track y movement. The controller 1060 may process signals from the head rangefinder 1042A and the head rangefinder 1042B to detect movement of the HMD 1000 relative to the user's eyes. This may allow 3D reconstruction to be performed by the controller 1060 only when movement of the HMD 1000 relative to the user's eyes is detected, thereby significantly reducing the power consumption of the eye tracking system. In some embodiments, instead of performing 3D reconstruction when movement of the HMD 1000 relative to the user's eye 1092 has been detected, the controller 1060 may determine the magnitude and direction of the detected movement, and the controller 1060 may adjust the 3D model of the user's eye 1092 previously generated by the 3D reconstruction method based on the magnitude and direction of the detected movement of the HMD 1000.

[0079] In order to track the relative movement of the user's eyes 1092 relative to the HMD 1000, the eye tracking system may also include an eye rangefinder 1044 placed at one or more locations in the HMD 1000 to enhance the eye tracking camera 1040. The eye rangefinder 1044 may be used as a low power component to track the relative movement of the user's eyes 1092 relative to the HMD 500 in the intervals between the processing of frames captured by the camera 1040. The 3D model generated by the controller 1060 based on the images captured by the eye tracking camera 1040 provides absolute gaze information, while the data captured by the eye rangefinder 1044 in the intervals between the processing of camera frames is processed by the controller 1060 to provide relative updates to the previously known and trusted 3D model. This may allow the frame rate of the eye tracking camera 1040 to be reduced, for example from 120 frames / second to 10 frames / second or less, and may also allow the controller 1060 to perform 3D reconstruction less frequently, thereby significantly reducing the power consumption of the eye tracking system. Enhanced eye tracking with an eye rangefinder 1044 can also significantly reduce bandwidth usage and latency of an eye tracking system. Examples of different techniques that can be used to implement an eye rangefinder 1044 in an eye tracking system are given below in the section entitled Exemplary Sensors.

[0080] As an alternative, in some embodiments, instead of using the eye rangefinder 1044 in the HMD 1000 to track eye movements in the intervals between frames, the eye tracking camera 1040 may capture low-resolution frames in the intervals between capturing high-resolution frames, for example, by binning pixels on the camera sensor, or by capturing horizontal and vertical strips or lines of pixels on the camera sensor rather than entire frames. This low-resolution information may be processed by the controller 1060 to track the relative movement of the user's eye 1092 relative to the HMD 1000 in the intervals between the full high-resolution frames captured by the eye tracking camera 1040 and processed by the controller 1060.

[0081] Fig.11 is a flow chart of an eye tracking method according to some embodiments, in which a head rangefinder is used to detect movement of the HMD relative to the user's eyes, and an eye rangefinder is used to track the movement of the eyes in intervals between processing of frames captured by an eye tracking camera. For example, Fig.11 The method can be found in Fig.10 As indicated at 1110, one or more frames may be captured by an eye tracking camera. As indicated at 1120, a 3D reconstruction method may be performed to generate a 3D model of the user's eyes using the frames captured by the eye tracking camera. For example, the 3D reconstruction may be performed at initialization / calibration of the HMD. The 3D model of the eye indicates the 3D position of the eye relative to the eye tracking camera, which allows the eye tracking algorithm executed by the controller to accurately track the movement of the eye relative to the HMD.

[0082] After performing an initial 3D reconstruction to generate a 3D model of the eye, the movement of the user's eyes relative to the HMD may be tracked in 2D space using data captured by and collected from an eye rangefinder on the HMD, as indicated at 1130. As indicated at 1140, the relative movement of the HMD relative to the user's head may be tracked using a head rangefinder on the HMD.

[0083] At 1150, if the controller detects movement of the HMD relative to the user's eyes based on data received from the head rangefinder, the method returns to element 1110 to perform the 3D reconstruction method again, thereby generating a new 3D model of the user's eyes using frames captured by the eye tracking camera. Alternatively, the magnitude and direction of the movement can be determined based on the head rangefinder signal. The controller can then use this information to adjust the 3D model, which allows the eye tracking algorithm executed by the controller to continue to accurately track the movement of the eyes relative to the HMD in 2D space without the need to perform a 3D reconstruction.

[0084] At 1150, if no movement of the HMD relative to the user's head is detected, and if the time limit has not passed at 1160, the eye tracking method may continue to track the movement of the user's eyes relative to the HMD using data from the eye rangefinder at 1130, and continue to track the movement of the HMD relative to the HMD using data from the head rangefinder at 1140. At 1160, if the time limit has passed, and if the confidence threshold for the eye tracking data has not been exceeded at 1170, the eye tracking method may continue to track the movement of the user's eyes relative to the HMD using data from the eye rangefinder at 1130, and continue to track the movement of the HMD relative to the HMD using data from the head rangefinder at 1140. At 1170, if the confidence threshold has been exceeded, the method returns to element 1110 to perform the 3D reconstruction method again, generating a new 3D model of the user's eyes using frames captured by the eye tracking camera.

[0085] like Fig.10 and Fig.11 Embodiments of the eye tracking system of the HMD that include both a head rangefinder and an eye rangefinder may, for example, further reduce the frequency with which 3D reconstruction is performed, and may also reduce the frequency with which 2D image processing of frames captured by an eye tracking camera is performed to further reduce power consumption of the eye tracking system.

[0086] Exemplary Sensors

[0087] The following provides methods that can be used to implement Figures 2 to 13 Non-limiting examples of different technologies for head motion sensors and eye motion sensors in the eye tracking system described. These technologies can be divided into three major categories: drift compensation (1D) technology, ranging (2D) technology and absolute tracking (5D) technology.

[0088] Drift compensation techniques may work by contact and may include ranging sensors in the HMD, such as ranging sensors in the nose pads of the HMD that make contact on the bridge of the user's nose, and ranging sensors at the temporal region of the HMD that make contact near the user's ears. The ranging sensors may measure force. Instead or in addition, the sensors may be capacitive sensors. As an alternative, an accelerometer with a smart baseline may be used.

[0089] Ranging technologies can be divided into five categories: electrical, optical, acoustic, radar, and pressure. Electrical ranging technologies may include, but are not limited to, EOG / EMG, ECG, and volumetric sensor (waveguide or triangulation) technologies. Optical ranging technologies may include, but are not limited to, photoelectric sensors, photodiode arrays, proximity sensors, and scanners (TOF, intensity, or phase) technologies. Acoustic ranging technologies may include, but are not limited to, sonar (continuous wave, TOF, or resonance) technologies. Radar ranging technologies may include, but are not limited to, continuous wave or TOF radar technologies. Pressure ranging technologies may include, but are not limited to, IMUs (inertial measurement units) or other sensors that sense changes in pressure from eye movement.

[0090] Absolute tracking technologies can be divided into two categories: optical and sonar technologies. Optical absolute tracking technologies may include, but are not limited to, single pixel TOF sensor technologies or various camera technologies. Camera technologies may include, but are not limited to, DVS, CIS (RGB+IR), compressed sensing, skip / merge (e.g., slit cameras), and thermal imaging (short, medium, long, focal thermoelectric or pyroelectric) technologies. Sonar absolute tracking technologies may include, but are not limited to, transceiver arrays, line-of-sight, and non-line-of-sight sonar technologies.

[0091] Fig.12 is a block diagram illustrating an exemplary VR / AR system according to some embodiments, the exemplary VR / AR system including: Figures 2 to 11 Components of the eye tracking system shown. In some embodiments, the VR / AR system may include HMD 2000, such as a headset, a helmet, goggles, or glasses. HMD 2000 may implement any of various types of virtual reality projector technologies. For example, HMD 2000 may include a VR projection system, which includes a projector 2020, and the projector 2020 displays a frame including a left image and a right image on a screen or display 2022A and a display 2022B viewed by a user through an eyepiece 2220A and an eyepiece 2220B. The VR projection system may be, for example, a DLP (digital light processing), LCD (liquid crystal display), or LCoS (liquid crystal on silicon) technology projection system. In order to create a three-dimensional (3D) effect in a 3D virtual view, objects at different depths or distances in the two images may be offset to the left or right as a function of the triangulation of the distance, with closer objects offset more than farther objects. It should be noted that in some embodiments, other types of projection systems may be used.

[0092] In some embodiments, the HMD 2000 may include a controller 2030 configured to implement the functions of the VR / AR system and generate frames (each frame including a left image and a right image) displayed by the projector 2020. In some embodiments, the HMD 2000 may also include a memory 2032 configured to store software (code 2034) of the VR / AR system executable by the controller 2030, and data 2038 that can be used by the VR / AR system when executed on the controller 2030. In some embodiments, the HMD 2000 may also include one or more interfaces (e.g., Bluetooth technology interface, USB interface, etc.), and one or more interfaces (e.g., Bluetooth technology interface, USB interface, etc.) are configured to communicate with the external device 2100 via a wired or wireless connection. In some embodiments, at least a portion of the functions described for the controller 2030 can be implemented by the external device 2100. External device 2100 may be or may include any type of computing system or computing device, such as a desktop computer, a notebook or laptop computer, a tablet or tablet device, a smart phone, a handheld computing device, a game controller, a gaming system, and the like.

[0093] In various embodiments, the controller 2030 may be a single processor system including one processor, or a multiprocessor system including several processors (e.g., two, four, eight or another suitable number). The controller 2030 may include a central processing unit (CPU) configured to implement any suitable instruction set architecture, and may be configured to execute instructions defined in the instruction set architecture. For example, in various embodiments, the controller 2030 may include a general purpose processor or embedded processor that implements any instruction set architecture in a variety of instruction set architectures (ISAs) such as x86, PowerPC, SPARC, RISC or MIPS ISAs, or any other suitable ISAs. In a multiprocessor system, each processor may implement the same ISA together, but it is not necessary. The controller 2030 may adopt any microarchitecture, including scalar, superscalar, pipeline, superpipeline, out of order, in order, speculative, non-speculative, etc., or a combination thereof. The controller 2030 may include a circuit that implements microcode technology. The controller 2030 may include one or more processing cores that are each configured to execute instructions. The controller 2030 may include one or more levels of cache memory, which may be of any size and in any configuration (set associative, direct mapping, etc.). In some embodiments, the controller 2030 may include at least one graphics processing unit (GPU), which may include any suitable graphics processing circuit. Typically, the GPU may be configured to render an object to be displayed into a frame buffer (e.g., a frame buffer including pixel data for an entire frame). The GPU may include one or more graphics processors that may execute graphics software to perform some or all of the graphics operations or hardware acceleration of certain graphics operations. In some embodiments, the controller 2030 may include one or more other components for processing and rendering video and / or images, such as an image signal processor (ISP), an encoder / decoder, etc.

[0094] The memory 2032 may include any type of memory, such as dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) SDRAM (including mobile versions of SDRAM, such as mDDR3, etc., or low power versions of SDRAM, such as LPDDR2, etc.), RAMBUS DRAM (RDRAM), static RAM (SRAM), etc. In some embodiments, one or more memory devices may be coupled to a circuit board to form a memory module, such as a single inline memory module (SIMM), a dual inline memory module (DIMM), etc. Alternatively, the device may be mounted in a chip stack configuration, a package stack configuration, or a multi-chip module configuration with an integrated circuit implementing the system.

[0095] In some embodiments, HMD 2000 may include one or more sensors 2050 that collect information about the user's environment (video, depth information, lighting information, etc.). Sensors 2050 may provide information to a controller 2030 of the VR / AR system. In some embodiments, sensors 2050 may include, but are not limited to, visible light cameras (e.g., video cameras).

[0096] like Figure 1 , Figure 2 , Figure 5 , Figure 7 and Fig.10As shown, HMD 2000 can be positioned on the user's head so that displays 2022A and 2022B and eyepieces 2220A and 2220B are disposed in front of the user's eyes 2292A and 2292B. IR light sources 2230A and 2230B (e.g., IR LEDs) can be positioned in HMD 2000 (e.g., around eyepieces 2220A and 2220B or elsewhere in HMD 2000) to illuminate the user's eyes 2292A and 2292B with IR light. Eye tracking cameras 2240A and 2240B (e.g., IR cameras, such as 400x400 pixel digital cameras or 600x600 pixel digital cameras operating at 850nm or 940nm or some other IR wavelength and capturing frames at a rate of 60-120 frames per second (FPS)) are located on each side of the user's face. In various embodiments, eye tracking camera 2240 may be positioned in HMD 2000 to provide a direct view of eye 2292, a view of eye 2292 through eyepiece 2220, or a view of eye 2292 via reflection from a thermal mirror or other reflective component. Note that the positions and angles of eye tracking camera 2240A and eye tracking camera 2240B are given by way of example and are not intended to be limiting. In some embodiments, there may be a single eye tracking camera 2240 located on each side of the user's face. In some embodiments, there may be two or more eye tracking cameras 2240 on each side of the user's face. For example, in some embodiments, a wide angle camera 2240 and a narrower angle camera 2240 may be used on each side of the user's face. A portion of the IR light emitted by the light sources 2230A and 2230B is reflected from the user's eyes 2292A and 2292B, received at the respective eye tracking cameras 2240A and 2240B, and captured by the eye tracking cameras 2240A and 2240B to image the user's eyes 2292A and 2292B. The eye tracking information captured by the cameras 2240A and 2240B may be provided to the controller 2030. The controller 2030 may analyze the eye tracking information (e.g., images of the user's eyes 2292A and 2292B) to determine eye positions and movements and / or other characteristics of the eyes 2292A and 2292B. In some embodiments, in order to accurately determine the position of the user's eyes 2292A and eyes 2292B relative to the eye tracking camera 2240A and eye tracking camera 2240B, the controller 2030 may perform 3D reconstruction using images captured by the eye tracking camera 2240A and eye tracking camera 2240B to generate a 3D model of the user's eyes 2292A and eyes 2292B.The 3D model of eye 2292A and eye 2292B indicates the 3D position of eye 2292A and eye 2292B relative to eye tracking camera 2240A and eye tracking camera 2240, which allows the eye tracking algorithm executed by the controller to accurately track eye movements.

[0097] In some embodiments, the HMD 2000 may include sensors 2242 (referred to herein as head motion sensors or head rangefinders) placed at one or more locations in or on the device, such as at or near the user's ears (2242A) to primarily track pitch and at or near the bridge of the nose (2242B) to primarily track y movement. Signals from the head rangefinder 2242A and the head rangefinder 2242B may be used to detect movement of the HMD 2000 relative to the user's eyes 2292A and eyes 2292B. This may allow 3D reconstruction to be performed only when movement of the HMD 2000 relative to the user's eyes 2292A and eyes 2292B is detected, thereby significantly reducing the power consumption of the eye tracking system. When movement of the HMD 2000 is not detected, 2D image processing (which is computationally much cheaper than 3D image processing) of frames captured by the eye tracking camera 2240A and the eye tracking camera 2240B may be performed to track the user's eyes 2292A and eyes 2292B. In some embodiments, even if movement of the HMD 2000 is not detected, 3D reconstruction may be performed periodically (e.g., once per second, or once every N frames) to prevent error / drift accumulation. In some embodiments, instead of performing 3D reconstruction when movement of the HMD 2000 relative to the user's eyes 2292A and eyes 2292B has been detected, the magnitude and direction of the detected movement may be determined, and the 3D model of the user's eyes 2292A and eyes 2292B previously generated by the 3D reconstruction method may be adjusted according to the magnitude and direction of the detected movement of the HMD 2000. In these embodiments, 3D reconstruction may be performed periodically (e.g., once per second, or once every N frames) to prevent error / drift accumulation.

[0098] In some embodiments, the HMD 2000 may include sensors 2244 (e.g., photosensors or photodiodes, referred to herein as eye movement sensors or eye rangefinders) placed at one or more locations in the device to enhance the eye tracking camera 2240A and the eye tracking camera 2240B. The eye rangefinders 2244A and the eye rangefinders 2244B may be used as low-power components to track the relative movement of the user's eyes 2292A and eyes 2292B relative to the camera 2240 in the intervals between the processing of frames captured by the camera 2240A and the camera 2240B. This may allow the frame rate of the eye tracking camera 2240A and the eye tracking camera 2240B to be reduced, for example, from 120 frames / second to 10 frames / second, and may also allow 3D reconstruction to be performed less frequently, thereby significantly reducing the power consumption of the eye tracking system. Reducing the frame rate of eye tracking camera 2240A and eye tracking camera 2240B by enhancing eye tracking with eye rangefinder 2244A and eye rangefinder 2244B also significantly reduces the bandwidth usage of the eye tracking system.

[0099] In some embodiments, instead of using sensors 2244A and 2244B to enhance eye tracking cameras 2240A and 2240B, eye tracking cameras 2240A and 2240B may capture low resolution frames, such as by merging pixels or by capturing horizontal and vertical strips of pixels instead of entire high resolution frames. The low resolution frames captured by cameras 2240A and 2240B may be used to track the relative movement of the user's eyes 2292A and 2292B relative to cameras 2240A and 2240B in the intervals between the processing of the full high resolution frames captured by eye tracking cameras 2240A and 2240B. This may allow 3D reconstruction to be performed less frequently, thereby significantly reducing the power consumption of the eye tracking system. In these embodiments, when capturing and processing lower resolution frames, eye tracking cameras 2240A and 2240B themselves may be considered "eye range finders".

[0100] The head rangefinder method and the eye rangefinder method described above may be used alone or in combination. An embodiment of HMD 2000 may include a head rangefinder 2242 to reduce the power consumption of the eye tracking system. An embodiment of HMD 2000 may include an eye rangefinder 2244 (implemented by a sensor or by an eye tracking camera) to reduce the power consumption and bandwidth usage of the eye tracking system. In addition, an embodiment of HMD 2000 may include both a head rangefinder 2242 and an eye rangefinder 2244. An embodiment including both a head rangefinder 2242 and an eye rangefinder 2244 may, for example, further reduce the frequency of performing 3D reconstruction, and may also reduce the frequency of performing 2D image processing of frames captured by the eye tracking camera 2240, to further reduce the power consumption of the eye tracking system.

[0101] The eye tracking information obtained and analyzed by the controller 2030 can be used by the controller to implement various VR or AR system functions. For example, the gaze point on the display 2022A and the display 2022B can be estimated from the images captured by the eye tracking camera 2240A and the eye tracking camera 2240B; the estimated gaze point can, for example, enable gaze-based interaction with the content shown on the display 2022A and the display 2022B. Other applications of eye tracking information may include, but are not limited to, creating eye image animations for avatars in VR or AR environments. As another example, in some embodiments, the information obtained from the eye tracking camera 2240A and the eye tracking camera 2240B can be used to adjust the rendering of the image to be projected, and / or adjust the projection of the image performed by the projector 220 of the HMD 2000 based on the direction and angle of the user's eyes. As another example, in some embodiments, the brightness of the projected image can be adjusted based on the pupil dilation of the user as determined by the eye tracking system.

[0102] In some embodiments, the HMD 2000 may be configured to render and display frames to provide an augmented or mixed reality (AR) view for the user based at least in part on the sensor 2050 input. The AR view may include rendering the user's environment, including rendering real objects in the user's environment based on video captured by one or more cameras, which capture high-quality, high-resolution video of the user's environment for display. The AR view may also include virtual content (e.g., virtual objects, virtual labels of real objects, an avatar of the user, etc.) generated by the VR / AR system and synthesized with the projected view of the user's real environment.

[0103] like Fig.12The embodiments of the HMD 2000 shown in FIG. 2000 may also be used in virtual reality (VR) applications to provide VR views to a user. In these embodiments, the controller 2030 of the HMD 2000 may render or obtain a virtual reality (VR) frame including virtual content, and the rendered frame may be provided to the projector 2020 of the HMD 2000 for display to the display 2022A and the display 2022B.

[0104] Although in Figures 2 to 12 Embodiments are generally described with respect to an exemplary VR / AR system and HMD that includes a near-eye display panel that a user views through an eyepiece, but embodiments of the low-power eye system method and apparatus may be implemented in other types of VR / AR systems and more generally in any system that uses an eye tracking camera to perform eye tracking. Fig.13 An exemplary alternative VR / AR device 3000 is shown that includes an eye tracking system having a head rangefinder 3042 that detects movement of the device 3000 relative to a user's eye 3092, and an eye rangefinder 3044 that tracks movement of the eye 3092 in intervals between processing of frames captured by the eye tracking camera 3040, in accordance with some embodiments. Note that Fig.13 The device 3000 shown is given by way of example and is not intended to be limiting. In various embodiments, the shape, size, and other features of the device 3000 may be different, and the location, number, type, and other features of the components of the device 3000 may vary. In the device 3000, the device 3000 does not have a display panel that the user observes through an eyepiece but includes a lens 3010 that allows light from the external environment to pass through and is also configured to reflect light representing virtual content emitted by the projector / controller 3080 toward the user's eye 3092, for example, using a holographic film. The device 3000 may include an eye tracking system that, in addition to the eye tracking camera 3040, includes a head rangefinder 3042 that detects movement of the device 3000 relative to the user's eye 3092 as described herein, an eye rangefinder 3044 that tracks movement of the eye 3092 in the intervals between processing of frames captured by the eye tracking camera 3040 as described herein, or both the head rangefinder 3042 and the eye rangefinder 3044 as described herein.

[0105] In different embodiments, the methods described herein can be implemented in software, hardware or a combination thereof. In addition, the order of the blocks of the method can be changed, and various elements can be added, reordered, combined, omitted, modified, etc. For those skilled in the art, benefiting from the present disclosure, it is obvious that various modifications and changes can be made. The various embodiments described herein are intended to be illustrative and not restrictive. Many variations, modifications, additions and improvements are possible. Therefore, multiple examples can be provided for the components described as a single example in this article. The boundaries between various components, operations and data repositories are arbitrary to a certain extent, and specific operations are shown in the context of a specific exemplary configuration. Other allocations of functions are expected, and they may fall within the scope of the appended claims. Finally, the structure and function presented as discrete components in the exemplary configuration may be implemented as a combined structure or component. These and other variations, modifications, additions and improvements may fall within the scope of the embodiments defined in the following claims.

Claims

1. A system for eye tracking, comprising: A head mounted device HMD, the head mounted device comprising: at least one camera configured to capture images of a user's eye at a frame rate of N frames per second; and one or more head motion sensors configured to detect movement of the HMD relative to the user's head; and A controller comprising one or more processors configured to: performing an initial 3D reconstruction based on at least one image captured by the at least one camera to generate an initial 3D model of the user's eye, wherein the initial 3D model indicates a position of the user's eye relative to the at least one camera; performing 2D image processing of one or more subsequent images captured by the at least one camera to track movement of the user's eyes relative to the HMD until it is detected that the HMD has been displaced on the user's head based on signals from the one or more head motion sensors; and Upon detecting that the HMD has shifted on the user's head based on signals from the one or more head motion sensors, updating the 3D model to indicate a new position of the user's eyes relative to the at least one camera.

2. The system of claim 1 , wherein, in order to update the 3D model to indicate a new position of the user's eyes relative to the at least one camera, the controller is configured to perform the 3D reconstruction based on at least one subsequent image captured by the at least one camera to generate a new 3D model of the user's eyes.

3. The system of claim 1 , wherein to update the 3D model to indicate a new position of the user's eyes relative to the at least one camera, the controller is configured to adjust the initial 3D model based on a magnitude and direction of movement of the HMD relative to the user's head determined from the signals from the one or more head motion sensors.

4. The system of claim 1 , wherein the one or more head motion sensors include at least one sensor located in the HMD at or near a bridge of the user's nose and at least one sensor located in the HMD at or near a temple of the user.

5. The system of claim 1 , wherein the HMD further comprises one or more eye movement sensors configured to detect movement of the user's eyes relative to the HMD, wherein the controller is further configured to process signals from the eye movement sensors to track movement of the user's eyes relative to the HMD in intervals between processing frames captured by the at least one camera.

6. The system of claim 1, wherein the one or more cameras are configured to capture high-resolution frames and low-resolution frames.

7. The system according to claim 6, wherein said 3D reconstruction for generating a 3D model of said user's eyes is performed on high resolution frames captured by said one or more cameras; and Wherein the 2D image processing is performed on low resolution frames captured by the one or more cameras.

8. The system of claim 6, wherein the controller is further configured to: processing the high resolution images captured by the at least one camera to track movement of the user's eyes relative to the HMD; and One or more low-resolution images captured by the at least one camera are processed to track movement of the user's eyes relative to the HMD in intervals between the high-resolution frames captured by the at least one camera and processed by the controller.

9. The system of claim 6, wherein the low resolution frame is captured by merging blocks of pixels on a camera sensor or by capturing horizontal and vertical lines of pixels on the camera sensor.

10. The system of claim 1, wherein the controller is a component of the HMD.

11. The system of claim 1 , wherein the HMD further comprises: at least one display screen, the at least one display screen configured to display a frame containing visual content for viewing by the user, wherein the controller is further configured to render the frame containing the visual content for display by the at least one display screen; as well as One or more light sources configured to emit light toward the user's eyes, wherein the at least one camera captures a portion of the light reflected from the user's eyes.

12. The system of claim 11, wherein the HMD further comprises left and right optical lenses located between the at least one display screen and the user's eyes.

13. A method for eye tracking, comprising: Executed by one or more processors: generating an initial 3D model of the user's eyes based on one or more images of the user's eyes captured by a camera of the head mounted device (HMD); processing subsequent images captured by the camera using 2D image processing to track movement of the user's eyes relative to the HMD until it is detected that the HMD has been displaced on the user's head based on signals from one or more head motion sensors of the HMD; as well as Upon detecting that the HMD has shifted on the user's head based on signals from one or more head motion sensors of the HMD, the 3D model of the user's eyes is updated to indicate a new position of the user's eyes relative to the camera.

14. The method of claim 13, wherein updating the 3D model to indicate a new position of the user's eyes relative to the camera comprises generating a new 3D model of the user's eyes based on at least one subsequent image captured by the camera.

15. The method of claim 13 , wherein updating the 3D model to indicate a new position of the user's eyes relative to the camera comprises adjusting the 3D model based on a magnitude and direction of movement of the HMD relative to the user's head determined from the signals from the one or more head motion sensors.

16. The method of claim 13, further comprising tracking movement of the user's eyes relative to the HMD in intervals between frames captured by the camera based on signals from one or more eye movement sensors of the HMD.

17. The method of claim 13, wherein the camera is configured to capture a high resolution image of the user's eye and a low resolution image of the user's eye, and wherein the one or more processors are further configured to iteratively perform: processing high resolution images captured by the camera to track movement of the user's eyes relative to the HMD; and One or more low-resolution images captured by the camera are processed to track movement of the user's eyes relative to the HMD between high-resolution images captured by the camera, wherein the low-resolution images are captured by merging blocks of pixels on the camera's sensor or by capturing horizontal and vertical lines of pixels on the camera's sensor.

18. A head mounted device comprising: at least one camera configured to capture a high-resolution image and a low-resolution image of an eye of a user; at least one head motion sensor configured to detect movement of the device relative to the user's head; as well as A controller comprising one or more processors configured to: processing one or more high-resolution images captured by the at least one camera to generate an initial 3D model of the user's eyes, wherein the 3D model indicates a position of the user's eyes relative to the device; signaling the at least one camera to switch to a low resolution mode; processing one or more subsequent low-resolution images captured by the at least one camera to track relative movement of the user's eyes with respect to the device until it is detected that the device has been displaced on the user's head based on a signal from the at least one head motion sensor; as well as Upon detecting, based on signals from the one or more head motion sensors, that the device has been displaced on the user's head: signaling the at least one camera to switch to a high resolution mode; as well as One or more subsequent high-resolution images captured by the at least one camera are processed to update the 3D model to indicate a new position of the user's eyes relative to the device.

19. A head mounted device according to claim 18, wherein the low resolution image is captured in a low resolution mode by merging blocks of pixels on a camera sensor or by capturing horizontal and vertical lines of pixels on the camera sensor.

20. The head mounted device of claim 18, wherein the high resolution image is processed using 3D image processing, and wherein the low resolution image is processed using 2D image processing.

Images