Eye tracking system for head-mounted display devices and operating method thereof

Abstract

To provide systems and methods for eye tracking for use in various applications such as virtual reality or augmented reality applications that include head-mounted display devices.SOLUTION: An eye tracking subsystem may be provided, comprising a plurality of assemblies, each including a light source such as an LED, a light detector such as a silicon photodiode, and a polarizer configured to prevent light reflected via specular reflection from reaching the light detectors so that only scattered light is detected by the light detectors. Machine learning or other techniques may be used to track or otherwise determine a gaze direction of a user, which may be used by one or more components of an HMD device to improve its functionality in various ways.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The following disclosure relates generally to techniques for eye-gaze tracking, and more particularly to techniques for eye-gaze tracking used in head-mounted display devices. [Background technology]

[0002] A head-mounted display (HMD) device or system is an electronic device that is worn on a user's head and, when so worn, fixes at least one electronic display within the viewable field of at least one of the user's eyes, regardless of the position or orientation of the user's head. HMD devices used to implement virtual reality systems typically completely surround the wearer's eyes and replace the actual view (or actual reality) in front of the wearer with a "virtual" reality, while HMD devices for augmented reality systems typically provide a semi-transparent or transparent overlay of one or more screens in front of the wearer's eyes so that the actual view is augmented with additional information. In the case of augmented reality systems, the "display" component of the HMD device is either transparent or peripheral to the user's field of vision so as not to completely obstruct the user's ability to view their external environment. [Brief explanation of the drawings]

[0003] [Figure 1] FIG. 1 is a schematic diagram of a networked environment including one or more systems suitable for performing at least some of the techniques described in this disclosure, including an embodiment of an eye-tracking subsystem.

[0004] [Figure 2] FIG. 1 illustrates an example environment in which at least some of the described techniques are used in conjunction with an example head-mounted display device that is coupled to a video-rendering computing system and provides a virtual reality display to a user.

[0005] [Figure 3] FIG. 1 is a front pictorial view of an HMD device having a binocular display subsystem.

[0006] [Figure 4] FIG. 1 illustrates a top view of an HMD device having a binocular display subsystem and various sensors according to an exemplary embodiment of the present disclosure.

[0007] [Figure 5] FIG. 10 illustrates an example of using a light source and a light detector to determine pupil position, for use in eye tracking, e.g., in an HMD device, according to the described techniques of this disclosure.

[0008] [Figure 6] 1 is a schematic diagram of an environment in which machine learning techniques may be used to implement an eye-tracking subsystem of an HMD device, according to one non-limiting example implementation.

[0009] [Figure 7] FIG. 1 is a flow diagram of a method of operating an HMD device including eye-tracking capabilities according to one non-limiting example implementation. Summary of the Invention

[0010] A head mounted display (HMD) system includes a support structure wearable on a user's head, an eye tracking subsystem coupled to the support structure, the eye tracking subsystem having a plurality of eye tracking assemblies each including a light source operable to emit light, a photodetector operable to detect light, and a polarizer positioned proximate to at least one of the light source and the photodetector, the polarizer configured to prevent light reflected via specular reflection from being received by the photodetector, at least one processor, and a memory for storing a set of instructions or data, the instructions or data being stored in the memory. The set of data may be summarized as comprising a memory that, upon execution, causes the HMD system to selectively cause the light sources of the plurality of eye tracking assemblies to emit light; receive light detection information captured by the light detectors of the plurality of eye tracking assemblies; provide the received light detection information as input to a predictive model; receive a determined gaze direction for the user's eye from the predictive model in response to providing the light detection information; and provide the determined gaze direction to a component associated with the HMD system for use by the HMD system. The light detection information may include a characteristic radiation pattern of each of the light sources after the light from the light sources has been reflected, scattered, or absorbed and removed by the user's face or eye. Each of the light sources may be directed toward an expected location of the user's pupil. The light sources may include light-emitting diodes emitting light having a wavelength between 780 nm and 1000 nm. The light detectors may include photodiodes. The eye tracking subsystem may include four eye tracking assemblies positioned to determine the gaze direction of the user's left eye and four eye tracking assemblies positioned to determine the gaze direction of the user's right eye. The polarizer may include two crossed linear polarizers. For each of the eye tracking assemblies, the polarizer may include a first polarizer positioned in an emission path of the light source and a second polarizer positioned in a light detection path of the photodetector.The polarizer may include at least one of a circular polarizer or a linear polarizer. For each of the plurality of eye tracking assemblies, the light source may be positioned away from the optical axis of the user's eye to provide darkfield illumination of the pupil. The predictive model may include a machine learning model or other type of function or model (e.g., polynomial, look-up table). The machine learning model may include a mixture density network (MDN) model. The machine learning model may include a recurrent neural network (RNN) model. The machine learning model may utilize past input information or eye movement information to determine the gaze direction. The machine learning model may be a model trained during field operation of a plurality of HMD systems. The HMD system may include at least one display, and the at least one processor may cause the at least one display to present user interface elements, selectively cause the light sources of the multiple eye tracking assemblies to emit light, receive light detection information captured by the light detectors of the multiple eye tracking assemblies, and update the machine learning model based, at least in part, on the received light detection information and known or inferred gaze direction information associated with the received light detection information. The user interface elements may include stationary or movable user interface elements. The HMD system may include at least one display, and the at least one processor may dynamically modify rendering output of the at least one display based, at least in part, on the determined gaze direction. The HMD system may include an interpupillary distance (IPD) adjustment component, and the at least one processor may cause the IPD adjustment component to align at least one component of the HMD system for the user based, at least in part, on the determined gaze direction.

[0011] A method of operating a head-mounted display (HMD) system may include an eye-tracking subsystem coupled to a support structure, the HMD system having multiple eye-tracking assemblies each including a light source, a photodetector, and a polarizer. The method may include selectively causing the light sources of the multiple eye-tracking assemblies to emit light, receiving light detection information captured by the multiple photodetectors, providing the received light detection information as input to a trained machine learning model, receiving a determined gaze direction for the user's eyes from the machine learning model in response to providing the light detection information, and providing the determined gaze direction to a component associated with the HMD system for use by the HMD system. Providing the received light detection information as input to a trained machine learning model may include providing the received light detection information as input to a mixture density network (MDN) model. Providing the received light detection information as input to a trained machine learning model may include providing the received light detection information as input to a recurrent neural network (RNN) model. Providing the received light detection information as input to a trained machine learning model may include providing the received light detection information to a machine learning model that utilizes past input information or eye movement information to determine the gaze direction.

[0012] The method may further include training the machine learning model during field operation of a plurality of HMD systems. The HMD systems may include at least one display, and the method may include causing the at least one display to present a user interface element; optionally, causing the light sources of the plurality of eye-tracking assemblies to emit light; receiving light detection information captured by the plurality of light detectors; and updating the machine learning model based, at least in part, on the received light detection information and known or inferred gaze direction information associated with the received light detection information. Causing the at least one display to present a user interface element may include causing the at least one display to present a stationary or movable user interface element. The HMD systems may include at least one display, and the method may include dynamically modifying an output of the at least one display based, at least in part, on the determined gaze direction.

[0013] The method may further comprise mechanically aligning at least one component of the HMD system for the user based at least in part on the determined gaze direction.

[0014] A head mounted display (HMD) system includes a support structure wearable on a user's head; an eye tracking subsystem coupled to the support structure, the eye tracking subsystem having a plurality of eye tracking assemblies each including a light emitting diode, a photodiode, and a polarizer positioned proximate to at least one of the light emitting diode and the photodiode, the polarizer configured to prevent light reflected via specular reflection from being received by the photodiode; and at least one processor and a memory storing a set of instructions or data, the set of instructions or data being operable to execute the eye tracking subsystem. As a result, the HMD system can be summarized as comprising memory that selectively causes the light-emitting diodes of the plurality of eye tracking assemblies to emit light; receives light detection information captured by the photodiodes of the plurality of eye tracking assemblies; provides the received light detection information as input to a trained machine learning model; receives a determined gaze direction for the user's eyes from the machine learning model in response to providing the light detection information; and dynamically modifies operation of components associated with the HMD system based at least in part on the determined gaze direction. DETAILED DESCRIPTION OF THE INVENTION

[0015] In the following description, certain specific details are set forth to provide a thorough understanding of various disclosed implementations. However, those skilled in the art will recognize that implementations can be practiced without one or more of these specific details, or with other methods, components, materials, etc. In other instances, well-known structures associated with computer systems, server computers, and / or communication networks have not been shown or described in detail to avoid unnecessarily obscuring the description of the implementations.

[0016] Unless the context requires otherwise, throughout this specification and the claims that follow, the term "comprising" is synonymous with "including" and is inclusive or open-ended (i.e., does not exclude additional, unrecited elements or method actions).

[0017] References throughout this specification to "one implementation" or "an implementation" mean that a particular feature, structure, or characteristic described in connection with that implementation is included in at least one implementation. Thus, the appearances of the phrase "in one implementation" or "in one implementation" in various places throughout this specification are not necessarily all referring to the same implementation. Furthermore, particular features, structures, or characteristics may be combined in any suitable manner in one or more implementations.

[0018] As used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It should also be noted that the term "or" is generally utilized to include "and / or" in its meaning unless the context clearly dictates otherwise.

[0019] The headings and abstract of the disclosure provided herein are for convenience only and do not interpret the scope or meaning of the implementation.

[0020] Eye tracking is a process by which the position, orientation, or movement of the eye can be measured, detected, sensed, determined, or monitored (collectively "measured"). In many applications, this is done for the purpose of determining a user's direction of gaze. Eye position, orientation, or movement may be measured in a variety of different ways, the least invasive of which may utilize one or more optical detectors or sensors to optically track the eye. Some techniques may involve illuminating or flooding the entire eye all at once with infrared light and measuring reflections with at least one optical sensor tuned to be sensitive to the infrared light. Information about how the infrared light reflects from the eye is analyzed to determine the position, orientation, and / or movement of one or more eye features, such as the cornea, pupil, iris, or retinal blood vessels.

[0021] Eye-tracking capabilities are highly advantageous in the application of wearable head-mounted display systems. Some examples of the usefulness of eye-tracking in head-mounted display systems include influencing where content is displayed within a user's field of view, conserving power, bandwidth, or computational resources by modifying the display of content outside the user's field of view (e.g., foveated rendering), influencing the content displayed to a user, determining where a user is looking or gazing, determining whether a user is looking at displayed content on a display, providing ways in which a user can control or interact with displayed content, and other applications.

[0022] The present disclosure generally relates to techniques for eye tracking. Such techniques may be used, for example, in head-mounted display ("HMD") devices used for VR or AR applications. Some or all of the techniques described herein may be performed through automated operation of an embodiment of an eye tracking subsystem, such as being implemented by one or more configured hardware processors and / or other configured hardware circuits. The one or more hardware processors or other configured hardware circuits of such a system or device may include, for example, one or more GPUs ("graphical processing units") and / or CPUs ("central processing units") and / or other microcontrollers ("MCUs") and / or other integrated circuits; for example, as discussed further below, the hardware processor may be part of an HMD device or other device incorporating one or more display panels on which image data will be displayed, or part of a computing system that generates or otherwise prepares image data to be sent to the display panels for display. More generally, such hardware processors or other configured hardware circuits may include, but are not limited to, one or more application specific integrated circuits (ASICs), standard integrated circuits, controllers (e.g., by executing appropriate instructions, and including microcontrollers and / or embedded controllers), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), digital signal processors (DSPs), programmable logic controllers (PLCs), etc. Further details are included elsewhere herein, including those discussed below with respect to FIG.

[0023] Technical benefits of at least some embodiments of the described techniques include addressing and mitigating increased media transmission bandwidth for image encoding by reducing image data size, improving the speed of controlling display panel pixels (e.g., based at least in part on the corresponding reduced image data size), improving foveated imaging systems and other techniques for reflecting subsets of display panels and / or images of particular interest, etc. Foveated imaging encoding systems take advantage of certain aspects of the human visual system (which may provide detailed information only at the point of focus and its surroundings), but often use specialized computational processing to avoid visual artifacts to which peripheral vision is highly susceptible (e.g., motion- and contrast-related artifacts in video and image data). For certain VR and AR displays, both bandwidth and computing usage for processing high-resolution media are amplified because certain display devices include two separate display panels (i.e., one for each eye), each having two separately addressable pixel arrays with appropriate resolution. Thus, the described techniques may be used, for example, to reduce transmission bandwidth for local and / or remote display of video frames or other images, while preserving resolution and detail in the viewer's "area of ​​interest" within the image, and while minimizing the computing usage for processing such image data. Furthermore, the use of lenses in head-mounted display devices and in conjunction with other displays may provide higher focus or resolution on a subset of the display panel, such that using such techniques to display lower resolution information on other portions of the display panel may further provide advantages when using such techniques in such embodiments.

[0024] For illustrative purposes, several embodiments are described below in which particular types of information are obtained and used in particular ways for particular types of structures and by using particular types of devices. However, it will be understood that such described techniques may be used in other ways in other embodiments, and therefore the present disclosure is not limited to the illustrative details provided. As one non-exclusive example, various embodiments discussed herein include the use of images that are video frames. However, while many examples described herein refer to "video frames" for convenience, it will be understood that the techniques described with reference to such examples may be utilized with one or more images of various types, including, but not limited to, the following non-exclusive examples: multiple consecutive video frames (e.g., at 30, 60, 90, 180, or some other number of frames per second), other video content, photographs, computer-generated graphical content, other visual media products, or any combination thereof. Additionally, various details are provided in the drawings and text for illustrative purposes and are not intended to limit the scope of the present disclosure. Additionally, as used herein, "pixel" refers to the smallest addressable picture element of a display that can be activated to provide all possible color values ​​for that display. Often, a pixel includes individual sub-elements (sometimes as separate "sub-pixels") that separately generate red, green, and blue light for perception by a human viewer, with separate color channels used to encode pixel values ​​for the differently colored sub-pixels. Pixel "value," as used herein, refers to a data value corresponding to the respective stimulus levels for one or more of those respective RGB elements of a single pixel.

[0025] 1 is a schematic diagram of a networked environment 100 including a local media rendering (LMR) system 110 (e.g., a gaming system) comprising a local computing system 120 and a display device 180 (e.g., an HMD device having two display panels) suitable for performing at least some of the techniques described herein. In the illustrated embodiment of FIG. 1, the local computing system 120 is communicatively connected to the display device 180 via a transmission link 115 (which may be wired or tethered, such as via one or more cables (cable 220) as shown in FIG. 2, or may alternatively be wireless). In other embodiments, the local computing system 120, whether in addition to or instead of the HMD device 180, may provide coded image data for display via a wired or wireless link to a panel display device (e.g., a TV, console, or monitor), each comprising one or more addressable pixel arrays. In various embodiments, local computing system 120 may include a general-purpose computing system, a gaming console, a video stream processing device, a mobile computing device (e.g., a mobile phone, PDA, or other mobile device), a VR or AR processing device, or other computing system.

[0026] In the illustrated embodiment, local computing system 120 has components including one or more hardware processors (e.g., central processing units, or "CPUs") 125, memory 130, various I / O ("input / output") hardware components 127 (e.g., a keyboard, a mouse, one or more gaming controllers, speakers, a microphone, an IR transmitter and / or receiver, etc.), a video subsystem 140 including one or more dedicated hardware processors (e.g., graphics processing units, or "GPUs") 144 and video memory (VRAM) 148, computer-readable storage 150, and a network connection 160. Also, in the illustrated embodiment, an embodiment of eye tracking subsystem 135 executes in memory 130 to perform the described techniques, such as by using CPU 125 and / or GPU 144 to perform automated operations implementing at least some of the described techniques, and memory 130 may optionally further execute one or more other programs 133 (e.g., for generating video or other images to be displayed, such as a game program) as part of automated operations implementing at least some of the techniques described herein. As part of automated operations implementing at least some of the techniques described herein, eye tracking subsystem 135 and / or program 133 executing in memory 130 may store or retrieve various types of data, including, in the example database, data structures in storage 150; in this example, the data used may include various types of image data information in database (“DB”) 154, various types of application data in DB 152, various types of configuration data in DB 157, and may include additional information, such as system data or other information.

[0027] LMR system 110, in the illustrated embodiment, is also communicatively connected via one or more computer networks 101 and network link 102 to an exemplary network-accessible media content provider 190 that may further provide content to LMR system 110 for display, whether in addition to or instead of image generator 133. For simplicity, some details about the network-accessible media content provider are not shown, but media content provider 190 may include one or more computing systems (not shown), each of which may have components similar to those of local computing system 120, including one or more hardware processors, I / O components, local storage devices, and memory.

[0028] 1, display device 180 is shown as distinct and separate from local computing system 120, it will be appreciated that in certain embodiments, some or all of the components of local media rendering system 110 may be integrated or housed within a single device, such as a mobile gaming device, a portable VR entertainment system, an HMD device, etc. In such embodiments, transmission link 115 may include, for example, one or more system bus and / or video bus architectures.

[0029] As one example involving operations performed locally by local media rendering system 120, assume that the local computing system is a gaming computing system, whereby application data 152 includes one or more gaming applications executing via CPU 125 using memory 130, and various video frame display data is generated and / or processed by image generation program 133, such as in combination with GPU 144 of video subsystem 140. To provide a high-quality gaming experience, a high volume of video frame data (corresponding to a high image resolution per video frame and a high "frame rate" of approximately 60-180 such video frames per second) is generated by local computing system 120 and provided to display device 180 via wired or wireless transmission link 115.

[0030] It will also be understood that computing system 120 and display device 180 are merely exemplary and are not intended to limit the scope of the present disclosure. Instead, computing system 120 may include multiple interacting computing systems or devices and may be connected to other devices not shown, including through one or more networks such as the Internet, via the Web, or via a private network (e.g., a mobile communications network, etc.). More generally, a computing system or other computing node may include any combination of hardware or software capable of interacting and performing functions of the types described, including, but not limited to, desktop or other computers, gaming systems, database servers, network storage devices and other network devices, PDAs, cellular phones, wireless phones, pagers, electronic organizers, Internet appliances, television-based systems (e.g., using set-top boxes and / or personal / digital video recorders), and various other consumer products that include appropriate communications capabilities. Display device 180 may similarly include one or more devices having one or more display panels of various types and forms, and may optionally include various other hardware and / or software components.

[0031] Additionally, functionality provided by the eye tracking subsystem 135 may be distributed across one or more components in some embodiments, and in some embodiments, some of the functionality of the eye tracking subsystem 135 may not be provided and / or other additional functionality may be available. It will also be understood that while various items are shown as being stored in memory or on storage during use, these items, or portions thereof, may be transferred between memory and other storage devices for memory management or data integrity purposes. Thus, in some embodiments, some or all of the described techniques may be performed by hardware, including one or more processors or other configured hardware circuits or memory or storage, such as when configured by one or more software programs (e.g., by the eye tracking subsystem 135 or components thereof) and / or data structures (e.g., by execution of software instructions of one or more software programs and / or storage of such software instructions and / or data structures). Some or all of the components, systems, and data structures may be stored (e.g., as software instructions or structured data) on a non-transitory computer-readable storage medium, such as a hard disk or flash drive or other non-volatile storage device, volatile or non-volatile memory (e.g., RAM), network storage device, or portable media product read by an appropriate drive (e.g., DVD disk, CD disk, optical disk, etc.) or via an appropriate connection. The systems, components, and data structures may, in some embodiments, be transmitted as a generated data signal (e.g., as part of a carrier wave or other analog or digital propagated signal) over a variety of computer-readable transmission media, including wireless-based media and wired / cable-based media, and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as a number of discrete digital packets or frames). Such computer program products may take other forms in other embodiments.Accordingly, the present invention may be practiced with other computer system configurations.

[0032] 2 illustrates an example environment 200 in which at least some of the described techniques are used in conjunction with an example HMD device 202 coupled to a video rendering computing system 204 via a tethered connection 220 (or a wireless connection in other embodiments) to provide a virtual reality display to a human user 206. The user wears the HMD device 202 and receives display information of a simulated environment that differs from the actual physical environment from the computing system 204 via the HMD device, which functions as an image rendering system that provides images of the simulated environment, such as images generated by a game program and / or other software program running on the computing system, to the HMD device for display to the user. The user is further able to move around within a tracked volume 201 of the actual physical environment 200 in this example and may further have one or more I / O (“input / output”) devices, which in this example include handheld controllers 208 and 210, that enable the user to further interact with the simulated environment.

[0033] In the illustrated example, the environment 200 may include one or more base stations 214 (two are shown, labeled base stations 214a and 214b) that may facilitate tracking of the HMD device 202 or the controllers 208 and 210. As the user moves location or changes orientation of the HMD device 202, the position of the HMD device is tracked, enabling, for example, corresponding portions of the simulated environment to be displayed to the user on the HMD device; the controllers 208 and 210 may further employ similar techniques for use in tracking the position of the controller (and, optionally, to use that information to help determine or verify the position of the HMD device). After the tracked position of the HMD device 202 is known, corresponding information is transmitted via tether 220 or wirelessly to the computing system 204, which uses the tracked position information to generate one or more subsequent images of the simulated environment for display to the user.

[0034] There are many different methods of position tracking that may be used in various implementations of the present disclosure, including, but not limited to, acoustic tracking, inertial tracking, magnetic tracking, optical tracking, combinations thereof, and the like.

[0035] In at least some implementations, the HMD device 202 may include one or more optical receivers or sensors that can be used to implement tracking functionality or other aspects of the present disclosure. For example, the base stations 214 may each scan optical signals across the tracked volume 201. Depending on the requirements of each particular implementation, each base station 214 may generate more than one optical signal. For example, while a single base station 214 is typically sufficient for six degrees of freedom tracking, in some embodiments, multiple base stations (e.g., base stations 214a, 214b) may be necessary or desirable to provide robust room-wide tracking of the HMD device and peripherals. In this example, optical receivers are integrated into the HMD device 202 and / or other tracked objects, such as the controllers 208 and 210. In at least some implementations, optical receivers may be paired with accelerometer and gyroscope inertial measurement units (“IMUs”) on each tracked device to support low-latency sensor fusion.

[0036] In at least some implementations, each base station 214 comprises two rotors that scan a linear beam across the tracked volume 201 on mutually orthogonal axes. At the beginning of each scanning cycle, the base station 214 may emit an omnidirectional light pulse (referred to as a "synchronization signal") visible to all sensors toward the tracked object. Each sensor then calculates a unique angular position in the scanning volume by timing the duration between the synchronization signal and the beam signal. Sensor range and orientation may be determined using multiple sensors fixed to a single rigid body.

[0037] One or more sensors positioned on the tracked object (e.g., HMD device 202, controllers 208 and 210) may include optoelectronic devices capable of detecting modulated light from the rotor. For visible or near-infrared (NIR) light, silicon photodiodes and suitable amplifier / detector circuitry may be used. Because the environment 200 may contain stationary and time-varying signals (optical noise) having wavelengths similar to those of the base station 214 signal, in at least some implementations, the base station light may be modulated to facilitate distinguishing it from any interfering signals and / or filtering the sensor from any wavelengths of radiation other than that of the base station signal.

[0038] Inside-out tracking is also a type of position tracking that can be used to track the position of the HMD device 202 and / or other objects (e.g., controllers 208 and 210, tablet computers, smartphones). Inside-out tracking differs from outside-in tracking by the location of the camera or other sensor used to determine the position of the HMD. With inside-out tracking, the camera or sensor is located on the HMD or the object being tracked, while in outside-out tracking, the camera or sensor is placed in a stationary position within the environment.

[0039] An HMD using inside-out tracking uses one or more cameras to "look out" to determine how its position changes relative to the environment. As the HMD moves, the sensors readjust its location within the room, and the virtual environment responds accordingly in real time. This type of positional tracking can be achieved with or without markers placed in the environment. Cameras placed on the HMD observe features of the surrounding environment. If markers are used, they are designed to be easily detected by the tracking system and placed within a specific area. With "markerless" inside-out tracking, the HMD system determines position and orientation using distinctive features (e.g., natural features) that are inherently present in the environment. The HMD system's algorithms identify specific images or shapes and use them to calculate the device's position in space. Data from the accelerometer and gyroscope can also be used to improve the accuracy of positional tracking.

[0040] FIG. 3 illustrates information 300 showing a front view of an exemplary HMD device 344 when worn on the head of a user 342. The HMD device 344 includes a front structure 343 supporting a front or forward-facing camera 346 and one or more types of multiple sensors 348a-348d (collectively 348). As one example, some or all of the sensors 348, such as optical sensors that detect and use optical information emitted from one or more external devices (not shown, e.g., base station 214 of FIG. 2), may help determine the location and orientation of the device 344 in space. As shown, the forward-facing camera 346 and sensors 348 are pointed forward toward a real-world scene or environment (not shown) in which the user 342 operates the HMD device 344. The real-world physical environment may include, for example, one or more objects (e.g., walls, ceilings, furniture, stairs, cars, trees, tracking markers, or any other type of object). The particular number of sensors 348 may be fewer or more than the number of sensors illustrated. The HMD device 344 may further comprise one or more additional components not attached to the front structure (e.g., internal to the HMD device), such as an IMU (Inertial Measurement Unit) 347 electronic device that measures and reports specific forces, angular velocities, and / or magnetic fields surrounding the HMD device (e.g., using a combination of accelerometers and gyroscopes, and optionally magnetometers) of the HMD device 344. The HMD device may further comprise additional components not shown, including one or more display panels and optical lens systems oriented toward the user's eyes (not shown), optionally having one or more attached internal motors for changing the alignment or other positioning of one or more of the optical lens systems and / or display panels within the HMD device, as discussed in more detail below with respect to FIG.

[0041] The depicted example of HMD device 344 is supported on the head of user 342 based at least in part on one or more straps 345 attached to the housing of HMD device 344 and extending wholly or partially around the user's head. Although not shown here, HMD device 344 may further include one or more external motors, such as those attached to one or more of straps 345, and the automated corrective action may include using such motors to adjust such straps to correct alignment or other positioning of the HMD device on the user's head. It will be understood that HMD devices may include other support structures not shown here (e.g., nosepieces, chin straps, etc.), whether in addition to or instead of the illustrated straps, and that some embodiments may include motors attached to one or more such other support structures to similarly adjust their shape and / or position to correct alignment or other positioning of the HMD device on the user's head. Other display devices that are not fixed to the user's head may similarly be attached to or be part of one or more structures that affect the positioning of the display device, and in at least some embodiments may be equipped with motors or other mechanical actuators that similarly modify their shape and / or position to modify the alignment or other positioning of the display device relative to one or more pupils of one or more users of the display device.

[0042] 4 shows a simplified top view 400 of an HMD device 405 comprising a pair of near-to-eye display systems 402 and 404. The HMD device 405 may be, for example, the same or similar HMD devices shown in FIGS. 1-3 or a different HMD device, and the HMD devices discussed herein may also be used in examples discussed further below. The near-to-eye display systems 402 and 404 of FIG. 4 comprise display panels 406 and 408 (e.g., OLED microdisplays), respectively, and respective optical lens systems 410 and 412, each having one or more optical lenses. The display systems 402 and 404 may be attached to or otherwise positioned within a housing (or frame) 414, which includes a front portion 416 (e.g., the same as or similar to the front surface 343 of FIG. 3 ), a left temple 418, a right temple 420, and an inner surface 421 that contacts or is adjacent to the face of a wearer, a user 424, when the HMD device is worn by the user. The two display systems 402 and 404 may be secured to the housing 414 in an eyeglass configuration that can be worn on the head 422 of the wearer, a user 424, with the left temple 418 and right temple 420 resting on the user's ears 426 and 428, respectively, while a nose assembly 492 may rest on the user's nose 430. 4, the HMD device 405 may be supported partially or entirely on the user's head by the nasal display and / or right and left ear temples, although in some embodiments, such as the embodiments shown in Figures 2 and 3, straps (not shown) or other structures may be used to secure the HMD device to the user's head. The housing 414 may be shaped and sized to position each of the two optical lens systems 410 and 412 in front of one of the user's eyes 432 and 434, respectively, so that the target position of each pupil 494 is centered vertically and horizontally in front of the respective optical lens system and / or display panel.It should be understood that while the housing 414 is shown in a simplified manner similar to eyeglasses for illustrative purposes, in practice more sophisticated structures (e.g., goggles, integrated headbands, helmets, straps, etc.) may be used to support and position the display systems 402 and 404 on the head 422 of the user 424.

[0043] 4, and other HMD devices discussed herein, are capable of presenting a virtual reality display to a user, such as via corresponding video presented at a display rate such as 30, 60, or 90 frames (or images) per second, while other embodiments of similar systems may present an augmented reality display to a user. Each of the displays 406 and 408 in FIG. 4 may generate light that passes through and is focused by respective optical lens systems 410 and 412 onto the eyes 432 and 434, respectively, of the user 424. The opening of each eye's pupil 494 through which light enters the eye typically has a pupil size ranging from 2 mm (millimeters) in diameter in very bright conditions to as little as 8 mm in dark conditions, while the larger iris in which the pupil is contained may have a size of approximately 12 mm, and the pupil (and surrounding iris) may typically move a few millimeters further horizontally and / or vertically within the visible portion of the eye under the open eyelid, also causing the pupil to move to different horizontal and vertical positions and to different depths from the display's optical lenses or other physical elements as the eyeball rotates about its center (resulting in a three-dimensional volume through which the pupil can move). Light entering the user's pupil is viewed as an image and / or video by the user 424. In some implementations, the distance between each of the optical lens systems 410 and 412 and the user's eyes 432 and 434 may be relatively short (e.g., less than 30 mm, less than 20 mm), which may advantageously make the HMD device feel lighter to the user (because the weight of the optical lens systems and display system is relatively closer to the user's face) and may also provide the user with a wider field of view. Although not shown here, some embodiments of such HMD devices may include various additional internal and / or external sensors.

[0044] In the illustrated embodiment, the HMD device 405 of FIG. 4 further comprises hardware sensors and additional components, such as one or more accelerometers and / or gyroscopes 490 (e.g., as part of one or more IMU units). As discussed in more detail elsewhere herein, values ​​from the accelerometers and / or gyroscopes may be used to locally determine the orientation of the HMD device. In addition, the HMD device 405 may comprise one or more front-facing cameras, such as camera 485 on the exterior of the front portion 416, the information of which may be used as part of the operation of the HMD device, such as to provide AR or positioning functionality. Furthermore, the HMD device 405 may further comprise other components 475 (e.g., electronic circuitry controlling the display of images on the display panels 406 and 408, internal storage, one or more batteries, a position tracking device that interacts with an external base station, etc.), as discussed in more detail elsewhere herein. Other embodiments may not comprise one or more of the components 475, 485, and / or 490. Although not shown here, some embodiments of such HMD devices may include various additional internal and / or external sensors, such as for tracking various other types of movements and positions of the user's body, eyes, controllers, etc.

[0045] 4 further comprises hardware sensors and additional components that may be used by disclosed embodiments as part of the described techniques for determining a user's pupils or gaze direction, which may be provided to one or more components associated with the HMD device for use by the HMD system, as discussed elsewhere herein. In this example, the hardware sensors comprise one or more eye-tracking assemblies 472 of an eye-tracking subsystem mounted on or near the display panels 406 and 408 and / or located on the interior surface 421 near the optical lens systems 410 and 412, for use in obtaining information regarding the actual positions of the user's pupils 494, e.g., separately for each pupil in this example.

[0046] Each of the eye tracking assemblies 472 may include one or more light sources (e.g., IR LEDs) and one or more photodetectors (e.g., silicon photodiodes). Additionally, while only four total eye tracking assemblies 472 are shown in FIG. 4 for clarity, it should be understood that in practice, a different number of eye tracking assemblies may be provided. In some embodiments, eight eye tracking assemblies 472 are provided, four for each eye of the user 424. Additionally, in at least some implementations, each eye tracking assembly includes a light source directed toward one of the eyes 432 and 434 of the user 424, a photodetector positioned to receive light reflected by the user's respective eye, and a polarizer positioned and configured to prevent light reflected via specular reflection from being impinged on the photodetector.

[0047] As discussed in more detail elsewhere herein, information from the eye tracking assembly 472 may be used to determine and track a user's direction of gaze while using the HMD device 405. Additionally, in at least some embodiments, the HMD device 405 may include one or more internal motors 438 (or other movement mechanisms) that may be used to move 439 the alignment and / or other positioning (e.g., vertically, horizontally left-to-right, and / or horizontally front-to-back) of one or more of the optical lens systems 410 and 412 and / or display panels 406 and 408 within the housing of the HMD device 405, such as to personalize or otherwise adjust a target pupil position of one or both of the near-to-eye display systems 402 and 404 that corresponds to the actual position of one or both of the pupils 494. Such motors 438 may be controlled, for example, by user manipulation of one or more controls 437 on the housing 414 and / or via user manipulation of one or more associated separate I / O controllers (not shown). In other embodiments, HMD device 405 may control the alignment and / or other positioning of optical lens systems 410 and 412 and / or display panels 406 and 408 without using such motors 438, such as through the use of adjustable positioning mechanisms (e.g., screws, sliders, ratchets, etc.) that are manually changed by a user via the use of controls 437. Additionally, although motors 438 are shown in FIG. 4 for only one of the near-to-eye display systems, each near-to-eye display system may have its own motor or motors in some embodiments, and in some embodiments, one or more motors may be used to control each of multiple near-to-eye display systems (e.g., independently).

[0048] While the described techniques may be used in some embodiments with display systems similar to those illustrated, in other embodiments, other types of display systems may be used, including those with a single optical lens and display device or those with multiple such optical lenses and display devices. Non-exclusive examples of other such devices include cameras, telescopes, microscopes, binoculars, spotting scopes, survey scopes, etc. Additionally, the described techniques may be used with a wide variety of display panels or other display devices that emit light to form images that one or more users view through one or more optical lenses. In other embodiments, a user may view one or more images through one or more optical lenses that are generated in a manner other than through a display panel, such as on a surface that partially or wholly reflects light from another light source.

[0049] Figure 5 illustrates an example of using multiple eye-tracking assemblies, each comprising a light source and a photodetector for determining a user's gaze position, in a particular manner in a particular embodiment according to the described techniques. In particular, Figure 5 includes information 500 to illustrate the operation of an example display panel 510 and associated optical lens 508 in providing image information to a user's eye 504 and focusing that information, for example, on the eye's pupil 506. In the illustrated embodiment, four eye-tracking assemblies 511a-511d (collectively 511) of the eye-tracking subsystem are mounted proximate the edge of the optical lens 508 and each are generally aimed at the pupil 506 of the eye 504 to emit light toward the eye 504 and capture light reflected from the pupil 506 or some or all of the surrounding iris 502.

[0050] In the illustrated example, each of the eye tracking assemblies 511 includes a light source 512, a photodetector 514, and a polarizer 516 positioned and configured to provide scattered light (diffuse reflection) to the photodetector while substantially blocking specularly reflected light from reaching the detector 514. In this example, the eye tracking assemblies 511 are located at positions including near the top of the optical lens 508 along the central vertical axis, near the bottom of the optical lens along the central vertical axis, near the left of the optical lens along the central horizontal axis, and near the right of the display panel along the central horizontal axis. In other embodiments, the eye tracking assemblies 511 may be positioned at other locations, and fewer or more eye tracking assemblies may be used.

[0051] The polarizer 516 of each eye-tracking assembly 511 may include one or more polarizers positioned in front of the light source 512 and / or the photodetector 514 to reduce or eliminate specularly reflected light from the light reaching the photodetector 514. In one example, the polarizer 516 of a particular eye-tracking assembly 511 may include two crossed linear polarizers: a first linear polarizer positioned in front of the light source 512 and a second linear polarizer oriented 90 degrees to the first polarizer positioned in front of the photodetector 514 to block specularly reflected light. In other implementations, one or more circular or linear polarizers may be used to prevent light reflected via specular reflection from reaching the photodetector 514, so that the photodetector receives substantially all light reflected via diffuse reflection.

[0052] It will be understood that the light sources and light detectors are shown for illustrative purposes only, and that other embodiments may include more or fewer light sources or detectors, and that the light sources or detectors may be located in other locations. Additionally, although not shown here, in some embodiments, additional hardware components may be used to assist in acquiring data from one or more of the light detectors. For example, an HMD device or other display device may include various light sources (e.g., infrared light, visible light, etc.) at different locations to illuminate the iris and pupil and reflect light back to one or more photodetectors, such as light sources mounted on or near the display panel 510, or alternatively elsewhere (e.g., between the optical lens 508 and the eye 504, such as on an interior surface (not shown) of an HMD device that includes the display panel 510 and the optical lens 508). In some such embodiments, light from such illumination sources may further bounce off the display panel before passing through the optical lens 508 to illuminate the iris and pupil.

[0053] FIG. 6 is a schematic diagram of an environment 600 in which machine learning techniques may be used to implement an eye-tracking subsystem of an HMD device, such as the eye-tracking subsystem discussed herein, according to one non-limiting example implementation. The environment 600 includes a model training unit 601 and an inference unit 603. In the training unit 601, training data 602 is fed to a machine learning algorithm 604 to generate a trained machine learning model 606. The training data may include, for example, labeled data from photodetectors that specify gaze positions. As a non-limiting example, in an embodiment including four photodetectors aimed at a user's eyes, each training sample may include outputs from each of the four photodetectors and a known or inferred gaze direction. In at least some implementations, the user's gaze direction may be known or inferred by instructing the user to gaze at a particular user interface element (e.g., a word, a dot, an "X," another shape or object, etc.) on the display of the HMD device, which may be stationary or movable on the display. The training data may also include samples in which one or more of the light sources or photodetectors are occluded, such as due to a user blinking, eyelashes, glasses, a hat, or other occlusion. For such training samples, the label may be "unknown" or "occluded" rather than a specified gaze direction.

[0054] The training data 602 may be obtained from multiple users of the HMD system and / or from a single user. The training data 602 may be obtained in a controlled environment and / or during actual use by a user (“field training”). Furthermore, in at least some implementations, the model 606 may be updated or calibrated from time to time (e.g., periodically, continuously, after specific events) to provide accurate gaze direction predictions.

[0055] In the inference unit 603, the runtime data 608 is provided as input to a trained machine learning model 606, which generates a gaze direction prediction 610. Continuing with the example above, photodetector output data may be provided as input to the trained machine learning model 606, which may process the data to predict a gaze position. The gaze direction prediction 610 may then be provided to one or more components associated with the HMD device, such as, for example, one or more VR or AR applications running on the HMD device, one or more display or rendering modules, one or more mechanical controls, one or more position tracking subsystems, etc.

[0056] The machine learning techniques utilized to implement the features discussed herein may include any type of suitable structure or technique. By way of non-limiting example, the machine learning model 606 may include one or more of a decision tree, a statistical hierarchical model, a support vector machine, an artificial neural network (ANN), such as a convolutional neural network (CNN) or a recurrent neural network (RNN) (e.g., a long short-term memory (LSTM) network), a mixture density network (MDN), a hidden Markov model, or others. In at least some implementations, such as those utilizing RNNs, the machine learning model 606 may utilize past input (memory, feedback) information to predict gaze direction. Such implementations may advantageously utilize motion information or sequential data to determine previous gaze direction predictions, which may provide more accurate real-time gaze direction predictions.

[0057] 7 is a flow diagram of an example embodiment of a method 700 for operating an eye-gaze tracking subsystem of an HMD device. Method 700 may be performed, for example, by eye-gaze tracking subsystem 135 of FIG. 1 or other systems discussed elsewhere herein. While the illustrated embodiment of 700 discusses performing operations for determining gaze direction for a single eye, it will be understood that the operations of method 700 may be applied simultaneously to both eyes of a user to track gaze direction substantially in real time. It will also be understood that the illustrated embodiment of method 700 may be implemented in software and / or hardware, as appropriate, and may be performed, for example, by a computing system associated with an HMD device, for example.

[0058] As discussed above, an HMD device may include a support structure wearable on a user's head, an eye-tracking subsystem coupled to the support structure, at least one processor, and a memory that stores a set of instructions or data. The eye-tracking subsystem may include multiple eye-tracking assemblies, each including a light source, a photodetector, and a polarizer.

[0059] The polarizer may be positioned proximate to at least one of the light source and the photodetector and may be configured to prevent light reflected via specular reflection from being received by the photodetector. In at least some implementations, the polarizer may include two crossed linear polarizers. In at least some implementations, for each of the plurality of eye-tracking assemblies, the polarizer includes a first polarizer positioned in the light emission path of the light source and a second polarizer positioned in the light detection path of the photodetector. More generally, the polarizer may include at least one of a circular polarizer or a linear polarizer.

[0060] Each of the light sources may be aimed at a target location on the user's pupil, which may allow for the use of lower-power light sources because the energy is concentrated at the target location. Additionally, various optical elements (e.g., lenses) or various types of light sources (e.g., IR lasers) may be used to focus the light on the target area. In at least some implementations, the light sources may include light-emitting diodes emitting light having wavelengths between 780 nm and 1000 nm, for example. The light sources may be positioned away from the optical axis of the user's eye to provide dark-field illumination of the pupil. In at least some implementations, the photodetector may include a silicon photodiode that provides an output signal dependent on incident optical power.

[0061] The illustrated embodiment of method 700 begins at 702, where at least one processor of an HMD device may selectively cause light sources of multiple eye-tracking assemblies to emit light. The at least one processor may cause the light sources to emit light simultaneously, sequentially, in separate patterns, or any combination thereof. At 704, the at least one processor may receive light detection information captured by light detectors of the multiple eye-tracking assemblies. For example, the at least one processor may store output data received from the multiple light detectors over one or more time periods.

[0062] At 706, the at least one processor may provide the received light detection information as input to a trained machine learning model, such as model 606 discussed above with respect to FIG. 6. As discussed above, the machine learning model may include an RNN, an MDN, or any other type of machine learning model suitable for providing accurate gaze direction predictions based on light data input received from multiple light detectors. As discussed elsewhere herein, in other implementations, predictive models or functions other than machine learning models may be used, such as 1D or 2D polynomials, one or more lookup tables, etc.

[0063] At 708, the at least one processor may receive a determined gaze direction for the user's eye from the machine learning model in response to providing the light detection information. The determined gaze direction may be provided in any suitable format.

[0064] To calibrate or update the machine learning model, the at least one processor may cause at least one display of the HMD to present user interface elements, selectively cause the light sources of the multiple eye tracking assemblies to emit light, and receive light detection information captured by the light detectors of the multiple eye tracking assemblies. The light detection information, along with corresponding known or inferred gaze direction information, may be used to update the machine learning model. The model may be updated various times as needed to provide accurate gaze direction predictions. The user interface elements may include stationary or movable user interface elements.

[0065] At 710, the at least one processor provides the determined gaze direction to a component associated with the HMD system for use by the HMD system. For example, the determined gaze direction may be provided to an image rendering subsystem of the HMD system to provide foveated rendering based on the determined gaze direction, as discussed above. As another example, the eye tracking subsystem may determine that the user's eyes are saccadic and may dynamically modify image rendering to take advantage of saccadic masking or saccadic suppression. For example, one or more characteristics of the image rendering, such as the resolution of all or a portion of the image, the spatial frequency of the image, the frame rate, or any other characteristic that may enable reduced bandwidth, lower computational requirements, or other technical benefits, may be modified during the saccadic eye movement.

[0066] As another example, in at least some implementations, an HMD device may include an interpupillary distance (IPD) adjustment component, the IPD adjustment component operable to automatically adjust one or more components of the HMD to account for variable IPD. In this example, the IPD adjustment component may receive a gaze direction and align at least one component of the HMD system for the user based, at least in part, on the determined gaze direction. For example, when a user is looking at a nearby object, the IPD may be relatively short, and the IPD adjustment may align one or more components of the HMD device accordingly. As yet another non-limiting example, the HMD device may automatically adjust the focus of the lenses based on the determined gaze direction.

[0067] While the above examples utilize machine learning techniques to determine gaze direction from light detection information, it should be understood that the features of the present disclosure are not limited to using machine learning techniques. Generally, any type of predictive model or function may be used. For example, in at least some implementations, rather than proceeding directly from light detection information to gaze direction, the system may do the reverse, i.e., predict light detection information given an input gaze direction. Such a method may find a predictive function or model that maps from gaze direction to a predicted light reading for that direction to perform this prediction. Because this function may be user-specific, the system may implement a calibration process to discover or customize the function. Once the predictive function is determined or generated, it may then be inverted (e.g., using a numerical solver) to generate real-time predictions during use. Specifically, given samples of light detection information from a real sensor, the solver is operable to find a gaze direction that minimizes the error between this actual reading and the predicted reading from the generated predictive function. In such implementations, the output is the determined gaze direction plus a residual error, which can be used to determine the quality of the solution. In at least some implementations, some additional correction may be applied to address various issues, such as the HMD system sliding around the user's face during operation. The prediction function may be any type of function. As an example, given a dataset of points captured from the user, a set of 2D polynomials may be used to map gaze angles to photodiode readings. In at least some other implementations, lookup tables or other techniques may also be used, including fitting ML systems to output predictions as discussed above.

[0068] It will be understood that in some embodiments, the functionality provided by the routines discussed above may be provided in alternative ways, such as being divided among more routines or consolidated into fewer routines. Similarly, in some embodiments, the illustrated routines may provide more or less functionality than described, such that other illustrated routines may alternatively lack or include such functionality, respectively, or the amount of functionality provided may vary. Additionally, while various operations may be shown as being performed in a particular way (e.g., sequentially or in parallel) and / or in a particular order, those skilled in the art will understand that in other embodiments, the operations may be performed in other orders and in other ways. Similarly, it will be understood that the data structures discussed above may be structured in different ways, including as databases or user interface screens / pages or other types of data structures, such as by dividing a single data structure into multiple data structures or by combining multiple data structures into a single data structure. Similarly, in some embodiments, the illustrated data structures may store more or less information than described, such as when other illustrated data structures may instead lack or include such information, respectively, or when the amount or type of information stored is changed.

[0069] Additionally, the sizes and relative positions of elements within the drawings, including the shapes and angles of various elements, are not necessarily drawn to scale; some elements have been enlarged and positioned to improve the readability of the drawings; and the particular shapes of at least some elements have been selected for ease of recognition without conveying information regarding the actual shape or scale of those elements. Additionally, some elements may be omitted for clarity and emphasis. Furthermore, reference numbers repeated in different drawings may refer to the same or similar elements.

[0070] From the foregoing, it will be understood that, while specific embodiments have been described herein for purposes of illustration, various modifications may be made without departing from the spirit and scope of the invention. Additionally, while certain aspects of the invention may sometimes be presented in particular claim forms, or may sometimes not be embodied in any claim, the inventors contemplate various aspects of the invention in any available claim form. For example, while only some aspects of the invention may be recited at a particular time as being embodied in a computer-readable medium, other aspects may likewise be so embodied.

[0071] U.S. Patent Application No. 16 / 773,840, filed January 27, 2020, from which this application claims priority, is hereby incorporated by reference in its entirety. According to this specification, the configurations described in the following items are also disclosed. (Item 1) A head-mounted display (HMD) system, comprising: a support structure wearable on a user's head; gaze tracking subsystems coupled to the support structure, each of the gaze tracking subsystems comprising: a light source operable to emit light; a photodetector operable to detect light; a polarizer positioned proximate to at least one of the light source and the photodetector, the polarizer configured to prevent light reflected via specular reflection from being received by the photodetector; an eye-tracking subsystem having a plurality of eye-tracking assemblies including: at least one processor; a memory for storing a set of instructions or data, the set of instructions or data being, as a result of execution, capable of causing the HMD system to: Optionally, causing the light sources of the plurality of eye-tracking assemblies to emit light; receiving light detection information captured by the light detectors of the plurality of eye tracking assemblies; providing the received light detection information as an input to a predictive model; receiving a determined gaze direction for the user's eye from the predictive model in response to providing the light detection information; providing the determined gaze direction to a component associated with the HMD system for use by the HMD system; memory and An HMD system comprising: (Item 2) Item 1. The HMD system of item 1, wherein the light detection information includes a characteristic radiation pattern of each of the light sources after the light from the light sources has been reflected, scattered, or absorbed and removed by the user's face or eyes. (Item 3) 3. The HMD system of claim 1, wherein each of the light sources is directed toward an expected position of the user's pupil. (Item 4) 4. The HMD system of any one of items 1 to 3, wherein the light source includes a light-emitting diode that emits light having a wavelength of 780 nm to 1000 nm. (Item 5) 5. The HMD system of any one of items 1 to 4, wherein the light detector includes a photodiode. (Item 6) 6. The HMD system of any one of items 1 to 5, wherein the gaze tracking subsystem has four gaze tracking assemblies positioned to determine the gaze direction of the user's left eye and four gaze tracking assemblies positioned to determine the gaze direction of the user's right eye. (Item 7) 7. The HMD system of any one of items 1 to 6, wherein the polarizer comprises two crossed linear polarizers. (Item 8) 8. The HMD system of any one of items 1 to 7, wherein for each of the plurality of gaze tracking assemblies, the polarizer includes a first polarizer positioned in an emission path of the light source and a second polarizer positioned in a light detection path of the light detector. (Item 9) 9. The HMD system of any one of items 1 to 8, wherein the polarizer comprises at least one of a circular polarizer or a linear polarizer. (Item 10) 10. The HMD system of any one of items 1 to 9, wherein for each of the plurality of eye tracking assemblies, the light source is positioned away from the optical axis of the user's eye to provide dark field illumination of the pupil. (Item 11) 11. The HMD system of any one of items 1 to 10, wherein the predictive model includes a machine learning model. (Item 12) Item 12. The HMD system of item 11, wherein the machine learning model includes a mixture density network (MDN) model. (Item 13) Item 13. The HMD system of item 11 or 12, wherein the machine learning model includes a recurrent neural network (RNN) model. (Item 14) 14. The HMD system of any one of items 11 to 13, wherein the machine learning model utilizes past input information or eye movement information to determine the gaze direction. (Item 15) 15. The HMD system of any one of items 11 to 14, wherein the machine learning model is a model trained during field operation of multiple HMD systems. (Item 16) 16. The HMD system of any one of items 11 to 15, wherein the predictive model comprises a polynomial or a lookup table. (Item 17) The HMD system includes at least one display, and the at least one processor: causing the at least one display to present user interface elements; selectively causing the light sources of the plurality of eye tracking assemblies to emit light; receiving light detection information captured by the light detectors of the plurality of eye tracking assemblies; 17. The HMD system of any one of items 11 to 16, wherein the machine learning model is updated based, at least in part, on the received light detection information and known or inferred gaze direction information associated with the received light detection information. (Item 18) Item 18. The HMD system of item 17, wherein the user interface element comprises a stationary user interface element or a movable user interface element. (Item 19) The HMD system includes at least one display, and the at least one processor: 19. The HMD system of any one of items 1 to 18, wherein the HMD system dynamically modifies the rendering output of the at least one display based at least in part on the determined gaze direction. (Item 20) 20. The HMD system of any one of items 1 to 19, wherein the HMD system includes an interpupillary distance (IPD) adjustment component, and the at least one processor causes the IPD adjustment component to align at least one component of the HMD system for the user based at least in part on the determined gaze direction. (Item 21) 1. A method of operating a head-mounted display (HMD) system, the HMD system comprising an eye-tracking subsystem coupled to a support structure, the eye-tracking subsystem having a plurality of eye-tracking assemblies each including a light source, a photodetector, and a polarizer, the method comprising: Optionally, causing the light sources of the plurality of eye-tracking assemblies to emit light; receiving light detection information captured by the plurality of light detectors; providing the received light detection information as input to a trained machine learning model; receiving a determined gaze direction for the user's eyes from the machine learning model in response to providing the light detection information; providing the determined gaze direction to a component associated with the HMD system for use by the HMD system; A method comprising: (Item 22) Item 22. The method of item 21, wherein the light detection information includes a characteristic radiation pattern of each of the light sources after the light from the light sources has been reflected, scattered, or absorbed and removed by the user's face or eyes. (Item 23) 23. The method of claim 21 or 22, wherein providing the received light detection information as input to a trained machine learning model comprises providing the received light detection information as input to a mixture density network (MDN) model. (Item 24) 24. The method of any one of claims 21 to 23, wherein providing the received light detection information as input to a trained machine learning model comprises providing the received light detection information as input to a recurrent neural network (RNN) model. (Item 25) 25. The method of any one of items 21 to 24, wherein providing the received light detection information as input to a trained machine learning model comprises providing the received light detection information to a machine learning model that utilizes past input information or eye movement information to determine the gaze direction. (Item 26) 26. The method of any one of items 21 to 25, further comprising training the machine learning model during field operation of a plurality of HMD systems. (Item 27) The HMD system includes at least one display, and the method includes: causing the at least one display to present user interface elements; Optionally, causing the light sources of the plurality of eye-tracking assemblies to emit light; receiving light detection information captured by the plurality of light detectors; updating the machine learning model based, at least in part, on the received light detection information and known or inferred gaze direction information associated with the received light detection information. 27. The method of any one of items 21 to 26, comprising: (Item 28) 28. The method of claim 27, wherein causing the at least one display to present a user interface element comprises causing the at least one display to present a stationary user interface element or a movable user interface element. (Item 29) The HMD system includes at least one display, and the method includes: 29. The method of any one of items 21 to 28, comprising dynamically modifying an output of the at least one display based at least in part on the determined gaze direction. (Item 30) 30. The method of any one of items 21 to 29, further comprising mechanically aligning at least one component of the HMD system for the user based at least in part on the determined gaze direction. (Item 31) A head-mounted display (HMD) system, comprising: a support structure wearable on a user's head; gaze tracking subsystems coupled to the support structure, each of the gaze tracking subsystems comprising: A light emitting diode, A photodiode; a polarizer positioned proximate to at least one of the light emitting diode and the photodiode, the polarizer configured to prevent light reflected via specular reflection from being received by the photodiode; and an eye-tracking subsystem having a plurality of eye-tracking assemblies including: at least one processor; a memory for storing a set of instructions or data, the set of instructions or data being, as a result of execution, capable of causing the HMD system to: selectively causing the light emitting diodes of the plurality of eye tracking assemblies to emit light; receiving light detection information captured by the photodiodes of the plurality of eye tracking assemblies; providing the received light detection information as an input to a predictive model; receiving a determined gaze direction for the user's eye from the predictive model in response to providing the light detection information; dynamically modifying operation of a component associated with the HMD system based at least in part on the determined gaze direction; and memory and An HMD system comprising:

Claims

1. a plurality of eye-tracking assemblies, each of the plurality of eye-tracking assemblies comprising: A light source and a photodetector; a polarizer positioned proximate to at least one of the light source and the photodetector, the polarizer configured to prevent light reflected via specular reflection from being received by the photodetector and to allow light reflected via diffuse reflection to be received by the photodetector, wherein light reflected via specular reflection includes light reflected from a surface at an angle equal to the angle of incidence and light reflected via diffuse reflection includes light scattered from a surface at many angles; and An eye-tracking system, including:

2. The eye-tracking system of claim 1 , wherein each of the light sources is aimed at an expected location of a user's pupil.

3. 3. The gaze tracking system of claim 1, wherein the light source comprises a light emitting diode that emits light having a wavelength between 780 nm and 1000 nm.

4. The gaze tracking system of claim 1 , wherein the light detector comprises a photodiode.

5. 5. The gaze tracking system of claim 1, comprising four gaze tracking assemblies positioned to determine the gaze direction of a user's left eye and four gaze tracking assemblies positioned to determine the gaze direction of the user's right eye.

6. The gaze tracking system of claim 1 , wherein the polarizer comprises two crossed linear polarizers.

7. 7. The gaze tracking system of claim 1, wherein for each of the plurality of gaze tracking assemblies, the polarizer includes a first polarizer positioned in an emission path of the light source and a second polarizer positioned in a light detection path of the photodetector.

8. The gaze tracking system of claim 1 , wherein the polarizer comprises at least one of a circular polarizer or a linear polarizer.

9. 9. The eye-tracking system of claim 1, wherein for each of the plurality of eye-tracking assemblies, the light source is positioned away from the optical axis of the user's eye to provide dark-field illumination of the eye's pupil.

10. 1. An eye-tracking system, comprising: a plurality of eye-tracking assemblies, each of the plurality of eye-tracking assemblies comprising: A light source and a photodetector; a polarizer positioned proximate to at least one of the light source and the photodetector, the polarizer configured to prevent light reflected via specular reflection from being received by the photodetector and to allow light reflected via diffuse reflection to be received by the photodetector, wherein light reflected via specular reflection includes light reflected from a surface at an angle equal to the angle of incidence and light reflected via diffuse reflection includes light scattered from a surface at many angles; and a control circuit that, during operation, causes the eye tracking system to Optionally, causing the light sources of the plurality of eye-tracking assemblies to emit light; receiving light detection information captured by the light detectors of the plurality of eye tracking assemblies; providing the received light detection information as an input to a predictive model; receiving a determined gaze direction for the user's eyes from the predictive model in response to providing the light detection information; storing the determined gaze direction in a non-transitory memory; the control circuit causing the An eye-tracking system, including:

11. 11. The gaze tracking system of claim 10, wherein the light detection information includes a characteristic radiation pattern of each of the light sources after the light from the light sources has been reflected, scattered, or absorbed and removed by the user's face or eyes.

12. 12. The eye-tracking system of claim 10 or 11, wherein the light source is aimed at the expected location of the user's pupils.

13. 13. The gaze tracking system of claim 10, wherein the light source comprises a light emitting diode that emits light having a wavelength between 780 nm and 1000 nm.

14. 14. The eye-tracking system of claim 10, wherein the light detector comprises a photodiode.

15. 15. The gaze tracking system of claim 10, comprising four gaze tracking assemblies positioned to determine the gaze direction of a user's left eye and four gaze tracking assemblies positioned to determine the gaze direction of the user's right eye.

16. 16. The eye-tracking system of claim 10, wherein the polarizer comprises two crossed linear polarizers.

17. 17. The gaze tracking system of claim 10, wherein for each of the plurality of gaze tracking assemblies, the polarizer comprises a first polarizer positioned in an emission path of the light source and a second polarizer positioned in a light detection path of the light detector.

18. 18. The gaze tracking system of claim 10, wherein the polarizer comprises at least one of a circular polarizer or a linear polarizer.

19. 19. The eye-tracking system of claim 10, wherein for each of the plurality of eye-tracking assemblies, the light source is positioned away from the optical axis of a user's eye to provide dark-field illumination of the eye's pupil.

20. The eye-tracking system of claim 10 , wherein the predictive model comprises a machine learning model.

21. and further comprising at least one display, wherein in operation the control circuitry: causing the at least one display to present user interface elements; Optionally, causing the light sources of the plurality of eye-tracking assemblies to emit light; receiving light detection information captured by the light detectors of the plurality of eye tracking assemblies; updating the predictive model based at least in part on the received light detection information and known or inferred gaze direction information associated with the received light detection information; 21. The eye-tracking system of claim 10, wherein the eye-tracking system executes the following:

22. The eye-tracking system of claim 21 , wherein the user interface element comprises a stationary user interface element or a movable user interface element.

23. and further comprising at least one display, wherein in operation the control circuitry:

23. The eye-tracking system of claim 21 or 22, wherein the system dynamically modifies a rendering output of the at least one display based at least in part on the determined gaze direction.

24. and an interpupillary distance (IPD) adjustment component, wherein during operation the control circuitry:

24. The eye-tracking system of claim 10, wherein the IPD adjustment component aligns at least one component of the eye-tracking system for a user based at least in part on the determined gaze direction.

25. 1. A method of operating an eye-tracking system comprising a plurality of eye-tracking assemblies, each including a light source, a photodetector, and a polarizer, comprising: Optionally, causing a plurality of the light sources of the plurality of eye-tracking assemblies to emit light; preventing light reflected via specular reflection from being received by the plurality of photodetectors and allowing light reflected via diffuse reflection to be received by the plurality of photodetectors with each of the polarizers of each of the plurality of eye tracking assemblies, wherein light reflected via specular reflection includes light reflected from a surface at an angle equal to the angle of incidence and light reflected via diffuse reflection includes light scattered from a surface at many angles; receiving light detection information captured by the plurality of light detectors, the light detection information including the light reflected by diffuse reflection; providing the received light detection information as input to a trained machine learning model; receiving, from the trained machine learning model in response to providing the light detection information, a determined gaze direction of the user's eyes; storing the determined gaze direction in a non-transitory memory; A method for providing the above.

26. 26. The method of claim 25, wherein the light detection information includes a characteristic radiation pattern of each of the light sources after the light from the light sources has been reflected, scattered, or absorbed and removed by the face or eyes of the user.

27. 27. The method of claim 25 or 26, wherein providing the received light detection information as input to the trained machine learning model comprises providing the received light detection information as input to a mixture density network (MDN) model.

28. 28. The method of claim 25, wherein providing the received light detection information as input to the trained machine learning model comprises providing the received light detection information as input to a recurrent neural network (RNN) model.

29. 29. The method of any one of claims 25 to 28, wherein providing the received light detection information as input to the trained machine learning model comprises providing the received light detection information to a machine learning model that utilizes past input information or eye movement information to determine the gaze direction.

30. 30. The method of any one of claims 25 to 29, further comprising training the trained machine learning model during field operation of a plurality of eye-tracking systems.

31. The eye-tracking system includes at least one display, and the method includes:

31. The method of any one of claims 25 to 30, comprising dynamically modifying the output of the at least one display based at least in part on the determined gaze direction.

32. 32. The method of any one of claims 25 to 31, further comprising mechanically aligning at least one component of an HMD system for the user based at least in part on the determined gaze direction.

Images