Optical tracking including imaging-based angle sensitive detector

Abstract

The present disclosure relates to systems and methods of object tracking for use in various applications, such as eye tracking in virtual reality or augmented reality applications including head-mounted display devices. An eye tracking system may be provided that includes a plurality of components, each component including a light detector, such as a quadrant photodetector, and an imaging lens configured to produce an image of an eye on the quadrant photodetector. Machine learning or other techniques may be used to track or otherwise determine a user's gaze direction, which may be used by one or more components of the HMD device, thereby improving its functionality in various ways.

Description

Technical Field

[0001] The following disclosure generally relates to techniques for tracking, and in at least some implementations, to techniques for eye tracking for use in head mounted display devices. Background Art

[0002] A head mounted display (HMD) device or system is an electronic device that is worn on a user's head and, when so worn, fixes at least one electronic display within the visible area of ​​at least one of the user's eyes, regardless of the position or orientation of the user's head. HMD devices used to implement virtual reality systems typically completely surround the wearer's eyes and replace the actual scene (or actual reality) in front of the wearer with a "virtual" reality, while HMD devices used for augmented reality systems typically provide a translucent or transparent covering of one or more screens in front of the wearer's eyes, thereby augmenting the actual scene with additional information. For augmented reality systems, the "display" component of the HMD device is either transparent or at the periphery of the user's field of view so that it does not completely block the user from seeing their external environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1 is a schematic diagram of a networked environment that includes one or more systems suitable for performing at least some of the techniques described in this disclosure, including an embodiment of an eye tracking subsystem.

[0004] Figure 2 is a diagram illustrating an example environment in which at least some of the described techniques are used with an example head-mounted display device that is connected to a video rendering computing system and provides a virtual reality display to a user.

[0005] Figure 3 is a front schematic diagram of an HMD device having a binocular display subsystem.

[0006] Figure 4 A top plan view of an HMD device with a binocular display subsystem and various sensors is shown according to an exemplary embodiment of the present disclosure.

[0007] Figure 5 An example of using one or more light sources and light detectors to determine pupil position, such as for eye tracking in an HMD device according to the techniques of this disclosure, is shown.

[0008] Figure 6 is a perspective view of an exemplary angle-sensitive optical detector that may be used in one or more implementations of the present disclosure.

[0009] Figure 7 yes Figure 6A cross-sectional view of an angle-sensitive optical detector is shown.

[0010] Figure 8 yes Figure 6 A top view of an angle-sensitive optical detector is shown.

[0011] Fig. 9 is a simplified illustration of an imaging lens and optical detector of an object tracking system according to one non-limiting illustrated implementation.

[0012] Fig. 10A is a perspective view of an exemplary angle-sensitive optical detector and an object to be detected according to one non-limiting illustrated implementation, wherein the object is located at a first position.

[0013] Fig. 10B According to a non-limiting illustrated implementation Fig. 10A A stereoscopic view of an angle-sensitive optical detector and an object to be detected, wherein the object is located at a second position. DETAILED DESCRIPTION

[0014] In the following description, certain specific details are set forth to provide a thorough understanding of various disclosed implementations. However, one skilled in the relevant art will recognize that the implementations may be practiced without one or more of these specific details or with other methods, components, materials, etc. In other instances, well-known structures associated with computer systems, server computers, and / or communication networks are not shown or described in detail to avoid unnecessarily obscuring the description of the implementations.

[0015] Throughout the specification and the appended claims, the word "comprising" is used synonymously with "including" and is inclusive or open-ended (ie, does not exclude other unrecited elements or methodological acts) unless the context requires otherwise.

[0016] References throughout this specification to "an implementation" or "implementations" mean that a particular feature, structure, or characteristic described in conjunction with that implementation is included in at least one implementation. Thus, the phrases "in one implementation" or "in implementations" appearing in various places throughout this specification do not necessarily all refer to the same implementation. Furthermore, particular features, structures, or characteristics may be combined in any suitable manner in one or more implementations.

[0017] As used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It should also be noted that the term "or" is generally employed in its sense including "and / or" unless the context clearly dictates otherwise.

[0018] The titles and abstracts of the disclosure provided herein are for convenience only and do not interpret the scope or meaning of the implementations.

[0019] Eye tracking is the process by which the position, orientation, or motion of the eye can be measured, detected, sensed, determined, or monitored (collectively referred to as "measured") . In many applications, this is done to determine the direction of a user's gaze. The position, orientation, or motion of the eye can be measured in a variety of different ways, the least invasive of which can employ one or more optical detectors or sensors to optically track the eye. Some techniques may include illuminating or flooding the entire eye with infrared light at once and measuring the reflection with at least one optical sensor tuned to be sensitive to infrared light. Information about how the infrared light reflects from the eye is analyzed to determine the position, orientation, and / or motion of one or more eye features (e.g., the cornea, pupil, iris, or retinal blood vessels).

[0020] Eye tracking functionality is very advantageous in applications of wearable head mounted display systems. Some examples of the utility of eye tracking in head mounted display systems include: affecting the location of displayed content in the user's field of view; saving power, bandwidth, or computing resources by modifying the display of content outside the user's field of view (e.g., foveated rendering); affecting what content is displayed to the user; determining where the user is looking or gazing; determining whether the user is looking at displayed content on the display; providing methods by which the user can control or interact with displayed content; and other applications.

[0021] The present disclosure generally relates to techniques for object tracking, such as eye tracking or tracking of other objects. Such techniques may be used, for example, in a head mounted display ("HMD") device for VR or AR applications. Some or all of the techniques described herein may be performed via automatic operation of an embodiment of an eye tracking subsystem, such as by one or more configured hardware processors or other configured hardware circuits. The one or more hardware processors or other configured hardware circuits of such a system or device may include, for example, one or more GPUs ("graphics processing units") and / or CPUs ("central processing units") and / or other microcontrollers ("MCUs") and / or other integrated circuits, such as (multiple) hardware processors being part of an HMD device or other device that includes one or more display panels on which image data is to be displayed, or (multiple) hardware processors being part of a computing system that generates or otherwise prepares image data for display to be sent to (multiple) display panels, as discussed further below. More generally, such hardware processors or other configured hardware circuits may include, but are not limited to, one or more application specific integrated circuits (ASICs), standard integrated circuits, controllers (e.g., executing appropriate instructions and including microcontrollers and / or embedded controllers), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), digital signal processors (DSPs), programmable logic controllers (PLCs), etc. Elsewhere in this document (including with respect to the discussion below), Figure 1 ) including other details.

[0022] Technical benefits in at least some embodiments of the technology include addressing and mitigating increases in media transmission bandwidth for image encoding by, for example, reducing image data size, increasing the speed of controlling display panel pixels (e.g., based at least in part on the corresponding reduced image data size), improving foveated image systems that reflect subsets of display panels and / or images of particular interest, and other techniques. Foveated image encoding systems exploit specific aspects of the human visual system (which can provide detailed information only at and around focus), but typically use specialized computational processing to avoid visual artifacts to which peripheral vision is very sensitive (e.g., artifacts related to motion and contrast in video and image data). In the case of certain VR and AR displays, both bandwidth and computational usage for processing high-resolution media are exacerbated because certain display devices involve two separate display panels (i.e., one for each eye) with two separately addressable pixel arrays, each pixel array involving an appropriate resolution. Thus, the technology can be used, for example, to reduce transmission bandwidth for local and / or remote display of video frames or other images, while maintaining resolution and detail in the viewer's "region of interest" within the image, while also minimizing computational usage for processing such image data. Additionally, the use of lenses in head mounted display devices and in conjunction with other displays can provide greater focus or resolution on a subset of the display panel, such that using this technique to display lower resolution information in other portions of the display panel can provide further benefits when this technique is used in these embodiments.

[0023] For the purpose of illustration, some embodiments of obtaining and using specific types of information in a specific type of manner for specific types of structures and by using specific types of devices are described below. However, it will be understood that the techniques described in this way can be used in other embodiments in other ways, and therefore the present disclosure is not limited to the exemplary details provided. As a non-exclusive example, various embodiments discussed in this article include the use of images as video frames, however, although for convenience, many examples described in this article relate to "video frames", it will be understood that the techniques described with reference to these examples can be used for one or more images of various types, including non-exclusive examples of multiple continuous video frames (e.g., at 30, 60, 90, 180 or some other amount of frames per second), other video content, photos, computer-generated graphics content, other visual media articles or some combination thereof. In addition, for exemplary purposes, various details are provided in the drawings and text, but are not intended to limit the scope of the present disclosure. In addition, as used herein, "pixel" refers to the smallest addressable image element of a display, which can be activated to provide all possible color values ​​for the display. In many cases, a pixel includes separate individual sub-elements (in some cases separate "sub-pixels") for respectively producing red, green, and blue light for perception by a human viewer, with separate color channels being used to encode pixel values ​​for the differently colored sub-pixels. A pixel "value" as used herein refers to a data value corresponding to a respective excitation level of one or more of those respective RGB elements of a single pixel.

[0024] Figure 1 1 is a schematic diagram of a networked environment 100 including a local media rendering (LMR) system 110 (e.g., a gaming system) including a local computing system 120 and a display device 180 (e.g., an HMD device with two display panels) suitable for performing at least some of the techniques described herein. Figure 1 In the illustrated embodiment, the local computing system 120 is connected via a transmission link 115 (which may be wired or cabled (tethered), such as via Figure 2The local computing system 120 is communicatively connected to the display device 180 via one or more cables (cable 220) as shown, or alternatively may be wireless. In other embodiments, the local computing system 120 may provide encoded image data for display to a panel display device (e.g., a TV, console, or monitor) via a wired or wireless link, whether in addition to or in lieu of the HMD device 180, and the display devices each include one or more addressable pixel arrays. In various embodiments, the local computing system 120 may include a general-purpose computing system, a game console, a video stream processing device, a mobile computing device (e.g., a cell phone, a PDA, or other mobile device), a VR or AR processing device, or other computing system.

[0025] In the illustrated embodiment, the local computing system 120 has a component 125 including one or more hardware processors (e.g., a centralized processing unit or "CPU"), memory 130, various I / O ("input / output") hardware components 127 (e.g., a keyboard, a mouse, one or more game controllers, speakers, a microphone, an IR transmitter and / or receiver, etc.), a video subsystem 140 including one or more dedicated hardware processors (e.g., a graphics processing unit or "GPU") 144 and video memory (VRAM) 148, a computer readable memory 150, and a network connection 160. Also in the illustrated embodiment, an embodiment of the eye tracking subsystem 135 is executed in the memory 130 to perform at least some of the described techniques, such as by using the CPU 125 and / or GPU 144 to perform automated operations that implement those described techniques, and the memory 130 may also optionally execute one or more other programs 133 (e.g., to generate video or other images to be displayed, such as a game program). As part of the automated operation of implementing at least some of the techniques described herein, the eye tracking subsystem 135 and / or program 133 executing in memory 130 may store or retrieve various types of data (including in an example database data structure in memory 150), in which example the data used may include various types of image data information 154 in a database ("DB"), various types of application data 152 in the DB, various types of configuration data 157 in the DB, and may include additional information, such as system data or other information.

[0026] In the depicted embodiment, the LMR system 110 is also communicatively connected via one or more computer networks 101 and network links 102 to an exemplary network-accessible media content provider 190, which may further provide content to the LMR system 110 for display, either in addition to or in lieu of the image generation program 133. The media content provider 190 may include one or more computing systems (not shown), each of which may have components similar to those of the local computing system 120, including one or more hardware processors, I / O components, local storage, and memory, although for the sake of brevity, some details are not shown for the network-accessible media content providers.

[0027] will understand that, despite Figure 1 In the illustrated embodiment of the display device 180 is described as being distinguished and separated from the local computing system 120, but in some embodiments, some or all components of the local media rendering system 110 may be integrated or housed in a single device, such as a mobile gaming device, a portable VR entertainment system, an HMD device, etc. In such an embodiment, the transmission link 115 may, for example, include one or more system buses or video bus architectures.

[0028] As an example involving operations performed locally by the local media rendering system 120, assuming that the local computing system is a gaming computing system, the application data 152 includes one or more gaming applications executed via the CPU 125 using the memory 130, and various video frame display data is generated and / or processed by the image generation program 133, for example, in conjunction with the GPU 144 of the video subsystem 140. In order to provide a high-quality gaming experience, a large amount of video frame data (corresponding to a high image resolution for each video frame and a high "frame rate" of about 60-180 such video frames per second) is generated by the local computing system 120 and provided to the display device 180 via the wired or wireless transmission link 115.

[0029] It will also be understood that the computing system 120 and the display device 180 are only illustrative and are not intended to limit the scope of the present disclosure. The computing system 120 may alternatively include multiple interactive computing systems or devices, and may be connected to other devices not shown, including connecting through one or more networks such as the Internet, via the Web, or via a dedicated network (e.g., a mobile communication network, etc.). More generally, a computing system or other computing node may include any combination of hardware or software that can interact and perform the functions of the type described, including but not limited to desktops or other computers, gaming systems, database servers, network storage devices and other network devices, PDAs, cellular phones, wireless phones, pagers, electronic organizers, Internet appliances, television-based systems (e.g., using set-top boxes and / or personal / digital video recorders) and various other consumer products including appropriate communication capabilities. The display device 180 may similarly include one or more devices having one or more display panels of various types and forms, and optionally include various other hardware and / or software components.

[0030] In addition, the functionality provided by the eye tracking subsystem 135 may be distributed among one or more components in some embodiments, and in some embodiments, some functionality of the eye tracking subsystem 135 may not be provided and / or other additional functionality may be available. It will also be understood that although various items are shown as being stored in memory or on storage when in use, these items or portions thereof may be transferred between memory and other storage devices for purposes of memory management or data integrity. Thus, in some embodiments, some or all of the techniques described may be performed by hardware including one or more processors or other configured hardware circuits or memory or storage, such as when configured by one or more software programs (e.g., by the eye tracking subsystem 135 or its components) and / or by data structures (e.g., by executing software instructions of one or more software programs and / or by storing such software instructions and / or data structures). Some or all of the components, systems, and data structures may also be stored (e.g., as software instructions or structured data) on a non-transitory computer-readable storage medium such as a hard disk or flash drive or other non-volatile storage device, a volatile or non-volatile memory (e.g., RAM), a network storage device, or a portable media protocol, so as to be read by an appropriate drive (e.g., a DVD disk, a CD disk, an optical disk, etc.) or via an appropriate connection. Systems, components, and data structures may also be transmitted on various computer-readable transmission media (including wireless and wire / cable-based media) as generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) in some embodiments, and may take various forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). In other embodiments, such a computer program product may also take other forms. Therefore, the present invention may be practiced with other computer system configurations.

[0031] Figure 2An exemplary environment 200 is shown in which at least some of the described techniques are used with an exemplary HMD device 202, which is connected to a video rendering computing system 204 via a cable connection 220 (or a wireless connection in other embodiments) to provide a virtual reality display to a human user 206. The user wears the HMD device 202 and receives display information of a simulated environment different from the actual physical environment from the computing system 204 via the HMD device, where the computing system acts as an image rendering system that provides images of the simulated environment (e.g., images generated by a game program and / or other software program executed on the computing system) to the HMD device for display to the user. In this example, the user is also able to move within a tracking volume 201 of the actual physical environment 200, and may also have one or more I / O ("input / output") devices to allow the user to further interact with the simulated environment, in this example, the one or more I / O ("input / output") devices include handheld controllers 208 and 210.

[0032] In the illustrated example, the environment 200 may include one or more base stations 214 (two are shown, labeled as base stations 214a and 214b), which may help track the HMD device 202 or the controllers 208 and 210. As the user moves the position of the HMD device 202 or changes the orientation of the HMD device 202, the position of the HMD device is tracked to allow the corresponding portion of the simulated environment to be displayed to the user on the HMD device, and the controllers 208 and 210 may also employ similar techniques to track the position of the controller (and optionally use this information to help determine or verify the position of the HMD device). After the tracked position of the HMD device 202 is known, the corresponding information is sent to the computing system 204 via the cable 220 or wirelessly, and the computing system 204 uses the tracked position information to generate one or more next images of the simulated environment for display to the user.

[0033] A variety of different position tracking methods may be used in various implementations of the present disclosure, including but not limited to acoustic tracking, inertial tracking, magnetic tracking, optical tracking, combinations thereof, and the like.

[0034] In at least some implementations, the HMD device 202 may include one or more optical receivers or sensors that may be used to implement tracking functionality or other aspects of the present disclosure. For example, the base stations 214 may each sweep an optical signal across the tracking volume 201. Each base station 214 may generate more than one optical signal, as required by each specific implementation. For example, while a single base station 214 is typically sufficient for six-degree-of-freedom tracking, in some implementations, multiple base stations (e.g., base stations 214a, 214b) may be required or desired to provide robust room-scale tracking for the HMD device and peripheral devices. In this example, the optical receiver is incorporated into the HMD device 202 and / or other tracked objects such as controllers 208 and 210. In at least some implementations, the optical receiver may be paired with an accelerometer and gyroscope inertial measurement unit ("IMU") on each tracked device to support low-latency sensor fusion.

[0035] In at least some embodiments, each base station 214 includes two rotors that sweep a linear beam across the tracking volume 201 on orthogonal axes. At the beginning of each scanning cycle, the base station 214 can transmit an omnidirectional light pulse (referred to as a "sync signal") visible to all sensors on the tracked object. Thus, each sensor calculates a unique angular position in the swept volume by timing the duration between the sync signal and the beam signal. Sensor distance and orientation can be resolved using multiple sensors fixed to a single rigid body.

[0036] One or more sensors positioned on the tracked object (e.g., HMD device 202, controllers 208, and 210) may include an optoelectronic device capable of detecting modulated light from the rotor. For visible light or near infrared (NIR) light, a silicon photodiode and appropriate amplifier / detector circuitry may be used. Because environment 200 may contain static and time-varying signals (optical noise) having wavelengths similar to those of base station 214 signals, in at least some implementations, the base station light may be modulated to make it easily distinguishable from any interfering signals and / or to cause the sensor to filter out radiation of any wavelength other than the base station signal.

[0037] Inside-out tracking is also a type of positional tracking that can be used to track the position of the HMD device 202 and / or other objects (e.g., controllers 208 and 210, tablet computers, smart phones). Inside-out tracking differs from outside-in tracking in the location of the camera or other sensor used to determine the position of the HMD. For inside-out tracking, the camera or sensor is located on the HMD or the object being tracked, while in outside-out tracking, the camera or sensor is placed in a fixed position in the environment.

[0038] HMDs that utilize inside-out tracking utilize one or more cameras to “look outside” to determine how their position has changed relative to the environment. As the HMD moves, the sensors readjust their position in the room, and the virtual environment responds accordingly in real time. This type of positional tracking can be implemented with or without markers placed in the environment. Cameras placed on the HMD observe features of the surrounding environment. When markers are used, the markers are designed to be easily detected by the tracking system and are placed in specific areas. With “markerless” inside-out tracking, the HMD system uses unique characteristics that initially exist in the environment (e.g., natural features) to determine position and orientation. The HMD system’s algorithms recognize specific images or shapes and use them to calculate the device’s position in space. Data from accelerometers and gyroscopes can also be used to increase the accuracy of positional tracking.

[0039] Figure 3 Information 300 is shown showing a front view of an exemplary HMD device 344 when worn on the head of a user 342. The HMD device 344 includes a front-facing structure 343 supporting a front-facing or front camera 346 and a plurality of sensors 348a-348d (collectively, 348) of one or more types. As an example, some or all of the sensors 348 can help determine the position and orientation of the device 344 in space, such as a light sensor for detecting and using signals from one or more external devices (not shown, for example, Figure 2 344). As shown, the front camera 346 and sensor 348 point forward to the actual scene or environment (not shown) in which the user 342 operates the HMD device 344. The actual physical environment may include, for example, one or more objects (e.g., walls, ceilings, furniture, stairs, cars, trees, tracking markers, or any other type of object). The specific number of sensors 348 may be less than or more than the number of sensors shown. The HMD device 344 may also include one or more additional components that are not attached to the forward structure (e.g., inside the HMD device), such as an IMU (inertial measurement unit) 347 electronic device that measures and reports (e.g., using a combination of accelerometers and gyroscopes, and optionally using a magnetometer) specific forces, angular rates, and / or magnetic fields around the HMD device 344. The HMD device may also include additional components not shown, including one or more display panels and optical lens systems that are oriented toward the user's eyes (not shown) and optionally have one or more attached internal motors to change the alignment or other positioning of the one or more optical lens systems and / or display panels within the HMD device, as described below with respect to Figure 4 discussed in more detail.

[0040] The example of the HMD device 344 shown is supported on the head of the user 342 based at least in part on one or more straps 345 that are attached to the housing of the HMD device 344 and extend completely or partially around the user's head. Although not shown here, the HMD device 344 may also have one or more external motors, for example, attached to one or more straps 345, and the automatic corrective action may include using such motors to adjust such straps to modify the alignment or other positioning of the HMD device on the user's head. It will be understood that the HMD device may include other support structures (e.g., nose bridge, chin strap, etc.) not shown here, either in addition to or in place of the straps shown, and some embodiments may include motors attached to one or more such other support structures to similarly adjust their shape and / or position to modify the alignment or other positioning of the HMD device on the user's head. Other display devices that are not secured to the user's head may similarly be attached to or be part of one or more structures that affect the positioning of the display device, and may, in at least some embodiments, include motors or other mechanical actuators to similarly modify their shape and / or position to modify the alignment or other positioning of the display device relative to one or more pupils of one or more users of the display device.

[0041] Figure 4 A simplified top plan view 400 of an HMD device 405 is shown that includes a pair of near-eye display systems 402 and 404. The HMD device 405 may be, for example, Figures 1 to 3 The same or similar HMD device or a different HMD device as shown in , and the HMD devices discussed herein can be further used in the examples discussed further below. Figure 4 The near-eye display systems 402 and 404 of the embodiment include display panels 406 and 408 (e.g., OLED microdisplays) and respective optical lens systems 410 and 412, each having one or more optical lenses. The display systems 402 and 404 can be mounted to or otherwise positioned within a housing (or frame or support structure) 414, which includes a forward portion 416 (e.g., with Figure 3 343 of the front surface 343 of the same or similar), left temple 418, right temple 420 and inner surface 421, when the user wears the HMD device, the inner surface 421 contacts or is close to the face of the wearer user 424. The two display systems 402 and 404 can be fixed to the housing 414 in a glasses arrangement, which can be worn on the head 422 of the wearer user 424, and the left temple 418 and the right temple 420 rest on the user's ears 426 and 428 respectively, and the nose assembly 492 can rest on the user's nose 430. Figure 4In the example of FIG. 4 , the HMD device 405 can be partially or completely supported on the user's head by the nose display and / or the left and right ear glasses arms, but in some embodiments, a strap (not shown) or other structure can be used to fix the HMD device to the user's head, for example Figure 2 and Figure 3 . The shape and size of the housing 414 can be designed to position each of the two optical lens systems 410 and 412 in front of one of the user's eyes 432 and 434, respectively, so that the target position of each pupil 494 is vertically and horizontally centered in front of the corresponding optical lens system and / or display panel. Although the housing 414 is shown in a simplified manner similar to glasses for illustrative purposes, it should be understood that more complex structures (e.g., goggles, integrated headbands, helmets, straps, etc.) can actually be used to support and position the display systems 402 and 404 on the head 422 of the user 424.

[0042] Figure 4 The HMD device 405 and other HMD devices discussed herein are capable of presenting a virtual reality display to a user, for example, via a corresponding video presented at a display rate of, for example, 30 or 60 or 90 frames (or images) per second, while other embodiments of similar systems may present an augmented reality display to a user. Figure 4 Each of the displays 406 and 408 can generate light that is transmitted and focused by the corresponding optical lens systems 410 and 412 onto the eyes 432 and 434 of the user 424, respectively. The pupil 494 aperture of each eye through which light enters the eye typically has a pupil size of 2 mm (millimeters) in diameter under very bright conditions to as much as 8 mm in dark conditions, while the larger iris containing the pupil can have a size of about 12 mm. The pupil (and the iris surrounding the pupil) can also typically move a few millimeters horizontally and / or vertically within the visible part of the eye with the eyelid open, which will also cause the pupil to move to different depths from the optical lenses or other physical elements of the display (creating a three-dimensional volume in which the pupil can move) at different horizontal and vertical positions as the eyeball rotates about its center. The light entering the user's pupil is viewed by the user 424 as an image or video. In some implementations, the distance between each of the optical lens systems 410 and 412 and the user's eyes 432 and 434 can be relatively short (e.g., less than 30 mm, less than 20 mm), which can help make the HMD device appear brighter to the user because the weight of the optical lens system and the display system is relatively close to the user's face, and this can also provide a larger field of view to the user. Although not shown here, some embodiments of such an HMD device can include various additional internal or external sensors.

[0043] In the illustrated embodiment, Figure 4 The HMD device 405 also includes hardware sensors and additional components, such as one or more accelerometers and / or gyroscopes 490 (e.g., as part of one or more IMU units). As discussed in more detail elsewhere herein, the values ​​from (multiple) accelerometers and / or gyroscopes can be used to locally determine the orientation of the HMD device. In addition, the HMD device 405 may include one or more forward-facing cameras, such as a camera 485 on the outside of the front 416, and its information can be used as part of the operation of the HMD device, such as for providing AR functions or positioning functions. In addition, the HMD device 405 may also include other components 475 (e.g., electronic circuits for controlling the display of images on the display panels 406 and 408, internal memory, one or more batteries, position tracking devices that interact with external base stations, etc.), as discussed in more detail elsewhere herein. Other embodiments may not include one or more of components 475, 485, and / or 490. Although not shown here, some embodiments of such HMD devices may include various additional internal and / or external sensors to track various other types of movements and positions of the user's body, eyes, controllers, etc.

[0044] In the illustrated embodiment, Figure 4 The HMD device 405 of the embodiment also includes hardware sensors and additional components that can be used by the disclosed embodiments as part of the described techniques for determining the user's pupils or gaze direction, which can be provided to one or more components associated with the HMD device for use, as discussed elsewhere herein. The hardware sensors in this example include one or more eye tracking components 472 of the eye tracking subsystem, which are mounted on or near the display panels 406 and 408 and / or on the inner surface 421 located near the optical lens systems 410 and 412, for obtaining information about the actual position of the user's pupils 494, such as for each pupil separately in this example.

[0045] Each of the eye tracking components 472 may include one or more photodetectors (e.g., silicon photodiodes, quadrant photodetectors with four active areas) and optionally one or more light sources (e.g., IR LEDs). Figure 44, but it should be understood that a different number of eye tracking components (e.g., one, three, six) may actually be provided. In some embodiments, a total of eight eye tracking components 472 are provided, four for each eye of user 424. In addition, in at least some implementations, each eye tracking component 472 includes a light source directed toward one of eyes 432 and 434 of user 424 and a light detector positioned to receive light reflected by the user's corresponding eye.

[0046] As discussed in greater detail elsewhere herein, information from eye tracking assembly 472 can be used to determine and track a user's gaze direction during use of HMD device 405. Additionally, in at least some embodiments, HMD device 405 can include one or more internal motors 438 (or other movement mechanisms) that can be used to move 439 the alignment and / or other positioning of one or more optical lens systems 410 and 412 and / or display panels 406 and 408 within the housing of HMD device 405 (e.g., in a vertical direction, a horizontal left-right direction, and / or a horizontal front-to-back direction) to personalize or otherwise adjust a target pupil position for one or both of near-eye display systems 402 and 404 to correspond to an actual position of one or both of pupils 494. Such motors 438 can be controlled, for example, by user manipulation of one or more controls 437 on housing 414 and / or via user manipulation of one or more associated separate I / O controllers (not shown). In other embodiments, HMD device 405 may control the alignment and / or other positioning of optical lens systems 410 and 412 and / or display panels 406 and 408 without requiring such motor 438, for example by using an adjustable positioning mechanism (e.g., a screw, a slider, a ratchet, etc.) that is manually changed by a user using controller 437. Figure 4 Motor 438 is shown for only one of the near-eye display systems, but in some embodiments, each near-eye display system can have its own one or more motors, and in some embodiments, one or more motors can be used to (e.g., independently) control each of multiple near-eye display systems.

[0047] While the technology can be used in some embodiments with a display system similar to the display system shown, other types of display systems can also be used in other embodiments, including use with a single optical lens and display device, or use with multiple such optical lenses and display devices. Non-exclusive examples of other such devices include cameras, telescopes, microscopes, binoculars, sights, measuring mirrors, etc. In addition, the technology can be used with various display panels or other display devices that emit light to form an image, and one or more users view the image through one or more optical lenses. In other embodiments, a user can view one or more images through one or more optical lenses that are generated in a manner other than via a display panel, such as one or more images on a surface that partially or completely reflects light from another light source.

[0048] Figure 5 An example of using multiple eye tracking components, each including a light source and a light detector, is shown to determine the user's gaze position in a specific manner in a specific implementation according to the described technology. Specifically, Figure 5 Information 500 is included to illustrate the operation of an exemplary display panel 510 and associated optical lens 508 in providing image information to a user's eye 504, such as focusing the information on the eye's pupil 506. In the illustrated embodiment, four eye tracking components 511a-511d (collectively 511) of the eye tracking subsystem are mounted near the edge of the optical lens 508, and each of the eye tracking components 511a-511d is generally oriented toward the pupil 506 of the eye 504 for emitting light toward the eye 504 and capturing light reflected from the pupil 506 or the iris 502 that partially or entirely surrounds the pupil 506.

[0049] In the example shown, each of the eye tracking components 511 includes a light source 512 and a light detector 514, although in other implementations the light detector 514 can be independent of the light source (e.g., four light detectors and one light source, or two light detectors and one light source, etc.). In this example, the eye tracking components 511 are placed at locations including near the top of the optical lens 508 along the central vertical axis, near the bottom of the optical lens along the central vertical axis, near the left side of the optical lens along the central horizontal axis, and near the right side of the display panel along the central horizontal axis. In other embodiments, the eye tracking components 511 can be positioned at other locations, and fewer or more eye tracking components can be used.

[0050] It will also be understood that the light sources and light detectors are shown for exemplary purposes only, and other embodiments may include more or fewer light sources or detectors, and the light sources or detectors may be located at other locations. In addition, although not shown here, in some embodiments, additional hardware components may be used to help obtain data from one or more light detectors. For example, an HMD device or other display device may include various light sources (e.g., infrared, visible light, etc.) at different locations to illuminate the iris and pupil and reflect back to one or more light detectors, such as including a light source mounted on or near the display panel 510 or alternatively elsewhere (e.g., between the optical lens 508 and the eye 504, such as on an unshown inner surface of the HMD device including the display panel 510 and the optical lens 508). In some such embodiments, light from such an illumination source may be further reflected from the display panel before passing through the optical lens 508 to illuminate the iris and pupil.

[0051] Figure 6 is a perspective view 600 of an exemplary angle-sensitive optical detector 603 that may be used in one or more implementations of the present disclosure. Figure 7 yes Figure 6 700 of a cross-sectional view of the angle sensitive optical detector 603 shown in FIG. Figure 8 800 is a top view of an angle sensitive optical detector. In this example, the optical detector 603 includes a quadrant photodetector 604 disposed on a common substrate 602. The quadrant photodetector 604 includes four active regions or cells 604a-604d separated by a small gap. It should be understood that other types of angle sensitive detectors may also be used, such as position sensitive detectors, photodiode detectors with fewer or more cells, etc.

[0052] The angle-sensitive detector 603 includes an opaque screen or cover 606 positioned above or in front of the photodetector 604 at a distance 612. The opaque screen 606 can be coupled to the substrate 602 or integrated with the substrate 602, such as as part of a housing. The opaque screen 606 has a hole 608 therein that allows light 616 from an object 614 to pass through. In this example, the object 614 is an eye of a user (including a dark pupil 615 of the eye), however the embodiments described herein can be used to track other objects (e.g., controllers, headphones, other objects). The optical detector 603 also includes an imaging lens 610 positioned within or near the hole 608 of the opaque screen 606. Advantageously, the imaging lens 610 is configured to focus an image 618 of the pupil 615 of the user's eye 614 onto the quadrant photodetector. That is, the imaging lens 610 can be designed to have an object plane that is substantially coplanar with the expected position of the pupil 615 of the user's eye 614 relative to the imaging lens 610 and an image plane that is substantially coplanar with the quadrant photodetector 604. The imaging lens 610 may include a single lens, or may include multiple lenses. In addition, the imaging lens 610 may include one or more optical components, such as one or more filters, coatings, etc.

[0053] In the non-limiting illustrated example, the active area (e.g., anode) of each element 604a-604d of the photodetector is individually usable so that light illuminating a single quadrant can be electrically characterized as being only in that quadrant. As light passes through the angle-sensitive detector 603, the light energy is distributed between adjacent elements 604a-604d, and the difference in the electrical contribution to each element defines the relative position of the light. The relative intensity distribution on the elements 604a-604d can be used to determine the position of the light applied to the unit.

[0054] As shown, light 616 reflected from the user's eye 614 through the lens 610 forms an image 618 of the eye 614, which includes a dark spot surrounded by a brighter area due to the dark pupil 615. This is because the dark pupil 615 reflects relatively less light than the light reflected by the eye 614 and the portion of the user's face surrounding the pupil. Since the dark pupil 615 will affect the intensity of the light on the unit, the image 618 (including the dark pupil) formed on the quadrant photodetector 604 can be electrically characterized to determine the position of the pupil 615 relative to the angle-sensitive detector 603. As discussed elsewhere herein, the position information can be used for gaze tracking and various other technologies that can be implemented using pupil position information. As discussed below, the systems and methods of the present disclosure can utilize multiple light sources and angle-sensitive detectors to determine the position of the user's eyes or the position of other components (e.g., components of an HMD system).

[0055] Fig. 99 is a simplified view 900 of an imaging lens 610 and an optical detector 603 of an object tracking system according to one non-limiting illustrated implementation. As shown, the imaging lens 610 is configured to focus an image 906 of an object 904 to be tracked onto the quadrant photodetector 604. That is, the imaging lens 610 can be designed to have an object plane or space that is substantially coplanar with the expected position of the object 904 to be tracked relative to the imaging lens 610 and an image plane or space that is substantially coplanar with the quadrant photodetector 604. In this manner, the units 604a-604d can be used to determine the position of the object 904 in the object space so that the object can be tracked, as discussed above. As described above, the imaging lens 610 can include a single lens, can include multiple lenses, and can include one or more other types of optical components (e.g., one or more filters, coatings, etc.).

[0056] Fig. 10A is a perspective view 1000 of an angle-sensitive optical detector 603 and an object 1002 to be detected, wherein the object is located in a first position. The object 1002 may include a defined pattern 1004 thereon. Fig. 10B yes Fig. 10A 603 and an object 1002 to be detected, wherein the object 1002 is located in a second position. As shown, the imaging lens 610 receives light 1006 reflected from the object 1002 and produces an image 1008 of the pattern 1004 on the photodetector units 604a-604b. Fig. 10A The first position shown in the Fig. 10B , the image 1008 of the pattern 1004 also moves its position accordingly. The control circuitry may be used to process the detector data received from the photodetector units 604a-604b to track the position of the object 1002 in space along one or more dimensions.

[0057] In the example shown, pattern 1004 can be designed to have alternating light and dark portions (e.g., a series of spaced apart dark bars), which can help provide more discrete signals at photodetector cells 604a-606b. In at least some implementations, providing discrete signals or steps can improve the ability of optical detector 603 to track object 1002 as the object moves in space along one or more dimensions.

[0058] In at least some implementations, according to a non-limiting illustrated implementation, machine learning techniques may be used to implement an eye tracking subsystem of an HMD device, such as the eye tracking subsystem discussed herein. For example, a model training portion and an inference portion of a machine learning system may be provided. In the training portion, training data is fed into a machine learning algorithm to generate a trained machine learning model. The training data may include, for example, labeled data from a light detector specifying a gaze position. As a non-limiting example, in an embodiment including four light detectors directed at the user's eyes, each training sample may include outputs from each of the four light detectors and a known or inferred gaze direction. In at least some implementations, the user's gaze direction may be learned or inferred by directing the user to gaze at a specific user interface element (e.g., a word, a dot, an "X", another shape or object, etc.) on the display of the HMD device, which may be static or may move on the display. The training data may also include samples in which one or more light sources or light detectors are blocked, for example, due to a user blinking, eyelashes, glasses, a hat, or other obstructions. For such training samples, the label may be "unknown" or "blocked" instead of specifying a gaze direction.

[0059] The training data may be obtained from multiple users and / or from a single user of the HMD system. The training data may be obtained in a controlled environment and / or during actual use by a user ("field training"). Furthermore, in at least some implementations, the model may be updated or calibrated from time to time (e.g., periodically, continuously, after certain events) to provide accurate gaze direction predictions.

[0060] In the inference portion, the runtime data may be provided as input to a trained machine learning model that generates a gaze direction prediction. Continuing with the example above, the output data of the light detector may be provided as input to a trained machine learning model that may process the data to predict the gaze position. The gaze direction prediction may then be provided to one or more components associated with the HMD device, such as, for example, one or more VR or AR applications executed on the HMD device, one or more display or rendering modules, one or more mechanical controllers, one or more position tracking subsystems, and the like.

[0061] Machine learning techniques for implementing the features discussed herein may include any type of appropriate structure or technique. As non-limiting examples, machine learning models may include one or more of a decision tree, a statistical hierarchical model, a support vector machine, an artificial neural network (ANN) such as a convolutional neural network (CNN) or a recursive neural network (RNN) (e.g., a long short-term memory (LSTM) network), a mixture density network (MDN), a hidden Markov model, etc., or other models may be used. In at least some implementations (e.g., implementations utilizing RNNs), the machine learning model may utilize past input (memory, feedback) information to predict gaze direction. This implementation may advantageously utilize sequence data to determine motion information or prior gaze direction predictions, which may provide more accurate real-time gaze direction predictions.

[0062] As another example, in at least some implementations, the HMD device may include an interpupillary distance (IPD) adjustment component that can be used to automatically adjust one or more components of the HMD to account for the variable IPD. In this example, the IPD adjustment component can receive a gaze direction, and can align at least one component of the HMD system for the user based at least in part on the determined gaze direction. For example, when the user is looking at a nearby object, the IPD can be relatively short, and the IPD adjustment can align one or more components of the HMD device accordingly. As yet another non-limiting example, the HMD device can automatically adjust the focus of the lens based on the determined gaze direction.

[0063] Although the above examples use machine learning techniques to determine the gaze direction from the light detection information, it should be understood that the features of the present disclosure are not limited to the use of machine learning techniques. In general, any type of prediction model or function can be used. For example, in at least some implementations, the system can do the opposite of going directly from the light detection information to the gaze direction—predicting the light detection information given the input gaze direction. This method can find a prediction function or model that performs the prediction, mapping from the gaze direction to the predicted light reading in that direction. The function can be user-specific, so the system can implement a calibration process to find or customize the function. Once the prediction function has been determined or generated, the prediction function can then be inverted (for example, using a numerical solver) to produce real-time predictions during use. Specifically, given a sample of light detection information from a real sensor, the solver can be used to find the gaze direction that minimizes the error between the real reading and the predicted reading from the generated prediction function. In this implementation, the output is the solved gaze direction plus the residual, which can be used to judge the quality of the solution. In at least some implementations, some additional corrections can be applied to handle various problems, such as the HMD system sliding on the user's face during operation. The prediction function can be any type of function. As an example, given a data set of points captured from a user, a set of 2D polynomials can be used to map gaze angles to photodiode readings. In at least some other implementations, lookup tables or other approaches can also be used, including adapting the ML system to output predictions as described above.

[0064] It will be understood that in some embodiments, the functions provided by the above routines can be provided in an alternative manner, such as split into more routines, or merged into fewer routines. Similarly, in some embodiments, such as when other illustrated routines lack or include such functions respectively, or when the amount of functions provided changes, the illustrated routines can provide more or less functions than described. In addition, although various operations can be shown as being performed in a particular manner (e.g., serial or parallel) and / or in a particular order, it will be understood by those skilled in the art that in other embodiments, these operations can be performed in other orders and in other ways. Similarly, it will be understood that the above data structure can be constructed in different ways, including being constructed in different ways for a database or user interface screen / page or other types of data structures, such as splitting a single data structure into multiple data structures or merging multiple data structures into a single data structure. Similarly, in some embodiments, such as when other illustrated data structures lack or include such information respectively, or when the amount or type of stored information changes, the illustrated data structure can store more or less information than described.

[0065] In addition, the sizes and relative positions of the elements in the drawings (including the shapes and angles of various elements) are not necessarily drawn to scale, some elements are enlarged and positioned to improve the readability of the drawings, and the specific shapes of at least some elements are selected for ease of identification without conveying information about the actual shapes or proportions of those elements. In addition, some elements may be omitted for clarity and emphasis. In addition, reference numerals repeated in different drawings may represent the same or similar elements.

[0066] It will be appreciated from the foregoing that, although specific embodiments are described herein for purposes of illustration, various modifications may be made without departing from the spirit and scope of the invention. In addition, although certain aspects of the invention are sometimes presented in certain claim forms, or sometimes may not be embodied in any claim, the inventors contemplate various aspects of the invention in any available claim form. For example, although only some aspects of the invention may be described as being implemented in a computer-readable medium at a particular time, other aspects may also be implemented in the same manner.

Claims

1. An eye tracking system, comprising: a support structure positionable on a user's head; as well as a plurality of optical detectors carried by the support structure, each of the plurality of optical detectors comprising: A quadrant photodetector comprising four optically active regions; an opaque screen spaced apart from and positioned in front of the quadrant photodetector, the opaque screen including an aperture therein; and An imaging lens is positioned adjacent the aperture in the opaque screen, the imaging lens being configured to focus an image of the user's eye onto the quadrant photodetector.

2. The eye tracking system according to claim 1, further comprising: The control circuit is configured to: receiving detector data from each of the plurality of optical detectors; processing the detector data received from the plurality of optical detectors; as well as The position of the eyes of the user is tracked based at least in part on the processing of the received detector data.

3. The eye tracking system of claim 1 , further comprising: A light source is carried by the support structure, the light source comprising a light emitting diode that emits light having a wavelength between 780 nm and 1000 nm.

4. The eye tracking system of claim 3, further comprising: The control circuit is configured to: causing the light source to emit light; receiving detector data from each of the plurality of optical detectors; processing the detector data received from the plurality of optical detectors; as well as The position of the eyes of the user is tracked based at least in part on the processing of the received detector data.

5. The eye tracking system according to claim 1, wherein: For each of the plurality of optical detectors, the imaging lens is designed to have: an object plane substantially coplanar with an expected position of a pupil of the eye of the user relative to the imaging lens; and An image plane is substantially coplanar with the quadrant photodetector.

6. The eye tracking system of claim 1, wherein: The imaging lens includes at least two lenses.

7. A head mounted display system, comprising: a support structure positionable on a user's head; a display, carried by the support structure; a light source carried by the support structure, the light source being configured to emit light toward an eye of the user; a plurality of optical detectors carried by the support structure, each of the plurality of optical detectors comprising: A quadrant photodetector comprising four optically active regions; an opaque screen spaced apart from and positioned in front of the quadrant photodetector, the opaque screen including an aperture therein; and an imaging lens positioned proximate the aperture of the opaque screen, the imaging lens configured to focus an image of the eye of the user onto the quadrant photodetector; and The control circuit is configured to: causing the light source to emit light; receiving detector data from each of the plurality of optical detectors; processing the detector data received from the plurality of optical detectors; and The position of the eyes of the user is tracked based at least in part on the processing of the received detector data.

8. The head mounted display system according to claim 7, wherein: The light source comprises a light emitting diode emitting light having a wavelength between 780 nm and 1000 nm.

9. The head mounted display system according to claim 7, wherein: For each of the plurality of optical detectors, the imaging lens is designed to have: an object plane substantially coplanar with an expected position of a pupil of the eye of the user relative to the imaging lens; and An image plane is substantially coplanar with the quadrant photodetector.

10. The head mounted display system according to claim 7, wherein: The imaging lens includes at least two lenses.

11. An optical detector comprising: A quadrant photodetector comprising four optically active regions; an opaque screen spaced apart from and positioned in front of the quadrant photodetector, the opaque screen including an aperture therein; as well as An imaging lens is positioned adjacent the aperture in the opaque screen, the imaging lens being configured to focus an image of an object to be tracked onto the quadrant photodetector.

12. The optical detector according to claim 11, wherein The imaging lens includes at least two lenses.

13. A method for tracking an object, the method comprising: A plurality of optical detectors are provided, each of the plurality of optical detectors comprising: A quadrant photodetector comprising four optically active regions; an opaque screen spaced apart from and positioned in front of the quadrant photodetector, the opaque screen including an aperture therein; and an imaging lens positioned proximate the aperture of the opaque screen, the imaging lens configured to focus an image of an object to be tracked onto the quadrant photodetector; receiving detector data from each of the plurality of optical detectors; processing the detector data received from the plurality of optical detectors; and A position of the object is tracked based at least in part on the processing of the received detector data.

14. The method according to claim 13, further comprising: The light source is caused to emit light having a wavelength between 780 nm and 1000 nm.

15. The method according to claim 13, wherein: For each of the plurality of optical detectors, the imaging lens is designed to have: an object plane substantially coplanar with an expected position of a pupil of the user's eye relative to the imaging lens; and An image plane is substantially coplanar with the quadrant photodetector.

16. The method according to claim 13, wherein: The imaging lens includes at least two lenses.

17. The method according to claim 13, wherein: The object comprises a human eye.

Images