Position tracking system and method for a head-mounted display system

Abstract

Systems and methods for tracking the position of one or more head-mounted display (HMD) system components. An HMD component can carry a plurality of angle-sensitive detectors or other types of detectors. An HMD system can be used to detect a damaged position tracking sample to allow such a sample to be ignored, thereby improving the position tracking process. Control circuitry causes a light source to emit light according to a specified pattern and receives sensor data from a plurality of detectors. The control circuitry can process the sensor data, e.g., using machine learning or other techniques, to track the position of one or more HMD components.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to position tracking for objects, such as head-mounted display systems and controllers associated with head-mounted display systems. BACKGROUND

[0002] Current generation virtual reality (“VR”) or augmented reality (“AR”) experiences are created using head-mounted displays (“HMDs”), which can be coupled to a stationary computer such as a personal computer (“PC”), a laptop, or a game console, combined and / or integrated with a smartphone and / or its associated display, or a self-contained display. Generally, an HMD is a display device that is worn on the head of a user, with a small display device in front of one eye (monocular HMD) or each eye (binocular HMD). The display unit is typically miniaturized and can include CRT, LCD, liquid crystal on silicon (LCos), or OLED technology, for example. Binocular HMDs have the potential to display different images to each eye. This capability is used to display stereoscopic images.

[0003] With the development of smartphones, high-definition televisions, and other electronic devices, there is an increased demand for displays with enhanced performance. The increasing popularity of virtual reality and augmented reality systems, particularly those using HMDs, further increases this demand. Virtual reality systems typically completely surround the eyes of the wearer and replace the actual or physical view in front of the wearer (or actual reality) with a “virtual” reality, while augmented reality systems typically provide a semi-transparent or transparent overlay of one or more screens in front of the eyes of the wearer, such that the actual view is augmented with additional information, and mediated reality systems can similarly present information to a viewer that combines real-world elements with virtual elements. In many virtual reality and augmented reality systems, the motion of the wearer of such a head-mounted display can be tracked in various ways, such as via sensors in the head-mounted display, controllers, or external sensors, in order to enable the displayed images to reflect the motion of the user and allow for an interactive environment.

[0004] Position tracking allows an HMD system to estimate the position of one or more components relative to each other and the surrounding environment. Position tracking can utilize a combination of hardware and software to enable detection of the absolute position of components of an HMD system. Position tracking is an important technology for AR or VR systems, enabling the tracking of the motion of an HMD (and / or controllers or other peripherals) in six degrees of freedom (6DOF).

[0005] Position tracking techniques can be used to change the user's point of view to reflect different actions, such as jumping or crouching, and can allow precise representation of the user's hands and other objects in the virtual environment. Position tracking can also increase the connection between the physical and virtual environments by, for example, using hand positions to move virtual objects by touch. Position tracking improves the user's 3D perception of the virtual environment due to parallax, which helps perception of distance. In addition, position tracking can help minimize motion sickness caused by disjunction between what the eyes see and what the user's vestibular system in the ears feels.

[0006] There are different position tracking methods. These methods can include acoustic tracking, inertial tracking, magnetic tracking, optical tracking, combinations thereof, and the like. BRIEF DESCRIPTION OF DRAWINGS

[0007] In the drawings, like reference numerals refer to like elements or acts throughout. The sizes and relative positions of elements in the drawings are not necessarily drawn to scale. For example, the shapes of various elements and angles are not drawn to scale, and some of these elements are arbitrarily enlarged and positioned to improve drawing legibility. Further, the particular shapes of the elements as drawn are not intended to convey any information regarding the actual shape of the particular elements, and can have been selected solely for ease of recognition in the drawings.

[0008] Figure 1 FIG. 1 is a schematic diagram of a networked environment including one or more systems suitable for performing at least some of the techniques described in this disclosure, including embodiments of a tracking subsystem.

[0009] Figure 2 FIG. 2 is a diagram showing an example environment in which at least some of the described techniques are used with an example head-mounted display device that is tethered to a video rendering computing system and provides a virtual reality display to a user.

[0010] Figure 3 FIG. 3 is an illustrative diagram of an HMD device with binocular display subsystems and multiple angle-sensitive detectors.

[0011] Figure 4 FIG. 4 is an illustrative diagram of a controller that can be used with an HMD device.

[0012] Figure 5 FIG. 5 is a schematic block diagram of an HMD device according to example embodiments of the present disclosure.

[0013] Figure 6 FIG. 6 is a schematic diagram of an environment according to one non-limiting illustration of implementation in which machine learning techniques can be used to implement a tracking subsystem of an HMD device.

[0014] Figure 7is a flowchart of a method of operating a position tracking system of an HMD system to track the position, orientation, and / or movement of components of the HMD system during use, in accordance with example implementations of the present disclosure.

[0015] Figure 8 is a perspective view of an example angle-sensitive detector that can be used in one or more implementations of the present disclosure.

[0016] Figure 9 is a first linear polarizer, a spatially varying polarizer, and a second linear polarizer of an angle-sensitive photodiode structure, and the polarization of light or light points reaching the photodiode through it.

[0017] Figure 10 is a first linear polarizer, a spatially varying polarizer, and a second linear polarizer of an angle-sensitive photodiode structure, and the polarization of light or light points reaching the photodiode through it.

[0018] Figure 11A is a top view of an example angle-sensitive detector that can be used in one or more implementations of the present disclosure.

[0019] Figure 11B is Figure 11A is a perspective view of an angle-sensitive detector.

[0020] Figure 12 is a simplified diagram showing the use of a light source and an angle-sensitive detector to determine the position of a component of an HMD system, according to one non-limiting illustration.

[0021] Figure 13 is a diagram depicting an example optical system of a light source and an angle-sensitive detector, according to one non-limiting illustration.

[0022] Figure 14 is a diagram depicting the operation of an example scatter detection module and a light source of a tracking system, according to one non-limiting illustration.

[0023] Figure 15 is a diagram depicting components of a light source and a scatter detection module of a tracking system, according to one non-limiting illustration.

[0024] Figure 16 is a diagrammatic representation of an HMD device having a binocular display subsystem, a plurality of angle-sensitive detectors, and a plurality of scatter detection modules that are operable to detect light that has been scattered or reflected, which can be used to ignore such scattered light during position tracking of the HMD device or components thereof.

[0025] Figure 17is a perspective view of components of a light source and scatter detection module of a tracking system according to an illustrative implementation.

[0026] Figure 18 is a flowchart of a method of operating a position tracking system of an HMD system to track position, orientation, and / or motion of components of the HMD system during use according to example embodiments of the present disclosure.

[0027] Figure 19 is a flowchart of a method of adaptively adjusting brightness of a plurality of light sources or optical detectors of a position tracking system of an HMD system according to example embodiments of the present disclosure.

[0028] Figure 20 is a flowchart of a method of adaptively adjusting brightness of a plurality of light sources or optical detectors of a position tracking system of an HMD system based on changes to one or more parameters according to example embodiments of the present disclosure.

[0029] Figure 21 is a flowchart of a method of compensating for non-uniform brightness of light sources of a position tracking system of an HMD system according to example embodiments of the present disclosure.

[0030] Figure 22 is a flowchart of a method of adaptively enabling and disabling components (e.g., light sources, optical detectors) of a tracking subsystem of an HMD system according to example embodiments of the present disclosure.

[0031] Figure 23 is a flowchart of a method of operating a position tracking system of an HMD system to track position, orientation, and / or motion of components of the HMD system by fusing inertial sensor data, optical sensor data, and image data according to example embodiments of the present disclosure. DETAILED DESCRIPTION

[0032] In the following description, certain specific details are set forth in order to provide a thorough understanding of various disclosed implementations. However, one skilled in the relevant arts will recognize that implementations can be practiced without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known structures associated with computer systems, server computers, and / or communication networks have not been described in detail to avoid unnecessarily obscuring descriptions of the implementations.

[0033] Unless the context requires otherwise, throughout the specification and claims which follow, the word "comprising" is synonymous with "including," "containing," or "comprising," and is inclusive or open-ended (i.e., does not exclude additional, unrecited elements or method acts).

[0034] References to "one implementation" or "an implementation" throughout this specification mean that a particular feature, structure, or characteristic described in connection with the implementation is included in at least one implementation. Thus, appearances of the phrases "in one implementation" or "in an implementation" as various places throughout the specification are not necessarily all referring to the same implementation. Furthermore, the particular features, structures, or characteristics can be combined in any suitable manner in one or more implementations.

[0035] As used in this specification and the appended claims, the singular forms "a," "an" and "the" include plural referents unless the context clearly dictates otherwise. It should also be noted that the term "or" is generally employed in its sense of "and / or" unless the context clearly dictates otherwise.

[0036] The title and summary of the disclosure presented herein provide a general description of the disclosure, and are not intended to be limiting.

[0037] One or more implementations of the present disclosure relate to systems and methods for precisely tracking the position of components of a head-mounted display (HMD) system (e.g., HMD, controllers, peripherals). In at least some implementations, the HMD includes a support structure that carries a forward-facing camera ("forward camera" or "front camera") and a plurality of angle-sensitive detectors or light sources. Similarly, one or more controllers can include a plurality of angle-sensitive detectors or light detectors. In other implementations, the HMD does not include a forward-facing camera. The forward-facing camera can capture image sensor data in a field of view of the forward-facing camera at a first frame rate (e.g., 30 Hz, 90 Hz). In at least some implementations, the HMD system can not include angle-sensitive detectors, or can include other types of optical detectors (e.g., photodiodes). Thus, where appropriate, the systems and methods described herein can utilize non-angle-sensitive detectors or angle-sensitive detectors.

[0038] In operation, one or more fixed or movable light sources (e.g., IR LEDs) can be made to emit light, as discussed further below. The light sources can be coupled to the HMD, a controller, a fixed object (e.g., a base station) located in the environment, etc. Each of the plurality of angle-sensitive detectors captures sensor data in a respective plurality of angle-sensitive detector fields of view at a second frame rate (e.g., 1000 Hz, 2000 Hz), which can be greater than the first frame rate of the forward-facing camera (when present). In at least some implementations, the angle-sensitive detector fields of view can be narrower than the forward-facing camera field of view, although this is not required. For example, the forward-facing camera can have a relatively wide forward-facing camera field of view of 90°, 120°, or 150°, and each angle-sensitive detector can have a relatively narrow sensor IC field of view (e.g., 25°, 45°, 75°). In at least some implementations, the angle-sensitive detector fields of view can collectively cover at least a majority of the forward-facing camera field of view, or even be greater than the forward-facing camera field of view, with each angle-sensitive detector field of view overlapping a different portion of the forward-facing camera field of view.

[0039] In operation, at least one processor operatively coupled to the plurality of angle-sensitive detectors can receive sensor data capturing light from a plurality of light sources (e.g., LEDs, lasers, other light sources). The at least one processor can process the received image sensor data based at least in part on processing of the received image sensor data to track a position of a component of a head-mounted display. For example, the at least one processor can fuse sensor data from the angle-sensitive detectors to track one or more features present in the environment. The at least one processor can utilize machine learning techniques, solvers, or other methods to process the sensor data to determine a position (e.g., location, orientation, movement) of one or more components of the HMD system. In at least some implementations, the sensor data can be fused with sensor data from other sensors, such as a forward-facing camera or inertial measurement unit (IMU) from a component of the HMD system. In at least some implementations, one or more scatter detection modules or “scatter detectors” can be used to detect when light has been scattered or reflected before reaching one or more angle-sensitive detectors, and such light can be ignored by the tracking system as its angle does not accurately indicate the location of the light source that emitted the light. With such techniques, the accuracy of the position tracking can be greatly improved. Various features of implementations of the present disclosure are discussed in detail below with reference to the figures.

[0040] Figure 1This is a schematic diagram of a networked environment 100 including a Local Media Rendering (LMR) system 110 (e.g., a gaming system), which includes a local computing system 120, a display device 180 (e.g., an HMD device with two display panels, one for each eye), and one or more controllers 182 adapted to perform at least some of the techniques described herein. Figure 1 In the depicted embodiment, the local computing system 120 is connected via a transmission link 115 (which may be wired or wired, such as via... Figure 2 One or more cables (cable 220) illustrated, or alternatively wireless, are communicatively connected to display device 180. Controller 182 may be coupled to local computing system 120 or display device 180 via suitable wired link 186 or wireless link 184, respectively. In other embodiments, local computing system 120 may provide encoded image data for display on panel display devices (e.g., TVs, consoles, or monitors) via wired or wireless links, whether or not it is an alternative to HMD device 180, and each display device includes one or more addressable pixel arrays. In various embodiments, local computing system 120 may include: general-purpose computing systems; game consoles; video streaming devices; mobile computing devices (e.g., cellular phones, PDAs, or other mobile devices); VR or AR processing devices; or other computing systems.

[0041] In the illustrated embodiment, the local computing system 120 has components including one or more hardware processors (e.g., a central processing unit or "CPU") 125, memory 130, various I / O ("input / output") hardware components 127 (e.g., a keyboard, a mouse, one or more game controllers, speakers, a microphone, an IR emitter and / or receiver, etc.), a video subsystem 140 including one or more specialized hardware processors (e.g., a graphics processing unit or "GPU") 144 and video memory (VRAM) 148, computer-readable storage 150, and a network connection 160. Also in the illustrated embodiment, an embodiment of the tracking subsystem 135 is executing in the memory 130 in order to perform at least some of the described techniques, such as by using the CPU 125 and / or GPU 144 to perform automated operations implementing those described techniques, and the memory 130 can optionally further execute one or more other programs 133 (e.g., a game program generating video or other images to be displayed). As part of the automated operations implementing at least some of the techniques described herein, the tracking subsystem 135 and / or programs 133 executing in the memory 130 can store or retrieve various types of data, including in example database data structures of the storage 150, in which case the data used can include various types of image data information in a database ("DB") 154, various types of application data in a DB 152, various types of configuration data in a DB 157, and can include additional information such as system data or other information.

[0042] In the depicted embodiment, the LMR system 110 is also communicably connected, via one or more computer networks 101 and network links 102, to an example network-accessible media content provider 190, which can further provide content to the LMR system 110 for display, either in addition to or instead of the image generation program 133. The media content provider 190 can include one or more computing systems (not shown), each of which can have components similar to those of the local computing system 120, including one or more hardware processors, I / O components, local storage, and memory, although some details are not illustrated for brevity.

[0043] It will be appreciated that, although in the depicted embodiment the tracking subsystem 135 is executing in the memory 130 of the local computing system 120, in other embodiments the tracking subsystem 135 can be executing in the memory of a remote computing system, such as the media content provider 190, or in a distributed manner across multiple computing systems. Figure 1The display device 180 is depicted in the illustrated embodiment as distinct and separate from the local computing system 120, but in some embodiments, some or all of the components of the local media rendering system 110 can be integrated or housed within a single device, such as a mobile gaming device, a portable VR entertainment system, an HMD device, etc. In such embodiments, the transport link 115 may, for example, comprise one or more system buses and / or video bus architectures.

[0044] As one example involving operations performed locally by the local media rendering system 120, assume that the local computing system is a gaming computing system, such that the application data 152 comprises one or more gaming applications executed via use of the CPU 125 of the memory 130, and various video frame display data is generated and / or processed by the image generation program 133, such as in conjunction with the GPU 144 of the video subsystem 140. To provide a high-quality gaming experience, a large amount of video frame data is generated by the local computing system 120 (corresponding to a high image resolution for each video frame, as well as a high "frame rate" of approximately 60-180 such video frames per second), and provided to the display device 180 via the wired or wireless transport link 115.

[0045] It will also be appreciated that the computing system 120 and the display device 180 are merely illustrative, and are not intended to limit the scope of the application. The computing system 120 can alternatively comprise multiple interacting computing systems or devices, and can be connected to other devices, including through one or more networks, such as the Internet, via a Web, or via a private network (e.g., a mobile communications network, etc.). More generally, the computing system or other computing nodes can comprise any combination of hardware or software that can interact and perform the types of functions described, including, but not limited to, desktop or other computers, gaming systems, database servers, network storage devices and other network devices, PDAs, cellular telephones, wireless telephones, pagers, electronic organizers, Internet appliances, television-based systems (e.g., using set-top boxes and / or personal / digital video recorders), and various other consumer products that include appropriate communication capabilities. The display device 180 can similarly comprise one or more devices having various types and forms of display panels, and optionally various other hardware and / or software components.

[0046] Additionally, in some embodiments, the functionality provided by the tracking subsystem 135 can be distributed among one or more components (e.g., local and remote computing systems, HMDs, controllers, base stations), and in some embodiments, some of the functionality of the tracking subsystem 135 can not be provided and / or other additional functionality can be available. It should also be understood that, while various items are illustrated as being stored in memory or on storage while being used, these items or portions of them can be transferred between memory and other storage devices, for example, as can serve memory management or data integrity. Thus, in some embodiments, some or all of the described techniques can be performed by hardware, for example, circuitry, including one or more processors or other configuration of hardware circuitry, or by hardware configured by software, for example, by one or more software programs (e.g., by the tracking subsystem 135 or components thereof) and / or data structures (e.g., by software instructions executed by one or more software programs and / or by storage of such software instructions and / or data structures). Some or all of the components, systems and data structures can also be stored (e.g., as software instructions or structured data) on non-transitory computer-readable storage media such as a hard disk or flash drive or other non-volatile storage device, volatile or non-volatile memory (e.g., RAM), a network storage device, or a portable media article to be read by an appropriate drive (e.g., a DVD disk, a CD disk, an optical disk, or the like), or via an appropriate connection. In some embodiments, the systems, components and data structures can also be transmitted as generated data signals (e.g., as part of a carrier wave or other analog or digital propagated signal) on a variety of computer- readable transmission media, including wireless-based and wired / cable-based media, and can take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). In other embodiments, such computer program products also can take other forms. Thus, the present application can be implemented on one or more computer systems that include components such as those described above.

[0047] Figure 2An example environment 200 is shown in which at least some of the described technology is used with an example HMD device 202 coupled to a video rendering computing system 204 via a tethered connection 220 (or a wireless connection in other embodiments) to provide a virtual reality display to a human user 206. The user wears the HMD device 202 and receives display information from the computing system 204 via the HMD device of a simulated environment that is different from the actual physical environment, with the computing system acting as an image rendering system that provides images of the simulated environment to the HMD device for display to the user, such as images generated by a game program and / or other software programs executing on the computing system. In this example, the user is also able to move within a tracked volume 201 of the actual physical environment 200 and can also have one or more I / O ("input / output") devices to allow the user to further interact with the simulated environment, including handheld controllers 208 and 210 in this example.

[0048] In the illustrated example, the environment 200 can include one or more base stations 214 (two are shown, labeled base stations 214a and 214b) that can help track the HMD device 202 or the controllers 208 and 210. As the user moves the location of the HMD device 202 or changes the orientation of the HMD device 202, the position of the HMD device is tracked in order to allow the corresponding portion of the simulated environment to be displayed to the user on the HMD device, and the controllers 208 and 210 can also use similar techniques to track the position of the controllers (and optionally use this information to help determine or verify the position of the HMD device). After the tracked position of the HMD device 202 is known, corresponding information is transmitted to the computing system 204 via the tether 220 or wirelessly, which uses the tracked position information to generate one or more next images of the simulated environment to display to the user.

[0049] The optical tracking described herein can be used in conjunction with various position tracking methods, including but not limited to acoustic tracking, inertial tracking, or magnetic tracking, among others.

[0050] In at least some implementations, at least one of the HMD device 202 and the controllers 208 and 210 can include one or more light receivers or sensors that can be used to implement the tracking functionality or other aspects of the present disclosure. In at least some implementations, at least one of the HMD device 202, the controllers 208 and 210, or other components can include one or more light sources (e.g., LEDs) that can emit light that is detected by one or more of the optical receivers. The light sources can be in a fixed position or can be located on a movable component, such as the HMD device or a controller.

[0051] In at least some implementations, in addition to or instead of generating a point light source, the base stations 214 can each sweep a light signal over the tracking volume 201. Depending on the requirements of each particular implementation, each base station 214 can generate more than one light signal. For example, while a single base station 214 is typically sufficient for six degrees of freedom tracking, in some embodiments multiple base stations (e.g., base stations 214a, 214b) can be needed or desired to provide robust room-scale tracking for HMD devices and peripherals. In this example, light receivers such as angle-sensitive detectors or scatter detectors are incorporated into the HMD device 202 and / or other tracked objects such as the controllers 208 and 210. In at least some implementations, the light receivers can be paired with accelerometers and gyroscopic inertial measurement units ("IMUs") on each tracked device to support low-latency sensor fusion.

[0052] In at least some implementations, each base station 214 includes two rotors that sweep linear beams over the tracking volume 201 on orthogonal axes. At the beginning of each sweep cycle, the base station 214 can emit an omnidirectional light pulse (referred to as a "sync signal") that is visible to all sensors on tracked objects. Thus, each sensor calculates a unique angular position in the swept volume by timing the duration between the sync signal and the beam signal. Sensor distance and orientation can be resolved using multiple sensors fixed to a single rigid body.

[0053] The one or more sensors located on tracked objects (e.g., the HMD device 202, the controllers 208 and 210) can include optoelectronic devices capable of detecting modulated light from the rotors. For visible or near-infrared (NIR) light, silicon photodiodes and appropriate amplifier / detector circuitry can be used. Because the environment 200 can include static and time-varying signals (light noise) having similar wavelengths as the base station 214 signals, in at least some implementations the base station light can be modulated in such a way that it is easily distinguished from any interfering signals, and / or the sensors are filtered from radiation at wavelengths other than the base station signals. As discussed further below, in at least some implementations angle-sensitive detectors are used to track one or more components of the HMD system, and one or more scatter detectors can be used to ignore light that can have scattered or reflected before being detected by the optical detectors.

[0054] Inertial-Extrinsic tracking is also a type of position tracking that can be used to track the position of the HMD device 202 and / or other objects (e.g., controllers 208 and 210, tablet computers, smartphones). Inertial-Extrinsic tracking differs from Extrinsic-Inertial tracking in the location of the cameras or other sensors used to determine the position of the HMD assembly. For inertial-extrinsic tracking, the cameras or sensors are located on the HMD assembly or the object being tracked, while in extrinsic-inertial tracking, the cameras or sensors are placed in a fixed location in the environment.

[0055] An HMD utilizing inertial-extrinsic tracking utilizes one or more sensors to "watch" to determine how its position changes relative to the environment. As the HMD moves, the sensors re-adjust their position in the room and the virtual environment responds in real-time accordingly. This type of position tracking can be implemented with or without markers placed in the environment. Cameras placed on the HMD watch the features of the surrounding environment. When markers are used, the markers are designed to be easily detected by the tracking system and are placed in specific areas. With "markerless" inertial-extrinsic tracking, the HMD system uses unique features that are initially present in the environment (e.g., natural features) to determine position and orientation. Algorithms of the HMD system recognize specific images or shapes and use them to calculate the device's position in space. Data from accelerometers and gyroscopes can also be used to improve the accuracy of the position tracking.

[0056] Figure 3 A front view of the HMD device 344 is shown when worn on the head of a user 342. The HMD device 344 includes a front-facing structure 343 that supports a front-facing or forward-facing camera 346 and one or more types of multiple angle-sensitive detectors 348a-348f (collectively, 348), or other types of optical detectors. As one example, some or all of the angle-sensitive detectors 348 can help determine the location and orientation of the device 344 in space, such as light sensors to detect and use light information emitted from one or more external devices (not shown, e.g., base stations 214, controllers) of the system 200. The angle-sensitive detectors 348 can be any type of detector for detecting the angle of arrival of light emitted from a light source. Non-limiting examples of angle-sensitive detectors include photodiode detectors (e.g., dual cell detectors, quadrant cell detectors), position sensitive detectors using resistive sheets, etc. Figure 2

[0057] ​As shown, forward-facing cameras 346 and angle-sensitive detectors 348 are directed forward toward the actual scene or environment (not shown) in which user 342 is operating HMD device 344. More generally, angle-sensitive detectors 348 can be directed to other areas (e.g., up, down, left, right, rearward) to detect light from various sources, such as a controller (e.g., held by user 342) or objects mounted at various locations (e.g., walls, ceilings). The actual physical environment can include, for example, one or more objects (e.g., walls, ceilings, furniture, stairs, cars, trees, tracking markers, light sources, or any other type of object). The particular number of sensors 348 can be fewer (e.g., 2, 4) or greater (e.g., 10, 20, 30, 40) than the number of sensors depicted. HMD device 344 can further include one or more additional components not attached to the forward-facing structure (e.g., internal to the HMD device), such as an IMU (inertial measurement unit) 347 electronic device that measures and reports particular forces, angular rates, and / or magnetic fields around the HMD device 344 (e.g., using a combination of accelerometers and gyroscopes and optionally magnetometers). HMD device 344 can also include additional components not shown, including one or more display panels and optical lens systems oriented toward the user's eyes (not shown) and optionally having one or more attached internal motors to change the alignment or other positioning of the one or more optical lens systems and / or display panels within the HMD device.

[0058] The illustrated example of HMD device 344 is supported at least in part on the head of user 342 by one or more straps 345 attached to the housing of HMD device 344 and extending around the user's head in whole or in part. Although not illustrated here, HMD device 344 can also have one or more external motors, such as attached to one or more straps 345, and automatic corrective actions can include using such motors to adjust such straps in order to modify the alignment or other positioning of the HMD device on the user's head. It will be appreciated that the HMD device can include other support structures (e.g., nose pieces, chin straps, etc.) not illustrated here, either in addition to or instead of the illustrated straps, and some embodiments can include motors attached to one or more such other support structures to similarly adjust their shape and / or location in order to modify the alignment or other positioning of the HMD device on the user's head. Other display devices not secured to the user's head can similarly be attached to or be part of one or more structures that affect the positioning of the display device, and in at least some embodiments can include motors or other mechanical actuators to similarly modify their shape and / or location in order to modify the alignment or other positioning of the display device relative to one or more pupils of one or more users of the display device.

[0059] Figure 4 An example of the manual controller 400 is shown in more detail. In practice, an HMD system may include... Figure 4 The manual controller 400 is similar to or identical to two manual controllers, which may be similar to or identical to the controllers 182, 208, and 210 described above. As shown, controller 400 has various surfaces on which angle-sensitive detectors 402 are located. The angle-sensitive detectors 402 are arranged to receive light signals from various different directions. Controller 400 may have buttons, sensors, lights, controllers, knobs, indicators, displays, etc., thereby allowing users to interact in various ways. Additionally, as mentioned above, in at least some implementations, one of controller 400 and HMD device 344 may include multiple light sources, while the other of controller and HMD device may include multiple angle-sensitive detectors or other types of detectors or sensors. The techniques described herein can be used for various types of position tracking, and are not limited to HMDs, controllers, etc.

[0060] Figure 5 A schematic block diagram of an HMD device 500 according to one or more implementations of this disclosure is shown. The HMD device 500 may be similar to or the same as HMD devices discussed elsewhere herein. Therefore, the above discussion regarding HMD devices can also be applied to the HMD device 500. Furthermore, at least some components of the HMD device 500 may be present in other components of the HMD system, such as controllers, base stations, etc. Therefore, at least some of the descriptions below are applicable to such other components.

[0061] HMD device 500 includes a processor 502, a forward-facing camera or front-facing camera 504, multiple angle-sensitive detectors 506 (e.g., quad-cell photodiodes, position-sensitive detectors), and optionally includes an IMU 507 or multiple light sources 509. In some implementations, HMD device 500 may include one of the light sources or angle-sensitive detectors, and other components (e.g., controllers, base stations) may include another of the angle-sensitive detectors or light sources. As described below, in at least some implementations, HMD device 500 may include one or more scattering detection modules or scattering detectors, which can be used to detect whether light received by one or more angle-sensitive detectors has been scattered or reflected and should therefore be ignored. HMD device 500 may include a display subsystem 508 (e.g., two displays and corresponding optical systems). HMD device 500 may also include a non-transitory data memory 510, which may store instructions or data for position tracking 512, instructions or data for display functions 514 (e.g., games), and / or other programs 516. HMD system 500 may include some of the functions of local computing system 120 or media content provider 190, or allow the functions of local computing system 120 or media content provider 190, such asFigure 1 as discussed above.

[0062] The HMD device 500 can also include various I / O components 518, which can include one or more user interfaces (e.g., buttons, touchpads, speakers), one or more wired or wireless communication interfaces, and the like. As an example, the I / O components 518 can include a communication interface that allows the HMD device 500 to communicate with an external device 520 over a wired or wireless communication link 522. As non-limiting examples, the external device 520 can include a host, a server, a mobile device (e.g., a smartphone, a wearable computer), a controller, and the like. The various components of the HMD device 500 can be housed in a single housing, can be housed in separate housings (e.g., a host), or any combination thereof.

[0063] It should be appreciated that the illustrated computing systems and devices are merely illustrative and are not intended to limit the scope of the disclosure. For example, the HMD 500 and / or the external device 520 can be connected to other devices not illustrated, including through one or more networks, such as the Internet or via the Web. More generally, such computing systems or devices can include any combination of hardware that is capable of interacting and performing functions of the type described, such as when appropriately programmed or configured with software, including but not limited to desktop computers, laptop computers, touchscreen tablet computers, tablet computers or other computers, smart phone computing devices and other cellular telephones, Internet appliances, PDAs and other electronic organizers, database servers, network storage devices and other network equipment, wireless telephones, pagers, television-based systems (e.g., using set-top boxes and / or personal / digital video recorders and / or game consoles and / or media servers), and various other consumer products that include appropriate interworking capabilities. For example, in at least some embodiments, the illustrated systems 500 and 520 can include executable software instructions and / or data structures that, when loaded into a particular computing system or device and / or executed by the particular computing system or device, can be used to program or otherwise configure those systems or devices, such as configuring processors of those systems or devices. Alternatively, some or all of the software systems can be executed in the memory of another device and communicated to the illustrated computing systems / devices via inter-computer communication. In addition, while various items are illustrated as being stored in memory or the storage device, these items or portions of them can be transferred between storage devices and memory (e.g., over a network), and / or stored on different storage devices at different times, e.g., back up storage, cache or other storage.

[0064] Thus, in at least some embodiments, the illustrated system is a software-based system that includes software instructions that, when executed by a processor and / or other processor device, program the processor to automatically perform the described operations for the system. Additionally, in some embodiments, some or all of the system can be implemented or provided in other manners, such as at least partially in firmware and / or hardware, including, but not limited to, one or more application-specific integrated circuits (ASICs), standard integrated circuits, controllers (e.g., by execution of appropriate instructions and including microcontrollers and / or embedded controllers), field- programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), etc. Some or all of the system and / or data structures can also be stored (e.g., as software instructions or structured data content) on a non-transitory computer- readable storage medium, such as a hard disk or flash drive or other non-volatile storage device, volatile or non-volatile memory (e.g., RAM), a network storage device, or a portable media article (e.g., a DVD disk, a CD disk, an optical disk, a flash drive, etc.) to be read by an appropriate drive or via an appropriate connection, in some embodiments, the system, modules, and data structures can also be transmitted as generated data signals (e.g., as part of a carrier wave or other analog or digital propagated signal) on a variety of computer- readable transmission mediums, including wireless-based and wired / cable-based mediums, and can take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). Such computer program product can also take other forms in other embodiments. Thus, the present disclosure can be practiced with other computer system configurations.

[0065] Figure 6is a schematic diagram of an environment 600 implemented in accordance with one non-limiting implementation in which machine learning techniques can be used to implement a tracking subsystem of an HMD device, one or more controllers, or other components, such as the tracking subsystems discussed herein. The environment 600 includes a model training portion 601 and an inference portion 603. In the training portion 601, training data 602 is fed into a machine learning algorithm 604 to generate a trained machine learning model 606. The training data can include, for example, labeled data from angle-sensitive detectors that specify a position and / or orientation of a particular object relative to one or more light sources (e.g., LEDs), labeled or unlabeled scatter detector data (discussed below), or other types of data. As a non-limiting example, in an embodiment that includes a component (e.g., an HMD, a controller) having 30 angle-sensitive detectors, each training sample can include output from each or a subset of the angle-sensitive detectors, a known or inferred position or orientation of the component, and information about the position or orientation of one or more light sources. As discussed below, each angle-sensitive detector can output a single data point (e.g., an angle), or can output multiple data points, such as two or four signals each indicative of a power or intensity of light received at a particular active element (e.g., a sub-detector or cell, a resistive sheet, etc.) of the angle-sensitive detector. The data can also include data from one or more scatter detectors, such as the scatter detectors discussed below. Such data can include polarization information (e.g., a type or degree of polarization), information about whether detected light has been scattered, or other types of data.

[0066] The training data 602 can be obtained from multiple users and / or from a single user of an HMD system. The training data 602 can be obtained in a controlled environment and / or during actual use by a user (“live training”). Additionally, in at least some implementations, the model 606 can be updated or calibrated from time to time (e.g., periodically, continuously, after certain events) to provide accurate position tracking predictions.

[0067] In inference portion 603, runtime data 608 is provided as input to trained machine learning model 606, which generates position tracking predictions 610. Continuing the example described above, output data (e.g., intensity data, angle data) of the angle-sensitive detector, optionally information regarding one or more light sources, and optionally information from one or more scatter detectors, can be provided as input to trained machine learning model 606, which can process the data to predict the position of the component. Tracking predictions 610 can then be provided to one or more components associated with the HMD device, such as one or more VR or AR applications, one or more display or rendering modules, one or more mechanical controls, one or more additional position tracking subsystems, etc.

[0068] Machine learning techniques used to implement features discussed herein can include any type of suitable structure or technique. As non-limiting examples, machine learning model 606 can include one or more of decision trees, statistical hierarchical models, support vector machines, artificial neural networks (ANN) such as convolutional neural networks (CNN) or recurrent neural networks (RNN) (e.g., long short-term memory (LSTM) networks), mixture density networks (MDN), hidden Markov models, or others that can be used. In at least some implementations, such as implementations utilizing RNNs, machine learning model 606 can utilize past input (memory, feedback) information to predict the position of one or more HMD components. Such implementations can advantageously utilize sequential data to determine motion information or previous position predictions, which can provide more accurate real-time position predictions.

[0069] Figure 7 is a flowchart of an example method 700 of operating an HMD system to track the position of HMD components during use. Method 700 can be performed by, for example, position tracking system or module 512 of HMD system 500 shown in FIG. 6. Figure 5 As described above, method 700 can be used to track the position of any component, such as an HMD device, one or more controllers, etc.

[0070] Implementations of the illustrated method 700 begin at act 702, where a first HMD system component having a plurality of angle-sensitive detectors is provided. The plurality of angle-sensitive detectors are operable to detect light emitted from one or more light sources, which can be fixedly positioned (e.g., mounted to a wall or ceiling) or movably positioned (e.g., coupled to an HMD or controller). In operation, each of the plurality of angle-sensitive detectors captures sensor data in a respective field of view of the angle-sensitive detector at a frame rate. The sensor data can include any type of data that can be used by control circuitry (e.g., a processor) to detect the presence and direction of a light source relative to an angle-sensitive detector. In at least some implementations, each of the angle-sensitive detectors can include one or more sensors (e.g., photodiodes) having image sensing circuitry and image processing circuitry thereon. The angle-sensitive detectors can output relatively raw data (e.g., light intensity or power data) or processed data (e.g., angle of incidence data).

[0071] At 704, a second HMD system component including a plurality of light sources (e.g., near-IR LEDs) can be provided. The second HMD system component can include a controller, an HMD device, or light sources located at a fixed location (e.g., a ceiling, a wall).

[0072] At 706, at least one processor of the HMD system can cause the light sources to emit light. The light sources can be illuminated in such a way that each of the angle-sensitive detectors can detect light from a single light source at a time, or more generally, in such a way that the system is able to determine from which light source the light detected by the angle-sensitive detectors was received. This can be accomplished by using any appropriate type of technique to modulate or multiplex the illumination of the light sources, such as time multiplexing, wavelength multiplexing, frequency multiplexing, polarization multiplexing, or other techniques that allow the system to know the light source from which light is received during use by each angle-sensitive detector.

[0073] As an example of time multiplexing, the at least one processor can illuminate only a subset of the light sources (e.g., one, two, four) at a time. For example, the at least one processor can sequentially illuminate the light sources one subset at a time, and collect sensor data in response to each light source.

[0074] As an example of wavelength multiplexing, different subsets of the light sources can emit different wavelengths of light, and different subsets of the angle-sensitive detectors can be used to sense different wavelengths of light. Thus, light sources having different wavelengths can be illuminated at the same time, and detected by respective wavelength-sensitive detectors.

[0075] As an example of frequency multiplexing, a subset of the light sources can be illuminated in a determined pattern or frequency that can be detected by the angle-sensitive detector to identify the particular light source of the light.

[0076] As an example of polarization multiplexing, a subset of the light sources can be polarized differently (e.g., linear, circular), and a corresponding subset of the angle-sensitive detector can be configured to detect certain polarized light (e.g., using a polarizer that passes light having a corresponding polarization), which allows multiple light sources to be illuminated simultaneously.

[0077] Other example techniques for illuminating the light sources can include one or more of frequency division multiple access or wavelength division multiple access (FDMA or WDMA), time division multiple access (TDMA), code division multiple access (CDMA), and orthogonal frequency division multiple access (OFDMA). In at least some implementations, the illumination pattern or scheme can be configured to be orthogonal with respect to one or more of time, wavelength, or frequency of the electrical system (e.g., 120 Hz, which is twice the 60 Hz frequency used in electrical systems in the United States, or 100 Hz, which is twice the 50 Hz frequency used in electrical systems in Europe). Additionally, a modulation scheme can be applied to the amplitude of the light of each individual marker or LED, or to one or more groups of two or more markers or LEDs, or to any combination thereof. In at least some implementations, two or more of the light-emitting assembly, the light-detecting assembly, or the processing assembly (e.g., the host system) can be synchronized with one another, which can provide further advantages, for example, during use of CDMA techniques or the like.

[0078] At 708, at least one processor associated with the HMD system can receive sensor data from the plurality of angle-sensitive detectors. As described above, for each angle-sensitive detector, the sensor data can indicate an angle of arrival of light emitted from a known light source. At 710, at least one processor associated with the HMD system can optionally receive sensor data from an inertial measurement unit (IMU) that can be operable to provide inertial tracking capabilities or sensor data from one or more additional sensors.

[0079] At 712, at least one processor associated with the HMD system can process the received sensor data. For example, the at least one processor can fuse some or all of the sensor data together to track one or more features present in the environment in which the HMD system is operating. The sensor data can include sensor data from multiple angle-sensitive detectors, and optionally sensor data from an IMU or from a camera. For example, the at least one processor can process the sensor data using a machine learning model (e.g., model 606) or another solver. As discussed further below, in at least some implementations, the at least one processor can disregard data from one or more sensors that are determined to be likely to receive light that has been scattered or reflected.

[0080] At 714, at least one processor associated with the HMD system can track, in real-time, a position (e.g., location, orientation, or movement) of a component of the HMD system during use of the HMD system by a user in the environment. As described above, method 700 can continue during operation of the HMD to continuously track the position of the component of the HMD system.

[0081] Figure 8 A perspective view of an example angle-sensitive detector 800 that can be used in one or more implementations of the present disclosure is shown. In this example, angle-sensitive detector 800 includes an angle-sensitive photodiode structure 804. Angle-sensitive photodiode structure 804 includes a photodiode 806, a second linear polarizer 808, a spatially-varying polarizer 810, and a first linear polarizer 812. Photodiode 806 can be any device that receives light, determines an intensity associated with the light, and outputs a signal (or data) representative of that intensity. First linear polarizer 812 and second linear polarizer 808 can each be any type of optical filter on which light is incident. First linear polarizer 812 and second linear polarizer 808 can output a linearly polarized component of the incident light (e.g., vertically polarized or horizontally polarized), and filter out (e.g., reflect or suppress, absorb) other components of the incident light.

[0082] In at least some implementations, spatially-varying polarizer 810 can be formed from a multi-twist retarder (MTR), which is a wave-plate like retardation film that provides precise and customized levels of broadband, narrowband, or multi-band retardation in a single thin film. More specifically, the MTR includes two or more twisted liquid crystal (LC) layers on a single substrate, and has a single alignment layer. Subsequent LC layers are directly aligned through the existing layer, allowing for simple fabrication, enabling automatic layer alignment, and resulting in a monolithic film with a continuously varying optical axis.

[0083] Spatial-variable polarizer 810 may include a wave delayer formed of a birefringent material. Birefringence is a property of a material having a refractive index that depends on the polarization and propagation direction of light. The wave delayer alters the polarization state or phase of light traveling through it. The wave delayer may have a slow axis (or unusual axis) and a fast axis (ordinary axis). When polarized light travels through the wave delayer, light along the fast axis travels faster than light along the slow axis.

[0084] The second linear polarizer 808, the spatially variable polarizer 810, and the first linear polarizer 812 can be stacked on the photodiode 806, such as Figure 8 As shown, and continuously layered on photodiode 806. It should be noted that although polarizers 812 and 808 are described herein as linear polarizers, in various embodiments, polarizers 812 and 808 may be nonlinear polarizers, and may be, for example, elliptical or circular polarizers. Polarizers 812 and 808 may have the same filtering characteristics and may similarly or identically suppress or transmit light with a specific polarization. In this simplified example, angle-sensitive detector 800 includes a cover 814 having an aperture 816 that allows light 818 from light source 820 to pass through. As shown, the light 818 passing through aperture 816 forms a light spot 822, which can be electrically characterized to determine the angle of light 818, and thus the angle of light source 820 relative to angle-sensitive detector 800. As described below, the systems and methods of this disclosure can utilize multiple light sources and angle-sensitive detectors to determine the location of components in an HMD system.

[0085] Figure 9 The diagram illustrates a first linear polarizer 812, a spatially variable polarizer 810, and a second linear polarizer 808 in an angle-sensitive photodiode structure 804, as well as the polarization of light 818 or a spot 822 reaching the photodiode 806 therethrough. Initially, light 818 illuminates the first linear polarizer 812. Light 818 can have any polarization and is therefore referred to as unpolarized in at least some implementations. In at least some implementations, the light can be linearly polarized, circularly polarized, or generally elliptically polarized.

[0086] The first linear polarizer 812 passes through the linear polarization component 824 of light 818 and suppresses (absorbs or reflects) the remaining polarization component of light 818. Although the first linear polarizer 812 is shown as a vertical polarization filter, in various embodiments, the first linear polarizer 812 may be a horizontal polarization filter or a circular polarization filter, etc.

[0087] The linearly polarized component 824 is then incident on the spatially- varying polarizer 810 located below the first linear polarizer 812. The spatially- varying polarizer 810 is adjusted to have light polarization characteristics that vary as a function of position on the spatially-varying polarizer 810 on which the linearly polarized component 824 (or any incident light) is incident. In Figure 9 In the example shown, the spatially-varying polarizer 810 changes the incident linearly polarized component 824.

[0088] The manner in which the spatially-varying polarizer 810 changes the incident linearly polarized component 824 varies as a function of the position on the spatially-varying polarizer 810 on which the incident linearly polarized component 824 is incident. This position can be substantially the same as the position on which the light 818 is incident on the angle-sensitive photodiode structure 804.

[0089] In this illustrative example, at a first end 826 (upper right as shown) of the spatially-varying polarizer 810, the spatially-varying polarizer 810 maintains the incident linearly polarized component 824 as a vertically polarized light signal. The spatially-varying polarizer 810 passes the vertically polarized light and blocks other polarization components. The linearly polarized component 824 incident on the first end 826 passes through as is.

[0090] As a non-limiting example, the polarization filtering characteristics of the spatially-varying polarizer 810 can vary gradually as a function of distance to the first end. At a second end 828 (lower left as shown) of the spatially-varying polarizer 810, the spatially-varying polarizer 810 almost converts the incident linearly polarized component 824 that is vertically polarized into a horizontally polarized light signal. In particular, at the second end 828, the spatially-varying polarizer 810 has a linear polarization direction of 175°. Thus, at the second end 828, the spatially-varying polarizer 810 outputs light having a greater horizontally polarized component than vertically polarized component. In contrast, near the center of the spatially-varying polarizer 810, the spatially-varying polarizer 810 has a linear polarization direction of about 135°, and, thus, the spatially-varying polarizer 810 rotates the polarization of the incident linearly polarized component 824 (which is vertically polarized) by an angle of about 45° toward horizontal polarization. The light exiting the spatially-varying polarizer 810 has a vertically polarized component near its center that has the same magnitude as its horizontally polarized component.

[0091] The spatially-varying characteristics of the spatially-varying polarizer 810 enable the position or possible positions of the linearly polarized component 824 to be identified. As Figure 9As shown, the spatially variable polarizer 810 transmits filtered light 830. The intensity of the horizontally or vertically polarized filtered light 830 represents the position, or possible position, of the linear polarization component 824 incident on the spatially variable polarizer 810. When the linear polarization component 824 incident on the first end 826, the filtered light 830 has the highest vertical polarization amplitude. The vertical polarization amplitude is inversely proportional to the distance from the first end 826.

[0092] Then, the filtered light 830 illuminates a second linear polarizer 808, which removes any horizontal component of the filtered light 830 and allows the vertical component to pass through. The second linear polarizer 808 allows the filtered linearly polarized component 832 to pass through. The second linear polarizer 808 ensures that the light passing to the photodiode 806 exclusively includes vertically polarized light and excludes horizontally polarized light.

[0093] Photodiode 806 receives the filtered linear polarization component 832 and detects its intensity. The intensity of the filtered linear polarization component 832 represents the position or position group of light 818 illuminating the spatially varying polarizer 810 and subsequently the angle-sensitive photodiode structure 804.

[0094] Note, reference Figure 9 The specific polarizations described are given by way of example for ease of description. In alternative embodiments, different polarizers, polarizations, or polarization modes may be employed. For example, instead of linear polarization, the first linear polarizer 812, the second linear polarizer 808, and the spatially variable polarizer 810 may utilize circular polarization, elliptical polarization, or any other type of polarization.

[0095] See again Figure 8 The size and position of the aperture 816 determine the size of the light spot 822 formed on the angle-sensitive photodiode structure 804. The intensity detected by the photodiode 806 represents the intensity of the light spot 822 that has passed through the first linear polarizer 812, the second linear polarizer 808, and the spatially variable polarizer 810 (or has been filtered by it). The intensity of the light spot 822 can be the sum of the intensities of the light rays constituting the light spot 822. The fact that the linear polarization component 824 of the light spot 822 illuminates a region of the spatially variable polarizer 810 instead of a point provides additional degrees of freedom in the design of the spatially variable polarizer 810 to achieve improved position detection. The spatially variable polarizer 810 can have region-varying characteristics to improve the position detection of the light spot 822.

[0096] Note that in some embodiments, one of the first linear polarizer 812 and the second linear polarizer 808 can be omitted. In embodiments, the first linear polarizer 812 can be omitted, and only horizontally polarized light can be emitted for position or angle determination.

[0097] In embodiments, improved position detection can be achieved by using a photodiode 806 having multiple area cells.

[0098] Figure 10 The first linear polarizer 812, the spatially-varying polarizer 810, and the second linear polarizer 808 of the angle-sensitive photodiode structure 804, and the polarization of light 818 or light spot 822 passing therethrough to the photodiode 806 are shown. Figure 10 In embodiments, the photodiode 806 is a four-quadrant cell (“four-way cell”) photodiode that includes four separate photodiode active regions or elements 802a-802d separated by small gaps. It should be understood that other types of angle-sensitive detectors can also be used, such as photodiode detectors having fewer or more cells, position sensitive detectors (PSDs), and the like.

[0099] The active region (e.g., anode) of each element 802a-802d is individually available such that a light spot illuminating a single quadrant can be electrically characterized only in that quadrant. The energy of the light spot is distributed between adjacent elements 802a-802d, and the difference in electrical contribution to each element defines the relative position of the light spot with respect to the center of the angle-sensitive detector. The relative intensity distribution across the elements 802a-802d can be used in conjunction with the relative intensity distribution of the spatially-varying polarizer 810 to determine the position of the light spot.

[0100] In embodiments, the spatially-varying polarizer 810 can be turned off to identify a baseline intensity. The spatially-varying polarizer 810 can be coupled to a controller. The controller can be a microcontroller or microprocessor, or one or more controllers 182 or processors 502 described herein can turn on or turn off the spatially-varying polarizer 810. When the spatially-varying polarizer 810 is turned on, the spatially-varying polarizer 810 filters light as described herein. Conversely, when the spatially-varying polarizer 810 is turned off, the spatially-varying polarizer 810 can stop polarizing filtering and pass the linear polarization component 824 as is.

[0101] When the spatially-varying polarizer 810 is turned off, the photodiode 806 detects the intensity of the light 818 (or light spot 822) without detecting the attenuation performed by the combination of the spatially-varying polarizer 810 with the first linear polarizer 812 and the second linear polarizer 808. The detected intensity can be used as a baseline intensity or maximum detected intensity. The baseline intensity or maximum detected intensity can correspond to the intensity of the light 818 impinging on the first end 826.

[0102] When the spatially-varying polarizer 810 is engaged, the photodiode 806 detects the intensity of the light 818 (or light spot 822) with the position-dependent polarization filter in place. When the position-dependent polarization filter matches the baseline intensity, the relationship between the detected intensities of the light 818 (or light spot 822) indicates the position or set of positions at which the light 818 impinges on the angle-sensitive photodiode structure 804.

[0103] As described herein, the polarization conversion performed by the spatially-varying polarizer 810 in combination with the filtering of the first linear polarizer 812 and the second linear polarizer 808 results in a spatially-varying amplitude (or intensity) attenuation of the light 818. The amplitude (or intensity) is, in turn, detected by the photodiode 806 and used for position determination.

[0104] Figure 11A and Figure 11B A top view and perspective view of an example angle-sensitive detector 1100 that can be used in one or more implementations of the present disclosure are shown, respectively. In this example, the angle-sensitive detector 1100 includes a four-quadrant cell ("four-way cell") photodiode that includes four separate photodiode active regions or elements 1102A-1102D separated by small gaps on a common substrate 1104. It should be appreciated that other types of angle-sensitive detectors can also be used, such as photodiode detectors with fewer or more cells, position-sensitive detectors, and the like.

[0105] In the illustrated non-limiting example, the active region (e.g., anode) of each element 1102A-1102D is individually addressable, such that a light spot illuminating a single quadrant can be electrically characterized as being in only that quadrant. As a light spot is translated across the angle-sensitive detector 1100, the energy of the light spot is distributed between adjacent elements 1102A-1102D, and the difference in electrical contribution to each element defines the relative position of the light spot with respect to the center of the angle-sensitive detector. The relative intensity distribution across the elements 1102A-1102D can be used to determine the position of the light spot.

[0106] In this simplified example, the angle-sensitive detector 1100 includes a cover 1110 having an aperture 1108 therein that allows light 1114 from a light source 1112 to pass through. As shown, the light 1114 forming a light spot 1106 through the aperture 1108 can be electrically characterized to determine the angle of the light 1114, and thus the angle of the light source 1112 with respect to the angle-sensitive detector 1100. As described below, the systems and methods of the present disclosure can utilize multiple light sources and angle-sensitive detectors to determine the position of components of an HMD system.

[0107] It should be appreciated that the angle-sensitive detectors of the present disclosure can include one or more of any appropriate type of detector, including four-cell photodiode detectors, position-sensitive detectors (PSDs) that utilize resistive sheets, photodiode detectors with fewer (e.g., 2) or more (e.g., 16) independent sensing elements, or any other detector capable of detecting an angle of arrival of light emitted from a light source. Additionally, as discussed below, in at least some implementations, the angle-sensitive detectors or light sources of the present disclosure can utilize various optical components, such as filters, lenses, polarizers, etc., to improve the functionality of the systems and methods discussed herein.

[0108] Figure 12 is a simplified diagram of an environment 1200 of an HMD system that uses light sources and angle-sensitive detectors to determine the position of components of the HMD system, according to one non-limiting illustrated implementation. In this example, a first component 1202, such as an HMD, includes a plurality of light sources 1206 (two are shown, 1206a and 1206b), and a second component 1204, such as a controller of the HMD system, includes a plurality of angle-sensitive detectors 1208 (two are shown, 1208a and 1208b). The angle-sensitive detectors 1208a and 1208b are separated from each other by a known distance di on the second component 1204, and the light sources 1206a and 1206b are separated from each other by a known distance d2 on the first component 1202. The first and second components can be any components of the HMD system, such as an HMD, a controller, a base station, a fixed or mobile light source, a fixed or mobile angle-sensitive detector, etc.

[0109] In this example, the angle-sensitive detector 1208a is used to determine that light arrives from the light source 1206a at an angle 1210, and that light arrives from the light source 1206b at an angle 1212. Similarly, the angle-sensitive detector 1208b is used to determine that light arrives from the light source 1206b at an angle 1214, and that light arrives from the light source 1206a at an angle 1216. Given the detected angles of arrival 1210, 1212, 1214, and 1216, and the known geometric relationships (e.g., distances di and d2) between the light sources 1206 and the detectors 1208, methods (e.g., triangulation) can be used to determine and track the relative position, orientation, or motion between the first component 1202 and the second component 1204. As described above, one or more solvers or machine learning methods can be used to determine the position of the components using sensor data from the angle-sensitive detectors and / or light source data indicative of information about the light sources of the HMD system.

[0110] Figure 13is an illustration 1300 of example light sources 1302 and angle-sensitive detectors 1304 of the present disclosure. The light sources 1302 and angle-sensitive detectors 1304 can be similar or identical to any of the light sources and angle-sensitive detectors discussed herein, and can be used in any implementation of the present disclosure. In the illustrated example, the light sources 1302 can include optical subsystems 1306, and the angle-sensitive detectors 1304 can include optical subsystems 1308. The optical subsystems 1306 and 1308 can be identical or different from each other, and can each include one or more optical components. The optical subsystems 1306 and 1308 can be integrated with the light sources 1302 and angle-sensitive detectors 1304, or can be separate components. Non-limiting examples of optical components include one or more lenses, one or more polarizers, one or more filters, one or more apertures, etc. In at least some implementations, a subset of light sources can include one type of optical subsystem, and one or more other subsets of light sources can include another type of optical subsystem. Similarly, a subset of angle-sensitive detectors can include one type of optical subsystem, and one or more other subsets of angle-sensitive detectors can include another type of optical subsystem. As an example, the optical subsystems can include filters that filter visible light or other types of light. In addition, as described above, the optical subsystems can include components that facilitate one or more of the various types of multiplexing described above that allow multiple light sources to be illuminated simultaneously without confusion as to the emitting light source.

[0111] Figure 14 is an illustration 1400 of a scatter detection module or scatter detector 1402 of the present disclosure, which can be used to determine whether light received by one or more optical detectors (e.g., angle-sensitive or other types of detectors) has been reflected or scattered before being received by the one or more optical detectors. Using this information, the at least one processor can be operable to ignore light data that is determined to be a scattered or reflected light signal, as such signals do not directly indicate the location of the light source from which the signal was emitted. In at least some implementations, the scatter detector 1402 can be a separate component used in conjunction with one or more optical detectors used for position tracking. In other implementations, the scatter detector 1402 can be integrated into one or more optical detectors (e.g., angle-sensitive detectors) used for position tracking. One or more scatter detectors 1402 can be used in any embodiment of the present disclosure. In addition, various machine learning or artificial intelligence-based methods can be used to process scatter detector data to improve the position tracking capabilities of the tracking systems of the present disclosure. For example, machine learning or other AI methods can be used to train the tracking systems to use polarization information to help improve tracking fidelity.

[0112] In the illustrated non-limiting example, a scatter detector 1402 is shown, and a first light source 1408 and a second light source 1410 are also shown. In practice, there can be multiple scatter detectors and multiple light sources. As a non-limiting example, the scatter detector 1402 can be located on one of an HMD and a controller, and the light sources 1408 and 1410 can be located on the other of the HMD and the controller. In at least some implementations, the scatter detector 1402 and one or more of the light sources 1408 and 1410 can be located on or coupled to a fixed object (e.g., a wall, a ceiling, a stand) or a movable object (e.g., an HMD, a controller). The scatter detector 1402 and the light sources 1408 and 1410 can be similar or identical to any of the light sources and scatter detectors discussed herein, and can be used in any implementation of the present disclosure.

[0113] In the illustrated example, the scatter detector 1402 can include an optical detector 1404, which can optionally be an angle-sensitive detector, and an optical subsystem 1406. The light source 1408 can include a light emitter 1412 (e.g., an LED) that emits light 1420 and an optical subsystem 1414, and the light source 1410 can include a light emitter 1416 that emits light 1422 and an optical subsystem 1414. Some or all of the optical subsystem 1406, the optical subsystem 1414, and the optical subsystem 1418 can be the same as or different from each other, and can each include one or more optical components. The optical subsystem 1406, the optical subsystem 1414, and the optical subsystem 1418 can be integrated with the detector 1404 and the light sources 1408 and 1410, respectively, or can be separate components. Non-limiting examples of optical components include one or more lenses, one or more polarizers, one or more wave retarders, one or more filters, one or more apertures, etc. In at least some implementations, a subset of the light sources can include one type of optical subsystem, and one or more other subsets of the light sources can include another type of optical subsystem. Similarly, a subset of the scatter detectors can include one type of optical subsystem, and one or more other subsets of the scatter detectors can include another type of optical subsystem. As an example, the optical subsystems can include filters that filter out visible light or other types of light. In addition, as described above, the optical subsystems can include components that facilitate one or more of the various types of multiplexing described above that allow multiple light sources to be illuminated simultaneously without being confused with the emitted light sources.

[0114] The designs of optical subsystem 1406, optical subsystem 1414, and optical subsystem 1418 can be coordinated such that scatter detector 1402 can be used to detect whether light from light sources 1408 and 1410 has been scattered or reflected, or whether the light has arrived directly at the scatter detector without scattering or reflection. For example, scatter detector 1402 can be used to detect changes in the type or degree of polarization of light emitted by the light sources due to scattering or reflection. In the illustrated example, light 1420 from light source 1408 is received directly by scatter detector 1402, while light 1422 from light source 1410 is reflected off surface 1423 as light 1424 received by scatter detector 1402. In this example, light 1420 indicates the relative position of light source 1408 with respect to scatter detector 1402, while light 1424 reflected from surface 1423 does not indicate the relative position of light source 1410 with respect to scatter detector 1402. Thus, by detecting that light 1424 has been scattered or reflected, when performing position tracking, the tracking system can ignore or suppress light signals from one or more sensors, such as sensors positioned in a similar orientation as the scatter detector, thereby improving the position tracking capabilities of the system.

[0115] There are a variety of configurations that can allow the scatter detector 1402 to be able to detect whether light from the light sources has been scattered or reflected and, as such, should be ignored by one or more detectors. Generally, in at least some implementations, the light emitted by the light sources 1408 and 1410 can be polarized in a determined manner by the optical systems 1414 and 1418, respectively, and the scatter detector 1402 can be configured to distinguish between light received directly from the light sources 1408 and 1410 and light from the light sources that has been scattered or reflected before being received by the scatter detector. For example, the type or degree of polarization of light from the light sources can change due to scattering or specular reflection, and the scatter detector 1402 can be configured to detect such a change. As one non-limiting example, the optical subsystems 1414 and 1418 of the light sources 1408 and 1410, respectively, can include one of a right- or left-handed circular polarizer, and the optical subsystem 1406 of the scatter detector 1402 can include the other of the right- or left-handed circular polarizer. For example, the optical subsystems 1414 and 1418 of the light sources 1408 and 1410, respectively, can include right-handed circular polarizers, and the optical subsystem 1406 of the scatter detector 1402 can include a left-handed circular polarizer. In such a configuration, the optical subsystem 1406 of the scatter detector 1402 can be used to detect light that is reflected from a depolarizing surface (e.g., has random polarization) or from a non-depolarizing surface (e.g., glass, metal, acrylic, etc.) and is left-circular polarized after reflection. If such light is above a determined threshold, the tracking system can ignore signals from one or more detectors that are determined to also likely receive the reflected or scattered light.

[0116] In Figure 15 An example of such a configuration is shown in the illustration 1500, Figure 15 A scatter detector 1502 and a light source 1504 are shown. The light source 1504 includes a light emitter 1506 (e.g., LED) and an optical subsystem that includes a right-handed circular polarizer 1508. In this implementation, the circular polarizer 1508 includes a linear polarizer 1510 and a quarter- wavelength retarder or waveplate 1512 and provides light 1522 with a right-handed circular polarization.

[0117] The scatter detector 1502 includes an optical detector 1514 (e.g., four-cell detector, single-cell detector) and an optical subsystem that includes a left-handed circular polarizer 1516. The left-handed circular polarizer 1516 includes a quarter- wavelength retarder or waveplate 1518 and a linear polarizer 1520. Because the scatter detector 1502 includes a circular polarizer that is counter-rotating to the circular polarizer 1508 of the light source, the scatter detector will detect light that is reflected via specular reflection due to the circular polarized light of the reflection switching to the opposite handedness (i.e., from right-handed to left-handed in this example).

[0118] In operation, when scatter detector 1502 detects light that has been scattered or reflected (e.g., above a determined threshold), the tracking system can suppress or ignore light from one or more optical sensors that can have received the same light (e.g., sensors positioned or oriented similarly to the scatter detector).

[0119] Figure 16 Information 1600 showing a front view of an example HMD device 1644 when worn on the head of a user 1642. HMD device 1644 includes a front-facing structure 1643 that supports a front-facing or forward-facing camera 1646 and a plurality of angle-sensitive detectors 1648a-1648f (collectively, 1648) of one or more types. As one example, some or all of angle-sensitive detectors 1648 can help determine the location and orientation of device 1644 in space, such as light sensors to detect and use light information emitted from one or more external devices (not shown, e.g., a base station 214, a controller) of a system 2100. Angle-sensitive detectors 1648 can be any type of detector for detecting an angle of arrival of light emitted from a light source. Non-limiting examples of angle-sensitive detectors include photodiode detectors (e.g., dual cell detectors, quadrant cell detectors), position-sensitive detectors using resistive sheets, etc. Figure 2

[0120] ​As shown, forward-facing cameras 1646 and angle-sensitive detectors 1648 are directed forward toward the actual scene or environment (not shown) in which user 1642 is operating HMD device 1644. More generally, angle-sensitive detectors 1648 can be directed to other areas (e.g., up, down, left, right, back) to detect light from various sources, such as a controller (e.g., held by user 1642) or objects mounted at various locations (e.g., walls, ceilings). The actual physical environment can include, for example, one or more objects (e.g., walls, ceilings, furniture, stairs, cars, trees, tracking markers, light sources, or any other type of object). The particular number of sensors 1648 can be fewer (e.g., 2, 4) or more (e.g., 10, 20, 30, 40) than the number of sensors depicted. HMD device 1644 can further include one or more additional components not attached to the forward-facing structure (e.g., inside the HMD device), such as an IMU (inertial measurement unit) 1647 electronic device that measures and reports the particular forces, angular rates, and / or magnetic fields around the HMD device 1644 (e.g., using a combination of accelerometers and gyroscopes, and optionally a magnetometer). HMD device 1644 can also include additional components not shown, including one or more display panels and optical lens systems that are oriented toward the user's eyes (not shown), and optionally one or more attached internal motors to change the alignment or other positioning of the one or more optical lens systems and / or display panels within the HMD device.

[0121] The illustrated example of HMD device 1644 is supported at least in part on the head of user 1642 by one or more straps 1645 that are attached to the housing of HMD device 1644 and extend fully or partially around the user's head. Although not illustrated here, HMD device 1644 can also have one or more external motors, such as attached to one or more straps 1645, and automatic corrective actions can include using such motors to adjust such straps in order to modify the alignment or other positioning of the HMD device on the user's head. It will be appreciated that the HMD device can include other support structures (e.g., nose pieces, chin straps, etc.) not shown here, either in addition to or instead of the illustrated straps, and some embodiments can include motors attached to one or more such other support structures to similarly adjust their shape and / or location in order to modify the alignment or other positioning of the HMD device on the user's head. Other display devices that are not secured to the user's head can similarly be attached to or be part of one or more structures that affect the positioning of the display device, and in at least some embodiments can include motors or other mechanical actuators to similarly modify their shape and / or location in order to modify the alignment or other positioning of the display device relative to one or more pupils of one or more users of the display device.

[0122] The HMD device 1644 also includes a plurality of scatter detectors 1650, 1652, and 1666. The scatter detectors 1650, 1652, and 1666 can be similar or identical to any of the scatter detectors discussed herein and can be effective to detect whether light 1660, 1664, and 1672 respectively associated with the HMD device 1644 from the light sources 1658, 1662, and 1670 has been reflected or scattered from a surface before reaching the HMD device. As discussed above, upon detection of light that has been scattered, sensor data from one or more sensors determined to likely receive the same light can be ignored.

[0123] In at least some implementations, a single scatter detector can be provided for all of the detectors 1648. In other implementations, a separate scatter detector can be provided for each of the detectors 1648, or a scatter detector can be included as part of one or more of the detectors 1648. In the simplified example illustrated, the scatter detector 1650 located to the right of the forward-facing structure 1643 corresponds to the detectors 1648a, 1648b, and 1648e, which are used to detect light from a light source (e.g., light source 1658) in the right-side region 1654 of the user 1642. That is, if the scatter detector 1650 detects light that has been reflected or scattered, the tracking system can ignore signals from one or more of the detectors 1648a, 1648b, and 1648e that are determined to likely receive the same light due to their similar orientation to the scatter detector 1650. Similarly, the scatter detector 1652 located to the left of the forward-facing structure 1643 corresponds to the detectors 1648c, 1648d, and 1648g, which are used to detect light from a light source (e.g., light source 1662) in the left-side region 1656 of the user 1642. The scatter detector 1666 on the upper region of the forward-facing structure 1643 corresponds to the detector 1648f, which is used to detect light from a light source (e.g., light source 1670) in the region 1668 above the user 1642. As discussed above, multiplexing (e.g., time, wavelength, pattern, code) can be used to allow the system to know which detector 1648, 1650, 1652, and 1666 received light from which light source or group of light sources.

[0124] Figure 17A perspective view of an angle-sensitive or scatter detector 1700 that can be used in one or more implementations of the present disclosure is shown. In this non-limiting example, the scatter detector 1700 includes a four-quadrant cell ("four-way cell") photodiode that includes four separate photodiode active regions or elements 1702A-1702D separated by small gaps on a common substrate 1704. It should be appreciated that other types of detectors can also be used, such as photodiode detectors with fewer or more cells, position-sensitive detectors, etc.

[0125] In the illustrated non-limiting example, the active region (e.g., anode) of each element 1702A-1702D is individually addressable, such that a light spot illuminating a single quadrant can be electrically characterized as being in only that quadrant. As the light spot translates across the detector 1700, the energy of the light spot is distributed between adjacent elements 1702A-1702D, and the difference in electrical contribution to each element defines the relative position of the light spot with respect to the center of the detector. The relative intensity distribution across the elements 1702A-1702D can be used to determine the position of the light spot.

[0126] In this simplified example, the detector 1700 includes an opaque cover or shroud 1710 having an aperture 1708 therein that allows light 1714 from a light source 1712 to pass therethrough. As shown, the light 1714 forming a light spot 1706 through the aperture 1708 can be electrically characterized to determine the angle of the light 1714, and thus the angle of the light source 1712 with respect to the detector 1700. As described below, the systems and methods of the present disclosure can utilize multiple light sources and detectors to determine the position of components of an HMD system.

[0127] In the illustrated example, a first circular polarizer 1716 is located proximate to (e.g., adjacent to) the light source 1712, while a second polarizer 1718 is located proximate to the detector 1700. In at least some implementations, light emitted by the light source 1712 can be polarized in a determinative manner by the first circular polarizer 1716, and the second circular polarizer 1718 of the scatter detector 1700 can be configured to distinguish between light received directly from the light source 1712 and light from the light source that was scattered or reflected before being received by the scatter detector 1700. As one non-limiting example, one of the first and second circular polarizers 1716, 1718 can include one of a right- or left-circular polarizer, respectively, while the other of the first and second circular polarizers 1716, 1718 can include the other of the right- or left-circular polarizer. By way of example, the first circular polarizer 1716 of the light source 1712 can include a right-circular polarizer, while the second circular polarizer 1718 of the scatter detector 1700 can include a left-circular polarizer. In this configuration, the second circular polarizer 1718 of the scatter detector 1700 can be used to detect light that is reflected from a depolarizing surface (e.g., having random polarization) or light that is reflected from a non-depolarizing surface (e.g., glass, metal, acrylic, etc.) and is left-circular polarized after reflection. If such light is above a determinative threshold, the tracking system can ignore signals from one or more detectors that are determined to also likely receive reflected or scattered light.

[0128] Figure 18 is a flowchart of an example method 1800 of operating an HMD system to track the position of HMD components during use. The method 1800 can be performed, for example, by the position tracking system or module 512 of the HMD system 500 shown. Figure 5 As described above, the method 1800 can be used to track the position of any component, such as an HMD device that is wearable on a user's head, one or more handheld controllers, etc.

[0129] Implementations of the illustrated method 1800 begin at act 1802, where a first HMD system component having a plurality of angle-sensitive detectors is provided. The plurality of angle-sensitive detectors can be used to detect light emitted from one or more light sources, which can be fixedly positioned (e.g., mounted to a wall or ceiling) or movably positioned (e.g., coupled to an HMD headset or controller). In operation, each of the plurality of angle-sensitive detectors captures sensor data in a respective field of view of the angle-sensitive detector at a frame rate. The sensor data can include any type of data that can be used by control circuitry (e.g., a processor) to detect the presence and direction of a light source relative to the angle-sensitive detector. In at least some implementations, each of the angle-sensitive detectors can include one or more sensors (e.g., photodiodes) with image sensing circuitry and optional image processing circuitry thereon. The angle-sensitive detectors can output relatively raw data (e.g., light intensity or power data) or processed data (e.g., angle of incidence data).

[0130] At 1804, a second HMD system component including a plurality of light sources (e.g., near-IR LEDs) can be provided. The second HMD system component can include a controller, an HMD headset, or light sources located in a fixed position (e.g., ceiling, wall).

[0131] At 1806, at least one processor of the HMD system can cause the light sources to emit light. The light sources can be illuminated in such a way that each of the angle-sensitive detectors can detect light from a single light source at a time, or more generally, in such a way that the system is able to determine from which light source the light detected by the angle-sensitive detectors was received. This can be accomplished by multiplexing the illumination of the light sources using any appropriate type of multiplexing, such as time multiplexing, wavelength multiplexing, frequency multiplexing, polarization multiplexing, or other techniques that allow the system to know the light source from which light is received during use by each angle-sensitive detector.

[0132] As an example of time multiplexing, the at least one processor can illuminate only a subset of the light sources (e.g., one, two, four) at a time. By way of example, the at least one processor can sequentially illuminate the light sources one subset at a time, and collect sensor data in response to each light source.

[0133] As an example of wavelength multiplexing, different subsets of the light sources can emit different wavelengths of light, and different subsets of the angle-sensitive detectors can be used to sense different wavelengths of light. Thus, light sources having different wavelengths can be illuminated and detected simultaneously by respective wavelength-sensitive detectors.

[0134] As an example of frequency multiplexing, a subset of the light sources can be illuminated in a determined pattern or frequency that can be detected by the angle-sensitive detectors to identify the particular light source of the light.

[0135] As an example of polarization multiplexing, a subset of the light sources can be polarized differently (e.g., linear, circular), and a corresponding subset of the angle-sensitive detectors can be configured to detect certain polarized light (e.g., using polarizers that pass light having a corresponding polarization), which allows multiple light sources to be illuminated simultaneously.

[0136] Other non-limiting example techniques for illuminating the light sources can include frequency division multiple access or wavelength division multiple access (FDMA or WDMA), time division multiple access (TDMA), code division multiple access (CDMA), orthogonal frequency division multiple access (OFDMA), etc.

[0137] At 1808, at least one processor associated with the HMD system can receive sensor data from the plurality of angle-sensitive detectors. As described above, for each angle-sensitive detector, the sensor data can indicate an angle of arrival of light emitted from a known light source. At 1810, at least one processor associated with the HMD system can optionally receive sensor data from an inertial measurement unit (IMU) that is operable to provide inertial tracking capabilities or sensor data from one or more additional sensors.

[0138] At 1812, at least one processor associated with the HMD system can process the received sensor data, including detection of corrupted data, as further discussed below. For example, the at least one processor can fuse some or all of the sensor data together to track one or more features present in an environment in which the HMD system is operating. The sensor data can include sensor data from the plurality of angle-sensitive detectors, and optionally sensor data from an IMU, camera, or other sensor data. For example, the at least one processor can use a machine learning model (e.g., model 606) or another solver to process the sensor data. As further discussed below, for example, in at least some implementations, the at least one processor can ignore data from one or more sensors determined to be corrupted, e.g., data from light that is not likely to have been received directly from a light source of the HMD system, but rather has been scattered, reflected, or received from another light source.

[0139] At 1814, at least one processor associated with the HMD system can track a position (e.g., location, orientation, or movement) of a component of the HMD system in real-time during use of the HMD system by a user in an environment. As described above, the method 1800 can continue during operation of the HMD to continuously track the position of the component of the HMD system.

[0140] In processing received sensor data, the control circuit can identify one or more corrupted sensor data samples, where each of the one or more corrupted sensor data samples includes a sensor data sample from one of the plurality of angle-sensitive detectors that is identified as likely not to represent light received by that one of the plurality of angle-sensitive detectors directly from one or more of the plurality of light sources. In at least some implementations, the control circuit is configured to ignore the corrupted sensor data samples during the tracking process, and can continue to ignore samples from that optical detector for a fixed or variable period of time.

[0141] Identification of the corrupted sensor data samples can be based at least in part on a known geometry of at least one of the first head-mounted display system component and the second head-mounted display system component. For example, a projection model of at least one of the first head-mounted display system component or the second head-mounted display system component can be utilized to determine which of the plurality of angle-sensitive detectors is likely not to receive light directly from one or more of the plurality of light sources, and data from those detectors can be ignored for a period of time. For example, the received sensor data samples can be compared to the at least one projection model, and received sensor data samples that do not match the at least one projection model within a defined threshold can be identified as corrupted sensor data samples that should be ignored.

[0142] Additionally or alternatively, identification of the one or more corrupted sensor data samples can be based at least in part on one or more of a past position or orientation of at least one of the first head-mounted display system component and the second head-mounted display system component, a current position or orientation of at least one of the first head-mounted display system component and the second head-mounted display system component, or a predicted future position or orientation of at least one of the first head-mounted display system component and the second head-mounted display system component.

[0143] In at least some implementations, the number of samples or the period of time during which one or more detectors are disabled can be selectively changed based on various criteria, such as actual or predicted movement (e.g., direction, velocity, rotation) of at least one of the first head-mounted display system component or the second head-mounted display system component. As an example, a comparison to a projection model can indicate that a detector is occluded by an object (e.g., a wall, a person, other component) or is turned away from a light source of the HMD system, and thus is unlikely to receive light directly from one of the light sources of the HMD system for a period of time. The control circuit can track the position or motion of the one or more components to determine a period of time after which the detector is expected to again receive light from at least one of the plurality of light sources, at which point the system can again use samples from the detector for tracking purposes.

[0144] A head-mounted display system component can include a head-mounted display device that is wearable on a user's head, a controller, a base station, or other HMD system component. As discussed elsewhere herein, to process received sensor data, a control circuit can provide the received sensor data as input to one or more trained machine learning models.

[0145] Figure 19 is a flowchart of a method of adaptively adjusting the brightness of a plurality of light sources of a position tracking system of an HMD system according to example embodiments of the present disclosure. The method 1900 can be performed by, for example, a position tracking system or module 512 of an HMD system 500 as shown in FIG. 5. As discussed above, the method 1900 can be implemented during tracking of the position of any component, such as an HMD device wearable on a user's head, one or more handheld controllers, etc., for example, in conjunction with the method 1800 of FIG. 18. Figure 5 Figure 18 Advantageously, the adaptive brightness features discussed herein can provide improved performance by utilizing a relatively greater dynamic range of the detector, and can also increase battery life by reducing power consumption.

[0146] The method 1900 begins at 1902, where a control circuit of an HMD system receives optical detector data from an optical detector (e.g., photodiode, angle sensitive detector). At 1904, the control circuit can process the received optical detector data, and at 1906, the control circuit can adaptively adjust the brightness of at least one light source of a plurality of light sources based at least in part on the processed optical detector data. In at least some implementations, the control circuit adaptively adjusts the brightness of at least one of the one or more light sources based on a dynamic range of the optical detector, for example, to maximize the dynamic range of the optical detector. To adjust the brightness, the pulse width of the signal provided to the one or more light sources can be selectively adjusted.

[0147] In at least some implementations, the control circuit can disable one or more light sources ("dark measurement"), and receive optical data from the optical detector while the one or more light sources are disabled. Such a feature can allow the brightness of the one or more light sensors to be adjusted based on the level of ambient light in the environment in which the HMD system is operating.

[0148] Figure 20 ​An example method 2000 for this feature is illustrated. At 2002, the control circuitry can disable one or more (e.g., all) light sources of the HMD system. At 2004, the control circuitry can capture sensor data from an optical detector while one or more light sources are disabled. At 2006, the control circuitry can adjust the brightness settings of one or more light sources based on the captured sensor data. At 2008, the control circuitry can optionally adjust the rate of sensor data capture for brightness adjustment based on changes to one or more parameters, such as movement of one or more components, elapsed time, number of samples, amount of ambient light, detected changes in ambient light, etc. To adaptively adjust the brightness of one or more light sources, the received optical detector data can be provided as input to one or more trained machine learning models, as discussed elsewhere herein. In at least some implementations, the control circuitry can periodically perform light measurements (e.g., every 5 samples, every 50 samples) and adjust the brightness of the light source after each measurement based on the results of the measurements.

[0149] Figure 21 This is a flowchart of a method for compensating for non-uniform brightness of a light source in a position tracking system of an HMD system according to an exemplary embodiment of this disclosure. In practice, an angle-sensitive detector such as a quadruple photodiode (QPD) may include multiple channels read sequentially in time (e.g., via multiplexing). Therefore, since measurements from all channels of the detector are not captured simultaneously, any non-uniformity in the brightness of the light source can lead to inaccurate measurements. For example, when an LED is lit for one lighting cycle, its intensity can vary over the lighting cycle due to heat and other effects (e.g., "droop"). As described below, method 2100 compensates for this effect, which advantageously provides more accurate measurements for position tracking.

[0150] Method 2100 can be derived from, for example Figure 5 The position tracking system or module 512 of the HMD system 500 shown is executed. As described above, method 2100 can be implemented during the tracking of the position of any component, such as an HMD device that can be worn on a user's head, one or more handheld controllers, etc., for example, in combination with Figure 18 Method 1800.

[0151] The method 2100 begins at 2102, where the control circuit can cause one or more light sources to emit light during an illumination period. At 2104, the control circuit can receive sensor data from an angle-sensitive detector, the angle-sensitive detector data including sensor cell samples from a plurality of sensor cells captured sequentially during the illumination period, as described above. For example, the control circuit can include an analog-to-digital converter (ADC), and a multiplexer can be used to sequentially read samples from each of a plurality (e.g., four) of the sensor cells of the angle-sensitive detector. Other methods capable of providing an illumination profile for the light sources with respect to time can be used to capture the illumination data.

[0152] At 2106, the control circuit can process the received sensor data, including determining a correction to account for non-uniform brightness of the one or more light sources during sequential capture of the sensor cell samples. At 2108, the control circuit can apply calibration data to the sensor cell samples using the determined correction, and track a position of the first head-mounted display system assembly using the calibrated sensor cell samples. For example, the calibration data can represent a characteristic slope of brightness of the one or more light sources during the illumination period.

[0153] In at least some implementations, the control circuit can iteratively determine updated calibration data, and can track the position of the first head-mounted display system assembly using the updated calibration data. For example, to determine updated calibration data, the control circuit can disable the one or more light sources for a calibration period, sequentially capture sensor cell samples from the plurality of sensor cells for the calibration period, interpolate the captured sensor cell samples, and determine updated calibration data based on the interpolation of the captured sensor cell samples.

[0154] The calibration data can additionally or alternatively be determined during a manufacturing or design process of the HMD system. For example, a characteristic slope or function of a light source (e.g., LED) can be empirically determined using an angle-sensitive detector or other type of optical detector, and this information can be provided to the HMD system to compensate for non-uniformity of the light source (or similar or identical light sources) during operation of the HMD system. More generally, a method of calibrating a head-mounted display system assembly can include causing one or more light sources to emit light for an illumination period, sequentially capturing sensor cell samples from a plurality of sensor cells of an angle-sensitive detector, processing the received sensor cell samples to generate calibration data that accounts for non-uniform brightness of the one or more light sources during the illumination period, and storing the calibration data in a non-transitory processor-readable storage medium for subsequent use in tracking a component of at least one head-mounted display system.

[0155] Figure 22is a flowchart of a method of adaptively enabling and disabling components of a tracking subsystem of an HMD system according to example embodiments of the present disclosure. In the context of light sources (e.g., LEDs), implementation of the method 2200 can be referred to as "adaptive startup" of the LEDs, for example, which can reduce power consumption and thus extend battery life. As discussed above, the method 2200 can be implemented during tracking of the position of any component, such as an HMD device wearable on a user's head, one or more handheld controllers, etc., for example, in conjunction with Figure 18 the method 1800 of FIG. 18.

[0156] The method 2200 begins at 2202, where the control circuit causes one or more of a plurality of light sources to emit light. At 2204, the control circuit receives sensor data from one or more of a plurality of optical detectors and tracks a position of a first head-mounted display system component based at least in part on the received sensor data, as discussed elsewhere herein.

[0157] At 2206, the control circuit can process the received sensor data, including determining whether to disable any of the optical detectors or light sources based on determined disabling criteria. At 2208, the control circuit can disable each of the optical detectors or light sources that meet the disabling criteria for a respective disabling time period during tracking of the position of the first head-mounted display system component.

[0158] In general, the disabling criteria provide or facilitate a determination that light emitted by a light source is unlikely to be received by any optical detector of the first head-mounted display system component. For example, the disabling criteria can determine that a particular light source is facing away from an optical detector of the HMD system, or is occluded by an object (e.g., a person, a component, furniture, a wall) in an environment in which the HMD system is operating. Similarly, the control circuit can determine that an optical detector is unlikely to receive light from a light source, and thus can be disabled for a fixed or variable time period.

[0159] The disabling criteria can be based at least in part on a determined relative position or motion between the first head-mounted display system component and a second head-mounted display system component, such as a relative position or motion between a controller and a headset, a relative position or motion between a controller and a base station, or a relative position or motion between a headset and a base station. As discussed above, one or more projection models can be used to assess whether light from a light source is predicted to be received directly by an optical sensor of the HMD system. In at least some implementations, position tracking information can be used to predict when a particular component (e.g., light source, detector) can be disabled, and when such a component should be re-enabled.

[0160] In at least some implementations, the disabling criteria provide or contribute to a determination that light emitted by each light source of a first subset of the plurality of light sources is likely to be received by at least one of the optical detectors of the first head-mounted display system assembly, and the disabling criteria are used to disable a second subset of the light sources of the first subset of light sources for respective disabling time periods. For example, the system can determine that a first subset of four spaced-apart light sources is likely to be detected by one or more detectors, and can disable two light sources of the first subset (i.e., a second subset) so that only two light sources are enabled for a period of time. This feature can increase battery life while still providing light sources that can be detected by the optical sensors. In at least some implementations, the light sources of the first subset of light sources that are not disabled can be light sources that are spaced apart from one another relative to provide a relatively large separation angle with respect to the optical detectors that detect light emitted by the light sources, which can improve measurement accuracy.

[0161] Figure 23 is a flowchart of a method of operating a position tracking system of an HMD system by fusing inertial sensor data, optical sensor data, and image data to track a position, orientation, and / or motion of a component of the HMD system during use in accordance with example embodiments of the present disclosure. As described above, the method 2300 can be implemented during tracking of the position of any component, such as an HMD device wearable on a user’s head, one or more handheld controllers, etc., e.g., in conjunction with Figure 18 the method 1800.

[0162] The head-mounted display system can include a first head-mounted display system assembly, an inertial measurement unit (IMU) carried by the first head-mounted display system assembly, a plurality of angle-sensitive optical detectors (or other types of optical detectors) carried by the first head-mounted display system assembly, and at least one camera carried by the first head-mounted display system assembly. At 2302, control circuitry associated with the HMD system can receive inertial sensor data from the inertial measurement unit. At 2304, the control circuitry can receive optical sensor data from one or more of the plurality of angle-sensitive optical detectors or other types of optical detectors. At 2306, the control circuitry can receive image sensor data from the camera. As an example, the camera can be a forward-facing camera of an HMD device wearable on a user’s head.

[0163] At 2308, the control circuit can process or fuse the received inertial sensor data, optical sensor data, and image sensor data. For example, the control circuit can utilize one or more sensor fusion algorithms, including but not limited to a central limit theorem algorithm, a Kalman filter, a Bayesian network, a Dempster-Shafer algorithm, or a convolutional neural network. At 2310, the control circuit can track a position of the first head-mounted display system component based at least in part on the processing of the received inertial sensor data, optical sensor data, and image sensor data. In at least some implementations, to process the received inertial sensor data, optical sensor data, and image sensor data, the control circuit can provide the inertial sensor data, optical sensor data, and image sensor data as input to one or more trained machine learning models, as discussed elsewhere herein (see, e.g., Figure 6

[0164] The foregoing detailed description has set forth various implementations of the devices and / or processes via the use of block diagrams, schematics, and examples. As will be appreciated by those skilled in the art, the foregoing merely sets forth various implementations, and that within the scope of the subject matter disclosed herein, those skilled in the art will recognize that the systems and / or processes described herein can be implemented in any number of ways, and that the illustrative implementations described herein are not the only way in which such systems and / or processes can be implemented. It should be understood that any combination of the above- described elements can be implemented as a means for achieving the functionality described herein. In the claims, means- plus-function clauses can be used in

[0165] Those skilled in the art will recognize that many of the methods or algorithms described herein can employ additional acts, may- omit some acts, and / or may- perform acts in an order different from that specified herein.

[0166] In addition, those skilled in the art will appreciate that the mechanisms taught herein can be distributed as an program product in a variety of forms, and that the illustrative implementations described herein apply regardless of the particular type of signal bearing media used to actually carry out the distribution. Examples of a signal bearing media include, but are not limited to, the following: a recordable type medium such as a floppy disk, a hard disk drive, a CD-ROM, a digital tape, and a computer memory. ​

[0167] The various implementations described above can be combined to provide further implementations. All of the U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, foreign patent applications, and non-patent publications referred to in this specification are incorporated by reference herein in their entireties, if any. Various aspects of the implementations can be modified, if necessary, to employ the systems, circuits, and concepts of the various patents, applications, and publications to provide yet further implementations.

[0168] These and other changes can be made to the implementations in light of the above Detailed Description. The terms used in the following claims should not be construed to limit the claims present application to the specific implementations disclosed in the specification and the claims. Rather, the scope of the application is to be determined entirely by the following claims, which are to be construed in accordance with established doctrines of claim interpretation.

Claims

1. A head-mounted display system, comprising: a first head-mounted display system component; a plurality of angle-sensitive detectors carried by the first head-mounted display system component, each of the plurality of angle-sensitive detectors, in operation, capturing sensor data indicative of an angle of arrival of light emitted from one or more light sources; a second head-mounted display system component comprising one or more light sources; and control circuitry to: cause the one or more light sources to emit light; receive sensor data from one or more of the plurality of angle-sensitive detectors, the sensor data comprising a plurality of sensor data samples; process the received sensor data, including identifying one or more corrupted sensor data samples, each of the one or more corrupted sensor data samples comprising a sensor data sample from one of the plurality of angle-sensitive detectors identified as likely not representing light received directly by the one of the plurality of angle-sensitive detectors from the one or more light sources; and track a position of the second head-mounted display system component based at least in part on the processing of the received sensor data wherein the identification of the one or more corrupted sensor data samples is based at least in part on a known geometry of at least one of the first head-mounted display system component and the second head-mounted display system component; or wherein the identification of the one or more corrupted sensor data samples is based at least in part on one or more of: a past position or orientation of at least one of the first head-mounted display system component and the second head-mounted display system component; a current position or orientation of at least one of the first head-mounted display system component and the second head-mounted display system component; a predicted future position or orientation of at least one of the first head-mounted display system component and the second head-mounted display system component; or wherein the identification of the one or more corrupted sensor data samples comprises utilizing at least one projection model of the first head-mounted display system component or the second head-mounted display system component to determine which of the plurality of angle-sensitive detectors likely did not receive light directly from the one or more of the plurality of light sources. To track the position of the second head-mounted display system component, the control circuitry ignores the corrupted sensor data samples.

2. The head-mounted display system of claim 1, wherein, Utilizing at least one projection model comprises comparing the sensor data samples to the at least one projection model; and identifying the sensor data samples that do not match the at least one projection model as corrupted sensor data samples.

3. The head-mounted display system of claim 1, wherein, Identifying the sensor data samples that do not match the at least one projection model as corrupted sensor data samples comprises identifying the sensor data samples that do not match the at least one projection model within a defined threshold as corrupted sensor data samples.

4. The head-mounted display system of claim 3, wherein, 5. The head-mounted display system of claim 1, further comprising: ​ For (i) a determined number of future sensor data samples, or for (ii) a period of time, one or more of the plurality of angle sensitive detectors is disabled, the one or more of the plurality of angle sensitive detectors generating identified corrupted sensor data samples.

6. The head-mounted display system of claim 5, wherein, Based on movement of at least one of the first head mounted display system component or the second head mounted display system component, the determined number of future sensor data samples or the period of time is selectively changed.

7. The head-mounted display system of claim 1, wherein, The second head mounted display system component comprises a head mounted display device or a handheld controller that is wearable on a head of a user.

8. The head-mounted display system of claim 1, wherein, Each of the plurality of angle sensitive detectors comprises one of a photodiode detector or a position sensitive detector.

9. The head-mounted display system of claim 1, wherein, Each of the plurality of angle sensitive detectors comprises a photodiode detector having at least four cells.

10. The head-mounted display system of claim 1, wherein, The first head mounted display system component comprises one of a head mounted display device, a controller, or a base station, and the second head mounted display system component comprises another one of a head mounted display device, a controller, or a base station.

11. The head-mounted display system of claim 1, wherein, At least one of the first head mounted display system component or the second head mounted display system component comprises a component that is fixed at a location proximate to an environment in which the head mounted display system operates.

12. The head mounted display system of claim 1, further comprising: a scatter detector that captures scatter detector data that is indicative of whether light received at one or more of the plurality of angle sensitive detectors has been reflected or scattered before reaching the one or more of the plurality of angle sensitive detectors, wherein the scatter detector data is used for identification of the one or more corrupted sensor data samples.

13. The head-mounted display system of claim 12, wherein, The control circuitry processes the scatter detector data to identify the one or more corrupted sensor data samples and to ignore the corrupted sensor data samples during tracking of a position of the second head mounted display system component.

14. The head mounted display system of claim 1, further comprising: a plurality of scatter detectors, each of the scatter detectors capturing scatter detector data that is indicative of whether light received at one or more of the plurality of angle sensitive detectors has been reflected or scattered before reaching the one or more of the plurality of angle sensitive detectors, wherein the scatter detector data is used for identification of the one or more corrupted sensor data samples.

15. The head-mounted display system of claim 1, wherein, To process the received sensor data, the control circuitry provides the received sensor data as input to a trained machine learning model.

16. The head-mounted display system of claim 1, wherein, In operation, the control circuitry causes the one or more light sources to emit light using multiplexing.

17. The head-mounted display system of claim 16, wherein, The multiplexing comprises at least one of time multiplexing, wavelength multiplexing, frequency multiplexing, or polarization multiplexing.

18. The head-mounted display system of claim 1, wherein, In operation, the control circuitry causes the one or more light sources to emit light using at least one of wavelength division multiplexing, time division multiplexing, code division multiplexing, or orthogonal frequency division multiplexing.

Images