Position tracking system and method for head-mounted display systems

Abstract

Systems and methods for tracking the position of one or more objects, such as components of a head-mounted display (HMD) system. The one or more objects may carry multiple angle-sensing optical detectors. Each of the optical detectors may include an optical subsystem configured to vary at least one of the phase or intensity of light provided on the optical detector. The optical subsystem may include one or more of a diffractive optical element, a lens array, an intensity mask, a phase mask, or the like. The optical detector may comprise a photodetector including multiple optically active areas, such as a quadrant photodetector, an image sensor having an array of photodiodes, or the like. A control circuit may cause a light source to emit light and receive sensor data from the multiple optical detectors. The control circuit may process the sensor data to track the position of the one or more objects.

Description

[Technical Field]

[0001] The present disclosure relates generally to position tracking for objects such as head-mounted display systems and controllers associated with head-mounted display systems. [Background technology]

[0002] One current generation of virtual reality (VR) or augmented reality (AR) experiences is created using a stationary computer (such as a personal computer (PC), laptop, or game console) that is combined with and / or integrated into a smartphone and / or a head-mounted display (HMD) that may be coupled to its associated display or may be self-contained. Generally, an HMD is a display device worn on a user's head with a small display device in front of one eye (monocular HMD) or each eye (compound HMD). These display units are typically miniaturized and may include, for example, CRT, LCD, liquid crystal on silicon (LCos), or OLED technology. Binocular HMDs have the potential to display different images to each eye. This capability is used to display stereoscopic images.

[0003] The development of smartphones, high-definition televisions, and other electronic devices has increased the demand for high-performance displays. The popularity of virtual reality and augmented reality systems, particularly those using HMDs, has further increased this demand. Virtual reality systems typically completely envelop the wearer's eyes and replace the actual or physical view (or actual reality) in front of the wearer with a "virtual" reality, while augmented reality systems typically provide a semi-transparent or transparent overlay of one or more screens in front of the wearer's eyes, allowing the actual view to be augmented with additional information, and mediated reality systems may similarly present information to the viewer that combines real-world elements with virtual elements. In many virtual reality and augmented reality systems, the movement of the wearer of such a head-mounted display may be tracked in various ways, such as via sensors in the head-mounted display, a controller, or external sensors, to enable images to be displayed that reflect the user's movement and to enable an interactive environment.

[0004] Positional tracking enables an HMD system to estimate the position of one or more components relative to each other and the surrounding environment. Positional tracking may use a combination of hardware and software to achieve absolute position detection of the components of an HMD system. Positional tracking is an important technology for AR or VR systems, allowing for tracking of the movement of the HMD (and / or controller or other peripherals) in six degrees of freedom (6DOF).

[0005] Positional tracking technology may be used to change a user's viewpoint to reflect different actions, such as jumping or crouching, and may enable accurate representation of the user's hands and other objects in the virtual environment. Positional tracking may also increase the connection between the physical and virtual environments, for example, by using the position of a hand to move virtual objects with touch. Positional tracking improves a user's three-dimensional perception of the virtual environment due to parallax and aids in the perception of distance. Positional tracking may also help minimize motion sickness caused by a mismatch between what the eyes see and what the user's vestibular system senses in their ears.

[0006] There are several different methods of position tracking, which may include acoustic tracking, inertial tracking, magnetic tracking, optical tracking, combinations thereof, etc. [Brief explanation of the drawings]

[0007] In the drawings, identical reference numbers identify similar elements or acts. The sizes and relative positions of elements within the drawings are not necessarily drawn to scale. For example, the shapes and angles of various elements are not necessarily drawn to scale, and some of these elements may be arbitrarily enlarged and positioned to improve the readability of the drawings. Furthermore, the particular drawn shapes of elements are not necessarily intended to convey any information regarding the actual shape of the particular element, but may merely be selected for ease of recognition within the drawings.

[0008] [Figure 1] FIG. 1 is a schematic diagram of a networked environment including one or more systems suitable for performing at least some of the techniques described in this disclosure, including an embodiment of a tracking subsystem.

[0009] [Figure 2]FIG. 1 illustrates an example environment in which at least some of the described techniques are used in conjunction with an example head-mounted display device that is coupled to a video-rendering computing system and provides a virtual reality display to a user.

[0010] [Figure 3] 1 is a pictorial diagram of an HMD device having a binocular display subsystem and multiple angle-sensing detectors.

[0011] [Figure 4] 1 is a pictorial diagram of a controller that may be used with an HMD device.

[0012] [Figure 5] FIG. 1 is a schematic block diagram of an HMD device according to an example embodiment of the present disclosure.

[0013] [Figure 6] 1 is a schematic diagram of an environment in which machine learning techniques may be used to implement a tracking subsystem of an HMD device, according to one non-limiting example implementation.

[0014] [Figure 7] FIG. 1 is a flow diagram of a method of operating a position tracking system of an HMD system to track the position, orientation, and / or movement of components of the HMD system during use, according to an example embodiment of the present disclosure.

[0015] [Figure 8] 1 shows a perspective view of an exemplary angle-sensing detector that may be used in one or more implementations of the present disclosure.

[0016] [Figure 9] 1 shows the first linear polarizer, the spatially varying polarizer, and the second linear polarizer of the angle-sensitive photodiode structure, and the polarization of the light or light spot passing through them to reach the photodiode.

[0017] [Figure 10] 1 shows the first linear polarizer, the spatially varying polarizer, and the second linear polarizer of the angle-sensitive photodiode structure, and the polarization of the light or light spot passing through them to reach the photodiode.

[0018] [Figure 11A] FIG. 1 is a top view of an exemplary angle-sensing detector that may be used in one or more implementations of the present disclosure.

[0019] [Figure 11B] FIG. 11B is a perspective view of the angle-sensing detector shown in FIG. 11A.

[0020] [Figure 12] FIG. 1 is a simplified diagram illustrating determining the position of components of an HMD system using a light source and an angle-sensing detector, according to one non-limiting illustrated implementation.

[0021] [Figure 13] FIG. 1 illustrates an exemplary optical system of a light source and an angle-sensitive detector according to one non-limiting illustrated implementation.

[0022] [Figure 14] FIG. 1 illustrates the operation of an example scattered light detection module and light source of a tracking system according to one non-limiting illustrated implementation.

[0023] [Figure 15] FIG. 1 illustrates components of a light source and scattered light detection module of a tracking system according to one non-limiting illustrated implementation.

[0024] [Figure 16]FIG. 1 is a pictorial diagram of an HMD device having a binocular display subsystem, multiple angle-sensing detectors, and multiple scattered light detection modules that act to detect scattered or reflected light and can be used to ignore such scattered light during positional tracking of the HMD device or its components.

[0025] [Figure 17] FIG. 1 is a perspective view of components of a light source and scattered light detection module of a tracking system, according to one non-limiting illustrated implementation.

[0026] [Figure 18] FIG. 1 is a flow diagram of a method of operating a position tracking system of an HMD system to track the position, orientation, or movement of components of the HMD system during use, according to an example embodiment of the present disclosure.

[0027] [Figure 19] FIG. 1 is a flow diagram of a method for adaptively adjusting the brightness of multiple light sources or optical detectors of a position tracking system of an HMD system, according to an example embodiment of the present disclosure.

[0028] [Figure 20] FIG. 10 is a flow diagram for a method for adaptively adjusting the brightness of multiple light sources or optical detectors of a position tracking system of an HMD system based on changes in one or more parameters, according to an example embodiment of the present disclosure.

[0029] [Figure 21] FIG. 1 is a flow diagram of a method for compensating for non-uniform brightness of a light source in a position tracking system of an HMD system, according to an example embodiment of the present disclosure.

[0030] [Figure 22]FIG. 1 is a flow diagram of a method for adaptively enabling and disabling components (e.g., light sources, optical detectors) of a tracking subsystem of an HMD system, according to an example embodiment of the present disclosure.

[0031] [Figure 23] FIG. 10 is a flow diagram for a method of operating a position tracking system of an HMD system to track the position, orientation, or movement of components of the HMD system by fusing inertial sensor data, optical sensor data, and image data, according to an exemplary embodiment of the present disclosure.

[0032] [Figure 24] FIG. 1 is a perspective view of an example angle-sensing optical detector that may be used in one or more implementations of the present disclosure.

[0033] [Figure 25] FIG. 25 is a cross-sectional view of the angle-sensitive optical detector shown in FIG. 24.

[0034] [Figure 26] FIG. 25 is a top view of the angle-sensitive optical detector shown in FIG. 24.

[0035] [Figure 27] FIG. 1 is a cross-sectional view of an angle-sensitive optical detector including intensity-varying and phase-varying optics and positioned below any industrial design surface.

[0036] [Figure 28] 1 is a cross-sectional view of an angle-sensitive optical detector including intensity or phase variation optics positioned adjacent to a photodetector of the angle-sensitive optical detector.

[0037] [Figure 29] 1 is a cross-sectional view of an angle sensitive optical detector including intensity or phase varying optics positioned within the physical aperture of the angle sensitive optical detector.

[0038] [Figure 30] FIG. 1 is a cross-sectional view of an angle-sensing optical detector including a controllable mechanical actuator according to one non-limiting illustrated implementation.

[0039] [Figure 31] 1 is a cross-sectional view of an angle-sensitive optical detector including a controllable liquid crystal layer according to one non-limiting illustrated implementation.

[0040] [Figure 32] 1 is a cross-sectional view of an angle-sensitive optical detector including a reflective telescope mirror according to one non-limiting illustrated implementation.

[0041] [Figure 33] FIG. 1 is a cross-sectional view of an angle-sensitive optical detector including pancake optics according to one non-limiting illustrated implementation.

[0042] [Figure 34] 1 is a graph illustrating the angular resolution of an angle-sensitive optical detector according to one non-limiting illustrated implementation.

[0043] [Figure 35] 1 is a graph illustrating the response of an angle-sensitive optical detector according to one non-limiting illustrated implementation.

[0044] [Figure 36] FIG. 1 is a schematic block diagram of a process for determining one or more optical subsystems of an angle-sensitive optical detector, according to one non-limiting illustrated implementation. DETAILED DESCRIPTION OF THE INVENTION

[0045] In the following description, certain specific details are set forth to provide a thorough understanding of various disclosed implementations. However, those skilled in the art will recognize that implementations may be practiced without one or more of these specific details, or with other methods, components, materials, etc. In other instances, well-known structures related to computer systems, server computers, and / or communication networks have not been shown or described in detail to avoid unnecessarily obscuring the description of the implementations.

[0046] Unless the context requires otherwise, throughout this specification and the claims that follow, the word "comprising" is synonymous with "including" and is inclusive or open-ended (i.e., does not exclude additional, unrecited elements or method acts).

[0047] Throughout this specification, a reference to "one implementation" or "an implementation" means that a particular feature, structure, or characteristic described in connection with that implementation is included in at least one implementation. Thus, the appearances of the phrase "in one implementation" or "in an implementation" in various places throughout this specification are not necessarily all referring to the same implementation. Furthermore, particular features, structures, or characteristics may be combined in any suitable manner in one or more implementations.

[0048] As used in this specification and the appended claims, the singular forms "a," "an," and "the," and the plural forms include plural referents, unless the context clearly dictates otherwise. It should also be noted that the term "or" is generally used in its sense including "and / or" unless the context clearly dictates otherwise.

[0049] The headings and abstracts provided herein are for convenience only and do not interpret the scope or meaning of the implementation.

[0050] One or more implementations of the present disclosure relate to systems and methods for accurately tracking the positions of components (e.g., HMD, controller, peripherals) of a head-mounted display (HMD) system. In at least some implementations, the HMD includes a support structure that holds a forward-facing camera ("front camera" or "front camera") and multiple angle-sensing detectors or light sources. Similarly, one or more controllers may include multiple angle-sensing detectors or light detectors. In other implementations, the HMD does not include a front camera. The front camera may capture image sensor data in the front camera's field of view at a first frame rate (e.g., 30 Hz, 90 Hz). In at least some implementations, the HMD system may not include an angle-sensing detector or may include other types of optical detectors (e.g., photodiodes). Accordingly, the systems and methods described herein may utilize non-angle-sensing detectors or angle-sensing detectors, as appropriate.

[0051] As described further below, during operation, one or more fixed or movable light sources (e.g., IR LEDs) may emit light. The light sources may be coupled to the HMD, the controller, a fixed object (e.g., a base station) located in the environment, etc. Each of the plurality of angle-sensing detectors captures sensor data in its respective field of view at a second frame rate (e.g., 1000 Hz, 2000 Hz) that may be greater than the first frame rate of the forward camera (if present). In at least some implementations, the field of view of the angle-sensing detectors may be narrower than the field of view of the forward camera, although this is not required. For example, the forward camera may have a relatively wide forward camera field of view of 90°, 120°, or 150°, and each of the angle-sensing detectors may have a relatively narrow sensor IC field of view (e.g., 25°, 45°, 75°). In at least some implementations, the angle-sensing detector fields of view may collectively cover at least a majority of the forward camera field of view, or a much larger portion than the forward camera field of view, with each of the angle-sensing detector fields of view overlapping a different portion of the forward camera field of view.

[0052] During operation, at least one processor operably coupled to the plurality of angle-sensing detectors may receive sensor data capturing light from a plurality of light sources (e.g., LEDs, lasers, other light sources). The at least one processor may process the received image sensor data and track the position of a component of the head-mounted display based at least in part on the processing of the received image sensor data. For example, the at least one processor may fuse the sensor data from the angle-sensing detectors to track one or more features present in the environment. The at least one processor may process the sensor data utilizing machine learning techniques, solvers, or another method to determine the position (e.g., location, orientation, movement) of one or more components of the HMD system. In at least some implementations, the sensor data may be fused with sensor data from other sensors, such as sensor data from a forward-facing camera or an inertial measurement unit (IMU) of an HMD system component. In at least some implementations, one or more scattered light detection modules, or "scattered light detectors," may be used to detect when light is scattered or reflected before reaching one or more angle-sensing detectors; such light may be ignored by the tracking system because the angle does not accurately indicate the location of the light source from which the light was emitted. Using this technique, the accuracy of position tracking may be greatly improved. Various features of implementations of the present disclosure are described in detail below with reference to the figures.

[0053] 1 is a schematic diagram of a networked environment 100 including a local media rendering (LMR) system 110 (e.g., a gaming system) that includes a local computing system 120 suitable for performing at least some of the techniques described herein, a display device 180 (e.g., an HMD device having two display panels (one for each eye)), and one or more controllers 182. In the illustrated embodiment of FIG. 1, the local computing system 120 is communicatively connected to the display device 180 via a transmission link 115 (which may be wired or tethered, such as via one or more cables (cable 220) as shown in FIG. 2, or may alternatively be wireless). The controller 182 may be coupled to the local computing system 120 or the display device 180 via suitable wired or wireless links 186 and 184, respectively. In other embodiments, local computing system 120 may provide encoded image data for display via a wired or wireless link to a panel display device (e.g., a TV, console, or monitor), each comprising one or more addressable pixel arrays, whether in addition to or instead of HMD device 180. In various embodiments, local computing system 120 may include a general-purpose computing system, a gaming console, a video stream processing device, a mobile computing device (e.g., a mobile phone, PDA, or other mobile device), a VR or AR processing device, or other computing system.

[0054] In the illustrated embodiment, local computing system 120 has components including one or more hardware processors (e.g., central processing units, or “CPUs”) 125, memory 130, various I / O (“input / output”) hardware components 127 (e.g., a keyboard, a mouse, one or more gaming controllers, speakers, a microphone, an IR transmitter and / or receiver, etc.), a video subsystem 140 including one or more dedicated hardware processors (e.g., graphics processing units, or “GPUs”) 144 and video memory (VRAM) 148, computer-readable storage 150, and a network connection 160. Also in the illustrated embodiment, one embodiment of a tracking subsystem 135 executes in memory 130 to perform the described techniques, such as by using CPU 125 and / or GPU 144 to perform automated operations implementing at least some of the described techniques, and memory 130 may optionally further execute one or more other programs 133 (e.g., for generating video or other images to be displayed, such as game programs). As part of automated operations implementing at least some of the techniques described herein, tracking subsystem 135 and / or program 133 executing in memory 130 may store or retrieve various types of data, including data structures in storage 150, in an example database; in this example, the data used may include various types of image data information in database (“DB”) 154, various types of application data in DB 152, various types of configuration data in DB 157, and may include additional information such as system data or other information.

[0055] LMR system 110, in the illustrated embodiment, is also communicatively connected via one or more computer networks 101 and network link 102 to an exemplary network-accessible media content provider 190 that may further provide content to LMR system 110 for display, whether in addition to or instead of image generator 133. For simplicity, some details about the network-accessible media content provider are not shown, but media content provider 190 may include one or more computing systems (not shown), each of which may have components similar to those of local computing system 120, including one or more hardware processors, I / O components, local storage devices, and memory.

[0056] 1, display device 180 is shown as distinct and separate from local computing system 120, it will be appreciated that in particular embodiments, some or all components of local media rendering system 110 may be integrated or housed within a single device, such as a mobile gaming device, a portable VR entertainment system, an HMD device, etc. In such embodiments, transmission link 115 may include, for example, one or more system bus and / or video bus architectures.

[0057] As one example involving operations performed locally by local media rendering system 120, assume that the local computing system is a gaming computing system, whereby application data 152 includes one or more gaming applications executing via CPU 125 using memory 130, and various video frame display data is generated and / or processed by image generation program 133, such as in combination with GPU 144 of video subsystem 140. To provide a high-quality gaming experience, a high volume of video frame data (corresponding to a high image resolution per video frame and a high "frame rate" of approximately 60-180 such video frames per second) is generated by local computing system 120 and provided to display device 180 via wired or wireless transmission link 115.

[0058] It will also be understood that computing system 120 and display device 180 are merely exemplary and are not intended to limit the scope of the present disclosure. Instead, computing system 120 may include multiple interacting computing systems or devices and may be connected to other devices not shown, including through one or more networks such as the Internet, via the Web, or via a private network (e.g., a mobile communications network, etc.). More generally, computing systems or other computing nodes may include any combination of hardware or software capable of interacting and performing the types of functions described, including, but not limited to, desktop or other computers, gaming systems, database servers, network storage devices and other network devices, PDAs, mobile phones, wireless phones, pagers, electronic organizers, Internet appliances, television-based systems (e.g., using set-top boxes and / or personal / digital video recorders), and various other consumer products that include appropriate communications capabilities. Display device 180 may similarly include one or more devices having one or more display panels of various types and forms and, optionally, may include various other hardware and / or software components.

[0059] Additionally, functionality provided by tracking subsystem 135 may be distributed across one or more components (e.g., local and remote processing systems, HMDs, controllers, base stations) in some embodiments, and in some embodiments, some of the functionality of tracking subsystem 135 may not be provided, and / or other additional functionality may be available. It will also be understood that while various items are shown as being stored in memory or on storage while in use, these items, or portions thereof, may be transferred between memory and other storage devices for memory management or data integrity purposes. Thus, in some embodiments, some or all of the described techniques may be performed by hardware including one or more processors or other configured hardware circuits or memory or storage, such as when configured by one or more software programs (e.g., by tracking subsystem 135 or components thereof) and / or data structures (e.g., by executing software instructions of one or more software programs and / or by storing such software instructions and / or data structures). Some or all of the components, systems and data structures may be stored (e.g., as software instructions or structured data) on a non-transitory computer-readable storage medium such as a hard disk or flash drive or other non-volatile storage device, volatile or non-volatile memory (e.g., RAM), network storage device, or portable media product read by an appropriate drive (e.g., DVD disk, CD disk, optical disk, etc.) or via an appropriate connection.The systems, components, and data structures may, in some embodiments, be transmitted as a generated data signal (e.g., as part of a carrier wave or other analog or digital propagated signal) over a variety of computer-readable transmission media, including wireless-based media and wire / cable-based media, and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as a number of discrete digital packets or frames). Such computer program products may take other forms in other embodiments. Accordingly, the invention may be practiced with other computer system configurations.

[0060] 2 illustrates an example environment 200 in which at least some of the described techniques are used in conjunction with an example HMD device 202 coupled to a video rendering computing system 204 via a tethered connection 220 (or a wireless connection in other embodiments) to provide a virtual reality display to a human user 206. The user wears the HMD device 202 and receives display information of a simulated environment that differs from the actual physical environment from the computing system 204 via the HMD device, which functions as an image rendering system that provides images of the simulated environment, such as images generated by a game program and / or other software program running on the computing system, to the HMD device for display to the user. The user is further able to move around within the tracked volume 201 of the actual physical environment 200 in this example and may further have one or more I / O (“input / output”) devices, which in this example include handheld controllers 208 and 210, that enable the user to further interact with the simulated environment.

[0061] In the illustrated example, the environment 200 may include one or more base stations 214 (two are shown, labeled base stations 214a and 214b) that may facilitate tracking of the HMD device 202 or the controllers 208 and 210. As the user moves from place to place or reorients the HMD device 202, the position of the HMD device is tracked, enabling, for example, corresponding portions of the simulated environment to be displayed to the user on the HMD device; the controllers 208 and 210 may further employ similar techniques for use in tracking the position of the controller (and, optionally, to use that information to help determine or verify the position of the HMD device). After the tracked position of the HMD device 202 is known, corresponding information is transmitted via tether 220 or wirelessly to the computing system 204, which uses the tracked position information to generate one or more subsequent images of the simulated environment for display to the user.

[0062] The optical tracking described herein may be used in combination with various methods of position tracking, including, but not limited to, acoustic tracking, inertial tracking, or magnetic tracking, among others.

[0063] In at least some implementations, at least one of the HMD device 202 and the controllers 208 and 210 may include one or more light receivers or sensors that may be used to implement tracking functions or other aspects of the present disclosure. In at least some implementations, at least one of the HMD device 202, the controllers 208 and 210, or other components may include one or more light sources (e.g., LEDs) that may emit light that is detected by one or more of the light receivers. The light sources may be in a fixed location or may be on a movable component, such as the HMD device or the controller.

[0064] In at least some implementations, in addition to or instead of generating fixed point light sources, the base stations 214 may each illuminate with an optical signal across the tracked volume 201. Depending on the requirements of each particular implementation, each base station 214 may generate more than one optical signal. For example, while a single base station 214 is typically sufficient for six degrees of freedom tracking, in some embodiments, multiple base stations (e.g., base stations 214a, 214b) may be necessary or desirable to provide robust room-wide tracking for the HMD device and peripherals. In this example, optical receivers, such as angle-sensing detectors or scattered light detectors, are incorporated into the HMD device 202 and / or other tracked objects, e.g., controllers 208 and 210. In at least some implementations, the optical receivers may be paired with accelerometer and gyroscope inertial measurement units (“IMUs”) on each tracked device to support low-latency sensor fusion.

[0065] In at least some implementations, each base station 214 comprises two rotors that scan a linear beam across the tracked volume 201 on mutually orthogonal axes. At the start of each scanning cycle, the base station 214 may emit an omnidirectional light pulse (referred to as a "synchronization signal") visible to all sensors toward the tracked object. Each sensor then calculates a unique angular position in the scanning volume by timing the duration between the synchronization signal and the beam signal. Sensor distance and orientation may be determined using multiple sensors fixed to a single rigid body.

[0066] One or more sensors positioned on the tracked object (e.g., HMD device 202, controllers 208 and 210) may include optoelectronic devices capable of detecting modulated light from the rotor. For visible or near-infrared (NIR) light, silicon photodiodes and suitable amplifier / detector circuitry may be used. Because the environment 200 may contain stationary and time-varying signals (optical noise) with wavelengths similar to those of the base station 214 signal, in at least some implementations, the base station light may be modulated to facilitate distinguishing from any interfering signals and / or filtering the sensor from any wavelengths of radiation other than those of the base station signal. As described further below, at least some implementations of angle-sensing detectors are used to track one or more components of the HMD system, and one or more scattered light detectors may be used to ignore light that may be scattered or reflected before being detected by the optical detectors.

[0067] Inside-out tracking is also a type of position tracking that can be used to track the position of the HMD device 202 and / or other objects (e.g., controllers 208 and 210, tablet computers, smartphones). Inside-out tracking differs from outside-in tracking by the location of the cameras or other sensors used to determine the location of the HMD components. With inside-out tracking, the cameras or sensors are located on the HMD components or objects being tracked, while in outside-out tracking, the cameras or sensors are placed in stationary positions within the environment.

[0068] An HMD using inside-out tracking uses one or more sensors to "look out" to determine how its position changes relative to the environment. As the HMD moves, the sensors readjust its location within the room, and the virtual environment responds accordingly in real time. This type of positional tracking can be achieved with or without markers placed in the environment. A camera placed on the HMD observes features of the surrounding environment. If markers are used, they are designed to be easily detected by the tracking system and placed within a specific area. With "markerless" inside-out tracking, the HMD system determines position and orientation using distinctive features (e.g., natural features) that are naturally present in the environment. The HMD system's algorithms identify specific images or shapes and use them to calculate the device's position in space. Data from the accelerometer and gyroscope can also be used to improve the accuracy of positional tracking.

[0069] FIG. 3 illustrates information 300 showing a front view of an exemplary HMD device 344 when worn on the head of a user 342. The HMD device 344 includes a front structure 343 supporting a front or forward-facing camera 346 and one or more types of angle-sensing detectors 348a-348f (collectively 348) or other types of optical detectors. As one example, some or all of the angle-sensing detectors 348, such as optical sensors that detect and use light information emitted from one or more external devices (not shown, e.g., base station 214, controller, FIG. 2), may help determine the location and orientation of the device 344 in space. The angle-sensing detectors 348 may be any type of detector that operates to detect the angle of arrival of light emitted from a light source. Non-limiting examples of angle-sensing detectors include photodiode detectors (e.g., bicell detectors, quadrantel detectors, image sensors), position-sensing detectors using resistive sheets, etc.

[0070] As shown, the forward camera 346 and angle-sensing detector 348 are pointed forward toward an actual scene or environment (not shown) in which the user 342 operates the HMD device 344. More generally, the angle-sensing detector 348 may be pointed toward other areas (e.g., above, below, left, right, behind) and detect light from various sources, such as a controller (e.g., held by the user 342) or objects mounted in various locations (e.g., walls, ceilings). The actual physical environment may include, for example, one or more objects (e.g., walls, ceilings, furniture, stairs, cars, trees, tracking markers, light sources, or any other type of object). The particular number of sensors 348 may be fewer (e.g., 2, 4) or more (e.g., 10, 20, 30, 40) than the number of sensors shown. The HMD device 344 may further comprise one or more additional components not attached to the front structure (e.g., internal to the HMD device), such as an IMU (Inertial Measurement Unit) 347 electronic device that measures and reports specific forces, angular velocities, and / or magnetic fields surrounding the HMD device (e.g., using a combination of accelerometers and gyroscopes, and optionally magnetometers) of the HMD device 344. The HMD device 344 may further comprise additional components not shown, including one or more display panels and optical lens systems that are directed toward the user's eyes (not shown), optionally having one or more attached internal motors for changing the alignment or other positioning of one or more of the optical lens systems and / or display panels within the HMD device.

[0071] The depicted example of HMD device 344 is supported on the head of user 342 based at least in part on one or more straps 345 attached to the housing of HMD device 344 and extending wholly or partially around the user's head. Although not shown here, HMD device 344 may further include one or more external motors, such as those attached to one or more of straps 345, and the automated corrective action may include using such motors to adjust such straps to correct alignment or other positioning of the HMD device on the user's head. It will be understood that HMD devices may include other support structures not shown here (e.g., nosepieces, chin straps, etc.), whether in addition to or instead of the illustrated straps, and that some embodiments may include motors attached to one or more such other support structures to similarly adjust their shape and / or location to correct alignment or other positioning of the HMD device on the user's head. Other display devices that are not fixed to the user's head may similarly be attached to or be part of one or more structures that affect the positioning of the display device, and in at least some embodiments may include motors or other mechanical actuators that similarly modify their shape and / or location to modify the alignment or other positioning of the display device relative to one or more pupils of one or more users of the display device.

[0072] FIG. 4 shows an example hand controller 400 in more detail. In practice, an HMD system may include two hand controllers similar to or identical to the hand controller 400 of FIG. 4 , which may be similar to or identical to the controllers 182, 208, and 210 described above. As shown, the controller 400 has various surfaces on which angle-sensing detectors 402 are positioned. The angle-sensing detectors 402 are positioned to receive optical signals from various different directions. The controller 400 may have buttons, sensors, light controls, knobs, indicators, displays, etc., to enable various modes of user interaction. Furthermore, as described above, in at least some implementations, one of the controller 400 and the HMD device 344 may include multiple light sources, and the other of the controller and the HMD device may include multiple angle-sensing detectors or other types of detectors or sensors. The techniques described herein may be used for various types of position tracking, including but not limited to HMDs, controllers, etc.

[0073] 5 shows a schematic block diagram of an HMD device 500 according to one or more implementations of the present disclosure. The HMD device 500 may be similar to or identical to HMD devices described elsewhere herein. Thus, the above description of the HMD device may also apply to the HMD device 500. Furthermore, at least some of the components of the HMD device 500 may reside in other components of an HMD system, such as a controller, a base station, etc. Thus, at least some of the description below may be applicable to such other components.

[0074] The HMD device 500 includes a processor 502, a front or forward-facing camera 504, multiple angle-sensing detectors 506 (e.g., quad-cell photodiodes, position-sensing detectors), and optionally an IMU 507 or multiple light sources 509. In some implementations, the HMD device 500 may include one of the angle-sensing detectors or the light sources, and another component (e.g., a controller, a base station) may include the other of the angle-sensing detectors or the light sources. As described below, in at least some implementations, the HMD device 500 may include one or more scattered light detection modules or scattered light detectors, which may be used to detect whether light received by one or more of the angle-sensing detectors is scattered or reflected, and therefore should be ignored. The HMD device 500 may include a display subsystem 508 (e.g., two displays and corresponding optical systems). The HMD device 500 may also include non-transitory data storage 510 that may store instructions or data for position tracking 512, instructions or data for display functions 514 (e.g., games), and / or other programs 516. The HMD system 500 may include or enable some of the functionality of the local computing system 120 or media content provider 190 shown in FIG. 1 and described above.

[0075] The HMD device 500 may also include various I / O components 518, which may include one or more user interfaces (e.g., buttons, a touchpad, a speaker, one or more wired or wireless communication interfaces, etc.). As an example, the I / O components 518 may include a communication interface that allows the HMD device 500 to communicate with an external device 520 via a wired or wireless communication link 522. By way of non-limiting example, the external device 520 may include a host computer, a server, a mobile device (e.g., a smartphone, a wearable computer), a controller, etc. The various components of the HMD device 500 may be housed in a single housing, in separate housings (e.g., a host computer), or any combination thereof.

[0076] It will be understood that the computing systems and devices shown are merely exemplary and are not intended to limit the scope of the present disclosure. For example, the HMD 500 and / or external device 520 may be connected to other devices not shown, such as through one or more networks such as the Internet or via the Web. More generally, such computing systems or devices may comprise any combination of hardware capable of interacting with and performing the types of functionality described, when programmed or otherwise configured with appropriate software, including, but not limited to, desktop computers, laptop computers, slate computers, tablet computers, or other computers, smartphone computing devices and other mobile phones, Internet appliances, PDAs and other electronic organizers, database servers, network storage devices and other network devices, wireless telephones, pagers, television-based systems (e.g., using set-top boxes, personal / digital video recorders, and / or game consoles and / or media servers), and various other consumer products including appropriate intercommunication capabilities. For example, example systems 500 and 520 may include executable software instructions and / or data structures in at least some embodiments that, when loaded and / or executed by a particular computing system or device, may be used to program or otherwise configure that system or device to configure the processor of that system or device. Alternatively, in other embodiments, some or all of the software systems may execute in memory on other devices and communicate with the illustrated computing systems / devices via computer-to-computer communications.Additionally, while various items are shown as being stored in memory or storage at various times (e.g., in use), these items or portions thereof may be transferred between memory and storage and / or between storage devices (e.g., in different locations) for purposes of memory management and / or data integrity.

[0077] Thus, in at least some embodiments, the illustrated system is a software-based system including software instructions that, when executed by a processor and / or other processor means, program the processor to automatically perform the described operations for the system. Furthermore, in some embodiments, some or all of the system may be implemented or provided in other manners, such as at least partially in firmware and / or hardware means, including, but not limited to, one or more application-specific integrated circuits (ASICs), standard integrated circuits, controllers (e.g., by executing appropriate instructions and including microcontrollers and / or embedded controllers), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), etc. Some or all of the system or data structures may also be stored (e.g., as software instruction content or structured data content) on a non-transitory computer-readable storage medium such as a hard disk or flash drive or other non-volatile storage device, volatile or non-volatile memory (e.g., RAM), network storage device, or portable media product (e.g., DVD disk, CD disk, optical disk, flash memory device, etc.) that is read by an appropriate drive or via an appropriate connection. The systems, modules, and data structures may also, in some embodiments, be transmitted as generated data signals (e.g., as part of a carrier wave or other analog or digital propagated signal) over a variety of computer-readable transmission media, including wireless-based and wired / cable-based media, which may take a variety of forms (e.g., as a single or multiplexed analog signal or as part of multiple discrete digital packets or frames). Such computer program products may take other forms in other embodiments. Accordingly, the present disclosure may be implemented with other computer system configurations.

[0078] 6 is a schematic diagram of an environment 600 in which machine learning techniques may be used to implement a tracking subsystem for tracking an HMD device, one or more controllers, or other components, such as the tracking subsystems described herein, according to one non-limiting illustrated implementation. The environment 600 includes a model trainer 601 and an inferer 603. In the trainer 601, training data 602 is fed to a machine learning algorithm 604 to generate a trained machine learning model 606. The training data may include, for example, labeled data from angle-sensing detectors, labeled or unlabeled scattered light detector data (described below), or other types of data that specify the position and / or orientation of a particular object relative to one or more light sources (e.g., LEDs). As a non-limiting example, in an embodiment including a component (e.g., HMD, controller) with 30 angle-sensing detectors, each training sample may include output from each or a subset of the angle-sensing detectors, the known or inferred position or orientation of the component, and information about the position or direction of one or more light sources. As described below, each angle-sensing detector may output a single data point (e.g., an angle) or may output multiple data points, such as two or four signals, each indicating the power or intensity of light received at a particular active element (e.g., a sub-detector or cell, a resistive sheet, etc.) of the angle-sensing detector. The data may also include data from one or more scattered light detectors, such as the scattered light detector described below. Such data may include polarization information (e.g., type or degree of polarization), information about whether the detected light is scattered, or other types of data.

[0079] The training data 602 may be obtained from multiple users of the HMD system and / or from a single user. The training data 602 may be obtained in a controlled environment and / or during actual use by a user (“field training”). Additionally, in at least some implementations, the model 606 may be updated or calibrated from time to time (e.g., periodically, continuously, after specific events) to provide accurate position tracking predictions.

[0080] In the reasoning unit 603, the runtime data 608 is provided as input to a trained machine learning model 606, which generates a position tracking prediction 610. Continuing with the above example, output data (e.g., intensity data, angle data) of the angle-sensing detector, optionally information about one or more light sources, and optionally information from one or more scattered light detectors may be provided as input to the trained machine learning model 606, which may process the data and predict the position of the component. The tracking prediction 610 may then be provided to one or more components associated with the HMD device, such as one or more VR or AR applications, one or more display or rendering modules, one or more mechanical controls, one or more additional position tracking subsystems, etc.

[0081] The machine learning techniques utilized to implement the features discussed herein may include any type of suitable structure or technique. By way of non-limiting example, the machine learning model 606 may include one or more of a decision tree, a statistical hierarchical model, a support vector machine, an artificial neural network (ANN), such as a convolutional neural network (CNN) or a recurrent neural network (RNN) (e.g., a long short-term memory (LSTM) network), a mixture density network (MDN), a hidden Markov model, or others. In at least some implementations, such as those utilizing RNNs, the machine learning model 606 may utilize past input (memory, feedback) information to predict the position of one or more HMD components. Such implementations may advantageously utilize motion information or sequential data to determine previous position predictions, which may provide more accurate real-time position predictions.

[0082] 7 is a flow diagram of an example method 700 of operating an HMD system to track the position of HMD components during use. Method 700 may be performed, for example, by the position tracking system or module 512 of the HMD system 500 shown in FIG. 5. As described above, method 700 may be used to track the position of any component of the HMD device, one or more controllers, etc.

[0083] The illustrated implementation of method 700 begins at operation 702, where a first HMD system component having a plurality of angle-sensing detectors is provided. The plurality of angle-sensing detectors may act to detect light emitted from one or more light sources that may be fixedly positioned (e.g., mounted on a wall or ceiling) or movable (e.g., coupled to the HMD or a controller). During operation, each of the plurality of angle-sensing detectors captures sensor data in the field of view of the respective angle-sensing detector at a frame rate. The sensor data may include any type of data that can be utilized by control circuitry (e.g., a processor) to detect the presence and orientation of the light source relative to the angle-sensing detector. In at least some implementations, each of the angle-sensing detectors may include one or more sensors (e.g., photodiodes) having image sensing circuitry and image processing circuitry. The angle-sensing detectors may output relatively raw data (e.g., light intensity or power data) or processed data (e.g., angle of incidence data).

[0084] A second HMD system component may be provided that includes a plurality of light sources (e.g., near-IR LEDs) at 704. The second HMD system component may include, for example, a controller, an HMD device, or a light source positioned in a fixed location (e.g., ceiling, wall).

[0085] At 706, at least one processor of the HMD system may cause the light sources to emit light. The light sources may be illuminated in a manner that allows each of the angle-sensing detectors to simultaneously detect light from a single light source, or more generally, in a manner that may enable the system to determine from which light source light detected by the angle-sensing detectors was received and illuminated. This may be achieved by modulating or multiplexing the illumination of the light sources using any suitable type of technique, such as time multiplexing, wavelength multiplexing, frequency multiplexing, polarization multiplexing, or other technique that allows the system to know the source of the light received from each of the angle-sensing detectors during use.

[0086] As an example of time multiplexing, the at least one processor may illuminate only a subset of the light sources simultaneously (e.g., 1, 2, 4). For example, the at least one processor may sequentially illuminate the light sources one subset at a time and collect sensor data responsive to each of the light sources.

[0087] As an example of wavelength multiplexing, different subsets of light sources may emit light of different wavelengths, and different subsets of angle-sensitive detectors may serve to detect light of the different wavelengths. Thus, light sources having different wavelengths may be simultaneously illuminated and detected by corresponding wavelength-sensitive detectors.

[0088] As an example of frequency multiplexing, a subset of light sources may be illuminated in a determined pattern or frequency that is detectable by an angle-sensitive detector to identify a particular source of light.

[0089] As an example of polarization multiplexing, a subset of the light sources may be differently polarized (e.g., linearly, circularly) and a corresponding subset of the angle-sensitive detectors may be configured to detect particular polarized light (e.g., using a polarizer that passes light with the corresponding polarization) allowing multiple light sources to be illuminated simultaneously.

[0090] Other exemplary techniques for illuminating the light source may include one or more of frequency or wavelength division multiple access (FDMA or WDMA), time division multiple access (TDMA), code division multiple access (CDMA), and orthogonal frequency division multiple access (OFDMA). In at least some implementations, the illumination pattern or scheme may be configured to be orthogonal with respect to one or more of the time, wavelength, or frequency of the electrical system (e.g., 120 Hz, which is twice the 60 Hz frequency used in the U.S. electrical system, or 100 Hz, which is twice the 50 Hz frequency used in the European electrical system). Furthermore, modulation schemes may be applied to the light amplitude of each individual marker or LED, or to one or more groups of two or more markers or LEDs, or any combination thereof. In at least some implementations, two or more of the light-emitting, light-detecting, or processing components (e.g., host systems) may be synchronized with one another, thereby providing additional benefits, for example, when using CDMA techniques or the like.

[0091] At 708, at least one processor associated with the HMD system may receive sensor data from a plurality of angle-sensing detectors. As described above, for each angle-sensing detector, the sensor data may indicate an angle of arrival of light emitted from a known light source. At 710, the at least one processor associated with the HMD system may optionally receive sensor data from an inertial measurement unit (IMU) that serves to provide inertial tracking functionality or sensor data from one or more additional sensors.

[0092] At 712, at least one processor associated with the HMD system may process the received sensor data. For example, the at least one processor may fuse some or all of the sensor data together to track one or more features present in the environment in which the HMD system operates. The sensor data may include sensor data from multiple angle-sensing detectors, and optionally, sensor data from an IMU or camera. The at least one processor may process the sensor data using, for example, a machine learning model (e.g., model 606) or another solver. As described further below, at least some implementations of the at least one processor may ignore data from one or more sensors determined to likely have scattered or reflected received light.

[0093] At 714, at least one processor associated with the HMD system may track the positions (e.g., location, orientation, or movement) of components of the HMD system in real time while the HMD system is being used by a user in the environment. During operation of the HMD, method 700 may continue to continuously track the positions of the components of the HMD system, as described above.

[0094] FIG. 8 shows a perspective view of an exemplary angle-sensitive detector 800 that may be used in one or more implementations of the present disclosure. In this example, the angle-sensitive detector 800 includes an angle-sensitive photodiode structure 804. The angle-sensitive photodiode structure 804 includes a photodiode 806, a second linear polarizer 808, a spatially varying polarizer 810, and a first linear polarizer 812. The photodiode 806 may be any device that receives light, determines the intensity associated with the light, and outputs a signal (or data) representing the intensity. The first and second linear polarizers 812, 808 may each be any type of optical filter onto which light is incident. The first and second linear polarizers 812, 808 may output linearly polarized components of the incident light (e.g., vertically polarized or horizontally polarized) and filter out (e.g., reflect, reject, or absorb) other components of the incident light.

[0095] In at least some implementations, the spatially varying polarizer 810 can be formed from a multi-twist retarder (MTR), a waveplate-like retardation film that provides precise, customized levels of broadband, narrowband, or multiband retardation in a single thin film. More specifically, an MTR includes two or more twisted liquid crystal (LC) layers on a single substrate along with a single alignment layer. Subsequent LC layers are directly aligned with previous layers, enabling simple fabrication, achieving automatic layer-to-layer alignment, and resulting in a monolithic film with a continuously varying optical axis.

[0096] The spatially varying polarizer 810 may include a retarder formed from a birefringent material. Birefringence is a property of a material that has a refractive index that depends on the polarization and propagation direction of the light. A retarder changes the polarization state or phase of light traveling through it. A retarder may have a slow axis (or special axis) and a fast axis (or ordinary axis). When polarized light travels through a retarder, light along the fast axis travels faster than light along the slow axis.

[0097] As shown in FIG. 8 , the second linear polarizer 808, the spatially varying polarizer 810, and the first linear polarizer 812 may be stacked on the photodiode 806, forming successive layers on the photodiode 806. Note that while the polarizers 812, 808 are described herein as linear polarizers, in various embodiments, the polarizers 812, 808 may be nonlinear polarizers, such as elliptical or circular polarizers. The polarizers 812, 808 may have identical light filtering characteristics and may similarly or identically reject or pass light having a particular polarization. In this simplified example, the angle-sensitive detector 800 includes a cover 814 having an opening 816 that allows light 818 from a light source 820 to pass through. As shown, the light 818 passing through the opening 816 forms a light spot 822 that can be electrically characterized to determine the angle of the light 818 and, therefore, the angle of the light source 820 relative to the angle-sensitive detector 800. As described below, the systems and methods of the present disclosure may utilize multiple light sources and angle-sensing detectors to determine the position of components of an HMD system.

[0098] 9 illustrates the first linear polarizer 812, the spatially varying polarizer 810, and the second linear polarizer 808 of the angle-sensing photodiode structure 804 and the polarization of light 818 or light spot 822 passing through them to reach the photodiode 806. Initially, the light 818 is incident on the first linear polarizer 812. The light 818 may have any polarization and, therefore, in at least some implementations, may be said to be unpolarized. In at least some implementations, the light may be linearly polarized, circularly polarized, or generally elliptically polarized.

[0099] The first linear polarizer 812 passes the linear polarization component 824 of the light 818 and rejects (absorbs or reflects) the remaining polarization components of the light 818. Although the first linear polarizer 812 is shown as a vertical polarization filter, in various embodiments, the first linear polarizer 812 can be a horizontal polarization filter or a circular polarization filter, among others.

[0100] The linearly polarized light component 824 then enters the spatially varying polarizer 810, which is positioned below the first linear polarizer 812. The spatially varying polarizer 810 is tuned so that the linearly polarized light component 824 (or any incident light) has light polarization properties that vary according to the position on the spatially varying polarizer 810 at which it enters the spatially varying polarizer 810. In the example shown in Figure 9, the spatially varying polarizer 810 modifies the linearly polarized light component 824 that is incident on it.

[0101] The manner in which the spatially varying polarizer 810 modifies the incident linearly polarized component 824 varies according to the position at which the incident linearly polarized component 824 is incident on the spatially varying polarizer 810. The position may be substantially the same as the position at which the light 818 is incident on the angle-sensitive photodiode structure 804.

[0102] In this illustrative example, at a first end 826 (shown at the top right) of the spatially varying polarizer 810, the spatially varying polarizer 810 preserves the incoming linear polarization component 824 as a vertically polarized optical signal. The spatially varying polarizer 810 passes the vertically polarized light and blocks other polarization components. The linear polarization component 824 incident on the first end 826 passes through unchanged.

[0103] The polarization filtering characteristics of the spatially varying polarizer 810 can vary gradually as a function of distance to the first end, as a non-limiting example. At the second end 828 of the spatially varying polarizer 810 (shown at the bottom left), the spatially varying polarizer 810 converts the incident vertically polarized linearly polarized component 824 into a horizontally polarized optical signal. In particular, at the second end 828, the spatially varying polarizer 810 has a linear polarization orientation of 175°. Thus, at the second end 828, the spatially varying polarizer 810 outputs light with a horizontally polarized component that is larger than the vertically polarized component. Conversely, near the center of the spatially varying polarizer 810, the spatially varying polarizer 810 has a linear polarization orientation of approximately 135°, and therefore, the spatially varying polarizer 810 rotates the polarization of the incident linearly polarized component 824 (which has a vertical polarization) by an angle of approximately 45° toward horizontal polarization. Light exiting the spatially varying polarizer 810 near its center has a vertically polarized component with the same magnitude as the horizontally polarized component.

[0104] The spatially varying characteristics of the space-varying polarizer 810 allow for identification of the position or potential position at which the linearly polarized component 824 is incident. The space-varying polarizer 810 passes filtered light 830, as shown in FIG. 9 . The intensity of the filtered light 830, in either horizontal or vertical polarization, represents the position or potential position at which the linearly polarized component 824 is incident on the space-varying polarizer 810. When the linearly polarized component 824 is incident on the first end 826, the filtered light 830 has the highest magnitude of vertical polarization. The magnitude of the vertical polarization may be inversely proportional to the distance from the first end 826.

[0105] The filtered light 830 then enters a second linear polarizer 808, which operates to remove any horizontal components of the filtered light 830 and pass the vertical light component. The second linear polarizer 808 passes the filtered linearly polarized component 832. The second linear polarizer 808 may ensure that the light passed to the photodiode 806 contains exclusively vertically polarized light and excludes horizontally polarized light.

[0106] Photodiode 806 receives filtered linearly polarized component 832 and detects the intensity of filtered linearly polarized component 832. The intensity of filtered linearly polarized component 832 is indicative of the position or set of positions at which light 818 is incident on spatially varying polarizer 810 and, consequently, angle-sensitive photodiode structure 804.

[0107] 9 is made as an example for ease of explanation. In alternative embodiments, different polarizers, polarizations, or polarization patterns may be utilized. For example, instead of linear polarization, the first and second linear polarizers 812, 808 and the spatially varying polarizer 810 may utilize circular polarization, elliptical polarization, or any other type of polarization.

[0108] Referring again to FIG. 8 , the size and position of the aperture 816 define the size of the light spot 822 formed on the angle-sensing photodiode structure 804. The intensity detected by the photodiode 806 represents the intensity of the light spot 822 that has passed through (or been filtered by) the first and second linear polarizers 812, 808 and the spatially varying polarizer 810. The intensity of the light spot 822 may be the sum of the intensities of the light rays that make up the light spot 822. The fact that the linearly polarized component 824 of the light spot 822 is incident on an area of ​​the spatially varying polarizer 810, rather than on a single point, provides an additional degree of freedom in the design of the spatially varying polarizer 810 to enable improved position detection. The spatially varying polarizer 810 may have characteristics that vary by region to enable improved position detection of the light spot 822.

[0109] Note that in some embodiments, one of the first and second linear polarizers 812, 808 may be omitted. In one embodiment, the first linear polarizer 812 may be omitted and only horizontally polarized light may be emitted for position or angle determination.

[0110] In one embodiment, improved position detection can be achieved by using a photodiode 806 with multiple local cells.

[0111] 10 illustrates the first linear polarizer 812, the spatially varying polarizer 810, and the second linear polarizer 808 of the angle-sensitive photodiode structure 804, and the polarization of the light 818 or light spot 822 passing through them to reach the photodiode 806. In FIG. 10, the photodiode 806 is a quad-cell photodiode that includes four separate photodiode active areas or elements 802a-802d separated by small gaps. It should be understood that other types of angle-sensitive detectors, such as photodiode detectors with fewer or more cells, position-sensitive detectors (PSDs), etc., may also be used.

[0112] The active area (e.g., anode) of each element 802a-802d is individually accessible, so that a spot of light illuminating a single quadrant can be electrically characterized as being in that quadrant only. The energy of the spot of light is distributed between adjacent elements 802a-802d, and the difference in the electrical contribution to each element defines the relative position of the spot of light with respect to the center of the angle-sensitive detector. The relative intensity profiles of the elements 802a-802d can be used in combination with the relative intensity profile of the spatially varying polarizer 810 to determine the position of the spot of light.

[0113] In one embodiment, the spatially varying polarizer 810 can be switched off to identify a baseline intensity. The spatially varying polarizer 810 can be coupled to a controller. The controller described herein, which can be a microcontroller or microprocessor, or, in particular, one or more controllers 182 or processor 502, can switch the spatially varying polarizer 810 on or off. When the spatially varying polarizer 810 is switched on, the spatially varying polarizer 810 filters light as described herein. Conversely, when the spatially varying polarizer 810 is switched off, the spatially varying polarizer 810 can cease polarization filtering and instead pass the linearly polarized light component 824 unchanged.

[0114] When the spatially varying polarizer 810 is switched off, the photodiode 806 detects the intensity of the light 818 (or light spot 822) without the attenuation performed by the spatially varying polarizer 810 in combination with the first and second linear polarizers 812, 808. The detected intensity may be treated as a baseline intensity or a maximum detected intensity. The baseline intensity or the maximum detected intensity may correspond to the intensity of the light 818 incident on the first end 826.

[0115] When the spatially varying polarizer 810 is switched on, the photodiode 806 detects the intensity of the light 818 (or light spot 822) with appropriate position-dependent polarization filtering. The relationship between the detected intensity of the light 818 (or light spot 822) when the position-dependent polarization filtering is appropriate to the baseline intensity indicates the position or set of positions at which the light 818 is incident on the angle-sensing photodiode structure 804.

[0116] As described herein, the polarization conversion performed by the spatially varying polarizer 810 in combination with the filtering of the first and second linear polarizers 812, 808 results in a spatially varying amplitude (or intensity) attenuation of the light 818. The amplitude (or intensity) is then detected by the photodiode 806 and used for position determination.

[0117] 11A and 11B show top and perspective views, respectively, of an exemplary angle-sensing detector 1100 that may be used in one or more implementations of the present disclosure. In this example, the angle-sensing detector 1100 includes a quad-cell photodiode that includes four separate photodiode active areas or elements 1102A-1102D separated by small gaps on a common substrate 1104. It should be understood that other types of angle-sensing detectors, such as photodiode detectors with fewer or more cells, position-sensing detectors, etc., may also be used.

[0118] In the non-limiting example shown, the active area (e.g., anode) of each element 1102A-1102D is individually accessible, so that a spot of light illuminating a single quadrant can be electrically characterized as being in that quadrant only. When a spot of light is converted by the angle-sensitive detector 1100, the energy of the spot of light is distributed among adjacent elements 1102A-1102D, and the difference in electrical contribution to each element defines the relative position of the spot of light with respect to the center of the angle-sensitive detector. The relative intensity profiles of the elements 1102A-1102D can be used to determine the position of the spot of light.

[0119] In this simplified example, angle-sensing detector 1100 includes a cover 1110 having an opening 1108 that allows light 1114 from a light source 1112 to pass through. As shown, light 1114 passing through opening 1108 to form light spot 1106 can be electrically characterized to determine the angle of light 1114 relative to angle-sensing detector 1100, and thus the angle of light source 1112. As described below, systems and methods of the present disclosure can utilize multiple light sources and angle-sensing detectors to determine the position of components of an HMD system.

[0120] It should be understood that the angle sensitive detectors of the present disclosure may include one or more of any suitable type of detector, including a quad-cell photodiode detector, a position sensitive detector (PSD) utilizing a resistive sheet, a photodiode detector having fewer (e.g., 2) or more (e.g., 16) independent sensing elements, or any other detector that allows for detecting the angle of arrival of light emitted from a light source. Additionally, as described below, in at least some implementations, the angle sensitive detectors or light sources of the present disclosure may utilize various optical components, such as filters, lenses, polarizers, etc., to improve the functionality of the systems and methods described herein.

[0121] 12 is a simplified diagram of an environment 1200 of an HMD system that uses light sources and angle-sensing detectors to determine the position of components of the HMD system, according to one non-limiting illustrated implementation. In this example, a first component 1202, such as an HMD, includes multiple light sources 1206 (two are shown, 1206a and 1206b), and a second component 1204, such as a controller of the HMD system, includes multiple angle-sensing detectors 1208 (two are shown, 1208a and 1208b). The angle-sensing detectors 1208a and 1208b are separated from each other by a known distance d1 on the second component 1204, and the light sources 1206a and 1206b are separated from each other by a known distance d2 on the first component 1202. The first and second components may be any components of the HMD system, such as an HMD, a controller, a base station, a fixed or mobile light source, a fixed or mobile angle-sensing detector, etc.

[0122] In this example, angle-sensing detector 1208a operates to determine that light arrives from light source 1206a at angle 1210 and that light arrives from light source 1206b at angle 1212. Similarly, angle-sensing detector 1208b operates to determine that light arrives from light source 1206b at angle 1214 and that light arrives from light source 1206a at angle 1216. Given the detected angles of arrival 1210, 1212, 1214, and 1216 and the known geometric relationship (e.g., distances d1 and d2) between light source 1206 and detector 1208, a method (e.g., triangulation) may be used to determine and track the relative position, orientation, or movement between first component 1202 and second component 1204. As described above, one or more solvers or machine learning methods may be used to determine the position of the components using sensor data from the angle-sensing detectors and / or light source data that indicates information about the light source of the HMD system.

[0123] 13 is an illustration 1300 of a light source 1302 and an angle-sensing detector 1304 of one example of the present disclosure. The light source 1302 and the angle-sensing detector 1304 may be similar to or identical to any of the light sources and angle-sensing detectors described herein and may be used in any of the implementations of the present disclosure. In the example shown, the light source 1302 may include an optical subsystem 1306, and the angle-sensing detector 1304 may include an optical subsystem 1308. The optical subsystems 1306 and 1308 may be identical to or different from one another and may each include one or more optical components. The optical subsystems 1306 and 1308 may be integrated into the light source 1302 and the angle-sensing detector 1304 or may be separate components. Non-limiting examples of optical components include one or more lenses, one or more polarizers, one or more filters, one or more apertures, etc. In at least some implementations, a subset of the light sources may include one type of optical subsystem, while one or more other subsets of the light sources may include another type of optical subsystem. Similarly, a subset of the angle-sensitive detectors may include one type of optical subsystem, while one or more other subsets of the angle-sensitive detectors may include another type of optical subsystem. As an example, the optical subsystem may include a filter that filters out visible light or other types of light. As described further above, the optical subsystem may include components that facilitate one or more of the various types of multiplexing described above, allowing multiple light sources to be illuminated simultaneously without confusion as to the source of the emitted light.

[0124] FIG. 14 is an illustration 1400 of a scattered light detection module or scattered light detector 1402 of the present disclosure, which may be used to determine whether light received by one or more optical detectors (e.g., angle-sensing or other types of detectors) was reflected or scattered before being received by the one or more optical detectors. Using such information, at least one processor may act to ignore light data determined to be scattered or reflected optical signals because such signals do not directly indicate the position of the light source from which the signals were emitted. In at least some implementations, the scattered light detector 1402 may be a separate component used in conjunction with one or more optical detectors used for position tracking. In other implementations, the scattered light detector 1402 may be integrated into one or more optical detectors (e.g., angle-sensing detectors) used for position tracking. One or more scattered light detectors 1402 may be used in any of the embodiments of the present disclosure. Additionally, various machine learning or artificial intelligence-based methods may be used to process scattered light detector data and improve the position tracking capabilities of the tracking system of the present disclosure. For example, machine learning or other AI methods may be used to train a tracking system to use polarization information to help improve tracking fidelity.

[0125] In the non-limiting example shown, a scattered light detector 1402 is shown, as are first and second light sources 1408 and 1410. In practice, there may be many scattered light detectors and many light sources. As a non-limiting example, the scattered light detector 1402 may be located on one of the HMD and the controller, and the light sources 1408 and 1410 may be located on the other of the HMD and the controller. In at least some implementations, the scattered light detector 1402 and one or more of the light sources 1408 and 1410 may be located on or coupled to a fixed object (e.g., a wall, ceiling, stand) or a movable object (e.g., the HMD, the controller). The scattered light detector 1402 and the light sources 1408 and 1410 may be similar to or identical to any of the light sources and scattered light detectors described herein and may be used in any of the implementations of the present disclosure.

[0126] In the example shown, scattered light detector 1402 may include an optical detector 1404 and an optical subsystem 1406, which may optionally be an angle-sensing detector. Light source 1408 may include a light emitter 1412 (e.g., an LED) and an optical subsystem 1414 that emits light 1420, and light source 1410 may include a light emitter 1416 and an optical subsystem 1414 that emits light 1422. Some or all of optical subsystems 1406, 1414, and 1418 may be the same as or different from one another and may each include one or more optical components. Optical subsystems 1406, 1414, and 1418 may be integrated into detector 1404 and light sources 1408 and 1410, respectively, or may be separate components. Non-limiting examples of optical components include one or more lenses, one or more polarizers, one or more retarders, one or more filters, one or more apertures, etc. In at least some implementations, a subset of the light sources may include one type of optical subsystem, while one or more other subsets of light sources may include another type of optical subsystem. Similarly, a subset of the scattered light detectors 1402 may include one type of optical subsystem, while one or more other subsets of the scattered light detectors may include another type of optical subsystem. As an example, the optical subsystem may include a filter that filters out visible light or other types of light. As described further above, the optical subsystem may include components that facilitate one or more of the various types of multiplexing described above, allowing multiple light sources to be illuminated simultaneously without confusion as to the source of the emitted light.

[0127] The design of the optical subsystems 1406, 1414, and 1418 may be adjusted so that the scattered light detector 1402 operates to detect whether light from the light sources 1408 and 1410 is scattered or reflected, or whether the light reaches the scattered light detector directly without scattering or reflection. For example, the scattered light detector 1402 may operate to detect a change in the type or degree of polarization of the light emitted by the light source due to scattering or reflection. In the example shown, light 1420 from the light source 1408 is received directly by the scattered light detector 1402, while light 1422 from the light source 1410 is reflected off a surface 1423 as light 1424 that is received by the scattered light detector 1402. In this example, the light 1420 indicates the relative position of the light source 1408 with respect to the scattered light detector 1402, while the light 1424 reflected off the surface 1423 does not indicate the relative position of the light source 1410 with respect to the scattered light detector 1402. Thus, by detecting that light 1424 is scattered or reflected, the tracking system can ignore or reject light signals from one or more sensors, such as scattered light detectors and similar position and orientation sensors, when performing position tracking, thereby improving the position tracking capabilities of the system.

[0128] There may be multiple configurations that may enable the scattered light detector 1402 to detect whether light from the light sources is scattered or reflected and therefore should be ignored by one or more detectors. Generally, in at least some implementations, the light emitted by the light sources 1408 and 1410 may be polarized in a manner determined by the optical systems 1414 and 1418, respectively, and the scattered light detector 1402 may be configured to distinguish between light received directly from the light sources 1408 and 1410 and light from the light sources that has been scattered or reflected before being received by the scattered light detector. For example, the type or degree of polarization of the light from the light sources may be altered as a result of scattering or specular reflection, and the scattered light detector 1402 may be configured to detect such alteration. As one non-limiting example, the optical subsystem 1414 of the light source 1408 and the optical subsystem 1418 of the light source 1410 may include one of a right-handed or left-handed circular polarizer, and the optical subsystem 1406 of the scattered light detector 1402 may include the other of a right-handed or left-handed circular polarizer. For example, the optical subsystem 1414 of light source 1408 and the optical subsystem 1418 of light source 1410 may include right-handed circular polarizers, and the optical subsystem 1406 of scattered light detector 1402 may include a left-handed circular polarizer. In this configuration, the optical subsystem 1406 of scattered light detector 1402 may be used to detect light reflected off a depolarizing surface (e.g., having random polarization) or light reflected off a non-depolarizing surface (e.g., glass, metal, acrylic, etc.) that is left-circularly polarized after reflection. If such light is above a decision threshold, the tracking system may ignore signals from one or more detectors determined to likely also have received reflected or scattered light.

[0129] An example of this configuration is shown in illustration 1500 of Figure 15, which shows a scattered light detector 1502 and a light source 1504. The light source 1504 includes an optical subsystem that includes a light emitter 1506 (e.g., an LED) and a right-handed circular polarizer 1508. The circular polarizer 1508 in this implementation includes a linear polarizer 1510 and a quarter-wave retarder or waveplate 1512 to provide light 1522 having right-handed circular polarization.

[0130] The scattered light detector 1502 includes an optical subsystem that includes an optical detector 1514 (e.g., a quad-cell detector, a single-cell detector) and a left-handed circular polarizer 1516. The left-handed circular polarizer 1516 includes a quarter-wave retarder or waveplate 1518 and a linear polarizer 1520. Because the scattered light detector 1502 includes a circular polarizer of opposite handedness to the light source's circular polarizer 1508, the scattered light detector detects light reflected via specular reflection due to the handedness of the reflected circularly polarized light switching to the opposite handedness (i.e., from right-handed to left-handed in this example).

[0131] During operation, when the scattered light detector 1502 detects scattered or reflected light (e.g., above a determined threshold), the tracking system may reject or ignore light from one or more optical sensors (e.g., sensors in a similar position or orientation to the scattered light detector) that may likely have received the same light.

[0132] FIG. 16 illustrates information 1600 showing a front view of an example HMD device 1644 when worn on the head of a user 1642. The HMD device 1644 includes a front structure 1643 that supports a front or forward-facing camera 1646 and one or more types of angle-sensing detectors 1648a-1648f (collectively 1648). As an example, some or all of the angle-sensing detectors 1648 may assist in determining the location and orientation of the device 1644 in space, such as optical sensors for detecting and using light information emitted from one or more external devices (e.g., base station 214, controller, not shown, of FIG. 2). The angle-sensing detectors 1648 may be any type of detector that operates to detect the angle of arrival of light emitted from a light source. Non-limiting examples of angle-sensing detectors include photodiode detectors (e.g., bicell detectors, quadrantel detectors), position-sensing detectors using resistive sheets, etc.

[0133] As shown, the forward camera 1646 and angle-sensing detector 1648 are pointed forward toward an actual scene or environment (not shown) in which the user 1642 operates the HMD device 1644. More generally, the angle-sensing detector 1648 may be pointed toward other areas (e.g., above, below, left, right, behind) to detect light from various sources, such as a controller (e.g., held by the user 1642) or objects mounted in various locations (e.g., walls, ceilings). The actual physical environment may include, for example, one or more objects (e.g., walls, ceilings, furniture, stairs, cars, trees, tracking markers, light sources, or any other type of object). The particular number of sensors 1648 may be fewer (e.g., 2, 4) or more (e.g., 10, 20, 30, 40) than the number of sensors shown. The HMD device 1644 may further comprise one or more additional components not attached to the front structure (e.g., internal to the HMD device), such as an IMU (Inertial Measurement Unit) 1647 electronic device that measures and reports specific forces, angular velocities, and / or magnetic fields surrounding the HMD device (e.g., using a combination of accelerometers and gyroscopes, and optionally magnetometers). The HMD device 1644 may further comprise additional components not shown, including one or more display panels and optical lens systems oriented toward the user's eyes (not shown), optionally having one or more attached internal motors for changing the alignment or other positioning of one or more of the optical lens systems and / or display panels within the HMD device.

[0134] The depicted example of an HMD device 1644 is supported on the head of a user 1642 based at least in part on one or more straps 1645 attached to the housing of the HMD device 1644 and extending wholly or partially around the user's head. Although not shown here, the HMD device 1644 may further include one or more external motors, such as those attached to one or more of the straps 1645, and the automated corrective action may include using such motors to adjust such straps to correct alignment or other positioning of the HMD device on the user's head. It will be understood that the HMD device may include other support structures not shown here (e.g., a nosepiece, a chinstrap, etc.), whether in addition to or instead of the illustrated straps, and that some embodiments may include motors attached to one or more such other support structures to similarly adjust their shape and / or location to correct alignment or other positioning of the HMD device on the user's head. Other display devices that are not fixed to the user's head may similarly be attached to or be part of one or more structures that affect the positioning of the display device, and in at least some embodiments may include motors or other mechanical actuators that similarly modify their shape and / or location to modify the alignment or other positioning of the display device relative to one or more pupils of one or more users of the display device.

[0135] The HMD device 1644 also includes multiple scattered light detectors 1650, 1652, and 1666. The scattered light detectors 1650, 1652, and 1666 may be similar to or identical to any of the scattered light detectors described herein and may operate to detect whether light 1660, 1664, and 1672, respectively, from light sources 1658, 1662, and 1670 associated with the HMD device 1644 has been reflected or scattered off a surface before reaching the HMD device. As described above, when scattered light is detected, sensor data from one or more sensors determined to have likely received the same light may be disregarded.

[0136] In at least some implementations, a single scattered light detector may be provided for all of the detectors 1648. In other implementations, a separate scattered light detector may be provided for each of the detectors 1648, or a scattered light detector may be included as part of one or more of the detectors 1648. In the simplified example shown, scattered light detector 1650 located on the right side of front structure 1643 corresponds to detectors 1648a, 1648b, and 1648e used to detect light from a light source (e.g., light source 1658) in region 1654 to the right side of user 1642. That is, if scattered light detector 1650 detects reflected or scattered light, the tracking system may ignore signals from one or more of detectors 1648a, 1648b, and 1648e determined to have likely received the same light due to a similar orientation as scattered light detector 1650. Similarly, scattered light detectors 1652 located on the left and right sides of front structure 1643 correspond to detectors 1648c, 1648d and 1648g used to detect light from a light source (e.g., light source 1662) in region 1656 to the left of user 1642. Scattered light detector 1666 in the upper region of front structure 1643 corresponds to detector 1648f used to detect light from a light source (e.g., light source 1670) in region 1668 above user 1642. As explained above, multiplexing (e.g., time, wavelength, pattern, code) can be used to enable the system to know which light sources or groups of light sources light is received by detectors 1648, 1650, 1652 and 1666.

[0137] 17 shows a perspective view of an angle-sensing or scattered light detector 1700 that may be used in one or more implementations of the present disclosure. In this non-limiting example, the scattered light detector 1700 comprises a quad-cell photodiode, including four separate photodiode active areas or elements 1702A-1702D separated by small gaps on a common substrate 1704. It should be understood that other types of detectors may also be used, such as photodiode detectors with fewer or more cells, position-sensing detectors, etc.

[0138] In the non-limiting example shown, the active area (e.g., anode) of each element 1702A-1702D is individually accessible, so that a spot of light illuminating a single quadrant can be electrically characterized as being in that quadrant only. When a spot of light is converted by the detector 1700, the energy of the spot of light is distributed among adjacent elements 1702A-1702D, and the difference in the electrical contribution to each element defines the relative position of the spot of light with respect to the center of the detector. The relative intensity profiles of the elements 1702A-1702D can be used to determine the position of the spot of light.

[0139] In this simplified example, detector 1700 includes an opaque cover or mask 1710 with an aperture 1708 that allows light 1714 from a light source 1712 to pass through. As shown, light 1714 passing through aperture 1708 forms a light spot 1706 that can be electrically characterized to determine the angle of light 1714, and thus the angle of light source 1712, relative to detector 1700. As described below, systems and methods of the present disclosure may utilize multiple light sources and detectors to determine the position of components of an HMD system.

[0140] In the example shown, the first circular polarizer 1716 is located proximate (e.g., next to) the light source 1712, and the second polarizer 1718 is located proximate to the detector 1700. In at least some implementations, the light emitted by the light source 1712 may be polarized in a manner determined by the first circular polarizer 1716 and the second circular polarizer 1718, and the scattered light detector 1700 may be configured to distinguish between light received directly from the light source 1712 and light from the light source that has been scattered or reflected before being received by the scattered light detector 1700. In one non-limiting example, one of the first and second circular polarizers 1716 and 1718 may each include one of a right- or left-handed circular polarizer, and the other of the first and second circular polarizers 1716 and 1718 may include the other of a right- or left-handed circular polarizer. For example, the first circular polarizer 1716 of the light source 1712 may comprise a right-handed circular polarizer, and the second circular polarizer 1718 of the scattered light detector 1700 may comprise a left-handed circular polarizer. In this configuration, the second circular polarizer 1718 of the scattered light detector 1700 may be used to detect light reflected off a depolarizing surface (e.g., having a random polarization) or light reflected off a non-depolarizing surface (e.g., glass, metal, acrylic, etc.) that is left-circularly polarized after reflection. If such light is above a decision threshold, the tracking system may ignore signals from one or more detectors determined to likely also have received reflected or scattered light.

[0141] 18 is a flow diagram of an example method 1800 of operating an HMD system to track the position of HMD components during use. Method 1800 may be performed, for example, by the position tracking system or module 512 of HMD system 500 shown in FIG. 5. As described above, method 1800 may be used to track the position of any component, such as an HMD device wearable on a user's head, one or more handheld controllers, etc.

[0142] The illustrated implementation of method 1800 begins at operation 1802, where a first HMD system component having a plurality of angle-sensing detectors is provided. The plurality of angle-sensing detectors may act to detect light emitted from one or more light sources that may be fixedly positioned (e.g., mounted on a wall or ceiling) or movable (e.g., coupled to an HMD headset or controller). During operation, each of the plurality of angle-sensing detectors captures sensor data at a frame rate in each of a plurality of angle-sensing detector fields of view. The sensor data may include any type of data that is usable by control circuitry (e.g., a processor) to detect the presence and orientation of the light source relative to the angle-sensing detector. In at least some implementations, each of the angle-sensing detectors may include one or more sensors (e.g., photodiodes) having image sensing circuitry and, optionally, image processing circuitry. The angle-sensing detectors may output relatively raw data (e.g., light intensity or power data) or processed data (e.g., angle of incidence data).

[0143] At 1804, a second HMD system component may be provided that includes a plurality of light sources (e.g., near-IR LEDs). The second HMD system component may include, for example, a light source located in a controller, an HMD headset, or a fixed location (e.g., ceiling, wall).

[0144] At 1806, at least one processor of the HMD system may cause the light sources to emit light. The light sources may be illuminated in a manner that allows each of the angle-sensing detectors to simultaneously detect light from a single light source, or more generally, in a manner that may enable the system to determine from which light source light detected by the angle-sensing detectors was received and illuminated. This may be achieved by multiplexing the illumination of the light sources using any suitable type of multiplexing, such as time multiplexing, wavelength multiplexing, frequency multiplexing, polarization multiplexing, or other technique that allows the system to know the source of light received from each of the angle-sensing detectors during use.

[0145] As an example of time multiplexing, the at least one processor may illuminate only a subset of the light sources simultaneously (e.g., 1, 2, 4). For example, the at least one processor may sequentially illuminate the light sources one subset at a time and collect sensor data responsive to each of the light sources.

[0146] As an example of wavelength multiplexing, different subsets of light sources may emit light of different wavelengths, and different subsets of angle-sensitive detectors may serve to detect light of the different wavelengths. Thus, light sources having different wavelengths may be simultaneously illuminated and detected by corresponding wavelength-sensitive detectors.

[0147] As an example of frequency multiplexing, a subset of light sources may be illuminated in a determined pattern or frequency that is detectable by an angle-sensitive detector to identify a particular source of light.

[0148] As an example of polarization multiplexing, a subset of the light sources may be differently polarized (e.g., linearly, circularly) and a corresponding subset of the angle-sensitive detectors may be configured to detect particular polarized light (e.g., using a polarizer that passes light with the corresponding polarization) allowing multiple light sources to be illuminated simultaneously.

[0149] Other non-limiting example techniques for illuminating the light source may include frequency or wavelength division multiple access (FDMA or WDMA), time division multiple access (TDMA), code division multiple access (CDMA), orthogonal frequency division multiple access (OFDMA), etc.

[0150] At 1808, at least one processor associated with the HMD system may receive sensor data from a plurality of angle-sensing detectors. As described above, for each angle-sensing detector, the sensor data may indicate an angle of arrival of light emitted from a known light source. At 1810, the at least one processor associated with the HMD system may optionally receive sensor data from an inertial measurement unit (IMU) that serves to provide inertial tracking functionality or sensor data from one or more additional sensors.

[0151] At 1812, at least one processor associated with the HMD system may process the received sensor data, including detecting corrupted data, as described further below. For example, the at least one processor may fuse some or all of the sensor data together to track one or more features present in the environment in which the HMD system operates. The sensor data may include sensor data from multiple angle-sensing detectors and, optionally, sensor data from an IMU, camera, or other sensor data. The at least one processor may process the sensor data using, for example, a machine learning model (e.g., model 606) or another solver. As described further below, in at least some implementations, the at least one processor may ignore data from one or more sensors determined to be corrupted, e.g., data from light that is unlikely to have been received directly from the light source of the HMD system and that is scattered, reflected, or received from another light source.

[0152] At 1814, at least one processor associated with the HMD system may track the positions (e.g., location, orientation, or movement) of components of the HMD system in real time while the HMD system is being used by a user in the environment. During operation of the HMD, method 1800 may continue to continuously track the positions of the components of the HMD system, as described above.

[0153] In processing the received sensor data, the control circuitry may identify one or more corrupted sensor data samples, where each of the one or more corrupted sensor data samples comprises a sensor data sample from one of the plurality of angle-sensing detectors that is identified as likely not representing light received directly by one of the plurality of angle-sensing detectors from the one or more light sources. In at least some implementations, the control circuitry is configured to ignore the corrupted sensor data sample during the tracking process, and may continue to ignore samples from that optical detector for a fixed or variable period of time.

[0154] The identification of corrupted sensor data samples may be based at least in part on the known geometry of at least one of the first and second head-mounted display system components. For example, a projection model of at least one of the first or second head-mounted display system components may be utilized to determine which of the plurality of angle-sensing detectors are likely not receiving light directly from one or more of the plurality of light sources, and data from such detectors may be ignored for a period of time. For example, received sensor data samples may be compared to the at least one projection model, and received sensor data samples that do not match the at least one projection model within a defined threshold may be identified as corrupted sensor data samples to be ignored.

[0155] Additionally or alternatively, identification of the one or more corrupted sensor data samples may be based, at least in part, on one or more of a past position or orientation of at least one of the first and second head mounted display system components, a current position or orientation of at least one of the first and second head mounted display system components, or a predicted future position or orientation of at least one of the first and second head mounted display system components.

[0156] In at least some implementations, the number of samples or time period during which one or more detectors are disabled may be selectively varied based on various criteria, such as the actual or predicted movement (e.g., direction, speed, rotation) of at least one of the first or second head-mounted display system components. As one example, comparison to the projection model may indicate that the detector is obstructed by an object (e.g., a wall, person, other component) or is facing away from the HMD system's light sources and is therefore unlikely to receive light directly from one of the HMD system's light sources for a period of time. The control circuitry may track the position or movement of the one or more components to determine a period of time after which the detector is expected to again receive light from at least one of the multiple light sources. At this time, the system may again use samples from the detector for tracking purposes.

[0157] The head-mounted display system components may include a head-mounted display device wearable on a user's head, a controller, a base station, or other HMD system components. As described elsewhere herein, to process the received sensor data, the control circuitry may provide the received sensor data as input to one or more trained machine learning models.

[0158] 19 is a flow diagram of a method for adaptively adjusting the brightness of multiple light sources of a position tracking system of an HMD system, according to an example embodiment of the present disclosure. Method 1900 may be performed, for example, by the position tracking system or module 512 of HMD system 500 shown in FIG. 5. As described above, method 1900 may be implemented, for example, in combination with method 1800 of FIG. 18, during tracking of the position of any component of a user's head-mountable HMD device, one or more handheld controllers, etc. Advantageously, the adaptive brightness features described herein may provide improved performance by utilizing the detector's larger dynamic range and may also increase battery life by reducing power consumption.

[0159] Method 1900 begins at 1902, where control circuitry of an HMD system receives optical detector data from an optical detector (e.g., a photodiode, an angle-sensing detector). At 1904, the control circuitry may process the received optical detector data, and at 1906, the control circuitry may adaptively adjust the brightness of at least one of a plurality of light sources based at least in part on the processed optical detector data. In at least some implementations, the control circuitry adaptively adjusts the brightness of at least one of the one or more light sources based on the dynamic range of the optical detector, e.g., to maximize the dynamic range of the optical detector. To adjust the brightness, the pulse width of a signal provided to the one or more light sources may be selectively adjusted.

[0160] In at least some implementations, the control circuitry may disable one or more light sources ("dark measurements") and receive optical data from the optical detector while the one or more light sources are disabled. Such a feature may allow the brightness of one or more light sensors to be adapted based on ambient light levels in the environment in which the HMD system is operated.

[0161] An example method 2000 of this feature is shown in FIG. 20. At 2002, the control circuitry may disable one or more (e.g., all) light sources of an HMD system. At 2004, the control circuitry may capture sensor data from the optical detector while the one or more light sources are disabled. At 2006, the control circuitry may adjust the brightness settings of the one or more light sources based on the captured sensor data. At 2008, the control circuitry may optionally adjust the rate of sensor data capture for brightness adjustment based on changes in one or more parameters, such as movement of one or more components, elapsed time, number of samples, amount of ambient light, detected changes in ambient light, etc. To adaptively adjust the brightness of the one or more light sources, the received optical detector data may be provided as input to one or more trained machine learning models, as described elsewhere herein. In at least some implementations, the control circuitry may perform light measurements periodically (e.g., every 5 samples, every 50 samples), and adjust the brightness of the light sources after each measurement based on the results of the measurements.

[0162] FIG. 21 is a flow diagram of a method for compensating for non-uniform brightness of a light source in a position tracking system of an HMD system, according to an example embodiment of the present disclosure. In practice, an angle-sensing detector, such as a quad photodiode (QPD), may include a number of channels that are read sequentially in time (e.g., via multiplexing). Therefore, measurements of all of the detector's channels are not captured simultaneously, so any non-uniformity in the brightness of the light source may cause inaccurate measurements. For example, when an LED is illuminated over an illumination period, its intensity may vary over the illumination period due to thermal and other effects (e.g., "droop"). As described below, method 2100 compensates for this effect, which advantageously provides more accurate measurements used for position tracking.

[0163] Method 2100 may be performed, for example, by the position tracking system or module 512 of the HMD system 500 shown in Figure 5. As described above, method 2100 may be implemented, for example, in combination with method 1800 of Figure 18, during tracking of the position of any component of a user's head-mountable HMD device, one or more handheld controllers, etc.

[0164] Method 2100 begins at 2102, where a control circuit may cause one or more light sources to emit light during an illumination period. At 2104, the control circuit may receive sensor data from the angle-sensing detector, which may include sequentially capturing sensor cell samples from multiple sensor cells during the illumination period, as described above. For example, the control circuit may include an analog-to-digital converter (ADC), and a multiplexer may be used to sequentially read sensor cell samples from each of multiple (e.g., four) sensor cells of the angle-sensing detector. Illumination data may be captured using other methods that are capable of providing an illumination profile for the light source with respect to time.

[0165] At 2106, the control circuitry may process the received sensor data, including determining a correction to account for non-uniform brightness of one or more light sources during sequential capture of sensor cell samples. At 2108, the control circuitry may use the determined correction to apply calibration data to the sensor cell samples and track the position of the first head mounted display system component using the calibrated sensor cell samples. The calibration data may represent, for example, a characteristic gradient in brightness of the one or more light sources during an illumination period.

[0166] In at least some implementations, the control circuitry may iteratively determine updated calibration data and may use the updated calibration data to track the position of the first head mounted display system component. As an example, to determine the updated calibration data, the control circuitry may disable one or more light sources for a calibration period, sequentially capture sensor cell samples from multiple sensor cells during the calibration period, interpolate the captured sensor cell samples, and determine the updated calibration data based on the interpolation of the captured sensor cell samples.

[0167] The calibration data may additionally or alternatively be determined during the manufacturing or design process of the HMD system. For example, a characteristic slope or function of a light source (e.g., an LED) may be empirically determined using an angle-sensing detector or other type of optical detector, and such information may be provided to the HMD system to compensate for non-uniformity of the light source (or similar or identical light source) during operation of the HMD system. More generally, a method of calibrating a head-mounted display system component may include causing one or more light sources to emit light over an illumination period; sequentially capturing sensor cell samples from a plurality of sensor cells of an angle-sensing detector; processing the received sensor cell samples to generate calibration data that accounts for non-uniform brightness of the one or more light sources over the illumination period; and storing the calibration data in a non-transitory processor-readable storage medium for subsequent use in tracking at least one component of the head-mounted display system.

[0168] 22 is a flow diagram of a method for adaptively enabling and disabling components of a tracking subsystem of an HMD system, according to an example embodiment of the present disclosure. In the context of light sources (e.g., LEDs), implementation of method 2200 may be referred to as "adaptive firing" of the LEDs, which may, for example, reduce power consumption and therefore extend battery life. As described above, method 2200 may be implemented, for example, in combination with method 1800 of FIG. 18, during tracking of the position of any component of a user's head-mountable HMD device, one or more handheld controllers, etc.

[0169] Method 2200 begins at 2202, where control circuitry causes one or more of a plurality of light sources to emit light. At 2204, the control circuitry receives sensor data from one or more of a plurality of optical detectors and tracks a position of a first head mounted display system component based at least in part on the received sensor data, as described elsewhere herein.

[0170] At 2206, the control circuitry may process the received sensor data, including determining whether to disable any one of the optical detectors or light sources based on the determined disablement criteria. At 2208, the control circuitry may disable each of the optical detectors or light sources that meet the disablement criteria for a respective disablement period during tracking of the position of the first head mounted display system component.

[0171] Generally, the disablement criteria provide or facilitate a determination that light emitted by the light source is unlikely to be received by any of the optical detectors of the first head mounted display system component. For example, the disablement criteria may determine that a particular light source faces away from the optical detectors of the HMD system or is obstructed by an object (e.g., a person, component, furniture, wall) in the environment in which the HMD system is operating. Similarly, the control circuit may determine that the optical detector is unlikely to receive light from the light source and therefore may be disabled for a fixed or variable period of time.

[0172] The disable criteria may be based at least in part on a determined relative position or movement between a first head-mounted display system component and a second head-mounted display system component, such as a relative position or movement between a controller and a headset, a relative position or movement between a controller and a base station, or a relative position or movement between a headset and a base station. As described above, one or more projection models may be used to evaluate whether light from a light source is expected to be directly received by an optical sensor of the HMD system. In at least some implementations, position tracking information may be used to predict when a particular component (e.g., light source, detector) may be disabled and when such component should be re-enabled.

[0173] In at least some implementations, the disablement criteria provide or facilitate a determination that light emitted by each light source in a first subset of the plurality of light sources is likely to be received by at least one of the optical detectors of the first head-mounted display system component, and the disablement criteria act to disable a second subset of light sources within the first subset of light sources for each disablement period. For example, the system may determine that a first subset of four distant light sources is likely to be detected by one or more detectors and may disable two light sources of the first subset (i.e., the second subset) so that only two light sources are enabled for a period of time. This feature may increase battery life while still providing light sources detectable by an optical sensor. In at least some implementations, the light sources in the first subset of light sources that are not disabled may be relatively distant from each other, thereby providing a relatively large angle of separation for the optical detectors that detect light emitted by the light sources, thereby improving measurement accuracy.

[0174] 23 is a flow diagram for a method of operating a position tracking system of an HMD system to track the position, orientation, and / or movement of components of the HMD system in use by fusing inertial sensor data, optical sensor data, and image data, according to an example embodiment of the present disclosure. As described above, method 2300 may be implemented, for example, in combination with method 1800 of FIG. 18, during tracking of the position of any component of a user's head-mountable HMD device, one or more handheld controllers, etc.

[0175] The head-mounted display system may include a first head-mounted display system component, an inertial measurement unit (IMU) carried by the first head-mounted display system component, a plurality of angle-sensing optical detectors (or other types of optical detectors) carried by the first head-mounted display system component, and at least one camera carried by the first head-mounted display system component. At 2302, control circuitry associated with the HMD system may receive inertial sensor data from the inertial measurement unit. At 2304, the control circuitry may receive optical sensor data from one or more of the plurality of angle-sensing optical detectors or other types of optical detectors. At 2306, the control circuitry may receive image sensor data from the camera. As an example, the camera may be a camera of a forward-facing HMD device that is wearable on a user's head.

[0176] At 2308, the control circuitry may process or fuse the received inertial sensor data, optical sensor data, and image sensor data. For example, the control circuitry may utilize one or more sensor fusion algorithms, including, but not limited to, a central limit theorem algorithm, a Kalman filter, a Bayesian network, a Dempster-Shafer algorithm, or a convolutional neural network. At 2310, the control circuitry may track the position of the first head mounted display system component based at least in part on processing the received inertial sensor data, optical sensor data, and image sensor data. In at least some implementations, the control circuitry may provide the inertial sensor data, optical sensor data, and image sensor data as input to one or more trained machine learning models to process the received inertial sensor data, optical sensor data, and image sensor data, as described elsewhere herein (e.g., see FIG. 6 ).

[0177] FIG. 24 is a perspective view 2400 of an example angle-sensitive optical detector 2403 that may be used in one or more implementations of the present disclosure. FIG. 25 is a cross-sectional view 2500 of the angle-sensitive optical detector 2403 shown in FIG. 24, and FIG. 26 is a top view 2600 of the angle-sensitive optical detector. In this example, the optical detector 2403 includes a quadrant photodetector 2404 disposed on a common substrate 2402. The quadrant photodetector 2404 includes four active areas or cells 2404a-d separated by small gaps. It should be understood that other types of angle-sensitive detectors, such as photodiode detectors, position-sensitive detectors, etc., having fewer or more cells may also be used.

[0178] The angle-sensing detector 2403 includes an opaque screen or cover 2406 positioned a distance 2412 above or in front of the optical detector 2404. The opaque screen 2406 may be coupled to the substrate 2402 or integrated therewith, for example, as part of the housing. The opaque screen 2406 has an opening 2408 therein that allows light 2416 from an object 2414 to pass therethrough. In this example, the object 2414 is a user's eye, including the eye's dark pupil 2415, although embodiments described herein may be used to track other objects (e.g., a controller, a headset, or other objects). The optical detector 2403 further includes an imaging lens 2410 positioned within or adjacent to the opening 2408 of the opaque screen 2406. Advantageously, the imaging lens 2410 is configured to focus an image 2418 of the pupil 2415 of the user's eye 2414 onto the quadrant optical detector. That is, the imaging lens 2410 may be designed to have an object plane that is substantially coplanar with the expected location of the pupil 2415 of the user's eye 2414 relative to the imaging lens 2410, and an image plane that is substantially coplanar with the quadrant photodetector 2404. The imaging lens 2410 may include a single lens, or may include multiple lenses. Additionally, the imaging lens 2410 may include one or more optical components, such as one or more filters, coatings, etc.

[0179] In the non-limiting illustrated example, the active area (e.g., anode) of each element 2404a-2404d of the photodetector is individually addressable, such that light illuminating a single quadrant can be electrically characterized as being in that quadrant only. As light translates across the angle-sensitive detector 2403, the light's energy is distributed between adjacent elements 2404a-2404d, and the difference in electrical contribution to each element defines the relative position of the light. The relative intensity profiles of the elements 2404a-2404d can be used to determine the position of the light imparted to the cell.

[0180] As shown, light 2416 reflected from the user's eye 2414 and passing through the lens 2410 forms an image 2418 of the eye 2414, which includes a dark spot surrounded by brighter areas due to the dark pupil 2415. This is because the dark pupil 2415 reflects a relatively small amount of light compared to the light reflected by portions of the eye 2414 and the user's face surrounding the pupil. Because the dark pupil 2415 will affect the intensity of light to the cells, the image 2418 (including the dark pupil) formed on the quadrant photodetector 2404 may be electrically characterized to determine the position of the pupil 2415 relative to the angle-sensing detector 2403. As discussed elsewhere herein, the position information may be used for gaze tracking and various other techniques that may be implemented using pupil location information. As discussed below, the systems and methods of the present disclosure may utilize multiple light sources and angle-sensing detectors to determine the position of the user's eye or other components, such as components of an HMD system.

[0181] 27 is a cross-sectional view of an angle-sensitive optical detector 2700 in accordance with one or more embodiments of the present disclosure. The optical detector 2700 includes a photodetector 2702 positioned beneath an optically transparent material 2704, such as resin, air, or the like. The angle-sensitive optical detector 2700 is configured to receive light 2706 from a light source 2708 attached to an object associated with the object tracking system, as described elsewhere herein. The light source 2708 can be any type of light source, such as an IR LED or other type of light source. The photodetector 2702 can be any photodetector including multiple optically active areas, such as a quadrant photodetector including four optically active areas, an image sensor having an array of photodiodes, or the like.

[0182] In the illustrated implementation, the angle-sensing optical detector 2700 includes intensity variation optics 2712 and phase variation optics 2714 positioned above the optically transparent layer 2704. In at least some implementations, the angle-sensing optical detector 2700 may include only one of the intensity variation optics 2712 and the phase variation optics 2714. In other implementations, a single component providing both intensity variation and phase variation capabilities may be used. An industrial design surface 2710 is positioned above the intensity variation optics 2712. The specific shape of the industrial design surface 2710 may be determined by the surface shape of the object associated with the angle-sensing optical detector 2700 (e.g., the controller of a headset or HMD). In at least some implementations, the industrial design surface 2710 may be configured to have specific optical properties and therefore may not be designed solely based on aesthetic or other non-optical criteria. In at least some implementations, the surface 2710 may be configured to have specific optical properties while matching the surrounding industrial design surfaces to provide an optical detector that is invisible, or at least less noticeable, to the user's eye.

[0183] Intensity variation optics 2712 and phase variation optics 2714 can be made of any number of different types of materials that modify at least one of the intensity or phase of light 2706 from light source 2708 received by photodetector 2702. Furthermore, intensity variation optics 2712 and phase variation optics 2714 can each include one or more components or can be integrated as a single component. As a non-limiting example, intensity variation optics 2712 can include a pattern screen printed on a glass or plastic substrate, where the pattern has transparent, semi-transparent, or opaque varying regions. As another example, one or both of optics 2712 and 2714 can include photolithographic film. For example, intensity variation optics 2712 can include a central transparent or semi-transparent region defining an "aperture" surrounded by opaque regions. In this manner, there is significant design freedom, allowing for countless pattern options for the intensity variation optics. Advantageously, this design freedom allows for the creation of intensity patterns or masks that are not constrained to patterns that are mechanically manufacturable.

[0184] Phase variation optics 2714 may include a diffractive optical element that can provide any diffraction order. Phase variation optics 2714 may include a lens array or any other suitable optical system. In at least some implementations, optics 2712 and 2714 may be formed from a single substrate (e.g., glass, plastic) etched with a diffraction pattern on one side and printed with an intensity variation pattern on the other side.

[0185] Optical systems 2712 and 2714 may be configured to produce a determined intensity profile at the photodetector. By way of non-limiting example, the optical systems may be configured to provide a Gaussian intensity profile or any other desired intensity profile. Additionally, optical systems 2712 and 2714 may be configured to modify or improve the angular resolution, linear response, or other characteristics of photodetector 2702. For example, optical systems 2712 and 2714 may be used to provide substantially the same angular resolution at the center of the field of view as at the edge of the field of view.

[0186] FIG. 28 is a cross-sectional view of an angle-sensing optical detector 2800 in accordance with one or more embodiments of the present disclosure. The optical detector 2800 includes a photodetector 2802 positioned beneath an optically transparent material 2804, such as resin, air, or the like. The angle-sensing optical detector 2800 is configured to receive light 2806 from a light source 2808 attached to an object associated with the object tracking system, as described elsewhere herein. The light source 2808 may be any type of light source, such as an IR LED or other type of light source. The photodetector 2802 may be any photodetector including multiple optically active areas, such as a quadrant photodetector including four optically active areas, an image sensor with an array of photodiodes, or the like. An industrial design surface 2810 is positioned above the optically transparent layer 2804.

[0187] In the implementation shown, angle-sensitive optical detector 2800 includes intensity or phase varying optics 2812 positioned adjacent to an upper surface of photodetector 2802. Intensity or phase varying optics 2812 may be configured to vary at least one of the intensity or phase of light 2806. In general, intensity or phase varying optics 2812 may be similar to or identical to optics discussed elsewhere herein, such as optics 2712 and 2714 shown in FIG.

[0188] FIG. 29 is a cross-sectional view of an angle-sensing optical detector 2900 according to one or more embodiments of the present disclosure. The optical detector 2900 includes a photodetector 2902 positioned beneath an optically transparent material 2904, such as resin, air, or the like. The angle-sensing optical detector 2900 is configured to receive light 2906 from a light source 2908 attached to an object associated with the object tracking system, as described elsewhere herein. The light source 2908 may be any type of light source, such as an IR LED or other type of light source. The photodetector 2902 may be any photodetector including multiple optically active areas, such as a quadrant photodetector including four optically active areas, an image sensor having an array of photodiodes, or the like. An industrial design surface (not shown) may be positioned above the optically transparent layer 2904.

[0189] In the implementation shown, angle-sensitive optical detector 2900 includes intensity or phase varying optics 2910 positioned within a mechanical aperture of photodetector 2902. Intensity or phase varying optics 2910 may be configured to vary at least one of the intensity or phase of light 2906 from light source 2908 received by photodetector 2902. In general, intensity or phase varying optics 2912 may be similar to or identical to optics discussed elsewhere herein, such as optics 2712 and 2714 shown in FIG.

[0190] FIG. 30 is a cross-sectional view of an angle-sensing optical detector 3000 according to one or more embodiments of the present disclosure. The optical detector 3000 includes a photodetector 3002 positioned beneath an optically transparent material 3004, such as resin, air, or the like. The angle-sensing optical detector 3000 is configured to receive light 3006 from a light source 3008 attached to an object associated with the object tracking system, as described elsewhere herein. The light source 3008 may be any type of light source, such as an IR LED or other type of light source. The photodetector 3002 may be any photodetector including multiple optically active areas, such as a quadrant photodetector including four optically active areas, an image sensor having an array of photodiodes, or the like. An industrial design surface (not shown) may be positioned above the optically transparent layer 3004, as discussed above with reference to other embodiments.

[0191] In the implementation shown, the angle-sensing optical detector 3000 includes intensity or phase varying optics 3010 that include a selectively controllable mechanical aperture. The intensity or phase varying optics 3010 may be configured to vary at least one of the intensity or phase of light 3006 from the light source 3008 received by the photodetector 3002. In general, the intensity or phase varying optics 3010 may be similar to or identical to the optics discussed elsewhere herein, such as optics 2712 and 2714 shown in FIG. 27 . The mechanical aperture may be operably coupled to a control circuit 3012 (“controller”) that may selectively adjust the size of the adjustable mechanical aperture during operation. In at least some implementations, the controller 3012 may selectively adjust the size or shape of the mechanical aperture based at least in part on feedback from a tracking system. For example, if it is desirable for the tracking system to increase the field of view of the optical detector 3000, the controller 3012 may increase the size of the mechanical aperture to provide a larger field of view. Similarly, if it is desirable for the tracking system to reduce the field of view of the optical detector 3000, the controller 3012 may reduce the size of the mechanical aperture to provide a smaller field of view.

[0192] FIG. 31 is a cross-sectional view of an angle-sensitive optical detector 3100 according to one or more embodiments of the present disclosure. The optical detector 3100 includes a photodetector 3102 positioned beneath an optically transparent material 3104, such as resin, air, or the like. The angle-sensitive optical detector 3100 is configured to receive light 3106 from a light source 3108 attached to an object associated with the object tracking system, as described elsewhere herein. The light source 3108 may be any type of light source, such as an IR LED or other type of light source. The photodetector 3102 may be any photodetector including multiple optically active areas, such as a quadrant photodetector including four optically active areas, an image sensor having an array of photodiodes, or the like. An industrial design surface (not shown) may be positioned above the optically transparent layer 3104, as discussed above with reference to other embodiments.

[0193] In the implementation shown, angle-sensitive optical detector 3100 includes intensity variation optics 3110 and phase variation optics 3112 positioned above optically transparent material 3104. Intensity variation optics 3110 may be configured to vary the intensity of light 3106 from light source 3108 that is received by photodetector 3102. Similarly, phase variation optics 3112 may be configured to vary the phase of light 3106 from light source 3108 that is received by photodetector 3102. In general, intensity or phase variation optics 3112 may be similar to or identical to optics discussed elsewhere herein, such as optics 2712 and 2714 shown in FIG.

[0194] In at least some implementations, at least one of the optical systems 3110 and 3112 may be selectively controllable, such as via the controller 3114. As a non-limiting example, at least one of the optical systems 3110 and 3112 may include a liquid crystal layer that may be controlled to provide various patterns of varying intensity, phase, polarization, or other optical properties. For example, the optical system 3110 may be controlled to provide one or more intensity mask patterns, one or more phase mask patterns, one or more polarization mask patterns, or any combination thereof.

[0195] FIG. 32 is a cross-sectional view of an angle-sensing optical detector 3200 according to one or more embodiments of the present disclosure. The optical detector 3200 includes a photodetector 3202 positioned toward an upper surface of an optically transparent material 3204, such as resin, air, or the like. The angle-sensing optical detector 3200 is configured to receive light 3206 from a light source 3208 attached to an object associated with the object tracking system, as described elsewhere herein. The light source 3208 may be any type of light source, such as an IR LED or other type of light source. The photodetector 3202 may be any photodetector including multiple optically active areas, such as a quadrant photodetector including four optically active areas, an image sensor having an array of photodiodes, or the like. An industrial design surface (not shown) may be positioned on the optically transparent layer 3204.

[0196] In the implementation shown, the angle-sensitive optical detector 3200 includes intensity or phase varying optics 3210 positioned adjacent to and below the photodetector 3202. The intensity or phase varying optics 3210 may be configured to vary at least one of the intensity or phase of the light 3206 from the light source 3208 received by the photodetector 3202. In general, the intensity or phase varying optics 3210 may be similar to or identical to the optics discussed elsewhere herein, such as optics 2712 and 2714 shown in FIG. 27 . In the implementation shown, a reflective telescope mirror 3212 is provided that acts to direct the light 3206 from the light source 3208 onto the photodetector 3202. While shown as a planar surface for simplicity, in practice the reflective mirror 312 may have a curvature that causes the light 3206 to reflect onto the photodetector 3202. Advantageously, in this configuration, it may be possible to achieve a larger field of view.

[0197] FIG. 33 is a cross-sectional view of an angle-sensing optical detector 3300 according to one or more embodiments of the present disclosure. The optical detector 3300 includes a photodetector 3302 positioned beneath an optically transparent material 3304, such as resin, air, or the like. The angle-sensing optical detector 3300 is configured to receive light 3306 from a light source 3308 attached to an object associated with the object tracking system, as described elsewhere herein. The light source 3308 may be any type of light source, such as an IR LED or other type of light source. The photodetector 3302 may be any photodetector including multiple optically active areas, such as a quadrant photodetector including four optically active areas, an image sensor having an array of photodiodes, or the like. An industrial design surface (not shown) may be positioned above the optically transparent layer 3304.

[0198] In the implementation shown, the angle-sensitive optical detector 3300 includes intensity or phase varying optics 3310 positioned within a mechanical aperture of the photodetector 3302. The intensity or phase varying optics 3310 may be configured to vary at least one of the intensity or phase of the light 3306 from the light source 3308 received by the photodetector 3302. In the implementation shown, the optics 3310 may include polarized catadioptric optics or “pancake” optics, which may allow for a wider field of view and / or thinner optics. In general, the intensity or phase varying optics 3312 may be similar to or identical to the optics discussed elsewhere herein, such as optics 2712 and 2714 shown in FIG. 27 .

[0199] 34 is a graph 3400 illustrating the angular resolution of an angle-sensing optical detector with respect to the field of view, according to one non-limiting illustrated implementation. Specifically, graph 3400 illustrates the angular resolution of a conventional optical detector in solid line 3402 and the angular resolution of an optical detector of the present disclosure in dotted line 3404. As illustrated by dotted line 3404, an optical detector of the present disclosure can be configured to have substantially the same angular resolution at the center of the field of view (left side of graph 3400) as at the edge of the field of view (right side of graph 3400).

[0200] 35 is a graph 3500 illustrating the output response of an angle-sensitive optical detector according to one non-limiting illustrated implementation. Specifically, graph 3500 illustrates the output of a conventional optical detector in solid line 3502 and the output of an example optical detector of the present disclosure in dotted line 3504. As illustrated by dotted line 3504, the optical detector of the present disclosure can be configured to output a step function by applying appropriate intensity and / or phase variation optics, as described above. This feature can advantageously improve the signal-to-noise ratio for the output of the optical detector.

[0201] 36 is a schematic block diagram 3600 of a process for determining one or more optical subsystems of an angle-sensitive optical detector, according to one non-limiting illustrated implementation. The optical subsystems to be determined can be any of the optical subsystems described herein, such as any of the intensity or phase variation optics of the present disclosure. In this example, a known position (P XYZ ) may be a point in space 3602 at a point in space 3602. The point may be, for example, the location of a light source coupled to the object being tracked. The angle-sensing detector may have an upper surface (e.g., a positively curved surface) with a defined forward optical function 3604. The upper surface may be any industrially designed surface and / or may be a surface with specified optical properties. Using an appropriate mathematical solver, the intensity and / or phase varying optics 3606 may be determined to provide a desired output 3608 from the optical detector. That is, given the design freedom of the implementation of the present disclosure, as described above, for any arbitrary optical function 3604, the intensity or phase varying optics 3606 may be mathematically determined to provide a desired output 3608, which may include any desired output for the optical detector (e.g., uniform angular resolution, increased linearity, larger or smaller field of view, etc.), as described herein.

[0202] The foregoing detailed description has illustrated various implementations of devices and / or processes through the use of block diagrams, schematic diagrams, and examples. To the extent that such block diagrams, schematic diagrams, and examples include one or more functions and / or operations, those skilled in the art will understand that each function and / or operation in such block diagrams, flow diagrams, or examples may be individually and / or collectively implemented by a wide range of hardware, software, firmware, or substantially any combination thereof. In one implementation, the subject matter may be implemented via an application-specific integrated circuit (ASIC). However, those skilled in the art will recognize that the implementations disclosed herein can equivalently be implemented in whole or in part on standard integrated circuits, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more controllers (e.g., microcontrollers), as one or more programs running on one or more processors (e.g., microprocessors), as firmware, or as substantially any combination thereof for which designing the circuitry and / or writing the software and / or firmware code is well within the skill of one of ordinary skill in the art in light of this disclosure.

[0203] Those skilled in the art will recognize that many of the methods or algorithm sets described herein may employ additional operations, omit some operations, and / or perform operations in a different order than specified.

[0204] Additionally, those skilled in the art will appreciate that the mechanisms taught herein can be distributed as a program product in a variety of formats, and that the exemplary implementations apply equally regardless of the particular type of signal-bearing medium used to actually accomplish the distribution, including, but not limited to, recordable types of media such as floppy disks, hard disk drives, CD-ROMs, digital tape, and computer memory.

[0205] The various implementations described above can be combined to provide further implementations. Unless inconsistent with the specific teachings and definitions herein, all U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, foreign patent applications, and non-patent publications referenced herein are incorporated herein by reference in their entirety. If desired, aspects of the implementations can be modified to employ systems, circuits, and concepts from various patents, applications, and publications to provide further implementations.

[0206] These and other changes can be made to the implementation in light of the above detailed description. Generally, in the following claims, the terms used should not be construed to limit the claims to the specific implementations disclosed in the specification and claims, but should be construed to include all possible implementations along the full range of equivalents to which such claims are subject. Accordingly, the scope of the claims is not limited by the disclosure.

Claims

1. 1. An object tracking system comprising: a support structure; and a plurality of optical detectors carried by the support structure; Equipped with Each of the plurality of optical detectors: a photodetector including a plurality of optically active areas configured to receive light emitted by an object of the object tracking system; and an optical subsystem positioned to receive the light before it is received by the photodetector, the optical subsystem including at least one optical system configured to vary at least one of an intensity of the light or a phase of the light; Including, Object tracking system.

2. The object tracking system of claim 1 , wherein the optical subsystem includes an intensity mask configured to generate a predetermined intensity profile.

3. The object tracking system of claim 1 , wherein the optical subsystem includes a patterned intensity variation layer that includes at least two regions having different levels of opacity.

4. The object tracking system of claim 1 , wherein the optical subsystem is configured to modify the angular resolution of the optical detector.

5. The object tracking system of claim 1 , wherein the optical subsystem is configured to cause the photodetector to generate discrete output values ​​as a function of the angle of incidence of the light.

6. The object tracking system of claim 1 , wherein the optical subsystem is configured such that the photodetector has substantially the same angular resolution at the center of the field of view and at the edge of the field of view.

7. The object tracking system of claim 1 , wherein the optical subsystem includes at least one of a diffractive optical element, a lens array, a pancake optic, or a reflective telescope mirror.

8. The object tracking system of claim 1 , wherein the optical subsystem includes a mask screen printed or etched onto a layer of material.

9. The object tracking system of claim 1 , wherein the optical subsystem includes an intensity mask comprising a photolithographic film.

10. 10. The object tracking system of claim 1, wherein the optical subsystem comprises a material having an etched pattern on a first side and a printed pattern on a second side, the second side being opposite the first side.

11. The object tracking system of claim 1 , wherein the optical subsystem includes an intensity mask layer and a separate phase mask layer.

12. 2. The object tracking system of claim 1, wherein each of the optical detectors includes an industrial design optical layer having an upper surface defined by an industrial design of the support structure, and the optical subsystem is configured to take into account the industrial design optical layer.

13. The object tracking system of claim 1 , wherein the photodetector comprises a quadrant photodetector having four optically active areas.

14. The object tracking system of claim 1 , wherein the optical subsystem includes a selectively controllable mechanical aperture.

15. The object tracking system of claim 1 , wherein the optical subsystem includes a liquid crystal layer selectively controllable to vary at least one of the intensity or phase of the light.

16. The object tracking system of claim 1 , wherein the optical subsystem is positioned adjacent to or within a physical aperture of the optical detector.

17. The object tracking system of claim 1 , wherein the optical subsystem is positioned adjacent to the plurality of optically active areas of the optical detector.

18. receiving detector data from each of the plurality of optical detectors; processing the detector data received from the plurality of optical detectors; and Tracking the position of the object based at least in part on the processing of the received detector data.

18. The object tracking system of claim 1, further comprising a control circuit configured to:

19. 1. A head mounted display system comprising: a support structure positionable on the user's head; a display carried by said support structure; a plurality of optical detectors carried by the support structure, each of the plurality of optical detectors comprising: a photodetector including a plurality of optically active areas configured to receive light emitted by an object of the object tracking system; and an optical subsystem positioned to receive the light before it is received by the photodetector, the optical subsystem including at least one optical system configured to vary at least one of an intensity of the light or a phase of the light; and receiving detector data from the plurality of optical detectors; processing the detector data received from the plurality of optical detectors; and Tracking a position of the head mounted display system based at least in part on the processing of the received detector data. A control circuit configured as follows: A head-mounted display system comprising:

20. 1. A method for tracking an object, comprising: providing a plurality of optical detectors, each of the plurality of optical detectors comprising: a photodetector including a plurality of optically active areas configured to receive light emitted by an object of the object tracking system; and an optical subsystem positioned to receive the light before it is received by the photodetector, the optical subsystem including at least one optical system configured to vary at least one of an intensity of the light or a phase of the light; Includes; receiving detector data from the plurality of optical detectors; processing the detector data received from the plurality of optical detectors; and Tracking a position of the object based at least in part on the processing of the received detector data. A method for providing the above.

Images