Position tracking systems and methods for head mounted display systems

Abstract

Systems and methods for tracking one or more objects, such as a position of a component of a head mounted display (HMD) system. One or more objects may carry a plurality of angle sensitive optical detectors. Each of the optical detectors may include an optical subsystem configured to vary at least one of a phase or an intensity of light impinging on the optical detector. The optical subsystem may include one or more of a diffractive optical element, a lens array, an intensity mask, a phase mask, etc. The optical detector may include a photodetector including a plurality of optically active regions, such as a quadrant cell photodetector, an image sensor with an array of photodiodes, and the like. The control circuit may cause the light source to emit light and may receive sensor data from the plurality of optical detectors. The control circuitry may process the sensor data to track the position of the one or more objects.

Description

Technical Field

[0001] The present disclosure generally relates to positional tracking of objects, such as head-mounted display systems and controllers associated with head-mounted display systems. Background Art

[0002] The current generation of virtual reality ("VR") or augmented reality ("AR") experiences are created using a head-mounted display ("HMD") that may be connected to a stationary computer (e.g., a personal computer ("PC"), laptop, or gaming console), may be combined and / or integrated with a smartphone and / or its associated display, or may be a standalone device. Typically, an HMD is a display device worn on the user's head with a small display device in front of one eye (monocular HMD) or each eye (binocular HMD). The display unit is typically miniaturized and may include, for example, CRT, LCD, liquid crystal on silicon (LCos), or OLED technology, among others. Binocular HMDs have the potential to display different images to each eye. This feature is used to display stereoscopic images.

[0003] With the development of smartphones, high-definition televisions, and other electronic devices, the demand for high-performance displays has increased. The increasing popularity of virtual reality and augmented reality systems, especially those using HMDs, has further increased this demand. Virtual reality systems typically completely cover the wearer's eyes and replace the real or physical scene (or real reality) in front of the wearer with a "virtual" reality, while augmented reality systems typically provide a translucent or transparent overlay of one or more screens in front of the wearer's eyes to enhance the real scene with additional information, and mediated reality systems can similarly present information to the viewer that combines real-world elements with virtual elements. In many virtual reality and augmented reality systems, the movement of the wearer of such a head-mounted display can be tracked in various ways, such as by sensors in the head-mounted display, controllers, or external sensors, so that the displayed image can reflect the user's movements and allow for interactive environments.

[0004] Position tracking allows an HMD system to estimate the position of one or more components relative to each other and the surrounding environment. Position tracking can utilize a combination of hardware and software to detect the absolute position of the components of the HMD system. Position tracking is an important technology for AR or VR systems, capable of tracking the movement of the HMD (and / or controllers or other peripherals) with six degrees of freedom (6DOF).

[0005] Positional tracking can be used to change the user's perspective to reflect different actions such as jumping or squatting, as well as to accurately represent the user's hands and other objects in the virtual environment. Positional tracking can also increase the connection between the physical and virtual environments by, for example, using hand position to move virtual objects by touch. Positional tracking improves the user's 3D perception of the virtual environment due to parallax, which helps with the perception of distance. Additionally, positional tracking can help minimize motion sickness caused by the disconnect between what the user's eyes see and the information perceived by the user's vestibular system.

[0006] There are different position tracking methods. These methods can include acoustic tracking, inertial tracking, magnetic tracking, optical tracking, and combinations thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In the drawings, the same reference numerals identify similar elements or steps. The sizes and relative positions of the elements in the drawings are not necessarily drawn to scale. For example, the shapes and angles of the various elements are not necessarily drawn to scale, and some of the elements may be arbitrarily enlarged and positioned to improve the clarity of the drawing. In addition, the particular shapes of the drawn elements are not necessarily intended to convey any information about the actual shape of the particular element, and may be selected solely for ease of identification in the drawings.

[0008] Figure 1 is a schematic diagram of a network environment that includes one or more systems suitable for performing at least some of the techniques described in this disclosure, including an implementation of a tracking subsystem.

[0009] Figure 2 is a diagram of an example environment showing at least some of the described techniques for use with an example head-mounted display device that is connected to a video rendering computing system and provides a virtual reality display to a user.

[0010] Figure 3 is a schematic diagram of an HMD device having a binocular display subsystem and multiple angle-sensitive detectors.

[0011] Figure 4 is a schematic diagram of a controller that may be used with the HMD device.

[0012] Figure 5 is a schematic block diagram of an HMD device according to an example embodiment of the present disclosure.

[0013] Figure 6 is a schematic diagram of an environment in which a tracking subsystem of an HMD device may be implemented using machine learning techniques, according to a non-limiting illustrative implementation.

[0014] Figure 7is a flow chart of a method of operating a position tracking system of an HMD system to track the position, orientation and / or movement of components of the HMD system during use, according to an example implementation of the present disclosure.

[0015] Figure 8 A perspective view of an example angle-sensitive detector that may be used in one or more implementations of the present disclosure is shown.

[0016] Fig. 9 The first linear polarizer, the spatially varying polarizer and the second linear polarizer of the angle sensitive photodiode structure are shown, as well as the polarization of the light or light spot passing through these polarizers to reach the photodiode.

[0017] Fig.10 The first linear polarizer, the spatially varying polarizer, and the second linear polarizer of the angle sensitive photodiode structure are shown, as well as the polarization of the light or light spot that passes through these polarizers to reach the photodiode.

[0018] Fig.11A is a top view of an example angle-sensitive detector that may be used in one or more implementations of the present disclosure.

[0019] Fig. 11B yes Fig.11A A perspective view of the angle-sensitive detector shown.

[0020] Fig.12 is a simplified diagram showing the use of light sources and angle-sensitive detectors to determine the positions of components of an HMD system, according to a non-limiting illustrative implementation.

[0021] Fig.13 is a diagram depicting an example optical system of a light source and an angle-sensitive detector according to a non-limiting illustrative implementation.

[0022] Fig.14 is a diagram describing the operation of an example scatter detection module and light source of a tracking system, according to a non-limiting illustrative implementation.

[0023] Fig.15 is a diagram depicting components of a light source and a scatter detection module of a tracking system, according to a non-limiting illustrative implementation.

[0024] Fig.16 is a schematic diagram of an HMD device having a binocular display subsystem, multiple angle-sensitive detectors, and multiple scatter detection modules for detecting light that has been scattered or reflected and for ignoring such scattered light during position tracking of the HMD device or its components.

[0025] Fig.17is a perspective view of components of a light source and a scatter detection module of a tracking system, according to a non-limiting illustrative implementation.

[0026] Fig.18 is a flow chart of a method of operating a position tracking system of an HMD system to track the position, orientation, or movement of a component of the HMD system during use, according to an example implementation of the present disclosure.

[0027] Fig.19 is a flow chart of a method of adaptively adjusting brightness of a plurality of light sources or optical detectors of a position tracking system of an HMD system according to an example embodiment of the present disclosure.

[0028] Fig. 20 is a flow chart of a method of adaptively adjusting brightness of multiple light sources or optical detectors of a position tracking system of an HMD system based on changes in one or more parameters according to an example embodiment of the present disclosure.

[0029] Fig.21 is a flow chart of a method of compensating for non-uniform brightness of a light source of a position tracking system of an HMD system according to an example implementation of the present disclosure.

[0030] Fig. 22 is a flow chart of a method of adaptively enabling and disabling components (eg, light sources, optical detectors) of a tracking subsystem of an HMD system according to an example implementation of the present disclosure.

[0031] Fig.23 is a flow chart of a method of operating a position tracking system of an HMD system to track the position, orientation, or movement of a component of the HMD system by fusing inertial sensor data, optical sensor data, and image data according to an example implementation of the present disclosure.

[0032] Fig.24 is a perspective view of an example angle-sensitive optical detector that may be used in one or more of the implementations of the present disclosure.

[0033] Fig.25 yes Fig.24 A cross-sectional view of an angle-sensitive optical detector is shown.

[0034] Fig.26 yes Fig.24 A top view of an angle-sensitive optical detector is shown.

[0035] Fig. 27 is a cross-sectional view of an angle-sensitive optical detector including intensity-varying optics and phase-varying optics and located beneath an arbitrary industrially designed surface.

[0036] Fig.28is a cross-sectional view of an angle sensitive optical detector including intensity or phase varying optics located proximate a photodetector of the angle sensitive optical detector.

[0037] Fig.29 is a cross-sectional view of an angle sensitive optical detector including intensity or phase varying optics positioned within a physical aperture of the angle sensitive optical detector.

[0038] Fig.30 is a cross-sectional view of an angle-sensitive optical detector including a controllable mechanical actuator according to a non-limiting illustrative implementation.

[0039] Fig.31 is a cross-sectional view of an angle-sensitive optical detector including a controllable liquid crystal layer according to a non-limiting illustrative implementation.

[0040] Fig.32 is a cross-sectional view of an angle-sensitive optical detector including a reflecting telescope mirror according to a non-limiting illustrative implementation.

[0041] Fig.33 is a cross-sectional view of an angle-sensitive optical detector including pancake-shaped optics according to a non-limiting illustrative implementation.

[0042] Fig.34 is a graph illustrating the angular resolution of an angle-sensitive optical detector according to a non-limiting illustrative implementation.

[0043] Fig.35 is a graph illustrating the response of an angle-sensitive optical detector according to a non-limiting illustrative implementation.

[0044] Fig.36 is a schematic block diagram of a process of determining one or more optical subsystems of an angle-sensitive optical detector, according to a non-limiting illustrative implementation. DETAILED DESCRIPTION

[0045] In the following description, certain specific details are set forth in order to provide a comprehensive understanding of the various implementations disclosed. However, those skilled in the art will recognize that the implementations may be practiced without one or more of these specific details, or with other methods, components, materials, etc. In other instances, well-known structures associated with computer systems, server computers, and / or communication networks are not shown or described in detail to avoid unnecessarily obscuring the description of the implementations.

[0046] Throughout the specification and claims, the term "comprising" is synonymous with "including," and is inclusive or open-ended (ie, not excluding additional, unrecited elements or method steps) unless the context requires otherwise.

[0047] In this specification, "an implementation" or "implementation" refers to at least one implementation including a particular feature, structure, or characteristic described in connection with the implementation. Thus, the phrases "in one implementation" or "in an implementation" appearing in various places in this specification do not necessarily refer to the same implementation. In addition, particular features, structures, or characteristics may be combined in any suitable manner in one or more implementations.

[0048] As used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It should also be noted that the term "or" generally includes "and / or" in its meaning unless the context clearly dictates otherwise.

[0049] The titles and abstracts of the disclosure provided herein are for convenience only and do not interpret the scope or meaning of the various implementations.

[0050] One or more implementations of the present disclosure relate to systems and methods for accurately tracking the positions of components (e.g., HMD, controllers, peripherals) of a head-mounted display (HMD) system. In at least some implementations, the HMD includes a support structure that carries a forward-facing camera ("front camera" or "forward-facing camera") and multiple angle-sensitive detectors or light sources. Similarly, one or more controllers may include multiple angle-sensitive detectors or light detectors. In other implementations, the HMD does not include a front camera. The front camera may capture image sensor data in the field of view of the front camera at a first frame rate (e.g., 30 Hz, 90 Hz). In at least some implementations, the HMD system may not include an angle-sensitive detector, or may include other types of optical detectors (e.g., photodiodes). Thus, where appropriate, the systems and methods described herein may use non-angle-sensitive detectors or angle-sensitive detectors.

[0051] In operation, as discussed further below, one or more fixed or movable light sources (e.g., IRLEDs) can be illuminated. The light source can be connected to an HMD, a controller, a fixed object located in an environment (e.g., a base station), etc. Each of the plurality of angle-sensitive detectors collects sensor data in the respective plurality of angle-sensitive detector fields of view at a second frame rate (e.g., 1000 Hz, 2000 Hz), and the second frame rate can be greater than the first frame rate of the front camera (if present). In at least some implementations, the angle-sensitive detector field of view can be narrower than the front camera field of view, although this is not required. For example, the front camera can have a wider front camera field of view of 90°, 120°, or 150°, and each of the angle-sensitive detectors can have a relatively narrow sensor IC field of view (e.g., 25°, 45°, 75°). In at least some implementations, the angle-sensitive detector field of view can collectively cover at least a majority of the front camera field of view, or even be greater than the front camera field of view, wherein each of the angle-sensitive detector fields of view overlaps a different portion of the front camera field of view.

[0052] In operation, at least one processor operably connected to a plurality of angle-sensitive detectors may receive sensor data that collects light from a plurality of light sources (e.g., LEDs, lasers, other light sources). The at least one processor may process the received image sensor data based at least in part on the processing of the received image sensor data to track the position of the components of the head-mounted display. For example, the at least one processor may fuse the sensor data from the angle-sensitive detector to track one or more features present in the environment. The at least one processor may process the sensor data using machine learning techniques, solvers, or other methods to determine the position (e.g., location, orientation, movement) of one or more components of the HMD system. In at least some implementations, the sensor data may be fused with sensor data from other sensors, such as sensor data from a front-facing camera or an inertial measurement unit (IMU) of an HMD system component. In at least some implementations, one or more scatter detection modules or "scatter detectors" may be used to detect when light has been scattered or reflected before reaching one or more angle-sensitive detectors, and because the angle of such light cannot accurately indicate the position of its outgoing light source, the tracking system may ignore such light. With this technique, the accuracy of position tracking may be greatly improved. Various features of the implementation of the present disclosure are discussed in detail below in conjunction with the accompanying drawings.

[0053] Figure 11 is a schematic diagram of a network environment 100 that includes a local media rendering (LMR) system 110 (e.g., a gaming system) that includes a local computing system 120, a display device 180 (e.g., an HMD device with two display panels, one for each eye), and one or more controllers 182 suitable for performing at least some of the techniques described herein. Figure 1 In the illustrated embodiment, the local computing system 120 is connected to the local computing system 120 via a transmission link 115 (which may be a wired link or a cable link, for example, via Figure 2 One or more cables (cable 220) shown, or may be a wireless link) are communicatively connected to the display device 180. The controller 182 can be connected to the local computing system 120 or the display device 180 via appropriate wired or wireless links 186 and 184, respectively. In other embodiments, whether in addition to or instead of the HMD device 180, the local computing system 120 can provide encoded image data to a panel display device (e.g., a TV, a game console, or a monitor) for display via a wired or wireless link, and the display devices each include one or more addressable pixel arrays. In various embodiments, the local computing system 120 may include: a general computing system; a game console; a video stream processing device; a mobile computing device (e.g., a cellular phone, a PDA, or other mobile device); a VR or AR processing device; or other computing systems.

[0054] In the illustrated embodiment, the local computing system 120 has: components including one or more hardware processors (e.g., centralized processing units or "CPUs") 125, memory 130, various I / O ("input / output") hardware components 127 (e.g., keyboard, mouse, one or more game controllers, speakers, microphones, IR transmitters and / or receivers, etc.); a video subsystem 140 including one or more dedicated hardware processors (e.g., graphics processing units or "GPUs") 144 and video memory (VRAM) 148; computer readable memory 150, and network connections 160. Additionally, in the illustrated embodiment, an embodiment of a tracking subsystem 135 runs in the memory 130 to perform at least some of the described techniques, such as by the CPU(s) 125 and / or GPU(s) 144 to perform automated operations that implement these described techniques, and the memory 130 may optionally further run one or more other programs 133 (e.g., to generate video or other images to be displayed, such as a game program). As part of implementing automated operations for at least some of the techniques described herein, tracking subsystem 135 and / or program 133 executing in memory 130 may store or retrieve various types of data included in an example database data structure of memory 150, where in this example, the data used may include various types of image data information in database ("DB") 154, various types of application data in DB 152, various types of configuration data in DB 157, and may include additional information such as system data or other information.

[0055] In the depicted embodiment, the LMR system 110 is also communicatively coupled via one or more computer networks 101 and network links 102 to an exemplary network-accessible media content provider 190, which may further provide content to the LMR system 110 for display, either in addition to or in lieu of the image generation program 133. The media content provider 190 may include one or more computing systems (not shown), each of which may each have components similar to those of the local computing system 120, including one or more hardware processors, I / O components, local storage devices, and memory, although for the sake of brevity, some details are not shown with respect to the network-accessible media content provider.

[0056] It should be understood that although Figure 1In the illustrated embodiment, the display device 180 is shown as being distinct and separate from the local computing system 120, but in some embodiments, some or all components of the local media rendering system 110 may be integrated or packaged in a single device, such as a mobile gaming device, a portable VR entertainment system, an HMD device, etc. In these embodiments, the transmission link 115 may include, for example, one or more system buses and / or video bus architectures.

[0057] As an example involving operations performed locally by the local media rendering system 120, assume that the local computing system is a gaming computing system, such that the application data 152 includes one or more gaming applications executed by the CPU 125 using the memory 130, and various video frame display data is generated and / or processed by the image generation program 133, such as the GPU 144 in conjunction with the video subsystem 140. In order to provide a high-quality gaming experience, the local computing system 120 generates a large amount of video frame data (corresponding to a high image resolution for each video frame and a high "frame rate" of approximately 60-180 video frames per second), and provides it to the display device 180 via the wired or wireless transmission link 115.

[0058] It will also be understood that the computing system 120 and the display device 180 are merely illustrative and are not intended to limit the scope of the present disclosure. Alternatively, the computing system 120 may include multiple interactive computing systems or devices, and may be connected to other devices not shown, including through one or more networks such as the Internet, through the World Wide Web (Web) or through a dedicated network (e.g., a mobile communication network, etc.). More generally, a computing system or other computing node may include any combination of hardware or software that may interact with each other and perform the functions of the described types, including but not limited to desktop computers or other computers, game systems, database servers, network storage devices and other network devices, PDAs, mobile phones, wireless phones, pagers, electronic notepads, Internet devices, television-based systems (e.g., using set-top boxes and / or personal / digital video recorders), and various other consumer products including appropriate communication capabilities. The display device 180 may similarly include one or more devices having one or more display panels of various types and forms, and may optionally include various other hardware and / or software components.

[0059] Additionally, in some embodiments, the functionality provided by the tracking subsystem 135 may be distributed among one or more components (e.g., local and remote computing systems, HMDs, one or more controllers, one or more base stations), and in some embodiments, some of the functionality of the tracking subsystem 135 may not be provided, and / or other additional functionality may not be available. It should also be understood that although various items are shown as being stored in memory or storage when in use, these items or portions thereof may be transferred between memory and other storage devices for purposes of memory management or data integrity. Thus, in some embodiments, some or all of the described techniques may be performed by hardware including one or more processors or other configured hardware circuits or memory or storage, such as when configured by one or more software programs (e.g., by the tracking subsystem 135 or its components) and / or data structures (e.g., by executing software instructions of one or more software programs and / or by storing such software instructions and / or data structures). Part or all of the components, systems, and data structures may also be stored (e.g., as software instructions or structured data) on a non-transitory computer-readable storage medium, such as a hard disk or flash drive or other non-volatile storage device, volatile or non-volatile memory (e.g., RAM), a network storage device, or a portable media product (e.g., a DVD disk, a CD disk, an optical disk, etc.) to be read by an appropriate drive or through an appropriate connection. In some embodiments, the systems, components, and data structures may also be transmitted as data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) generated on various computer-readable transmission media (including wireless and wire / cable-based media), and may take various forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). In other embodiments, these computer program products may also take other forms. Therefore, the present invention may be implemented with other computer system configurations.

[0060] Figure 2An example environment 200 is shown in which at least some of the described techniques are used with an example HMD device 202, which is connected to a video rendering computing system 204 via a cable connection 220 (or a wireless connection in other embodiments) to provide a virtual reality display to a human user 206. The user wears the HMD device 202 and receives display information of a simulated environment that is different from the real physical environment from the computing system 204 through the HMD device, wherein the computing system acts as an image rendering system that provides images of the simulated environment to the HMD device for display to the user, wherein the images of the simulated environment are images generated by, for example, a game program and / or other software program running on the computing system. In this example, the user is also able to move around in a tracking space 201 of the real physical environment 200, and may also have one or more I / O ("input / output") devices to allow the user to further interact with the simulated environment, which in this example include handheld controllers 208 and 210.

[0061] In the illustrated example, the environment 200 may include one or more base stations 214 (two are shown, labeled base stations 214a and 214b), which may facilitate tracking of the HMD device 202 or controllers 208 and 210. As the user moves the position of the HMD device 202 or changes the orientation of the HMD device 202, the position of the HMD device is tracked to allow the corresponding portion of the simulated environment to be displayed to the user on the HMD device, and the controllers 208 and 210 may also use similar techniques to track the position of the controllers (and optionally use this information to assist in determining or verifying the position of the HMD device). After the tracked position of the HMD device 202 is known, the corresponding information is sent to the computing system 204 via the cable 220 or wirelessly, and the computing system 204 uses the tracked position information to generate one or more subsequent images of the simulated environment for display to the user.

[0062] The optical tracking described herein can be used in conjunction with various position tracking methods, including but not limited to acoustic tracking, inertial tracking, or magnetic tracking, among others.

[0063] In at least some implementations, at least one of the HMD device 202 and the controllers 208 and 210 may include one or more optical receivers or sensors that may be used to implement tracking functionality or other aspects of the present disclosure. In at least some implementations, at least one of the HMD device 202, the controllers 208 and 210, or other components may include one or more light sources (e.g., LEDs) that may emit light that is detected by one or more of the optical receivers. The light source may be located in a fixed location or on a movable component such as the HMD device or the controller.

[0064] In at least some implementations, in addition to or in lieu of generating a fixed point light source, the base stations 214 can each sweep an optical signal across the tracking space 201. Each base station 214 can generate more than one optical signal, as required by each particular implementation. For example, while a single base station 214 is typically sufficient for six degrees of freedom tracking, in some embodiments, multiple base stations (e.g., base stations 214a, 214b) may be required or desired to provide robust room-scale tracking for the HMD device and peripherals. In this example, an optical receiver such as an angle-sensitive detector or a scatter detector is integrated into the HMD device 202 and / or other tracking objects, such as controllers 208 and 210. In at least some implementations, the optical receiver can be paired with an accelerometer and gyroscopic inertial measurement unit (“IMU”) on each tracking device to support low-latency sensor fusion.

[0065] In at least some implementations, each base station 214 includes two rotors that sweep a linear beam across the tracking space 201 on orthogonal axes. At the beginning of each scanning cycle, the base station 214 can emit an omnidirectional light pulse (called a "sync signal") that is visible to all sensors on the tracked object. Thus, each sensor calculates a unique angular position in the scanning space by timing the duration between the sync signal and the beam signal. Multiple sensors fixed to a single rigid body can be used to solve sensor distance and orientation problems.

[0066] One or more sensors located on the tracked object (e.g., HMD device 202, controllers 208, and 210) may include an optoelectronic device capable of detecting modulated light from the rotor. For visible light or near infrared (NIR) light, a silicon photodiode and appropriate amplifier / detector circuitry may be used. Because the environment 200 may include static and time-varying signals (optical noise) having similar wavelengths to the base station 214 signal, in at least some implementations, the base station light may be modulated to make it easier to distinguish from any interfering signals and / or to cause the sensor to filter out any radiation wavelength other than the base station signal wavelength. As discussed further below, in at least some embodiments, an angle-sensitive detector is used to track one or more components of the HMD system, and one or more scatter detectors may be used to ignore light that may have been scattered or reflected before being detected by the optical detector.

[0067] Inside-out tracking is also a type of positional tracking that can be used to track the position of the HMD device 202 and / or other objects (e.g., controllers 208 and 210, tablets, smartphones). Inside-out tracking differs from outside-in tracking in the location of the camera or other sensor used to determine the position of the HMD component. For inside-out tracking, the camera or sensor is located on the HMD component or object being tracked, while in outside-out tracking, the camera or sensor is placed at a fixed location in the environment.

[0068] An HMD that employs inside-out tracking "looks out" using one or more sensors to determine how its position relative to the environment is changing. As the HMD moves, the sensors readjust its position in the room, and the virtual environment responds accordingly in real time. This positional tracking can be achieved with or without markers placed in the environment. Cameras placed on the HMD can observe features of the surrounding environment. When markers are used, the markers are designed to be easily detected by the tracking system and placed in specific areas. With "markerless" inside-out tracking, the HMD system uses unique features that are initially present in the environment (e.g., natural features) to determine position and orientation. The HMD system's algorithms recognize specific images or shapes and use them to calculate the device's position in space. Data from accelerometers and gyroscopes can also be used to improve the accuracy of positional tracking.

[0069] Figure 3 Information 300 is shown presenting a front view of an example HMD device 344 when worn on the head of a user 342. The HMD device 344 includes a forward-facing structure 343 that supports a forward-facing or front-facing camera 346 and one or more types of multiple angle-sensitive detectors 348a to 348f (collectively, 348) or other types of optical detectors. As an example, some or all of the angle-sensitive detectors 348 can assist in determining the position and orientation of the device 344 in space, such as light sensors, to detect and use information from one or more external devices (not shown, for example, Figure 2 The angle sensitive detector 348 can be any type of detector that can detect the angle of arrival of light emitted from the light source. Non-limiting examples of angle sensitive detectors include photodiode detectors (e.g., dual-unit detectors, quadrant unit detectors, image sensors), position sensitive detectors using resistor sheets, etc.

[0070] As shown, the front camera 346 and the angle-sensitive detector 348 face forward toward the real scene or environment (not shown) in which the user 342 operates the HMD device 344. More generally, the angle-sensitive detector 348 can be pointed to other areas (e.g., up, down, left, right, back) to detect light from various sources, such as controllers (e.g., held by the user 342) or objects mounted in different locations (e.g., walls, ceilings). The real physical environment may include, for example, one or more objects (e.g., walls, ceilings, furniture, stairs, cars, trees, tracking markers, light sources, or any other type of object). The specific number of sensors 348 can be less (e.g., 2, 4) or more (e.g., 10, 20, 30, 40) than the number of sensors shown. The HMD device 344 may also include one or more additional components that are not attached to the forward-facing structure (e.g., inside the HMD device), such as an IMU (Inertial Measurement Unit) 347 electronic device that measures and reports specific force, angular rate, and / or magnetic field surrounding the HMD device 344 (e.g., using a combination of accelerometers and gyroscopes and optionally magnetometers). The HMD device 344 may also include additional components not shown, including one or more display panels and optical lens systems oriented toward the user's eyes (not shown), and optionally having one or more attached internal motors to change the alignment or other positioning of one or more of the optical lens systems and / or the display panels within the HMD device.

[0071] The illustrated example of the HMD device 344 is supported on the head of the user 342 based at least in part on one or more straps 345 attached to the housing of the HMD device 344 and extending completely or partially around the user's head. Although not shown herein, the HMD device 344 may also have one or more external motors, for example, attached to one or more straps 345, and the automatic corrective action may include using such motors to adjust these straps to modify the alignment or other positioning of the HMD device on the user's head. It should be understood that in addition to or in lieu of the illustrated straps, the HMD device may include other support structures not shown here (e.g., nose pads, chin straps, etc.), and some embodiments may include motors to attach one or more such other support structures to similarly adjust their shape and / or position to modify the alignment or other positioning of the HMD device on the user's head. Other display devices that are not secured to the user's head may be similarly attached to one or more structures, or portions of such structures, that affect the positioning of the display device, and in at least some embodiments may include motors or other mechanical actuators to similarly modify their shape and / or position to modify the alignment or other positioning of the display device relative to one or more pupils of one or more users of the display device.

[0072] Figure 4 An example of a handheld controller 400 is shown in more detail. In practice, an HMD system may include Figure 4 Two handheld controllers similar or identical to the handheld controller 400 of the present invention can be similar or identical to the controllers 182, 208 and 210 discussed above. As shown in the figure, the controller 400 has respective surfaces, and the angle-sensitive detector 402 is located on these surfaces. The angle-sensitive detector 402 is arranged to receive optical signals from various different directions. The controller 400 can have buttons, sensors, lights, controls, knobs, indicators, displays, etc., thereby allowing users to interact in various ways. In addition, as discussed above, in at least some implementations, one of the controller 400 and the HMD device 344 can include multiple light sources, and the other of the controller and the HMD device can include multiple angle-sensitive detectors or other types of detectors or sensors. The technology described herein can be used for various types of position tracking, and is not limited to HMD, controllers, etc.

[0073] Figure 5A schematic block diagram of an HMD device 500 according to one or more implementations of the present disclosure is shown. The HMD device 500 may be similar or identical to the HMD devices discussed elsewhere herein. Thus, the above discussion of the HMD device may also apply to the HMD device 500. In addition, at least some of the components of the HMD device 500 may be present in other components of the HMD system, such as a vertical controller, a base station, etc. Thus, at least some of the following descriptions may apply to these other components.

[0074] The HMD device 500 includes a processor 502, a front-facing or front-facing camera 504, a plurality of angle-sensitive detectors 506 (e.g., a four-unit photodiode, a position-sensitive detector), and optionally an IMU 507 or a plurality of light sources 509. In some implementations, the HMD device 500 may include one of the angle-sensitive detector or the light source, and other components (e.g., a controller, a base station) may include the other of the angle-sensitive detector or the light source. As discussed above, in at least some embodiments, the HMD device 500 may include one or more scatter detection modules or scatter detectors that may be used to detect whether light received by one or more of the angle-sensitive detectors has been scattered or reflected, and if so, should be ignored. The HMD device 500 may include a display subsystem 508 (e.g., two displays and corresponding optical systems). The HMD device 500 may also include a non-transitory data memory 510 that may store instructions or data for position tracking 512, instructions or data for display functions 514 (e.g., games), and / or other programs 516. The HMD system 500 may include Figure 1 Some of the functions of the local computing system 120 or media content provider 190 shown and discussed above may support such as Figure 1 The functionality of the local computing system 120 or media content provider 190 is shown and discussed above.

[0075] The HMD device 500 may also include various I / O components 518, which may include one or more user interfaces (e.g., buttons, touchpads, speakers), one or more wired or wireless communication interfaces, etc. As an example, the I / O components 518 may include a communication interface that enables the HMD device 500 to communicate with an external device 520 via a wired or wireless communication link 522. As non-limiting examples, the external device 520 may include a host, a server, a mobile device (e.g., a smartphone, a wearable computer), a controller, etc. The various components of the HMD device 500 may be packaged in a single housing, or in separate housings (e.g., a host), or any combination thereof.

[0076] It should be understood that the computing systems and devices shown are merely illustrative and are not intended to limit the scope of the present disclosure. For example, HMD 500 and / or external device 520 can be connected to other devices not shown, including through one or more networks such as the Internet or through the World Wide Web. More generally, such computing systems or devices can include any combination of hardware that can interact and perform the described type of functions, for example, when programmed or otherwise configured using appropriate software, including but not limited to desktop computers, laptops, tablet computers, tablet computers or other computers, smart phone computing devices and other mobile phones, Internet devices, PDAs and other electronic managers, database servers, network storage devices and other network devices, wireless phones, pagers, television-based systems (e.g., using set-top boxes and / or personal / digital video recorders and / or game consoles and / or media servers) and various other consumer products including appropriate inter-communication capabilities. For example, in at least some embodiments, the illustrated systems 500 and 520 may include executable software instructions and / or data structures that, when loaded and / or executed by a particular computing system or device, may be used to program or otherwise configure such systems or devices to configure the processors of such systems or devices. Alternatively, in other embodiments, some or all of the software systems may be executed in the memory of another device and communicate with the illustrated computing system / device via inter-computer communication. Additionally, although various items are illustrated as being stored in memory or storage at different times (e.g., when in use), such items or portions thereof may be transferred between memory and storage and / or between storage devices (e.g., at different locations) for purposes of memory management and / or data integrity.

[0077] Thus, in at least some embodiments, the illustrated system is a software-based system including software instructions that, when executed by one or more processors and / or other processor devices, program the one or more processors to automatically perform the described operations of the system. In addition, in some embodiments, part or all of the system may be implemented or provided in other ways, such as at least partially implemented or provided in firmware and / or hardware devices, including but not limited to one or more application specific integrated circuits (ASICs), standard integrated circuits, controllers (e.g., by running appropriate instructions, and including microcontrollers and / or embedded controllers), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), etc. Part or all of the system or data structure may be stored (e.g., as software instruction content or structured data content) on a non-transitory computer-readable storage medium, such as a hard disk or flash drive or other non-volatile storage device, volatile or non-volatile memory (e.g., RAM), a network storage device, or a portable media product (e.g., a DVD disk, a CD disk, an optical disk, a flash memory device, etc.), so as to be read by an appropriate drive or via an appropriate connection. In some embodiments, the system, modules, and data structures may also be transmitted as a generated data signal (e.g., as part of a carrier wave or other analog or digital propagation signal) on various computer-readable transmission media (including wireless and wired / wired based media), which includes wireless and wired / wired based media, and may take various forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). In other embodiments, these computer program products may also take other forms. Therefore, the present disclosure may be implemented with other computer system configurations.

[0078] Figure 66 is a schematic diagram of an environment 600 in which, according to a non-limiting illustrative implementation, a tracking subsystem may be implemented using machine learning techniques to track an HMD device, one or more controllers, or other components, such as the tracking subsystems discussed herein. The environment 600 includes a model training portion 601 and an inference portion 603. In the training portion 601, training data 602 is input into a machine learning algorithm 604 to generate a trained machine learning model 606. The training data may include, for example: labeled data from an angle-sensitive detector that specifies the position and / or orientation of a particular object relative to one or more light sources (e.g., LEDs); labeled or unlabeled scatter detector data (as discussed below); or other types of data. As a non-limiting example, in an embodiment including a component (e.g., HMD, controller) having 30 angle-sensitive detectors, each training sample may include outputs from each or a subset of the angle-sensitive detectors, a known or inferred position or orientation of the component, and information about the position or orientation of one or more light sources. As discussed below, each angle-sensitive detector may output a single data point (e.g., an angle), or may output multiple data points, e.g., two or four signals, each of which indicates the power or intensity of light received at a particular active element (e.g., a sub-detector or cell, a resistor, etc.) of the angle-sensitive detector. The data may also include data from one or more scatter detectors, such as the scatter detectors discussed below. Such data may include polarization information (e.g., type or degree of polarization), information about whether the detected light has been scattered, or other types of data.

[0079] Training data 602 can be obtained from multiple users and / or a single user of the HMD system. Training data 602 can be obtained in a controlled environment and / or during real-world use by a user ("field training"). Further, in at least some implementations, model 606 can be updated or calibrated from time to time (e.g., periodically, continuously, after a specific event) to provide accurate position tracking predictions.

[0080] In the inference portion 603, the runtime data 608 is provided as input to a trained machine learning model 606, which generates a position tracking prediction 610. Continuing with the above example, the output data of the angle-sensitive detector (e.g., intensity data, angle data), optionally information about one or more light sources, and optionally information from one or more scatter detectors can be provided as input to the trained machine learning model 606, which can process the data to predict the position of the component. The tracking prediction 610 can then be provided to one or more components associated with the HMD device, such as one or more VR or AR applications, one or more display or rendering modules, one or more mechanical controls, one or more additional position tracking subsystems, and the like.

[0081] The machine learning techniques used to implement the features discussed herein may include any suitable structure or type of technique. As non-limiting examples, the machine learning model 606 may include one or more of a decision tree, a statistical hierarchical model, a support vector machine, an artificial neural network (ANN) such as a convolutional neural network (CNN) or a recurrent neural network (RNN) (e.g., a long short-term memory (LSTM) network), a mixture density network (MDN), a hidden Markov model, or other models that may be used. In at least some implementations, such as implementations utilizing RNNs, the machine learning model 606 may utilize past input (memory, feedback) information to predict the position of one or more HMD components. These implementations may advantageously utilize sequential data to determine motion information or prior position predictions, which may provide more accurate real-time position predictions.

[0082] Figure 7 is a flow chart of an example method 700 of operating an HMD system to track the position of HMD components during use. The method 700 may be performed by, for example Figure 5 The method 700 is performed by the position tracking system or module 512 of the illustrated HMD system 500. As discussed above, the method 700 may be used to track the position of any component, such as the position of an HMD device, one or more controllers, and the like.

[0083] The illustrated implementation of method 700 begins at step 702, where a first HMD system component having a plurality of angle-sensitive detectors is provided. The plurality of angle-sensitive detectors can be operable to detect light emitted from one or more light sources, which can be fixedly positioned (e.g., mounted to a wall or ceiling), or movable (e.g., coupled to an HMD or controller). In operation, each of the plurality of angle-sensitive detectors collects sensor data in a respective plurality of angle-sensitive detector fields of view at a frame rate. The sensor data can include any type of data that can be used by a control circuit (e.g., a processor) to detect the presence and orientation of the light source relative to the angle-sensitive detector. In at least some implementations, each of the angle-sensitive detectors can include one or more sensors (e.g., photodiodes) having image sensing circuitry and image processing circuitry thereon. The angle-sensitive detectors can output relatively raw data (e.g., light intensity or power data) or processed data (e.g., incident angle data).

[0084] At 704, a second HMD system component including a plurality of light sources (eg, near infrared LEDs) may be provided. For example, the second HMD system component may include a controller, an HMD device, or a light source located at a fixed location (eg, ceiling, wall).

[0085] At 706, at least one processor of the HMD system may illuminate the light source. The light source may be illuminated in a manner that the angle-sensitive detectors may each detect light from a single light source at a time, or more generally, in a manner that the system may be able to determine which light source the light detected by the angle-sensitive detectors is from. This may be accomplished by modulating or multiplexing the illumination of the light source using any suitable type of technique, such as time division multiplexing, wavelength division multiplexing, frequency division multiplexing, polarization multiplexing, or other techniques that allow the system to know the source of the light received from each angle-sensitive detector during use.

[0086] As an example of time division multiplexing, the at least one processor may illuminate only a subset (e.g., one, two, four) of the light sources at a time. For example, the at least one processor may illuminate the light sources in sequence, a subset at a time, and collect sensor data in response to each of the light sources.

[0087] As an example of wavelength division multiplexing, different subsets of light sources can emit light of different wavelengths, and different subsets of angle-sensitive detectors can be operated to sense light of different wavelengths. Thus, light sources with different wavelengths can be illuminated at the same time and detected by corresponding wavelength-sensitive detectors.

[0088] As an example of frequency division multiplexing, a subset of light sources may be illuminated at a determined pattern or frequency detectable by an angle sensitive detector to identify a particular light source of light.

[0089] As an example of polarization multiplexing, subsets of light sources can be differently polarized (e.g., linear polarization, circular polarization), and corresponding subsets of angle-sensitive detectors can be configured to detect light of a specific polarization (e.g., using polarizers that allow light with the corresponding polarization to pass), which allows multiple light sources to be illuminated simultaneously.

[0090] Other example techniques for illuminating the light sources may include one or more of frequency division multiple access or wavelength division multiple access (FDMA or WDMA), time division multiple access (TDMA), code division multiple access (CDMA), and orthogonal frequency division multiple access (OFDMA). In at least some implementations, the illumination patterns or schemes may be configured to be orthogonal in one or more of the time, wavelength, or frequency of the electrical system (e.g., 120 Hz, twice the 60 Hz frequency used in the U.S. electrical system, or 100 Hz, twice the 50 Hz frequency used in the European electrical system). In addition, the modulation scheme may be applied to the amplitude of the light of each individual tag or LED, or to one or more groups of two or more tags or LEDs, or to any combination thereof. In at least some implementations, two or more of the light emitting components, light detecting components, or processing components (e.g., a host system) may be synchronized with each other, which may provide further advantages, for example, during use of CDMA techniques, etc.

[0091] At 708, at least one processor associated with the HMD system may receive sensor data from a plurality of angle-sensitive detectors. As discussed above, for each angle-sensitive detector, the sensor data may indicate an angle of arrival of light emitted from a known light source. At 710, at least one processor associated with the HMD system may optionally receive sensor data from an inertial measurement unit (IMU) for providing inertial tracking functionality, or from one or more additional sensors.

[0092] At 712, at least one processor associated with the HMD system may process the received sensor data. For example, the at least one processor may fuse some or all of the sensor data together to track one or more features present in the environment in which the HMD system operates. The sensor data may include sensor data from multiple angle-sensitive detectors, and optionally sensor data from an IMU or from a camera. For example, the at least one processor may process the sensor data using a machine learning model (e.g., model 606) or another solver. As discussed further below, in at least some implementations, the at least one processor may ignore data from one or more sensors that are determined to have likely received light that has been scattered or reflected.

[0093] At 714, at least one processor associated with the HMD system can track the position (e.g., location, orientation, or movement) of components of the HMD system in real time during use of the HMD system by a user in the environment. As discussed above, method 700 can continue during operation of the HMD to continuously track the position of components of the HMD system.

[0094] Figure 8 A perspective view of an example angle-sensitive detector 800 that can be used in one or more implementations of the present disclosure is shown. In this example, the angle-sensitive detector 800 includes an angle-sensitive photodiode structure 804. The angle-sensitive photodiode structure 804 includes a photodiode 806, a second linear polarizer 808, a spatially varying polarizer 810, and a first linear polarizer 812. The photodiode 806 can be any device that receives light, determines the intensity associated with the light, and outputs a signal (or data) representing the intensity. The first linear polarizer 812 and the second linear polarizer 808 can each be any type of filter on which light is impinged. The first linear polarizer 812 and the second linear polarizer 808 can output a linear polarization component (e.g., vertical polarization or horizontal polarization) of the impinging light, and filter out (e.g., reflect or block, absorb) other components of the impinging light.

[0095] In at least some implementations, the spatially varying polarizer 810 can be formed of a multi-twisted retarder (MTR), which is a wave plate-like retardation film that provides precise and customized levels of broadband, narrowband, or multi-band retardation in a single thin film. More specifically, the MTR includes two or more twisted liquid crystal (LC) layers on a single substrate, and has a single alignment layer. Subsequent LC layers are aligned directly through the previous layers, allowing simple manufacturing, enabling automatic layer alignment, and producing a monolithic film with a continuously varying optical axis.

[0096] The spatially varying polarizer 810 may include a wave retarder formed of a birefringent material. Birefringence is the property of a material having a refractive index that depends on the polarization and propagation direction of the light. The wave retarder changes the polarization state or phase of the light propagating through the wave retarder. The wave retarder may have a slow axis (or extraordinary axis) and a fast axis (ordinary axis). When polarized light travels through the wave retarder, light along the fast axis travels faster than light along the slow axis.

[0097] The second linear polarizer 808, the spatially varying polarizer 810, and the first linear polarizer 812 may be stacked on the photodiode 806, as shown in FIG. Figure 8As shown, and stacked continuously on the photodiode 806. It should be noted that although polarizers 812, 808 are described herein as linear polarizers, in various embodiments, polarizers 812, 808 can be non-linear polarizers, and can be, for example, elliptical polarizers or circular polarizers. Polarizers 812, 808 can have the same filtering characteristics, and can similarly or identically suppress or pass light with a specific polarization. In this simplified example, the angle-sensitive detector 800 includes a cover 814 having a hole 816 therein, which allows light 818 from a light source 820 to pass through. As shown, the light 818 passing through the hole 816 forms a spot 822, which is electrically characterized to determine the angle of the light 818, and therefore the angle of the light source 820 relative to the angle-sensitive detector 800. As discussed below, the systems and methods of the present disclosure can utilize multiple light sources and angle-sensitive detectors to determine the position of components of the HMD system.

[0098] Fig. 9 The first linear polarizer 812, the spatially varying polarizer 810, and the second linear polarizer 808 of the angle-sensitive photodiode structure 804 are shown, as well as the polarization of light 818 or light spot 822 that passes through them to the photodiode 806. Initially, the light 818 impinges on the first linear polarizer 812. The light 818 can have any polarization and thus can be referred to as unpolarized in at least some implementations. In at least some implementations, the light can be linearly polarized, circularly polarized, or substantially elliptically polarized.

[0099] The first linear polarizer 812 allows the linear polarization component 824 of the light 818 to pass through, and blocks (absorbs or reflects) the remaining polarization components of the light 818. Although the first linear polarizer 812 is shown as a vertical polarization filter, in various embodiments, the first linear polarizer 812 can be a horizontal polarization filter or a circular polarization filter, etc.

[0100] The linear polarization component 824 then impinges upon the spatially varying polarizer 810 located below the first linear polarizer 812. The spatially varying polarizer 810 is tuned to have light polarization characteristics that vary depending on the position at which the linear polarization component 824 (or any impinging light) on the spatially varying polarizer 810 impinges upon the spatially varying polarizer 810. Fig. 9 In the example shown, the spatially varying polarizer 810 varies the illumination linear polarization component 824 .

[0101] The manner in which the spatially varying polarizer 810 alters the illumination linear polarization component 824 varies depending on the location at which the illumination linear polarization component 824 impinges on the spatially varying polarizer 810. This location may be substantially the same as the location at which the light 818 impinges on the angle-sensitive photodiode structure 804.

[0102] In this illustrative example, at the first end 826 (upper right as shown in the figure) of the spatially varying polarizer 810, the spatially varying polarizer 810 maintains the impinging linear polarization component 824 as a vertically polarized light signal. The spatially varying polarizer 810 allows vertically polarized light to pass through, and blocks other polarization components. The linear polarization component 824 impinging on the first end 826 passes through as is.

[0103] As a non-limiting example, the polarization filtering characteristics of the spatially varying polarizer 810 can gradually change with the distance from the first end. At the second end 828 (lower left as shown in the figure) of the spatially varying polarizer 810, the spatially varying polarizer 810 almost converts the vertically polarized irradiation linear polarization component 824 into a horizontally polarized light signal. In particular, at the second end 828, the spatially varying polarizer 810 has a linear polarization orientation of 175°. Thus, at the second end 828, the spatially varying polarizer 810 outputs light having a horizontal polarization component greater than the vertical polarization component. In contrast, near the center of the spatially varying polarizer 810, the spatially varying polarizer 810 has a linear polarization orientation of approximately 135°, and thus, the spatially varying polarizer 810 rotates the polarization of the irradiation linear polarization component 824 (having vertical polarization) toward horizontal polarization by an angle of approximately 45°. Light leaving from near the center of the spatially varying polarizer 810 has a vertical polarization component having the same amplitude as its horizontal polarization component.

[0104] The spatially varying properties of spatially varying polarizer 810 allow identification of the location or likely location of linear polarization component 824. Fig. 9 As shown, spatially varying polarizer 810 allows filtered light 830 to pass through. The intensity of the horizontally polarized or vertically polarized filtered light 830 represents the location or possible location of the linear polarization component 824 impinging on the spatially varying polarizer 810. When the linear polarization component 824 impinges on the first end 826, the filtered light 830 has the highest vertical polarization amplitude. The vertical polarization amplitude can be inversely proportional to the distance from the first end 826.

[0105] The filtered light 830 then strikes a second linear polarizer 808, which is used to remove any horizontal components of the filtered light 830 and allow the vertical light component to pass through. The second linear polarizer 808 allows the filtered linear polarization component 832 to pass through. The second linear polarizer 808 can ensure that the light passing through the photodiode 806 includes only vertically polarized light and does not include horizontally polarized light.

[0106] Photodiode 806 receives filtered linear polarization component 832 and detects the intensity of filtered linear polarization component 832. The intensity of filtered linear polarization component 832 represents the location or series of locations where light 818 impinges on spatially varying polarizer 810, and thus, on angle-sensitive photodiode structure 804.

[0107] It should be noted that the reference Fig. 9 The specific polarizations described are by way of example for ease of description. In alternative embodiments, different polarizers, polarizations, or polarization patterns may be employed. For example, instead of linear polarization, the first linear polarizer 812 and the second linear polarizer 808 and the spatially varying polarizer 810 may utilize circular polarization, elliptical polarization, or any other type of polarization.

[0108] Refer to Figure 8 , the size and position of the hole 816 determine the size of the spot 822 formed on the angle-sensitive photodiode structure 804. The intensity detected by the photodiode 806 represents the intensity of the spot 822 that has passed through the first linear polarizer 812 and the second linear polarizer 808 and the spatially varying polarizer 810 (or filtered by them). The intensity of the spot 822 can be the sum of the intensities of the light rays that constitute the spot 822. The fact that the linear polarization component 824 of the spot 822 is irradiated on an area of ​​the spatially varying polarizer 810 rather than on a point provides additional degrees of freedom in the design of the spatially varying polarizer 810 to achieve improved position detection. The spatially varying polarizer 810 can have characteristics that vary by area so that the position detection of the spot 822 can be improved.

[0109] It should be noted that in some embodiments, one of the first linear polarizer 812 and the second linear polarizer 808 may be omitted. In embodiments, the first linear polarizer 812 may be omitted, and only horizontally polarized light may be emitted for position or angle determination.

[0110] In an embodiment, improved position detection may be achieved by using a photodiode 806 having multiple area units.

[0111] Fig.10 The first linear polarizer 812, the spatially varying polarizer 810, and the second linear polarizer 808 of the angle sensitive photodiode structure 804 are shown, as well as the polarization of light 818 or spot 822 that passes through them to the photodiode 806. Fig.10, photodiode 806 is a quadrant unit ("quad-unit") photodiode, including four independent photodiode active regions or elements 802a to 802d separated by a small gap. It should be understood that other types of angle-sensitive detectors may also be used, such as photodiode detectors with fewer or more units, position-sensitive detectors (PSDs), etc.

[0112] The active region (e.g., anode) of each element 802a to 802d is individually usable so that a light spot that illuminates a single quadrant can be electrically characterized as being only in that quadrant. The energy distribution of the light spot between adjacent elements 802a to 802d, and the difference in the electrical contribution to each element define the relative position of the light spot relative to the center of the angle-sensitive detector. The relative intensity distribution on elements 802a to 802d combined with the relative intensity distribution of the spatially varying polarizer 810 can be used to determine the position of the light spot.

[0113] In an embodiment, the spatially varying polarizer 810 can be turned off to identify the baseline intensity. The spatially varying polarizer 810 can be coupled to a controller. The controller can be a microcontroller or microprocessor, etc., or can be one or more controllers 182 or processors 502 described herein, which can turn the spatially varying polarizer 810 on or off. When the spatially varying polarizer 810 is turned on, the spatially varying polarizer 810 filters the light as described herein. Conversely, when the spatially varying polarizer 810 is turned off, the spatially varying polarizer 810 can stop polarization filtering and allow the linear polarization component 824 to pass as is.

[0114] When the spatially varying polarizer 810 is turned off, the photodiode 806 detects the intensity of the light 818 (or the light spot 822) without being affected by the attenuation caused by the spatially varying polarizer 810 acting in conjunction with the first linear polarizer 812 and the second linear polarizer 88. The detected intensity can be used as a baseline intensity or a maximum detected intensity. The baseline intensity or the maximum detected intensity can correspond to the intensity of the light 818 irradiated on the first end 826.

[0115] When spatially varying polarizer 810 is turned on, photodiode 806 detects the intensity of light 818 (or light spot 822) with the position-dependent polarization filter in place. The relationship between the detected intensity of light 818 (or light spot 822) with the position-dependent polarization filter in place and the baseline intensity indicates the position or range of positions at which light 818 impinges on angle-sensitive photodiode structure 804.

[0116] As described herein, the polarization conversion performed by spatially varying polarizer 810 combined with the filtering by first linear polarizer 812 and second linear polarizer 808 results in a spatially varying amplitude (or intensity) attenuation of light 818. The amplitude (or intensity) is in turn detected by photodiode 806 and used for position determination.

[0117] Fig.11A and Fig. 11B A top view and a perspective view, respectively, of an example angle-sensitive detector 1100 that may be used in one or more of the implementations of the present disclosure are shown. In this example, the angle-sensitive detector 1100 includes a quadrant-unit ("quad-unit") photodiode that includes four independent photodiode active regions or elements 1102A-1102D separated by a small gap on a common substrate 1104. It should be understood that other types of angle-sensitive detectors may also be used, such as photodiode detectors with fewer or more units, position-sensitive detectors, etc.

[0118] In the non-limiting illustrated example, the active region (e.g., anode) of each element 1102A to 1102D is individually usable so that a light spot illuminating a single quadrant can be electrically characterized as being only in that quadrant. As the light spot translates across the angle-sensitive detector 1100, the energy distribution of the light spot between adjacent elements 1102A to 1102D, and the difference in electrical contribution to each element defines the relative position of the light spot relative to the center of the angle-sensitive detector. The relative intensity distribution across elements 1102A to 1102D can be used to determine the position of the light spot.

[0119] In this simplified example, the angle-sensitive detector 1100 includes a cover 1110 having a hole 1108 therein that allows light 1114 from a light source 1112 to pass therethrough. As shown, the light 1114 passing through the hole 1108 forms a light spot 1106 that can be electrically characterized to determine the angle of the light 1114, and thus the angle of the light source 1112 relative to the angle-sensitive detector 1100. As discussed below, the systems and methods of the present disclosure can utilize multiple light sources and angle-sensitive detectors to determine the position of components of an HMD system.

[0120] It should be understood that the angle sensitive detectors of the present disclosure may include one or more of any suitable type of detector, including a four-element photodiode detector, a position sensitive detector (PSD) using a resistive sheet, a photodiode detector having fewer (e.g., 2) or more (e.g., 16) independent sensitive elements, or any other detector capable of detecting the angle of arrival of light emitted from a light source. In addition, as discussed below, in at least some implementations, the angle sensitive detectors or light sources of the present disclosure may utilize various optical components, such as filters, lenses, polarizers, etc., to improve the functionality of the systems and methods discussed herein.

[0121] Fig.12 12 is a simplified diagram of an environment 1200 of an HMD system that uses light sources and angle-sensitive detectors to determine positions of components of the HMD system, according to a non-limiting illustrative implementation. In this example, a first component 1202, such as an HMD, includes a plurality of light sources 1206 (two are shown, 1206a and 1206b), and a second component 1204, such as a controller of the HMD system, includes a plurality of angle-sensitive detectors 1208 (two are shown, 1208a and 1208b). The angle-sensitive detectors 1208a and 1208b are spaced apart from each other on the second component 1204 by a known distance d. 1 , and light sources 1206a and 1206b are spaced apart from each other by a known distance d on the first component 1202 2 The first component and the second component may be any components of the HMD system, such as an HMD, a controller, a base station, a fixed or mobile light source, a fixed or mobile angle-sensitive detector, etc.

[0122] In this example, angle-sensitive detector 1208a is operable to determine that light arrives from light source 1206a at angle 1210, and that light arrives from light source 1206b at angle 1212. Similarly, angle-sensitive detector 1208b is operable to determine that light arrives from light source 1206b at angle 1214, and that light arrives from light source 1206a at angle 1216. Given the detected arrival angles 1210, 1212, 1214, and 1216 and a known geometric relationship between light source 1206 and detector 1208 (e.g., distance d 1 and d 2 ), methods (e.g., triangulation) may be used to determine and track the relative position, orientation, or movement between the first component 1202 and the second component 1204. As discussed above, one or more solvers or machine learning methods may be used to determine the positions of the components using sensor data from an angle-sensitive detector and / or light source data indicating information about the light sources of the HMD system.

[0123] Fig.131300 is an illustration of an example light source 1302 and an angle-sensitive detector 1304 of the present disclosure. The light source 1302 and the angle-sensitive detector 1304 may be similar or identical to any light source and angle-sensitive detector discussed herein, and may be used in any implementation of the present disclosure. In the illustrated example, the light source 1302 may include an optical subsystem 1306, and the angle-sensitive detector 1304 may include an optical subsystem 1308. The optical subsystems 1306 and 1308 may be the same or different from each other, and may each include one or more optical components. The optical subsystems 1306 and 1308 may be integrated with the light source 1302 and the angle-sensitive detector 1304, or may be separate components. Non-limiting examples of optical components include one or more lenses, one or more polarizers, one or more filters, one or more apertures, and the like. In at least some implementations, a subset of the light sources may include one type of optical subsystem, and one or more other subsets of the light sources may include another type of optical subsystem. Similarly, a subset of the angle sensitive detectors may include one type of optical subsystem, and one or more other subsets of the angle sensitive detectors may include another type of optical subsystem. As an example, the optical subsystem may include a filter to filter out visible light or other types of light. In addition, as discussed above, the optical subsystem may include components that facilitate one or more of the various types of multiplexing discussed above, which allows multiple light sources to be illuminated simultaneously without confusion as to the source of the emitted light.

[0124] Fig.14 1400 is an illustration of a scatter detection module or scatter detector 1402 of the present disclosure, which can be used to determine whether light received by one or more optical detectors (e.g., angle-sensitive or other types of detectors) has been reflected or scattered before being received by the one or more optical detectors. Using this information, at least one processor can be operated to ignore light data determined to be scattered or reflected light signals because such signals do not directly indicate the location of the light source from which the signal was emitted. In at least some implementations, the scatter detector 1402 can be a separate component used in conjunction with one or more optical detectors for position tracking. In other implementations, the scatter detector 1402 can be integrated into one or more optical detectors (e.g., angle-sensitive detectors) for position tracking. In any embodiment of the present disclosure, one or more scatter detectors 1402 can be used. In addition, various machine learning or artificial intelligence based methods can be used to process scatter detector data to improve the position tracking capabilities of the tracking system of the present disclosure. For example, machine learning or other AI methods can be used to train the tracking system to use polarization information to help improve tracking accuracy.

[0125] In the non-limiting illustrated example, a scatter detector 1402 is shown, and a first light source 1408 and a second light source 1410 are also shown. In practice, there may be multiple scatter detectors and multiple light sources. As a non-limiting example, the scatter detector 1402 may be located on one of the HMD and the controller, and the light sources 1408 and 1410 may be located on the other of the HMD and the controller. In at least some implementations, one or more of the scatter detector 1402 and the light sources 1408 and 1410 may be located on a fixed object (e.g., a wall, a ceiling, a bracket) or a movable object (e.g., an HMD, a controller), or coupled to a fixed object (e.g., a wall, a ceiling, a bracket) or a movable object (e.g., an HMD, a controller). The scatter detector 1402 and the light sources 1408 and 1410 may be similar or identical to any light source and scatter detector discussed herein, and may be used in any implementation of the present disclosure.

[0126] In the illustrated example, the scatter detector 1402 may include an optical detector 1404 and an optical subsystem 1406, which may optionally be an angle-sensitive detector. The light source 1408 may include a light emitter 1412 (e.g., an LED) that emits light 1420 and an optical subsystem 1414, and the light source 1410 may include a light emitter 1416 that emits light 1422 and an optical subsystem 1414. Some or all of the optical subsystems 1406, 1414, and 1418 may be the same or different from each other, and may each include one or more optical components. The optical subsystems 1406, 1414, and 1418 may be integrated with the detector 1404 and the light sources 1408 and 1410, respectively, or may be separate components. Non-limiting examples of optical components include one or more lenses, one or more polarizers, one or more wave retarders, one or more filters, one or more apertures, and the like. In at least some implementations, a subset of the light sources may include one type of optical subsystem, and one or more other subsets of the light sources may include another type of optical subsystem. Similarly, a subset of scatter detectors 1402 may include one type of optical subsystem, and one or more other subsets of scatter detectors may include another type of optical subsystem. As an example, the optical subsystem may include a filter to filter out visible light or other types of light. In addition, as discussed above, the optical subsystem may include components that facilitate one or more of the various types of multiplexing discussed above, which allows multiple light sources to be illuminated simultaneously without confusion as to the source of the emitted light.

[0127] The designs of optical subsystems 1406, 1414, and 1418 can be coordinated so that scatter detector 1402 can be operated to detect whether light from light sources 1408 and 1410 has been scattered or reflected, or whether the light reaches the scatter detector directly without scattering or reflection. For example, scatter detector 1402 can be operated to detect changes in the polarization type or degree of polarization of light emitted by the light source due to scattering or reflection. In the illustrated example, light 1420 from light source 1408 is directly received by scatter detector 1402, while light 1422 from light source 1410 is reflected off surface 1423 as light 1424 received by scatter detector 1402. In this example, light 1420 indicates the relative position of light source 1408 relative to scatter detector 1402, while light 1424 reflected from surface 1423 does not indicate the relative position of light source 1410 relative to scatter detector 1402. Thus, by detecting that light 1424 has been scattered or reflected, the tracking system can ignore or block light signals from one or more sensors, such as sensors positioned in a similar orientation as the scatter detectors, when performing position tracking, thereby improving the system's position tracking capabilities.

[0128] There are a variety of configurations that can allow scatter detector 1402 to detect whether light from a light source has been scattered or reflected and should therefore be ignored by one or more detectors. In general, in at least some implementations, light emitted by light sources 1408 and 1410 can be polarized in a determined manner by optical systems 1414 and 1418, respectively, and scatter detector 1402 can be configured to distinguish between light received directly from light sources 1408 and 1410 and light from the light sources that has been scattered or reflected before being received by the scatter detector. For example, the type or degree of polarization of light from the light sources can change due to scattering or specular reflection, and scatter detector 1402 can be configured to detect such a change. As a non-limiting example, optical subsystems 1414 and 1418 of light sources 1408 and 1410, respectively, can include one of a right-handed circular polarizer or a left-handed circular polarizer, and optical subsystem 1406 of scatter detector 1402 can include the other of a right-handed circular polarizer or a left-handed circular polarizer. For example, optical subsystems 1414 and 1418 of light sources 1408 and 1410 can include right-handed circular polarizers, respectively, and optical subsystem 1406 of scatter detector 1402 can include left-handed circular polarizers. In this configuration, optical subsystem 1406 of scatter detector 1402 can be used to detect light that is reflected from a depolarized surface (e.g., having random polarization) or light that is reflected from a non-depolarized surface (e.g., glass, metal, acrylic, etc.) and is left-handed circularly polarized after reflection. If such light is above a determined threshold, the tracking system can ignore signals from one or more detectors that are determined to be likely to also receive reflected light or scattered light.

[0129] exist Fig.15 An example of such a configuration is shown in diagram 1500 of Fig.15 Scatter detector 1502 and light source 1504 are shown. Light source 1504 includes a light emitter 1506 (e.g., an LED) and an optical subsystem including a right-handed circular polarizer 1508. In this implementation, circular polarizer 1508 includes a linear polarizer 1510 and a quarter wave retarder or wave plate 1512, and provides light 1522 having right-handed circular polarization.

[0130] The scatter detector 1502 includes an optical detector 1514 (e.g., a four-element detector, a single-element detector) and an optical subsystem including a left-handed circular polarizer 1516. The left-handed circular polarizer 1516 includes a quarter-wave retarder or wave plate 1518 and a linear polarizer 1520. Since the scatter detector 1502 includes a circular polarizer of opposite handedness to the circular polarizer 1508 of the light source, the scatter detector will detect light reflected via specular reflection due to the switching of the handedness of the reflected circularly polarized light to the opposite handedness (i.e., from right-handed to left-handed in this example).

[0131] In operation, when scatter detector 1502 detects light that has been scattered or reflected (e.g., above a determined threshold), the tracking system can block or ignore light from one or more optical sensors (e.g., sensors positioned or oriented similarly to the scatter detector) that may have received the same light.

[0132] Fig.16 Information 1600 shows a front view of an example HMD device 1644 when worn on the head of a user 1642. The HMD device 1644 includes a front-facing structure 1643 supporting a front-facing or front-facing camera 1646 and a plurality of angle-sensitive detectors 1648a-1648f (collectively, 1648) of one or more types. As an example, some or all of the angle-sensitive detectors 1648 can help determine the position and orientation of the device 1644 in space, such as light sensors, to detect and use information from one or more external devices (not shown, for example, Figure 2 The angle sensitive detector 1648 can be any type of detector operable to detect the angle of arrival of light emitted from the light source. Non-limiting examples of angle sensitive detectors include photodiode detectors (e.g., dual-unit detectors, quadrant unit detectors), position sensitive detectors using resistor sheets, and the like.

[0133] As shown, the front camera 1646 and the angle-sensitive detector 1648 face forward toward the real scene or environment (not shown) in which the user 1642 operates the HMD device 1644. More generally, the angle-sensitive detector 1648 can be directed to other areas (e.g., up, down, left, right, back) to detect light from various sources such as a controller (e.g., held by the user 1642) or objects mounted in different locations (e.g., walls, ceilings). The real physical environment may include, for example, one or more objects (e.g., walls, ceilings, furniture, stairs, cars, trees, tracking markers, light sources, or any other type of object). The specific number of sensors 1648 can be less (e.g., 2, 4) or more (e.g., 10, 20, 30, 40) than the number of sensors shown. The HMD device 1644 may also include one or more additional components that are not attached to the forward-facing structure (e.g., inside the HMD device), such as an IMU (inertial measurement unit) 1647 electronic device that measures and reports the specific force, angular rate, and / or magnetic field surrounding the HMD device 1644 (e.g., using a combination of accelerometers and gyroscopes and, optionally, a magnetometer). The HMD device 1644 may also include additional components not shown, including one or more display panels and optical lens systems that face toward the user's eyes (not shown), and optionally have one or more attached internal motors to change the alignment or other positioning of the one or more optical lens systems and / or display panels within the HMD device.

[0134] The illustrated example of the HMD device 1644 is supported on the head of the user 1642 based at least in part on one or more straps 1645 attached to the housing of the HMD device 1644 and extending completely or partially around the user's head. Although not shown herein, the HMD device 1644 may also have one or more external motors, for example, attached to one or more straps 1645, and the automatic corrective action may include using such motors to adjust these straps to modify the alignment or other positioning of the HMD device on the user's head. It should be understood that in addition to or in lieu of the illustrated straps, the HMD device may include other support structures (e.g., nose pads, chin straps, etc.) not shown here, and some embodiments may include motors to attach one or more such other support structures to similarly adjust their shape and / or position to modify the alignment or other positioning of the HMD device on the user's head. Other display devices that are not secured to the head of a user may be similarly attached to one or more structures or portions of such structures that affect the positioning of the display device, and in at least some embodiments may include motors or other mechanical actuators to similarly modify their shape and / or position to modify the alignment or other positioning of the display device relative to one or more pupils of one or more users of the display device.

[0135] The HMD device 1644 also includes a plurality of scatter detectors 1650, 1652, and 1666. The scatter detectors 1650, 1652, and 1666 may be similar or identical to any of the scatter detectors discussed herein, and may be effective to detect whether light 1660, 1664, and 1672 from light sources 1658, 1662, and 1670, respectively, associated with the HMD device 1644, has been reflected or scattered from a surface before reaching the HMD device. As discussed above, when light that has been scattered is detected, sensor data from one or more sensors that are determined to be likely to have received the same light may be ignored.

[0136] In at least some implementations, a single scatter detector may be provided for all of the detectors 1648. In other implementations, a separate scatter detector may be provided for each of the detectors 1648, or a scatter detector may be included as part of one or more of the detectors 1648. In the illustrated simplified example, the scatter detectors 1650 located to the right of the forward-facing structure 1643 correspond to the detectors 1648a, 1648b, and 1648e, which are used to detect light from a light source (e.g., light source 1658) in the area 1654 to the right of the user 1642. That is, if the scatter detector 1650 detects light that has been reflected or scattered, the tracking system may ignore signals from one or more of the detectors 1648a, 1648b, and 1648e that are determined to be likely receiving the same light due to their similar orientation to the scatter detector 1650. Similarly, scatter detector 1652 located on the left side of forward facing structure 1643 corresponds to detectors 1648c, 1648d, and 1648g, which are used to detect light from a light source (e.g., light source 1662) in region 1656 to the left of user 1642. Scatter detector 1666 on the upper region of forward facing structure 1643 corresponds to detector 1648f, which is used to detect light from a light source (e.g., light source 1670) in region 1668 above user 1642. As discussed above, multiplexing (e.g., time, wavelength, mode, code) can be used to allow the system to know from which light source or groups of light sources detectors 1648, 1650, 1652, and 1666 are receiving light.

[0137] Fig.17 A perspective view of an angle sensitive detector or scatter detector 1700 that may be used in one or more implementations of the present disclosure is shown. In this non-limiting example, the scatter detector 1700 includes a quadrant cell ("quad cell") photodiode that includes four independent photodiode active regions or elements 1702A through 1702D separated by a small gap on a common substrate 1704. It should be understood that other types of detectors may also be used, such as photodiode detectors with fewer or more cells, position sensitive detectors, etc.

[0138] In the non-limiting illustrated example, the active area (e.g., anode) of each element 1702A to 1702D is individually usable so that a light spot illuminating a single quadrant can be electrically characterized as being only in that quadrant. As the light spot is translated across the detector 1700, the energy distribution of the light spot between adjacent elements 1702A to 1702D, and the difference in electrical contribution to each element defines the relative position of the light spot relative to the center of the detector. The relative intensity distribution across the elements 1702A to 1702D can be used to determine the position of the light spot.

[0139] In this simplified example, the detector 1700 includes an opaque cover or mask 1710 having a hole 1708 that allows light 1714 from a light source 1712 to pass through. As shown, the light 1714 passing through the hole 1708 forms a light spot 1706, which can be electrically characterized to determine the angle of the light 1714, and thus the angle of the light source 1712 relative to the detector 1700. As discussed below, the systems and methods of the present disclosure can utilize multiple light sources and detectors to determine the position of components of the HMD system.

[0140] In the illustrated example, the first circular polarizer 1716 is located near (e.g., adjacent) the light source 1712, and the second polarizer 1718 is located near the detector 1700. In at least some implementations, light emitted by the light source 1712 can be polarized in a determined manner by the first circular polarizer 1716 of the scatter detector 1700, and the second circular polarizer 1718 can be configured to distinguish between light received directly from the light source 1712 and light from the light source that has been scattered or reflected before being received by the scatter detector 1700. As a non-limiting example, one of the first circular polarizer 1716 and the second circular polarizer 1718 can include one of a right-handed or left-handed circular polarizer, respectively, and the other of the first circular polarizer 1716 and the second circular polarizer 1718 can include the other of the right-handed or left-handed circular polarizers. For example, the first circular polarizer 1716 of the light source 1712 can include a right-handed circular polarizer, and the second circular polarizer 1718 of the scatter detector 1700 can include a left-handed circular polarizer. In this configuration, the second circular polarizer 1718 of the scatter detector 1700 can be used to detect light that is reflected from a depolarized surface (e.g., having random polarization) or light that is reflected from a non-depolarized surface (e.g., glass, metal, acrylic, etc.) and is left circularly polarized after reflection. If such light is above a determined threshold, the tracking system can ignore signals from one or more detectors that are determined to be likely to also receive reflected light or scattered light.

[0141] Fig.18 1 is a flow chart of an example method 1800 of operating an HMD system to track the position of HMD components during use. The method 1800 may be performed by, for example, Figure 5 The method 1800 is performed by the position tracking system or module 512 of the illustrated HMD system 500. As discussed above, the method 1800 may be used to track the position of any component, such as an HMD device wearable on a user's head, one or more handheld controllers, and the like.

[0142] The illustrated implementation of method 1800 begins at step 1802, where a first HMD system component having a plurality of angle-sensitive detectors is provided. The plurality of angle-sensitive detectors may be operable to detect light emitted from one or more light sources, which may be fixedly positioned (e.g., mounted to a wall or ceiling), or movable (e.g., coupled to an HMD headset or controller). In operation, each of the plurality of angle-sensitive detectors collects sensor data in a respective field of view of the plurality of angle-sensitive detectors at a frame rate. The sensor data may include any type of data that a control circuit (e.g., a processor) can use to detect the presence and orientation of a light source relative to the angle-sensitive detector. In at least some implementations, each of the angle-sensitive detectors may include one or more sensors (e.g., photodiodes) having image sensing circuitry and optional image processing circuitry thereon. The angle-sensitive detectors may output relatively raw data (e.g., light intensity or power data) or processed data (e.g., incident angle data).

[0143] At 1804, a second HMD system component including a plurality of light sources (e.g., near infrared LEDs) may be provided. For example, the second HMD system component may include a controller, an HMD headset, or a light source located at a fixed location (e.g., ceiling, wall).

[0144] At 1806, at least one processor of the HMD system may illuminate the light source. The light source may be illuminated in a manner that the angle-sensitive detectors may each detect light from a single light source at a time, or more generally, in a manner that the system may determine which light source the light detected by the angle-sensitive detectors is from. This may be accomplished by multiplexing the illumination of the light source using any suitable type of multiplexing, such as time division multiplexing, wavelength division multiplexing, frequency division multiplexing, polarization multiplexing, or other techniques that allow the system to know the source of the light received from each angle-sensitive detector during use.

[0145] As an example of time multiplexing, at least one processor may illuminate only a subset (e.g., one, two, four) of the light sources at a time. For example, at least one processor may sequentially illuminate the light sources, a subset at a time, and collect sensor data in response to each of the light sources.

[0146] As an example of wavelength division multiplexing, different subsets of light sources can emit light of different wavelengths, and different subsets of angle-sensitive detectors can be used to sense light of different wavelengths. Thus, light sources with different wavelengths can be illuminated at the same time and detected by corresponding wavelength-sensitive detectors.

[0147] As an example of frequency division multiplexing, a subset of light sources may be illuminated at a determined pattern or frequency detectable by an angle sensitive detector to identify a particular light source of light.

[0148] As an example of polarization multiplexing, subsets of light sources can be differently polarized (e.g., linear polarization, circular polarization), and corresponding subsets of angle-sensitive detectors can be configured to detect light of a specific polarization (e.g., using polarizers that allow light with the corresponding polarization to pass), thereby allowing multiple light sources to be illuminated simultaneously.

[0149] Other non-limiting example technologies for illuminating the light source may include frequency division multiple access or wavelength division multiple access (FDMA or WDMA), time division multiple access (TDMA), code division multiple access (CDMA), orthogonal frequency division multiple access (OFDMA), etc.

[0150] At 1808, at least one processor associated with the HMD system may receive sensor data from a plurality of angle-sensitive detectors. As discussed above, for each angle-sensitive detector, the sensor data may indicate an angle of arrival of light emitted from a known light source. At 1810, at least one processor associated with the HMD system may optionally receive sensor data from an inertial measurement unit (IMU) operable to provide inertial tracking functionality, or receive sensor data from one or more additional sensors.

[0151] At 1812, at least one processor associated with the HMD system may process the received sensor data, including detecting corrupted data, as further discussed below. For example, at least one processor may fuse some or all of the sensor data together to track one or more features present in the environment in which the HMD system operates. The sensor data may include sensor data from multiple angle-sensitive detectors, and optionally sensor data from an IMU, camera, or other sensor data. For example, at least one processor may process the sensor data using a machine learning model (e.g., model 606) or another solver. As further discussed below, in at least some implementations, at least one processor may ignore data from one or more sensors that is determined to be corrupted, for example, data from light that is unlikely to be received directly from a light source of the HMD system, but from light that has been scattered, reflected, or from another light source.

[0152] At 1814, at least one processor associated with the HMD system may track the position (e.g., location, orientation, or movement) of components of the HMD system in real time during use of the HMD system by a user in the environment. As discussed above, method 1800 may continue during operation of the HMD to continuously track the position of components of the HMD system.

[0153] In processing the received sensor data, the control circuitry may identify one or more corrupted sensor data samples, wherein each of the one or more corrupted sensor data samples includes sensor data samples from one of the plurality of angle-sensitive detectors that are determined to be unlikely to represent light that the angle-sensitive detector has directly received from the one or more light sources. In at least some implementations, the control circuitry is configured to ignore the corrupted sensor data samples during the tracking process, and may continue to ignore samples from the optical detector for a fixed or variable period of time.

[0154] Identification of corrupted sensor data samples may be based at least in part on a known geometry of at least one of the first head mounted display system component and the second head mounted display system component. For example, at least one projection model of the first head mounted display system component or the second head mounted display system component may be utilized to determine which of a plurality of angle sensitive detectors may not be directly receiving light from one or more of a plurality of light sources, and data from these detectors may be ignored for a period of time. For example, received sensor data samples may be compared to at least one projection model, and received sensor data samples that do not match at least one projection model within a defined threshold may be identified as corrupted sensor data samples that should be ignored.

[0155] Additionally or alternatively, identification of one or more corrupted sensor data samples can be based at least in part on one or more of a past position or orientation of at least one of the first head mounted display system component and the second head mounted display system component, a current position or orientation of at least one of the first head mounted display system component and the second head mounted display system component, or a predicted future position or orientation of at least one of the first head mounted display system component and the second head mounted display system component.

[0156] In at least some implementations, the number of samples or the time period for which one or more detectors are disabled can be selectively varied based on various criteria, such as based on actual or predicted movement (e.g., direction, speed, rotation) of at least one of the first head mounted display system component or the second head mounted display system component. As an example, a comparison with a projection model can indicate that a detector is blocked by an object (e.g., a wall, a person, another component) or facing away from a light source of the HMD system, and thus is unlikely to receive light directly from one of the light sources of the HMD system for a period of time. The control circuitry can track the position or movement of the one or more components to determine a time period after which the detector is expected to again receive light from at least one of the plurality of light sources, at which time the system can again use samples from the detector for tracking purposes.

[0157] The head mounted display system components may include a head mounted display device, a controller, a base station, or other HMD system components that can be worn on the head of a user. As discussed elsewhere herein, to process the received sensor data, the control circuitry may provide the received sensor data as input to one or more trained machine learning models.

[0158] Fig.19 1900 is a flowchart of a method for adaptively adjusting the brightness of multiple light sources of a position tracking system of an HMD system according to an example embodiment of the present disclosure. The method 1900 may be performed by, for example, Figure 5 The method 1900 is performed by the position tracking system or module 512 of the HMD system 500 shown. As discussed above, the method 1900 can be implemented during the tracking of the position of any component, such as an HMD device wearable on the user's head, one or more handheld controllers, etc., for example, in conjunction with Fig.18 Method 1800. Advantageously, the adaptive brightness features discussed herein can provide improved performance by utilizing relatively more of the dynamic range of the detector, and can also increase battery life by reducing power consumption.

[0159] Method 1900 begins at 1902, where control circuitry of an HMD system receives optical detector data from an optical detector (e.g., a photodiode, an angle-sensitive detector). At 1904, the control circuitry may process the received optical detector data, and at 1906, the control circuitry may adaptively adjust a brightness of at least one of a plurality of light sources based at least in part on the processed optical detector data. In at least some implementations, the control circuitry adaptively adjusts a brightness of at least one of the one or more light sources based on a dynamic range of the optical detector, e.g., to maximize the dynamic range of the optical detector. To adjust the brightness, a pulse width of a signal provided to the one or more light sources may be selectively adjusted.

[0160] In at least some implementations, the control circuitry can disable one or more light sources ("dark measurement") and receive optical data from the optical detector while the one or more light sources are disabled. This feature can allow the brightness of one or more light sensors to be adjusted based on the ambient light level in the environment in which the HMD system is operating.

[0161] Fig. 20An example method 2000 of this feature is shown. At 2002, the control circuit may disable one or more (e.g., all) light sources of the HMD system. At 2004, the control circuit may collect sensor data from an optical detector while the one or more light sources are disabled. At 2006, the control circuit may adjust the brightness setting of the one or more light sources based on the collected sensor data. At 2008, the control circuit may optionally adjust the sensor data collection rate for brightness adjustment based on changes in one or more parameters, such as movement of one or more components, elapsed time, number of samples, amount of ambient light, detected changes in ambient light, etc. In order to adaptively adjust the brightness of one or more light sources, the received optical detector data may be provided as input to one or more trained machine learning models, as discussed elsewhere herein. In at least some implementations, the control circuit may periodically perform light measurements (e.g., every 5 samples, every 50 samples), and after each measurement, adjust the brightness of the light source based on the measurement results.

[0162] Fig.21 21 is a flow chart of a method for compensating for non-uniform brightness of a light source of a position tracking system of an HMD system, according to an example embodiment of the present disclosure. In practice, an angle-sensitive detector such as a quadrant photodiode (QPD) may include multiple channels that are read sequentially in time (e.g., via multiplexing). Thus, since measurements from all channels of the detector are not collected simultaneously, any non-uniformity in the brightness of the light source may result in inaccurate measurements. For example, when an LED is illuminated during an illumination cycle, its intensity may vary during the illumination cycle due to thermal and other effects (e.g., "droop"). As discussed below, method 2100 compensates for such effects, which advantageously provides more accurate measurements for position tracking.

[0163] Method 2100 may be performed by, for example Figure 5 The method 2100 is performed by the position tracking system or module 512 of the HMD system 500 shown. As discussed above, the method 2100 can be implemented during the tracking of the position of any component, such as an HMD device wearable on the user's head, one or more handheld controllers, etc., for example, in conjunction with Fig.18 Method 1800.

[0164] The method 2100 begins at 2102, where a control circuit may cause one or more light sources to emit light during an illumination period. At 2104, the control circuit may receive sensor data from an angle-sensitive detector, including sequentially collecting sensor unit samples from a plurality of sensor units during an illumination period, as discussed above. For example, the control circuit may include an analog-to-digital converter (ADC), and a multiplexer may be used to sequentially read samples from each of a plurality of (e.g., four) sensor units of the angle-sensitive detector in the sensor unit. Other methods that can provide an illumination profile of the light source that varies over time may be used to collect illumination data.

[0165] At 2106, the control circuitry may process the received sensor data, including determining a correction to account for non-uniform brightness of the one or more light sources during the period in which the sensor unit samples are collected. At 2108, the control circuitry may use the determined correction to apply calibration data to the sensor unit samples, and use the calibrated sensor unit samples to track the position of the first head mounted display system component. For example, the calibration data may represent a characteristic slope of brightness of the one or more light sources during an illumination period.

[0166] In at least some implementations, the control circuitry can iteratively determine updated calibration data, and can use the updated calibration data to track the position of the first head mounted display system component. For example, to determine the updated calibration data, the control circuitry can disable one or more light sources during a calibration period, sequentially collect sensor unit samples from a plurality of sensor units during the calibration period, interpolate the collected sensor unit samples, and determine updated calibration data based on the interpolation of the collected sensor unit samples.

[0167] The calibration data may additionally or alternatively be determined during a manufacturing or design process of the HMD system. For example, a characteristic slope or function of a light source (e.g., an LED) may be determined empirically using an angle-sensitive detector or other type of optical detector, and such information may be provided to the HMD system to compensate for non-uniform brightness of the light source (or a similar or identical light source) during operation of the HMD system. More generally, a method of calibrating a component of a head-mounted display system may include causing one or more light sources to emit light during an illumination period, sequentially collecting sensor unit samples from a plurality of sensor units of the angle-sensitive detector, processing the received sensor unit samples to generate calibration data for addressing non-uniform brightness of one or more light sources during the illumination period, and storing the calibration data in a non-transitory processor-readable storage medium for subsequent use in tracking at least one component of the head-mounted display system.

[0168] Fig. 222200 is a flow chart of a method for adaptively enabling and disabling components of a tracking subsystem of an HMD system according to an example implementation of the present disclosure. In the case of a light source (e.g., an LED), implementation of method 2200 may be referred to as "adaptive activation" of the LED, which may, for example, reduce power consumption and, thus, extend battery life. As discussed above, method 2200 may be implemented during tracking of the position of any component, such as an HMD device wearable on a user's head, one or more handheld controllers, and the like, for example, in conjunction with Fig.18 Method 1800.

[0169] The method 2200 begins at 2202, where control circuitry causes one or more of a plurality of light sources to emit light. At 2204, the control circuitry receives sensor data from one or more of a plurality of optical detectors and tracks a position of a first head mounted display system component based at least in part on the received sensor data, as discussed elsewhere herein.

[0170] At 2206, the control circuitry may process the received sensor data, including determining whether to disable any of the optical detectors or light sources based on the determined disabling criteria. At 2208, the control circuitry may disable each of the optical detectors or light sources that meets the disabling criteria for a corresponding disabling time period during tracking the position of the first head mounted display system component.

[0171] Typically, the disabling criteria provide or facilitate a determination that light emitted by a light source is unlikely to be received by any optical detector of the first head mounted display system component. For example, the disabling criteria may determine that a particular light source faces away from an optical detector of the HMD system, or is blocked by an object (e.g., a person, component, furniture, wall) in an environment in which the HMD system operates. Similarly, the control circuitry may determine that an optical detector is unlikely to receive light from a light source, and thus may be disabled for a fixed or variable period of time.

[0172] The disabling criteria can be based at least in part on a determined relative position or movement between a first head mounted display system component and a second head mounted display system component, such as a relative position or movement between a controller and a headset, a relative position or movement between a controller and a base station, or a relative position or movement between a headset and a base station. As discussed above, one or more projection models can be used to assess whether light from a light source is directly received by an optical sensor of the HMD system. In at least some embodiments, position tracking information can be used to predict when specific components (e.g., light sources, detectors) can be disabled, and when these components should be re-enabled.

[0173] In at least some implementations, the disabling criteria provide or facilitate determining that light emitted by each of a first subset of a plurality of light sources is likely to be received by at least one of the optical detectors of the first head mounted display system component, and the disabling criteria are operable to disable a second subset of light sources in the first subset of light sources for a corresponding disabling time period. For example, the system may determine that a first subset of four spaced-apart light sources is likely to be detected by one or more detectors, and two light sources of the first subset (i.e., the second subset) may be disabled so that only two light sources are enabled for a period of time. This feature may increase battery life while still providing light sources that are detectable by the optical sensor. In at least some implementations, the light sources in the first subset of light sources that are not disabled may be light sources that are relatively spaced apart from one another, which provides a relatively large separation angle relative to the optical detector that detects light emitted by the light sources, which may improve measurement accuracy.

[0174] Fig.23 2300 is a flow chart of a method of operating a position tracking system of an HMD system to track the position, orientation, and / or movement of a component of the HMD system by fusing inertial sensor data, optical sensor data, and image data during use according to an example embodiment of the present disclosure. As discussed above, the method 2300 may be implemented during tracking the position of any component, such as an HMD device wearable on a user's head, one or more handheld controllers, etc., for example, in conjunction with Fig.18 Method 1800.

[0175] The head mounted display system may include a first head mounted display system component, an inertial measurement unit (IMU) carried by the first head mounted display system component, a plurality of angle sensitive optical detectors (or other types of optical detectors) carried by the first head mounted display system component, and at least one camera carried by the first head mounted display system component. At 2302, control circuitry associated with the HMD system may receive inertial sensor data from the inertial measurement unit. At 2304, the control circuitry may receive optical sensor data from one or more of the plurality of angle sensitive optical detectors or other types of optical detectors. At 2306, the control circuitry may receive image sensor data from the camera. As an example, the camera may be a forward facing camera of an HMD device wearable on a user's head.

[0176] At 2308, the control circuit may process or fuse the received inertial sensor data, optical sensor data, and image sensor data. For example, the control circuit may utilize one or more sensor fusion algorithms, including but not limited to a central limit theorem algorithm, a Kalman filter, a Bayesian network, a DS (Dempster-Shafer) algorithm, or a convolutional neural network. At 2310, the control circuit may track the position of the first head-mounted display system component based at least in part on the processing of the received inertial sensor data, optical sensor data, and image sensor data. In at least some implementations, in order to process the received inertial sensor data, optical sensor data, and image sensor data, the control circuit may provide the inertial sensor data, optical sensor data, and image sensor data as input to one or more trained machine learning models, as discussed elsewhere herein (see, e.g., Figure 6 ).

[0177] Fig.24 is a perspective view 2400 of an example angle-sensitive optical detector 2403 that may be used in one or more implementations of the present disclosure. Fig.25 yes Fig.24 A cross-sectional view 2500 of the angle sensitive optical detector 2403 is shown, Fig.26 2600 is a top view of an angle sensitive optical detector. In this example, the optical detector 2403 includes a quadrant photodetector 2404 disposed on a common substrate 2402. The quadrant photodetector 2404 includes four active regions or cells 2404a to 2404d separated by a small gap. It should be understood that other types of angle sensitive detectors may also be used, such as photodiode detectors with fewer or more cells, position sensitive detectors, etc.

[0178] The angle-sensitive detector 2403 includes an opaque screen or cover 2406 located at a distance 2412 above or in front of the photodetector 2404. The opaque screen 2406 can be connected to the substrate 2402 or integrated with the substrate 2402, for example as part of a housing. The opaque screen 2406 has a hole 2408 therein that allows light 2416 from an object 2414 to pass through. In this example, the object 2414 is the user's eye, including the dark pupil 2415 of the eye, but the embodiments described herein can also be used to track other objects (e.g., controllers, headphones, other objects). The optical detector 2403 also includes an imaging lens 2410 located within or near the hole 2408 of the opaque screen 2406. Advantageously, the imaging lens 2410 is configured to focus an image 2418 of the pupil 2415 of the user's eye 2414 onto the quadrant photodetector. That is, the imaging lens 2410 can be designed to have: an object plane that is substantially coplanar with the expected position of the pupil 2415 of the user's eye 2414 relative to the imaging lens 2410; and an image plane that is substantially coplanar with the quadrant photodetector 2404. The imaging lens 2410 can include a single lens, or can include multiple lenses. In addition, the imaging lens 2410 can include one or more optical components, such as one or more filters, coatings, etc.

[0179] In the non-limiting illustrated example, the active region (e.g., anode) of each element 2404a to 2404d of the photodetector is individually usable so that a light spot illuminating a single quadrant can be electrically characterized as being only in that quadrant. As light translates across the angle-sensitive detector 2403, the energy of the light is distributed between adjacent elements 2404a to 2404d, and the difference in the electrical contribution to each element defines the relative position of the light. The relative intensity distribution across the elements 2404a to 2404d can be used to determine the position of the light applied to the unit.

[0180] As shown, light 2416 reflected from the user's eye 2414 through the lens 2410 forms an image 2418 of the eye 2414, which includes a dark spot surrounded by a brighter area due to the dark pupil 2415. This is because the dark pupil 2415 reflects relatively less light than the portion of the eye 2414 around the pupil and the light reflected by the user's face. Since the dark pupil 2415 will affect the intensity of the light on the unit, the image 2418 (including the dark pupil) formed on the quadrant photodetector 2404 can be electrically characterized to determine the position of the pupil 2415 relative to the angle-sensitive detector 2403. As discussed elsewhere herein, the position information can be used for gaze tracking, and various other technologies can be implemented using the pupil position information. As discussed below, the systems and methods of the present disclosure can utilize multiple light sources and angle-sensitive detectors to determine the position of the user's eyes or the position of other components (e.g., components of the HMD system).

[0181] Fig. 27 2700 is a cross-sectional view of an angle-sensitive optical detector 2700 according to one or more embodiments of the present disclosure. The optical detector 2700 includes a photodetector 2702 located below an optically transparent material 2704, such as a resin, air, etc. The angle-sensitive optical detector 2700 is configured to receive light 2706 from a light source 2708, which is attached to an object associated with an object tracking system, as discussed elsewhere herein. The light source 2708 can be any type of light source, such as an IR LED or other type of light source. The photodetector 2702 can be any photodetector including multiple optically active areas, such as a quadrant photodetector including four optically active areas, an image sensor with a photodiode array, etc.

[0182] In the illustrated implementation, the angle-sensitive optical detector 2700 includes an intensity-varying optical device 2712 and a phase-varying optical device 2714 located above the optically transparent layer 2704. In at least some implementations, the angle-sensitive optical detector 2700 may include only one of the intensity-varying optical device 2712 and the phase-varying optical device 2714. In other implementations, a single component that provides both intensity-varying and phase-varying capabilities may be used. An industrial design surface 2710 is located above the intensity-varying optical device 2712. The specific shape of the industrial design surface 2710 may be determined by the surface shape of an object associated with the angle-sensitive optical detector 2700 (e.g., a headset or controller of an HMD). In at least some implementations, the industrial design surface 2710 may be configured to have specific optical properties, and thus may not be designed solely based on aesthetic criteria or other non-optical criteria. In at least some implementations, the surface 2710 may be configured to have specific optical properties while matching the surrounding industrial design surfaces to provide an optical detector that is invisible or at least less noticeable to the user's eyes.

[0183] The intensity change optic 2712 and the phase change optic 2714 can be any of a variety of different types of materials that change at least one of the intensity or phase of the light 2706 from the light source 2708 received by the photodetector 2702. In addition, the intensity change optic 2712 and the phase change optic 2714 can each include one or more components, or can be integrated into a single component. As a non-limiting example, the intensity change optic 2712 can include a pattern screen printed onto a glass or plastic substrate, wherein the pattern has transparent, translucent, or opaque varying regions. As another example, one or both of the optics 2712 and 2714 can include a photolithographic film. For example, the intensity change optic 2712 can include a central transparent or translucent region that defines a "hole" surrounded by opaque regions. In this way, there is considerable design freedom, which allows a myriad of patterns to be selected for the intensity change optic. Advantageously, with this design freedom, intensity patterns or masks can be produced that are not limited to patterns that can be mechanically manufactured.

[0184] Phase-varying optics 2714 may include a diffractive optical element that may provide any diffraction order. Phase-varying optics 2714 may include a lens array, or any other suitable optic. In at least some implementations, optics 2712 and 2714 may be formed from a single substrate (e.g., glass, plastic) that has a diffraction pattern etched on one side and an intensity-varying pattern printed on the other side.

[0185] The optical devices 2712 and 2714 can be configured to produce a determined intensity distribution at the photodetector. As a non-limiting example, the optical devices can be configured to provide a Gaussian intensity distribution or any other desired intensity distribution. In addition, the optical devices 2712 and 2714 can be configured to modify or improve the angular resolution, linear sensitivity, or other characteristics of the photodetector 2702. For example, the optical devices 2712 and 2714 can be used to provide an angular resolution at the center of the field of view that is substantially the same as the resolution at the edge of the field of view.

[0186] Fig.28 2800 is a cross-sectional view of an angle-sensitive optical detector 2800 according to one or more embodiments of the present disclosure. The optical detector 2800 includes a photodetector 2802 located below an optically transparent material 2804 such as a resin, air, etc. The angle-sensitive optical detector 2800 is configured to receive light 2806 from a light source 2808, which is attached to an object associated with an object tracking system, as discussed elsewhere herein. The light source 2808 can be any type of light source, such as an IR LED or other type of light source. The photodetector 2802 can be any photodetector including multiple optically active areas, such as a quadrant photodetector including four optically active areas, an image sensor with a photodiode array, etc. An industrial design surface 2810 is located above the optically transparent layer 2804.

[0187] In the illustrated implementation, the angle-sensitive optical detector 2800 includes an intensity or phase-changing optic 2812 positioned adjacent to a top surface of the photodetector 2802. The intensity or phase-changing optic 2812 can be configured to change at least one of the intensity or phase of the light 2806. In general, the intensity or phase-changing optic 2812 can be similar or identical to the optics discussed elsewhere herein, e.g., Fig. 27 Optics 2712 and 2714 are shown to be similar or identical.

[0188] Fig.29 is a cross-sectional view of an angle-sensitive optical detector 2900 according to one or more embodiments of the present disclosure. The optical detector 2900 includes a photodetector 2902 located below an optically transparent material 2904, such as a resin, air, or the like. The angle-sensitive optical detector 2900 is configured to receive light 2906 from a light source 2908, which is attached to an object associated with an object tracking system, as discussed elsewhere herein. The light source 2908 can be any type of light source, such as an IR LED or other type of light source. The photodetector 2902 can be any photodetector including multiple optically active areas, such as a quadrant photodetector including four optically active areas, an image sensor with an array of photodiodes, or the like. An industrial design surface (not shown) can be located above the optically transparent layer 2904.

[0189] In the illustrated implementation, the angle-sensitive optical detector 2900 includes an intensity or phase-changing optical device 2910 located within a mechanical aperture of a photodetector 2902. The intensity or phase-changing optical device 2910 can be configured to change at least one of the intensity or phase of light 2906 from a light source 2908 received by the photodetector 2902. In general, the intensity or phase-changing optical device 2912 can be similar or identical to the optical devices discussed elsewhere herein, for example, Fig. 27 Optics 2712 and 2714 are shown to be similar or identical.

[0190] Fig.30 is a cross-sectional view of an angle-sensitive optical detector 3000 according to one or more embodiments of the present disclosure. The optical detector 3000 includes a photodetector 3002 located below an optically transparent material 3004, such as a resin, air, or the like. The angle-sensitive optical detector 3000 is configured to receive light 3006 from a light source 3008, which is attached to an object associated with an object tracking system, as discussed elsewhere herein. The light source 3008 can be any type of light source, such as an IR LED or other type of light source. The photodetector 3002 can be any photodetector including multiple optically active areas, such as a quadrant photodetector including four optically active areas, an image sensor with an array of photodiodes, and the like. An industrial design surface (not shown) can be located above the optically transparent layer 3004, as discussed above with reference to other embodiments.

[0191] In the illustrated implementation, the angle-sensitive optical detector 3000 includes an intensity or phase-varying optical device 3010 that includes a selectively controllable mechanical aperture. The intensity or phase-varying optical device 3010 can be configured to vary at least one of the intensity or phase of light 3006 from a light source 3008 received by a photodetector 3002. In general, the intensity or phase-varying optical device 3010 can be similar or identical to the optical devices discussed elsewhere herein, for example, with Fig. 27 The illustrated optics 2712 and 2714 are similar or identical. The mechanical aperture can be operably coupled to a control circuit 3012 ("controller") that can selectively adjust the size of the adjustable mechanical aperture in operation. In at least some implementations, the controller 3012 can selectively adjust the size or shape of the mechanical aperture based at least in part on feedback from the tracking system. For example, if it is desired for the tracking system to increase the field of view of the optical detector 3000, the controller 3012 can increase the size of the mechanical aperture to provide a larger field of view. Similarly, if it is desired for the tracking system to decrease the field of view of the optical detector 3000, the controller 3012 can decrease the size of the mechanical aperture to provide a smaller field of view.

[0192] Fig.31 is a cross-sectional view of an angle-sensitive optical detector 3100 according to one or more embodiments of the present disclosure. The optical detector 3100 includes a photodetector 3102 located below an optically transparent material 3104, such as a resin, air, or the like. The angle-sensitive optical detector 3100 is configured to receive light 3106 from a light source 3108, which is attached to an object associated with an object tracking system, as discussed elsewhere herein. The light source 3108 can be any type of light source, such as an IR LED or other type of light source. The photodetector 3102 can be any photodetector including multiple optically active areas, such as a quadrant photodetector including four optically active areas, an image sensor with an array of photodiodes, and the like. An industrial design surface (not shown) can be located above the optically transparent layer 3104, as discussed above with reference to other embodiments.

[0193] In the illustrated implementation, the angle-sensitive optical detector 3100 includes an intensity-varying optic 3110 and a phase-varying optic 3112 positioned above the optically transparent material 3104. The intensity-varying optic 3110 can be configured to vary the intensity of light 3106 from a light source 3108 received by the photodetector 3102. Similarly, the phase-varying optic 3112 can be configured to vary the phase of light 3106 from a light source 3108 received by the photodetector 3102. In general, the intensity or phase-varying optic 3112 can be similar or identical to the optics discussed elsewhere herein, for example, with Fig. 27 Optics 2712 and 2714 are shown to be similar or identical.

[0194] In at least some implementations, at least one of the optical devices 3110 and 3112 can be selectively controllable, such as via a controller 3114. As a non-limiting example, at least one of the optical devices 3110 and 3112 can include a liquid crystal layer that can be controlled to provide various patterns of different intensities, phases, polarizations, or other optical properties. For example, the optical device 3110 can be controlled to provide one or more intensity mask patterns, one or more phase mask patterns, one or more polarization mask patterns, or any combination thereof.

[0195] Fig.32is a cross-sectional view of an angle-sensitive optical detector 3200 according to one or more embodiments of the present disclosure. The optical detector 3200 includes a photodetector 3202 positioned toward a top surface of an optically transparent material 3204, such as a resin, air, or the like. The angle-sensitive optical detector 3200 is configured to receive light 3206 from a light source 3208 attached to an object associated with an object tracking system, as discussed elsewhere herein. The light source 3208 can be any type of light source, such as an IR LED or other type of light source. The photodetector 3202 can be any photodetector including a plurality of optically active regions, such as a quadrant photodetector including four optically active regions, an image sensor having an array of photodiodes, or the like. An industrial design surface (not shown) can be located above the optically transparent layer 3204.

[0196] In the illustrated implementation, the angle-sensitive optical detector 3200 includes an intensity or phase-changing optical element 3210 located adjacent to and below the photodetector 3202. The intensity or phase-changing optical element 3210 can be configured to change at least one of the intensity or phase of light 3206 from the light source 3208 received by the photodetector 3202. In general, the intensity or phase-changing optical device 3210 can be similar or identical to the optical devices discussed elsewhere herein, for example, with Fig. 27 The illustrated optics 2712 and 2714 are similar or identical. In the illustrated implementation, a reflecting telescope 3212 is provided for directing light 3206 from a light source 3208 onto a photodetector 3202. Although shown as a flat surface for simplicity, in practice, the reflector 312 may have a curvature that reflects the light 3206 onto the photodetector 3202. Advantageously, in this configuration, a larger field of view may be achieved.

[0197] Fig.33 is a cross-sectional view of an angle-sensitive optical detector 3300 according to one or more embodiments of the present disclosure. The optical detector 3300 includes a photodetector 3302 located below an optically transparent material 3304 such as a resin, air, etc. The angle-sensitive optical detector 3300 is configured to receive light 3306 from a light source 3308, which is attached to an object associated with an object tracking system, as discussed elsewhere herein. The light source 3308 can be any type of light source, such as an IR LED or other type of light source. The photodetector 3302 can be any photodetector including multiple optically active areas, such as a quadrant photodetector including four optically active areas, an image sensor with an array of photodiodes, etc. An industrial design surface (not shown) can be located above the optically transparent layer 3304.

[0198] In the illustrated embodiment, the angle sensitive optical detector 3300 includes an intensity or phase varying optic 3310 positioned within a mechanical aperture of a photodetector 3302. The intensity or phase varying optic 3310 can be configured to vary at least one of the intensity or phase of light 3306 from a light source 3308 that is received by the photodetector 3302. In the illustrated embodiment, the optic 3310 can include polarized catadioptric optics or "flat optics," which can allow for a wider field of view and / or thinner optics. In general, the intensity or phase varying optic 3312 can be similar or identical to the optics discussed elsewhere herein, for example, with Fig. 27 Optics 2712 and 2714 are shown to be similar or identical.

[0199] Fig.34 3400 is a graph illustrating the angular resolution of an angle-sensitive optical detector relative to a field of view, according to a non-limiting illustrative implementation. Specifically, graph 3400 illustrates the angular resolution of a conventional optical detector at a solid line 3402 and the angular resolution of an optical detector of the present disclosure at a dashed line 3404. As indicated by dashed line 3404, the optical detector of the present disclosure may be configured to have substantially the same angular resolution at the center of the field of view (left side of graph 3400) as at the edge of the field of view (right side of graph 3400).

[0200] Fig.35 3500 is a graph showing the output response of an angle sensitive optical detector according to a non-limiting illustrative implementation. Specifically, graph 3500 shows the output of a conventional optical detector at solid line 3502 and the output of an example optical detector of the present disclosure at dashed line 3504. As shown by dashed line 3504, the optical detector of the present disclosure can be configured to output a step function by applying appropriate one or more intensity and / or phase varying optical elements, as discussed above. This feature can advantageously improve the output signal-to-noise ratio of the optical detector.

[0201] Fig.36 3600 is a schematic block diagram of a process of determining one or more optical subsystems of an angle-sensitive optical detector according to a non-limiting illustrative implementation. The optical subsystem to be determined may be any optical subsystem discussed herein, such as any intensity or phase-changing optical device of the present disclosure. In this example, there may be a known position (P XYZ) point 3602. The point can be, for example, the location of a light source coupled to an object to be tracked. The angle-sensitive detector can have a top surface (e.g., a positive curved surface) having a defined front optical function 3604. The top surface can be an arbitrary industrial design surface, and / or can be a surface with specific optical properties. Using an appropriate mathematical solver, the intensity and / or phase-changing optical device 3606 can be determined to provide a desired output 3608 from the optical detector. That is, as discussed above, given the design freedom of the implementation of the present disclosure, for any arbitrary optical function 3604, the intensity or phase-changing optical device 3606 can be mathematically determined to provide a desired output 3608, which can include any desired output for the optical detector, as discussed herein (e.g., uniform angular resolution, higher linearity, larger or smaller field of view, etc.).

[0202] The foregoing detailed description has described various implementations of devices and / or processes using block diagrams, schematic diagrams, and examples. Where such block diagrams, schematic diagrams, and examples contain one or more functions and / or operations, it will be understood by those skilled in the art that each function and / or operation within such block diagrams, flow charts, or examples may be implemented individually and / or collectively by a variety of hardware, software, firmware, or almost any combination thereof. In one implementation, the subject matter may be implemented by an application specific integrated circuit (ASIC). However, it will be appreciated by those skilled in the art that the implementations disclosed herein may be implemented in whole or in part in a standard integrated circuit, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more controllers (e.g., microcontrollers) (as one or more programs running on one or more processors (e.g., microprocessors)), as firmware, or as almost any combination thereof, and in view of the content of the present disclosure, designing circuits and / or writing software and / or firmware code is entirely within the capabilities of those of ordinary skill in the art.

[0203] Those skilled in the art will recognize that many of the methods or algorithms described herein may employ additional steps, may omit steps, and / or may perform steps in an order other than that specified.

[0204] In addition, it will be appreciated by those skilled in the art that the mechanisms taught herein can be distributed as a program product in a variety of forms, and that the illustrative implementations are equally applicable regardless of the particular type of signal-bearing medium actually used to implement the distribution. Examples of signal-bearing media include, but are not limited to, the following: recordable media such as floppy disks, hard drives, CD ROMs, digital tapes, and computer memory.

[0205] The various implementations described above can be combined to provide more implementations. All U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, foreign patent applications, and non-patent publications mentioned in this specification are incorporated herein by reference in their entirety, provided that they do not conflict with the specific teachings and definitions herein. If necessary, various aspects of the implementation can be modified to provide more implementations using the systems, circuits, and concepts of various patents, applications, and publications.

[0206] These and other changes can be made to the implementations in light of the above detailed description. Generally, in the claims that follow, the terms used should not be interpreted as limiting the claims to the specific implementations disclosed in the specification and claims, but should be interpreted to include all possible implementations and the full range of equivalents to which these claims are entitled. Thus, the claims are not limited by this disclosure.

Claims

1. An object tracking system, comprising: Support structure; as well as a plurality of optical detectors carried by the support structure, each of the plurality of optical detectors comprising: a photodetector comprising a plurality of optically active regions configured to receive light emitted by an object of the object tracking system; as well as An optical subsystem is positioned to receive the light prior to receipt of the light by the photodetector, the optical subsystem comprising at least one optical device configured to change at least one of an intensity of the light or a phase of the light.

2. The object tracking system according to claim 1, wherein: The optical subsystem includes an intensity mask configured to generate a predetermined intensity distribution.

3. The object tracking system according to claim 1, wherein: The optical subsystem includes a patterned intensity variation layer including at least two regions having different opacity levels.

4. The object tracking system according to claim 1, wherein: The optical subsystem is configured to modify an angular resolution of the optical detector.

5. The object tracking system according to claim 1, wherein: The optical subsystem is configured to cause the photodetector to generate discrete output values ​​that vary with an incident angle of the light.

6. The object tracking system according to claim 1, wherein: The optical subsystem is configured such that the photodetector has substantially the same angular resolution at a center of a field of view and at an edge of the field of view.

7. The object tracking system according to claim 1, wherein: The optical subsystem includes at least one of a diffractive optical element, a lens array, a pancake optic, or a reflecting telescope mirror.

8. The object tracking system according to claim 1, wherein: The optical subsystem includes a mask that is screen printed or etched onto a material layer.

9. The object tracking system according to claim 1, wherein: The optical subsystem includes an intensity mask including a photoresist sheet.

10. The object tracking system according to claim 1, wherein: The optical subsystem includes a material having an etched pattern on a first side and a printed pattern on a second side, the second side opposite the first side.

11. The object tracking system according to claim 1, wherein: The optical subsystem includes an intensity mask layer and a separate phase mask layer.

12. The object tracking system according to claim 1, wherein: Each of the optical detectors includes an industrial design optical layer having a top surface defined by an industrial design of the support structure, and the optical subsystem is configured to take into account the industrial design optical layer.

13. The object tracking system according to claim 1, wherein: The photodetector includes a quadrant photodetector having four optically active regions.

14. The object tracking system according to claim 1, wherein: The optical subsystem includes a selectively controllable mechanical aperture.

15. The object tracking system according to claim 1, wherein: The optical subsystem includes a liquid crystal layer that is selectively controllable to change at least one of an intensity or a phase of the light.

16. The object tracking system of claim 1, wherein: The optical subsystem is positioned adjacent to or within the physical aperture of the optical detector.

17. The object tracking system of claim 1, wherein: The optical subsystem is positioned adjacent to the plurality of active regions of the optical detector.

18. The object tracking system of claim 1, further comprising: The control circuit is configured as: receiving detector data from each of the plurality of optical detectors; processing the detector data received from the plurality of optical detectors; as well as Based at least in part on the processing of the received detector data, the position of the object is tracked.

19. A head mounted display system comprising: a support structure capable of being positioned on a user's head; a display, carried by the support structure; a plurality of optical detectors carried by the support structure, each of the plurality of optical detectors comprising: a photodetector comprising a plurality of optically active regions configured to receive light emitted by an object of an object tracking system; and an optical subsystem positioned to receive the light prior to receipt of the light by the photodetector, the optical subsystem comprising at least one optical device configured to change at least one of an intensity of the light or a phase of the light; and The control circuit is configured as: receiving detector data from the plurality of optical detectors; processing the detector data received from the plurality of optical detectors; and Based at least in part on the processing of the received detector data, a position of the head mounted display system is tracked.

20. A method for tracking an object, the method comprising: A plurality of optical detectors are provided, each of the plurality of optical detectors comprising: a photodetector comprising a plurality of optically active regions configured to receive light emitted by an object of an object tracking system; and an optical subsystem positioned to receive the light prior to receipt of the light by the photodetector, the optical subsystem comprising at least one optical device configured to change at least one of an intensity of the light or a phase of the light; receiving detector data from the plurality of optical detectors; processing the detector data received from the plurality of optical detectors; and Based at least in part on the processing of the received detector data, the position of the object is tracked.

Images