Position tracking system for head-mounted display systems, including a high-sensitivity angle detector.

Abstract

A system and method for tracking the position of a head-mounted display (HMD) system component, such as a wearable HMD device or handheld controller. The HMD component may include a support structure holding multiple highly sensitive angle detectors capable of detecting the angle of arrival of light emitted from a light source. A processor may cause the light source to emit light according to a specified pattern and receive sensor data from the multiple highly sensitive angle detectors. The processor may process the received sensor data using machine learning or other techniques and track the position of the head-mounted display component based on the processing of the received sensor data.

Description

Technical Field

[0001] The present disclosure generally relates to position tracking for objects such as head-mounted displays and controllers associated with head-mounted displays.

Background Art

[0002] One current generation of virtual reality (“VR”) or augmented reality (“AR”) experiences is formed using a head-mounted display (“HMD”), which can be coupled to a stationary computer (such as a personal computer (“PC”), laptop, or game console), combined and / or integrated with a smartphone and / or an associated display, or can be self-contained. Generally, an HMD is a display device that is worn on a user's head and has a small display device in front of one eye (monocular HMD) or each eye (binocular HMD). This display unit is typically miniaturized and may include, for example, CRT, LCD, liquid crystal on silicon (LCos), or OLED technology. A binocular HMD has the possibility of displaying different images for each eye. This function is used to display stereoscopic images.

[0003] Advances in smartphones, high-definition televisions, and other electronic devices are driving the demand for higher-performance displays. The growing popularity of virtual and augmented reality systems, particularly those using head-mounted displays (HMDs), is further increasing this demand. Virtual reality systems typically completely enclose the wearer's eyes, replacing the wearer's actual or physical field of view (or actual reality) with "virtual" reality, while augmented reality systems typically provide a translucent or transparent overlay of one or more screens in front of the wearer's eyes, thereby extending the actual field of view with additional information, and mediated reality systems may similarly present the viewer with information that combines elements of the real world with virtual elements. In many virtual and augmented reality systems, the wearer's movements in such head-mounted displays may be tracked in various ways, for example, via sensors within the head-mounted display, controllers, or external sensors, to enable the displayed images to reflect the user's movements and to allow for an interactive environment.

[0004] Position tracking allows an HMD system to estimate the position of one or more components relative to each other and to the surrounding environment. Position tracking may utilize a combination of hardware and software to achieve absolute position detection of components in an HMD system. Position tracking is a critical technology for AR or VR systems, enabling tracking of the movement of a 6-degree-of-freedom (6DOF) HMD (and / or controller or other peripherals).

[0005] Position tracking technology may be used to change the user's viewpoint to reflect different actions such as jumping or crouching, and may enable accurate representation of the user's hands and other objects within the virtual environment. Position tracking may also extend the connection between the physical and virtual environments, for example, by using hand position to move virtual objects by touch. Position tracking improves the user's 3D perception of the virtual environment due to parallax and assists in distance perception. Position tracking may also help minimize or reduce motion sickness caused by the disconnect between what the user sees and what the user perceives through their ear's vestibular system.

[0006] There are various methods for position tracking. These methods may include acoustic tracking, inertial tracking, magnetic tracking, optical tracking, and combinations thereof. [Disclosure of the Invention]

[0007] The head-mounted display system may be summarized as comprising a first head-mounted display system component wearable by a user, and a plurality of highly sensitive angle detectors held by the first head-mounted display system component, each of which, during operation, captures sensor data indicating the angle of arrival of light emitted from one or more light sources; at least one non-temporary processor-readable storage medium storing at least one processor-executable instruction or data; and at least one processor operably coupled to the plurality of highly sensitive angle detectors and the at least one non-temporary processor-readable storage medium, the at least one processor receiving sensor data from the plurality of highly sensitive angle detectors, processing the received sensor data, and tracking the position of the first head-mounted display system component, at least in part, based on the processing of the received sensor data. The first head-mounted display system component may include a head-mounted display device or a handheld controller wearable on the user's head. Each of the plurality of highly sensitive angle detectors may include one of a photodiode detector or a highly sensitive position detector. Each of the multiple high-sensitivity angle detectors may include a photodiode detector having at least four cells.

[0008] The head-mounted display system may further include a second head-mounted display system component which includes multiple light sources. The first head-mounted display system component may include one of a head-mounted display device, a controller, or a base station, and the second head-mounted display system component may include another of a head-mounted display device, a controller, or a base station. The second head-mounted display system component may include a component fixed in close proximity to the environment in which the head-mounted display system operates. The multiple light sources may include LED light sources. During operation, the second head-mounted display system component may illuminate a subset of the multiple light sources for a given time, the subset not including all of the multiple light sources of the second head-mounted display system component. During operation, the second head-mounted display system component may sequentially illuminate the multiple light sources. During operation, the second head-mounted display system component may illuminate the multiple light sources using multiplexing. The multiplexing may include at least one of time multiplexing, wavelength multiplexing, frequency multiplexing, or polarization multiplexing. Each light source may include an optical subsystem which includes at least one of a lens, filter, or polarizer. Each high-sensitivity angle detector may include an optical subsystem comprising at least one of a lens, filter, or polarizer. Each high-sensitivity angle detector may include a lens in the high-sensitivity element of the high-sensitivity angle detector that causes the off-axis light to have similar dimensions to the on-axis light. To process the received sensor data, at least one processor may provide the received sensor data as input to a trained machine learning model. The processor may be configured to receive training data and train the machine learning model using the training data.A first head-mounted display system component may include an inertial measuring unit (IMU) sensor operably coupled to at least one processor, and during operation, at least one processor receives IMU sensor data from the IMU sensor, processes the IMU sensor data and sensor data received from a plurality of optically sensitive angle detectors, and tracks the position of the first head-mounted display system component, at least in part, based on the received IMU sensor data and the processing of the received sensor data. The frame rate of the highly sensitive angle detectors may be greater than or equal to 1000 frames per second.

[0009] A method for operating a head-mounted display system, the head-mounted display system comprising a first head-mounted display system component wearable by a user and a plurality of high-sensitivity angle detectors held by the first head-mounted display system component, the method may be summarized as comprising the steps of: capturing sensor data indicating the angle of arrival of light emitted from a light source via each of the plurality of high-sensitivity angle detectors; receiving the sensor data from the plurality of high-sensitivity angle detectors by at least one processor; processing the received sensor data by at least one processor; and tracking the position of the first head-mounted display system component by at least one processor based at least in part on the processing of the received sensor data.

[0010] The head-mounted display system may be summarized as comprising a first head-mounted display system component wearable by a user, a plurality of highly sensitive angle detectors held by the first head-mounted display system component, each of the plurality of highly sensitive angle detectors capturing sensor data indicating the angle of arrival of light emitted from a light source during operation, a second head-mounted display system component including a plurality of highly sensitive angle detectors and a plurality of light sources, at least one non-temporary processor-readable storage medium storing at least one processor-executable instruction or data, and at least one processor operably coupled to the plurality of highly sensitive angle detectors, the plurality of light sources, and the at least one non-temporary processor-readable storage medium during operation, the at least one processor causing the plurality of light sources to emit light, receiving sensor data from the plurality of highly sensitive angle detectors, processing the received sensor data, and tracking the position of at least one of the first head-mounted display system component and the second head-mounted display system component, at least in part based on the processing of the received sensor data. Each of the plurality of highly sensitive angle detectors may include one of a photodiode detector or a highly sensitive position detector. The first head-mounted display system component may include one of a head-mounted display device, a controller, or a base station, and the second head-mounted display system component may include another of a head-mounted display device, a controller, or a base station. Multiple light sources may include LED light sources that emit invisible light. During operation, the second head-mounted display system component may illuminate a subset of the multiple light sources for a given time, the subset not including all of the multiple light sources of the second head-mounted display system component. During operation, the second head-mounted display system component may sequentially illuminate the multiple light sources. During operation, the second head-mounted display system component may illuminate the multiple light sources using multiplexing. Multiplexing may include at least one of time multiplexing, wavelength multiplexing, frequency multiplexing, or polarization multiplexing.Each light source may include an optical subsystem comprising at least one of a lens, filter, or polarizer. Each high-sensitivity angle detector may include an optical subsystem comprising at least one of a lens, filter, or polarizer. [Brief explanation of the drawing]

[0011] In drawings, the same reference numerals identify similar elements or functions. The size and relative position of elements in drawings are not necessarily depicted to scale. For example, the shapes and angles of various elements are not necessarily depicted to scale, and some of these elements may be arbitrarily enlarged and positioned to improve the readability of the drawing. Furthermore, the specific shapes on which these elements are depicted may not necessarily be intended to convey any information about the actual shape of those elements, but may simply be chosen to facilitate recognition in the drawing.

[0012] [Figure 1] This is a schematic diagram of a network environment comprising one or more systems suitable for performing at least some of the techniques described herein, including embodiments of a tracking subsystem.

[0013] [Figure 2] This figure shows an exemplary environment in which at least some of the described technologies are used, having an exemplary head-mounted display device that is tethered to a video rendering computing system and provides a virtual reality display to the user.

[0014] [Figure 3] This is a diagram of an HMD device having a binocular display subsystem and multiple high-sensitivity angle detectors. [Figure 4] This is a diagram of a controller that may be used with an HMD device.

[0015] [Figure 5]Schematic block diagram of an HMD device according to an exemplary embodiment of the present disclosure.

[0016] [Figure 6] Schematic diagram of an environment in which machine learning techniques may be used to implement a tracking subsystem of an HMD device according to one non - limiting illustrated implementation.

[0017] [Figure 7] Flow diagram for a method of operating a position tracking system of an HMD system to track the position, orientation, and / or movement of components of the HMD system in use according to an exemplary embodiment of the present disclosure.

[0018] [Figure 8A] Top view of an exemplary high - sensitivity angle detector that may be used in one or more implementations of the present disclosure. [Figure 8B] Perspective view of the high - sensitivity angle detector shown in FIG. 8A.

[0019] [Figure 9] Simplified diagram showing the use of a light source and a high - sensitivity angle detector to determine the position of components of an HMD system according to one non - limiting illustrated implementation.

[0020] [Figure 10] Diagram depicting an exemplary optical system of a light source and a high - sensitivity angle detector according to one non - limiting illustrated implementation.

DETAILED DESCRIPTION OF THE INVENTION

[0021] In the following description, specific details are set forth in order to provide a thorough understanding of the various disclosed implementations. However, one of ordinary skill in the art will recognize that these implementations may be practiced without one or more of these specific details, or with other methods, components, materials, etc. In other instances, well-known structures associated with computer systems, server computers, and / or communication networks have not been shown or described in detail in order to avoid obscuring the description of these implementations needlessly.

[0022] Unless the context requires otherwise, throughout the following specification and claims, the word "comprise" and variations thereof such as "comprises" and "comprising" are to be interpreted in an inclusive sense, that is, as "including, but not limited to."

[0023] References throughout this specification to "one implementation" or "an implementation" mean that a particular feature, structure, or characteristic described in connection with the implementation is included in at least one implementation. Thus, the appearances of the phrases "in one implementation" or "in an implementation" throughout this specification are not necessarily all referring to the same implementation. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more implementations.

[0024] As used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It should also be noted that the term "or" is generally employed in its sense including "and / or" unless the context clearly dictates otherwise.

[0025] The headings and abstracts provided herein are for convenience only and do not interpret the scope or meaning of the implementations.

[0026] One or more implementations of this disclosure relate to a system and method for accurately tracking the position of components (e.g., the HMD, controllers, and peripherals of a head-mounted display (HMD) system). In at least some implementations, the HMD includes a front camera ("forward camera" or "front camera") and a support structure that holds multiple high-sensitivity angle detectors or light sources. Similarly, one or more controllers may include multiple high-sensitivity angle detectors or photodetectors. In other implementations, the HMD does not include a forward camera. The forward camera may capture image sensor data in the forward camera field of view at a first frame rate (e.g., 30Hz, 90Hz).

[0027] During operation, one or more fixed or movable light sources (e.g., IR LEDs) may be made to emit light, as further described below. The light sources may be coupled to the HMD, controller, fixed objects placed in the environment (e.g., base station), etc. Each of the multiple high-sensitivity angle detectors captures sensor data in each of the multiple high-sensitivity angle detector fields of view at a second frame rate (e.g., 1000Hz, 2000Hz) which may be greater than the first frame rate of the forward camera (if any). In at least some implementations, the high-sensitivity angle detector field of view may be narrower than the forward camera field of view, but this is not required. For example, the forward camera may have a relatively wide forward camera field of view of 90°, 120°, or 150°, and each of the high-sensitivity angle detectors may have a relatively narrow sensor IC field of view (e.g., 25°, 45°, 75°). In at least some implementations, the high-sensitivity angle detector field of view may collectively cover at least a large portion of the forward camera field of view, or a much larger portion, by overlapping each of the high-sensitivity angle detector field of view with a different portion of the forward camera field of view.

[0028] During operation, at least one processor operably coupled with multiple high-sensitivity angle detectors may receive sensor data capturing light from multiple light sources (e.g., LEDs, lasers, other light sources). At least one processor may process the received image sensor data and, at least in part, track the positions of components of the head-mounted display based on the processing of the received image sensor data. For example, at least one processor may combine sensor data from high-sensitivity angle detectors to track one or more features present in the environment. At least one processor may use machine learning techniques, solvers, or other methods of processing sensor data to determine the positions (e.g., position, orientation, motion) of one or more components of the HMD system. In at least some implementations, the sensor data may be combined with sensor data from other sensors, such as sensor data from a forward camera or an inertial measurement unit (IMU) of an HMD system component. Various features of the implementation of this disclosure are described in detail later with reference to the figures.

[0029] Figure 1 is a schematic diagram of a network environment 100 including a local computing system 120, a display device 180 (e.g., an HMD device having two display panels), and a local media rendering (LMR) system 110 (e.g., a gaming system) including one or more controllers 182 suitable for performing at least some of the techniques described herein. In the illustrated embodiment of Figure 1, the local computing system 120 is communicably connected to the display device 180 via a transmission link 115 (which may be wired or tethered via one or more cables (cable 220) as shown in Figure 2, or alternatively wireless). The controllers 182 may be coupled to the local computing system 120 or the display device 180 via suitable wired or wireless links 186 and 184, respectively. In other embodiments, the local computing system 120 may, in addition to or instead of the HMD device 180, provide encoded image data via a wired or wireless link for display on a panel display device (e.g., a television, console, or monitor), each display device comprising one or more addressable pixel arrays. In various embodiments, the local computing system 120 may include a general-purpose computing system, a gaming console, a video stream processing device, a mobile computing device (e.g., a mobile phone, PDA, or other mobile device), a VR or AR processing device, or other computing system.

[0030] In the shown embodiment, the local computing system 120 has components including one or more hardware processors (e.g., a central processing unit, or "CPU") 125, memory 130, various I / O ("input / output") hardware components 127 (e.g., a keyboard, mouse, one or more gaming controllers, speakers, microphones, IR transmitters and / or receivers, etc.), a video subsystem 140 including one or more special-purpose hardware processors (e.g., a graphics processing unit, or "GPU") 144 and video memory (VRAM) 148, computer-readable storage 150, and network connectivity 160. Also in the shown embodiment, embodiments of the tracking subsystem 135 run in memory 130, for example, using the CPU 125 and / or GPU 144 to perform automated operations that implement those described techniques. Memory 130 may optionally further run one or more other programs 133 (e.g., game programs, for generating videos or other images to be displayed). As part of an automated operation to implement at least some of the techniques described herein, a tracking subsystem 135 and / or program 133 running in memory 130 may store or retrieve various types of data in an exemplary database, including data structures in storage 150. In this example, the data used may include various types of image data information in database ("DB") 154, various types of application data in DB 152, various types of configuration data in DB 157, and additional information such as system data or other information.

[0031] The LMR system 110 is also, in the illustrated embodiment, communicatively connected to an exemplary network-accessible media content provider 190 via one or more computer networks 101 and network links 102. This may further provide the LMR system 110 with content for display in addition to, or in lieu of, the image generation program 133. The media content provider 190 may include one or more computing systems (not shown). Each of these may have components including one or more hardware processors, I / O components, local storage devices, and memory, similar to those of the local computing system 120. However, some details of the network-accessible media content provider are not shown for brevity.

[0032] Although the display device 180 is shown as separate and isolated from the local computing system 120 in the embodiment shown in Figure 1, it will be understood that in certain embodiments, some or all components of the local media rendering system 110 may be integrated or housed within a single device such as a mobile gaming device, a portable VR entertainment system, or an HMD device. In such embodiments, the transmission link 115 may include, for example, one or more system bus and / or video bus architectures.

[0033] As an example involving operations performed locally by the local media rendering system 120, the local computing system is a gaming computing system, where application data 152 includes one or more gaming applications executed using memory 130 via the CPU 125, and various video frame display data is generated and / or processed by an image generation program 133, for example, together with the GPU 144 of the video subsystem 140. To provide a high-quality gaming experience, a large amount of video frame data (corresponding to high image resolution, along with a high "frame rate" of approximately 60-180 such video frames per second for each video frame) is generated by the local computing system 120 and provided to the display device 180 via a wired or wireless transmission link 115.

[0034] It will also be understood that the computing system 120 and the display device 180 are merely illustrative and are not intended to limit the scope of this disclosure. The computing system 120 may instead include multiple interacting computing systems or devices, which may be connected to other devices not shown, including being connected through one or more networks such as the Internet, via the Web, or via a private network (e.g., a mobile communications network). More generally, a computing system or other computing node may include any combination of hardware or software, which may be capable of interacting with and performing the types of functions described, and which may include, but are not limited to, desktops or other computers, game systems, database servers, network storage devices, and other network devices, PDAs, mobile phones, wireless phones, pagers, electronic organizers, internet equipment, television-based systems (e.g., using set-top boxes and / or personal / digital video recorders), and various other consumer goods including appropriate communication functions. The display device 180 may similarly include one or more devices having one or more display panels of various types and forms, and may optionally include various other hardware and / or software components.

[0035] Furthermore, the functions provided by the tracking subsystem 135 may, in some embodiments, be distributed across one or more components (e.g., local and remote computing systems, HMDs, controllers, base stations), and in some embodiments, some of the functions of the tracking subsystem 135 may not be provided, and / or other additional functions may be available. Although various items are shown as being stored in memory or on storage during use, it will also be understood that these items or some of them may be transferred between memory and other storage devices for memory management or data integrity purposes. Thus, in some embodiments, some or all of the techniques described may be performed when configured by hardware including one or more processors or other configured hardware circuits or memory or storage, for example by one or more software programs (e.g., by the tracking subsystem 135 or its components), and / or by data structures (e.g., by the execution of software instructions of one or more software programs, and / or by the storage of such software instructions and / or data structures). Some or all of the components, systems, and data structures may also be stored (for example, as software instructions or structured data) on non-temporary computer-readable storage media such as hard disks or flash drives or other non-volatile storage devices, volatile or non-volatile memory (e.g., RAM), network storage devices, or portable media products read via a suitable drive (e.g., DVD discs, CD discs, optical discs, etc.) or a suitable connection.The system, components, and data structures may also be transmitted in some embodiments as generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) over a variety of computer-readable transmission media, including wireless-based and wired / cable-based media, and may take various forms (e.g., as part of a single or multiplexed analog signal, or as multiple distinct digital packets or frames). Such computer program products may also take other forms in other embodiments. Accordingly, the present invention may be implemented by other computer system configurations.

[0036] Figure 2 shows an exemplary environment 200 in which at least some of the techniques described are used with an exemplary HMD device 202 coupled to a video rendering computing system 204 via a tethering connection 220 (or a wireless connection in other embodiments) to provide a virtual reality display to a human user 206. The user wears the HMD device 202 and receives displayed information from the computing system 204 of a simulated environment distinct from the actual physical environment, via the HMD device. Here, the computing system acts as an image rendering system that supplies images of the simulated environment to the HMD device for display to the user. For example, the images are generated by a game program and / or other software program running on the computing system. The user may further have one or more I / O ("input / output") devices that allow the user to move around within a tracked volume 201 of the actual physical environment 200 and further enable the user to interact with the simulated environment. This includes, in this example, handheld controllers 208 and 210.

[0037] In the example shown, the environment 200 may include one or more base stations 214 (two are shown, designated 214a and 214b). This may facilitate tracking of the HMD device 202 or controllers 208 and 210. As the user moves or changes the orientation of the HMD device 202, the position of the HMD device is tracked, and, for example, the corresponding portion of the simulation environment becomes available to the user on the HMD device. Controllers 208 and 210 may further employ similar techniques for use in tracking the controller's position (and optionally for using information to assist in positioning or verifying the HMD device). After the tracked position of the HMD device 202 is known, the corresponding information is transmitted to the computing system 204 via tethering 220 or wirelessly. This uses the tracked position information to generate and display one or more subsequent images of the simulation environment to the user.

[0038] There are numerous different methods of position tracking that may be used in various implementations of this disclosure, including, but are not limited to, acoustic tracking, inertial tracking, magnetic tracking, optical tracking, and combinations thereof.

[0039] In at least some implementations, the HMD device 202 and at least one of the controllers 208 and 210 may include one or more optical receivers or sensors which may be used to implement the tracking functions or other embodiments of the present disclosure. In at least some implementations, the HMD device 202, the controllers 208 and 210, or at least one of the other components may include one or more light sources (e.g., LEDs) which may emit light that is detected by one or more optical receivers. The light sources may be in a fixed position or on a movable component such as the HMD device or controllers.

[0040] In at least some implementations, in addition to generating a fixed point light source, or instead thereof, each base station 214 may sweep an optical signal over the tracked quantity 201. Depending on the requirements of each specific implementation, each base station 214 may generate more than one optical signal. For example, if a single base station 214 is typically sufficient for 6-degree-of-freedom tracking, multiple base stations (e.g., base stations 214a, 214b) may be required or desirable in some embodiments to provide robust room-scale tracking to the HMD device and peripherals. In this example, the optical receiver is integrated into the HMD device 202 and / or other tracked objects such as controllers 208 and 210. In at least some implementations, the optical receiver may be paired with an accelerometer and gyroscope inertial measuring unit ("IMU") on each tracked device to support low-latency sensor fusion.

[0041] In at least some implementations, each base station 214 includes two rotors that sweep a linear beam over a tracking quantity 201 on the vertical axis. At the start of each sweep cycle, the base station 214 may emit an omnidirectional light pulse (referred to as the “synchronization signal”) visible to all sensors on the object being tracked. Each sensor then calculates its intrinsic angular position over the sweep by measuring the period between the synchronization signal and the beam signal. Sensor distances and orientations may be derived using multiple sensors fixed to a single rigid body.

[0042] One or more sensors placed on the object being tracked (e.g., HMD device 202, controllers 208 and 210) may include optoelectronic devices capable of detecting modulated light from the rotor. For visible or near-infrared (NIR) light, a silicon photodiode and a suitable amplifier / detector circuit may be used. Since the environment 200 may contain static and time-varying signals (optical noise) with wavelengths similar to those of the base station 214 signal, in at least some implementations, the base station light may be modulated in a manner that facilitates its distinction from any interfering signals and / or filters the sensor from any emitted wavelengths other than those of the base station signal. Furthermore, as will be described later, in at least some implementations, a highly sensitive angle detector is used to track one or more components of the HMD system.

[0043] Inside-out tracking is also a type of position tracking and may be used to track the position of the HMD device 202 and / or other objects (e.g., controllers 208 and 210, tablet computer, smartphone). Inside-out tracking differs from outside-in tracking in that it is the position of the camera or other sensor used to determine the position of the HMD component. In inside-out tracking, the camera or sensor is positioned on the HMD component or the object being tracked, while in outside-in tracking, the camera or sensor is positioned at a stationary location in the environment.

[0044] HMDs utilizing inside-out tracking use one or more sensors to "look out" and determine how their position changes relative to the environment. If the HMD moves, the sensors readjust their position within the room, and the virtual environment responds accordingly in real time. This type of position tracking can be achieved with or without markers placed in the environment. Cameras placed on the HMD observe the features of the surrounding environment. If markers are used, they are designed to be easily detected by the tracking system and are placed in specific areas. In "markerless" inside-out tracking, the HMD system determines position and orientation using prominent features that are inherently present in the environment (e.g., natural features). The HMD system's algorithms identify specific images or shapes and use them to calculate the device's position in space. Data from accelerometers and gyroscopes may also be used to improve the accuracy of position tracking.

[0045] Figure 3 shows information 300 indicating the forward field of view of an exemplary HMD device 344 when worn on the head of a user 342. The HMD device 344 includes a front structure 343 supporting a front or forward camera 346 and a plurality of high-sensitivity angle detectors 348a-348f (collectively 348) of one or more types. As an example, some or all of the high-sensitivity angle detectors 348, e.g., optical sensors that detect and use light information emitted from one or more external devices (not shown, e.g., base station 214 in Figure 2, controller), may help determine the position and orientation of the device 344 in space. The high-sensitivity angle detectors 348 may be any type of detector capable of operating to detect the angle of arrival of light emitted from a light source. Non-limiting examples of high-sensitivity angle detectors include photodiode detectors (e.g., bicell detectors, quadrandcell detectors), high-sensitivity position detectors using resistive sheets, etc.

[0046] As shown, the front camera 346 and the high-sensitivity angle detector 348 are oriented forward toward the actual scene or environment (not shown) in which the user 342 operates the HMD device 344. More generally, the high-sensitivity angle detector 348 is oriented toward other areas (e.g., upward, downward, left, right, backward) to detect light from various sources such as a controller or objects mounted in various positions (e.g., walls, ceilings). The actual physical environment may include, for example, one or more objects (e.g., walls, ceilings, furniture, stairs, cars, trees, tracking markers, light sources, or any other kind of object). The specific number of sensors 348 may be less than (e.g., 2, 4) or more than (e.g., 10, 20, 30, 40) the number of sensors shown. The HMD device 344 may further include one or more additional components not mounted in the front structure (e.g., inside the HMD device). For example, the IMU (Inertial Measuring Unit) 347 measures and reports specific forces, angular velocities, and / or magnetic fields around the electronic device HMD device 344 (e.g., using a combination of an accelerometer and a gyroscope, and optionally a magnetometer). The HMD device 344 may further include additional components (not shown) which include one or more display panels and optical lens systems having one or more mounted internal motors that are oriented toward the user's eyes (not shown) and optionally change the alignment or other positioning of one or more of the optical lens systems and / or display panels within the HMD device.

[0047] The shown example of the HMD device 344 is supported on the head of a user 342, at least partially based on one or more straps 345 attached to the housing of the HMD device 344, which extend all or partially over the user's head. Although not shown herein, the HMD device 344 may further have one or more external motors attached to one or more of the straps 345, for example. Automated compensatory actions may include adjusting such straps using such motors to correct the alignment or other positioning of the HMD device on the user's head. The HMD device may include, in addition to or in place of the shown straps, other support structures not shown herein (e.g., nosepiece, chin strap, etc.), and it will be understood that some embodiments may include motors attached to one or more such other support structures to adjust their shape and / or position as well, and to correct the alignment or other positioning of the HMD device on the user's head. Other display devices not fixed to the user's head may also be attached to or part of one or more structures that affect the positioning of the display devices, and in at least some embodiments may include motors or other mechanical actuators, thereby similarly correcting their shapes and / or positions and correcting the alignment or other positioning of the display devices with respect to one or more pupils of one or more users of the display devices.

[0048] Figure 4 shows an example of the hand controller 400 in more detail. In practice, the HMD system may include two hand controllers similar to or identical to the hand controller 400 in Figure 4, which may be similar to or identical to the controllers 182, 208, and 210 described above. As shown, the controller 400 has various surfaces on which the high-sensitivity angle detector 402 is positioned. The high-sensitivity angle detector 402 is positioned to receive optical signals from various different directions. The controller 400 may have buttons, sensors, lights, controls, knobs, indicators, displays, etc., enabling various forms of interaction by the user. Furthermore, as described above, in at least some implementations, one of the controller 400 and the HMD device 344 may include multiple light sources, and the other of the controller and the HMD device may include multiple high-sensitivity angle detectors or other types of detectors or sensors. The techniques described herein may be used for various types of position tracking, but are not limited to HMDs, controllers, etc.

[0049] Figure 5 shows a schematic block diagram of an HMD device 500 according to one or more implementations of the present disclosure. The HMD device 500 may be the same as or identical to the HMD devices described elsewhere in this specification. Accordingly, the above-mentioned descriptions with respect to HMD devices may also apply to the HMD device 500. Furthermore, at least some of the components of the HMD device 500 may be present in other components of the HMD system, such as a controller or base station. Accordingly, at least some of the following descriptions may be applicable to such other components.

[0050] The HMD device 500 includes a processor 502, a front or forward camera 504, and a plurality of high-sensitivity angle detectors 506 (e.g., quad-cell photodiodes, high-sensitivity position detectors), and optionally includes an IMU 507 or a plurality of light sources 509. In some implementations, the HMD device 500 may include one of the high-sensitivity angle detectors or light sources, and other components (e.g., controller, base station) may include other high-sensitivity angle detectors or light sources. The HMD device 500 may also include a display subsystem 508 (e.g., two displays and corresponding optical systems). The HMD device 500 may also include non-temporary data storage 510 that can store instructions or data for position tracking 512, instructions or data for display functions 514 (e.g., games), and / or other programs 516. The HMD system 500 may include some or all of the functions of the local computing system 120 or media content provider 190 described above, as shown in Figure 1.

[0051] The HMD device 500 may also include various I / O components 518, which may include one or more user interfaces (e.g., buttons, touchpads, speakers), one or more wired or wireless communication interfaces, etc. As an example, the I / O component 518 may include a communication interface that enables the HMD device 500 to communicate with an external device 520 via a wired or wireless communication link 522. In a non-limiting example, the external device 520 may include a host computer, a server, a mobile device (e.g., a smartphone, a wearable computer), a controller, etc. The various components of the HMD device 500 may be housed in a single housing, in separate housings (e.g., a host computer), or in any combination thereof.

[0052] It will be understood that the computing systems and devices shown are merely illustrative and are not intended to limit the scope of this disclosure. For example, the HMD500 and / or external devices 520 may be connected to other devices not shown, including being connected through one or more networks such as the Internet or via the Web. More generally, such computing systems or devices may include any combination of hardware capable of interacting with and performing the functions of the described type, when programmed or otherwise configured with appropriate software, including but not limited to, desktop computers, laptop computers, slate computers, tablet computers, or other computers, smartphone computing devices and other mobile phones, internet devices, PDAs and other electronic organizers, database servers, network storage devices and other network devices, wireless telephones, pagers, television-based systems (e.g., set-top boxes, and / or personal / digital video recorders, and / or game consoles and / or media servers), and various other consumer goods including appropriate intercommunication capabilities. For example, the illustrative systems 500 and 520 may, in at least some embodiments, include executable software instructions and / or data structures that, when loaded and / or executed by a specific computing system or device, may be used to program or otherwise configure that system or device, for example, to constitute the processor of that system or device. Alternatively, in other embodiments, some or all of the software systems may run in memory on another device and communicate with the illustrated computing system / device via intercomputer communication.Furthermore, although various items are shown to be stored in memory or storage at different times (e.g., while in use), these items or parts thereof may be transferred between memory and storage and / or between storage devices (e.g., in different locations) for memory management and / or data integrity purposes.

[0053] Accordingly, in at least some embodiments, the exemplary system is a software-based system that includes software instructions for programming a processor to automatically perform the described operations for the system when executed by a processor and / or other processor means. Furthermore, in some embodiments, some or all of the system may be implemented or provided in other ways, such as firmware and / or hardware means, at least in part, including, but not limited to, one or more application-specific integrated circuits (ASICs), standard integrated circuits, controllers (e.g., by executing appropriate instructions and including microcontrollers and / or embedded controllers), field-programmable gate arrays (FPGAs), complex-programmable logic devices (CPLDs), etc. Some or all of the system or data structures may also be stored (e.g., as software instruction content or structured data content) on non-temporary computer-readable storage media such as hard disks or flash drives or other non-volatile storage devices, volatile or non-volatile memory (e.g., RAM), network storage devices, or portable media products (e.g., DVD discs, CD discs, optical discs, flash memory devices, etc.) that are read by an appropriate drive or via an appropriate connection. The systems, modules, and data structures may also be transmitted as generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) on various computer-readable transmission media, which may take various forms (e.g., as part of a single or multiplexed analog signal or as part of multiple separate digital packets or frames), including wireless-based and wired / cable-based media in some embodiments. Such computer program products may also take other forms in other embodiments. Accordingly, the present disclosure may be implemented by other computer system configurations.

[0054] Figure 6 is a schematic diagram of an environment 600 in which machine learning techniques may be used to implement a tracking subsystem that tracks an HMD device, one or more controllers, or other components, such as the tracking subsystem described herein, in one non-limiting illustrated implementation. The environment 600 includes a modeled training unit 601 and an inference unit 603. In the training unit 601, training data 602 is supplied to a machine learning algorithm 604 to generate a trained machine learning model 606. The training data may include, for example, labeled data from a highly sensitive angle detector that specifies the position and / or orientation of a specific object for one or more light sources (e.g., LEDs). In a non-limiting example, in an embodiment including components (e.g., HMD, controllers) having 30 highly sensitive angle detectors, each training sample may include output from each or a subset of the highly sensitive angle detectors, known or inferred position or orientation of components, and information about the position or orientation of one or more light sources. As will be described later, each high-sensitivity angle detector may output a single data point (e.g., an angle), or it may output multiple data points, such as two or four signals, each indicating the power or intensity of light received in a specific active element of the high-sensitivity angle detector (e.g., a sub-detector or cell, a resistive sheet, etc.).

[0055] Training data 602 may be obtained from multiple users and / or from a single user of the HMD system. Training data 602 may be obtained in a controlled environment and / or during actual use by a user ("field training"). Furthermore, in at least some implementations, model 606 may be updated or calibrated from time to time (e.g., periodically, continuously, after specific events) to provide accurate position tracking predictions.

[0056] In the inference unit 603, runtime data 608 is provided as input to the trained machine learning model 606 to generate a position tracking prediction 610. Continuing the example above, output data from a high-sensitivity angle detector (e.g., intensity data, angle data) and optionally information about one or more light sources may be provided as input to the trained machine learning model 606, which may process the data to predict the position of the components. The tracking prediction 610 may then be provided to one or more components associated with the HMD device, such as one or more VR or AR applications, one or more display or rendering modules, one or more mechanical controls, one or more additional position tracking subsystems, etc.

[0057] The machine learning techniques used to implement the features described herein may include any kind of preferred structure or technique. As a non-limiting example, machine learning model 606 may include one or more decision trees, statistical hierarchical models, support vector machines, convolutional neural networks (CNNs) or recurrent neural networks (RNNs) (e.g., long short-term memory (LSTM) networks), mixed density networks (MDNs), artificial neural networks (ANNs) such as hidden Markov models, or other models may be available. In at least some implementations, such as those utilizing RNNs, machine learning model 606 may use past input (memory, feedback) information to predict the position of one or more HMD components. Such implementations may, advantageously, use sequential data to determine motion information or previous position predictions that may provide more accurate real-time position predictions.

[0058] Figure 7 is a flowchart of an exemplary method 700 for operating an HMD system and tracking the position of HMD components during use. Method 700 may be performed, for example, by the position tracking system or module 512 of the HMD system 500 shown in Figure 5. As described above, method 700 may be used to track the position of any component, such as the HMD device, one or more controllers, etc.

[0059] An implementation of Method 700, beginning with Operation 702, provides a first HMD system component having a plurality of high-sensitivity angle detectors. The plurality of high-sensitivity angle detectors may be operable to detect light emitted from one or more light sources, which may be fixed in position (e.g., mounted on a wall or ceiling) or movable (e.g., coupled to an HMD or controller). During operation, each of the plurality of high-sensitivity angle detectors captures sensor data at a frame rate in each of the plurality of high-sensitivity angle detector fields of view. The sensor data may include any type of data that a processor can use to detect the presence and direction of a light source relative to the high-sensitivity angle detectors. In at least some implementations, each of the high-sensitivity angle detectors may include one or more sensors (e.g., photodiodes) having image sensing and image processing circuits. The high-sensitivity angle detectors may output relatively raw data (e.g., light intensity or power data) or processed data (e.g., incident angle data).

[0060] In 704, a second HMD system component may be provided which includes multiple light sources (e.g., near-infrared LEDs). The second HMD system component may include, for example, a controller, an HMD device, or light sources placed in a fixed position (e.g., ceiling, wall).

[0061] In 706, at least one processor of the HMD system may cause a light source to emit light. The light source may be illuminated in such a manner that each sensitive angle detector can detect light from a single light source at once, or more generally, in such a manner that the system can determine from which light source the light detected by the sensitive angle detector was received. This may be achieved by multiplexing the illumination of the light source using any preferred type of multiplexing, such as time multiplexing, wavelength multiplexing, frequency multiplexing, polarization multiplexing, or other techniques that enable the system to know the source of the light received from each of the sensitive angle detectors during use.

[0062] As an example of time multiplexing, at least one processor may illuminate only a subset of light sources (e.g., 1, 2, and 4) at a time. For example, at least one processor may sequentially illuminate one light source at a time and collect sensor data in response to each light source.

[0063] As an example of wavelength division multiplexing, different subsets of light sources may emit light of different wavelengths, and different subsets of high-sensitivity angle detectors may be capable of detecting light of different wavelengths. Therefore, light sources with different wavelengths may be simultaneously irradiated and detected by corresponding wavelength-sensitive detectors.

[0064] As an example of frequency multiplexing, a subset of light sources may be illuminated in a determined pattern or frequency, detectable by a highly sensitive angle detector to identify specific light sources.

[0065] As an example of polarization multiplexing, a subset of light sources may be polarized in different ways (e.g., linearly, circularly), and a corresponding subset of high-sensitivity angle detectors may be configured to detect specific polarizations, thereby enabling simultaneous illumination by multiple light sources.

[0066] In 708, at least one processor associated with the HMD system may receive sensor data from a plurality of high-sensitivity angle detectors. As described above, for each high-sensitivity angle detector, the sensor data may indicate the angle of arrival of light emitted from a known light source. In 710, at least one processor associated with the HMD system may optionally receive sensor data from an inertial measuring unit (IMU) capable of operating to provide inertial tracking functionality, or from one or more additional sensors.

[0067] In 712, at least one processor associated with the HMD system may process the received sensor data. For example, at least one processor may combine some or all of the sensor data together to track one or more features present in the environment in which the HMD system operates. The sensor data may include sensor data from multiple high-sensitivity angle detectors and optionally include sensor data from an IMU or a camera. At least one processor may process the sensor data using, for example, a machine learning model (e.g., Modeling 606) or another solver.

[0068] In 714, at least one processor associated with the HMD system may track the position (e.g., position, orientation, or motion) of the components of the HMD system in real time while the user is using the HMD system in the environment. Method 700 may continue to track the position of the components of the HMD system continuously while the HMD is operating, as described above.

[0069] Figures 8A and 8B show a top view and a perspective view, respectively, of an exemplary high-sensitivity angle detector 800 that may be used in one or more implementations of the present disclosure. In this example, the high-sensitivity angle detector 800 includes a quadrand-cell ("quad-cell") photodiode. This includes four separate photodiode active regions or elements 802A-802D, which are separated by small gaps on a common substrate 804. It should be understood that other types of high-sensitivity angle detectors, such as photodiode detectors with fewer or more cells, and high-sensitivity position detectors, may also be used.

[0070] In the non-restrictive illustrated example, the active region (e.g., anode) of each element 802A-802D is available individually so that the light spot illuminating a single quadrant can be electrically characterized as being located only in that quadrant. As the light spot translates across the high-sensitivity angle detector 800, the energy of the light spot is distributed among the adjacent elements 802A-802D, and the difference in the electrical contributions to each element defines the relative position of the light spot with respect to the center of the high-sensitivity angle detector. The relative intensity profile of elements 802A-802D may be used to determine the position of the light spot.

[0071] In this simplified example, the high-sensitivity angle detector 800 includes a cover 810 having an aperture 808 therein, allowing light 814 from a light source 812 to pass through it. As shown, the light 814 passing through the aperture 808 to form a light spot 806 may be electrically characterized to determine the angle of the light 814, and thus the angle of the light source 812 relative to the high-sensitivity angle detector 800. As will be discussed later, the systems and methods of the present disclosure may utilize multiple light sources and high-sensitivity angle detectors to determine the position of components in an HMD system.

[0072] It should be understood that the high-sensitivity angle detectors of this disclosure may include one or more of any suitable types of detectors, including quad-cell photodiode detectors, high-sensitivity position detectors (PSDs) utilizing resistive sheets, photodiode detectors having fewer (e.g., 2) or more (e.g., 16) independent high-sensitivity elements, or any other detectors capable of detecting the angle reached by light emitted from a light source. Furthermore, as will be discussed later, in at least some implementations, the high-sensitivity angle detectors or light sources of this disclosure may utilize various optical components such as filters, lenses, polarizers, etc., to improve the functionality of the systems and methods described herein.

[0073] Figure 9 is a simplified diagram of an environment 900 for an HMD system in which the position of components of the HMD system is determined using light sources and high-sensitivity angle detectors, in one non-restrictive illustrated implementation. In this example, a first component 902, such as an HMD, includes multiple light sources 906 (two are shown, 906a and 906b), and a second component 904, such as a controller for the HMD system, includes multiple high-sensitivity angle detectors 908 (two are shown, 908a and 908b). The high-sensitivity angle detectors 908a and 908b are separated from each other by a known distance d1 on the second component 904, and the light sources 906a and 906b are separated from each other by a known distance d2 on the first component 902. The first and second components may be any components of the HMD system, such as an HMD, controller, base station, stationary or mobile light sources, stationary or mobile high-sensitivity angle detectors, etc.

[0074] In this example, the high-sensitivity angle detector 908a is operable to determine that light arrives from light source 906a at angle 910 and light arrives from light source 906b at angle 912. Similarly, the high-sensitivity angle detector 908b is operable to determine that light arrives from light source 906b at angle 914 and light arrives from light source 906a at angle 916. As can be understood, given the arrival angles 910, 912, 914, and 916 and the known geometric relationships (e.g., distances d1 and d2) between light source 906 and detector 908, a method (e.g., triangulation) may be used to determine the relative position, orientation, or motion between the first component 902 and the second component 904. As described above, one or more solvers or machine learning methods may be used to determine the position of the components using sensor data from the high-sensitivity angle detectors and / or light source data indicating information about the light sources of the HMD system.

[0075] Figure 10 is an example 1000 of an example of a light source 1002 and a high-sensitivity angle detector 1004 of the present disclosure. The light source 1002 and the high-sensitivity angle detector 1004 may be similar to or identical to any of the light sources and high-sensitivity angle detectors described herein and may be used in any of the implementations of the present disclosure. In the illustrated example, the light source 1002 may include an optical subsystem 1006, and the high-sensitivity angle detector 1004 may include an optical subsystem 1008. The optical subsystems 1006 and 1008 may be the same as or different from each other, and each may include one or more optical components. The optical subsystems 1006 and 1008 may be integrated with the light source 1002 and the high-sensitivity angle detector 1004, or they may be separate components. Non-limiting examples of optical components include one or more lenses, one or more polarizers, one or more filters, one or more apertures, and the like. In at least some implementations, a subset of light sources may include one type of optical subsystem, and one or more other subsets of light sources may include another type of optical subsystem. Similarly, a subset of high-sensitivity angle detectors may include one type of optical subsystem, and one or more other subsets of high-sensitivity angle detectors may include another type of optical subsystem. As an example, the optical subsystem may include a filter that filters out visible light or other types of light. Furthermore, as described above, the optical subsystem may include components that facilitate one or more of the various types of multiplexing described above, enabling multiple light sources to illuminate simultaneously without confusion regarding the source of the emitted light.

[0076] The detailed description above illustrates various implementations of devices and / or processes through the use of block diagrams, schematics, and examples. To the extent that such block diagrams, schematics, and examples include one or more functions and / or operations, those skilled in the art will understand that each function and / or operation within the scope of such block diagrams, flowcharts, or examples can be implemented individually and / or collectively by a wide range of hardware, software, firmware, or virtually, or any combination thereof. In one implementation, the subject matter may be implemented via application-specific integrated circuits (ASICs). However, a person skilled in the art will recognize that the implementations disclosed herein can be equivalently implemented in whole or in part on a standard integrated circuit as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more controllers (e.g., microcontrollers), as one or more programs running on one or more processors (e.g., microprocessors), as firmware, or virtually any combination thereof, provided that the design of the circuit and / or the writing of the software or firmware code is well within the skill of a person skilled in the art in light of this disclosure.

[0077] Those skilled in the art will recognize that many of the methods or algorithm sets described herein may employ additional operations, omit some operations, and / or perform operations in an order different from that specified.

[0078] Furthermore, those skilled in the art will understand that the mechanisms taught herein can be distributed as program products in various forms, and that exemplary implementations apply equally regardless of the specific type of signal-carrying medium used to actually carry out the distribution. Examples of signal-carrying media include, but are not limited to, recordable media such as floppy disks, hard disk drives, CD-ROMs, digital tapes, and computer memory.

[0079] The various implementations described above may be combined to provide further implementations. To the extent that they do not conflict with the specific teachings and definitions herein, all U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, foreign patent applications, and non-patent literature referred to herein, including U.S. Patent Application No. 16 / 752,478 filed on January 24, 2020, are incorporated herein by reference. Where necessary, the embodiments of the implementations may be modified to adopt systems, circuits, and concepts from various patents, applications, and publications to provide yet another implementation.

[0080] In light of the detailed description above, these and other modifications may be made to the implementation. In general, the terms used in the following claims should not be interpreted as limiting the claims to any specific implementation disclosed in the specification and claims, but rather as encompassing all possible implementations along the entire scope of equivalents to which rights are granted. Accordingly, the claims are not limited by this disclosure.

Claims

1. A first head-mounted display system component that can be worn by the user, A plurality of high-sensitivity angle detectors mounted at known and separated positions on the first head-mounted display system component, each of the plurality of high-sensitivity angle detectors includes a single cover located in front of four photodiodes on a common substrate, the cover having an opening that allows light to pass through, each photodiode operating to generate an electrical signal indicating the intensity value of the light shining on the photodiode through the opening, the relative values ​​of each of the intensity values ​​being usable to generate sensor data indicating the angle of arrival of the light detected by the high-sensitivity angle detector, each of the plurality of high-sensitivity angle detectors includes a lens in the high-sensitivity element of the high-sensitivity angle detector that causes off-axis light to have similar dimensions to on-axis light, and the plurality of high-sensitivity angle detectors operate at a frame rate of 1000 fps or more, A second head-mounted display system component comprising discrete multiple light-emitting diodes (LEDs), each LED uniquely multiplexed by wavelength or polarization, At least one non-temporary processor-readable storage medium for storing at least one processor-executable instruction or data, The plurality of high-sensitivity angle detectors and the at least one processor operably coupled to the at least one non-temporary processor-readable storage medium and Equipped with, During operation, the at least one processor, By causing one or more of the aforementioned LEDs to light up, The sensor data indicating the angle of arrival of light emitted from one or more of the aforementioned LEDs is received from the aforementioned multiple high-sensitivity angle detectors. Processing the received sensor data, and the processing includes decoding the LED identity from the multiplexed signal and triangulating the pose of the first head-mounted display system component from multiple reach angles measured directly from the LED without using an image of the environmental surface, wherein the angle measurement is generated from the intensity ratio between the photodiodes. A head-mounted display system that tracks the position of at least one of the first head-mounted display system component or the second head-mounted display system component based at least in part on the processing of the received sensor data.

2. The head-mounted display system according to claim 1, wherein the first head-mounted display system component includes a head-mounted display device or handheld controller that can be worn on the user's head.

3. The head-mounted display system according to claim 1, wherein the first head-mounted display system component includes one of a head-mounted display device, a controller, or a base station, and the second head-mounted display system component includes another of a head-mounted display device, a controller, or a base station.

4. The head-mounted display system according to any one of claims 1 to 3, wherein the second head-mounted display system component includes a component fixed in close proximity to the environment in which the head-mounted display system operates.

5. The head-mounted display system according to any one of claims 1 to 4, wherein during operation, the second head-mounted display system component illuminates a subset of the plurality of LEDs for a given time, the subset does not include all of the plurality of LEDs of the second head-mounted display system component.

6. The head-mounted display system according to any one of claims 1 to 5, wherein during operation, the second head-mounted display system component sequentially illuminates the plurality of LEDs.

7. The head-mounted display system according to any one of claims 1 to 6, wherein, in order to process the received sensor data, the at least one processor provides the received sensor data as input to a trained machine learning model.

8. The head-mounted display system according to claim 7, wherein the processor is configured to receive training data and train the machine learning model using the training data.

9. The first head-mounted display system component includes an inertial measuring unit (IMU) sensor operably coupled to the at least one processor, and during operation, the at least one processor, The IMU sensor receives IMU sensor data from the aforementioned IMU sensor. The IMU sensor data and the sensor data received from the plurality of high-sensitivity angle detectors are processed. Based at least in part on the received IMU sensor data and the processing of the received sensor data, the position of at least one of the first head-mounted display system component or the second head-mounted display system component is tracked. A head-mounted display system according to any one of claims 1 to 8.

10. A method for operating a head-mounted display system, the head-mounted display system comprising a first head-mounted display system component wearable by a user, and a plurality of high-sensitivity angle detectors mounted on the first head-mounted display system component at known and isolated positions, each of the plurality of high-sensitivity angle detectors comprising a single cover located in front of four photodiodes on a common substrate, the cover having an opening that allows light to pass through, each photodiode operating to generate an electrical signal indicating an intensity value of light irradiated onto the photodiode through the opening, the relative values ​​of each of the intensity values ​​being usable to generate sensor data indicating the angle of arrival of light emitted from one or more of a plurality of discrete light-emitting diodes (LEDs), each LED being uniquely multiplexed by wavelength or polarization, the method, A step of capturing sensor data indicating the angle of arrival of light emitted from one or more of the multiple LEDs via each of the multiple high-sensitivity angle detectors, wherein each of the multiple high-sensitivity angle detectors includes a lens in its high-sensitivity element that causes the off-axis light to have similar dimensions to the on-axis light, and the multiple high-sensitivity angle detectors operate at a frame rate of 1000 fps or more. The process involves at least one processor receiving the sensor data from the plurality of high-sensitivity angle detectors, The step of processing the received sensor data by at least one processor includes decoding the LED identity from the multiplexed signals and triangulating the pose of the first head-mounted display system component from a plurality of reach angles measured directly from the LEDs without using an image of the environmental surface, wherein the angle measurement is generated from the intensity ratio between the photodiodes, The steps include: tracking the position of the first head-mounted display system component based at least in part on the processing of the received sensor data by the at least one processor; A method for providing this.

11. A first head-mounted display system component that can be worn by the user, A plurality of high-sensitivity angle detectors mounted at known and separated positions on the first head-mounted display system component, each of the plurality of high-sensitivity angle detectors includes a single cover located in front of four photodiodes on a common substrate, the cover having an opening that allows light to pass through, each photodiode operating to generate an electrical signal indicating the intensity value of the light shining on the photodiode through the opening, the relative values ​​of each of the intensity values ​​being usable to generate sensor data indicating the angle of arrival of the light detected by the high-sensitivity angle detector, each of the plurality of high-sensitivity angle detectors includes a lens in the high-sensitivity element of the high-sensitivity angle detector that causes off-axis light to have similar dimensions to on-axis light, and the plurality of high-sensitivity angle detectors operate at a frame rate of 1000 fps or more, A second head-mounted display system component comprising discrete light-emitting diodes (LEDs), each LED uniquely multiplexed by wavelength or polarization, At least one non-temporary processor-readable storage medium for storing at least one processor-executable instruction or data, The plurality of high-sensitivity angle detectors, the plurality of LEDs, and the at least one processor operably coupled to the at least one non-temporary processor-readable storage medium. Equipped with, During operation, the at least one processor, One or more of the aforementioned LEDs are made to emit light. Sensor data is received from the aforementioned multiple high-sensitivity angle detectors. Processing the received sensor data, and the processing includes decoding the LED identity from the multiplexed signal and triangulating the pose of the first head-mounted display system component from multiple reach angles measured directly from the LED without using an image of the environmental surface, wherein the angle measurement is generated from the intensity ratio between the photodiodes. A head-mounted display system that tracks the position of at least one of the first head-mounted display system component or the second head-mounted display system component based at least in part on the processing of the received sensor data.

12. The head-mounted display system according to claim 11, wherein the first head-mounted display system component includes one of a head-mounted display device, a controller, or a base station, and the second head-mounted display system component includes another of a head-mounted display device, a controller, or a base station.

13. The head-mounted display system according to claim 11 or 12, wherein the plurality of LEDs include an LED light source that emits invisible light.

14. The head-mounted display system according to any one of claims 11 to 13, wherein during operation, the second head-mounted display system component illuminates a subset of the plurality of LEDs for a given time, the subset does not include all of the plurality of LEDs of the second head-mounted display system component.

15. The head-mounted display system according to any one of claims 11 to 14, wherein during operation, the second head-mounted display system component sequentially illuminates the plurality of LEDs.

Images