Position tracking system including head-mounted display system with angle-sensitive detector

Abstract

Systems and methods for tracking the position of head-mounted display (HMD) system components, such as wearable HMD devices or handheld controllers. The HMD components may include a support structure carrying multiple angle-sensitive detectors capable of detecting the angle of arrival of light emitted from a light source. A processor causes the light source to emit light according to a specified pattern and receives sensor data from the multiple angle-sensitive detectors. The processor may use machine learning or other techniques to process the received sensor data to track the position of the head-mounted display component based on the processing of the received sensor data.

Description

Technical Field

[0001] This disclosure generally relates to position tracking of objects, such as head-mounted displays and controllers associated with them. Background Technology

[0002] Current-generation virtual reality (“VR”) or augmented reality (“AR”) experiences are created using head-mounted displays (“HMDs”), which can be connected to a stationary computer (such as a personal computer (“PC”), laptop computer, or game console), combined with and / or integrated with a smartphone and / or associated display, or self-contained. Typically, an HMD is a display device worn on a user's head, with a small display device in front of one eye (monocular HMD) or each eye (binocular HMD). The display units are typically miniaturized and can include technologies such as CRT, LCD, liquid crystal on silicon (LCos), or OLED. Binocular HMDs have the potential to display different images to each eye. This capability is used to display stereoscopic images.

[0003] With the development of smartphones, high-definition televisions, and other electronic devices, the demand for displays with enhanced performance has increased. The growing popularity of virtual reality and augmented reality systems, particularly those using head-mounted displays (HMDs), has further amplified this demand. Virtual reality systems typically completely surround the wearer's eyes, replacing the actual or physical view (or real reality) in front of the wearer with a "virtual" reality, while augmented reality systems typically provide a semi-transparent or transparent overlay of one or more screens in front of the wearer's eyes, allowing the actual view to be enhanced with additional information. This mediates how the augmented reality system can similarly combine real-world elements with virtual elements to present information to the viewer. In many virtual reality and augmented reality systems, the wearer's movement can be tracked in various ways, such as through sensors in the head-mounted display, controllers, or external sensors, enabling the display of images to reflect the user's movement and allowing for interaction with the environment.

[0004] Position tracking allows an HMD system to estimate the position of one or more components relative to each other and relative to the surrounding environment. Position tracking can be achieved using a combination of hardware and software to detect the absolute position of components within an HMD system. Position tracking is a crucial technology for AR or VR systems, enabling the tracking of motion of the HMD (and / or controllers or other peripherals) in six degrees of freedom (6DOF).

[0005] Position tracking technology can be used to alter a user's perspective to reflect different actions, such as jumping or crouching, and can allow for accurate representation of a user's hands and other objects in a virtual environment. Position tracking can also increase the connection between the physical and virtual environments, for example, by moving virtual objects through touch using gestures. Position tracking improves the user's 3D perception of the virtual environment, which stems from parallax, aiding in distance perception. Furthermore, position tracking can help minimize motion sickness caused by a disconnect between the input of what the eyes see and what the user's vestibular system senses.

[0006] There are different position tracking methods. These methods can include acoustic tracking, inertial tracking, magnetic tracking, optical tracking, and combinations thereof. Summary of the Invention

[0007] A head-mounted display system can be broadly categorized as including: a first head-mounted display system component that can be worn by a user; a plurality of angle-sensitive detectors carried by the first head-mounted display system component, each of which, in operation, captures sensor data indicating the angle of arrival of light emitted from one or more light sources; at least one non-transitory processor-readable storage medium storing at least one of processor-executable instructions and data; and at least one processor operatively connected to the plurality of angle-sensitive detectors and the at least one non-transitory processor-readable storage medium, wherein, in operation, the at least one processor receives sensor data from the plurality of angle-sensitive detectors; processes the received sensor data; and tracks the position of the first head-mounted display system component at least in part based on the processing of the received sensor data. The first head-mounted display system component may include a head-mounted display device or a handheld controller that can be worn on a user's head. Each of the plurality of angle-sensitive detectors may include one of a photodiode detector or a position-sensitive detector. Each of the plurality of angle-sensitive detectors may include a photodiode detector having at least four units.

[0008] The head-mounted display system may further include a second head-mounted display system component, which includes multiple light sources. The first head-mounted display system component may include one of a head-mounted display device, a controller, and a base station, and the second head-mounted display system component may include others of the head-mounted display device, controller, and base station. The second head-mounted display component may include a component fixed at a location near the operating environment of the head-mounted display system. The multiple light sources may include LED light sources. In operation, the second head-mounted display system component may illuminate a subset of the multiple light sources at a given time, which does not include all of the multiple light sources of the second head-mounted display system component. In operation, the second head-mounted display system component may illuminate the multiple light sources sequentially. In operation, the second head-mounted display system component may use multiplexing to illuminate the multiple light sources. Multiplexing may include at least one of time multiplexing, wavelength multiplexing, frequency multiplexing, and polarization multiplexing. Each light source may include an optical subsystem, which includes at least one of a lens, a filter, and a polarizer. Each angle-sensitive detector may include an optical subsystem, which includes at least one of a lens, a filter, and a polarizer. Each angle-sensitive detector may include a lens that causes off-axis light to have a similar size at the sensitive element of the angle-sensitive detector as on-axis light. To process the received sensor data, at least one processor may provide the received sensor data as input to a trained machine learning model. The processor may be configured to receive training data and use the training data to train the machine learning model. A first head-mounted display system component may include an inertial measurement unit (IMU) sensor operatively connected to at least one processor, wherein, in operation, at least one processor receives IMU sensor data from the IMU sensor; processes the IMU sensor data and sensor data received from a plurality of optical angle-sensitive detectors; and tracks the position of the first head-mounted display system component at least in part based on the processing of the received IMU sensor data and the received sensor data. The frame rate of the angle-sensitive detector may be greater than or equal to 1000 frames per second.

[0009] A method of operating a head-mounted display system, the head-mounted display system including a first head-mounted display system component wearable by a user and a plurality of angle-sensitive detectors carried by the first head-mounted display system component. The method can be summarized as including: capturing sensor data indicating the angle of arrival of light emitted from a light source via each of the plurality of angle-sensitive detectors; receiving the sensor data from the plurality of angle-sensitive detectors by at least one processor; processing the received sensor data by at least one processor; and tracking the position of the first head-mounted display system component by at least one processor based at least in part on the processing of the received sensor data.

[0010] A head-mounted display system can be broadly categorized as including: a first head-mounted display system component that can be worn by a user; a plurality of angle-sensitive detectors carried by the first head-mounted display system component, each of which, in operation, captures sensor data indicating the angle of arrival of light emitted from a light source; a second head-mounted display system component that includes a plurality of light sources; at least one non-transitory processor-readable storage medium storing at least one of processor-executable instructions and data; and at least one processor operatively connected to the plurality of angle-sensitive detectors, the plurality of light sources, and the at least one non-transitory processor-readable storage medium, wherein, in operation, the at least one processor causes the plurality of light sources to emit light; receiving sensor data from the plurality of angle-sensitive detectors; processing the received sensor data; and tracking the position of at least one of the first head-mounted display system component and the second head-mounted display system component, at least in part based on the processing of the received sensor data. Each of the plurality of angle-sensitive detectors may include one of a photodiode detector and a position-sensitive detector. The first head-mounted display system component may include one of a head-mounted display device, a controller, and a base station, and the second head-mounted display system component may include the other of a head-mounted display device, a controller, and a base station. Multiple light sources may include LED light sources that emit invisible light. In operation, the second head-mounted display system component may illuminate a subset of the multiple light sources at a given time, which does not include all of the multiple light sources of the second head-mounted display system component. In operation, the second head-mounted display system component may illuminate the multiple light sources sequentially. In operation, the second head-mounted display system component may use multiplexing to illuminate the multiple light sources. Multiplexing may include at least one of time multiplexing, wavelength multiplexing, frequency multiplexing, and polarization multiplexing. Each light source may include an optical subsystem that includes at least one of a lens, a filter, and a polarizer. Each angle-sensitive detector may include an optical subsystem that includes at least one of a lens, a filter, and a polarizer. Attached Figure Description

[0011] In the accompanying drawings, the same reference numerals denote similar elements or actions. The dimensions and relative positions of the elements in the drawings need not be drawn to scale. For example, the shapes and angles of various elements need not be drawn to scale, and some of these elements may be arbitrarily enlarged and positioned to improve the readability of the drawing. Furthermore, the specific shapes of the elements shown in the figures are not necessarily intended to convey any information about the actual shape of the particular element and may be chosen solely for ease of identification in the accompanying drawings.

[0012] Figure 1 It is a schematic diagram of a networked environment including one or more systems adapted to perform at least some of the techniques described in this disclosure, including implementations of a tracking subsystem.

[0013] Figure 2 The diagram illustrates an example environment in which at least some of the described technologies are used with an example head-mounted display device connected to a video rendering computing system and providing a virtual reality display to a user.

[0014] Figure 3 This is a schematic diagram of an HMD device with a binocular display subsystem and multiple angle-sensitive detectors.

[0015] Figure 4 This is a schematic diagram of a controller that can be used with HMD devices.

[0016] Figure 5 This is a schematic block diagram of an HMD device according to an exemplary embodiment of the present disclosure.

[0017] Figure 6 This is a schematic diagram of an environment according to a non-limiting embodiment, in which machine learning techniques can be used to implement a tracking subsystem for an HMD device.

[0018] Figure 7 This is a flowchart of a method for operating a position tracking system of an HMD system according to an exemplary embodiment of the present disclosure to track the position, orientation, and / or movement of components of the HMD system during use.

[0019] Figure 8A This is a top view of an exemplary angle-sensitive detector that can be used in one or more embodiments of this disclosure.

[0020] Figure 8B yes Figure 8A The image shows a perspective view of the angle-sensitive detector.

[0021] Figure 9 This is a simplified diagram illustrating the use of a light source and an angle-sensitive detector to determine the position of components in an HMD system according to a non-limiting embodiment.

[0022] Figure 10 This is a diagram illustrating an example optical system of a light source and an angle-sensitive detector according to a non-limiting embodiment shown. Detailed Implementation

[0023] In the following description, certain specific details are set forth to provide a thorough understanding of the disclosed embodiments. However, those skilled in the art will recognize that embodiments can be practiced without using one or more of these specific details, or using other methods, components, materials, etc. In other instances, well-known structures associated with computer systems, server computers, and / or communication networks have not been shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0024] Unless the context otherwise requires, throughout the specification and the appended claims, the word “comprising” is synonymous with “including” and is inclusive or open-ended (i.e., does not exclude other unlisted elements or method actions).

[0025] Throughout this specification, references to "one embodiment" or "implementation" mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment. Therefore, the phrases "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, a particular feature, structure, or characteristic may be combined in one or more embodiments in any suitable manner.

[0026] Unless the context clearly specifies otherwise, the singular forms “a,” “an,” and “the” used in this specification and the appended claims include plural indicators. It should also be noted that, unless the context clearly specifies otherwise, the term “or” is generally used in the sense of including “and / or.”

[0027] The titles and abstracts provided herein are for convenience only and do not constitute an explanation of the scope or meaning of implementation.

[0028] One or more embodiments of this disclosure relate to systems and methods for accurately tracking the position of components (e.g., HMD, controller, peripheral devices) of a head-mounted display (HMD) system. In at least some embodiments, the HMD includes a support structure that carries a forward-facing camera (“front-facing camera”) and multiple angle-sensitive detectors or light sources. Similarly, one or more controllers may include multiple angle-sensitive detectors or light detectors. In other embodiments, the HMD does not include a front-facing camera. The front-facing camera may capture image sensor data in its field of view at a first frame rate (e.g., 30Hz, 90Hz).

[0029] In operation, as discussed further below, one or more fixed or movable light sources (e.g., IR LEDs) may emit light. The light source may be connected to an HMD, controller, a fixed object in the environment (e.g., a base station), etc. Each of the plurality of angle-sensitive detectors captures sensor data from its corresponding field of view at a second frame rate (e.g., 1000Hz, 2000Hz), which may be greater than the first frame rate of the front-facing camera (if present). In at least some embodiments, although not required, the field of view of the angle-sensitive detector may be narrower than that of the front-facing camera. For example, the front-facing camera may have a relatively wide field of view of 90°, 120°, or 150°, and each angle-sensitive detector may have a relatively narrow sensor IC field of view (e.g., 25°, 45°, 75°). In at least some embodiments, the field of view of the angle-sensitive detector may generally cover at least a large portion of the front-facing camera's field of view, or even be larger than the front-facing camera's field of view, wherein each angle-sensitive detector's field of view overlaps with different portions of the front-facing camera's field of view.

[0030] In operation, at least one processor operatively connected to multiple angle-sensitive detectors can receive sensor data capturing light from multiple light sources (e.g., LEDs, lasers, other light sources). The at least one processor can process the received image sensor data and track the position of the head-mounted display components based at least in part on the processing of the received image sensor data. For example, the at least one processor can fuse sensor data from the angle-sensitive detectors to track one or more features present in the environment. The at least one processor can utilize machine learning techniques, solvers, or other methods to process the sensor data to determine the position (e.g., location, orientation, movement) of one or more components of the HMD system. In at least some embodiments, this sensor data can be fused with sensor data from other sensors, such as sensor data from the front-facing camera or inertial measurement unit (IMU) of the HMD system components. Various features of embodiments of this disclosure are discussed in detail below with reference to the accompanying drawings.

[0031] Figure 1 This is a schematic diagram of a networked environment 100 including a Local Media Rendering (LMR) system 110 (e.g., a gaming system), the LMR system 110 including a local computing system 120, a display device 180 (e.g., an HMD device with two display panels), and one or more controllers 182 adapted to perform at least some of the techniques described herein. Figure 1 In the illustrated embodiment, the local computing system 120 is connected via a transmission link 115 (which may be wired or tethered, for example via...) Figure 2One or more cables (cable 220) shown, or wirelessly, are communicatively connected to display device 180. Controller 182 may be connected to local computing system 120 or display device 180 via suitable wired or wireless links 186 and 184, respectively. In other embodiments, whether in conjunction with or replacing HMD device 180, local computing system 120 may provide encoded image data for display via wired or wireless links to panel display devices (e.g., TVs, consoles, or monitors), each display device including one or more addressable pixel arrays. In various embodiments, local computing system 120 may include general-purpose computing systems, game consoles, video streaming devices, mobile computing devices (e.g., cellular phones, PDAs, or other mobile devices), VR or AR processing devices, or other computing systems.

[0032] In the illustrated embodiment, the local computing system 120 includes the following components: one or more hardware processors (e.g., a centralized processing unit or "CPU") 125, memory 130, various I / O ("input / output") hardware components 127 (e.g., keyboard, mouse, one or more game controllers, speakers, microphones, IR transmitters and / or receivers, etc.), a video subsystem 140 including one or more dedicated hardware processors (e.g., a graphics processing unit or "GPU") and video memory (VRAM) 148, computer-readable storage 150, and network connectivity 160. Additionally, in the illustrated embodiment, an implementation of the tracking subsystem 135 is executed in memory 130 to perform at least some of the described techniques, such as using the CPU 125 and / or GPU 144 to perform automated operations that implement those described techniques, and memory 130 may optionally further execute one or more other programs 133 (e.g., generating videos or other images to be displayed, such as game programs). As part of automated operation for implementing at least some of the techniques described herein, the tracking subsystem 135 and / or program 133 executing in memory 130 may store or retrieve various types of data included in an example database data structure in memory 150, in which the data used may include various types of image data information in database (“DB”) 154, various types of application data in DB 152, various types of configuration data in DB 157, and additional information (such as system data or other information).

[0033] In the described implementation, the LMR system 110 is also communicatively connected to an exemplary network-accessible media content provider 190 via one or more computer networks 101 and network links 102. Whether in conjunction with or in place of the image generation program 133, the network-accessible media content provider 190 can further provide content to the LMR system 110 for display. The media content provider 190 may include one or more computing systems (not shown), each of which may have components similar to those of the local computing system 120, including one or more hardware processors, I / O components, local storage devices, and memory; however, for the sake of brevity, some details of the network-accessible media content provider are not shown.

[0034] It should be understood that, despite Figure 1 In the illustrated embodiment, the display device 180 is described as separate from and independent of the local computing system 120; however, in some embodiments, some or all components of the local media presentation system 110 may be integrated or housed within a single device, such as a mobile gaming device, a portable VR entertainment system, an HMD device, etc. In such embodiments, the transmission link 115 may, for example, include one or more system buses and / or video bus structures.

[0035] As an example involving operations performed locally by the local media presentation system 120, assuming the local computing system is a game computing system, application data 152 includes one or more game applications executed by the CPU 125 using memory 130, and various video frame display data generated and / or processed by the image generation program 133, such as the GPU 144 combined with the video subsystem 140. To provide a high-quality gaming experience, a large amount of video frame data (corresponding to a high image resolution per video frame and a high "frame rate" of approximately 60-180 such video frames per second) is generated by the local computing system 120 and provided to the display device 180 via a wired or wireless transmission link 115.

[0036] It will also be understood that computing system 120 and display device 180 are illustrative only and are not intended to limit the scope of this disclosure. Computing system 120 may alternatively include multiple interactive computing systems or devices and may be connected to other devices not shown via one or more networks such as the Internet, via the Web, or via a dedicated network (e.g., a mobile communication network, etc.). More generally, computing systems or other computing nodes may include any combination of hardware or software capable of interacting and performing functions of the types described, including but not limited to desktop or other computers, gaming systems, database servers, network storage devices and other network devices, PDAs, cellular phones, wireless phones, pagers, electronic notebooks, Internet devices, television-based systems (e.g., using set-top boxes and / or personal / digital video recorders), and various other consumer products including suitable communication capabilities. Display device 180 may similarly include one or more devices having one or more display panels of various types and forms, and optionally include various other hardware and / or software components.

[0037] Furthermore, in some embodiments, the functionality provided by the tracking subsystem 135 may be distributed across one or more components (e.g., local and remote computing systems, HMDs, controllers, base stations), and in some embodiments, some functions of the tracking subsystem 135 may not be provided and / or additional functionalities may be available. It should also be understood that although various entries are shown as stored in memory or on storage devices during use, these entries, or portions thereof, may be transferred between memory and other storage devices for memory management or data integrity purposes. Therefore, in some embodiments, such as when configured via one or more software programs (e.g., via the tracking subsystem 135 or its components) and / or data structures (e.g., by executing software instructions of one or more software programs and / or by storing such software instructions and / or data structures), some or all of the described techniques may be executed by hardware including one or more processors or other configured hardware circuitry or memory. Some or all of the components, systems, and data structures may also be stored (e.g., as software instructions or structured data) on non-transitory computer-readable storage media such as hard disks or flash drives or other non-volatile storage devices, volatile or non-volatile memory (e.g., RAM), network storage devices, or portable media articles (DVDs, CDs, optical discs, etc.) to be read by a suitable drive or by a suitable connection. In some embodiments, the systems, components, and data structures may also be transmitted as generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) in various forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames) on various computer-readable transmission media, including wireless-based and wired / cable-based media. In other embodiments, such computer program products may also take other forms. Therefore, this disclosure can be implemented with other computer system configurations.

[0038] Figure 2An exemplary environment 200 is illustrated, in which at least some of the described technologies are used with an exemplary HMD device 202, which is connected via a tethered connection 220 (or a wireless connection in other embodiments) to a video rendering computing system 204 to provide a virtual reality display to a human user 206. The user wears the HMD device 202 and receives display information of a simulated environment different from the actual physical environment from the computing system 204, where the computing system acts as an image rendering system, providing images of the simulated environment (e.g., images generated by a game program and / or other software program running on the computing system) to the HMD device for display to the user. In this example, the user is also able to move within a tracking volume 201 of the actual physical environment 200 and may also have one or more I / O (“input / output”) devices to allow further interaction with the simulated environment; in this example, one or more I / Os include handheld controllers 208 and 210.

[0039] In the illustrated example, environment 200 may include one or more base stations 214 (two are shown, labeled base stations 214a and 214b) that can help track HMD device 202 or controllers 208 and 210. The position of HMD device 202 is tracked when the user moves its location or changes its orientation, allowing the corresponding portion of the simulated environment to be displayed to the user on the HMD device. Controllers 208 and 210 may also use similar techniques to track their position (and optionally use this information to help determine or verify the position of the HMD device). After the tracked position of HMD device 202 is known, the corresponding information is transmitted via tether 220 or wirelessly to computing system 204, which uses the tracked position information to generate one or more subsequent images of the simulated environment for display to the user.

[0040] Various position tracking methods can be used in various embodiments of this disclosure, including but not limited to acoustic tracking, inertial tracking, magnetic tracking, optical tracking, and combinations thereof.

[0041] In at least some embodiments, at least one of the HMD device 202 and controllers 208 and 210 may include one or more optical receivers or sensors that can be used to implement the tracking functions or other aspects of this disclosure. In at least some embodiments, at least one of the HMD device 202, controllers 208 and 210, or other components may include one or more light sources (e.g., LEDs) that can emit light detected by one or more optical receivers. The light source may be in a fixed position or may be located on a movable component such as the HMD device or controller.

[0042] In at least some embodiments, instead of generating a fixed-point light source, each base station 214 may scan a light signal across the tracking volume 201. Depending on the requirements of each particular embodiment, each base station 214 may generate more than one light signal. For example, while a single base station 214 is generally sufficient for six-degree-of-freedom tracking, in some embodiments, multiple base stations (e.g., base stations 214a, 214b) may be required or desired to provide robust room-scale tracking for the HMD device and peripherals. In this example, an optical receiver is incorporated into the HMD device 202 and / or other tracked objects, such as controllers 208 and 210. In at least some embodiments, the optical receiver on each tracked device may be paired with an accelerometer and gyroscope inertial measurement unit (“IMU”) to support low-latency sensor fusion.

[0043] In at least some embodiments, each base station 214 includes two rotors that scan a linear beam across the tracking volume 201 on orthogonal axes. At the beginning of each scanning cycle, base station 214 may emit an omnidirectional light pulse (referred to as a "synchronization signal") that is visible to all sensors on the tracked object. Therefore, each sensor calculates a unique angular position within the scanning volume by timing the duration between the synchronization signal and the beam signal. Multiple sensors fixed to a single rigid body can be used to determine sensor distance and orientation.

[0044] One or more sensors located on the tracked object (e.g., HMD device 202, controllers 208 and 210) may include optoelectronic devices capable of detecting modulated light from the rotor. For visible or near-infrared (NIR) light, silicon photodiodes and suitable amplifier / detector circuitry may be used. Because the environment 200 may contain static and time-varying signals (optical noise) with wavelengths similar to the signal from base station 214, in at least some embodiments, the base station light may be modulated in a manner that makes it easily distinguishable from any interfering signals and / or filters the sensor from radiation of any wavelength other than the base station signal. As discussed further below, in at least some embodiments, angle-sensitive detectors are used to track one or more components of the HMD system.

[0045] Inside-out tracking is also a type of location tracking that can be used to track the position of HMD device 202 and / or other objects (e.g., controllers 208 and 210, tablet computers, smartphones). The difference between inside-out tracking and outside-out tracking lies in the position of the camera or other sensors used to determine the position of the HMD component. In inside-out tracking, the camera or sensor is located on the HMD component or the object being tracked, while in outside-out tracking, the camera or sensor is placed in a fixed position within the environment.

[0046] HMDs utilizing inside-out tracking employ one or more sensors to "look out" to determine how their position changes relative to the environment. As the HMD moves, the sensors readjust their positions within the room, and the virtual environment responds accordingly in real time. This type of position tracking can be achieved with or without markers placed in the environment. Cameras placed on the HMD observe features of the surrounding environment. When markers are used, they are designed to be easily detected by the tracking system and placed in specific areas. Regarding "markerless" inside-out tracking, the HMD system uses unique features inherent in the environment (e.g., natural features) to determine position and orientation. The HMD system's algorithms identify specific images or shapes and use them to calculate the device's position in space. Data from accelerometers and gyroscopes can also be used to increase the accuracy of position tracking.

[0047] Figure 3 Information 300 shows a front view of an exemplary HMD device 344 when worn on the head of user 342. The HMD device 344 includes a front-facing structure 343 supporting a forward-facing or front-mounted camera 346 and multiple angle-sensitive detectors 348a-348f (collectively referred to as 348) of one or more types. As an example, some or all of the angle-sensitive detectors 348 may help determine the position and orientation of the device 344 in space, such as light sensors, which detect and utilize signals from one or more external devices (not shown, e.g., [unclear]). Figure 2 The base station 214 (controller) transmits light information. The angle-sensitive detector 348 can be any type of detector used to detect the angle of arrival of light emitted from the light source. Non-limiting examples of angle-sensitive detectors include photodiode detectors (e.g., dual-cell detectors, quadrant cell detectors), position-sensitive detectors using resistive elements, etc.

[0048] As shown, the front-facing camera 346 and angle-sensitive detector 348 are pointed forward at the actual scene or environment (not shown) in which the user 342 operates the HMD device 344. More generally, the angle-sensitive detector 348 can be pointed at other areas (e.g., up, down, left, right, back) to detect light from various sources such as controllers or objects mounted in various locations (e.g., walls, ceilings). The actual physical environment can include, for example, one or more objects (e.g., walls, ceilings, furniture, stairs, cars, trees, tracking markers, light sources, or any other type of object). The specific number of sensors 348 can be fewer (e.g., 2, 4) or more (e.g., 10, 20, 30, 40) than the number of sensors shown. HMD device 344 may further (e.g., within the HMD device) include one or more additional components not attached to the front-facing structure, such as IMU (Inertial Measurement Unit) 347 electronics, which measures and reports specific forces, angular rates, and / or magnetic fields around the HMD device 344 (e.g., using a combination of accelerometers and gyroscopes (optionally, magnetometers)). HMD device 344 may also include additional components (not shown) including one or more display panels and optical lens systems facing the user's eyes (not shown) and optionally having one or more attached internal motors to change the alignment or other positioning of the one or more optical lens systems and / or display panels within the HMD device.

[0049] The illustrated HMD device 344 is at least partially supported on the head of the user 342 by one or more straps 345 attached to the housing of the HMD device 344 and extending wholly or partially around the user's head. Although not shown here, the HMD device 344 may also have one or more external motors, for example, attached to one or more straps 345, and automatic correction actions may include using such motors to adjust such straps to modify the alignment or other positioning of the HMD device on the user's head. It should be understood that, whether as a supplement to or alternative to the straps shown, the HMD device may include other support structures (e.g., nose clips, chin rests, etc.) not shown here, and some embodiments may include motors attached to one or more of these other support structures to similarly adjust their shape and / or position to modify the alignment or other positioning of the HMD device on the user's head. Other display devices not fixed to the user's head may be similarly attached to one or more structures or parts thereof that affect the positioning of the display device, and in at least some embodiments may include motors or other mechanical actuators to similarly modify their shape and / or position, thereby modifying the alignment or other positioning of the display device relative to one or more pupils of one or more users of the display device.

[0050] Figure 4 An example of the handheld controller 400 is shown in more detail. In fact, an HMD system can include... Figure 4 Two handheld controllers, similar to or identical to those described above (controllers 182, 208, and 210), are provided for the handheld controller 400. As shown, controller 400 has various surfaces on which angle-sensitive detectors 402 are located. The angle-sensitive detector 402 is configured to receive light signals from various directions. Controller 400 may include buttons, sensors, lights, controllers, knobs, indicators, displays, etc., allowing the user to interact in various ways. Furthermore, as described above, in at least some embodiments, one of controller 400 and HMD device 344 may include multiple light sources, while the other may include multiple angle-sensitive detectors or other types of detectors or sensors. The techniques described herein can be used for various types of position tracking, but are not limited to HMDs, controllers, etc.

[0051] Figure 5 A schematic block diagram of an HMD device 500 according to one or more embodiments of the present disclosure is shown. The HMD device 500 may be similar to or the same as HMD devices discussed elsewhere herein. Therefore, the above discussion regarding HMD devices can also be applied to the HMD device 500. Furthermore, at least some components of the HMD device 500 may be present in other components of an HMD system, such as a controller, base station, etc. Therefore, at least some of the descriptions below are applicable to such other components.

[0052] HMD device 500 includes a processor 502, a front-facing or front-mounted camera 504, multiple angle-sensitive detectors 506 (e.g., quad-cell photodiodes, position-sensitive detectors), and optionally includes an IMU 507 or multiple light sources 509. In some implementations, HMD device 500 may include one of the angle-sensitive detectors or light sources, and other components (e.g., controllers, base stations) may include another of the angle-sensitive detectors or light sources. HMD device 500 may include a display subsystem 508 (e.g., two displays and corresponding optical systems). HMD device 500 may also include a non-transitory data memory 510, which may store instructions or data for position tracking 512, instructions or data for display functions 514 (e.g., games), and / or other programs 516. HMD system 500 may include several such... Figure 1 The functions or permissions of the local computing system 120 or media content provider 190 shown and discussed above, as well as those of the local computing system 120 or media content provider 190, are as follows: Figure 1 The functions of the local computing system 120 or media content provider 190 shown and discussed above.

[0053] HMD device 500 may also include various I / O components 518, which may include one or more user interfaces (e.g., buttons, touchpads, speakers), one or more wired or wireless communication interfaces, etc. As an example, I / O components 518 may include communication interfaces that allow HMD device 500 to communicate with external device 520 via a wired or wireless communication link 522. As a non-limiting example, external device 520 may include a host, server, mobile device (e.g., smartphone, wearable computer), controller, etc. The various components of HMD device 500 may be housed in a single enclosure, in separate enclosures (e.g., a host), or any combination thereof.

[0054] It should be understood that the computing systems and devices shown are merely illustrative and not intended to limit the scope of this disclosure. For example, the HMD 500 and / or external device 520 can be connected to other devices not shown, including via one or more networks such as the Internet or via the Web. More generally, such computing systems or devices may include any combination of hardware capable of interacting and performing functions of the types described, such as when programmed or configured with appropriate software, including but not limited to desktop computers, laptop computers, tablet touch computers, tablet computers or other computers, smartphone computing devices and other cellular phones, Internet devices, PDAs and other electronic organizers, database servers, network storage devices and other network devices, wireless phones, pagers, television-based systems (e.g., using set-top boxes and / or personal / digital video recorders and / or game consoles and / or media servers), and various other consumer products including appropriate intercommunication capabilities. For example, in at least some embodiments, illustrated systems 500 and 520 may include executable software instructions and / or data structures that, when loaded onto and / or executed by a particular computing system or device, can be used to program or configure those systems or devices, such as configuring the processors of those systems or devices. Alternatively, in other embodiments, some or all of the software system may be executed in memory on another device and communicate with the illustrated computing system / device via inter-computer communication. Furthermore, although various entries are shown to be stored in memory or on storage at various times (e.g., when in use), these entries or portions thereof may be transferred between memory and storage and / or between storage devices (e.g., in different locations) for memory management and / or data integrity purposes.

[0055] Therefore, in at least some embodiments, the illustrated system is a software-based system comprising software instructions that, when executed by a processor and / or other processor device, program the processor to automatically perform the operations of the system. Furthermore, in some embodiments, some or all of the system may be implemented or provided in other ways, for example, at least partially as firmware and / or hardware devices, including but not limited to one or more application-specific integrated circuits (ASICs), standard integrated circuits, controllers (e.g., by executing appropriate instructions, and including microcontrollers and / or embedded controllers), field-programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), etc. Some or all of the system or data structures may also be stored (e.g., as software instruction content or structured data content) on non-transitory computer-readable storage media such as hard disks or flash drives or other non-volatile storage devices, volatile or non-volatile memories (e.g., RAM), network storage devices, or portable media articles (e.g., DVDs, CDs, optical discs, flash memory devices, etc.) that can be read by appropriate drives or via appropriate connections. In some embodiments, the system, modules, and data structures can also be transmitted as generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) in various forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames) over various computer-readable transmission media, including wireless-based and wired / cable-based media. In other embodiments, such computer program products may also take other forms. Therefore, this disclosure can be implemented with other computer system configurations.

[0056] Figure 6This is a schematic diagram of an environment 600 according to a non-limiting embodiment, wherein machine learning techniques can be used to implement a tracking subsystem, such as the tracking subsystem discussed herein, for tracking an HMD device, one or more controllers, or other components. Environment 600 includes a model training section 601 and an inference section 603. In the training section 601, training data 602 is fed into a machine learning algorithm 604 to generate a trained machine learning model 606. The training data may include, for example, labeled data from angle-sensitive detectors, which specifies the position and / or orientation of a particular object relative to one or more light sources (e.g., LEDs). As a non-limiting example, in an embodiment including components (e.g., an HMD, a controller) having 30 angle-sensitive detectors, each training sample may include the output from each or a subset of the angle-sensitive detectors, the known or inferred position or orientation of the component, and information regarding the position or orientation of one or more light sources. As described below, each angle-sensitive detector may output a single data point (e.g., angle), or may output multiple data points, such as two or four signals, each signal indicating the power or intensity of light received at a specific active element (e.g., sub-detector or unit, resistor, etc.) of the angle-sensitive detector.

[0057] Training data 602 can be obtained from multiple users and / or a single user of the HMD system. Training data 602 can be obtained in a controlled environment and / or during actual user use (“field training”). Furthermore, in at least some embodiments, model 606 can be updated or calibrated in a timely manner (e.g., periodically, continuously, after certain events) to provide accurate position tracking predictions.

[0058] In the inference section 603, runtime data 608 is provided as input to a trained machine learning model 606, which generates a position tracking prediction 610. Continuing the example above, output data from an angle-sensitive detector (e.g., intensity data, angle data) and optionally information about one or more light sources can be provided as input to the trained machine learning model 606, which can process the data to predict the position of a component. The tracking prediction 610 can then be provided to one or more components associated with the HMD device, such as one or more VR or AR applications, one or more display or presentation modules, one or more mechanical controls, one or more additional position tracking subsystems, etc.

[0059] Machine learning techniques used to implement the features discussed herein can include any type of suitable architecture or technique. As a non-limiting example, machine learning model 606 may include one or more of decision trees, statistical hierarchical models, support vector machines, artificial neural networks (ANNs) such as convolutional neural networks (CNNs) or recurrent neural networks (RNNs) (e.g., long short-term memory (LSTM) networks), hybrid density networks (MDNs), hidden Markov models, or others. In at least some implementations, such as utilizing an RNN implementation, machine learning model 606 can use past input (memory, feedback) information to predict the position of one or more HMD components. Such implementations can advantageously utilize sequential data to determine motion information or pre-position predictions, which can provide more accurate real-time position predictions.

[0060] Figure 7 This is a flowchart of an example method 700 for operating an HMD system to track the position of HMD components during use. Method 700 can be, for example... Figure 5 The position tracking system or module 512 of the HMD system 500 shown is executed. As described above, method 700 can be used to track the position of any component, such as an HMD device, one or more controllers, etc.

[0061] The implementation of method 700 shown begins at action 702, wherein a first HMD system component having a plurality of angle-sensitive detectors is provided. The plurality of angle-sensitive detectors can be used to detect light emitted from one or more light sources, which can be fixedly positioned (e.g., mounted to a wall or ceiling) or movably positioned (e.g., connected to an HMD or controller). In operation, each of the plurality of angle-sensitive detectors captures sensor data in its respective field of view at a frame rate. The sensor data can include any type of data used by a processor to detect the presence and orientation of the light source relative to the angle-sensitive detector. In at least some embodiments, each angle-sensitive detector can include one or more sensors (e.g., photodiodes) having image sensing circuitry and image processing circuitry thereon. The angle-sensitive detectors can output relatively raw data (e.g., light intensity or power data) or processed data (e.g., angle of incidence data).

[0062] At 704, a second HMD system component comprising multiple light sources (e.g., near-IR LEDs) may be provided. The second HMD system component may include a controller, an HMD device, or a light source located in a fixed position (e.g., ceiling, wall).

[0063] In 706, at least one processor of the HMD system can make a light source emit light. The light source can be illuminated in such a way that each of the angle-sensitive detectors can detect light from a single light source at a time, or more generally, in such a way that the system can determine from which light source the light detected by the angle-sensitive detector was received. This can be achieved by using any suitable type of multiplexing to multiplex the illumination of the light source, such as time multiplexing, wavelength multiplexing, frequency multiplexing, polarization multiplexing, or other techniques that allow the system to know the light source received from each angle-sensitive detector during use.

[0064] As an example of time multiplexing, at least one processor can illuminate only a subset of light sources (e.g., one, two, four) at a time. For example, at least one processor can illuminate light sources sequentially, one at a time, and collect sensor data in response to each light source.

[0065] As an example of wavelength multiplexing, different subsets of a light source can emit light of different wavelengths, and different subsets of an angle-sensitive detector can be used to sense light of different wavelengths. Therefore, light sources with different wavelengths can be lit simultaneously and detected by corresponding wavelength-sensitive detectors.

[0066] As an example of frequency reuse, a subset of light sources can be illuminated in a defined pattern or frequency, which can be detected by an angle-sensitive detector to identify the specific light source.

[0067] As an example of polarization multiplexing, a subset of light sources can be polarized differently (e.g., linear, circular), and a corresponding subset of angle-sensitive detectors can be configured to detect certain polarized light, which allows multiple light sources to be lit simultaneously.

[0068] At 708, at least one processor associated with the HMD system can receive sensor data from multiple angle-sensitive detectors. As described above, for each angle-sensitive detector, the sensor data can indicate the angle of arrival of light emitted from a known light source. At 710, at least one processor associated with the HMD system can optionally receive sensor data from an inertial measurement unit (IMU) used to provide inertial tracking capability or sensor data from one or more additional sensors.

[0069] At 712, at least one processor associated with the HMD system can process the received sensor data. For example, at least one processor can fuse some or all of the sensor data to track one or more features present in the environment in which the HMD system operates. The sensor data may include sensor data from multiple angle-sensitive detectors, and optionally sensor data from an IMU or a camera. At least one processor can process the sensor data, for example, using a machine learning model (e.g., model 606) or another solver.

[0070] At 714, at least one processor associated with the HMD system can track the position (e.g., location, orientation, or movement) of components of the HMD system in real time during user use of the HMD system in the environment. As described above, method 700 can continue during the operation of the HMD to continuously track the position of components of the HMD system.

[0071] Figure 8A and 8B Top and perspective views of an exemplary angle-sensitive detector 800 that may be used in one or more embodiments of this disclosure are shown. In this example, the angle-sensitive detector 800 includes a quadrant-cell photodiode comprising four separate photodiode active regions or elements 802A-802D separated by a small gap on a common substrate 804. It should be understood that other types of angle-sensitive detectors may also be used, such as photodiode detectors with fewer or more cells, position-sensitive detectors, etc.

[0072] In the non-limiting example shown, the active region (e.g., anode) of each element 802A-802D is individually available, such that a light spot illuminating a single quadrant can be electrically characterized as existing only in that quadrant. As the light spot translates over the angle-sensitive detector 800, the energy distribution of the light spot is between adjacent elements 802A-802D, and the difference in the electrical contribution to each element defines the relative position of the light spot with respect to the center of the angle-sensitive detector. The relative intensity distribution on elements 802A-802D can be used to determine the position of the light spot.

[0073] In this simplified example, the angle-sensitive detector 800 includes a cover 810 having an aperture 808 that allows light 814 from a light source 812 to pass through. As shown, the light 814 forming a light spot 806 through the aperture 808 can be electrically characterized to determine the angle of the light 814, and thus the angle of the light source 812 relative to the angle-sensitive detector 800. As described below, the systems and methods of this disclosure can utilize multiple light sources and angle-sensitive detectors to determine the position of components in an HMD system.

[0074] It should be understood that the angle-sensitive detector of this disclosure may include one or more of any suitable type of detector, including a four-unit photodiode detector, a position-sensitive detector (PSD) utilizing a resistive sheet, a photodiode detector having fewer (e.g., 2) or more (e.g., 16) independent sensing elements, or any other detector capable of detecting the angle of arrival of light emitted from a light source. Furthermore, as discussed below, in at least some embodiments, the angle-sensitive detector or light source of this disclosure may utilize various optical components, such as filters, lenses, polarizers, etc., to improve the functionality of the systems and methods discussed herein.

[0075] Figure 9 This is a simplified diagram of an environment 900 of an HMD system according to a non-limiting embodiment, which uses light sources and angle-sensitive detectors to determine the positions of components of the HMD system. In this example, a first component 902, such as an HMD, includes a plurality of light sources 906 (two shown, 906a-906b), and a second component 904, such as a controller of the HMD system, includes a plurality of angle-sensitive detectors 908 (two shown, 908a-908b). Angle-sensitive detectors 908a and 908b are separated from each other by a known distance d1 on the second component 904, and light sources 906a and 906b are separated from each other by a known distance d2 on the first component 902. The first and second components can be any components of the HMD system, such as an HMD, controller, base station, fixed or mobile light sources, fixed or mobile angle-sensitive detectors, etc.

[0076] In this example, angle-sensitive detector 908a is used to determine that light arrives from light source 906a at an angle of 910 and from light source 906b at an angle of 912. Similarly, angle-sensitive detector 908b is used to determine that light arrives from light source 906b at an angle of 914 and from light source 906a at an angle of 916. It can be understood that, given the angles of arrival 910, 912, 914, and 916 and the known geometric relationships (e.g., distances d1 and d2) between light source 906 and detector 908, methods (e.g., triangulation) can be used to determine the relative position, orientation, or movement between the first component 902 and the second component 904. As described above, one or more solvers or machine learning methods can be used to determine the position of the components using sensor data from the angle-sensitive detectors and / or light source data indicating information about the light sources in the HMD system.

[0077] Figure 10This is a schematic diagram 1000 of an example light source 1002 and an angle-sensitive detector 1004 of this disclosure. The light source 1002 and angle-sensitive detector 1004 may be similar to or the same as any light source and angle-sensitive detector discussed herein and may be used in any implementation of this disclosure. In the example shown, the light source 1002 may include an optical subsystem 1006, and the angle-sensitive detector 1004 may include an optical subsystem 1008. Optical subsystems 1006 and 1008 may be the same as or different from each other and may each include one or more optical components. Optical subsystems 1006 and 1008 may be integrated with the light source 1002 and the angle-sensitive detector 1004, or they may be separate components. Non-limiting examples of optical components include one or more lenses, one or more polarizers, one or more filters, one or more apertures, etc. In at least some embodiments, a subset of the light source may include one type of optical subsystem, and one or more other subsets of the light source may include another type of optical subsystem. Similarly, a subset of the angle-sensitive detector may include one type of optical subsystem, and one or more other subsets of the angle-sensitive detector may include another type of optical subsystem. As an example, the optical subsystem may include filters to remove visible light or other types of light. Furthermore, as described above, the optical subsystem may include components that facilitate multiplexing of one or more of the aforementioned types, allowing multiple light sources to be illuminated simultaneously without confusion with the emitting light source.

[0078] Various implementations of the device and / or process have been illustrated in the foregoing detailed description using block diagrams, schematic diagrams, and examples. Where such block diagrams, schematic diagrams, and examples contain one or more functions and / or operations, those skilled in the art will understand that each function and / or operation within such block diagrams, flowcharts, or examples can be implemented individually and / or collectively by a wide range of hardware, software, firmware, or virtually any combination thereof. In one implementation, this subject matter can be implemented using an application-specific integrated circuit (ASIC). However, those skilled in the art will recognize that the implementations disclosed herein can be implemented, in whole or in part, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more controllers (e.g., microcontrollers), as one or more programs running on one or more processors (e.g., microprocessors), as firmware, or as virtually any combination thereof, equivalently implemented in a standard integrated circuit, and that designing circuits and / or writing code for software and / or firmware according to this disclosure will be within the skill level of those skilled in the art.

[0079] Those skilled in the art will recognize that many of the methods or algorithms described herein may employ additional actions, omit some actions, and / or perform actions in a different order than specified.

[0080] Furthermore, those skilled in the art will understand that the mechanisms taught herein can be distributed as program products in various forms, and the illustrative implementations are equally applicable regardless of the specific type of signal-bearing medium used to actually perform the distribution. Examples of signal-bearing media include, but are not limited to, recordable media such as floppy disks, hard disk drives, CD-ROMs, digital magnetic tapes, and computer memory.

[0081] The various implementations described above can be combined to provide further implementations. All U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, foreign patent applications, and non-patent publications (including U.S. Patent Application No. 16 / 752,478, filed January 24, 2020) referred to herein are incorporated herein by reference in their entirety. If necessary, aspects of the implementations can be modified to incorporate various patented, applied, and published systems, circuits, and concepts to provide further implementations.

[0082] Based on the detailed description above, these and other changes can be made to the embodiments. Generally, the terminology used in the following claims should not be construed as limiting the claims to the specific embodiments disclosed in the specification and claims, but should be interpreted to include all possible embodiments and the full scope of equivalents conferred by these claims. Therefore, the claims are not limited to this disclosure.

Claims

1. A head-mounted display system, comprising: The first wearable head-mounted display system component; Multiple angle-sensitive detectors carried by the first head-mounted display system component, each of the multiple angle-sensitive detectors including a cover having a hole therein and multiple separate photodiode units, and each photodiode unit being operable to generate an electrical signal indicating an intensity value of light incident on the photodiode unit, wherein the relative values ​​of the respective intensity values ​​can be used to generate sensor data indicating the angle of arrival of light detected by the angle-sensitive detector; The second head-mounted display system component includes multiple light sources; At least one non-transitory processor-readable storage medium storing at least one of processor-executable instructions and data; as well as At least one processor operatively connected to the plurality of angle-sensitive detectors and the at least one non-transitory processor-readable storage medium, wherein, in operation, the at least one processor: This causes one or more of the plurality of light sources to emit light; Receive sensor data from the plurality of angle-sensitive detectors, the sensor data indicating the angle of arrival of light emitted from one or more of the plurality of light sources; The received sensor data is processed; as well as The position of at least one of the first head-mounted display system component and the second head-mounted display system component is tracked based at least in part on the processing of received sensor data.

2. The head-mounted display system according to claim 1, wherein, The first head-mounted display system component includes a head-mounted display device or a handheld controller that can be worn on a user's head.

3. The head-mounted display system according to claim 1, wherein, Each of the plurality of angle-sensitive detectors includes four photodiode units separated by gaps on a common substrate.

4. The head-mounted display system according to claim 1, wherein, Each of the plurality of angle-sensitive detectors includes at least four photodiode units.

5. The head-mounted display system according to claim 1, wherein, The second head-mounted display system component includes multiple light-emitting diodes.

6. The head-mounted display system according to claim 1, wherein, The first head-mounted display system component includes one of a head-mounted display device, a controller, and a base station, and the second head-mounted display system component includes another of a head-mounted display device, a controller, and a base station.

7. The head-mounted display system according to claim 1, wherein, The second head-mounted display system component includes a component fixed at a location near the operating environment of the head-mounted display system.

8. The head-mounted display system according to claim 1, wherein, The plurality of light sources includes LED light sources.

9. The head-mounted display system according to claim 1, wherein, In operation, the second head-mounted display system component illuminates a subset of the plurality of light sources at a given time, the subset not including all of the plurality of light sources of the second head-mounted display system component.

10. The head-mounted display system according to claim 1, wherein, During operation, the second head-mounted display system component sequentially illuminates the plurality of light sources.

11. The head-mounted display system according to claim 1, wherein, In operation, the second head-mounted display system component uses multiplexing to illuminate the plurality of light sources.

12. The head-mounted display system according to claim 11, wherein, The multiplexing includes at least one of time multiplexing, wavelength multiplexing, frequency multiplexing, and polarization multiplexing.

13. The head-mounted display system according to claim 1, wherein, Each of the light sources includes an optical subsystem, which includes at least one of a lens, a filter, and a polarizer.

14. The head-mounted display system according to claim 1, wherein, Each of the angle-sensitive detectors includes an optical subsystem, which includes at least one of a lens, a filter, and a polarizer.

15. The head-mounted display system according to claim 1, wherein, Each of the angle-sensitive detectors includes a lens that causes off-axis light to have a similar size to on-axis light at the sensitive element of the angle-sensitive detector.

16. The head-mounted display system according to claim 1, wherein, In order to process the received sensor data, the at least one processor provides the received sensor data as input to a trained machine learning model.

17. The head-mounted display system according to claim 16, wherein, The processor is configured to receive training data and use the training data to train the machine learning model.

18. The head-mounted display system according to claim 1, wherein, The first head-mounted display system component includes an inertial measurement unit (IMU) sensor operatively connected to the at least one processor, wherein, in operation, the at least one processor: Receive IMU sensor data from the IMU sensor; Processing the IMU sensor data and the sensor data received from the plurality of angle-sensitive detectors; and The position of at least one of the first head-mounted display system component and the second head-mounted display system component is tracked at least in part based on the processing of received IMU sensor data and received sensor data.

19. The head-mounted display system according to claim 1, wherein, The frame rate of the angle-sensitive detector is greater than or equal to 1000 frames per second.

20. A method of operating a head-mounted display system, the head-mounted display system comprising a first head-mounted display system component wearable by a user and a plurality of angle-sensitive detectors carried by the first head-mounted display system component, each of the plurality of angle-sensitive detectors including a cover having an aperture therein, and a plurality of separate photodiode units, each photodiode unit operable to generate an electrical signal indicating an intensity value of light incident on the photodiode unit, wherein relative values ​​of the respective intensity values ​​are available for generating sensor data indicating the angle of arrival of light emitted from one or more light sources, the head-mounted display system further comprising a second head-mounted display system component including a plurality of light sources, the method comprising: This causes one or more of the plurality of light sources to emit light; Sensor data indicating the angle of arrival of light emitted from the one or more light sources is captured via each of the plurality of angle-sensitive detectors; The sensor data is received from the plurality of angle-sensitive detectors by at least one processor; The received sensor data is processed by the at least one processor; as well as The position of at least one of the first head-mounted display system components and the second head-mounted display system component is tracked by the at least one processor based at least in part on the processing of the received sensor data.

21. A head-mounted display system, comprising: The first wearable head-mounted display system component; The plurality of angle-sensitive detectors carried by the first head-mounted display system component, in operation, each of the plurality of angle-sensitive detectors includes a cover having a hole therein and a plurality of separate photodiode units, and each photodiode unit is operable to generate an electrical signal indicating an intensity value of light incident on the photodiode unit, wherein the relative values ​​of the respective intensity values ​​can be used to generate sensor data indicating the angle of arrival of light detected by the angle-sensitive detector; The second head-mounted display system component includes multiple light sources; Control circuit, during operation: To cause one or more of the plurality of light sources to emit light; Receive sensor data from the plurality of angle-sensitive detectors; The received sensor data is processed; as well as The position of at least one of the first head-mounted display system component and the second head-mounted display system component is tracked based at least in part on the processing of received sensor data.

22. The head-mounted display system according to claim 21, wherein, Each of the plurality of angle-sensitive detectors includes a plurality of photodiode units separated by gaps on a common substrate.

23. The head-mounted display system according to claim 21, wherein, The first head-mounted display system component includes one of a head-mounted display device, a controller, and a base station, and the second head-mounted display system component includes another of a head-mounted display device, a controller, and a base station.

24. The head-mounted display system according to claim 21, wherein, The plurality of light sources includes LED light sources that emit invisible light.

25. The head-mounted display system according to claim 21, wherein, In operation, the second head-mounted display system component illuminates a subset of the plurality of light sources at a given time, the subset not including all of the plurality of light sources of the second head-mounted display system component.

26. The head-mounted display system according to claim 21, wherein, During operation, the second head-mounted display system component sequentially illuminates the plurality of light sources.

27. The head-mounted display system according to claim 21, wherein, In operation, the second head-mounted display system component uses multiplexing to illuminate the plurality of light sources.

28. The head-mounted display system according to claim 27, wherein, The multiplexing includes at least one of time multiplexing, wavelength multiplexing, frequency multiplexing, and polarization multiplexing.

29. The head-mounted display system according to claim 21, wherein, Each of the light sources includes an optical subsystem, which includes at least one of a lens, a filter, and a polarizer.

30. The head-mounted display system according to claim 21, wherein, Each of the angle-sensitive detectors includes an optical subsystem, which includes at least one of a lens, a filter, and a polarizer.

Images