Wireless data transmission system for head mounted displays

Abstract

A system and method for providing low-latency, high-bandwidth wireless data transmission for various applications, such as virtual reality, augmented reality, or video applications, is provided. A wireless data transmission system is provided that includes an electrically steerable antenna, such as a phased array antenna, operable to selectively steer its beam based on a control input. The wireless data transmission system includes a tracking subsystem operable to track a portable wireless device (e.g., a head-mounted display (HMD), a tablet computer, a smartphone) as the portable wireless device moves about within a tracked volume. The wireless data transmission system utilizes a known position, orientation, or movement of the portable wireless device to receive data (e.g., video data) from the tracking subsystem and compensates for the movement of the portable wireless device by selectively adjusting a beamforming pattern of the steerable antenna based at least in part on the tracking data received from the tracking subsystem.

Description

[Technical field]

[0001] The present disclosure relates generally to wireless data transmission systems and methods. [Background technology]

[0002] One current generation of virtual reality ("VR") experiences is created using head mounted displays ("HMDs"). HMDs may be tethered to a stationary computer (such as a personal computer ("PC"), laptop, game console, etc.), may be combined and / or integrated with a smartphone and / or its associated display, or may be self-contained. In general, an HMD is a display device that is worn on the user's head and has a small display device in front of one eye (monocular HMD) or both eyes (binocular HMD). The display unit is usually miniaturized and may include, for example, CRT, LCD, liquid crystal on silicon (LCOS), or OLED technologies. Binocular HMDs may be able to display a different image to each eye. This capability is used to display stereoscopic images.

[0003] VR systems require large amounts of video data to provide a high-quality VR experience. Furthermore, the video data must be provided with low latency (e.g., less than 20 ms) to prevent users from experiencing dizziness and discomfort due to the time lag between the user's head movement and the corresponding change in the scene display. This requirement is called "Motion to Photon" or MTP. Thus, the amount of data required to provide suitable performance can be very large (e.g., 1 Gb / s, 3 Gb / s, or more). Current wireless systems, such as IEEE 802.11ac, struggle to maintain high throughput levels, especially when the user moves around wearing the HMD or when the presence of other devices can cause interference. Thus, to achieve the required throughput, most VR systems currently utilize a wired tether between a portable HMD worn by the user and a stationary computer that transmits the video data for display by the HMD. Summary of the Invention

[0004] A wireless data transmission system may be summarized as including: a tracking subsystem that, upon operation, generates tracking data indicative of a physical location of a portable wireless device as the portable wireless device moves within a tracked volume; an antenna array selectively configurable to provide a plurality of beamforming patterns; a radio frequency (RF) transmitter operably coupled to the antenna array, where, upon operation, the RF transmitter wirelessly transmits data to the portable wireless device via the antenna array; and a beamforming controller operably coupled to the tracking subsystem and the antenna array, where, upon operation, the beamforming controller receives the generated tracking data from the tracking subsystem and dynamically adjusts the beamforming pattern of the antenna array based at least in part on the tracking data received from the tracking subsystem.

[0005] The antenna array may include a planar phased array antenna having a plurality of antenna sub-elements. To dynamically adjust the beamforming pattern of the antenna array, the beamforming controller may dynamically adjust the beamforming weights of each of the plurality of antenna sub-elements to modify at least one of the phase shift or amplitude of each of the plurality of antenna sub-elements. The beamforming controller may directly calculate the values ​​of the beamforming weights of each of the plurality of antenna sub-elements based on the tracking data received from the tracking subsystem. The beamforming controller may retrieve the values ​​of the beamforming weights of each of the plurality of antenna sub-elements from a memory based on the tracking data received from the tracking subsystem. The RF transmitter may operate in one or more frequency bands between 30 GHz and 300 GHz. The RF transmitter may operate in a frequency band of 60 GHz.

[0006] The tracking data generated by the tracking subsystem may further include orientation data indicative of an orientation of the portable wireless device within the tracked volume. The tracking subsystem may generate tracking data indicative of a physical location of at least one object within the tracked volume other than the tracked portable wireless device. The beamforming controller may dynamically adjust a beamforming pattern of the antenna array based at least in part on the tracking data indicative of a physical location of the at least one other object. The at least one other object may include at least one of a handheld controller, a part of a human body, a stationary object, or a physical boundary of the tracked volume. The tracking data may include predictive data indicative of a future physical location of the portable wireless device. The beamforming controller may dynamically adjust a beamforming pattern of the antenna array based at least in part on the predictive data indicative of a future physical location of the portable wireless device. The beam controller may receive quality data indicative of a quality of a wireless connection between the wireless data transmission system and the portable wireless device, and dynamically adjust a beamforming pattern of the antenna array based at least in part on the received quality data. The beam controller may receive quality data indicative of a quality of the wireless connection between the wireless data transmission system and the portable wireless device, store the quality data and corresponding tracking data, analyze the stored quality data and corresponding tracking data, and dynamically adjust a beamforming pattern of the antenna array based at least in part on the analyzed quality data and corresponding tracking data. The RF transmitter may wirelessly transmit video data to the portable wireless device via the antenna array.

[0007] The RF transmitter may be a first RF transmitter, and the wireless data transmission system may further include a second RF transmitter that, in operation, transmits non-video data using a different RF band than the first RF transmitter. The tracking subsystem may implement at least one of video tracking, simultaneous localization and mapping (SLAM) tracking, acoustic tracking, inertial tracking, magnetic tracking, or optical tracking. The beam controller may dynamically adjust the beamforming pattern of the antenna array based at least in part on the past tracking data and the past quality data. The beam controller may receive quality data indicative of a quality of a wireless connection between the wireless data transmission system and the portable wireless device, compare the received quality data to a quality limit value, and dynamically adjust the beamforming pattern of the antenna array based on criteria other than the tracking data in response to a determination that the quality data does not meet the quality limit value. The other criteria may include the received quality data indicative of a quality of the wireless connection. The portable wireless device may include a head mounted display (HMD), a tablet computer, a smartphone, or a wearable computer. In operation, the beamforming controller may include a multiple-input multiple-output (MIMO) controller that receives pre-derived channel state information having position dependency and selects one or more of coefficients, weights, and stream encodings based at least in part on the received channel state information.

[0008] A method of operating a wireless data transmission system may be summarized as comprising: generating tracking data indicative of a physical location of a portable wireless device within a tracked volume, receiving the generated tracking data from a tracking subsystem, dynamically adjusting a beamforming pattern of an antenna array based on the tracking data, and wirelessly transmitting video data via the antenna array while the portable wireless device moves within the tracked volume. Dynamically adjusting a beamforming pattern of the antenna array may include dynamically adjusting beamforming weights of each of a plurality of antenna sub-elements of the antenna array to modify at least one of a phase shift or an amplitude of each of the plurality of antenna sub-elements. Generating tracking data may include generating orientation data indicative of an orientation of the portable wireless device within the tracked volume.

[0009] The method may further include receiving quality data indicative of a quality of a wireless connection between the wireless data transmission system and the portable wireless device, and dynamically adjusting a beamforming pattern of the antenna array based at least in part on the received quality data.

[0010] The method may further include receiving quality data indicative of a quality of a wireless connection between the wireless data transmission system and the portable wireless device, storing the quality data and corresponding tracking data, analyzing the stored quality data and corresponding tracking data, and dynamically adjusting a beamforming pattern of the antenna array based at least in part on the analyzed quality data and corresponding tracking data.

[0011] The method may further include transmitting non-video data to the portable wireless device using an RF band different from the RF band used to transmit the video data. Generating the tracking data may include implementing at least one of video tracking, Simultaneous Localization and Mapping (SLAM) tracking, acoustic tracking, inertial tracking, magnetic tracking, or optical tracking.

[0012] The method may further include receiving quality data indicative of a quality of a wireless connection between the wireless data transmission system and the portable wireless device, comparing the received quality data to a quality limit value, and in response to determining that the quality data does not meet the quality limit value, dynamically adjusting a beamforming pattern of the antenna array based on the received quality data instead of the received tracking data.

[0013] A method of operating a wireless data transmission system may be summarized as including: wirelessly transmitting video data via a steerable phased array antenna to a portable wireless device present within a tracked volume; repeatedly tracking a physical location of the portable wireless device during the wireless transmission of the video data; and dynamically adjusting a beamforming pattern of the steerable phased array antenna to track a movement of the portable wireless device within the tracked volume. The dynamically adjusting the beamforming pattern may include dynamically adjusting beamforming weights of the steerable phased array antenna. [Brief description of the drawings]

[0014] In the drawings, identical reference numbers identify similar elements or acts. The sizes and relative positions of elements within the drawings are not necessarily drawn to scale. For example, the shapes and angles of various elements are not necessarily drawn to scale. Also, some of these elements may be arbitrarily enlarged and positioned to improve the readability of the drawings. Furthermore, the particular shapes of the elements as drawn are not necessarily intended to convey any information regarding the actual shape of the particular elements, but may only be selected to make the drawings easier to understand.

[0015] [Figure 1] FIG. 1 is a schematic diagram of a network environment including one or more systems suitable for implementing at least some of the techniques described in this disclosure.

[0016] [Diagram 2] FIG. 1 is a pictorial diagram illustrating an exemplary environment in which at least some of the described techniques are used in conjunction with an exemplary head-mounted display device that is wirelessly coupled to a local video-rendering computing system and provides a virtual reality display to a user.

[0017] [Diagram 3] FIG. 1 is a schematic diagram of one embodiment of a wireless data transmission system in which a steerable phased array antenna system is used to wirelessly communicate with a portable wireless device, such as an HMD device, according to one non-limiting exemplary implementation.

[0018] [Figure 4A] 1 illustrates an exemplary planar phased array antenna including four antenna sub-elements according to one non-limiting example implementation.

[0019] [Figure 4B]1 illustrates an exemplary planar phased array antenna including eight antenna subelements arranged in two rows of four subelements, according to one non-limiting illustrative implementation.

[0020] [Diagram 5] 1 is a flow diagram of a method of operating a wireless data transmission system that provides data to a portable wireless device, according to one non-limiting example implementation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] In the following description, certain specific details are set forth to provide a thorough understanding of the various implementations disclosed. However, those skilled in the art will recognize that implementations may be practiced without one or more of these specific details, or with other methods, components, materials, etc. In other instances, well-known structures associated with computer systems, server computers, and / or communication networks have not been shown or described in detail so as not to unnecessarily obscure the description of the implementations.

[0022] Unless the context requires otherwise, throughout this specification and the claims that follow, the word "comprising" is synonymous with "including" and is inclusive or open-ended (i.e., does not exclude other elements or method operations not recited).

[0023] Throughout this specification, a reference to "one implementation" or "an implementation" means that a particular feature, structure, or characteristic described in connection with that implementation is included in at least one implementation. Thus, the appearances of the phrase "in one implementation" or "in an implementation" in various places throughout this specification do not necessarily all refer to the same implementation. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more implementations.

[0024] As used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It should also be noted that the term "or" is generally used in its sense to include "and / or" unless the context clearly dictates otherwise.

[0025] The headings and abstracts provided herein are for convenience only and do not interpret the scope or meaning of the implementations.

[0026] One or more implementations of the present disclosure relate to systems and methods for providing low latency and high bandwidth wireless data transmission for various applications, such as virtual reality applications, augmented reality applications, or video applications. In at least some implementations, a wireless data transmission system is provided that includes an electrically steerable antenna, such as a phased array antenna, operable to selectively steer its beam based on electronic control inputs. The wireless data transmission system may include a tracking subsystem operable to track a portable wireless device (e.g., HMD, tablet computer, smartphone) as it moves around within a tracked volume. Advantageously, the wireless data transmission system utilizes a known current position or a predicted future position and / or orientation of the portable wireless device that receives data (e.g., video data) from the tracking subsystem, and compensates for the motion of the portable wireless device by selectively adjusting a beamforming pattern of the steerable antenna based at least in part on the tracking data received from the tracking subsystem.

[0027] Among other things, this feature allows for a reduction in the amount of buffering, thereby reducing latency, which is a critical parameter in some applications, such as VR applications, where MTP is an important industry metric. Furthermore, at least in some implementations, tracking data from other objects, such as one or more handheld controllers, may also be used to detect when a user's hand is moving in a direction that may block the receiver antenna of the portable wireless device. It has been found that even partial obstruction by the user's hand can cause significant data loss, for example, without heavy reliance on buffering, interleaving, or forward error correction.

[0028] FIG. 1 is a schematic diagram of a network environment 100 including a local media rendering (LMR) system 102 (e.g., a gaming system, a desktop computing system, a laptop computer). The LMR system 102 includes a local computing system 104 suitable for performing at least some of the techniques described herein and a portable display device 106 (e.g., an HMD with two display panels, a tablet computer, a laptop computer, a smartphone). In the embodiment shown in FIG. 1, the local computing system 104 is communicatively connected to the portable display device 106 via a wireless transmission link 108, as described further below. In at least some embodiments, the local computing system 104 may provide image data for display on a panel display device (not shown), such as a TV, console, or monitor, via a wired or wireless link. In various embodiments, the local computing system 104 may include, by way of non-limiting examples, a general-purpose computing system, a gaming console, a video stream processing device, a portable computing device (e.g., a mobile phone, a PDA, or other portable device), a VR or AR processing device, or other computing system.

[0029] In the illustrated embodiment, the local computing system 104 has components including one or more general-purpose hardware processors (e.g., central processing units or "CPUs") 110, processor-readable memory or storage 112, various I / O ("input / output") hardware components 114 (e.g., a keyboard, a mouse, one or more gaming controllers, speakers, a microphone, an IR transmitter and / or receiver, a camera, biofeedback sensors, etc.), a video subsystem 116 including one or more dedicated hardware processors (e.g., graphics processing units or "GPUs") 118 and video memory (VRAM) 120, computer-readable storage 150, and one or more wired or wireless network connections 122. In at least some implementations, the local computing system 104 may implement a unified memory architecture, in which a pool of managed memory is shared between the one or more CPUs and one or more GPUs.

[0030] The local computing system 104, in the illustrated embodiment, may also be communicatively connected via one or more computer networks 134 and network link 136 to an exemplary network-accessible media content provider 140 that may further provide content to the local computing system 104 for display via one or more links 138. The media content providers 140 may include one or more computing systems (not shown), each of which may have similar components as the local computing system 104 (including one or more hardware processors, I / O components, local storage devices and memory), although some details are not shown in the media content server computing systems for ease of explanation.

[0031] It will also be understood that the local computing system 104 and the portable wireless device 106 are merely examples and are not intended to limit the scope of the present disclosure. A computing system may instead include multiple computing systems or devices interacting with each other and may be connected to other devices not shown, such as through one or more networks such as the Internet, the Web, or private networks (e.g., cellular networks, etc.). More generally, a computing system or other computing node may include any combination of hardware or software capable of interacting with each other and performing the types of functions described. Such combinations include, but are not limited to, desktop or other computers, gaming systems, database servers, network storage devices and other network devices, PDAs, mobile phones, wireless phones, pagers, electronic organizers, Internet appliances, television-based systems (e.g., using set-top boxes and / or personal / digital video recorders), and various other consumer products that include appropriate communications capabilities.

[0032] The portable wireless device 106 may also include one or more devices having one or more display panels of various types and formats, and may include various other hardware and / or software components, as appropriate. For example, in the illustrated implementation, the portable wireless device 106 is shown as including a controller 144 (e.g., one or more microcontrollers, processors), one or more displays 146, a data acquisition subsystem 148 that may be used to provide tracking data to the local computing system 104 and is described further below, and a wireless communication subsystem 150 and associated antenna 152 that may be used to facilitate wireless communication with the local computing system 104 or other devices (e.g., one or more game controllers).

[0033] It will be understood that while various items are shown as being stored in memory or storage during use, these items or portions thereof may be transferred between memory and other storage devices for purposes of memory management and data integrity. Thus, in some embodiments, some or all of the described techniques may be performed by hardware means including one or more processors or other configured hardware circuits and / or memory and / or storage, such as when configured by one or more software programs and / or data structures (e.g., by executing software instructions of one or more software programs and / or by storing such software instructions and / or data structures). Some or all of the components, systems, and data structures may be stored (e.g., as software instructions or structured data) in a non-transitory computer-readable storage medium, such as a hard disk or flash drive or other non-volatile storage device, a volatile or non-volatile memory (e.g., RAM), a network storage device, or a removable media product (e.g., DVD disk, CD disk, optical disk, etc.) read by an appropriate drive or via an appropriate connection. The systems, components, and data structures may in some embodiments be transmitted as a generated data signal (e.g., as part of a carrier wave or other analog or digital propagated signal), over a variety of computer-readable transmission media, including wireless-based and wired / cable-based media, and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple separate digital packets or frames). Such computer program products may take other forms in other embodiments. Thus, the invention may be practiced with other computer system configurations.

[0034] The local computing system 104 may also include a tracking subsystem 132 operable to track a predicted future position, orientation, and / or trend of movement of the portable wireless device 106 as it moves about within the tracked volume (e.g., while playing a VR video game). As described further below, the tracking subsystem 132 may receive data from a data acquisition subsystem 148 via a wireless transmission link 108. In at least some implementations, the tracking subsystem 132 or the data acquisition subsystem 148 may utilize one or more tracking base stations 142 fixedly positioned within or proximate to the tracked volume, which facilitate tracking of the portable wireless device 106 within the tracked volume. Various non-limiting examples of implementations of the tracking subsystem 132 are described below.

[0035] The local computing system 104 may also include a radio frequency (RF) transceiver 124 operatively coupled to an electronically steerable antenna array module 126 ("antenna array") that includes a plurality of antenna sub-elements 128. The local computing system 104 also includes a beamforming controller 130 operable to dynamically and selectively adjust the beamforming pattern of the antenna array 126. In at least some implementations, the beamforming controller 130 dynamically adjusts beamforming weights or coefficients 131 to selectively modify at least one of the phase shift and amplitude of each of the plurality of antenna sub-elements to provide constructive / destructive interference to effectively steer or adjust the beamforming pattern of the antenna array 126 in a desired direction. As described further below, the beamforming controller 130 receives tracking data 133 from a tracking subsystem 132 and utilizes the tracking data to compensate for movement of the portable wireless device 106 within the tracked volume by causing the beamforming pattern of the antenna array 126 to track the portable wireless device 106. This reduces the amount of buffering required, thereby reducing delays.

[0036] The RF transceiver 124 (or RF transmitter), beamforming controller 130, and antenna array module 126 may have any type or combination of components to achieve the functions described herein. In general, these components include a phased antenna array or "electronically scanned array," which may be a planar or non-planar array of multiple antenna subelements, where the beam can be electronically steered to point in any direction over a wide angle in front of the array without physically moving the antenna. In operation, current from the transmitter 124 is fed to each component antenna through a phase shifter in the antenna array module 126. The phase shifter is controlled by a controller (e.g., the beamforming controller 130) by varying the individual beamforming weights or coefficients for each of the multiple subelements. By changing the relative phase and amplitude of the fed currents, the beam can be instantly steered in different directions. Multiple antenna sub-elements may be fed from a single transmitter or receiver through phase shifters, and each antenna sub-element may have its own transmitter and / or receiver module controlled by the beamforming controller 130.

[0037] In at least some implementations, the RF transceiver 124 (or RF transmitter) operates in the mmWave band (i.e., 30 GHz to 300 GHz), for example at 60 GHz (e.g., IEEE 802.11ad). The mmWave band is advantageous for extending communication distances at multi-gigabit speeds using transmit and / or receive beamforming. At such mmWaves, the path loss proportional to the carrier frequency is generally too large to allow acceptable communication distances without using beamforming on one or both sides of the wireless link. In current applications, beamforming solutions are optimized for immovable objects such as televisions or monitors. To use beamforming for VR / AR applications, the system may use buffering or other methods that meet the needs of the beamforming controller to reconfigure itself for optimal data throughput as the user moves around within the tracked object volume. In fixed receiver applications, the beamforming controller only needs to compensate for chance variations, and therefore less buffering is required. Advantageously, one or more implementations of the present disclosure use tracking data 133 (e.g., position data, orientation data) from the tracking subsystem 132 to dynamically adjust the beamforming pattern of the antenna array 126 to track the portable wireless device 106 as it moves into the tracked volume.

[0038] 2 illustrates an example environment 200 in which at least some of the described techniques are employed in conjunction with an example HMD device 202 that is coupled via a wireless connection to a video rendering computing system 204 to provide a virtual reality display to a human user 206. In this example, wireless communication functionality may be provided by a separate wireless communication module 212 that is operably coupled to the computing system 204. In other implementations, wireless communication functionality may be provided in the computing system 204 or using two or more computing systems.

[0039] A user wears an HMD device 202 and receives display information of a simulated environment, different from the real physical environment, from a computing system 204 via the HMD device. The computing system acts as an image rendering system that provides images of the simulated environment (such as images generated by game programs and / or other software programs running on the computing system) to the HMD device for display to the user. The user can further move around within the tracked volume 201 of the real physical environment 200 in this example, and may further have one or more I / O ("input / output") devices that allow the user to further interact with the simulated environment, including handheld controllers 208 and 210 in this example.

[0040] The environment 200 may include one or more base stations 214 (two base stations 214a and 214b are shown labeled) that may facilitate tracking of the HMD device 202 and / or the controllers 208 and 210. As the user moves from place to place and / or turns the HMD device 202, the position of the HMD device is tracked so that a corresponding portion of the simulated environment can be displayed to the user on the HMD device, and the controllers 208 and 210 may further utilize similar techniques for use in tracking the position of the controller (and, if necessary, for using this information to help determine and / or confirm the position of the HMD device). After confirming the tracked position of the HMD device 202, corresponding information is wirelessly transmitted to the computing system 204. The computing system 204 uses the tracked position information to generate one or more subsequent images of the simulated environment to be displayed to the user and dynamically steer the beamforming pattern of the antenna array 126.

[0041] There are many different methods of position tracking that may be used in various implementations of the present disclosure, including, but not limited to, acoustic tracking, inertial tracking, magnetic tracking, optical tracking, combinations thereof, and the like.

[0042] In at least some implementations, the HMD device 202 may include one or more optical receivers or sensors that may be used to implement tracking functions or other aspects of the disclosure. For example, each base station 214 may scan an optical signal across the tracked volume 201. Depending on the requirements of each particular implementation, each base station 214 may generate more than one optical signal. For example, while one base station 214 is typically sufficient for six degrees of freedom tracking, multiple base stations (e.g., base stations 214a, 214b) may be necessary or desirable in some embodiments to provide robust room-scale tracking to the HMD device and peripherals. In this example, optical receivers are incorporated into the HMD device 202 and / or other tracked objects (such as controllers 208 and 210). In at least some implementations, these optical receivers may be combined with inertial measurement units ("IMUs") incorporating accelerometers and gyroscopes in each tracked device to support low-latency sensor fusion.

[0043] In at least some implementations, each base station 214 includes two rotors to scan a linear beam across the tracked volume 201 on orthogonal axes. At the start of each scanning cycle, the base station 214 may emit an omnidirectional light pulse (called a "synchronization signal") that is visible to all sensors on the tracked object. Each sensor then calculates its own angular position within the scanned volume by timing the interval between the synchronization signal and the beam signal. Sensor distance and orientation may be determined using multiple sensors fixed to a single rigid body.

[0044] One or more sensors located on the tracked object (e.g., HMD device 202, controllers 208 and 210) may include optoelectronic devices capable of detecting modulated light from the rotor. For visible or near infrared (NIR) light, silicon photodiodes and suitable amplifier / detector circuitry may be used. Because the environment 200 may contain static time-varying signals (optical noise) with wavelengths similar to the base station 214 signals, in at least some implementations, the light from the base station may be modulated in a manner that makes it easier to distinguish from any interfering signals and / or to filter the sensor from any emission wavelengths other than those of the base station signal.

[0045] Inside-out tracking is also a type of position tracking that may be used to track the position of the HMD device 202 and / or other objects (e.g., controllers 208 and 210, tablet computers, smartphones). Inside-out tracking differs from outside-in tracking by the position of a camera or other sensor used to determine the position of the HMD. With inside-out tracking, the camera or sensor is located in the HMD, i.e., the object being tracked. In contrast, with outside-in tracking, the camera or sensor is located in a fixed position in the environment.

[0046] HMDs that use inside-out tracking use one or more cameras that "look out" to determine how the HMD's position changes relative to the environment. As the HMD moves, the sensors recalibrate its location in the room, and the virtual environment responds accordingly in real time. This type of positional tracking can be achieved with or without markers placed in the environment. The cameras placed in the HMD observe features of the surrounding environment. When markers are used, they are designed and placed in specific areas so that they are easily detected by the tracking system. In "marker-less" inside-out tracking, the HMD system uses salient features that are inherently present in the environment (e.g., terrain) to determine position and orientation. The HMD system's algorithms identify specific images or shapes and use the images or shapes to calculate the device's position in space. Data from accelerometers and gyroscopes may also be used to improve the accuracy of positional tracking.

[0047] 3 is a schematic block diagram of an environment 300, in which a wireless data transmission system 302 may be used to transmit data, such as video data, to a portable wireless device 306 using the high-bandwidth and low-latency techniques described herein according to one non-limiting illustrated implementation. As a non-limiting example, the wireless data transmission system 302 may be similar to or identical to the local computing systems 104 and 204 of FIGS. 1 and 2, respectively, and the portable wireless device 306 may be similar to or identical to the portable wireless devices 106 and 202 of FIGS. 1 and 2, respectively. The wireless data transmission system 302 includes an RF transceiver and a steerable antenna array 304, as described above. The portable wireless device 306 includes an RF antenna 308 and associated receiver circuitry.

[0048] In the simplified diagram provided in Figure 3, the portable wireless device 306 is shown in three different positions at times T1 (top of Figure 3), T2 (middle of Figure 3), and T3 (bottom of Figure 3) as the portable wireless device moves through the environment. As the portable wireless device 306 moves through the environment, the wireless data transmission system 302 tracks the movement of the portable wireless device using the techniques described above and dynamically adjusts the beamforming pattern of the antenna array 304 to be directed toward the portable wireless device.

[0049] In the example shown, the RF transceiver of the wireless data transmission system 302 transmits an outbound RF signal containing outbound data via the array antenna 304 at time T1 with a beam 310 directed to the location of the portable wireless device 306 at time T1, transmits a beam 312 at time T2 directed to the location of the portable wireless device 306 at time T2, and transmits a beam 314 at time T3 directed to the location of the portable wireless device 306 at time T3. More generally, the wireless data transmission system 302 dynamically configures the antenna array 304 to provide a number of different beamforming patterns that can be dynamically directed towards the portable wireless device 306 as it moves about in the environment.

[0050] FIG. 4A illustrates a non-limiting example of a planar antenna array 400 that may be used to implement features of the present disclosure. The antenna array 400 includes two rows of two antenna subelements 402 (i.e., a 2×2 array). FIG. 4B illustrates a non-limiting example of a planar antenna array 404 that may be used to implement features of the present disclosure. The antenna array 404 includes two rows of four antenna subelements 406 (i.e., a 2×4 array). It should be understood that the shape, configuration, and number of antenna subelements may be selected based on the specific requirements of a given application. Thus, other antenna subelement configurations such as 2×8, 2×10, 2×12, 8×8, 4×10, non-rectangular shapes, etc. may be used to implement the features of the present disclosure.

[0051] Fig. 5 shows a flow diagram of a method 500 of operating a wireless data transmission system for wirelessly transmitting data with high bandwidth and low latency to a portable wireless device such as an HMD device. The method 500 may be implemented by a device similar to or identical to one of the wireless data transmission systems 104 and 204 of Fig. 1 and Fig. 2, respectively, and the data may be received by a device similar to or identical to one of the portable wireless devices 106 and 202 of Fig. 1 and Fig. 2, respectively.

[0052] At 502, the wireless data transmission system may generate tracking data indicative of the physical location of the portable wireless device within the tracked volume. For example, the wireless data transmission system may receive data acquired by an acquisition unit (e.g., acquisition subsystem 148) of the wireless portable device, which may include data obtained from one or more sensors responsive to signals detected from one or more base stations, as described above. The tracking data may further include orientation data or motion data indicative of the orientation or movement of the portable wireless device within the tracked volume.

[0053] In at least some implementations, the tracking data may include data indicative of the physical position, orientation, or movement of at least one object other than the tracked portable wireless device within the tracked volume, such as one or more controllers, one or more stationary objects (e.g., walls, ceilings, floors), one or more movable objects (parts of the user's body or another person's body), etc. In at least some implementations, the tracking data may include historical data that may be used to confirm past movement or predict future movement of the one or more objects, thereby facilitating improved tracking capabilities. The tracking subsystem may utilize any of a number of types of tracking technologies, including, but not limited to, video tracking, simultaneous localization and mapping (SLAM) tracking, acoustic tracking, inertial tracking, magnetic tracking, optical tracking, combinations thereof, and the like.

[0054] At 504, a beamforming controller of the wireless data transmission system may receive the generated tracking data from the tracking subsystem. As described above, the beamforming controller is operably coupled to a steerable antenna array, such as a phased array antenna. In at least some implementations, the antenna array includes a planar phased array antenna having a plurality of antenna sub-elements coupled to a corresponding plurality of adjustable phase shifters. As an example, as a user wearing a portable wireless device moves around within a tracked volume, such as when playing a VR game, the beamforming controller may repeatedly receive updated tracking data periodically (e.g., every 10 ms, every 20 ms, every 100 ms). The tracking data may include position data, orientation data indicating the orientation of the portable wireless device, motion data indicating past or predicted movement of the portable wireless device, or tracking data associated with other stationary objects (e.g., walls, ceilings, furniture, obstacles) or movable objects (e.g., handheld controller, user's body).

[0055] At 506, the beamforming controller may dynamically adjust a beamforming pattern of the steerable antenna array based at least in part on the received tracking data. As described above, the beamforming controller may dynamically adjust the beamforming pattern of the antenna array by modifying beamforming weights or coefficients that selectively adjust the phase shift and amplitude of each of multiple antenna subelements of the antenna array.

[0056] At 508, the wireless data transmission system may wirelessly transmit data (e.g., video data) to the portable device using an antenna array, with beams of the wireless data transmission system being dynamically directed to the portable wireless device as it moves about within the tracked volume. As mentioned above, in at least some implementations the wireless data transmission system may utilize an RF transmitter operating in the mmWave band (e.g., the 60 GHz band), which is advantageous for utilizing transmit beamforming and / or receive beamforming to increase bandwidth and provide low latency.

[0057] In at least some implementations, the beamforming controller dynamically adjusts the beamforming pattern of the antenna array based at least in part on tracking data indicative of the current or predicted future physical location of at least one other object, such as the location of the controller, a body part of the user, or a stationary or moving obstacle. As an example, the beamforming controller may analyze the tracking data to determine whether the controller is currently blocking or will soon block a straight line path between the antenna array of the wireless data transmission system and the receiver of the mobile device. In response to such detection, the beamforming controller may modify the beamforming pattern in a confirmed or predicted direction to provide a good wireless connection. For example, the beamforming controller may aim the beamforming pattern at a ceiling, floor, or wall to provide a reflected path rather than a direct collimated path. The beamforming may be repeated or scanned using multiple patterns in search of a path that provides a high quality wireless connection. Such a path determined to be suitable or optimal may be stored for use in future similar situations (e.g., whenever the receiver is subsequently blocked).

[0058] In at least some implementations, such alternate paths may be selected based on real-time tracking data and connection quality data (e.g., received signal strength indicator (RSSI)) using an iterative process (e.g., scanning beamforming patterns while monitoring quality data). In at least some implementations, alternate paths may be selected based on historical tracking data and historical quality data, and the system learns over time, using a calibration process or during normal use, which beam pattern is likely to provide the best wireless connection for a particular situation (e.g., receiver obstruction, location within the tracked volume, type of user movement).

[0059] Data other than video data, such as control data, status data, tracking data, or other data that does not require the same bandwidth as video, may be communicated between the wireless data transmission system and the portable wireless device (or other device) using one or more communication channels different from the channel used to transmit the video data. For example, the video data may be transmitted by a first RF transmitter operating in the mmWave band as described above, and the other data may be communicated via one or more other transmitters operating at 2.4 GHz (e.g., IEEE 802.11g, Bluetooth) or 5.0 GHz (e.g., IEEE 802.11ac).

[0060] In at least some implementations, the beam controller may compare the received connection quality data (e.g., RSSI) to a quality limit value. In such implementations, in response to a determination that the quality data does not meet the quality limit value while using the tracking data, the beam controller may dynamically adjust the beamforming pattern of the antenna array based on criteria other than the tracking data. For example, the beam controller may determine that the tracking data does not provide a satisfactory wireless connection for some reason (e.g., because the signal path is blocked) and in response to such a determination, the beam controller may control the beam pattern using the received connection quality data to find a good path. In such a case, the beam controller may iteratively return to beam steering using the tracking data to see if the performance when using the tracking data has improved to a better level than could be achieved using conventional beamforming methods. Thus, in such implementations, conventional beamforming methods may serve as a performance floor, and the tracking-based beamforming methods described herein are used to improve performance if possible.

[0061] Advantageously, by using the tracking data described herein, optimal coefficients for a phased array antenna may be determined much faster than in conventional systems. In conventional "no tracking" systems, a beamforming training process may be performed that includes one or more directional scans to determine the optimal antenna configuration for a transmitter / receiver combination. For example, in some conventional systems, an initial broad scan may be performed to determine one of several directional regions or areas to point the beam. A second fine-tuning may then be performed in the selected area to further optimize the beam direction. Such a process is time consuming and must be repeated whenever a disturbance or significant movement occurs. In the tracking-enabled systems disclosed herein, the coefficients may be calculated directly without the need for such a scan, since the physical location of the receiving system has been ascertained from the tracking data. The system may also store coefficients determined to work well for multiple locations in memory, and may retrieve the appropriate coefficients from memory based on the tracked location of the receiving system. Additionally or alternatively, the system may search a relatively small area (e.g., narrow scan) using the known location of the receiving system, rather than one or more broad scans, thereby significantly reducing the time required to determine the coefficients that result in the optimal antenna configuration. Furthermore, as described above, the system can predict the future location of the receiving system, so that the coefficients can be adapted before problematic jamming or motion events occur. This further reduces the time required to steer the beam toward the receiving system. In addition to improving delay and buffering, selecting good coefficients allows the data link to operate frequently at its high data rate. Unsatisfactory routing forces the transmitter to use a less aggressive modulation scheme, resulting in a lower data rate being used to provide an acceptable error rate.

[0062] In at least some implementations, the system can dynamically renegotiate the buffer length based on the data link quality, which can significantly reduce latency (e.g., to 2 ms, 5 ms). For example, if the system determines that link integrity is high, the system can reduce the size of the buffer (e.g., circular buffer) to reduce system latency. Because the tracking system allows for good route selection that results in high link integrity, as described above, the system may generally utilize small buffers that result in reduced latency. If link integrity is low (e.g., due to environmental conditions or receiver blockage), the buffer size may be dynamically increased until link integrity improves.

[0063] In at least some implementations, the systems and methods provided herein may utilize multiple-input multiple-output (MIMO) techniques to perform beamforming. In such cases, the transmitting system may include multiple antennas (e.g., an antenna array) and the receiving system may include one or more antennas. MIMO utilizes the fact that data transmitted between the transmitting and receiving antennas is subject to filtering from the environment. Signals may be reflected off objects (e.g., walls, furniture), and these reflections will have associated delays, attenuations, and directions of arrival. To utilize multiple paths to provide fast and reliable data transfer, spatial channels between antenna elements may be characterized. This characterization is called channel state information (CSI), which is essentially a collection of spatial transfer functions between each antenna and the receiving system. The CSI may be used to digitally encode and decode data transmitted from and received by the antenna array. The CSI may be updated as necessary to maintain accurate information (e.g., as the receiving system moves through the environment). Thus, in such implementations, the beam controller utilizes MIMO techniques, and the MIMO controller may receive pre-derived CSI having position dependency, and may select one or more of the coefficients, weights, or stream encodings based on the received CSI. Furthermore, because MIMO variables are more difficult to directly compute than non-MIMO beamforming, in at least some implementations, it may be advantageous to store various combinations of coefficients, weights, and / or stream encodings using a memory for lookup, rather than directly computing such variables on the fly.

[0064] The foregoing detailed description has described various implementations of devices and / or processes using block diagrams, schematics, and examples. To the extent that such block diagrams, schematics, and examples include one or more functions and / or operations, those skilled in the art will appreciate that each function and / or operation included in such block diagrams, flow charts, or examples, individually and / or collectively, may be implemented by a wide range of hardware, software, firmware, or substantially any combination thereof. In one implementation, the subject matter may be implemented using application specific integrated circuits (ASICs). However, those skilled in the art will recognize that the implementations disclosed herein may be equivalently implemented, in whole or in part, as one or more computer programs running on one or more computers (e.g., one or more programs running on one or more computer systems), as one or more programs running on one or more controllers (e.g., microcontrollers), as one or more programs running on one or more processors (e.g., microprocessors), as firmware, or as substantially any combination thereof, on standard integrated circuits, and that designing circuitry and / or writing code for the software and / or firmware is well within the skills of those skilled in the art in light of this disclosure.

[0065] Those skilled in the art will recognize that many of the methods or algorithms detailed herein may utilize alternative operations, omit certain operations, and / or perform operations in an order different from that stated.

[0066] Moreover, those skilled in the art will appreciate that the mechanisms taught herein may be distributed in a variety of forms as a program product, and that the exemplary implementations apply equally regardless of the particular type of signal-bearing medium used to actually effect the distribution, including, but not limited to, write-once media such as floppy disks, hard disk drives, CD-ROMs, digital tape, and computer memory.

[0067] The various implementations described above may be combined to produce further implementations. These and other changes may be made to the implementations in light of the above detailed description. In general, in the following claims, the terms used should not be construed to limit the scope of the claims to the specific implementations disclosed in the specification and claims, but should be construed to include all possible implementations along with the full scope of equivalent forms to which such claims are subject. Therefore, the scope of the claims is not limited by this disclosure.

[0068] U.S. Patent Application No. 16 / 258,342, filed January 25, 2019, from which this application claims priority, is hereby incorporated by reference in its entirety.

Claims

1. As the portable wireless device moves through the tracked volume, the physical location of the portable wireless device is tracked in operation, and a physical location of at least one object within the tracked volume other than the portable wireless device being tracked, the physical location including at least one of a handheld controller, a part of a human body, a stationary object, or a physical boundary of the tracked volume; a tracking subsystem that generates tracking data indicative of an antenna array selectively configurable to provide a plurality of beamforming patterns; a radio frequency transmitter (RF transmitter) operably coupled to the antenna array, wherein in operation the RF transmitter wirelessly transmits data to the portable wireless device via the antenna array; and a beamforming controller operatively coupled to the tracking subsystem and the antenna array, wherein in operation the beamforming controller comprises: receiving the generated tracking data from the tracking subsystem; Dynamically adjusting a beamforming pattern of the antenna array based at least in part on the tracking data received from the tracking subsystem. Beamforming Controller and Equipped with the tracking data includes predictive data indicative of a future physical location of the portable wireless device; the beamforming controller analyzes the tracking data to determine whether the at least one object currently blocks or will block a straight line path between the antenna array and the portable wireless device; The beamforming controller dynamically adjusts the beamforming pattern of the antenna array based on a result of the determination.

2. 2. The wireless data transmission system of claim 1, wherein the antenna array comprises a planar phased array antenna including a plurality of antenna subelements.

3. 3. The wireless data transmission system of claim 2, wherein to dynamically adjust the beamforming pattern of the antenna array, the beamforming controller dynamically adjusts beamforming weights of each of the plurality of antenna sub-elements to modify at least one of a phase shift or an amplitude of each of the plurality of antenna sub-elements.

4. The wireless data transmission system of claim 3 , wherein the beamforming controller directly calculates the beamforming weight values ​​for each of the plurality of antenna sub-elements based on the tracking data received from the tracking subsystem.

5. The wireless data transmission system of claim 3 , wherein the beamforming controller retrieves the beamforming weight values ​​for each of the plurality of antenna sub-elements from a memory based on the tracking data received from the tracking subsystem.

6. 6. A wireless data transmission system according to claim 1, wherein the RF transmitter operates in one or more frequency bands between 30 GHz and 300 GHz.

7. 7. The wireless data transmission system according to claim 1, wherein the RF transmitter operates in the 60 GHz frequency band.

8. 8. The wireless data transmission system of claim 1, wherein the tracking data generated by the tracking subsystem further comprises orientation data indicative of an orientation of the portable wireless device within the tracked volume.

9. 9. The wireless data transmission system of claim 1, wherein the beamforming controller dynamically adjusts the beamforming pattern of the antenna array based at least in part on the predictive data indicative of the future physical location of the portable wireless device.

10. The beamforming controller includes: receiving quality data indicative of a quality of a wireless connection between the wireless data transmission system and the portable wireless device; 10. The wireless data transmission system of claim 1, further comprising: dynamically adjusting the beamforming pattern of the antenna array based at least in part on the received quality data.

11. The beamforming controller includes: receiving quality data indicative of a quality of a wireless connection between the wireless data transmission system and the portable wireless device; storing said quality data and corresponding tracking data; Analyzing the stored quality data and the corresponding tracking data; dynamically adjusting the beamforming pattern of the antenna array based at least in part on the analyzed quality data and the corresponding tracking data; 11. The wireless data transmission system according to claim 1, wherein the dynamic adjustment of the beamforming pattern includes selection of one of a plurality of alternative paths, including a reflected path.

12. 12. The wireless data transmission system according to claim 1, wherein the RF transmitter wirelessly transmits video data to the portable wireless device via the antenna array.

13. 13. The wireless data transmission system of claim 12, wherein the RF transmitter is a first RF transmitter, and the wireless data transmission system further comprises a second RF transmitter that, in operation, transmits non-video data using a different RF band than the first RF transmitter.

14. 14. The wireless data transmission system of claim 1, wherein the tracking subsystem implements at least one of video tracking, Simultaneous Localization and Mapping (SLAM) tracking, acoustic tracking, inertial tracking, magnetic tracking, or optical tracking.

15. 15. The wireless data transmission system of claim 1, wherein the beamforming controller dynamically adjusts the beamforming pattern of the antenna array based at least in part on past tracking data and past quality data.

16. The beamforming controller includes: receiving quality data indicative of a quality of a wireless connection between the wireless data transmission system and the portable wireless device; Comparing the received quality data with a quality limit; 16. The wireless data transmission system of claim 1 , further comprising: dynamically adjusting the beamforming pattern of the antenna array based on criteria other than the tracking data in response to determining that the quality data does not meet the quality limit.

17. 17. The wireless data transmission system of claim 16, wherein the other criteria other than the tracking data comprises the received quality data indicative of the quality of the wireless connection.

18. The wireless data transmission system according to claim 1 , wherein the portable wireless device includes a head mounted display (HMD), a tablet computer, a smartphone, or a wearable computer.

19. 19. The wireless data transmission system of claim 1, wherein the beamforming controller comprises a multiple-input multiple-output (MIMO) controller that, in operation, receives pre-derived channel state information having position dependency and selects one or more of coefficients, weights, or stream encodings based at least in part on the received channel state information.

20. a tracking subsystem that, upon operation, generates tracking data indicative of a physical location of the portable wireless device as the portable wireless device moves within a tracked volume; an antenna array selectively configurable to provide a plurality of beamforming patterns; a radio frequency transmitter (RF transmitter) operably coupled to the antenna array, wherein in operation the RF transmitter wirelessly transmits data to the portable wireless device via the antenna array; a beamforming controller operatively coupled to the tracking subsystem and the antenna array, wherein in operation the beamforming controller: receiving the generated tracking data from the tracking subsystem; Dynamically adjusting a beamforming pattern of the antenna array based at least in part on the tracking data received from the tracking subsystem. Beamforming Controller and Equipped with The beamforming controller, in operation, receives pre-derived channel state information having position dependency, and has a multiple-input multiple-output (MIMO) controller that selects one or more of coefficients, weights, or stream encodings based at least in part on the received channel state information.

21. 1. A method of operating a wireless data transmission system, the method comprising: the physical location of the portable wireless device within the tracked volume; and a physical location of at least one object within the tracked volume other than the portable wireless device being tracked, the physical location including at least one of a handheld controller, a part of a human body, a stationary object, or a physical boundary of the tracked volume; generating, by a tracking subsystem, tracking data indicative of a future physical location of the portable wireless device, the tracking data including predictive data indicative of a future physical location of the portable wireless device; receiving the generated tracking data from a tracking subsystem; analyzing the tracking data to determine whether the at least one object currently blocks or will block a direct line path between an antenna array and the portable wireless device; dynamically adjusting a beamforming pattern of the antenna array based on the tracking data and a result of the determining; wirelessly transmitting video data via the antenna array while the portable wireless device is moving within the tracked volume; A method for providing the above.

22. 22. The method of claim 21 , wherein dynamically adjusting the beamforming pattern of the antenna array comprises dynamically adjusting beamforming weights of each of a plurality of antenna sub-elements of the antenna array to modify at least one of a phase shift or an amplitude of each of the plurality of antenna sub-elements.

23. 23. A method according to claim 21 or 22, wherein generating tracking data comprises generating orientation data indicative of an orientation of the portable wireless device within the tracked volume.

24. receiving quality data indicative of a quality of a wireless connection between the wireless data transmission system and the portable wireless device; dynamically adjusting the beamforming pattern of the antenna array based at least in part on the received quality data; 24. The method of any one of claims 21 to 23, further comprising:

25. receiving quality data indicative of a quality of a wireless connection between the wireless data transmission system and the portable wireless device; storing the quality data and corresponding tracking data; analyzing the stored quality data and the corresponding tracking data; dynamically adjusting the beamforming pattern of the antenna array based at least in part on the analyzed quality data and the corresponding tracking data.

25. The method of any one of claims 21 to 24, further comprising:

26. 26. The method of any one of claims 21 to 25, further comprising transmitting non-video data to the portable wireless device using an RF band different from an RF band used to transmit the video data.

27. 27. The method of any one of claims 21 to 26, wherein generating the tracking data comprises implementing at least one of video tracking, Simultaneous Localization and Mapping (SLAM) tracking, acoustic tracking, inertial tracking, magnetic tracking, or optical tracking.

28. receiving quality data indicative of a quality of a wireless connection between the wireless data transmission system and the portable wireless device; comparing the received quality data with a quality limit; dynamically adjusting the beamforming pattern of the antenna array based on the received quality data instead of the received tracking data in response to determining that the quality data is below the quality limit.

28. The method of any one of claims 21 to 27, further comprising:

29. 1. A method of operating a wireless data transmission system, the method comprising: wirelessly transmitting the video data via a steerable phased array antenna to a handheld wireless device located within the tracked volume; the physical location of said portable wireless device at the time of said wireless transmission of video data; and a physical location of at least one object within the tracked volume other than the portable wireless device being tracked, the physical location including at least one of a handheld controller, a part of a human body, a stationary object, or a physical boundary of the tracked volume; to generate tracking data, the tracking data including predictive data indicative of a future physical location of the portable wireless device; analyzing the tracking data to determine whether the at least one object currently blocks or will block a direct line path between an antenna array and the portable wireless device; dynamically adjusting a beamforming pattern of the steerable phased array antenna of the antenna array based on the tracking data and the result of the determination to track movement of the portable wireless device within the tracked volume. A method for providing the above.

30. 30. The method of claim 29, wherein dynamically adjusting a beamforming pattern comprises dynamically adjusting beamforming weights of the steerable phased array antenna.

Images