Wireless data transmission system for head mounted display
Through the coordination of the tracking subsystem and the beamforming controller, the beamforming mode of the antenna array is dynamically adjusted, which solves the stability and quality problems of data transmission when HMD is moved in the VR system, and realizes wireless data transmission with low latency and high bandwidth .
Patent Information
- Application Number
- CN202080010969.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-01-25
- Filing Date
- 2020-01-23
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2040-01-23
AI Technical Summary
Existing wireless data transmission systems are difficult to maintain stable connections when providing a virtual reality (VR) experience with high bandwidth and low latency, especially when users move with a head-mounted display (HMD) and are prone to data loss. and interference, cannot effectively solve the motion to photon (MTP) problem.
The tracking subsystem and beamforming controller are used to cooperate with the antenna array to dynamically adjust the beamforming mode, and the configuration of the antenna array is adjusted in real time based on the tracking data and quality data to ensure the stability and efficiency of data transmission.
By dynamically adjusting the beamforming mode, the buffering amount is reduced, the waiting time is reduced, the stability and quality of data transmission is improved, the MTP problem is solved, and the performance of VR applications is enhanced.
Smart Images

Figure CN113366338B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure generally relates to wireless data transmission systems and methods. Background Art
[0002] The current generation of virtual reality ("VR") experiences are created using a head-mounted display ("HMD"), which can be tethered to a stationary computer (such as a personal computer ("PC"), laptop, or game console), combined and / or integrated with a smartphone and / or its associated display, or standalone. Typically, an HMD is a display device worn on the user's head that has a small display device in front of one eye (monocular HMD) or each eye (binocular HMD). The display unit is typically miniaturized and may include, for example, CRT, LCD, liquid crystal on silicon (LCos), or OLED technology. Binocular HMDs have the potential to display different images to each eye. This capability is used to display stereoscopic images.
[0003] VR systems require large amounts of video data to provide a quality VR experience. In addition, the video data must be provided with low latency (e.g., less than 20ms) to prevent the user from feeling dizzy or nauseous due to the lag between the movement of their head and the corresponding changes in the scene display. This requirement is called "motion to photons" or MTP. Therefore, the amount of data required to provide adequate performance can be very high (e.g., 1Gb / s, 3Gb / s, or higher). Current wireless systems, such as IEEE 802.11ac, struggle to maintain high throughput levels, especially during movement of the HMD worn by the user, or if there are other devices that may cause interference. Therefore, in order to achieve the required throughput, most VR systems today utilize a wired connection between a mobile HMD worn by the user and a fixed computer that sends video data for display by the HMD. Summary of the invention
[0004] The wireless data transmission system can be summarized as including a tracking subsystem, an antenna array, a radio frequency (RF) transmitter, and a beamforming controller, wherein the tracking subsystem generates tracking data indicating the physical location of the mobile wireless device as the mobile wireless device moves in a tracked volume; the antenna array can be selectively configured to provide multiple beamforming modes; the radio frequency (RF) transmitter can be operably connected to the antenna array, and the RF transmitter wirelessly sends data to the mobile wireless device via the antenna array during operation; the beamforming controller can be operably connected to the tracking subsystem and the antenna array, and the beamforming controller receives the generated tracking data from the tracking subsystem during operation, and dynamically adjusts the beamforming mode of the antenna array based at least in part on the tracking data received from the tracking subsystem.
[0005] The antenna array may include a planar phased array antenna including a plurality of antenna sub-elements. In order to dynamically adjust the beamforming pattern of the antenna array, a beamforming controller may dynamically adjust the beamforming weights of each of the plurality of antenna sub-elements to modify at least one of the phase shift or amplitude of each of the plurality of antenna sub-elements. The beamforming controller may directly calculate the value of the beamforming weights of each of the plurality of antenna sub-elements based on the tracking data received from the tracking subsystem. The beamforming controller may obtain the value of the beamforming weights of each of the plurality of antenna sub-elements from a memory based on the tracking data received from the tracking subsystem. The RF transmitter may operate in one or more frequency bands between 30 GHz and 300 GHz. The RF transmitter may operate in a 60 GHz frequency band.
[0006] The tracking data generated by the tracking subsystem may also include orientation data indicating the orientation of the mobile wireless device in the tracked volume. The tracking subsystem may generate tracking data indicating the physical location of at least one other object in addition to the tracked mobile wireless device in the tracked volume. The beamforming controller may dynamically adjust the beamforming pattern of the antenna array based at least in part on the tracking data indicating the physical location of the at least one other object. The at least one other object may include at least one of a handheld controller, a part of a human body, a stationary object, or a physical boundary of the tracked volume. The tracking data may include predicted data indicating the future physical location of the mobile wireless device. The beamforming controller may dynamically adjust the beamforming pattern of the antenna array based at least in part on the predicted data indicating the future physical location of the mobile wireless device. The beamforming controller may receive quality data indicating the quality of the wireless connection between the wireless data transmission system and the mobile wireless device; and dynamically adjust the beamforming pattern of the antenna array based at least in part on the received quality data. The beam controller may receive quality data indicating the quality of the wireless connection between the wireless data transmission system and the mobile wireless device; store the quality data and corresponding tracking data; analyze the stored quality data and corresponding tracking data; and dynamically adjust the beamforming pattern of the antenna array based at least in part on the analyzed quality data and corresponding tracking data. The RF transmitter may wirelessly send video data to the mobile wireless device via the antenna array.
[0007] The RF transmitter is a first RF transmitter, and the wireless data transmission system may further include a second RF transmitter that transmits non-video data via an RF frequency band different from the first RF transmitter in operation. The tracking subsystem may implement at least one of video tracking, simultaneous localization and mapping (SLAM) tracking, acoustic tracking, inertial tracking, magnetic tracking, or optical tracking. The beam controller may dynamically adjust the beamforming pattern of the antenna array based at least in part on historical tracking data and historical quality data. The beam controller may receive quality data indicating the quality of a wireless connection between the wireless data transmission system and the mobile wireless device; compare the received quality data with a quality threshold; and in response to determining that the quality data is below the quality threshold, dynamically adjust the beamforming pattern of the antenna array based on criteria other than the tracking data. Other criteria may include the received quality data indicating the quality of the wireless connection. The mobile wireless device may include a head mounted display (HMD), a tablet computer, a smart phone, or a wearable computer. The beamforming controller may include a multiple-input multiple-output (MIMO) controller that, in operation, receives position-dependent previously derived channel state information and selects one or more of coefficients, weights, or stream coding based at least in part on the received channel state information.
[0008] A method of operating a wireless data transmission system may be summarized as including generating tracking data indicating a physical location of a mobile wireless device in a tracked volume; receiving the generated tracking data from a tracking subsystem; dynamically adjusting a beamforming pattern of an antenna array based on the tracking data; and wirelessly transmitting video data via the antenna array while the mobile wireless device moves in the tracked volume. Dynamically adjusting the beamforming pattern of the antenna array may include dynamically adjusting beamforming weights of each of a plurality of antenna subelements of the antenna array to modify at least one of a phase shift or an amplitude of each of the plurality of antenna subelements. Generating the tracking data may include generating orientation data indicating an orientation of the mobile wireless device in the tracked volume.
[0009] The method may also include receiving quality data indicative of a quality of a wireless connection between the wireless data transmission system and the mobile wireless device; and dynamically adjusting a beamforming pattern of the antenna array based at least in part on the received quality data.
[0010] The method may also include receiving quality data indicating the quality of the wireless connection between the wireless data transmission system and the mobile wireless device; storing the quality data and corresponding tracking data; analyzing the stored quality data and corresponding tracking data; and dynamically adjusting the beamforming pattern of the antenna array based at least in part on the analyzed quality data and corresponding tracking data.
[0011] The method may also include transmitting non-video data to the mobile wireless device via an RF frequency band different from the RF frequency band used to transmit the video data. Generating the tracking data may include implementing at least one of video tracking, simultaneous localization and mapping (SLAM) tracking, acoustic tracking, inertial tracking, magnetic tracking, or optical tracking.
[0012] The method may also include receiving quality data indicating the quality of the wireless connection between the wireless data transmission system and the mobile wireless device; comparing the received quality data to a quality threshold; and in response to determining that the quality data is below the quality threshold, dynamically adjusting the beamforming pattern of the antenna array based on the received quality data rather than the received tracking data.
[0013] A method of operating a wireless data transmission system can be summarized as including wirelessly transmitting video data to a mobile wireless device present in a tracked volume via a steerable phased array antenna; and during the wireless transmission of the video data, iteratively performing the following operations: tracking the physical location of the mobile wireless device; and dynamically adjusting a beamforming pattern of the steerable phased array antenna to track the movement of the mobile wireless device within the tracked volume. Dynamically adjusting the beamforming pattern can include dynamically adjusting beamforming weights of the steerable phased array antenna. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] In the drawings, the same reference numerals represent similar elements or actions. The sizes and relative positions of the elements in the drawings are not necessarily drawn to scale. For example, the shapes of various elements and angles are not necessarily drawn to scale, and some of these elements may be arbitrarily enlarged and positioned to improve the readability of the drawings. In addition, the particular shapes of the drawn elements are not necessarily intended to convey any information about the actual shape of the particular element, and may be selected solely for ease of identification in the drawings.
[0015] Figure 1 is a schematic diagram of a networked environment that includes one or more systems suitable for performing at least some of the techniques described in this disclosure.
[0016] Figure 2 is a diagram showing an example environment in which at least some of the described techniques are used with an example head-mounted display device that wirelessly couples to a local video rendering computing system and provides a virtual reality display to a user.
[0017] Figure 3 is a schematic diagram of an example of a wireless data transmission system using a steerable phased array antenna system for wirelessly communicating with a mobile wireless device, such as an HMD device, according to one non-limiting illustrative embodiment.
[0018] Figure 4A An exemplary planar phased array antenna including four antenna subelements is shown according to one non-limiting illustrated embodiment.
[0019] Figure 4B An exemplary planar phased array antenna is shown according to one non-limiting illustrated embodiment and includes eight antenna subelements arranged in two rows of four subelements.
[0020] Figure 5 is a flow chart of a method of operating a wireless data transmission system to provide data to a mobile wireless device according to one non-limiting illustrative embodiment. DETAILED DESCRIPTION
[0021] In the following description, certain specific details are set forth to provide a thorough understanding of the various embodiments disclosed. However, one skilled in the relevant art will recognize that the embodiments may be practiced without one or more of these specific details, or with other methods, components, materials, etc. In other instances, well-known structures associated with computer systems, server computers, and / or communication networks are not shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.
[0022] Throughout the specification and following claims, the word "comprising" is synonymous with "including" and is inclusive or open-ended (ie, does not exclude other unrecited elements or methodological acts) unless the context requires otherwise.
[0023] Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the phrases "in one embodiment" or "in an embodiment" appearing in various places throughout the specification do not necessarily all refer to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0024] As used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It should also be noted that the term "or" is generally employed in its sense including "and / or" unless the context clearly dictates otherwise.
[0025] The titles and abstracts of the disclosure provided herein are for convenience only and do not interpret the scope or meaning of the embodiments.
[0026] One or more embodiments of the present disclosure relate to systems and methods for providing low latency and high bandwidth wireless data transmission for various applications (e.g., virtual reality, augmented reality, or video applications). In at least some embodiments, a wireless data transmission system is provided, the wireless data transmission system including an electrically steerable antenna, such as a phased array antenna, which is operable to selectively steer its beam based on an electronic control input. The wireless data transmission system may include a tracking subsystem that is operable to track a mobile wireless device (e.g., an HMD, a tablet computer, a smart phone) as the mobile wireless device moves back and forth in a tracked volume. Advantageously, the wireless data transmission system utilizes the known current or predicted future position and / or orientation of a mobile wireless device that receives data (e.g., video data) from the tracking subsystem, and compensates for the movement of the mobile wireless device by selectively adjusting the beamforming pattern of the steerable antenna based at least in part on the tracking data received from the tracking subsystem.
[0027] Additionally, this feature allows for a reduced amount of buffering, which reduces latency - a critical parameter in some applications such as VR applications where MTP is an important industrial metric. Additionally, in at least some embodiments, tracking data from other objects such as one or more handheld controllers may also be used to detect when a user's hand moves in a direction that may occlude a receiver antenna of the mobile wireless device. It has been found, for example, that even partial occlusion of a user's hand may result in significant data loss without relying heavily on buffering, interleaving, or forward error correction.
[0028] Figure 1 is a schematic diagram of a networked environment 100 that includes a local media presentation (LMR) system 102 (e.g., a gaming system, a desktop computing system, a laptop computer) that includes a local computing system 104 and a mobile display device 106 (e.g., an HMD with two display panels, a tablet computer, a laptop computer, a smartphone) that is suitable for performing at least some of the techniques described herein. Figure 1 In the illustrated embodiment, the local computing system 104 is communicatively connected to the mobile display device 106 via a wireless transmission link 108, which will be discussed further below. In at least some embodiments, the local computing system 104 can provide image data for display to a panel display device (not shown) such as a TV, console, or monitor via a wired or wireless link. In various embodiments, as non-limiting examples, the local computing system 104 may include a general computing system, a game console, a video streaming device, a mobile computing device (e.g., a cellular phone, a PDA, or other mobile device), a VR or AR processing device, or other computing system.
[0029] In the illustrated embodiment, the local computing system 104 has components including one or more general purpose hardware processors (e.g., centralized processing units or "CPUs") 110, processor readable storage or memory 112, various I / O ("input / output") hardware components 114 (e.g., keyboard, mouse, one or more game controllers, speakers, microphones, IR transmitters and / or receivers, cameras, biofeedback sensors, etc.), a video subsystem 116 including one or more specialized hardware processors (e.g., graphics processing units or "GPUs") 118 and video memory (VRAM) 120, computer readable memory 150, and one or more wired or wireless network connections 122. In at least some embodiments, the local computing system 104 may implement a unified memory architecture in which a pool of managed memory is shared between one or more CPUs and one or more GPUs.
[0030] In the depicted embodiment, the local computing system 104 is also communicatively connected via one or more computer networks 134 and network link 136 to an exemplary network-accessible media content provider 140 via one or more links 138, which may also provide content for display to the local computing system 104. The media content provider 140 may include one or more computing systems (not shown), each of which may have components similar to those of the local computing system 104, including one or more hardware processors, I / O components, local memory devices, and storage, although some details of the media content server computing systems are not shown for the sake of brevity.
[0031] It should also be understood that the local computing system 104 and the mobile wireless device 106 are merely illustrative and are not intended to limit the scope of the present disclosure. The computing system may alternatively include multiple interactive computing systems or devices, and may be connected to other devices not shown, including through one or more networks such as the Internet, via the World Wide Web, or via a dedicated network (e.g., a mobile communication network, etc.). More generally, a computing system or other computing node may include any combination of hardware or software that can interact and perform the functions of the type described, including but not limited to desktop or other computers, gaming systems, database servers, network memory devices and other network devices, PDAs, cellular phones, wireless phones, pagers, electronic organizers, Internet appliances, television-based systems (e.g., using set-top boxes and / or personal / digital video recorders), and various other consumer products that include appropriate communication capabilities.
[0032] The mobile wireless device 106 may similarly include one or more devices having one or more display panels of various types and forms, and optionally include various other hardware and / or software components. For example, in the illustrated embodiment, the mobile wireless device 106 is shown to include a controller 144 (e.g., one or more microcontrollers, processors), one or more displays 146, a data acquisition subsystem 148 (which may be used to provide tracking data to the local computing system 104, which will be discussed further below), and a wireless communication subsystem 150 and associated antenna 152, which may be used to facilitate wireless communication with the local computing system 104 or other devices (e.g., one or more game controllers).
[0033] It should also be understood that although the entries are shown as being stored in storage or on memory while in use, these entries or portions thereof may be transferred between storage and other memory devices for purposes of memory management and data integrity. Thus, in some embodiments, some or all of the described techniques may be performed by a hardware device including one or more processors or other configured hardware circuits and / or storage and / or memory, such as when configured by one or more software programs and / or data structures (e.g., by executing software instructions of one or more software programs and / or by storing such software instructions and / or data structures). Some or all of the components, systems, and data structures may also be stored (e.g., as software instructions or structured data) on a non-transitory computer-readable memory medium, such as a hard disk or flash drive or other non-transitory memory device, volatile or non-transitory memory (e.g., RAM), a network memory device, or a portable media product to be read by an appropriate drive (e.g., DVD disk, CD disk, optical disk, etc.) or via an appropriate connection. In some embodiments, system components and data structures may also be transmitted as generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) on various computer-readable transmission media, including wireless-based and wire / cable-based media, and may take various forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). In other embodiments, such computer program products may also take other forms. Therefore, the present disclosure may be implemented with other computer system configurations.
[0034] The local computing system 104 may also include a tracking subsystem 132 operable to track the current position, orientation, and / or movement of the predicted future position of the mobile wireless device 106 as the mobile wireless device 106 moves around in the tracked volume (e.g., during the performance of a VR video game). As discussed further below, the tracking subsystem 132 may receive data from the data acquisition subsystem 148 via the wireless transmission link 108. In at least some embodiments, the tracking subsystem 132 or the data acquisition subsystem 148 may utilize one or more tracking base stations 142 that are fixedly positioned in or near the tracked volume to facilitate tracking of the mobile wireless device 106 in the tracked volume. Various non-limiting examples of embodiments of the tracking subsystem 132 are discussed below.
[0035] The local computing system 104 may also include a radio frequency (RF) transceiver 124 operably coupled to an electronically steerable antenna array module 126 ("antenna array") including a plurality of antenna sub-elements 128. The local computing system 104 also includes a beamforming controller 130 operable to dynamically and selectively adjust a beamforming pattern of the antenna array 126. In at least some embodiments, the beamforming controller 130 dynamically adjusts beamforming weights or coefficients 131 to selectively modify at least one of a phase shift and an amplitude of each antenna sub-element to provide constructive / destructive interference, thereby effectively steering or adjusting the beamforming pattern of the antenna array 126 in a desired direction. As further described below, the beamforming controller 130 receives tracking data 133 from the tracking subsystem 132 and utilizes the tracking data to compensate for movement of the mobile wireless device 106 within the tracked volume by causing the beamforming pattern of the antenna array 126 to track the mobile wireless device 106, thereby reducing the amount of buffering required, which reduces latency.
[0036] The RF transceiver 124 (or RF transmitter), the beamforming controller 130, and the antenna array module 126 may include any type and combination of components to implement the functionality discussed in the present disclosure. Typically, the components include a phased antenna array or "electronically scanned array" that may be a planar or non-planar array of antenna sub-elements, where the beam can be electronically steered to point in any direction over a wide angle in front of the array without physically moving the antenna. In operation, current from the transmitter 124 is fed to each component antenna through a phase shifter in the antenna array module 126, which is controlled by a controller (e.g., a beamforming controller 130) by changing the individual beamforming weights or coefficients of each sub-element. By changing the relative phase and amplitude of the feed current, the beam can be instantly directed in different directions. The antenna sub-elements may be fed from a single transmitter or receiver through phase shifters, or each antenna sub-element may have its own transmitter and / or receiver module controlled by the beamforming controller 130.
[0037] In at least some embodiments, the RF transceiver 124 (or RF transmitter) operates in a millimeter wave band (i.e., 30 GHz-300 GHz), such as 60 GHz (e.g., IEEE 802.11ad), and the RF transceiver 124 advantageously utilizes transmit and / or receive beamforming to improve communication range at multiple gigabit speeds. At this millimeter wavelength, the propagation path loss proportional to the carrier frequency is generally so severe that an acceptable communication range cannot be obtained without deploying beamforming on either or both sides of the wireless link. In the present application, the beamforming solver is optimized for non-moving targets, such as televisions or monitors. In order to use beamforming for VR / AR applications, the system may utilize buffering or other methods to accommodate the need for the beamforming controller to reconfigure itself as the user moves around the tracked volume to obtain optimal data throughput. In non-moving receiver applications, the beamforming controller only needs to compensate for occasional interference, which requires less buffering. Advantageously, one or more embodiments of the present disclosure use tracking data 133 (e.g., position data, orientation data) from the tracking subsystem 132 to dynamically adjust the beamforming pattern of the antenna array 126 to track the mobile wireless device 106 as the mobile wireless device 106 moves within the tracked volume.
[0038] Figure 2An exemplary environment 200 is shown in which at least some of the described techniques are used with an exemplary HMD device 202 that is coupled to a video rendering computing system 204 via a wireless connection to provide a virtual reality display to a human user 206. In this example, wireless communication functionality may be provided by a separate wireless communication module 212 that is operably coupled to the computing system 204. In other implementations, wireless communication functionality may be provided within the computing system 204, or provided via two or more computing systems.
[0039] The user wears the HMD device 202 and receives display information of a simulated environment different from the actual physical environment from the computing system 204 via the HMD device, wherein the computing system acts as an image rendering system that provides images of the simulated environment (such as images generated by a game program and / or other software programs executed on the computing system) to the HMD device for display to the user. In this example, the user can also move around within the tracked volume 201 of the actual physical environment 200, and can also have one or more I / O ("input / output") devices to allow the user to further interact with the simulated environment, in this example, the I / O ("input / output") devices include handheld controllers 208 and 210.
[0040] The environment 200 may include one or more base stations 214 (two are shown, labeled base stations 214a and 214b) that may help track the HMD device 202 and / or the controllers 208 and 210. As the user moves the position of the HMD device 202 and / or changes the orientation of the HMD device 202, the position of the HMD device is tracked to allow the corresponding portion of the simulated environment to be displayed to the user on the HMD device, and the controllers 208 and 210 may also use similar techniques for tracking the position of the controllers (and optionally using this information to help determine and / or verify the position of the HMD device). After the tracked position of the HMD device 202 is known, the corresponding information is wirelessly transmitted to the computing system 204, which uses the tracked position information to generate one or more subsequent images of the simulated environment for display to the user and dynamically manipulate the beamforming pattern of the antenna array 126.
[0041] A variety of different position tracking methods may be used in various embodiments of the present disclosure, including but not limited to acoustic tracking, inertial tracking, magnetic tracking, optical tracking, combinations thereof, and the like.
[0042] In at least some embodiments, the HMD device 202 may include one or more optical receivers or sensors that can be used to implement tracking functions or other aspects of the present disclosure. For example, the base stations 214 may each scan an optical signal across the tracked volume 201. Each base station 214 may generate more than one optical signal as required by each specific embodiment. For example, while a single base station 214 is generally sufficient for six-degree-of-freedom tracking, in some embodiments, multiple base stations (e.g., base stations 214a, 214b) may be required or desired to provide robust room-scale tracking for the HMD device and peripherals. In this example, the optical receiver is incorporated into the HMD device 202 and / or other tracked objects, such as the controller 208 and the controller 210. In at least some embodiments, the optical receiver may be paired with an accelerometer and gyroscope inertial measurement unit ("IMU") on each tracked device to support low-latency sensor fusion.
[0043] In at least some embodiments, each base station 214 includes two rotors that scan linear beams across the tracked volume 201 on orthogonal axes. At the beginning of each scanning cycle, the base station 214 can transmit an omnidirectional light pulse (referred to as a "sync signal") that is visible to all sensors on the tracked object. Thus, each sensor calculates a unique angular position in the scanned volume by timing the duration between the sync signal and the beam signal. Sensor distance and orientation can be resolved using multiple sensors fixed to a single rigid body.
[0044] One or more sensors located on the tracked object (e.g., HMD device 202, controllers 208, and 210) may include optoelectronic devices capable of detecting modulated light from the rotor. For visible light or near infrared (NIR) light, a silicon photodiode and appropriate amplifier / detector circuits may be used. Since environment 200 may contain static and time-varying signals (optical noise) having wavelengths similar to those of base station 214 signals, in at least some embodiments, the base station light may be modulated to make it easily distinguishable from any interfering signals, and / or the sensor may be filtered from radiation of any wavelength other than that of the base station signal.
[0045] Inside-out tracking is also a type of position tracking that can be used to track the position of the HMD device 202 and / or other objects (e.g., controllers 208 and 210, tablet computers, smart phones). Inside-out tracking differs from outside-in tracking in the location of the camera or other sensor used to determine the position of the HMD. For inside-out tracking, the camera or sensor is located on the HMD or the object being tracked, while in outside-out tracking, the camera or sensor is placed in a fixed position in the environment.
[0046] An HMD using inside-out tracking utilizes one or more cameras to "look outside" to determine how its position has changed relative to the environment. As the HMD moves, the sensors readjust its position in the room, and the virtual environment responds accordingly in real time. This type of positional tracking can be implemented with or without markers placed in the environment. Cameras placed on the HMD observe features of the surrounding environment. When markers are used, the markers are designed to be easily detected by the tracking system and are placed in specific areas. Using "marker-less" inside-out tracking, the HMD system uses unique features (e.g., natural features) that are initially present in the environment to determine position and orientation. The HMD system's algorithm recognizes specific images or shapes and uses the specific images or shapes to calculate the device's position in space. Data from accelerometers and gyroscopes can also be used to increase the accuracy of positional tracking.
[0047] Figure 3 is a schematic block diagram of an environment 300 in which a wireless data transmission system 302 may be used to transmit data, such as video data, to a mobile wireless device 306 using the high bandwidth and low latency techniques described in the present disclosure, according to a non-limiting illustrated embodiment. As a non-limiting example, the wireless data transmission system 302 may be respectively Figure 1 The local computing system 104 and Figure 2 The local computing system 204 is similar or identical to the local computing system 204, and the mobile wireless device 306 can be respectively Figure 1 The mobile wireless device 106 and Figure 2 The mobile wireless device 306 is similar or identical to the mobile wireless device 202. As described above, the wireless data transmission system 302 includes an RF transceiver and a steerable antenna array 304. The mobile wireless device 306 includes an RF antenna 308 and associated receiver circuitry.
[0048] exist Figure 3 In the simplified illustration provided, the mobile wireless device 306 is at time T1 ( Figure 3 The top of T2( Figure 3 middle) and T3( Figure 3 306 is shown in three different positions to indicate that the mobile wireless device is moving within the environment. As the mobile wireless device 306 moves within the environment, the wireless data transmission system 302 tracks the movement of the mobile wireless device using the techniques discussed above and dynamically adjusts the beamforming pattern of the antenna array 304 to point to the mobile wireless device.
[0049] In the illustrated example, the RF transceiver of the wireless data transmission system 302 transmits an outbound RF signal containing outbound data via the array antenna 304 at time T1 using a beam 310 directed at the location of the mobile wireless device 306 at time T1, transmits a beam 312 directed at the location of the mobile wireless device 306 at time T2 at time T2, and transmits a beam 314 directed at the location of the mobile wireless device 306 at time T3 at time T3. More generally, the wireless data transmission system 302 dynamically configures the antenna array 304 to provide a plurality of different beamforming patterns that can be dynamically directed at the mobile wireless device 306 as the device moves around in an environment.
[0050] Figure 4A A non-limiting example of a planar antenna array 400 that can be used to implement features of the present disclosure is shown. Antenna array 400 includes two rows of two antenna sub-elements 402 (ie, a 2x2 array). Figure 4B A non-limiting example of a planar antenna array 404 that can be used to implement features of the present disclosure is shown. Antenna array 404 includes two rows of four antenna sub-elements 406 (i.e., a 2×4 array). It should be understood that the shape, configuration, and number of antenna sub-elements can be selected based on the specific requirements of a given application. Therefore, other antenna sub-element configurations (such as 2×8, 2×10, 2×12, 8×8, 4×10, non-rectangular shapes, etc.) can be used to implement the functionality of the present disclosure.
[0051] Figure 5 A flow chart of a method 500 of operating a wireless data transmission system to wirelessly transmit data to a mobile wireless device such as an HMD device with high bandwidth and low latency is shown. The method 500 may be performed by respectively Figure 1 The wireless data transmission system 104 and Figure 2 A similar or identical device in the wireless data transmission system 204 is implemented, and the data can be transmitted by respectively Figure 1 The mobile wireless device 106 and Figure 2 The mobile wireless device 202 may be used to receive the signal from a device similar to or identical to the mobile wireless device 202.
[0052] At 502, the wireless data transmission system may generate tracking data indicating the physical location of the mobile wireless device in the tracked volume. For example, the wireless data transmission system may receive data acquired by an acquisition unit (e.g., acquisition subsystem 148) of the wireless mobile device, which may include data obtained from one or more sensors in response to signals detected from one or more base stations, as described above. The tracking data may additionally include orientation data or motion data indicating the orientation or movement of the mobile wireless device in the tracked volume.
[0053] In at least some embodiments, the tracking data may include data indicating the physical position, orientation, or movement of at least one object other than the tracked mobile wireless device (such as one or more controllers, one or more stationary objects (e.g., walls, ceilings, floors), one or more movable objects (part of the body of a user or another person), etc.) in the tracked volume. In at least some embodiments, the tracking data may include historical data that can be used to determine past motion of one or more objects or predict future motion of one or more objects, which can help improve tracking capabilities. The tracking subsystem can utilize any of a variety of types of tracking technologies, including but not limited to video tracking, simultaneous localization and mapping (SLAM) tracking, acoustic tracking, inertial tracking, magnetic tracking, optical tracking, combinations thereof, and the like.
[0054] At 504, a beamforming controller of the wireless data transmission system may receive the generated tracking data from the tracking subsystem. As described above, the beamforming controller may be operably coupled to a steerable antenna array, such as a phased array antenna. In at least some embodiments, the antenna array includes a planar phased array antenna, which includes a plurality of antenna sub-elements coupled to a corresponding plurality of adjustable phase shifters. As an example, when a user wearing a mobile wireless device moves back and forth in a tracked volume (such as when playing a VR game), the beamforming controller may periodically (e.g., every 10ms, every 20ms, every 100ms) repeatedly receive updated tracking data. The tracking data may include position data, directional data indicating the orientation of the mobile wireless device, motion data indicating past movement or predicted movement of the mobile wireless device, or tracking data associated with other stationary objects (e.g., walls, ceilings, furniture, obstacles) or moving objects (e.g., handheld controllers, user's body).
[0055] At 506, the beamforming controller may dynamically adjust a beamforming pattern of the steerable antenna array based at least in part on the received tracking data. As described above, the beamforming controller may dynamically adjust the beamforming pattern of the antenna array by modifying beamforming weights or coefficients that selectively adjust the phase shift and amplitude of each antenna sub-element of the antenna array.
[0056] At 508, the wireless data transmission system may wirelessly transmit data (e.g., video data) to the mobile device using the antenna array, while the beam of the wireless data transmission system is dynamically directed toward the mobile wireless device as the device moves around the tracked volume. As described above, in at least some embodiments, the wireless data transmission system may use an RF transmitter operating in a millimeter wave frequency band (e.g., a 60 GHz frequency band) that advantageously utilizes transmit and / or receive beamforming to improve bandwidth and provide low latency.
[0057] In at least some embodiments, a beamforming controller dynamically adjusts a beamforming pattern of an antenna array based at least in part on tracking data indicating a current physical location or predicted future physical location of at least one other object, such as a location of a controller, a part of a user's body, or a stationary or moving obstacle. As an example, a beamforming controller may analyze tracking data to determine that the controller is currently or about to block a direct path between an antenna array of a wireless data transmission system and a receiver of a mobile device. In response to the detection, the beamforming controller may modify the beamforming pattern to a known direction or a predicted direction to provide a better wireless connection. For example, the beamforming controller may direct the beamforming pattern toward a ceiling, floor, or wall to provide a reflected path rather than a directly observed path. Beamforming may cycle or scan multiple patterns in the search for a path that provides a quality wireless connection. Such a path determined to be suitable or optimal may be stored for future similar situations (e.g., whenever the receiver is subsequently blocked).
[0058] In at least some embodiments, the alternative path can be selected based on real-time tracking data and connection quality data (e.g., received signal strength indicator (RSSI)) via an iterative process (e.g., scanning beamforming modes while monitoring quality data). In at least some embodiments, the alternative path can be selected based on historical tracking data and historical quality data, where the system learns over time, either via a calibration process or during normal use, which beam modes are likely to provide the best wireless connection for a particular environment (e.g., an obscured receiver, location in a tracked volume, type of user movement).
[0059] Data other than video data (such as control data, status data, tracking data, or other data that does not require the same bandwidth as the video) can be communicated between the wireless data transmission system and the mobile wireless device (or other device) using one or more different communication channels that are different from the channel used to send the video data. For example, as described above, the video data can be sent by a first RF transmitter operating in the millimeter wave frequency band, and the other data can be communicated by one or more different transmitters operating at 2.4 GHz (e.g., IEEE 802.11g) or 5.0 GHz (e.g., IEEE 802.11ac).
[0060] In at least some embodiments, the beam controller may compare received connection quality data (e.g., RSSI) to a quality threshold. In this embodiment, in response to determining that the quality data is below the quality threshold while using tracking data, the beam controller may dynamically adjust the beamforming pattern of the antenna array based on criteria other than the tracking data. For example, the beam controller may determine that the use of tracking data provides a poor wireless connection for some reason (e.g., the signal path is blocked), and in response, the received connection quality data may be used to control the beam pattern to find a better path. In this case, the beam controller may iteratively return to using the tracking data for beam steering to determine whether the performance using the tracking data has improved to a level better than the performance that can be achieved using traditional beamforming methods. Therefore, in this embodiment, traditional beamforming methods may serve as a basis for performance while using the tracking-based beamforming methods discussed in the present disclosure to improve performance when possible.
[0061] Advantageously, by using the tracking data discussed in the present disclosure, the optimal coefficients of the phased array antenna can be determined more quickly than in conventional systems. In conventional "no tracking" systems, a beamforming training process including one or more directional scans can be implemented to determine the optimal antenna configuration for the transmitter / receiver pair. For example, in some conventional systems, an initial wide scan can be performed to determine one of several directional areas or sectors in which the beam is guided. Then, a second refinement can be performed in the selected sector to further optimize the direction of the beam. This process is time-consuming and must be repeated whenever there is a disturbance or significant movement. In the tracking system of the present disclosure, the coefficients can be calculated directly without the above-mentioned scan because the physical location of the receiving system can be known from the tracking data. The system can also store coefficients that have been determined to work well for multiple locations in memory, and can retrieve appropriate coefficients from memory based on the tracked location of the receiving system. Additionally or alternatively, without the need for one or more wide scans, the system can use the known location of the receiving system to search a relatively narrow area (e.g., using a narrow scan), which significantly reduces the time required to determine the coefficients that provide the optimal antenna configuration. Additionally, as described above, because the system can predict the future position of the receiving system, the coefficients can be adjusted before problematic obstructions or motion occur, further reducing the time required to steer the beam toward the receiving system. In addition to improvements in latency and buffering, better coefficient selection enables the data link to operate at its higher data rate more often. Poor path selection forces the transmitter to use a more conservative modulation scheme, which results in the use of a lower data rate to provide an acceptable error rate.
[0062] In at least some embodiments, the system can dynamically renegotiate buffer lengths based on data link quality, which can significantly reduce latency (e.g., by 2ms, 5ms). For example, if the system determines that link integrity is high, the system can reduce the size of a buffer (e.g., a circular buffer) to reduce the latency of the system. As described above, because the tracking system allows for better path selection for higher link integrity, on average the system can utilize smaller buffers that provide reduced latency. When link integrity is low (e.g., due to environmental conditions or a blocked receiver), the buffer size can be dynamically increased until link integrity improves.
[0063] In at least some embodiments, the systems and methods provided by the present disclosure may utilize multiple-input multiple-output (MIMO) technology to implement beamforming. In this case, the transmission system may include multiple antennas (e.g., an antenna array), and the receiving system may include one or more antennas. MIMO utilizes the fact that the data transmitted between the transmitting antenna and the receiving antenna is filtered by the environment. Signals may be reflected from objects (e.g., walls, furniture), and these reflections will have associated delays, attenuations, and directions of arrival. In order to utilize multiple paths to provide faster and more reliable data transmission, the spatial channels between antenna elements may be characterized. This characterization is called channel state information (CSI), which is actually a collection of spatial transfer functions between each antenna and the receiving system. CSI can be used to digitally encode and decode data transmitted and received from an antenna array. CSI can be updated as needed to maintain accurate information (e.g., when the receiving system moves in the environment). Therefore, in this embodiment in which the beam controller uses MIMO technology, the MIMO controller may receive previously derived CSI related to the position, and may select one or more of the coefficients, weights, or stream coding based on the received CSI. Additionally, because MIMO variables are more difficult to compute directly than non-MIMO beamforming, in at least some embodiments, it may be advantageous to use memory to store various combinations of coefficients, weights, and / or stream encodings for lookup rather than directly and quickly computing such variables.
[0064] The foregoing detailed description has set forth various embodiments of the device and / or process via the use of block diagrams, schematic diagrams, and examples. Where the block diagrams, schematic diagrams, and examples contain one or more functions and / or operations, it will be understood by those skilled in the art that each function and / or operation within the block diagram, flowchart, or example may be implemented individually and / or collectively by a wide range of hardware, software, firmware, or almost any combination thereof. In one embodiment, the subject matter may be implemented via an application specific integrated circuit (ASIC). However, it will be appreciated by those skilled in the art that the embodiments disclosed herein may be implemented in whole or in part in a standard integrated circuit equivalently as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more controllers (e.g., microcontrollers) (e.g., as one or more programs running on one or more processors), as one or more programs running on one or more processors (e.g., microprocessors), as firmware, or as any combination thereof in practice, and that circuits designed for software and / or firmware and / or writing code for software and / or firmware will be completely within the technical scope of those skilled in the art in accordance with the present disclosure.
[0065] Those skilled in the art will recognize that many of the methods or algorithms set forth herein may employ additional acts, may omit some acts, and / or may perform acts in an order different from that specified.
[0066] Furthermore, those skilled in the art will appreciate that the mechanisms taught herein can be distributed as a program product in a variety of forms, and that the illustrative embodiments apply equally regardless of the particular type of signal-bearing medium used to actually perform the distribution. Examples of signal-bearing media include, but are not limited to, the following: recordable media such as floppy disks, hard drives, CD ROMs, digital tapes, and computer memory.
[0067] The various embodiments described above can be combined to provide further embodiments. Based on the above detailed description, the above and other changes can be made to the embodiments. Generally, in the following claims, the terms used should not be interpreted as limiting the claims to the specific embodiments disclosed in the specification and claims, but should be interpreted as including all possible embodiments and the full range of equivalent expressions authorized by these claims. Therefore, the claims are not limited by this disclosure.
[0068] This application claims priority to U.S. Patent Application No. 16 / 258,342, filed on January 25, 2019, the entire contents of which are incorporated herein by reference.
Claims
1. A wireless data transmission system, comprising: a tracking subsystem that tracks the mobile wireless device as it moves within a tracked volume, the tracking subsystem being operable to generate tracking data indicative of a physical location of at least one other object in the tracked volume other than the tracked mobile wireless device, wherein the at least one other object comprises at least one of: a handheld controller, a portion of a human body, a stationary object, or a physical boundary of the tracked volume; an antenna array selectively configurable to provide a plurality of beamforming patterns; a radio frequency (RF) transmitter operably coupled to the antenna array and operable to wirelessly transmit data to the mobile wireless device via the antenna array; A beamforming controller is operably coupled to the tracking subsystem and the antenna array, and the beamforming controller, in operation: receiving generated tracing data from the tracing subsystem; and Based at least in part on the tracking data indicating the physical location of the at least one other object, it is determined that a direct path between the antenna array and the mobile wireless device is currently or about to be blocked, and a beamforming pattern of the antenna array is dynamically adjusted.
2. The wireless data transmission system according to claim 1, wherein: The antenna array includes a planar phased array antenna including a plurality of antenna sub-elements.
3. The wireless data transmission system according to claim 2, wherein: To dynamically adjust the beamforming pattern of the antenna array, the beamforming controller dynamically adjusts the beamforming weights of each of the plurality of antenna sub-elements to modify at least one of a phase shift or an amplitude of each of the plurality of antenna sub-elements.
4. The wireless data transmission system according to claim 3, wherein: The beamforming controller calculates values of the beamforming weights directly for each of the plurality of antenna subelements based on tracking data received from the tracking subsystem.
5. The wireless data transmission system according to claim 3, wherein: The beamforming controller obtains from memory a value of the beamforming weight for each of the plurality of antenna subelements based on tracking data received from the tracking subsystem.
6. The wireless data transmission system according to claim 1, wherein: The RF transmitter operates at one or more frequency bands between 30 GHz and 300 GHz.
7. The wireless data transmission system according to claim 1, wherein: The RF transmitter operates at the 60 GHz frequency band.
8. The wireless data transmission system according to claim 1, wherein: The tracking data generated by the tracking subsystem also includes orientation data indicating an orientation of the mobile wireless device within the tracked volume.
9. The wireless data transmission system according to claim 1, wherein: The tracking data includes predictive data indicative of a future physical location of the mobile wireless device.
10. The wireless data transmission system according to claim 9, wherein: The beamforming controller dynamically adjusts a beamforming pattern of the antenna array based at least in part on predicted data indicative of a future physical location of the mobile wireless device.
11. The wireless data transmission system according to claim 1, wherein: The beamforming controller: receiving quality data indicative of a quality of a wireless connection between the wireless data transmission system and the mobile wireless device; as well as A beamforming pattern of the antenna array is dynamically adjusted based at least in part on the received quality data.
12. The wireless data transmission system according to claim 1, wherein: The beamforming controller: receiving quality data indicative of a quality of a wireless connection between the wireless data transmission system and the mobile wireless device; storing the quality data and corresponding tracking data; analyzing the stored quality data and the corresponding tracking data; and A beamforming pattern of the antenna array is dynamically adjusted based at least in part on the analyzed quality data and the corresponding tracking data.
13. The wireless data transmission system according to claim 1, wherein: The RF transmitter wirelessly transmits video data to the mobile wireless device via the antenna array.
14. The wireless data transmission system according to claim 13, wherein: The RF transmitter is a first RF transmitter, and the wireless data transmission system further comprises a second RF transmitter operable to transmit non-video data via a different RF frequency band than the first RF transmitter.
15. The wireless data transmission system according to claim 1, wherein: The tracking subsystem implements at least one of video tracking, simultaneous localization and mapping (SLAM) tracking, acoustic tracking, inertial tracking, magnetic tracking, or optical tracking.
16. The wireless data transmission system according to claim 1, wherein: The beamforming controller dynamically adjusts a beamforming pattern of the antenna array based at least in part on historical tracking data and historical quality data.
17. The wireless data transmission system according to claim 1, wherein: The beamforming controller: receiving quality data indicative of a quality of a wireless connection between the wireless data transmission system and the mobile wireless device; comparing the received quality data to a quality threshold; as well as In response to determining that the quality data is below the quality threshold, a beamforming pattern of the antenna array is dynamically adjusted based on criteria other than the tracking data.
18. The wireless data transmission system according to claim 17, wherein: The other criteria include received quality data indicative of a quality of the wireless connection.
19. The wireless data transmission system according to claim 1, wherein: The mobile wireless device includes a head mounted display HMD, a tablet computer, a smart phone or a wearable computer.
20. The wireless data transmission system according to claim 1, wherein: The beamforming controller comprises a multiple-input multiple-output (MIMO) controller operable to receive position-dependent previously derived channel state information and to select one or more of coefficients, weights, or stream coding based at least in part on the received channel state information.
21. A method of operating a wireless data transmission system, the method comprising: generating tracking data indicating a physical location of at least one other object other than the tracked mobile wireless device in the tracked volume, wherein the at least one other object comprises at least one of: a handheld controller, a part of a human body, a stationary object, or a physical boundary of the tracked volume; receiving generated tracing data from a tracing subsystem; determining that a direct path between an antenna array and the mobile wireless device is currently or about to be blocked based at least in part on the tracking data indicating the physical location of the at least one other object, and dynamically adjusting a beamforming pattern of the antenna array; and Video data is wirelessly transmitted via the antenna array while the mobile wireless device moves within the tracked volume.
22. The method according to claim 21, wherein: Dynamically adjusting a beamforming pattern of the antenna array includes dynamically adjusting a beamforming weight of each of a plurality of antenna sub-elements of the antenna array to modify at least one of a phase shift or an amplitude of each of the plurality of antenna sub-elements.
23. The method according to claim 21, wherein: Generating tracking data includes generating orientation data indicative of an orientation of the mobile wireless device in the tracked volume.
24. The method of claim 21, further comprising: receiving quality data indicative of a quality of a wireless connection between the wireless data transmission system and the mobile wireless device; as well as A beamforming pattern of the antenna array is dynamically adjusted based at least in part on the received quality data.
25. The method of claim 21, further comprising: receiving quality data indicative of a quality of a wireless connection between the wireless data transmission system and the mobile wireless device; storing the quality data and corresponding tracking data; analyzing the stored quality data and the corresponding tracking data; and A beamforming pattern of the antenna array is dynamically adjusted based at least in part on the analyzed quality data and the corresponding tracking data.
26. The method of claim 21, further comprising: The non-video data is transmitted to the mobile wireless device via an RF frequency band different from an RF frequency band used to transmit the video data.
27. The method of claim 21, wherein: Generating the tracking data includes implementing at least one of video tracking, simultaneous localization and mapping (SLAM) tracking, acoustic tracking, inertial tracking, magnetic tracking, or optical tracking.
28. The method of claim 21, further comprising: receiving quality data indicative of a quality of a wireless connection between the wireless data transmission system and the mobile wireless device; comparing the received quality data to a quality threshold; as well as In response to determining that the quality data is below the quality threshold, a beamforming pattern of the antenna array is dynamically adjusted based on the received quality data instead of the received tracking data.
29. A method of operating a wireless data transmission system, the method comprising: wirelessly transmitting video data via a steerable phased array antenna to a mobile wireless device present in the tracked volume; as well as The following operations are iteratively performed during the wireless transmission of the video data: Track the physical location of: said mobile wireless device; as well as at least one other object in the tracked volume other than the tracked mobile wireless device, wherein the at least one other object comprises at least one of: a handheld controller, a part of a human body, a stationary object, or a physical boundary of the tracked volume; A beamforming pattern of the steerable phased array antenna is dynamically adjusted to track the position of the at least one other object and to track movement of the mobile wireless device within a tracked volume.
30. The method of operating a wireless data transmission system according to claim 29, wherein: Dynamically adjusting the beamforming pattern includes dynamically adjusting beamforming weights of the steerable phased array antenna.
