Dual Detail Coding in Distributed Systems

Abstract

This specification describes, among other things, techniques for implementing a dual detail encoding scheme in a distributed display system. At a host computer, an application can render a frame of a scene at a first resolution. The host computer can generate a two-dimensional (2D) array of pixels of the frame, encode the 2D array of pixels, and transmit the encoded pixel data to a display device. The 2D array of pixels includes, for each eye, a copy of the frame downscaled to a second resolution lower than the first resolution and a copy of a sub-region of the scene. At the display device, the encoded pixel data is decoded to obtain the 2D array of pixels of the frame, and the 2D array of pixels is modified by upscaling the copy of the frame such that an image can be generated using the upscaled copy of the frame and the copy of the sub-region using blending.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Patent Application No. 18 / 087,431, filed December 22, 2022. U.S. Patent Application No. 18 / 087,431 is incorporated herein by reference in its entirety. [Background technology]

[0002] Wireless streaming technology is used both within and outside the video game industry. In the video game industry, some virtual reality (VR) gaming systems utilize wireless streaming technology to leverage the high computing power of the host computer to run the video game, while also providing wearers of wireless VR headsets with greater mobility compared to tethered (non-wireless) headsets. Despite these advantages, the data rates of existing wireless protocols limit the amount of data that can be transmitted in a relatively short period of time. This presents a challenge for VR gaming systems that operate at high frame rates, especially when playing graphics-intensive video games.

[0003] Technical solutions are provided herein to improve and enhance these and other systems. [Brief explanation of the drawings]

[0004] DETAILED DESCRIPTION OF THE INVENTION The detailed description is described with reference to the accompanying drawings. In the drawings, the leftmost digit(s) of a reference number identifies the figure in which the reference number first appears. Use of the same reference number in different figures indicates similar or identical components or features.

[0005] [Figure 1] FIG. 1 illustrates an exemplary distributed display system including a host computer and a display device in the form of a head-mounted display (HMD), according to embodiments disclosed herein. [Figure 2] 1A-1C illustrate images presented on a display panel of an HMD according to embodiments disclosed herein. [Figure 3A] 1 illustrates an exemplary packing layout for a two-dimensional (2D) array of pixels according to embodiments disclosed herein. [Figure 3B] 10 illustrates another example packing layout for a 2D array of pixels according to embodiments disclosed herein. [Figure 4] 1 shows a flow diagram of an exemplary process for implementing dual detail coding in a distributed display system, according to embodiments disclosed herein. [Figure 5A] 1 illustrates an alternative setup of a system for streaming data from a host computer to a display device according to embodiments disclosed herein. [Figure 5B] 1 illustrates an alternative setup of a system for streaming data from a host computer to a display device according to embodiments disclosed herein. [Figure 5C] 1 illustrates an alternative setup of a system for streaming data from a host computer to a display device according to embodiments disclosed herein. [Figure 6] 1 illustrates exemplary components of a display device, such as an HMD (e.g., a VR headset), and a host computer in which the techniques disclosed herein may be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0006] Described herein are, among other things, techniques, devices, and systems for streaming pixel data from a host computer to a display device using a dual-detail encoding scheme. This dual-detail encoding scheme enables, among other data, pixel data to be transported from a host computer to a display device in a timely manner without compromising the perceived fidelity of the image displayed on the display device. This is particularly beneficial when streaming real-time content, such as video game content. Unlike, for example, pre-recorded content, real-time content cannot be streamed in advance and buffered on the client until the content is ready to be displayed. Instead, frames are rendered milliseconds before the content is displayed based on data received from the display device.

[0007] The following description is not limited to gaming systems, or even VR systems, although many of the examples described herein relate to a VR gaming system including a host computer and an HMD communicatively coupled to the host computer. However, it should be understood that the HMD is merely one example of a “display device” that may be utilized to display streamed content, and other types of display devices are contemplated herein. Furthermore, the techniques described herein may be implemented in wireless and / or wired distributed systems (e.g., a host computer and display devices connected via a cable). In a distributed system including an HMD, the HMD may be worn by a user, possibly for the purpose of immersing the user in a VR or augmented reality (AR) environment. One or more display panels of the HMD are configured to present images based on data generated by an application (e.g., a video game). The application runs on the host computer and generates pixel data for each frame of a series of frames. The pixel data is encoded and transmitted (or transmitted) from the host computer to the HMD to present an image viewed by the user through optics included in the HMD, causing the user to perceive the image as if the user were immersed in the VR or AR environment.

[0008] In some embodiments, the HMD is configured to transmit data to a host computer, which is configured to use the data in various ways, such as generating pixel data for a given frame. For example, the host computer can use head tracking data received from the HMD to generate pose data indicating a predicted pose the HMD will be in at the time that light-emitting elements of the HMD's display panel illuminate for the frame. This pose data can be input into a running application (e.g., a video game) to generate pixel data for the given frame. In some examples, as described in more detail below, eye tracking data received from the HMD can be used to determine subregions of a scene that will ultimately be displayed with higher quality than the rest of the displayed image, such that the user sees more detail in the image, making the image appear sharper. Generally, the HMD can receive encoded pixel data from the host computer and display an image on the HMD's display panel based on the received pixel data. As described in more detail below, the HMD can reconstruct and / or modify the pixel data as necessary before rendering the final image so that the user perceives the content as intended to be perceived consistent with the user's head orientation and line of sight.

[0009] As mentioned above, the limited data rates of existing communication protocols (e.g., wireless protocols) present challenges in distributed display systems, particularly in distributed wireless VR gaming systems while playing graphics-intensive video games. For example, depending on the wireless protocol, typical data rates for transporting data wirelessly from a host computer to a display device can range from 50 to 300 megabits per second (Mbps). Consider an example where the available data rate is on the order of 100 Mbps. In this example, a video game running on a host computer can render frames at a resolution of 2.5k pixels per eye (i.e., an image having 2500 x 2500 pixels), which equates to 12,500,000 pixels across two images, one image per eye. At 3 bytes per pixel, this means that the amount of data transporting the image of a given frame is on the order of 300 megabits. In a 120 Hertz (Hz) VR gaming system, to transmit all of the image data in a timely manner, the host computer must therefore transmit pixel data at a rate of 36,000 Mbps (or 36 Gigabits per second (Gbps)). However, this amount of pixel data cannot be transported fast enough in a distributed system constrained by a 100 Mbps data rate.

[0010] Described herein are techniques, devices, and systems for implementing a dual detail encoding scheme that enables timely transport of pixel data, among other data, from a host computer to a display device without compromising the perceived fidelity of the image displayed on the display device. An exemplary process for streaming pixel data from a host computer to a display device may include rendering a frame for a scene at a first resolution by a graphics rendering application running on the host computer, generating a 2D array of pixels for the frame, encoding the 2D array of pixels to obtain encoded pixel data for the frame, and transmitting the encoded pixel data to the display device. In particular, the 2D array of pixels encoded in the above process provides two levels of detail per eye. That is, the 2D array of pixels comprises, for each eye, a copy of the frame downscaled to a second resolution lower than the first resolution and a copy of a subregion of the scene. Additionally, because the frame includes the subregions, the 2D array of pixels includes, for each eye, two copies of the subregion (one copy of the subregion itself and one included in the downscaled copy of the frame). In some examples, the pixels representing a copy of a subregion of a scene have a one-to-one correspondence with the original pixels of the frame, thereby enabling the image to be presented via a display device, with the subregion rendered with the same or similar quality as was rendered by the application in the original frame. Furthermore, by transmitting copies of the subregion (a subregion having fewer pixels than the full image) and copies of the downscaled frame, the host computer can transmit a reduced amount of pixel data at data rates typically available in distributed display systems, including wireless systems.Thus, the dual detail encoding techniques described herein enable a host computer to transmit a much smaller amount of data (i.e., a 2D array of pixels), and the display device is configured to reconstruct an image from the 2D array of pixels such that a user of the display device perceives the presented image as a high-quality image (e.g., a clear, crisp-looking image, etc.).

[0011] An exemplary process for displaying an image based on pixel data received from a host computer may include receiving, from the host computer, encoded pixel data for a frame for a scene; decoding the encoded pixel data to obtain a 2D array of pixels for the frame; enlarging the first copy of the frame based at least in part on a first pixel of the 2D array of pixels to obtain a first enlarged copy of the frame; and enlarging the second copy of the frame based at least in part on a second pixel of the 2D array of pixels to obtain a second enlarged copy of the frame. The process may continue by generating a first image based at least in part on a first enlarged copy of the frame and a third pixel of the 2D array of pixels representing a first copy of a sub-region of the scene, where a subset of the third pixel at the periphery of the sub-region is blended in the first image, and generating a second image based at least in part on a second enlarged copy of the frame and a fourth pixel of the 2D array of pixels representing a second copy of the sub-region, where a subset of the fourth pixel at the periphery of the sub-region is blended in the second image. The first and second images may then be presented on respective display panels of a display device, such as left and right display panels, or on a single display panel.

[0012] By using the dual detail coding techniques described herein to reduce the amount of data transmitted from a host computer to a display device, a distributed display system can stream pixel data in a timely manner for display of a corresponding image on the display device, and the user perceives the image on the display device as a high-fidelity image. For example, even if portions of the displayed image outside the subregion may be presented at a relatively low quality (e.g., a low resolution), the user's gaze is likely to be directed to the subregion of the scene where the image quality is relatively high, and the image appears to the user as if it were of high quality. In other words, a distributed display system can implement the dual detail coding techniques described herein to avoid transmitting pixel data for areas of a scene that the user is unlikely to see when the corresponding image is presented, but which the user will not perceive as being of lower quality than the originally rendered frame.

[0013] Also disclosed herein are devices, systems, and non-transitory computer-readable media that store computer-executable instructions for implementing the techniques and processes disclosed herein. While the techniques and systems disclosed herein are discussed, by way of example, in the context of video game applications, and specifically VR game applications, it should be understood that the techniques and systems described herein may provide benefits in other applications, including, but not limited to, non-VR applications (e.g., AR applications, mixed reality (MR) applications), and / or non-gaming applications such as industrial machinery applications, defense applications, robotics applications, etc.

[0014] FIG. 1 illustrates an exemplary distributed display system 100 including a host computer 102 and a display device in the form of an HMD 104, according to embodiments disclosed herein. FIG. 1 depicts the HMD 104 being worn by a user 106. FIG. 1 also depicts exemplary implementations of the host computer 102 in the form of a laptop 102(1), for example, carried in a backpack, or a personal computer (PC) 102(N), for example, that may be located in the household of the user 106. However, it should be understood that these exemplary types of host computer 102 are not limiting of the present disclosure. For example, the host computer 102 may be implemented as any type and / or number of computing devices, including, but not limited to, a PC, a laptop computer, a desktop computer, a personal digital assistant (PDA), a mobile phone, a tablet computer, a set-top box, a game console, a portable gaming device, a server computer, a wearable computer (e.g., a smart watch, etc.), or any other electronic device capable of transmitting data to and receiving data from other devices. The host computer 102 can be co-located with the HMD 104, such as in the household of the user 106 wearing the HMD 104. Alternatively, the host computer 102 can be located remotely relative to the HMD 104, such as in the form of a server computer located at a geographic location remote from the geographic location of the HMD 104. In a remote host computer 102 implementation, the host computer 102 can be communicatively coupled to the HMD 104 via a wide area network such as the Internet. In a local host computer 102 implementation, the host computer 102 can be co-located in an environment (e.g., a household) with the HMD 104, whereby the host computer 102 and the HMD 104 can be communicatively coupled to each other either directly or via a local area network (LAN) via an intermediate network device.

[0015] It should also be understood that the exemplary HMD 104 is merely one exemplary type of display device that may be implemented using the techniques and systems described herein. For example, other types and / or numbers of display devices may be utilized in place of or in conjunction with the HMD 104 described in many of the examples herein. Accordingly, the HMD 104 may be more generally referred to herein as a “display device,” with it being understood that other suitable types of display devices may exchange data with the host computer 102. Such display devices may include, but are not limited to, televisions, portable gaming devices with displays, laptop computers, desktop computers, PDAs, mobile phones, tablet computers, wearable computers (e.g., smart watches, etc.), or any other electronic device capable of transmitting / receiving data and displaying images on a display screen or panel.

[0016] 1 , the HMD 104 and the host computer 102 are communicatively coupled together and configured to work cooperatively together to render a given frame and present a corresponding image on the display panel 108 of the HMD 104. This collaboration repeats over a series of frames to render a series of images (e.g., images for a VR game) on the HMD 104. In the illustrated implementation, the HMD 104 includes one or more processors 110 and memory 112 (e.g., computer-readable media 112). In some implementations, the processor 110 may include a central processing unit (CPU), a graphics processing unit (GPU) 114, both a CPU and a GPU 114, a microprocessor, a digital signal processor, or other processing unit or component known in the art. Alternatively, or in addition, the functionality described herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-a-chip (SOCs), etc. Additionally, each of the processors 110 may have its own local memory that may also store program modules, program data, and / or one or more operating systems. In some embodiments, each of the processors 110 may have its own encoder and / or decoder hardware.

[0017] Memory 112 may include volatile and nonvolatile memory, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Such memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc ROM (CD-ROM) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, redundant array of independent disks (RAID) storage systems, or any other medium that can be used to store desired information and that can be accessed by a computing device. Memory 112 may be implemented as a computer-readable storage medium (CRSM), which can be any available physical medium accessible by processor 110 to execute instructions stored on memory 112. In one basic implementation, CRSM may include RAM and flash memory. In other implementations, the CRSM may include, but is not limited to, a ROM, an EEPROM, or any other tangible medium that may be used to store desired information and that may be accessed by the processor 110.

[0018] Generally, the HMD 104 may include logic (e.g., software, hardware, and / or firmware) configured to implement the techniques, functions, and / or operations described herein. The computer-readable medium 112 may include various modules, such as instructions, data stores, etc., that may be configured to execute on the processor 110 to perform the techniques, functions, and / or operations described herein. While an exemplary functional module in the form of a compositor 116 is shown as stored on the computer-readable medium 112 and executable on the processor 110, the same functionality may alternatively be implemented in hardware, firmware, or as a system-on-a-chip (SOC) and / or other logic. Furthermore, additional or different functional modules may be stored on the computer-readable medium 112 and executable on the processor 110. The compositor 116 is configured to modify pixels (or pixel data) to generate an image and output the modified pixel data representing the image to a frame buffer (e.g., a stereo frame buffer) so that the image may be presented on the display panel 108 of the HMD 104. For example, the compositor 116 is configured to reconstruct frames using pixel data received from the host computer 102, as described in more detail below. Additionally, the compositor 116 may apply other corrections to the pixel data before outputting the final pixel data to a frame buffer. For example, the compositor 116 may be configured to perform adjustments for geometric distortion, chromatic aberration, reprojection, etc. At least some of these adjustments may compensate for distortions in the near-to-eye optical subsystem (e.g., lenses and other optics) of the HMD 104, while other adjustments, such as reprojection adjustments, may compensate for slight inaccuracies in the HMD 104's original pose estimation and / or compensate for applications that cannot perform frame rate and / or compensate for late-arriving data packets.For example, reprojected frames may be generated using pixel data from application-rendered frames by transforming the application-rendered frames (e.g., through rotation and reprojection calculations) in a manner that takes into account an updated prediction of the pose of the HMD 104, which may be more accurate than the original pose prediction. Thus, the modified pixel data obtained from applying the reprojection adjustments (and / or other adjustments) may be used to present an image on the display panel 108 of the HMD 104 for a given frame, and this process may be repeated over a series of frames.

[0019] The HMD 104 may further include a head tracking system 118 that generates head tracking data. The head tracking system 118 may utilize one or more sensors (e.g., infrared (IR) light sensors mounted on the HMD 104) and one or more tracking beacons (e.g., IR light emitters co-located in the environment with the HMD 104) to track head movement or translation, including head rotation, of the user 106. This exemplary head tracking system 118 is non-limiting, and other types of head tracking systems 118 (e.g., camera-based, inertial measurement unit (IMU)-based, etc.) may be utilized.

[0020] The HMD 104 may further include an eye tracking system 120 that generates eye tracking data. The eye tracking system 120 may include, without limitation, a camera or other optical sensor within the HMD 104 to capture image data (or information) of the user's eyes, and the eye tracking system 120 can use the captured data / information to determine the 3D position of each eye relative to the HMD 104, including motion vectors, interpupillary distance, interocular distance, magnitude of twist and rotation (i.e., roll, pitch, and yaw), and gaze direction of each eye. In one example, infrared light is emitted within the HMD 104 and reflected from each eye. The reflected light is received or detected by a camera of the eye tracking system 120 and analyzed to extract eye rotation from changes in the infrared light reflected by each eye. Many methods for tracking the eyes of the user 106 may be used by the eye tracking system 120.

[0021] 1 depicts an HMD 104 transmitting data 122 to a host computer 102. This data 122 may include, but is not limited to, head tracking data and / or eye tracking data, as described above. Additionally, the data 122 may be transmitted via a communication interface 124 of the HMD 104 during runtime, when frames are being rendered at the host computer 102 and when corresponding images are being presented at the HMD 104. The communication interface 124 of the HMD 104 may include wired and / or wireless components (e.g., chips, ports, etc.) to facilitate wired and / or wireless data transmission / reception to / from the host computer 102, either directly or through one or more intermediary devices, such as a wireless access point (WAP). For example, the communication interface 124 may include a wireless network interface controller. The communications interface 124 may conform to the Institute of Electrical and Electronics Engineers (IEEE) 502.11 standard and may include a radio (e.g., having one or more antennas) for facilitating wireless connectivity with the host computer 102 and / or other devices and for transmitting and receiving data packets using radio frequency (RF) communications. The communications interface 124 may be incorporated into the HMD 104 or coupled to the HMD 104 (e.g., a wireless adapter, peripheral, accessory, etc.) to adapt the HMD 104 to operate as a 502.11-compliant wireless communications device. In some embodiments, the communications interface 124 may comprise a Universal Serial Bus (USB) wireless adapter (e.g., a USB dongle) that plugs into a USB port on the HMD 104. In other embodiments, the communications interface 124 may be coupled to electrical components of the HMD 104 via a Peripheral Component Interconnect (PCI) interface.It should be appreciated that the communications interface 124 may further include one or more physical ports to facilitate a wired connection (e.g., via a cable) with the host computer 102 and / or another device (e.g., a plug-in network device that communicates with another wireless network).

[0022] Turning to the host computer 102, the host computer 102 is shown as including one or more processors 126 and memory 128 (e.g., computer-readable medium 128). In some implementations, the processor 126 may include a CPU, a GPU 130, both a CPU and a GPU 130, a microprocessor, a digital signal processor, or other processing units or components known in the art. Alternatively, or in addition, the functionality described herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include FPGAs, ASICs, ASSPs, SOCs, etc. Additionally, each of the processors 126 may have its own local memory, which may also store program modules, program data, and / or one or more operating systems. In some embodiments, each of the processors 126 may have its own encoder and / or decoder hardware. Compared to the GPU 114 of the HMD 104, the GPU 130 of the host computer 102 may be a higher-performance GPU with greater computing power. For example, the computing power of the GPU 114 of the HMD 104 may be insufficient to process the graphics of a particular graphics-intensive video game, which is why the system 100 is distributed to leverage the relatively high computing power of the host computer 102 to run the graphics-intensive video game.

[0023] The memory 128 of the host computer 102 may include volatile and nonvolatile memory, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Such memory includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, RAID storage systems, or any other medium that can be used to store desired information and that can be accessed by a computing device. The memory 128 may be implemented as a CRSM, which can be any available physical medium accessible by the processor 126 to execute instructions stored in the memory 128. In one basic implementation, the CRSM may include RAM and flash memory. In other implementations, the CRSM may include, but is not limited to, ROM, EEPROM, or any other tangible medium that can be used to store desired information and that can be accessed by the processor 126.

[0024] Generally, host computer 102 may include logic (e.g., software, hardware, and / or firmware) configured to implement the techniques, functions, and / or operations described herein. Computer-readable medium 128 may include various modules, such as instructions, data stores, etc., that may be configured to execute on processor 126 to perform the techniques, functions, and / or operations described herein. An exemplary functional module in the form of application 132 is illustrated in FIG. 1. Application 132 may represent a graphics-rendering application or a graphics-based application, such as video game 132(1). In some examples, host computer 102 may include a client application or other software (e.g., a video game client) configured to execute one or more applications 132. For example, host computer 102 may have game software installed for playing video game 132(1).

[0025] Generally, the communication interface 134 of the host computer 102 may include wired and / or wireless components (e.g., chips, ports, etc.) to facilitate wired and / or wireless data transmission / reception to / from the HMD 104 (or any other type of display device), either directly or via one or more intermediary devices such as a WAP. In some examples, the communication interface 134 may include a wireless network interface controller. The communication interface 134 may conform to the IEEE 502.11 standard and may include a radio (e.g., having one or more antennas) to facilitate wireless connection with the HMD 104 and / or other devices and to transmit and receive data packets using RF communications. The communication interface 134 may be incorporated into or coupled to the host computer 102 (e.g., a wireless adapter, peripheral, accessory, etc.) to adapt the host computer 102 to operate as a 502.11-compliant wireless communication device. In some embodiments, the communication interface 134 may comprise a USB wireless adapter (e.g., a USB dongle) that plugs into a USB port on the host computer 102. In other embodiments, the communication interface 134 may be coupled to electrical components of the host computer 102 via a PCI interface. It should be appreciated that the communication interface 134 may further include one or more physical ports to facilitate a wired connection (e.g., via a cable) with the HMD 104 and / or another device (e.g., a plug-in network device that communicates with another wireless network).

[0026] In some embodiments, HMD 104 may represent a VR headset for use in a VR system, such as for use with a VR gaming system, in which case video game 132(1) may represent a VR video game 132(1). However, HMD 104 may additionally or alternatively be implemented as an AR headset for use in AR applications, an MR headset for use in MR applications, or a headset usable for non-gaming-related VR, AR, and / or MR applications (e.g., industrial applications). In AR, user 106 sees virtual objects overlaid on a real-world environment, whereas in VR, user 106 typically does not see a real-world environment but is fully immersed in a virtual environment as perceived through the display panel 108 and optics (e.g., lenses) of HMD 104. It should be understood that in some VR systems, a pass-through image of user 106's real-world environment may be displayed in conjunction with the virtual image to create an augmented VR environment within the VR system, whereby the VR environment is augmented with the real-world image (e.g., overlaid on the virtual world). Although the examples described herein relate primarily to a VR-based HMD 104, it should be understood that the HMD 104 is not limited to implementation in VR applications.

[0027] Generally, as described above, the application 132 executing on the host computer 102 may represent a graphics-based application 132 (e.g., a video game 132(1)). The application 132 is configured to generate pixel data for a series of frames, which are ultimately used to present corresponding images on the display panel 108 of the HMD 104. In a system 100 having an HMD 104, components of the host computer 102 may determine a predicted "lighting time" for a frame. This predicted "lighting time" for a frame represents the time that light-emitting elements of the display panel 108 of the HMD 104 will illuminate for the frame. This prediction may take into account, among other things, the estimated time it will take to transmit data over a wireless communications link between the host computer 102 and the HMD 104, as well as the predicted rendering time of the application 132 and / or the known scan-out time of pixels from a frame buffer. In some embodiments, the prediction may be different for a wireless communications link than for a wired communications link. In some embodiments, the illumination time can be predicted as a future amount of time (e.g., approximately 20-40 milliseconds into the future), where the amount of time can vary depending on the connection type (e.g., wired vs. wireless connection).

[0028] The host computer 102 can receive data 122 (e.g., head tracking data, eye tracking data, etc.) from the HMD 104. The data 122 can be generated and / or transmitted at any suitable frequency, such as a frequency corresponding to a target frame rate and / or refresh rate of the HMD 104 or a different (e.g., faster) frequency, such as 360 Hertz (Hz) (or one sensor reading every 2.7 milliseconds). Components of the host computer 102 can determine a predicted pose that the HMD 104 will be in at a predicted illumination time based at least in part on the head tracking data included in the data 122. The pose data indicative of the predicted pose can be provided to an executing application 132 to render a frame (e.g., generate pixel data for the frame) based on the predicted pose, and the application 132 can output pixel data associated with the frame. This pixel data can correspond to an array of pixels on the display panel 108 of the HMD 104. For example, pixel data output by application 132 based on pose data may include a two-dimensional array of per-pixel values ​​(e.g., color values) for an array of pixels on display panel 108 of HMD 104. In an illustrative example, HMD 104 may include a stereo pair of display panels 108, and application 132 may render frames of a scene at a first resolution. For example, the first resolution at which frames for each display panel 108 are rendered may be an array of 2500 x 2500 pixels. In this illustrative example, the pixel data for an individual frame of display panel 108 may include 2500 x 2500 pixel values ​​(or 12,500,000 pixel values ​​for both display panels 108). In some embodiments, the pixel data may include data for each pixel represented by a single set of color and alpha values ​​(e.g., one color value for a red channel, one color value for a green channel, one color value for a blue channel, and one or more values ​​for one or more alpha channels).

[0029] It should be understood that the logic of the host computer 102 may also generate extra data besides (or in addition to) the pixel data generated to present an image on the display panel 108 of the HMD 104, and at least some of this extra data may be transmitted to the HMD 104 as well. In some embodiments, the extra data may be packaged with the pixel data and transmitted to the HMD 104 as data 136. At least some of the extra data packaged in data 136 may be used by the logic of the HMD 104 to present an image corresponding to the frame on the display panel 108 of the HMD 104. The extra data may include, but is not limited to, pose data indicating a predicted pose of the HMD 104, depth data, motion vector data, parallax occlusion data, extra pixel data, and / or audio data. For example, in addition to generating pixel data corresponding to an image to be rendered, the application 132 may also generate depth data (e.g., Z-buffer data) and / or extra pixel data (sometimes referred to herein as “out-of-bounds pixel data” or “additional pixel data”) for the frame. Additionally or alternatively, to aid in processing of the pixel data on the HMD 104, motion vector data based at least in part on the head tracking data received from the HMD 104 may be generated and transmitted to the HMD 104. For example, the motion vector data may be generated based on a comparison of head tracking data generated at two different times (e.g., a comparison of head tracking data separated by several milliseconds). Logic in the HMD 104 may utilize some or all of the redundant data to modify the pixel data to correct for errors in pose predictions previously made in the host computer 102. For example, adjustments made by the HMD 104 may include, but are not limited to, adjustments for geometric distortion, chromatic aberration, reprojection, etc.

[0030] Before the pixel data is transmitted to the HMD 104, the host computer 102 can implement the dual detail encoding techniques disclosed herein, which enable timely transmission of data 136 to the HMD 104 under data rate constraints on the order of 100 Mbps. As depicted in FIG. 1 , the host computer 102 can generate a 2D array of pixels 138 for a frame with a reduced number of pixels compared to the number of pixels of the application-rendered frame (which, as described in the example above, may be approximately 12.5 million pixels). The 2D array of pixels 138 can include first pixels representing a first copy of the frame 140(1) downscaled to a second resolution lower than the first resolution at which the application 132 rendered the frame, and second pixels representing a second copy of the frame 140(2) downscaled to the second resolution. As used herein, "downscale" means to reduce resolution and may also be referred to as "down-resolution" or "down-convert." In an illustrative example, each copy of frame 140(1), 140(2) on each display panel 108 of HMD 104 may be downscaled from a first resolution of, for example, 2500 x 2500 pixels to a second resolution of, for example, 500 x 500 pixels. This is exemplary, and any suitable amount or degree of downscaling may be implemented. 2D array of pixels 138 may further include a third pixel representing a first copy of sub-region 142(1) of the scene and a fourth pixel representing a second copy of sub-region 142(2). A sub-region may be a region of the scene for a given frame.

[0031] In some examples, the subregion may be a predetermined (or fixed) subregion. In VR, the user 106 looks straight ahead most of the time, which means that most of the time the user 106 sees a very small portion of the image presented on a near-eye display, such as the HMD 104. This portion of the image corresponds to a central subregion of the scene, offset from the center by 10 to 30 degrees. In either case, the user 106 rarely, if ever, looks at the periphery of the scene (e.g., the corners and far edges of the image), and in fact, the user's eyes may not be able to see the periphery of the scene outside of the user's eyes' peripheral vision in a near-eye display because the angles are too extreme in the near-eye display. Thus, the subregion of the scene may be predetermined in some examples, and the predetermined subregion may correspond to the central subregion of the scene. That is, in some examples, for each frame that is rendered, 2D array of pixels 138 may include pixels that represent a copy of sub-region 142(1) of the scene that has been predetermined as a central sub-region of the scene, 142(1), so that the sub-region is the same part of the scene for every frame, and for every 2D array of pixels 138 that is generated and transported, but the scene itself is different from frame to frame.

[0032] In some examples, the subregions may be dynamically determined for each frame. For example, the subregions may be determined for each frame based at least in part on eye tracking data generated by the eye tracking system 120 of the HMD 104 and received by the host computer 102 during runtime in data 122. The eye tracking data received by the host computer 102 may indicate a predicted location on the display panel 108 of the HMD 104 where the user 106 will be looking when an image is presented on the display panel 108 based on the encoded pixel data transmitted to the HMD 104 in data 136. In some examples, the host computer 102 makes a prediction of where the user 106 will be looking when an image is presented. In some examples, the eye tracking system 120 makes the prediction and transmits data representing the prediction to the host computer 102. An advantage of the host computer 102 making the prediction is that the prediction may be made closer in time to when the image is presented in the HMD 104 and / or the relatively higher computing power of the host computer 102 may be leveraged to make the prediction. In either case, in examples where the sub-region is determined dynamically (e.g., on the fly) for each frame, the location of the sub-region can change from frame to frame, such that when the user's 106 gaze steers left, the sub-region is selected as the sub-region of the scene that is left of center, or when the user's 106 gaze steers up, the sub-region is selected as the sub-region of the scene that is above the center of the scene. In other words, eye tracking can be used to "steer" a more detailed (e.g., higher resolution) internal target, referred to herein as a "sub-region" of the scene. In this way, the more detailed sub-region can be presented in the image on the display panel 108 at the location where the user 106 is looking, such that the user 106 is looking directly at a more detailed (or higher quality) portion of the image in the scene, while the less detailed (or lower quality) image is viewed through the user's peripheral vision, if at all.

[0033] The copies of subregions 142(1), 142(2) may be any suitable shape. While the example in Figure 1 shows the copies of subregions 142(1), 142(2) as rectangles, the copies of subregions 142(1), 142(2) may be different shapes, such as ovals, squares, squares with rounded corners (e.g., "squircle shapes"), triangles, pentagons, or any other suitable polygons.

[0034] In some examples, the pixels in each copy of subregions 142(1), 142(2) in 2D array 138 may have a one-to-one correspondence with the original pixels of the frame representing the subregion. In other words, each copy of subregions 142(1), 142(2) in 2D array 138 may be an exact copy that has not been resampled (e.g., downscaled) in any way. This means that each pixel in the subregion of the scene for each application-rendered image for a frame may be copied as is into each copy of subregions 142(1), 142(2) in 2D array 138. In some examples, the copies of subregions 142(1), 142(2) in 2D array 138 are downscaled to a reduced resolution (i.e., a lower resolution than the resolution at which application 132 rendered the subregion in a given frame). In these examples, copies of subregions 142(1), 142(2) in 2D array 138 may be downscaled with idealized sampling, such as by downscaling the subregions according to an integer ratio (e.g., 1:2). This idealized sampling of the subregions may enable relatively high-quality resampling when copies of the subregions are enlarged in HMD 104. In other words, downscaling copies of subregions 142(1), 142(2) according to an integer ratio may reduce artifacts (e.g., aliasing, blurring, etc.) in the corresponding portions of the image presented in HMD 104, while downscaling copies of subregions 142(1), 142(2) by 10%, 15%, etc. may cause such artifacts to appear in the resulting image in HMD 104.

[0035] The copies of subregions 142(1), 142(2) can be any suitable size. The tradeoff for making the subregions as large as possible (to obscure relatively low-quality portions of the image outside the subregions) is that more data is required to transport pixels for larger subregions. Therefore, because data rate constraints may vary depending on the implementation, the optimal size for a subregion may vary, and there may be practical limits to increasing the size of a subregion. Furthermore, application 132, in some examples, may change the resolution at which frames are rendered from frame to frame. In other words, application 132 may render a first frame in a series of frames at a first resolution, and application 132 may then render a second frame (e.g., the next frame) in the series at a second resolution that is different from the first resolution. This means that even if the size of a subregion is fixed at M×N pixels, the proportion of the scene (or image) that corresponds to the subregion may change from frame to frame. In the current example, the subregion may be an 800×800 pixel subregion. 1 , 2D array of pixels 138 may be, for example, an array of 800 by 3200 pixels (or 2,560,000 pixel values ​​for both display panels 108). Another example is 2D array 138 of 960 by 3840 pixels (or 3,686,400 pixel values ​​for both display panels 108). In other words, the render target size for 2D array 138 may have an aspect ratio of 4:1. Thus, by generating 2D array of pixels 138, far fewer pixels are ultimately transmitted to HMD 104 compared to the 12,500,000 pixel values ​​of the frame rendered by application 132 at the first resolution.

[0036] The host computer 102 can encode the 2D array of pixels 138 to obtain encoded pixel data for a frame, and the host computer 102 can transmit the encoded pixel data to the HMD 104 as part of the data 136 transmitted to the HMD 104, as shown in Figure 1. In some examples, the host computer 102 can utilize encoder hardware in the GPU 130 to encode the 2D array of pixels 138. Furthermore, the data 136 including the encoded pixel data (and possibly extra data) is transmitted to the HMD 104 during runtime. In this manner, the host computer 102 is configured to encode and transport smaller images, compared to the application-rendered images, that contain redundant data and provide different levels of detail for each eye of the user 106, such that the HMD 104 can reconstruct the images presented on the display panel 108 of the HMD 104 in such a way that the user 106 perceives them as high-fidelity images, even though some of the pixel data for a given frame is sacrificed (e.g., not transmitted), by downscaling copies of frames 140(1), 140(2) in the 2D array of pixels 138, as described above.

[0037] Upon receiving data 136 including the encoded pixel data (and possibly extra data) at the HMD 104, the HMD 104 may decode the encoded pixel data to obtain a 2D array 138 of pixels for a frame. In some examples, the HMD 104 may utilize decoder hardware in the GPU 114 to decode the encoded pixel data received from the host computer 102. The compositor 116 of the HMD 104 may then modify the pixels of the 2D array 138 to reconstruct an image of the frame, as well as to adjust for geometric distortion, chromatic aberration, reprojection, etc.

[0038] 2 illustrates images 200(1) and 200(2) presented on display panels 108(1) and 108(2), respectively, of HMD 104, according to an embodiment disclosed herein. As described above, HMD 104 is configured to decode encoded pixel data received from host computer 102 to obtain 2D arrays of pixels 138 of frames shown in FIG. 1, and compositor 116 modifies the pixels of 2D arrays 138 to reconstruct images 200(1), 200(2) of frames. To reconstruct right image 200(2) (e.g., for right display panel 108(2) of a stereo pair of display panels 108), compositor 116 may upscale a first copy of frame 140(1) based at least in part on corresponding pixels in 2D array 138 to obtain a first upscaled copy of frame 202(1), and may generate right image 200(2) based at least in part on the first upscaled copy of frame 202(1) and pixels in 2D array 138 that represent a first copy of subregion 142(1) of the scene, wherein a subset of pixels in periphery 204(1) of the subregion are blended in right image 200(2). To reconstruct the left of the two images, compositor 116 may upscale a second copy of frame 140(2) based at least in part on corresponding pixels of 2D array 138 to obtain a second upscaled copy of frame 202(2), and may generate left image 200(1) based at least in part on the second upscaled copy of frame 202(2) and the pixels of 2D array 138 representing a second copy of subregion 142(2), wherein a subset of pixels in periphery 204(2) of the subregion are blended in left image 200(1). Left image 200(1) and right image 200(2) may then be presented on respective display panels 108(1) and 108(2) of HMD 104.

[0039] Because the copies of frames 140(1), 140(2) in 2D array of pixels 138 were downscaled, the enlarged copies of frames 202(1), 202(2) may appear blurry in displayed images 200(1), 200(2). In particular, areas of images 200(1), 200(2) outside of respective subregions 142(1), 142(2) will appear blurry if viewed directly, but because the eyes of user 106 are likely to be looking directly at subregions 142(1), 142(2), the relatively lower quality images outside subregions 142(1), 142(2) are likely to go unnoticed due to those portions of the images being within user 106's peripheral vision. As explained above, using eye tracking to dynamically determine sub-regions can reduce the instances in which a user 106 notices a relatively poor quality image outside of sub-regions 142(1), 142(2), but even with predetermined (or fixed) sub-regions, the dual detail coding scheme disclosed herein makes it difficult for an average user 106 to notice any degradation in image quality.

[0040] The blended perimeters 204(1), 204(2) obscure the boundary between the subregions 142(1), 142(2) and the enlarged copies of the frames 202(1), 202(2). That is, without blending the subregion perimeters 204(1), 204(2), the user 106 can perceive a clear boundary (or boundaries) around the subregions 142(1), 142(2) in the presented images 200(1), 200(2). The blending in the perimeters 204(1), 204(2) of the subregions 142(1), 142(2) may include fading between overlapping pixel data in the two areas of each image 200(1), 200(2). For example, blending may involve interpolating between pixel values ​​of subregions 142(1), 142(2) and overlapping pixel values ​​of enlarged copies of frames 202(1), 202(2), where the interpolation is closer to the pixel values ​​of subregions 142(1), 142(2) at points closer to the center of each image 200(1), 200(2), and gradually changing for points farther from the image center such that the interpolation is closer to the pixel values ​​of the enlarged copies of frames 202(1), 202(2) at points farther from the image center. The particular shape of subregions 142(1), 142(2) may result in some blending at peripheries 204(1), 204(2). The end result of the image reconstruction for each image 200 is an image 200 having three areas, including a sharp and crisp sub-region 142, a somewhat blurry area outside of the sub-region 142, and an area between these two areas that are blended to make the image 200 appear smooth (i.e., the user 106 cannot tell that there are three separate areas of the image 200).

[0041] FIG. 3A shows a first example packing layout 300A for 2D array of pixels 138 described above, and FIG. 3B shows a second example packing layout 300B for 2D array of pixels 138. These are merely example packing layouts 300; others may be implemented. In the example of FIG. 3A, packing layout 300A represents a vertical packing layout 300A having pixels representing a first copy of subregion 142(1) of a scene at the top of 2D array of pixels 138, followed by pixels representing a first copy of frame 140(1) (downscaled to a reduced resolution) below the pixels representing the first copy of subregion 142(1), followed by pixels representing a second copy of subregion 140(2) at the bottom of the pixels representing the first copy of frame 142(1), and finally, pixels representing the second copy of frame 140(2) (downscaled to a reduced resolution) at the bottom of 2D array of pixels 138. This vertical packing layout 300A may allow host computer 102 to send packets carrying portions of encoded pixel data while other pixel data is still being encoded. For example, host computer 102 may use the encoder of GPU 130 to encode the pixels of a first copy of subregion 142(1) and begin transmitting the encoded pixel data for the first copy of subregion 142(1) while the pixels representing the first copy of frame 140(1) are being encoded. This may reduce the latency of transmitting the encoded pixel data compared to waiting for the entire 2D array 138 of pixels to be encoded before transmitting the encoded pixel data to HMD 104. Furthermore, it should be understood that the ordering of the four subarrays of 2D array 138 shown in FIG. 3A is exemplary. For example, there may be as many as 24 different vertical packing layouts having 24 different ordering permutations for the four subarrays.

[0042] FIG. 3A also shows that the dimensions of each of the four groups of pixels in 2D array of pixels 138 are M by N pixels. In other words, the first copy of subregion 142(1) is a rectangle having M pixels in the horizontal dimension and N pixels in the vertical dimension. As shown in FIG. 3A, the first copy of frame 140(1) is downscaled to a rectangle having M pixels in the horizontal dimension and N pixels in the vertical dimension. The same is true for the second copy of subregion 142(2) and the second copy of frame 140(2). In some examples, M is equal to N. In some examples, M is different from N. In examples where M is equal to N, M and N may be 500 pixels, although this is just one example dimension that may be implemented. Thus, as described above, 2D array of pixels 138 may be 500 by 3200 pixels, which is far fewer pixels than the application-rendered frame.

[0043] Packing layout 300B shown in FIG. 3B is another possible layout of groups of pixels within 2D array of pixels 138. Exemplary packing layout 300B has pixels representing a first copy of frame 140(1) (downscaled to a reduced resolution) at the top of 2D array of pixels 138, followed by pixels representing a second copy of frame 140(2) (downscaled to a reduced resolution) below the pixels representing the first copy of frame 140(1), with pixels representing the first and second copies of sub-regions 142(1), 142(2) arranged side-by-side at the bottom of 2D array of pixels 138. Sub-arrays 142(1) and 142(2) in packing layout 300B may have M pixels in the horizontal dimension and N pixels in the vertical dimension, similar to those shown in FIG. 3A.

[0044] The processes described herein are illustrated as a collection of blocks in a logical flow graph, which represent a sequence of operations that can be implemented in hardware, software, firmware, or combinations thereof (i.e., logic). In the software context, the blocks represent computer-executable instructions, which, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as limiting, and any some of the described blocks can be combined in any order and / or in parallel to implement a process.

[0045] 4 shows a flow diagram of an exemplary process 400 for implementing dual detail coding in a distributed display system 100, according to an embodiment disclosed herein. For discussion purposes, the process 400 will be described with reference to the previous figure.

[0046] At 402, logic in host computer 102 may execute an application 132 (e.g., a graphics rendering application), such as a video game 132(1) tasked with rendering one frame of a series of frames for the purpose of creating visual video game content to be displayed on a display device (e.g., HMD 104). The following blocks 404-438 of process 400 may represent sub-operations involved in rendering one frame of the series of frames and presenting a corresponding image on display panel 108 of HMD 104.

[0047] At 404, the HMD 104 may transmit data 122 to the host computer 102, which is communicatively coupled to the HMD 104, for use by the host computer 102 to render the frame. In the case of an HMD 104, the data transmitted at block 404 may include head tracking data generated by the head tracking system 118, eye tracking data generated by the eye tracking system 120, and / or other data. In the case of another type of display device, such as a portable gaming device having a display, the data 122 transmitted at block 404 may comprise control input data (e.g., data corresponding to controls (e.g., buttons) pressed, touched, or otherwise manipulated on the gaming device). In the case of a handheld controller, positional controller data and / or auxiliary input data (e.g., touch-sensing data, finger tracking data, etc.) may be transmitted to the host computer 102 at block 404. Positional data for one or more handheld controllers may be predicted forward to the locations where the controllers will be when the frames are displayed to the user 106. In implementations in which the host computer 102 is wirelessly coupled to the HMD 104, the data 122 may be transmitted wirelessly from the HMD 104 to the host computer 102 in block 404. While any suitable wireless communication protocol may be used, in some examples, the 502.11 protocol may be used to transmit the data 122, 136 between the HMD 104 and the host computer 102.

[0048] At 406, the host computer 102 may receive data 122 from a display device (e.g., HMD 104). Although the data 122 may be received in various ways depending on the implementation (e.g., wirelessly, via a wired connection, via a wide area network, etc.), in at least one embodiment, the data may be received wirelessly at block 406 via the communication interface 134 of the host computer 102.

[0049] At 408, the application 132 executing on the host computer 102 can render frames of the scene at a first resolution. In some examples, the frames are rendered based on a predicted illumination time, which represents the time that light-emitting elements of the display panel 108 of the display device (e.g., HMD 104) will illuminate for a given frame. That is, logic in the host computer 102 can determine the time that photons associated with the image presented for a given frame will actually reach the eyes of the user 106. This predicted illumination time is a future time (e.g., 20-40 ms into the future) because it takes time for the application 132 to generate pixel data for the frame and for the pixel data to be encoded and transmitted from the host computer 102 to the HMD 104. It also takes time for the pixel data to be decoded, modified, and scanned out on the HMD 104 before the corresponding image is finally presented on the display panel 108 of the display device (e.g., HMD 104).

[0050] In some examples, the frame is rendered in block 408 based on a predicted pose that the HMD 104 will be in at the predicted illumination time. For example, the head tracking system 118 of the HMD 104 may be configured to track up to six degrees of freedom (e.g., 3D position, roll, pitch, and yaw) of the HMD 104, which may be transmitted in data 122 to the host computer 102 to determine a predicted pose of the HMD 104 (e.g., taking into account predicted head movements that result in a future pose of the HMD 104). Thus, pose data indicative of the predicted pose may be provided to the application 132 for rendering the frame in block 408. For example, the application 132 may call a function to receive the pose data, and in response to the function call, the application 132 may be provided with the requested pose data (predicted for the target illumination time of the given frame and predicted based at least in part on the head tracking data received from the HMD 104) such that the application 132 can render the given frame according to the pose data that corresponds to the virtual camera pose used to render the scene. The rendering at block 408 may involve application 132 generating pixel data for the rendered frame at a first resolution (e.g., 2.5k resolution, as described above). The pixel data generated at block 408 may include pixel values ​​for individual pixels in an array of pixels, as described herein.

[0051] If the subregion of the scene is not predetermined (or fixed) at 410, process 400 may follow a "No" route from block 410 to block 412, where logic in the host computer 102 determines the subregion based at least in part on eye tracking data received from a display device (e.g., HMD 104). This eye tracking data may have been transmitted in data 122 at block 404 and may indicate where the user 106 will look at the predicted illumination time of the frame. Thus, if the user 106 is predicted to look at the center of the display panel 108 of the HMD 104, the subregion may be determined to be a central subregion of the scene (or the center of the scene) at block 412. If the user 106 is predicted to look at the left side of the display panel 108 of the HMD 104, the subregion may be determined to be a subregion to the left of the center in the scene at block 412. If the sub-regions of the scene have been predetermined (ie, following the “Yes” route from block 410 ), or after determining the sub-regions in block 412 , process 400 may proceed to block 414 .

[0052] At 414, logic in host computer 102 may generate 2D array of pixels 138 for the frame. As indicated by sub-blocks 416 and 418, 2D array of pixels 138 may include two copies of sub-regions 142(1), 142(2) of the scene and two copies of frames 140(1), 140(2) downscaled to a second resolution lower than the first resolution at which the frames were rendered in block 408. For example, 2D array of pixels 138 may include first pixels (e.g., a first group / subarray of pixels) representing a first copy of frame 140(1) downscaled to a second, reduced resolution, second pixels (e.g., a second group / subarray of pixels) representing a second copy of frame 140(2) downscaled to a second, reduced resolution, third pixels (e.g., a third group / subarray of pixels) representing a first copy of subregion 142(1) of the scene, and fourth pixels (e.g., a fourth group / subarray of pixels) representing a second copy of subregion 142(2). As described herein, the pixels in 2D array 138 may be arranged in any suitable packing layout, such as packing layout 300A, packing layout 300B, or any other suitable packing layout. In the vertical packing layout 300A, the third pixel is at the top of the 2D array 138, the first pixel is below the third pixel, the fourth pixel is below the first pixel, and the second pixel is at the bottom of the 2D array of pixels, but this is just an example packing layout 300A that may be implemented in block 414.

[0053] As described above, the pixels of each copy of subregions 142(1), 142(2) in 2D array 138 may have a one-to-one correspondence with the original pixels of the frame rendered in block 408. In other words, each copy of subregions 142(1), 142(2) in 2D array 138 may be an exact copy of the subregion in the rendered image of the frame without resampling (e.g., without downscaling) the application-rendered subregion. In examples in which copies of subregions 142(1), 142(2) in 2D array 138 are downscaled, the downscaling of the subregions may be performed using idealized sampling, such as by downscaling the subregions according to an integer ratio (e.g., 1:2), as described above.

[0054] At 420, logic in the host computer 102 may encode the 2D array of pixels 138 to obtain encoded pixel data for the frame. In some embodiments, the encoding and / or decoding may be performed at least partially in hardware (e.g., in silicon embedded in the device's respective GPU and / or CPU). Thus, the host computer 102 may utilize the encoder hardware of the GPU 130 to encode the 2D array of pixels 138 at block 420. In some examples, the encoding performed at block 420 may include compressing and / or serializing data to be transmitted to a display device (e.g., the HMD 104) for purposes of rendering an image associated with the frame. In some embodiments, the encoding performed at block 420 includes reformatting the pixel data. For example, the encoding at block 420 may utilize a video encoding standard such as High Efficiency Video Coding (HEVC) and / or an extension of HEVC, sometimes referred to as H.265 and / or MPEG-H Part 2. While HEVC is one example of a standard that may be utilized in block 420, it should be understood that other suitable video encoding / compression standards, such as H.264 or VP9, ​​may be utilized in block 420. HEVC utilizes motion estimation to compress pixel data with the goal of transmitting less data than uncompressed pixel data in systems where bandwidth constraints exist, but the compressed data is sufficient to approximate the original uncompressed data on the receiving end (e.g., at HMD 104). In some examples, the operations performed in block 414 to generate 2D array 138 of pixels for a frame are performed as part of the encoding operations in block 420.

[0055] In some examples, the encoding in block 420 may include allocating bits to encode each group / subarray of pixels in 2D array 138 uniformly or with variability. For example, the encoder of GPU 130 may allocate fewer bits to encode a first pixel (e.g., a first group / subarray of pixels) representing a first copy of frame 140(1) downscaled to the second reduced resolution and a second pixel (e.g., a second group / subarray of pixels) representing a second copy of frame 140(2) downscaled to the second reduced resolution. Meanwhile, the encoder of GPU 130 may allocate more bits to encode a third pixel (e.g., a third group / subarray of pixels) representing a first copy of subregion 142(1) of the scene and a fourth pixel (e.g., a fourth group / subarray of pixels) representing a second copy of subregion 142(2). In other words, a first number of bits may be allocated to encode the first and second pixels, and a second, greater number of bits may be allocated to encode the third and fourth pixels, such that more data is used to encode the copies of sub-regions 142(1), 142(2) than the copies of downscaled frames 140(1), 140(2). In some examples, logic in host computer 102 may determine a subset of pixels representing the copies of downscaled frames 140(1), 140(2) that will be occluded by pixels of sub-regions 142(1), 142(2) in images 200(1), 200(2), and the encoder may allocate zero bits to the subset of pixels that will be occluded in images 200(1), 200(2) in block 420.

[0056] At 422, the host computer 102 may transmit (or send) the encoded pixel data to a display device (e.g., HMD 104) via the communication interface 134. In some examples, one or more data packets carrying the encoded pixel data are transmitted wirelessly via a transceiver of the host computer 102 at block 422. In some examples, the transmission at block 422 may utilize a transmission data rate of approximately 100 Mbps.

[0057] At 424, the display device (e.g., HMD 104) may receive encoded pixel data for a frame (for a scene) from the host computer 102 via the communication interface 124. In some examples, one or more data packets carrying the encoded pixel data are received wirelessly via a transceiver of the display device (e.g., HMD 104) at block 424. In some examples, the reception at block 424 may utilize a receive data rate of approximately 100 Mbps.

[0058] At 426, logic of the display device (e.g., HMD 104) may decode the encoded pixel data to obtain a 2D array 138 of pixels for the frame. In some embodiments, the display device (e.g., HMD 104) may utilize decoder hardware of the GPU 114 to decode the encoded pixel data at block 426. In some examples, the decoding performed at block 426 may include decompressing and / or deserializing the data for purposes of rendering an image associated with the frame.

[0059] At 428, logic of a display device (e.g., HMD 104), such as compositor 116, may modify pixels in 2D array 138 to obtain modified pixel data. As indicated by sub-blocks 430 and 432, at least some of the modifications in block 428 may be to reconstruct images 200(1), 200(2) of frames. Generally, at block 430, two copies of previously downscaled frames 140(1), 140(2) may be upscaled to obtain two upscaled copies of frames 202(1), 202(2). At 432, images 200(1), 200(2) are generated based on these enlarged copies of frames 202(1), 202(2) and two copies of subregions 142(1), 142(2) with blending at perimeters 204(1), 204(2) of subregions 142(1), 142(2). For example, to reconstruct right image 200(2) (e.g., for right display panel 108(2) of a stereo pair of display panels 108), compositor 116 may, in block 430, upscale a first copy of frame 140(1) based at least in part on corresponding pixels in 2D array 138 to obtain a first upscaled copy of frame 202(1), and, in block 432, generate right image 200(2) based at least in part on the first upscaled copy of frame 202(1) and pixels in 2D array 138 that represent a first copy of subregion 142(1) of the scene, wherein a subset of pixels in periphery 204(1) of the subregion are blended in right image 200(2).To reconstruct the left of the two images, compositor 116 may, in block 430, upscale a second copy of frame 140(2) based at least in part on corresponding pixels in 2D array 138 to obtain a second upscaled copy of frame 202(2), and may generate left image 200(1) based at least in part on the second upscaled copy of frame 202(2) and the pixels in 2D array 138 representing the second copy of subregion 142(2), wherein a subset of pixels in periphery 204(2) of the subregion are blended in left image 200(1).

[0060] As indicated by sub-block 434, other modifications may be performed in block 428, such as modifications to compensate for geometric distortion, chromatic aberration, reprojection, etc. In one example, the other modifications applied to the pixels in block 434 may include reprojection adjustments based at least in part on an updated pose that the HMD 104 will be in at the illumination time of a given frame. For example, logic in the HMD 104 may determine an updated pose that the HMD 104 will be in at the illumination time for a given frame, representing the time that light-emitting elements of the display panel 108 of the HMD 104 will illuminate for the given frame, based at least in part on updated head tracking data generated by the head tracking system 118 of the HMD 104. Because this determination is closer in time to the illumination time of the given frame, the pose prediction by the HMD 104 may be more accurate (e.g., have less error) than a pose prediction determined by the host computer 102 further ahead of the illumination time. Thus, the display device (e.g., HMD 104) can compare the original predicted pose (which was based on the head tracking data received by host computer 102 in block 406) with the updated pose, which can reveal a delta (or difference) between the compared poses, and the reprojection adjustments can include rotation calculations to compensate for this delta (e.g., by shifting and / or rotating pixel data in one way or another, depending on the delta between the two pose determinations). In other words, the head tracking data is used to "correct" the projection of the image rendered on the display panel 108.

[0061] At 436, logic (e.g., compositor 116) of the display device (e.g., HMD 104) can output the modified pixels (or modified pixel data) to a frame buffer of the display device. Because the modified pixels represent images 200(1), 200(2) depicted in FIG. 2 , this is also described herein as outputting images 200(1), 200(2) to a frame buffer. In the case of a display device (e.g., HMD 104) with a pair of display panels 108(1) and 108(2), this modified pixel data may correspond to frames representing the pair of images displayed on the pair of display panels 108(1) and 108(2) and may be output to the stereo frame buffer accordingly.

[0062] In some examples, images 200(1), 200(2) may be output to a frame buffer in a single write operation or multiple write operations. In a single-write implementation, pixel values ​​output to a frame buffer are output in a single pass, and pixel values ​​are not overwritten. This single-write implementation may involve logic (e.g., compositor 116) of a display device (e.g., HMD 104) performing a lookup for each pixel to determine the pixel value to output based on the pixel modifications made in block 428. In a multi-write implementation, compositor 116 may output pixel values ​​corresponding to an enlarged copy of frames 202(1), 202(2) to the frame buffer in a first pass, even if those pixel values ​​are ultimately occluded or blended in the overlap with subregions 142(1), 142(2), and then in a second pass, compositor 116 may output pixel values ​​corresponding to copies of subregions 142(1), 142(2) to the frame buffer with blending at the periphery of subregions 142(1), 142(2), which may overwrite some of the previously written pixel values ​​from the first pass. In some examples, blending is performed in a third write operation (or third pass).

[0063] At 438, logic of the display device (e.g., HMD 104) may cause images 200(1), 200(2) to be presented on respective display panels 108(1), 108(2) of the display device (e.g., HMD 104) based on the pixel values ​​(or pixel data) output to the frame buffer in block 436. This may include scanning the pixel data out to display panels 108(1) and 108(2) of the display device (e.g., HMD 104) and illuminating light-emitting elements of display panels 108(1), 108(2) to illuminate pixels on display panels 108(1), 108(2).

[0064] Thus, process 400 is an exemplary technique for implementing a dual detail coding scheme in distributed display system 100. This dual detail coding scheme reduces the amount of data transmitted from host computer 102 to a display device (e.g., HMD 104) so ​​that pixel data can be streamed in a timely manner for display of a corresponding image on the display device (e.g., HMD 104) and in such a manner that user 106 perceives the image as a high-fidelity image on the display device (e.g., HMD 104). For example, even though portions of displayed images 200(1), 200(2) outside of subregions 142(1), 142(2) may be presented at a relatively lower quality (e.g., lower resolution), user 106's gaze is likely to be directed toward subregions 142(1), 142(2) of the scene where the image quality is relatively higher, and user 106 will see the image as if it were of higher quality.

[0065] An alternative technique for transporting pixel data in distributed display system 100 under data rate constraints is to distort the application-rendered images by enlarging a central portion of each image so that visual information at the center of each image is scaled to a larger size, encoding the distorted image to obtain encoded pixel data, and transmitting the encoded pixel data to a display device (e.g., HMD 104). At the display device (e.g., HMD 104), the encoded pixel data is decoded to obtain a distorted image, and the distortion is reversed in HMD 104 to generate the image presented on display panels 108(1), 108(2). This alternative technique is referred to herein as the “squinch” technique. In contrast to the dual detail encoding technique described herein, the squinch technique provides a single level of detail per eye (as opposed to two levels of detail) in that the 2D array of pixels generated based on the above-described distortion of the application-rendered image contains, for each eye, one copy of the frame distorted by enlarging the central portion of the scene. The application of squinching techniques can result in a 2D array of pixels having fewer pixels than the application-rendered frame, making it suitable for transporting pixel data under data rate constraints of, for example, 100 Mbps. For example, a video game running on the host computer 102 may be rendering frames at a resolution of 2.5k pixels per eye (i.e., images having 2500 x 2500 pixels), which equates to 12,500,000 pixels across two images of an HMD 104 having two display panels 108(1), 108(2), and the squinching techniques described above can result in a 2D array of pixels on the order of 1500 x 1500 pixels per eye, which equates to 4,500,000 pixels being transmitted to the HMD 104 for a given frame.Thus, while squinch solutions can reduce the amount of data transported in distributed display system 100, the dual detail coding scheme described herein can reduce the amount of data to a greater extent. Furthermore, squinch solutions can suffer from resampling artifacts on the display device, such as aliasing, blurring, etc.

[0066] 5A, 5B, and 5C illustrate alternative setups of a system for streaming data from a host computer 102 to a display device according to embodiments disclosed herein. Referring briefly to FIG. 1 , an exemplary implementation is where the host computer 102 is co-located in an environment with a display device in the form of an HMD 104 worn by a user 106. For example, the host computer 102 may be located in the user's 106 home while the user 106 uses the HMD 104 therein, regardless of whether the host computer 102 is located in the same room as the HMD 104 or a different room. Alternatively, the host computer 102 in the form of a mobile computing device (e.g., a tablet or laptop) may be carried (e.g., in a backpack on the user's 106 back), thereby enabling greater mobility. For example, the user 106 may be located in a public park while using such a system.

[0067] 5A shows an alternative implementation in which the host computer 102 represents one or more server computers located at a geographically remote location relative to the HMD 104. In this case, the HMD 104 may be communicatively coupled to the host computer 102 via an access point (AP) 500, such as a wireless AP (WAP), base station, USB wireless dongle, or the like. In an illustrative example, data is exchanged (e.g., streamed) between the host computer 102 and the HMD 104 via the AP 500, such as by streaming data over the Internet. In this implementation, the host computer 102 and / or the HMD 104 may implement one or more of the dual detail encoding techniques described herein.

[0068] 5B shows yet another alternative implementation in which the host computer 102 is communicatively coupled to the HMD 104 via an intermediate computing device 504, such as a laptop or tablet computer. The difference between FIG. 5A and FIG. 5B is that the AP 500 in FIG. 5A may function as a data routing device for a server computer 102 that is remotely located with respect to the HMD 104, while the intermediate computing device 504 in FIG. 5B may function as a data routing device for a host computer 102 that is co-located in the same environment as the HMD 104. The intermediate computing device 504 may even perform part of the rendering workload (e.g., reconstructing pixel data and / or modifying pixel data as described herein) so that the HMD 104 can remain as “lightweight” as possible.

[0069] FIG. 5C illustrates yet another alternative implementation in which the HMD 104 is replaced by another type of display device in the form of a portable gaming device 502 having a display. In the setup of FIG. 5C , as described herein primarily with respect to the HMD 104, the portable gaming device 502 can send control input data to the host computer 102 as the user 106 manipulates the controls of the portable gaming device 502 with their hands, and the host computer 102 can generate pixel data based at least in part on the control input data and transmit the encoded pixel data to the portable gaming device 502. Thus, conventional game streaming systems, which are challenged by adverse environments on most home networks, can benefit from the dual detail encoding techniques and systems described herein. For example, a home personal computer running a gaming platform / application 132 may enable continuous playback of video streams and gameplay interaction with significant data rate constraints. The use of game streaming services with the techniques and systems described herein may enable a higher-quality gameplay experience in environments with limited bandwidth. Furthermore, although the present technology is primarily described herein with reference to VR gaming, some or all of the technology may be equally applicable to 2D / 3D gaming applications.

[0070] FIG. 6 illustrates exemplary components of a wearable device, such as an HMD 104 (e.g., a VR headset), and a host computer 102 in which the techniques disclosed herein may be implemented, according to embodiments disclosed herein. However, it should be understood that the relevant components described with respect to the HMD 104 may be implemented in other types of display devices (e.g., a portable gaming device 502), except that unrelated components may be omitted from those other types of display devices. The HMD 104 may be implemented as a connected device that is communicatively coupled to the host computer 102 during operation and / or as a standalone device. In either mode of operation, the HMD 104 will be worn by the user 106 (e.g., on the head of the user 106). In some embodiments, the HMD 104 may be head-mountable, such as by allowing the user 106 to secure the HMD 104 to their head using a fastening mechanism (e.g., an adjustable band) sized to fit around the head of the user 106. In some embodiments, the HMD 104 comprises a VR, AR, and / or MR headset that includes a near-eye or near-to-eye display. Accordingly, the terms “wearable device,” “wearable electronic device,” “VR headset,” “AR headset,” “MR headset,” and “head-mounted display (HMD)” may be used interchangeably herein to refer to the device 104 of FIG. 6 . However, it should be understood that these types of devices are merely examples of the HMD 104, and that the HMD 104 may be implemented in a variety of other form factors. It should also be understood that some or all of the components shown in FIG. 6 may be implemented on the HMD 104. Accordingly, in some embodiments, a subset of the components shown as implemented in the HMD 104 may be implemented on another computing device separate from the host computer 102 or the HMD 104.

[0071] In the illustrated implementation, the HMD 104 includes the above-mentioned processor 110, which may include one or more GPUs 114, as well as memory 112 that stores a compositor 116 executable by the processor 110, a display panel 108, a head tracking system 118, an eye tracking system 120, and a communication interface 124.

[0072] The HMD 104 may include a single display panel 108 or multiple display panels 108, such as a left display panel 108(1) and a right display panel 108(2) of a stereo pair of display panels. One or more display panels 108 of the HMD 104 may be used to present a series of image frames (referred to herein as “frames”) viewable by a user 106 wearing the HMD 104. It should be understood that the HMD 104 may include any number of display panels 108 (e.g., three or more display panels, a pair of display panels, or a single display panel). Thus, the term “display panel,” as used herein in the singular, may refer to any display panel 108 of a pair of display panels of a two-panel HMD 104, or may refer to a single display panel 108 of an HMD 104 having any number of display panels (e.g., a single-panel HMD 104 or a multi-panel HMD 104). In a two-panel HMD 104, a stereo frame buffer may render pixels on both display panels of the HMD 104. In a single-panel HMD 104, the HMD 104 may include a single display panel 108 and a pair of lenses, one for each eye, for viewing a corresponding image displayed on a portion of the display panel 108.

[0073] The display panel 108 of the HMD 104 may utilize any suitable type of display technology, such as an emissive display that utilizes light-emitting elements (e.g., light-emitting diodes (LEDs)) or laser illumination to emit light during the presentation of frames on the display panel 108. By way of example, the display panel 108 of the HMD 104 may include a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an inorganic light-emitting diode (ILED) display, or any other suitable type of display technology for HMD applications.

[0074] The display panel 108 of the HMD 104 can operate at any suitable refresh rate, such as a 90 Hertz (Hz) refresh rate, a 120 Hz refresh rate, etc., which can be a fixed refresh rate or a variable refresh rate that changes dynamically over a range of refresh rates. The "refresh rate" of a display is the number of times per second that the display redraws the screen. The number of frames displayed per second can be limited by the refresh rate of the display when a fixed refresh rate is used. Thus, a series of frames can be processed (e.g., rendered) and displayed as an image on the display such that a single frame of the series is displayed per screen refresh. That is, to present a series of images on the display panel 108, the display panel 108 can transition from frame to frame in a series of frames at the refresh rate of the display, illuminating pixels per screen refresh.

[0075] The display system of the HMD 104 may implement any suitable type of display driving scheme, such as a global flashing type display driving scheme, a rolling band type display driving scheme, or any other suitable type of display driving scheme. In a global flashing type display driving scheme, an array of light-emitting elements of the display illuminate simultaneously with each screen refresh, thereby flashing globally at the refresh rate. In a rolling band type display driving scheme, individual subsets of the light-emitting elements of the display may be illuminated independently and sequentially in a rolling band of illumination during an illumination period. These types of display driving schemes may be enabled by individually addressable light-emitting elements. Where the array of pixels and the array of light-emitting elements on the display panel 108 are arranged in rows and columns (but not necessarily one light-emitting element per pixel), in a rolling band type display driving scheme, individual rows and / or individual columns of light-emitting elements may be addressed in sequence, and / or individual groups of consecutive rows and / or individual groups of consecutive columns of light-emitting elements may be addressed in sequence.

[0076] Generally, as used herein, "illuminating a pixel" means illuminating a light-emitting element corresponding to that pixel. For example, an LCD illuminates a light-emitting element of a backlight to illuminate a corresponding pixel of the display. Furthermore, as used herein, a "subset of pixels" may include an individual pixel or multiple pixels (e.g., a group of pixels). To drive the display panel 108, the HMD 104 may include, among other things, a display controller, a display driver circuit, and similar electronics for driving the display panel 108. The display driver circuit may be coupled to the array of light-emitting elements of the display panel 108 via conductive paths, such as metal traces on a flexible printed circuit. In one example, the display controller may be communicatively coupled to the display driver circuit and configured to provide signals, information, and / or data to the display driver circuit. The signals, information, and / or data received by the display driver circuit may cause the display driver circuit to illuminate the light-emitting elements in a particular manner. That is, the display controller can determine which light-emitting elements to illuminate, when to illuminate the elements, and the level of light output to be emitted by the light-emitting elements, and can communicate appropriate signals, information, and / or data to the display driver circuitry to accomplish that goal.

[0077] The computer-readable medium 112 may store additional functional modules executable on the processor 110, although the same functionality may alternatively be implemented in hardware, firmware, or as a SOC and / or other logic. For example, the operating system module 600 may be configured to manage the hardware in and coupled to the HMD 104 for the benefit of other modules. Additionally, in some instances, the HMD 104 may include one or more applications 602 stored in the memory 112 or otherwise accessible to the HMD 104. For example, the applications 602 may include, but are not limited to, a video game application (e.g., a basic video game with graphics that are not very computationally intensive to process), a video playback application (e.g., an application that accesses a video content library stored on the HMD 104 and / or in the cloud), etc. The HMD 104 may include any number or type of applications 602 and is not limited to the specific examples described herein.

[0078] Generally, the HMD 104 has input devices 604 and output devices 606. The input devices 604 may include control buttons. In some implementations, one or more microphones may function as the input devices 604 to receive audio input, such as user voice input. In some implementations, one or more cameras or other types of sensors (e.g., inertial measurement units (IMUs)) may function as the input devices 604 to receive gestural input, such as hand and / or head movements of the user 106. In some embodiments, additional input devices 604 may be provided in the form of a keyboard, keypad, mouse, touchscreen, joystick, etc. In other embodiments, the HMD 104 may omit a keyboard, keypad, or other similar form of mechanical input. Instead, the HMD 104 may be implemented with a relatively simple configuration of input devices 604, a network interface (wireless or wire-based), power, and processing / memory capabilities. For example, a limited set of one or more input components (e.g., dedicated buttons for initiating configuration, power on / off, etc.) may be employed so that the HMD 104 can be subsequently used. In one implementation, the input device 604 may include controls such as basic volume control buttons for increasing / decreasing the volume, as well as power and reset buttons.

[0079] The output device(s) 606 may include a display panel 108, which may include one or more display panels 108 (e.g., a stereo pair of display panels 108), as described herein. The output device(s) 606 may further include, without limitation, light elements (e.g., LEDs), vibrators for producing tactile sensations, speakers (e.g., headphones), etc. There may also be simple light elements (e.g., LEDs) to indicate a status, such as when the power is on.

[0080] The HMD 104 may further include a communication interface 124 including one or more antennas 910 (e.g., of a transceiver) to facilitate a wireless connection to a network and / or a second device, such as, but not limited to, the host computer 102, as described herein. The communication interface 124 may implement one or more of a variety of wireless technologies, such as Wi-Fi, Bluetooth, radio frequency (RF), etc. It should be appreciated that the communication interface 124 of the HMD 104 may further include a physical port to facilitate a wired connection to a network and / or a second device, such as the host computer 102.

[0081] The HMD 104 may further include an optical subsystem 612 that directs light from the display panel 108 to the user's eyes using one or more optical elements. The optical subsystem 612 may include various types and combinations of different optical elements, including, but not limited to, apertures, lenses (e.g., Fresnel lenses, convex lenses, concave lenses, etc.), filters, etc. In some embodiments, one or more optical elements in the optical subsystem 612 may have one or more coatings, such as anti-reflective coatings. The magnification of image light by the optical subsystem 612 may allow the display panel 108 to be physically smaller, lighter, and consume less power than larger displays. Additionally, the magnification of image light may increase the field of view (FOV) of the displayed content (e.g., images). For example, the FOV of the displayed content may be such that the displayed content is presented using nearly all (e.g., 120-150 degrees diagonal), or in some cases all, of the user's FOV. AR applications may have a narrower FOV (e.g., an FOV of approximately 40 degrees). The optical subsystem 612 may be designed to correct one or more optical errors, such as, but not limited to, barrel distortion, pincushion distortion, longitudinal chromatic aberration, lateral aberration, spherical aberration, coma, field curvature, astigmatism, etc. In some embodiments, content provided to the display panel 108 for display is pre-distorted (e.g., by applied geometric distortion adjustments and / or chromatic aberration adjustments described herein), and the optical subsystem 612 corrects the distortion when receiving image light from the display panel 108 generated based on the content.

[0082] The HMD 104 may further include one or more sensors 614, such as sensors used to generate movement, position, and orientation data. These sensors 614 may be or include gyroscopes, accelerometers, magnetometers, video cameras, color sensors, or other movement, position, and orientation sensors. The sensors 614 may also include a sub-portion of sensors, such as a series of active or passive markers that can be viewed externally by a camera or color sensor to generate movement, position, and orientation data. For example, a VR headset may include multiple markers on its exterior, such as reflectors or lights (e.g., infrared or visible light), that, when viewed by an external camera or illuminated with light (e.g., infrared or visible light), may provide one or more reference points for interpretation by software to generate movement, position, and orientation data. The HMD 104 may include a light sensor sensitive to light (e.g., infrared or visible light) projected or broadcast by a base station in the HMD 104's environment.

[0083] In one example, the sensors 614 may include an inertial measurement unit (IMU) 616. The IMU 616 may be an electronic device that generates calibration data based on measurement signals received from accelerometers, gyroscopes, magnetometers, and / or other sensors suitable for detecting motion and correcting errors associated with the IMU 616, or some combination thereof. Based on the measurement signals, such motion-based sensors, such as the IMU 616, may generate calibration data that indicates an estimated position of the HMD 104 relative to an initial position of the HMD 104. For example, multiple accelerometers may measure translational motion (forward / backward, up / down, left / right), and multiple gyroscopes may measure rotational motion (e.g., pitch, yaw, and roll). The IMU 616 may, for example, rapidly sample the measurement signals and calculate an estimated position of the HMD 104 from the sampled data. For example, the IMU 616 may integrate measurement signals received from an accelerometer over time to estimate a velocity vector, and integrate the velocity vector over time to determine an estimated position of a reference point on the HMD 104. A reference point is a point that can be used to describe the position of the HMD 104. While a reference point may generally be defined as a point in space, in various embodiments, the reference point is defined as a point within the HMD 104 (e.g., the center of the IMU 616). Alternatively, the IMU 616 provides sampled measurement signals to an external console (or other computing device), which determines calibration data. The sensors 614 may include sensors in one or more handheld controllers that are part of the HMD system. Thus, in some embodiments, the controller can transmit sensor data to the host computer 102, which can fuse the sensor data received from the HMD 104 and the handheld controller.

[0084] The sensor 614 may operate at a relatively high frequency to provide sensor data at a high rate. For example, the sensor data may be generated at a rate of 1000 Hz (or one sensor reading every millisecond). In this manner, 1000 readings are generated per second. If the sensor generates this much data at this rate (or even faster), the data set used to predict movement becomes very large, even over a relatively short period of time, on the order of tens of milliseconds.

[0085] As mentioned above, in some embodiments, the sensor 614 may include an optical sensor sensitive to light emitted by a base station in the environment of the HMD 104 for purposes of tracking the position and / or orientation, posture, etc. of the HMD 104 in 3D space. The calculation of the position and / or orientation may be based on timing characteristics of the light pulses and the presence or absence of light detected by the sensor 614.

[0086] The HMD 104 may further include the head tracking system 118 described above. The head tracking system 118 may utilize one or more of the sensors 614 to track head movement, including head rotation, of the user 106, as described above. For example, the head tracking system 118 may track up to six degrees of freedom (i.e., 3D position, roll, pitch, and yaw) of the HMD 104. These calculations may be performed for each frame in a series of frames so that the application 132 can determine how to render the scene in the next frame according to the head position and orientation. In some embodiments, the head tracking system 118 is configured to generate head tracking data that can be used to predict the future pose (position and / or orientation) of the HMD 104 based on current and / or past data and / or based on known / implied scan-out latencies of individual subsets of pixels in the display system. This is because the application 132 is required to render frames before the user 106 actually sees the light (and therefore the image) on the display panel 108. Thus, the next frame may be rendered based on this future prediction of head position and / or orientation made at an earlier point in time. The rotation data provided by the head tracking system 118 may be used to determine both the direction of rotation of the HMD 104 and the amount of rotation of the HMD 104 in any suitable units of measure. For example, the direction of rotation may be simply output in terms of positive or negative horizontal directions and positive or negative vertical directions corresponding to left, right, up, and down. The amount of rotation may be in terms of degrees, radians, etc. Angular velocity may be calculated to determine the rate of rotation of the HMD 104.

[0087] The HMD 104 may further include the eye tracking system 120 described above that generates eye tracking data. The eye tracking system 120 may include, without limitation, a camera or other optical sensor within the HMD 104 to capture image data (or information) of the user's eyes, and the eye tracking system 120 can use the captured data / information to determine the 3D position of each eye relative to the HMD 104, including motion vectors, interpupillary distance, interocular distance, magnitude of twist and rotation (i.e., roll, pitch, and yaw), and gaze direction of each eye. In one example, infrared light is emitted within the HMD 104 and reflected from each eye. The reflected light is received or detected by a camera of the eye tracking system 120 and analyzed to extract eye rotation from changes in the infrared light reflected by each eye. Many methods for tracking the eyes of the user 106 may be used by the eye tracking system 120. Thus, the eye tracking system 120 may track up to six degrees of freedom (i.e., 3D position, roll, pitch, and yaw) for each eye, and at least a subset of the tracked quantities may be combined from the two eyes of the user 106 to estimate a point of gaze (i.e., a 3D location or position within the virtual scene the user is viewing), which may be mapped to a location on the display panel 108 to predict where the user 106 is looking with respect to an individual subset (e.g., a row) or group of consecutive subsets (e.g., a group of consecutive rows) of pixels of the display panel 108. For example, the eye tracking system 120 may integrate information from past measurements, measurements identifying the position of the user 106's head, and 3D information describing the scene presented by the display panel 108. Thus, information regarding the position and orientation of the user 106's eyes is used to determine a point of gaze within the virtual scene presented by the HMD 104 at which the user 106 is viewing and map the point of gaze to a location on the display panel 108 of the HMD 104.

[0088] In the illustrated implementation, the host computer 102 includes the aforementioned processor 126 , which may include one or more GPUs 130 , as well as memory 128 that stores an application 132 , and a communication interface 134 .

[0089] Memory 128 may further store an operating system module 618 configured to manage hardware within and coupled to host computer 102 for the benefit of other modules. Memory 128 may further store a video game client 620 configured to execute one or more video games, such as video game 132(1), within video game library 622. A video game within video game library 622 may be retrieved and executed by loading video game client 620. In one example, user 106 may select to play one of multiple video games purchased and downloaded to video game library 622 by loading video game client 620 and selecting video game 132(1) to begin execution of video game 132(1). Video game client 620 may enable a user to log in to the video game service using credentials (e.g., user account, password, etc.).

[0090] The host computer 102 may further include a communication interface 134 including one or more antennas 624 (e.g., of a transceiver) to facilitate a wireless connection to a network and / or a second device, such as, but not limited to, the HMD 104. The communication interface 134 may implement one or more of a variety of wireless technologies, such as Wi-Fi, Bluetooth, radio frequency (RF), etc. It should be appreciated that the communication interface 134 of the host computer 102 may further include a physical port to facilitate a wired connection to a network and / or a second device, such as the HMD 104, or another type of display device, such as the portable gaming device 502.

[0091] Generally, the host computer 102 has input devices 626 and output devices 628. The input devices 626 can be a keyboard, keypad, mouse, touch screen, joystick, control buttons, microphone, camera, etc. The output devices 628 can include, but are not limited to, a display, light elements (e.g., LEDs), a vibrator for generating tactile sensations, speakers (e.g., headphones), etc.

[0092] Although the present subject matter has been described in language specific to structural features, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features described. Rather, the specific features are disclosed as example forms of implementing the claims.

Claims

1. a host computer, a processor; and a memory storing computer-executable instructions that, when executed by the processor, cause the host computer to: executing a graphics rendering application to render frames of the scene at a first resolution; a two-dimensional (2D) array of pixels of the frame, the 2D array of pixels comprising: first pixels representing a first copy of the frame downscaled to a second resolution lower than the first resolution; second pixels representing a second copy of the frame downscaled to the second resolution; a third pixel representing a first copy of the sub-region of the scene; a fourth pixel representing a second copy of the sub-region; encoding the 2D array of pixels to obtain encoded pixel data for the frame; a host computer that transmits the encoded pixel data to a display device;

2. the third pixels have a one-to-one correspondence with the original pixels of the frame representing the sub-region; 2. The host computer of claim 1, wherein the fourth pixels correspond one-to-one with the original pixels.

3. the third pixel represents the first copy of the sub-region downscaled according to an integer ratio; 2. The host computer of claim 1, wherein the fourth pixel represents the second copy of the sub-region downscaled according to the integer ratio.

4. 2. The host computer of claim 1, wherein the sub-regions are predetermined.

5. The host computer of claim 1 , wherein the computer-executable instructions, when executed by the processor, further cause the host computer to determine the sub-region based at least in part on eye tracking data received from the display device.

6. Generating the 2D array of pixels includes arranging the pixels of the frame in a vertical packing layout, the vertical packing layout comprising: the third pixel at the top of the 2D array of pixels; the first pixel below the third pixel; the fourth pixel below the first pixel; and the second pixel at the bottom of the 2D array of pixels.

7. Encoding the 2D array of pixels comprises: allocating a first number of bits to encode the first pixel and the second pixel; allocating a second number of bits to encode the third pixel and the fourth pixel; 2. The host computer of claim 1, wherein the first number of bits is less than the second number of bits.

8. 10. The host computer of claim 1, further comprising a transceiver, wherein transmitting the encoded pixel data to the display device comprises wirelessly transmitting one or more data packets carrying the encoded pixel data via the transceiver.

9. 1. A method comprising: Rendering frames of the scene at a first resolution by a graphics rendering application executing on the host computer; a two-dimensional (2D) array of pixels of the frame, the 2D array of pixels comprising: first pixels representing a first copy of the frame downscaled to a second resolution lower than the first resolution; second pixels representing a second copy of the frame downscaled to the second resolution; a third pixel representing a first copy of the sub-region of the scene; a fourth pixel representing a second copy of the sub-region; and encoding the 2D array of pixels to obtain encoded pixel data for the frame; transmitting the encoded pixel data to a display device.

10. the third pixels have a one-to-one correspondence with the original pixels of the frame representing the sub-region; The method of claim 9 , wherein the fourth pixels have a one-to-one correspondence with the original pixels.

11. The method of claim 9 , wherein the sub-region is predetermined and corresponds to a center of the scene.

12. The method of claim 9 , further comprising determining the sub-region based at least in part on eye-tracking data received from the display device.

13. The generating the 2D array of pixels includes arranging the pixels of the frame in a vertical packing layout, the vertical packing layout comprising: the third pixel at the top of the 2D array of pixels; the first pixel below the third pixel; the fourth pixel below the first pixel; and the second pixel at the bottom of the 2D array of pixels.

14. The encoding of the 2D array of pixels comprises: allocating a first number of bits to encode the first pixel and the second pixel; allocating a second number of bits to encode the third pixel and the fourth pixel; The method of claim 9 , wherein the first number of bits is less than the second number of bits.

15. 10. The method of claim 9, wherein the transmitting the encoded pixel data to the display device comprises wirelessly transmitting one or more data packets carrying the encoded pixel data via a transceiver of the host computer.

16. the sub-region is a rectangle having M pixels in the horizontal dimension and N pixels in the vertical dimension; 10. The method of claim 9, wherein the first copy of the frame and the second copy of the frame are each downscaled to M pixels in the horizontal dimension and N pixels in the vertical dimension.

17. 1. A display device, comprising: at least one display panel; a processor; and a memory storing computer-executable instructions that, when executed by the processor, cause the display device to: receiving, from a host computer, encoded pixel data for a frame of a scene; a two-dimensional (2D) array of pixels of the frame, the 2D array of pixels comprising: first pixels representing a first copy of the frame at a second resolution lower than the first resolution at which the frame was rendered; second pixels representing a second copy of the frame at the second resolution; a third pixel representing a first copy of the sub-region of the scene; a fourth pixel representing a second copy of the sub-region; and enlarging the first copy of the frame based at least in part on the first pixels to obtain a first enlarged copy of the frame; enlarging the second copy of the frame based at least in part on the second pixels to obtain a second enlarged copy of the frame; generating a first image based at least in part on the first enlarged copy of the frame and the third pixels, wherein a subset of the third pixels at a periphery of the sub-region are blended in the first image; generating a second image based at least in part on the second enlarged copy of the frame and the fourth pixels, wherein a subset of the fourth pixels at the periphery of the sub-region are blended in the second image; A display device that causes the first image and the second image to be presented on the at least one display panel.

18. 20. The display device of claim 17, wherein the computer-executable instructions, when executed by the processor, further cause the display device to output the first image and the second image to a frame buffer in a single write operation.

19. 20. The display device of claim 17, wherein the computer-executable instructions, when executed by the processor, further cause the display device to output the first image and the second image to a frame buffer in multiple write operations.

20. 20. The display device of claim 17, wherein the computer-executable instructions, when executed by the processor, further cause the display device to transmit eye tracking data to the host computer for use by the host computer in determining the subregions.

Images