Dual detail coding in distributed system

Abstract

Techniques, among other things, are described herein for implementing a dual-detail coding scheme in a distributed display system. At a host computer, an application may render a frame of a scene at a first resolution. A host computer may generate a two-dimensional (2D) pixel array of a frame, encode the 2D pixel array, and transmit the encoded pixel data to a display device. For each eye, the 2D pixel array includes a copy of the frame shrunk to a second resolution less than the first resolution and a copy of a sub-region of the scene. At a display device, the encoded pixel data is decoded to obtain a 2D pixel array of the frame, which array is modified by magnifying a copy of the frame such that an image can be generated by mixing using a magnified copy of the frame and a copy of the sub-region.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims the priority of U.S. Patent Application Serial No. 18 / 087,431, filed on December 22, 2022. The application serial number 18 / 087,431 is incorporated herein by reference in its entirety. Background Art

[0003] Wireless streaming technology is used both inside and outside the video game industry. In the video game industry, some virtual reality (VR) game systems use wireless streaming technology to utilize the high computing capacity of a host computer to execute video games while providing greater mobility for the wearer of a wireless VR headset compared to a tethered (non - wireless) headset. Despite these advantages, the data rate of existing wireless protocols limits the amount of data that can be transmitted in a relatively short time. This poses a challenge to VR game systems operating at high frame rates, especially when playing graphically intensive video games.

[0004] This document provides technical solutions for improving and enhancing these and other systems. Brief Description of the Drawings

[0005] The detailed description is described with reference to the accompanying drawings. In the drawings, the left - most digit of the reference numeral identifies the drawing in which the reference numeral first appears. The same reference numerals are used in different drawings to represent similar or identical components or features.

[0006] Figure 1 is a diagram showing an exemplary distributed display system according to an embodiment disclosed herein, the exemplary distributed display system including a host computer and a display device in the form of a head - mounted display (HMD).

[0007] Figure 2 is a diagram showing an image presented on the display panel of the HMD according to an embodiment disclosed herein.

[0008] Figure 3A shows an exemplary packaging layout of a two - dimensional (2D) pixel array according to an embodiment disclosed herein.

[0009] Figure 3B shows another exemplary packaging layout of a 2D pixel array according to an embodiment disclosed herein.

[0010] Figure 4 shows a flowchart of an exemplary process for implementing dual - detail encoding in a distributed display system according to an embodiment disclosed herein.

[0011] Figure 5A 、 Figure 5B and Figure 5CShows an alternative arrangement of a system for streaming data from a host computer to a display device in accordance with embodiments disclosed herein.

[0012] Figure 6 Shows exemplary components of a display device such as an HMD (e.g., a VR headset) and a host computer in which the techniques disclosed herein may be implemented. Detailed Description

[0013] Techniques, devices, systems, etc. for streaming pixel data from a host computer to a display device using a dual-detail encoding scheme are described herein. The dual-detail encoding scheme allows pixel data and other data to be transmitted from the host computer to the display device in a timely manner without degrading the perceived fidelity of the image displayed on the display device. This is particularly beneficial when streaming real-time content such as video game content. For example, unlike pre-recorded content, real-time content cannot be pre-streamed and buffered on the client before the content is ready to be displayed. Instead, frames are rendered milliseconds before the content is displayed based on data received from the display device.

[0014] Although the following description is not limited to game systems, or even to VR systems, many of the examples described herein relate to a VR game system that includes a host computer and an HMD communicatively coupled thereto. However, it should be understood that the HMD is just one example of a "display device" that can be used to display streamed content, and other types of display devices are also contemplated herein. Additionally, the techniques described herein may be implemented in wireless distributed systems and / or wired distributed systems (e.g., a host computer and a display device connected via a cable). In a distributed system that includes an HMD, the HMD may, depending on the circumstances, be worn by a user to immerse the user in a VR environment or an augmented reality (AR) environment. One or more display panels of the HMD are configured to present images based on data generated by an application (e.g., a video game). The application is executed on the host computer and generates pixel data for each frame in a series of frames. The pixel data is encoded and transmitted (or sent) from the host computer to the HMD to present an image that the user views through the optics included in the HMD, causing the user to perceive the image as if the user were immersed in a VR or AR environment.

[0015] In some embodiments, the HMD is configured to transmit data to a host computer, and the host computer is configured to use the data in various ways, such as to generate pixel data for a given frame. For example, the host computer may use head tracking data received from the HMD to generate pose data that indicates the predicted pose the HMD will be in when the light-emitting elements of the HMD's display panel will illuminate within the frame. This pose data may be input into an executing application (e.g., a video game) to generate pixel data for a given frame. In some examples, as will be described in more detail below, eye tracking data received from the HMD may be used to determine a sub-region of the scene that is ultimately displayed with a higher quality than the rest of the displayed image, such that the user is looking at a more detailed part of the image, making the image appear sharp. Generally, the HMD may receive encoded pixel data from the host computer and display an image on the HMD's display panel based on the received pixel data. As will be described in more detail below, before rendering the final image, the HMD may reconstruct and / or modify the pixel data as needed such that the user perceives the content as if the content is intended to be perceived in accordance with the user's head orientation and gaze.

[0016] As described above, the limited data rates of existing communication protocols (e.g., wireless protocols) pose challenges in distributed display systems, especially in distributed wireless VR game systems playing graphics-intensive video games. For example, depending on the wireless protocol, the typical data rate for wirelessly transmitting data from a host computer to a display device can range from 50 megabits per second to 300 megabits per second (Mbps). Consider an example where the available data rate is approximately 100 Mbps. In this example, a video game executed on the host computer can render frames at a resolution of 2.5k pixels per eye (i.e., an image with 2500×2500 pixels), which is equivalent to 12,500,000 pixels on two images, one image per eye. At 3 bytes per pixel, this means that the amount of data for transmitting an image of a given frame is approximately 300 megabits. In a VR game system at 120 Hertz (Hz), in order to send all the image data in a timely manner, the host computer must therefore send pixel data at a rate of 36,000 Mbps (or 36 gigabits per second (Gbps)). However, in a distributed system limited to a 100 Mbps data rate, this amount of pixel data cannot be transmitted fast enough.

[0017] This document describes techniques, devices, and systems for implementing a dual-detail encoding scheme that allows pixel data and other data to be transferred from a host computer to a display device in a timely manner without compromising the perceived fidelity of the image displayed on the display device. An exemplary process for streaming pixel data from a host computer to a display device may include: rendering a frame of a scene by a graphics rendering application executing on the host computer at a first resolution, generating a 2D pixel array of the frame, encoding the 2D pixel array to obtain encoded pixel data of the frame, and sending the encoded pixel data to the display device. In particular, the 2D pixel array encoded in the foregoing process provides two levels of detail for each eye. That is, for each eye, the 2D pixel array includes a copy of the frame reduced to a second resolution smaller than the first resolution and a copy of a sub-region of the scene. And because the frame includes the sub-region, for each eye, the 2D pixel array includes two copies of the sub-region (one copy is a copy of the sub-region itself and the other copy is included in the copy of the reduced frame). In some examples, the pixels representing the copy of the sub-region of the scene correspond one-to-one with the original pixels of the frame, which allows the image to be presented via the display device, where the sub-regions are rendered with the same or similar quality as when they were rendered by the application in the original frame. Additionally, by sending a copy of the sub-region (which has fewer pixels than the full image) and a copy of the reduced frame, the host computer is able to send a reduced amount of pixel data at data rates typically available in distributed display systems, including wireless systems. Thus, the dual-detail encoding technique described herein allows the host computer to transmit a smaller amount of data (i.e., the 2D pixel array), and the display device is configured to reconstruct the image from the 2D pixel array such that a user of the display device perceives the presented image as a high-quality image (e.g., an image that looks clear and sharp, etc.).

[0018] An exemplary process for displaying an image based on pixel data received from a host computer may include: receiving encoded pixel data of a frame of a scene from the host computer, decoding the encoded pixel data to obtain a 2D pixel array of the frame, magnifying a first copy of the frame at least in part based on a first pixel of the 2D pixel array to obtain a first magnified copy of the frame, and magnifying a second copy of the frame at least in part based on a second pixel of the 2D pixel array to obtain a second magnified copy of the frame. The process may continue: generating a first image at least in part based on the first magnified copy of the frame and a third pixel of the 2D pixel array representing a first copy of a sub-region of the scene, wherein a subset of the third pixels at the periphery of the sub-region is blended in the first image; and generating a second image at least in part based on the second magnified copy of the frame and a fourth pixel of the 2D pixel array representing a second copy of the sub-region, wherein a subset of the fourth pixels at the periphery of the sub-region is blended in the second image. Then, the first image and the second image may be presented on respective display panels (such as a left display panel and a right display panel) of the display device, or on a single display panel.

[0019] By using the dual-detail encoding technique described herein to reduce the amount of data to be sent from the host computer to the display device, a distributed display system can stream pixel data in a timely manner to display corresponding images on the display device, and a user perceives the images on the display device as high-fidelity images. For example, even though portions of the displayed image outside the sub-region may be presented at a relatively low quality (e.g., low resolution), a user's gaze may be directed to the sub-region of the scene where the image quality is relatively high, such that to the user, it may appear as if the image has high quality. In other words, the distributed display system may implement the dual-detail encoding technique described herein to avoid sending pixel data of scene regions that a user is less likely to look at when presenting the corresponding image, yet the user does not have the perception that the quality of the image is lower than the original rendered frame.

[0020] Devices, systems, and non-transitory computer-readable media storing computer-executable instructions for implementing the techniques and processes disclosed herein are also disclosed. Although the techniques and systems disclosed herein are discussed by way of example in the context of video game applications, particularly VR game applications, it should be understood that the techniques and systems described herein may provide benefits for other applications, including but not limited to non-VR applications (e.g., AR applications, mixed reality (MR) applications), and / or non-game applications, such as industrial machine applications, defense applications, robotic applications, and the like.

[0021] Figure 1 FIG. 10 is a diagram showing an exemplary distributed display system 100 according to an embodiment disclosed herein. The exemplary distributed display system includes a host computer 102 and a display device in the form of an HMD 104.Figure 1 Depicts the HMD 104 worn by the user 106. Figure 1 Also depicted is an exemplary implementation of the host computer 102 in the form of, for example, a laptop computer 102(1) carried in a backpack or a personal computer (PC) 102(N) that may be located, for example, in the home of the user 106. However, it should be understood that these exemplary types of host computers 102 do not limit the present disclosure. For example, the host computer 102 may be implemented as any type and / or any number of computing devices, including but not limited to a PC, a laptop computer, a desktop computer, a portable digital assistant (PDA), a mobile phone, a tablet computer, a set-top box, a gaming console, a portable gaming device, a server computer, a wearable computer (e.g., a smartwatch, etc.), or any other electronic device that can send data to and receive data from other devices. The host computer 102 may be co-located with the HMD 104 in the same environment, such as the home of the user 106 wearing the HMD 104. Alternatively, the host computer 102 may be located remotely relative to the HMD 104, such as a host computer 102 in the form of a server computer located at a remote geographical location relative to the geographical location of the HMD 104. In a remote host computer 102 implementation, the host computer 102 may be communicatively coupled to the HMD 104 via a wide area network, such as the Internet. In a local host computer 102 implementation, the host computer 102 may be co-located with the HMD 104 in an environment (e.g., a home), whereby the host computer 102 and the HMD 104 may be communicatively coupled together directly or via an intermediate network device through a local area network (LAN).

[0022] It should also be understood that the exemplary HMD 104 is just one exemplary type of display device that can be implemented using the techniques and systems described herein. For example, other types and / or numbers of display devices may be used instead of or in combination with the HMD 104 described in many of the examples herein. Thus, the HMD 104 may be more generally referred to herein as a "display device," and it should be understood that other suitable types of display devices may exchange data with the host computer 102. Such display devices may include but are not limited to a television, a portable gaming device having a display, a laptop computer, a desktop computer, a PDA, a mobile phone, a tablet computer, a wearable computer (e.g., a smartwatch, etc.), or any other electronic device that can send / receive data and display an image on a display screen or panel.

[0023] In Figure 1In the example, the HMD 104 and the host computer 102 are communicatively coupled and configured to work together in a collaborative manner to render a given frame and present a corresponding image on the display panel 108 of the HMD 104. This collaboration iterates over a series of frames to render a series of images (e.g., images of a VR game) on the HMD 104. In the specific implementation shown, the HMD 104 includes one or more processors 110 and a memory 112 (e.g., a computer-readable medium 112). In some specific implementations, the processor 110 may include a central processing unit (CPU), a graphics processing unit (GPU) 114, both the CPU and the GPU 114, a microprocessor, a digital signal processor, or other processing units or components known in the art. Alternatively or additionally, the functions described herein may be performed at least in part by one or more hardware logic components. For example, illustrative types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on a chip (SOCs), etc., but are not limited thereto. Additionally, each processor in the processor 110 may have its own local memory, which may also store program modules, program data, and / or one or more operating systems. In some embodiments, each processor in the processor 110 may have its own encoder and / or decoder hardware.

[0024] The memory 112 may include volatile and non-volatile memory, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Such memory includes but is not limited to random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or other memory technologies, compact disc read-only memory (CD-ROM) or other optical storage devices, magnetic tape cartridges, tapes, magnetic disk storage devices, or other magnetic storage devices, redundant array of independent disks (RAID) storage systems, or any other medium that can be used to store the required information and can be accessed by a computing device. The memory 112 may be implemented as a computer-readable storage medium (CRSM), which may be any available physical medium that can be accessed by the processor 110 to execute the instructions stored on the memory 112. In a basic specific implementation, the CRSM may include RAM and flash memory. In other specific implementations, the CRSM may include but is not limited to ROM, EEPROM, or any other tangible medium that can be used to store the required information and can be accessed by the processor 110.

[0025] Generally, the HMD 104 may include logic (e.g., software, hardware, and / or firmware, etc.) configured to implement the techniques, functions, and / or operations described herein. The computer-readable medium 112 may include various modules (such as instructions, data storage, etc.) that may be configured to execute on the processor 110 to implement the techniques, functions, and / or operations described herein. An example functional module in the form of a synthesizer 116 is shown as being stored in the computer-readable medium 112 and may execute on the processor 110, but the same function may alternatively be implemented in hardware, firmware, or as a system-on-chip (SOC) and / or other logic. Additionally, additional or different functional modules may be stored in the computer-readable medium 112 and may execute on the processor 110. The synthesizer 116 is configured to modify pixels (or pixel data) to generate an image and output the modified pixel data representing the image to a frame buffer (e.g., a stereoscopic frame buffer) such that the image may be presented on the display panel 108 of the HMD 104. For example, the synthesizer 116 is configured to reconstruct a frame using pixel data received from the host computer 102, as will be described in more detail below. Additionally, the synthesizer 116 may apply other modifications to the pixel data before outputting the final pixel data to the frame buffer. For example, the synthesizer 116 may be configured to adjust for geometric distortion, chromatic aberration, reprojection, etc. At least some of these adjustments may compensate for the distortion of the near-eye optical subsystem (e.g., lenses and other optics) of the HMD 104, while other adjustments such as reprojection adjustments may compensate for minor inaccuracies in the original pose prediction of the HMD 104 and / or compensate for an application failure to produce a frame rate and / or compensate for late data packets. For example, a reprojection frame may be generated by transforming (e.g., through rotation and reprojection calculations) an application-rendered frame using pixel data from the application-rendered frame in a manner that takes into account an updated prediction of the pose of the HMD 104, which may be more accurate than the original pose prediction. Thus, the modified pixel data obtained from applying the reprojection adjustment (and / or other adjustments) may be used to present an image of a given frame on the display panel 108 of the HMD 104, and this process may be iterated over a series of frames.

[0026] The HMD 104 may also include a head tracking system 118 that generates head tracking data. The head tracking system 118 may utilize one or more sensors (e.g., infrared (IR) light sensors mounted on the HMD 104) and one or more tracking beacons (e.g., IR light emitters collocated with the HMD 104 in the environment) to track the head movement or motion (including head rotation) of the user 106. This exemplary head tracking system 118 is non-limiting, and other types of head tracking systems 118 (e.g., camera-based, inertial measurement unit (IMU)-based, etc.) may be used.

[0027] The HMD 104 may also include an eye tracking system 120 that generates eye tracking data. The eye tracking system 120 may include, but is not limited to, a camera or other optical sensor inside the HMD 104 to capture image data (or information) of the user's eyes, and the eye tracking system 120 may use the captured data / information to determine the motion vectors of each eye relative to the HMD 104, the interpupillary distance, the distance between the two eyes, 3D positioning, including the amplitude of torsion and rotation (i.e., roll, pitch, and yaw) of each eye, and the gaze direction. In one example, infrared light is emitted within the HMD 104 and reflected from each eye. The reflected light is received or detected by the camera of the eye tracking system 120 and analyzed to extract eye rotation from the changes in the infrared light reflected from each eye. The eye tracking system 120 may use a variety of methods for tracking the eyes of the user 106.

[0028] Figure 1 Depicted is the HMD 104 transmitting data 122 to the host computer 102. As described above, the data 122 may include, but is not limited to, head tracking data and / or eye tracking data. Additionally, the data 122 may be sent during runtime via the communication interface 124 of the HMD 104 when a frame is rendered at the host computer 102 and when the corresponding image is presented at the HMD 104. The communication interface 124 of the HMD 104 may include wired and / or wireless components (e.g., chips, ports, etc.) to facilitate sending / receiving wired and / or wireless data to / from the host computer 102 directly or via one or more intermediate devices (such as a wireless access point (WAP)). For example, the communication interface 124 may include a wireless network interface controller. The communication interface 124 may comply with the Institute of Electrical and Electronics Engineers (IEEE) 502.11 standard and may include a radio (e.g., having one or more antennas) to facilitate a wireless connection with the host computer 102 and / or other devices and for sending and receiving data packets using radio frequency (RF) communication. The communication interface 124 may be built into the HMD 104 or coupled to the HMD 104 (e.g., wireless adapter, peripheral, accessory, etc.) to enable the HMD 104 to operate as a 502.11-compliant wireless communication device. In some embodiments, the communication interface 124 may include a universal serial bus (USB) wireless adapter (e.g., USB dongle) inserted into a USB port on the HMD 104. In other embodiments, the communication interface 124 may be coupled to the electrical components of the HMD 104 via a peripheral component interconnect (PCI) interface. It should be understood that the communication interface 124 may also include one or more physical ports to facilitate a wired connection (e.g., via a cable) with the host computer 102 and / or another device (e.g., a plug-in network device communicating with other wireless networks).

[0029] Turning to host computer 102, host computer 102 is shown as including one or more processors 126 and a memory 128 (e.g., computer-readable medium 128). In some embodiments, processor 126 may include a CPU, a GPU 130, both a CPU and a GPU 130, a microprocessor, a digital signal processor, or other processing units or components known in the art. Alternatively or additionally, the functions described herein may be performed at least in part by one or more hardware logic components. For example, illustrative types of hardware logic components that may be used include, but are not limited to, FPGAs, ASICs, ASSPs, SOCs, etc. Additionally, each processor in processor 126 may have its own local memory, which may also store program modules, program data, and / or one or more operating systems. In some embodiments, each processor in processor 126 may have its own encoder and / or decoder hardware. Compared to the GPU 114 of HMD 104, the GPU 130 of host computer 102 may be a higher-performance GPU with a higher computing capacity. For example, the computing capacity of the GPU 114 of HMD 104 may not be sufficient to handle the graphics of certain graphics-intensive video games, which is why system 100 is distributed; this is to utilize the relatively high computing capacity of host computer 102 to execute graphics-intensive video games.

[0030] The memory 128 of host computer 102 may include volatile and non-volatile memory, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Such memory includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technologies, CD-ROM or other optical storage devices, magnetic tape cassettes, tapes, magnetic disk storage devices, or other magnetic storage devices, RAID storage systems, or any other medium that can be used to store the required information and can be accessed by a computing device. The memory 128 may be implemented as a CRSM, which may be any available physical medium that can be accessed by processor 126 to execute the instructions stored on the memory 128. In a basic embodiment, the CRSM may include RAM and flash memory. In other embodiments, the CRSM may include, but is not limited to, ROM, EEPROM, or any other tangible medium that can be used to store the required information and can be accessed by processor 126.

[0031] Generally, host computer 102 may include logic (e.g., software, hardware, and / or firmware, etc.) configured to implement the technologies, functions, and / or operations described herein. The computer-readable medium 128 may include various modules (such as instructions, data storage, etc.), which may be configured to be executed on processor 126 to implement the technologies, functions, and / or operations described herein. InFigure 1 An example functional module in the form of application 132 is shown. Application 132 may represent a graphics rendering application or a graphics-based application, such as video game 132(1). In some examples, host computer 102 may include a client application or other software (e.g., a video game client) configured to execute one or more applications 132. For example, game software may be installed on host computer 102 to play video game 132(1).

[0032] Generally, the communication interface 134 of host computer 102 may include wired and / or wireless components (e.g., chips, ports, etc.) to facilitate sending / receiving wired and / or wireless data to / from HMD 104 (or any other type of display device) directly or via one or more intermediate devices (such as a WAP). In some examples, communication interface 134 may include a wireless network interface controller. Communication interface 134 may comply with the IEEE 502.11 standard and may include a radio (e.g., having one or more antennas) to facilitate wireless connection with HMD 104 and / or other devices and for sending and receiving data packets using RF communication. Communication interface 134 may be built into or coupled to host computer 102 (e.g., a wireless adapter, peripheral, accessory, etc.) to enable host computer 102 to operate as a 502.11-compliant wireless communication device. In some embodiments, communication interface 134 may include a USB wireless adapter (e.g., a USB dongle) inserted into a USB port on host computer 102. In other embodiments, communication interface 134 may be coupled to the electrical components of host computer 102 via a PCI interface. It should be understood that communication interface 134 may also include one or more physical ports to facilitate a wired connection (e.g., via a cable) with HMD 104 and / or another device (e.g., a plug-in network device communicating with other wireless networks).

[0033] In some embodiments, the HMD 104 may represent a VR head-mounted device for use in a VR system (such as for use with a VR gaming system), in which case the video game 132(1) may represent a VR video game 132(1). However, the HMD 104 may additionally or alternatively be implemented as an AR head-mounted device for AR applications, an MR head-mounted device for MR applications, or a head-mounted device for VR, AR, and / or MR applications unrelated to gaming (e.g., industrial applications). In AR, the user 106 sees virtual objects overlaid on the real-world environment, while in VR, the user 106 generally does not see the real-world environment but is fully immersed in a virtual environment, as perceived via the display panel 108 and optics (e.g., lenses) of the HMD 104. It should be understood that in some VR systems, a through-image of the user 106's real-world environment may be displayed in combination with the virtual image to create an enhanced VR environment in the VR system, whereby the VR environment is enhanced with real-world images (e.g., overlaid on the virtual world). The examples described herein mainly relate to the VR-based HMD 104, but it should be understood that the HMD 104 is not limited to specific implementations in VR applications.

[0034] Generally speaking, and as mentioned above, the application 132 executed on the host computer 102 may represent a graphics-based application 132 (e.g., the video game 132(1)). The application 132 is configured to generate pixel data for a series of frames, and this pixel data is ultimately used to render a corresponding image on the display panel 108 of the HMD 104. In the system 100 with the HMD 104, components of the host computer 102 may determine a predicted "illumination time" for the frame. This predicted "illumination time" for the frame represents the time during which the light-emitting elements of the display panel 108 of the HMD 104 will be illuminated within the frame. This prediction may take into account the estimated time it will take to send data over the wireless communication link between the host computer 102 and the HMD 104, as well as the predicted rendering time of the application 132 and / or the known scan output time of the pixels from the frame buffer, etc. In some embodiments, the prediction for the wireless communication link may be different from that for a wired communication link. In some embodiments, the illumination time may be predicted as an amount of time in the future (e.g., about 20 milliseconds to 40 milliseconds in the future), and this amount of time may vary depending on the connection type (e.g., wired connection and wireless connection).

[0035] The host computer 102 may receive data 122 (e.g., head tracking data, eye tracking data, etc.) from the HMD 104. The data 122 may be generated and / or transmitted at any suitable frequency, such as a frequency corresponding to the target frame rate and / or refresh rate of the HMD 104; or generated and / or transmitted at a different (e.g., faster) frequency, such as 360 Hertz (Hz) (or 1 sensor reading every 2.7 milliseconds). Components of the host computer 102 may determine a predicted pose that the HMD 104 will be in at a predicted illumination time, at least in part based on the head tracking data included in the data 122. Pose data indicating the predicted pose may be provided to the execution application 132 to render a frame (e.g., generate pixel data for the frame) based on the predicted pose, and the application 132 may output pixel data associated with the frame. The pixel data may correspond to the pixel array of the display panel 108 of the HMD 104. For example, the pixel data output by the application 132 based on the pose data may include a two-dimensional array of per-pixel values (e.g., color values) of the pixel array on the display panel 108 of the HMD 104. In an illustrative example, the HMD 104 may include a pair of stereoscopic display panels 108, and the application 132 may render frames of a scene at a first resolution. For example, the first resolution at which frames of each display panel 108 are rendered may be an array of 2500×2500 pixels. In this illustrative example, the pixel data for a single frame of the display panel 108 may include 2500×2500 pixel values (or two display panels 108 include 12,500,000 pixel values). In some embodiments, the pixel data may include data for each pixel represented by a single set of color and alpha values (e.g., one color value for the red channel, one color value for the green channel, one color value for the blue channel, and one or more values for one or more alpha channels).

[0036] It should be understood that in addition to the pixel data generated for presenting an image on the display panel 108 of the HMD 104, the logic of the host computer 102 may generate additional data, and at least some of this additional data may also be transmitted to the HMD 104. In some embodiments, the additional data may be encapsulated with the pixel data and transmitted to the HMD 104 as data 136. At least some of the additional data encapsulated in the data 136 may be used by the logic of the HMD 104 to present an image corresponding to the frame on the display panel 108 of the HMD 104. The additional data may include, but is not limited to, pose data indicating the predicted pose of the HMD 104, depth data, motion vector data, parallax occlusion data, additional pixel data, and / or audio data. For example, in addition to generating pixel data corresponding to the image to be rendered, the application 132 may also generate depth data (e.g., Z-buffer data) and / or additional pixel data (sometimes referred to herein as "out-of-bounds pixel data" or "additional pixel data") for the frame. Additionally or alternatively, motion vector data may be generated at least in part based on the head tracking data received from the HMD 104, and the motion vector data may be sent to the HMD 104 to assist in processing the pixel data on the HMD 104. For example, the motion vector data may be generated based on a comparison of the head tracking data generated at two different points in time (e.g., a comparison of the head tracking data separated by a few milliseconds). The logic of the HMD 104 may use some or all of the additional data to modify the pixel data to correct for errors in the pose prediction made in advance at the host computer 102. For example, the adjustments made by the HMD 104 may include, but are not limited to, adjustments for geometric distortion, chromatic aberration, reprojection, etc.

[0037] Before the pixel data is transmitted to the HMD 104, the host computer 102 may implement the dual-detail encoding technique disclosed herein, which allows the data 136 to be sent to the HMD 104 in a timely manner under a data rate limit of approximately 100 Mbps. As Figure 1As depicted, compared to the number of pixels in the application-rendered frame (which can be approximately 12.5 million pixels, as described in the example above), the host computer 102 can generate the 2D pixel array 138 of the frame with a reduced number of pixels. The 2D pixel array 138 can include first pixels representing a first copy 140(1) of the frame scaled down to a second resolution and second pixels representing a second copy 140(2) of the frame scaled down to the second resolution, where the second resolution is less than the first resolution at which the application 132 renders the frame. As used herein, "scaling down" means reducing the resolution and is sometimes also referred to as "downsampling" or "downconverting". In an illustrative example, each copy 140(1), 140(2) of the frame of the individual display panel 108 of the HMD 104 can be scaled down from a first resolution of, for example, 2500×2500 pixels to a second resolution of, for example, 500×500 pixels. This is exemplary, and any suitable amount or degree of downscaling can be achieved. The 2D pixel array 138 can also include third pixels representing a first copy 142(1) of a sub-region of the scene and fourth pixels representing a second copy 142(2) of the sub-region. The sub-region can be a region of the scene of a given frame.

[0038] In some examples, the sub-region can be a pre-determined (or fixed) sub-region. In VR, the user 106 looks straight ahead most of the time, which means that most of the time, the user 106 is looking at a very small portion of the image presented on the near-eye display (such as the HMD 104). This portion of the image corresponds to the central sub-region of the scene, 10 to 30 degrees off-center. In any case, the user 106 rarely (if at all) looks at the periphery of the scene (e.g., the corners and far edges of the image), and in fact, the user's eyes may not be able to view the periphery of the scene in the near-eye display except through their peripheral vision because the angles in the near-eye display are too extreme. Thus, in some examples, the sub-region of the scene can be pre-determined, and the pre-determined sub-region can correspond to the central sub-region of the scene. That is, in some examples, for each frame rendered, the 2D pixel array 138 can include pixels representing copies 142(1), 142(1) of the sub-region of the scene that is pre-determined to be the central sub-region of the scene, such that the sub-region is the same part of the scene for each frame and for each 2D pixel array 138 generated and transmitted, but the scene itself changes frame by frame.

[0039] In some examples, sub-regions of individual frames can be determined dynamically. For example, the sub-regions of individual frames can be determined at least in part based on eye-tracking data generated by the eye-tracking system 120 of the HMD 104 and received by the host computer 102 at runtime in the data 122. The eye-tracking data received by the host computer 102 can indicate a predicted location on the display panel 108 of the HMD 104 where the user 106 will look when an image is presented on the display panel 108 based on the encoded pixel data sent to the HMD 104 in the data 136. In some examples, the host computer 102 predicts where the user 106 will look when presenting the image. In some examples, the eye-tracking system 120 makes the prediction and sends data representing the prediction to the host computer 102. The advantage of the host computer 102 making the prediction is that the prediction can be made closer to the time when the image is presented at the HMD 104, and / or the relatively high computational capacity of the host computer 102 can be utilized to make the prediction. In any case, in examples where the sub-regions are determined dynamically on a frame-by-frame basis (e.g., instantaneously), the positioning of the sub-regions can change from frame to frame such that when the user 106's gaze turns to the left, the sub-region is selected as a sub-region that is left of center of the scene, or when the user 106's gaze turns upward, the sub-region is selected as a sub-region that is above the center of the scene. In other words, eye-tracking can be used to "steer" to a more detailed (e.g., higher resolution) internal target, which is referred to herein as a "sub-region" of the scene. In this way, a more detailed sub-region can be presented in the image at the location on the display panel 108 where the user 106 is looking, such that the user 106 looks directly at the more detailed (or higher quality) part of the image in the scene and views a less detailed (or lower quality) image (if any) through the user's peripheral vision.

[0040] Copies 142(1), 142(2) of the sub-region can have any suitable shape. Figure 1 Examples show copies 142(1), 142(2) of the sub-region as rectangles, but copies 142(1), 142(2) of the sub-region can have different shapes, such as oval, square, square with rounded corners (e.g., "squircle"), triangle, pentagon, or any other suitable polygon.

[0041] In some examples, the pixels of the respective copies 142(1), 142(2) of the sub-regions in the 2D array 138 may correspond one-to-one with the original pixels of the frame representing the sub-region. In other words, the respective copies 142(1), 142(2) of the sub-regions in the 2D array 138 may be exact copies that are not resampled (e.g., downscaled) in any way. This means that each pixel in the sub-region of the scene of each image rendered with in-frame application can be copied as-is into the respective copies 142(1), 142(2) of the sub-regions in the 2D array 138. In some examples, the copies 142(1), 142(2) of the sub-regions in the 2D array 138 have been downscaled to a reduced resolution (i.e., a resolution less than the resolution at which the application 132 renders the sub-region in a given frame). In these examples, the copies 142(1), 142(2) of the sub-regions in the 2D array 138 can be downscaled with idealized sampling, such as by downscaling the sub-region according to an integer ratio (e.g., 1:2). When the copies of the sub-regions are magnified at the HMD 104, the idealized sampling of the sub-regions can allow for relatively high-quality resampling. In other words, downscaling the copies 142(1), 142(2) of the sub-regions according to an integer ratio can mitigate artifacts (e.g., aliasing, blurring, etc.) in the corresponding portions of the image presented at the HMD 104, while downscaling the copies 142(1), 142(2) of the sub-regions by 10%, 15%, etc. may result in such artifacts in the resulting image at the HMD 104.

[0042] The copies 142(1), 142(2) of the sub-regions can have any suitable size. The trade-off of making the sub-regions as large as possible (so that the relatively low-quality portions of the image outside the sub-regions are less noticeable) is that more data is needed to transmit the pixels of the larger sub-regions. Thus, because data rate limitations may vary with the specific implementation, the optimal size of the sub-regions may vary, and there may be practical limitations to increasing the size of the sub-regions. Additionally, in some examples, the application 132 can change the resolution at which it renders frames on a per-frame basis. In other words, the application 132 can render the first frame in a series of frames at a first resolution, and the application 132 can subsequently render the second frame (e.g., the next frame) in the series at a second resolution different from the first resolution. This means that the percentage of the scene (or image) corresponding to the sub-region can vary from frame to frame, even if the size of the sub-region is fixed at M×N pixels. In the running example, the sub-region can be a sub-region of 800×800 pixels. Thus, in Figure 1In an example, the 2D pixel array 138 can be, for example, an array of 800 × 3200 pixels (or two display panels 108 have 2,560,000 pixel values). Another example is a 2D array 138 of 960 × 3840 pixels (or two display panels 108 have 3,686,400 pixel values). In other words, the size of the rendering target of the 2D array 138 can have an aspect ratio of 4:1. Thus, compared to the 12,500,000 pixel values of the frame rendered by the application 132 at the first resolution, generating the 2D pixel array 138 results in far fewer pixels ultimately sent to the HMD 104.

[0043] As Figure 1 shown, the host computer 102 can encode the 2D pixel array 138 to obtain the encoded pixel data of the frame, and the host computer 102 can send the encoded pixel data to the HMD 104 as part of the data 136 sent to the HMD 104. In some examples, the host computer 102 can use the encoder hardware of the GPU 130 to encode the 2D pixel array 138. In addition, the data 136 including the encoded pixel data (and possibly additional data) is sent to the HMD 104 during runtime. In this way, the host computer 102 is configured to encode and transmit an image smaller than the application-rendered image, and these smaller images include redundant data and provide different levels of detail for each eye of the user 106, such that the HMD 104 can reconstruct the image presented on the display panel 108 of the HMD 104, such that the user 106 perceives them as high-fidelity images, even by sacrificing (e.g., not sending) some of the pixel data in the pixel data of the frame in the copy 140(1), 140(2) of the frame in the 2D pixel array 138, as described above.

[0044] At the HMD 104, upon receiving the data 136 including the encoded pixel data (and possibly additional data), the HMD 104 can decode the encoded pixel data to obtain the 2D pixel array 138 of the frame. In some examples, the HMD 104 can use the decoder hardware of the GPU 114 to decode the encoded pixel data received from the host computer 102. The synthesizer 116 of the HMD 104 can then modify the pixels of the 2D array 138 in order to reconstruct the image of the frame, as well as adjust for geometric distortion, chromatic aberration, reprojection, etc.

[0045] Figure 2 is a diagram showing images 200(1) and 200(2) respectively presented on the display panels 108(1) and 108(2) of the HMD 104 according to an embodiment disclosed herein. As mentioned above, the HMD 104 is configured to decode the encoded pixel data received from the host computer 102 to obtainFigure 1 the 2D pixel array 138 of the frame shown in, and the synthesizer 116 modifies the pixels of the 2D array 138 to reconstruct the images 200(1), 200(2) of the frame. To reconstruct the right image 200(2) (e.g., for the right display panel 108(2) of a pair of stereoscopic display panels 108), the synthesizer 116 may at least partially magnify a first copy 140(1) of the frame based on corresponding pixels of the 2D array 138 to obtain a first magnified copy 202(1) of the frame, and may generate the right image 200(2) at least partially based on the first magnified copy 202(1) of the frame and the pixels of the 2D array 138 representing a first copy 142(1) of a sub-region of the scene, wherein a subset of the pixels at the periphery 204(1) of the sub-region is blended in the right image 200(2). To reconstruct the left image in both images, the synthesizer 116 may at least partially magnify a second copy 140(2) of the frame based on corresponding pixels of the 2D array 138 to obtain a second magnified copy 202(2) of the frame, and may generate the left image 200(1) at least partially based on the second magnified copy 202(2) of the frame and the pixels of the 2D array 138 representing a second copy 142(2) of the sub-region, wherein a subset of the pixels at the periphery 204(2) of the sub-region is blended in the left image 200(1). The left image 200(1) and the right image 200(2) may then be presented on corresponding display panels 108(1) and 108(2) of the HMD 104.

[0046] Because the copies 140(1), 140(2) of the frame in the 2D pixel array 138 are scaled down, the magnified copies 202(1), 202(2) of the frame may appear blurry in the displayed images 200(1), 200(2). Specifically, if looked at directly, the images 200(1), 200(2) will appear blurry in regions outside the respective sub-regions 142(1), 142(2), but since the user 106's eyes may look directly at the sub-regions 142(1), 142(2), the relatively low-quality images outside the sub-regions 142(1), 142(2) may be ignored because those parts of the image are in the user 106's peripheral vision. As described above, by using eye tracking to dynamically determine the sub-regions, the instances where the user 106 notices the relatively low-quality images outside the sub-regions 142(1), 142(2) may be reduced, but even for pre-determined (or fixed) sub-regions, it is difficult for an average user 106 to notice any degradation in image quality due to the dual-detail encoding scheme disclosed herein.

[0047] The blended perimeters 204(1), 204(2) make the boundaries between the sub-regions 142(1), 142(2) and the magnified copies 202(1), 202(2) of the frames less distinct. That is, without blending the perimeters 204(1), 204(2) of the sub-regions, the user 106 might notice different boundaries (or borders) around the sub-regions 142(1), 142(2) in the presented images 200(1), 200(2). The blending at the perimeters 204(1), 204(2) of the sub-regions 142(1), 142(2) can include a fade between the overlapping pixel data of the two regions of each image 200(1), 200(2). For example, the blending can involve interpolating between the pixel values of the sub-regions 142(1), 142(2) and the overlapping pixel values of the magnified copies 202(1), 202(2) of the frames, where the interpolation is closer to the pixel values of the sub-regions 142(1), 142(2) at points closer to the center of the respective images 200(1), 200(2), and for points farther from the image center, the interpolation gradually changes to be closer to the pixel values of the magnified copies 202(1), 202(2) of the frames at points farther from the image center. Certain shapes of the sub-regions 142(1), 142(2) can result in more or less blending at the perimeters 204(1), 204(2). The final result of the image reconstruction of each image 200 is an image 200 with three regions, which include a clear and distinct sub-region 142, a somewhat blurred region outside the sub-region 142, and a region between the two that is blended to make the image 200 look smooth (i.e., the user 106 cannot discern that the image 200 has three different regions).

[0048] Figure 3A shows a first exemplary packaging layout 300A of the 2D pixel array 138 described above, while Figure 3B shows a second exemplary packaging layout 300B of the 2D pixel array 138. These are merely exemplary packaging layouts 300, and other packaging layouts can be implemented. In Figure 3AIn the example of, the packaging layout 300A represents a vertical packaging layout 300A that has pixels representing a first copy 142(1) of a sub-region representing a scene at the top of the 2D pixel array 138, followed by pixels representing a first copy 140(1) of a frame (scaled down to a reduced resolution) below the pixels representing the first copy 142(1) of the sub-region, followed by pixels representing a second copy 142(2) of the sub-region below the pixels representing the first copy 140(1) of the frame, and pixels representing a second copy 140(2) of the frame (scaled down to a reduced resolution) at the bottom of the 2D pixel array 138. This vertical packaging layout 300A may allow the host computer 102 to transmit packets carrying some of the encoded pixel data in the encoded pixel data while other pixel data is still being encoded. For example, using the encoder of the GPU 130, the host computer 102 may encode the pixels of the first copy 142(1) of the sub-region and may start sending the encoded pixel data of the first copy 142(1) of the sub-region while the pixels representing the first copy 140(1) of the frame are being encoded. This may reduce the waiting time for sending the encoded pixel data compared to waiting for the entire 2D pixel array 138 to be encoded before sending the encoded pixel data to the HMD 104. Additionally, it should be understood that Figure 3A The sorting of the four sub-arrays of the 2D array 138 shown in is exemplary. For example, for four sub-arrays, there may be up to 24 different vertical packaging layouts and 24 different sorting permutations.

[0049] Figure 3A It is also shown that each of the four groups of pixels in the 2D pixel array 138 is sized M×N pixels. In other words, the first copy 142(1) of the sub-region is a rectangle that has M pixels in the horizontal dimension and N pixels in the vertical dimension. As Figure 3A shown, the first copy 140(1) of the frame is scaled down to a rectangle that has M pixels in the horizontal dimension and N pixels in the vertical dimension. The same is true for the second copy 142(2) of the sub-region and the second copy 140(2) of the frame. In some examples, M is equal to N. In some examples, M is different from N. In the example where M is equal to N, M and N can be 500 pixels, but this is just one exemplary size that can be achieved. Thus, and as mentioned above, the 2D pixel array 138 can be 500×3200 pixels, and the number of pixels in this case is much less than the application-rendered frames.

[0050] Figure 3BThe packaging layout 300B shown in [description] is another possible layout of multiple groups of pixels in the 2D pixel array 138. The exemplary packaging layout 300B has pixels representing the first copy 140(1) of the frame (scaled down to a reduced resolution) at the top of the 2D pixel array 138, followed by pixels representing the second copy 140(2) of the frame (scaled down to a reduced resolution) below the pixels of the first copy 140(1) of the frame, and has pixels representing the first copy 142(1) and the second copy 142(2) of the sub-region in a side-by-side arrangement at the bottom of the 2D pixel array 138. Similar to Figure 3A as shown, the sub-arrays 142(1) and 142(2) in the packaging layout 300B can have M pixels in the horizontal dimension and N pixels in the vertical dimension.

[0051] The processes described herein are shown as a collection of blocks in a logical flowchart, which represent a sequence of operations that can be implemented in hardware, software, firmware, or a combination thereof (i.e., logic). In the context of software, the blocks represent computer-executable instructions that, when executed by one or more processors, perform the operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific abstract data types. The order of description of the operations is not intended to be construed as a limitation, and any number of the described blocks can be combined in any order and / or in parallel to implement these processes.

[0052] Figure 4 A flowchart of an exemplary process 400 for implementing dual-detail encoding in the distributed display system 100 according to an embodiment disclosed herein is shown. For discussion purposes, the process 400 is described with reference to the previous figures.

[0053] At 402, the logic of the host computer 102 can execute an application 132 (e.g., a graphics rendering application) thereon, such as a video game 132(1), whose task is to render one frame in a series of frames to create visual video game content to be displayed on a display device (e.g., the HMD 104). The next blocks 404 - 438 of the process 400 can represent the sub-operations involved in rendering one frame in a series of frames and presenting the corresponding image on the display panel 108 of the HMD 104.

[0054] At 404, the HMD 104 may transmit data 122 to the host computer 102 communicatively coupled to the HMD 104, which data 122 the host computer 102 will use to render frames. In the case of the HMD 104, the data transmitted at block 404 may include head tracking data generated by the head tracking system 118, eye tracking data generated by the eye tracking system 120, and / or other data. In the case of another type of display device, such as a portable gaming device having a display, the data 122 transmitted at block 404 may include control input data (e.g., data corresponding to controls (e.g., buttons) pressed, touched, or otherwise manipulated on the gaming device). In the case of a handheld controller, at block 404, position controller data and / or auxiliary input data (e.g., touch sensing data, finger tracking data, etc.) may be transmitted to the host computer 102. The position data of one or more handheld controllers may be forward predicted to the position the controller will be in when the frame is displayed to the user 106. For a particular implementation where the host computer 102 is wirelessly coupled to the HMD 104, at block 404, the data 122 may be wirelessly sent from the HMD 104 to the host computer 102. Although any suitable wireless communication protocol may be used, in some examples, the 802.11 protocol may be used to send the data 122, 136 between the HMD 104 and the host computer 102.

[0055] At 406, the host computer 102 may receive the data 122 from the display device (e.g., the HMD 104). Although the data 122 may be received in various ways depending on the particular implementation (e.g., wirelessly, via a wired connection, via a wide area network, etc.), in at least one implementation, at block 406, the data may be received wirelessly via the communication interface 134 of the host computer 102.

[0056] At 408, the application 132 executing on the host computer 102 may render a frame of the scene at a first resolution. In some examples, the frame is rendered based on a predicted illumination time representing the time during which the light emitting elements of the display panel 108 of the display device (e.g., the HMD 104) will be illuminated within a given frame. That is, the logic of the host computer 102 may determine the time at which photons associated with the image presented in the given frame will actually reach the eyes of the user 106. This predicted illumination time is a future time (e.g., 20 ms to 40 ms in the future), as it takes time for the application 132 to generate the pixel data for the frame, and it also takes time for the pixel data to be encoded and sent from the host computer 102 to the HMD 104. It also takes time to decode, modify, and scan out the pixel data on the HMD 104 before the corresponding image is finally presented on the display panel 108 of the display device (e.g., the HMD 104).

[0057] In some examples, at block 408, frames are rendered based on a predicted pose that the HMD 104 will be in at a predicted illumination time. For example, the head tracking system 118 of the HMD 104 may be configured to track up to six degrees of freedom of the HMD 104 (e.g., 3D position, roll, pitch, and yaw), which degrees of freedom may be communicated in the data 122 to the host computer 102 to determine the predicted pose of the HMD 104 (e.g., taking into account predicted head movements that result in future poses of the HMD 104). Thus, at block 408, pose data indicating the predicted pose may be provided to the application 132 to render the frames. For example, the application 132 may call a function to receive the pose data, and in response to the function call, the requested pose data (predicted for a target illumination time of a given frame and predicted at least in part based on head tracking data received from the HMD 104) may be provided to the application 132 such that the application 132 may render the given frame according to the pose data, which corresponds to a virtual camera pose for rendering the scene. The rendering at block 408 may involve the application 132 generating pixel data for the frame rendered at a first resolution (e.g., a resolution of 2.5k as described above). As described herein, the pixel data generated at block 408 may include pixel values for individual pixels in a pixel array.

[0058] At 410, if a sub-region of the scene is not pre-determined (or fixed), the process 400 may follow the "no" route from block 410 to block 412, where the logic of the host computer 102 determines the sub-region based at least in part on eye tracking data received from a display device (e.g., the HMD 104). At block 404, this eye tracking data may have been communicated in the data 122, and it may indicate where the user 106 will be looking at the predicted illumination time of the frame. Thus, if it is predicted that the user 106 is looking at the center of the display panel 108 of the HMD 104, at block 412, the sub-region may be determined as the center sub-region (or the center) of the scene. If it is predicted that the user 106 is looking at the left side of the display panel 108 of the HMD 104, at block 412, the sub-region may be determined as a left-of-center sub-region in the scene. If the sub-region of the scene is pre-determined (i.e., along the "yes" route from block 410), or after the sub-region is determined at block 412, the process 400 may proceed to block 414.

[0059] At 414, the logic of host computer 102 may generate a 2D pixel array 138 of the frame. As shown in sub - boxes 416 and 418, the 2D pixel array 138 may include two copies 142(1), 142(2) of a sub - region of the scene and two copies 140(1), 140(2) of the frame scaled down to a second resolution that is less than the first resolution at which the frame is rendered at block 408. For example, the 2D pixel array 138 may include a first pixel (e.g., a first group of pixels / a first pixel sub - array) representing the first copy 140(1) of the frame scaled down to the second reduced resolution, a second pixel (e.g., a second group of pixels / a second pixel sub - array) representing the second copy 140(2) of the frame scaled down to the second reduced resolution, a third pixel (e.g., a third group of pixels / a third pixel sub - array) representing the first copy 142(1) of the sub - region of the scene, and a fourth pixel (e.g., a fourth group of pixels / a fourth pixel sub - array) representing the second copy 142(2) of the sub - region. As described herein, the pixels in the 2D array 138 may be arranged in any suitable packing layout (such as packing layout 300A, packing layout 300B, or any other suitable packing layout). In the vertical packing layout 300A, the third pixel is at the top of the 2D array 138, the first pixel is below the third pixel, the fourth pixel is below the first pixel, and the second pixel is at the bottom of the 2D pixel array, but this is merely an exemplary packing layout 300A that may be implemented at block 414.

[0060] As described above, the pixels of the respective copies 142(1), 142(2) of the sub - region in the 2D array 138 may correspond one - to - one with the original pixels of the frame rendered at block 408. In other words, the respective copies 142(1), 142(2) of the sub - region in the 2D array 138 may be exact copies of the sub - region of the rendered image of the frame without any resampling (e.g., without scaling down) of the applied rendered sub - region. In an example where the copies 142(1), 142(2) of the sub - region in the 2D array 138 are scaled down, the scaling down of the sub - region may be performed by idealized sampling, such as by scaling down the sub - region according to an integer ratio (e.g., 1:2), as described above.

[0061] At 420, the host computer 102 may encode the 2D pixel array 138 to obtain the encoded pixel data for the frame. In some embodiments, encoding and / or decoding may be performed at least partially in hardware (e.g., in silicon built into the respective GPU and / or CPU of the device). Thus, at block 420, the host computer 102 may use the encoder hardware of the GPU 130 to encode the 2D pixel array 138. In some examples, the encoding performed at block 420 may include compressing and / or serializing the data to be transmitted to a display device (e.g., the HMD 104) to render an image associated with the frame. In some embodiments, the encoding performed at block 420 involves changing the format of the pixel data. For example, the encoding at block 420 may use a video coding standard such as High Efficiency Video Coding (HEVC) and / or HEVC extensions (which are sometimes referred to as H.265 and / or MPEG-H Part 2). HEVC is an example of a standard that may be used at block 420, but it should be understood that other suitable video coding / compression standards may be used at block 420, such as, for example, H.264 or VP9. HEVC uses motion estimation to compress the pixel data, with the goal of transmitting less data than the uncompressed pixel data in a system with bandwidth limitations, yet the compressed data is sufficient to approximate the original uncompressed data at the receiving end (e.g., at the HMD 104). In some examples, the operation of generating the 2D pixel array 138 for the frame performed at block 414 is performed as part of the encoding operation at block 420.

[0062] In some examples, the encoding of block 420 may include allocating bits to encode each group of pixels / each pixel sub-array in the 2D array 138 either uniformly or variably. For example, the encoder of the GPU 130 may allocate fewer bits to encode the first pixels (e.g., the first group of pixels / first pixel sub-array) of the first copy 140(1) of the frame downscaled to a second reduced resolution and the second pixels (e.g., the second group of pixels / second pixel sub-array) of the second copy 140(2) of the frame downscaled to the second reduced resolution. At the same time, the encoder of the GPU 130 may allocate more bits to encode the third pixels (e.g., the third group of pixels / third pixel sub-array) of the first copy 142(1) of the sub-region of the scene and the fourth pixels (e.g., the fourth group of pixels / fourth pixel sub-array) of the second copy 142(2) of the sub-region. In other words, a first quantity of bits may be allocated to encode the first and second pixels, and a second larger quantity of bits may be allocated to encode the third and fourth pixels, such that more data is used to encode the copies 142(1), 142(2) of the sub-region compared to the copies 140(1), 140(2) of the downscaled frames. In some examples, the logic of the host computer 102 may determine a subset of the pixels of the copies 140(1), 140(2) of the downscaled frames that are occluded by the pixels of the sub-regions 142(1), 142(2) in the images 200(1), 200(2), and at block 420, the encoder may allocate zero bits to the subset of the pixels in the images 200(1), 200(2) that will be occluded.

[0063] At 422, the host computer 102 may send (or transmit) the encoded pixel data to the display device (e.g., the HMD 104) via the communication interface 134. In some examples, at block 422, one or more data packets carrying the encoded pixel data are wirelessly sent via the transceiver of the host computer 102. In some examples, the transmission at block 422 may use a transmission data rate of approximately 100 Mbps.

[0064] At 424, the display device (e.g., the HMD 104) may receive the encoded pixel data of the frame (of the scene) from the host computer 102 via the communication interface 124. In some examples, at block 424, one or more data packets carrying the encoded pixel data are wirelessly received via the transceiver of the display device (e.g., the HMD 104). In some examples, the reception at block 424 may use a reception data rate of approximately 100 Mbps.

[0065] At 426, the logic of the display device (e.g., HMD 104) can decode the encoded pixel data to obtain the 2D pixel array 138 of the frame. In some embodiments, at block 426, the display device (e.g., HMD 104) can use the decoder hardware of the GPU 114 to decode the encoded pixel data. In some examples, the decoding performed at block 426 can include decompressing and / or deserializing the data to render the image associated with the frame.

[0066] At 428, the logic of the display device (e.g., HMD 104) (such as the synthesizer 116) can modify the pixels in the 2D array 138 to obtain modified pixel data. As shown in sub-blocks 430 and 432, at least some of the modifications in block 428 can be for reconstructing the images 200(1), 200(2) of the frame. Generally, at block 430, two copies 140(1), 140(2) of the previously scaled-down frame can be enlarged to obtain two enlarged copies 202(1), 202(2) of the frame. At 432, based on these enlarged copies 202(1), 202(2) of the frame and two copies 142(1), 142(2) of the sub-region and the blending at the perimeters 204(1), 204(2) of the sub-regions 142(1), 142(2), the images 200(1), 200(2) are generated. For example, to reconstruct the right image 200(2) (e.g., for the right display panel 108(2) of a pair of stereoscopic display panels 108), the synthesizer 116 can, at block 430, at least partially based on the corresponding pixels of the 2D array 138, enlarge the first copy 140(1) of the frame to obtain the first enlarged copy 202(1) of the frame, and can, at block 432, at least partially based on the first enlarged copy 202(1) of the frame and the pixels of the 2D array 138 representing the first copy 142(1) of the sub-region of the scene, generate the right image 200(2), where a subset of the pixels at the perimeter 204(1) of the sub-region is blended in the right image 200(2). To reconstruct the left image in the two images, the synthesizer 116 can, at block 430, at least partially based on the corresponding pixels of the 2D array 138, enlarge the second copy 140(2) of the frame to obtain the second enlarged copy 202(2) of the frame, and can generate the left image 200(1) at least partially based on the second enlarged copy 202(2) of the frame and the pixels of the 2D array 138 representing the second copy 142(2) of the sub-region, where a subset of the pixels at the perimeter 204(2) of the sub-region is blended in the left image 200(1).

[0067] As shown in sub - box 434, other modifications may be performed at box 428, such as modifications that compensate for geometric distortion, color aberration, reprojection, etc. In one example, other modifications applied to pixels at box 434 may include reprojection adjustments that are at least partially based on the updated pose that the HMD 104 will be in at the illumination time of a given frame. For example, the logic of the HMD 104 may determine the updated pose that the HMD 104 will be in at the illumination time of a given frame at least partially based on updated head - tracking data generated by the head - tracking system 118 of the HMD 104, where the illumination time represents the time during which the light - emitting elements of the display panel 108 of the HMD 104 will be illuminated within the given frame. Since this determination is closer in time to the illumination time of the given frame, the pose prediction made by the HMD 104 may be more accurate (e.g., have less error) than the pose prediction determined by the host computer 102 prior to the illumination time. Thus, the display device (e.g., HMD 104) may compare the original predicted pose (which is based on the head - tracking data received by the host computer 102 at box 406) with the updated pose, which may reveal the amount (or difference) between the compared poses, and the reprojection adjustment may include rotation calculations to compensate for that amount (e.g., by shifting and / or rotating the pixel data in one way or another, depending on the amount between the two pose determinations). In other words, the head - tracking data is used to “correct” the projection of the image rendered onto the display panel 108.

[0068] At 436, the logic (e.g., the compositor 116) of the display device (e.g., HMD 104) may output the modified pixels (or modified pixel data) to the frame buffer of the display device. Since the modified pixels represent Figure 2 the images 200(1), 200(2) depicted in, this is also described herein as outputting the images 200(1), 200(2) to the frame buffer. For a display device (e.g., HMD 104) having a pair of display panels 108(1) and 108(2), the modified pixel data may correspond to a frame representing a pair of images to be displayed on the pair of display panels 108(1) and 108(2), and may be output to the stereoscopic frame buffer accordingly.

[0069] In some examples, images 200(1), 200(2) may be output to the frame buffer in a single write operation or multiple write operations. In a single-write implementation, the pixel values output to the frame buffer are output in a single pass and the pixel values are not overwritten. This single-write implementation may involve the logic of the display device (e.g., HMD 104) (e.g., compositor 116) performing a lookup for each pixel to determine the pixel value to output based on the pixel modifications made at block 428. In a multi-write implementation, in a first pass, the compositor 116 may output pixel values corresponding to the magnified copies 202(1), 202(2) of the frame to the frame buffer, even those pixel values that will ultimately be occluded or blended in the overlap with sub-regions 142(1), 142(2); then, in a second pass, the compositor 116 may output pixel values corresponding to the copies of sub-regions 142(1), 142(2) and the blending at the periphery of sub-regions 142(1), 142(2), which may overwrite some of the previously written pixel values from the first pass. In some examples, the blending is completed in a third write operation (or third pass).

[0070] At 438, the logic of the display device (e.g., HMD 104) may cause images 200(1), 200(2) to be presented on the respective display panels 108(1), 108(2) of the display device (e.g., HMD 104) based on the pixel values (or pixel data) output to the frame buffer at block 436. This may involve scanning the pixel data out to the display panels 108(1) and 108(2) of the display device (e.g., HMD 104) and illuminating the light-emitting elements of the display panels 108(1), 108(2) to illuminate the pixels on the display panels 108(1), 108(2).

[0071] Thus, process 400 is an exemplary technique for implementing a dual-detail encoding scheme in distributed display system 100. This dual-detail encoding scheme reduces the amount of data to be sent from host computer 102 to the display device (e.g., HMD 104) such that the pixel data can be streamed in a timely manner to display the corresponding images on the display device (e.g., HMD 104) and in this way enable user 106 to perceive the images as high-fidelity images on the display device (e.g., HMD 104). For example, even though the portions of the displayed images 200(1), 200(2) outside of sub-regions 142(1), 142(2) may be presented at a relatively low quality (e.g., low resolution), the gaze of user 106 may be directed to sub-regions 142(1), 142(2) of the scene where the image quality is relatively high such that to user 106 it appears as if the images have high quality.

[0072] An alternative technique for transmitting pixel data in a distributed display system 100 under a data rate constraint is to distort the application-rendered image by magnifying the central portion of each image such that the visual information at the center of the corresponding image is scaled to a larger size; encode the distorted image to obtain encoded pixel data; and transmit the encoded pixel data to a display device (e.g., HMD 104). At the display device (e.g., HMD 104), decode the encoded pixel data to obtain the distorted image, and invert the distortion at the HMD 104 to generate an image that is presented on display panels 108(1), 108(2). This alternative technique is referred to herein as the "foveation" technique. Compared to the dual-detail encoding technique described herein, the foveation technique provides a single level of detail for each eye (as opposed to two levels of detail) because the 2D pixel array generated based on the aforementioned distortion of the application-rendered image includes one copy of the frame distorted by magnifying the central portion of the scene for each eye. Application of the foveation technique can result in a 2D pixel array having fewer pixels than the application-rendered frame, making it suitable for transmitting pixel data under a data rate constraint such as 100 Mbps. For example, a video game executed on host computer 102 can render frames at a resolution of 2.5k pixels per eye (i.e., an image having 2500×2500 pixels), which corresponds to 12,500,000 pixels on two images for an HMD 104 having two display panels 108(1), 108(2), and the foveation technique described above can result in a 2D pixel array of approximately 1500×1500 pixels per eye, which corresponds to 4,500,000 pixels to be sent to the HMD 104 within a given frame. Thus, the foveation solution can reduce the amount of data to be transmitted in the distributed display system 100, but the dual-detail encoding solution described herein can reduce the amount of data to a greater extent. Additionally, the foveation solution may be subject to resampling artifacts (such as aliasing, blurring, etc.) on the display device.

[0073] Figure 5A , Figure 5B and Figure 5C illustrates an alternative arrangement of a system for streaming data from a host computer 102 to a display device according to embodiments disclosed herein. Briefly refer to Figure 1, an exemplary embodiment is where a host computer 102 and a display device in the form of an HMD 104 worn by a user 106 are juxtaposed in an environment. For example, when user 106 is using HMD 104 in a house, host computer 102 can be located in user 106's house, regardless of whether host computer 102 is in the same room or a different room as HMD 104. Alternatively, a host computer 102 in the form of a portable computing device (e.g., a tablet computer or a laptop computer) can be carried (e.g., in a backpack on the back of user 106), allowing for greater mobility. For example, when using such a system, user 106 can be located in a park.

[0074] Figure 5A Another alternative embodiment is shown, where host computer 102 represents one or more server computers located at a geographically remote location relative to HMD 104. In this case, HMD 104 can be communicatively coupled to host computer 102 via an access point (AP) 500 (such as a wireless AP (WAP), a base station, a USB dongle, etc.). In an illustrative example, data is exchanged (e.g., streamed) between host computer 102 and HMD 104 via AP 500, such as streaming data over the Internet. In this embodiment, host computer 102 and / or HMD 104 can implement one or more of the dual-detail encoding techniques described herein.

[0075] Figure 5B Yet another alternative embodiment is shown, where host computer 102 is communicatively coupled to HMD 104 via an intermediate computing device 504 (such as a laptop computer or a tablet computer). Figure 5A and Figure 5B The difference between Figure 5A is that the AP 500 in Figure 5B can be used as a data routing device for server computer 102 located remotely relative to HMD 104, while the intermediate computing device 504 in

[0076] Figure 5C can be used as a data routing device for host computer 102 juxtaposed with HMD 104 in the same environment. The intermediate computing device 504 can even perform a portion of the rendering workload (e.g., reconstructing pixel data and / or modifying pixel data, as described herein), such that HMD 104 can remain as "lightweight" as possible. Figure 5CIn the setting, when user 106 manipulates the controls of the portable gaming device 502 with his / her hand, the portable gaming device 502 can transmit control input data to the host computer 102, and the host computer 102 can generate pixel data at least partially based on the control input data and send the encoded pixel data to the portable gaming device 502, as described herein mainly with respect to the HMD 104. Thus, traditional game streaming systems challenged by adverse environments on most home networks can benefit from the dual-detail encoding techniques and systems described herein. For example, a home personal computer running a game platform / application 132 can allow continuous playback of video streams and game interactions with significant data rate limitations. The use of game streaming services according to the techniques and systems described herein can achieve a higher quality gaming experience in an environment with limited bandwidth. Additionally, although these techniques are mainly described herein with reference to VR games, some or all of these techniques can be equivalently applied to the 2D / 3D gaming field.

[0077] Figure 6 Exemplary components of a wearable device such as the HMD 104 (e.g., VR headset) and the host computer 102 in which the techniques disclosed herein can be implemented according to embodiments disclosed herein are shown. However, it should be understood that the relevant components described with respect to the HMD 104 can be implemented in other types of display devices (e.g., the portable gaming device 502), and only the irrelevant components can be omitted from those other types of display devices. The HMD 104 can be implemented as a connected device communicatively coupled to the host computer 102 during operation and / or as a stand-alone device. In either operating mode, the HMD 104 will be worn by the user 106 (e.g., on the head of the user 106). In some embodiments, the HMD 104 can be head-mounted, such as by allowing the user 106 to secure the HMD 104 to his / her head using a securing mechanism (e.g., an adjustable strap) sized to fit around the head of the user 106. In some embodiments, the HMD 104 includes a VR, AR, and / or MR headset that includes a near-eye display. Thus, the terms "wearable device", "wearable electronic device", "VR headset", "AR headset", "MR headset", and "head-mounted display (HMD)" can be used interchangeably herein to refer to Figure 6 device 104. However, these types of devices are merely examples of the HMD 104, and it should be understood that the HMD 104 can be implemented in a variety of other form factors. It should also be understood that Figure 6Some or all of the components shown may be implemented on the HMD 104. Thus, in some embodiments, a subset of the components shown as being implemented in the HMD 104 may be implemented on the host computer 102 or on another computing device separate from the HMD 104.

[0078] In the specific embodiment shown, the HMD 104 includes the aforementioned processor 110, which may include one or more GPUs 114; and a memory 112 that stores a compositor 116 executable by the processor 110, the display panel 108, the head tracking system 118, the eye tracking system 120, and the communication interface 124.

[0079] The HMD 104 may include a single display panel 108 or multiple display panels 108, such as a left display panel 108(1) and a right display panel 108(2) of a pair of stereoscopic display panels. One or more display panels 108 of the HMD 104 may be used to present a series of image frames (referred to herein as "frames") viewable by a user 106 wearing the HMD 104. It should be understood that the HMD 104 may include any number of display panels 108 (e.g., more than two display panels, a pair of display panels, or a single display panel). Thus, the term "display panel" used herein in the singular may refer to a display panel 108 in a pair of display panels of a two-panel HMD 104, or may refer to a single display panel 108 of an HMD 104 having any number of display panels (e.g., a single-panel HMD 104 or a multi-panel HMD 104). In a two-panel HMD 104, a stereoscopic frame buffer may render pixels on the two display panels of the HMD 104. In a single-panel HMD 104, the HMD 104 may include a single display panel 108 and a pair of lenses, one lens for each eye, to view corresponding images displayed on a portion of the display panel 108.

[0080] The display panel 108 of the HMD 104 may use any suitable type of display technology, such as an emissive display that uses light-emitting elements (e.g., light-emitting diodes (LEDs)) or laser illumination to emit light during the presentation of frames on the display panel 108. As an example, the display panel 108 of the HMD 104 may include a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an inorganic light-emitting diode (ILED) display, or any other suitable type of display technology for HMD applications.

[0081] The display panel 108 of the HMD 104 can operate at any suitable refresh rate (such as a 90 Hertz (Hz) refresh rate, a 120 Hz refresh rate, etc.), which can be a fixed refresh rate or a variable refresh rate that varies dynamically within a range of refresh rates. The "refresh rate" of a display is the number of times the display redraws the screen per second. If a fixed refresh rate is used, the number of frames displayed per second may be limited by the refresh rate of the display. Thus, a series of frames can be processed (e.g., rendered) and displayed as images on the display such that a single frame in the series of frames is displayed with each screen refresh. That is, in order to present a series of images on the display panel 108, the display panel 108 can transition frame by frame at the refresh rate of the display in the series of frames, thereby lighting up pixels with each screen refresh.

[0082] The display system of the HMD 104 can implement any suitable type of display driving scheme, such as a global blinking type display driving scheme, a scrolling band type display driving scheme, or any other suitable type of display driving scheme. In a global blinking type display driving scheme, the array of light-emitting elements of the display is simultaneously illuminated with each screen refresh, thereby globally blinking at the refresh rate. In a scrolling band type display driving scheme, during an illumination period, individual subsets of the light-emitting elements of the display can be independently and sequentially lit in a scrolling illumination band. These types of display driving schemes can be implemented with individually addressable light-emitting elements. If the pixel array and the light-emitting element array on the display panel 108 are arranged in rows and columns (but not necessarily with each light-emitting element corresponding to a pixel), the light-emitting elements of individual rows and / or individual columns can be sequentially addressed, and / or groups of successive multiple rows and / or groups of successive multiple columns of light-emitting elements can be sequentially addressed for a scrolling band type display driving scheme.

[0083] Generally, as used herein, "lighting a pixel" means lighting the light-emitting element corresponding to that pixel. For example, an LCD lights the light-emitting elements of a backlight to light the corresponding pixels of a display. Additionally, as used herein, a "subset of pixels" may include a single pixel or multiple pixels (e.g., a group of pixels). To drive display panel 108, HMD 104 may include a display controller, display driver circuitry, and similar electronics for driving display panel 108. The display driver circuitry may be coupled to the array of light-emitting elements of display panel 108 via conductive paths (such as metal traces) on a flexible printed circuit. In an example, the display controller may be communicatively coupled to the display driver circuitry and configured to provide signals, information, and / or data to the display driver circuitry. The signals, information, and / or data received by the display driver circuitry may cause the display driver circuitry to light the light-emitting elements in a particular manner. That is, the display controller may determine which light-emitting element(s) will be lit, when the element(s) will be illuminated, and the level of light output that the element(s) will emit, and may transmit appropriate signals, information, and / or data to the display driver circuitry to achieve that goal.

[0084] Computer-readable medium 112 may store additional functional modules executable on processor 110, but the same functionality may alternatively be implemented in hardware, firmware, or as an SOC and / or other logic. For example, operating system module 600 may be configured to manage the hardware within HMD 104 and be coupled to the HMD to facilitate other modules. Additionally, in some cases, HMD 104 may include one or more applications 602 stored in memory 112 or otherwise accessible by HMD 104. For example, applications 602 may include, but are not limited to, video game applications (e.g., basic video games with less computationally intensive graphics), video playback applications (e.g., applications that access a library of video content stored on HMD 104 and / or in the cloud), etc. HMD 104 may include any number or type of applications 602 and is not limited to the specific examples described herein.

[0085] Typically, the HMD 104 has an input device 604 and an output device 606. The input device 604 may include control buttons. In some embodiments, one or more microphones may be used as the input device 604 to receive audio input, such as user voice input. In some embodiments, one or more cameras or other types of sensors (e.g., inertial measurement unit (IMU)) may be used as the input device 604 to receive gesture input, such as the hand and / or head movement of the user 106. In some implementations, additional input devices 604 may be provided in the form of a keyboard, keypad, mouse, touch screen, joystick, etc. In other implementations, the HMD 104 may omit a keyboard, keypad, or other similar form of mechanical input. Instead, the HMD 104 may be implemented in a relatively simple form with an input device 604, a network interface (wireless or wired), a power supply, and processing / memory capabilities. For example, a limited set of one or more input components (e.g., dedicated buttons for startup configuration, power on / off, etc.) may be employed such that the HMD 104 can then be used. In one embodiment, the input device 604 may include control mechanisms, such as basic volume control buttons for increasing / decreasing volume and power and reset buttons.

[0086] The output device 606 may include a display panel 108, which may include one or more display panels 108 (e.g., a pair of stereoscopic display panels 108), as described herein. The output device 606 may also include, but is not limited to, light elements (e.g., LEDs), vibrators that produce a tactile sensation, speakers (e.g., headphones), etc. There may also be simple light elements (e.g., LEDs) to indicate status (such as when powered on).

[0087] The HMD 104 may also include a communication interface 124, which includes but is not limited to one or more antennas 910 (e.g., antennas of a transceiver) to facilitate wireless connection to a network and / or a second device (such as the host computer 102 described herein). The communication interface 124 may implement one or more wireless technologies among various wireless technologies, such as Wi-Fi, Bluetooth, radio frequency (RF), etc. It should be understood that the communication interface 124 of the HMD 104 may also include physical ports to facilitate wired connection to a network and / or a second device (such as the host computer 102).

[0088] The HMD 104 may also include an optical subsystem 612 that uses one or more optical elements to direct light from the display panel 108 to the user's eyes. The optical subsystem 612 may include various types and combinations of different optical elements, including but not limited to, such as an aperture, lenses (e.g., Fresnel lenses, convex lenses, concave lenses, etc.), filters, and the like. In some embodiments, one or more of the optical elements in the optical subsystem 612 may have one or more coatings, such as an anti-reflection coating. The magnification of the image light by the optical subsystem 612 allows the display panel 108 to be physically smaller, lighter, and consume less power than a larger display. Additionally, the magnification of the image light may increase the field of view (FOV) of the displayed content (e.g., an image). For example, the FOV of the displayed content causes the displayed content to be presented using nearly all (e.g., 120 degrees to 150 degrees diagonal) of the user's FOV and, in some cases, the entire FOV of the user. AR applications may have a narrower FOV (e.g., approximately 40 degrees FOV). The optical subsystem 612 may be designed to correct one or more optical errors, such as but not limited to barrel distortion, pincushion distortion, longitudinal chromatic aberration, lateral chromatic aberration, spherical aberration, coma aberration, field curvature, astigmatism, and the like. In some embodiments, the content provided to the display panel 108 for display is pre-distorted (e.g., by the geometric distortion adjustment and / or chromatic aberration adjustment described herein), and the optical subsystem 612 corrects the distortion when it receives the image light generated based on the content from the display panel 108.

[0089] The HMD 104 may also include one or more sensors 614, such as sensors for generating motion, positioning, and orientation data. These sensors 614 may be or may include gyroscopes, accelerometers, magnetometers, video cameras, color sensors, or other motion, positioning, and orientation sensors. The sensors 614 may also include sub-parts of the sensors, such as a series of active or passive markers that can be viewed externally through a camera or color sensor to generate motion, positioning, and orientation data. For example, a VR headset may include multiple markers on its exterior, such as reflectors or lights (e.g., infrared light or visible light), which can provide one or more reference points for software to interpret when viewed through an external camera or illuminated by a light (e.g., infrared light or visible light) to generate motion, positioning, and orientation data. The HMD 104 may include a light sensor that is sensitive to light (e.g., infrared light or visible light) projected or broadcast by a base station in the environment of the HMD 104.

[0090] In an example, the sensor 614 may include an inertial measurement unit (IMU) 616. The IMU 616 may be an electronic device that generates calibration data based on measurement signals received from an accelerometer, a gyroscope, a magnetometer, and / or other sensors suitable for detecting motion, correcting errors associated with the IMU 616, or some combination thereof. Based on the measurement signals, a motion-based sensor such as the IMU 616 may generate calibration data that indicates an estimated orientation of the HMD 104 relative to an initial orientation of the HMD 104. For example, multiple accelerometers may measure translational motion (front / back, up / down, left / right), and multiple gyroscopes may measure rotational motion (e.g., pitch, yaw, and roll). The IMU 616 may, for example, sample the measurement signals rapidly and calculate the estimated orientation of the HMD 104 based on the sampled data. For example, the IMU 616 may integrate the measurement signals received from the accelerometer over time to estimate a velocity vector and integrate the velocity vector over time to determine the estimated orientation of a reference point on the HMD 104. The reference point is a point that may be used to describe the orientation of the HMD 104. Although the reference point may generally be defined as a point in space, in various embodiments, the reference point is defined as a point within the HMD 104 (e.g., the center of the IMU 616). Alternatively, the IMU 616 provides the sampled measurement signals to an external console (or other computing device) to determine the calibration data. The sensor 614 may include sensors of one or more handheld controllers that are part of the HMD system. Thus, in some embodiments, the controller may send sensor data to the host computer 102, and the host computer 102 may fuse the sensor data it receives from the HMD 104 and the handheld controllers.

[0091] The sensor 614 may operate at a relatively high frequency to provide sensor data at a high rate. For example, the sensor data may be generated at a rate of 1000 Hz (or 1 sensor reading per 1 millisecond). In this way, one thousand readings are taken per second. When the sensor generates so much data at this rate (or a higher rate), the dataset for predicting motion is quite large, even over a relatively short period of time of about a few tens of milliseconds.

[0092] As mentioned, in some embodiments, the sensor 614 may include a light sensor that is sensitive to light emitted by a base station in the environment of the HMD 104 to track the orientation, pose, etc. of the HMD 104 in 3D space. The calculation of the orientation and / or pose may be based on the timing characteristics of the light pulses and the presence or absence of light detected by the sensor 614.

[0093] The HMD 104 may also include the aforementioned head tracking system 118. As described above, the head tracking system 118 may utilize one or more sensors 614 to track the head movements of the user 106, including head rotations. For example, the head tracking system 118 may track up to six degrees of freedom of the HMD 104 (i.e., 3D positioning, roll, pitch, and yaw). These calculations may be performed for each frame in a series of frames such that the application 132 can determine how to render the scene in the next frame based on the head position and orientation. In some embodiments, the head tracking system 118 is configured to generate head tracking data that can be used to predict the future pose (position and / or orientation) of the HMD 104 based on current and / or past data and / or based on the known / implied scan output latency of individual pixel subsets in the display system. This is because the application 132 is required to render a frame before the user 106 actually sees the light on the display panel 108 (and thus sees the image). Thus, the next frame may be rendered based on this future prediction of the head position and / or orientation made at an earlier point in time. The rotation data provided by the head tracking system 118 may be used to determine the direction of rotation of the HMD 104 as well as the amount of rotation of the HMD 104 in any suitable unit of measure. For example, the direction of rotation may be simplified and output in positive or negative horizontal directions and positive or negative vertical directions (which correspond to left, right, up, and down). The amount of rotation may be expressed in degrees, radians, etc. The angular velocity may be calculated to determine the rate of rotation of the HMD 104.

[0094] The HMD 104 may also include the aforementioned eye tracking system 120 that generates eye tracking data. The eye tracking system 120 may include, but is not limited to, a camera or other optical sensors inside the HMD 104 to capture image data (or information) of the user's eyes, and the eye tracking system 120 may use the captured data / information to determine the motion vectors of each eye relative to the HMD 104, the interpupillary distance, the distance between the two eyes, the 3D positioning, including the amplitude of the torsion and rotation (i.e., roll, pitch, and yaw) of each eye, and the gaze direction. In one example, infrared light is emitted within the HMD 104 and reflected from each eye. The reflected light is received or detected by the camera of the eye tracking system 120 and analyzed to extract the eye rotation from the change in the infrared light reflected from each eye. The eye tracking system 120 may use a variety of methods for tracking the eyes of the user 106. Thus, the eye tracking system 120 may track up to six degrees of freedom of each eye (i.e., 3D positioning, roll, pitch, and yaw), and may combine at least a subset of the tracking amounts of the two eyes of the user 106 to estimate the fixation point (i.e., the 3D position or location in the virtual scene that the user is looking at), which may be mapped to a position on the display panel 108 for predicting where the user 106 will look based on a single subset of pixels (e.g., a row of pixels) or a set of consecutive subsets of pixels (e.g., a set of consecutive multiple rows of pixels) of the display panel 108. For example, the eye tracking system 120 may integrate information from past measurements, measurements identifying the head positioning of the user 106, and 3D information describing the scene presented by the display panel 108. Thus, the information on the positioning and orientation of the eyes of the user 106 is used to determine the fixation point in the virtual scene presented by the HMD 104 that the user 106 is looking at, and map that fixation point to a position on the display panel 108 of the HMD 104.

[0095] In the specific implementation shown, the host computer 102 includes the aforementioned processor 126, which may include one or more GPUs 130; and a memory 128 and a communication interface 134 that stores the application 132.

[0096] The memory 128 may also store an operating system module 618 that is configured to manage the hardware within the host computer 102 and is coupled to the host computer to facilitate other modules. The memory 128 may also store a video game client 620 that is configured to execute one or more video games within a video game library 622, such as video game 132(1). The video games within the video game library 622 may be retrieved and executed by loading the video game client 620. In an example, the user 106 may start executing video game 132(1) by loading the video game client 620 and selecting video game 132(1) to play one of the multiple video games that they have purchased and downloaded to the video game library 622. The video game client 620 may allow the user to log in to a video game service using credentials (e.g., user account, password, etc.).

[0097] The host computer 102 may also include a communication interface 134 that includes, but is not limited to, one or more antennas 624 (e.g., antennas of a transceiver) to facilitate a wireless connection to a network and / or a second device (such as the HMD 104). The communication interface 134 may implement one or more of various wireless technologies, such as Wi-Fi, Bluetooth, radio frequency (RF), etc. It should be understood that the communication interface 134 of the host computer 102 may also include physical ports to facilitate a wired connection to a network and / or a second device (such as the HMD 104) or another type of display device (such as the portable gaming device 502).

[0098] Generally, the host computer 102 has an input device 626 and an output device 628. The input device 626 may be a keyboard, keypad, mouse, touch screen, joystick, control buttons, microphone, camera, etc. The output device 628 may include, but is not limited to, a display, light elements (e.g., LEDs), a vibrator that generates a tactile sensation, speakers (e.g., headphones), etc.

[0099] Although the subject matter has been described in language specific to structural features, it should be understood that the subject matter defined in the appended claims need not be limited to the specific features described. Rather, the specific features are disclosed as illustrative forms of implementing the claims.

Claims

1. A host computer, the host computer comprising: a processor; and a memory that stores computer-executable instructions that, when executed by the processor, cause the host computer to: execute a graphics rendering application to render a frame of a scene at a first resolution; generate a two-dimensional (2D) pixel array of the frame, the 2D pixel array comprising: a first pixel that represents a first copy of the frame scaled down to a second resolution that is less than the first resolution; a second pixel that represents a second copy of the frame scaled down to the second resolution; a third pixel that represents a first copy of a sub-region of the scene; and a fourth pixel that represents a second copy of the sub-region; encode the 2D pixel array to obtain encoded pixel data of the frame; and send the encoded pixel data to a display device.

2. The host computer according to claim 1, wherein: the third pixel corresponds one-to-one with the original pixels of the frame representing the sub-region; and the fourth pixel corresponds one-to-one with the original pixels.

3. The host computer according to claim 1, wherein: wherein the third pixel represents the first copy of the sub-region scaled down according to an integer ratio; and wherein the fourth pixel represents the second copy of the sub-region scaled down according to the integer ratio.

4. The host computer according to claim 1, wherein the sub-region is pre-determined.

5. The host computer according to claim 1, wherein the computer-executable instructions, when executed by the processor, further cause the host to determine the sub-region at least in part based on eye-tracking data received from the display device.

6. The host computer according to claim 1, wherein generating the 2D pixel array comprises arranging the pixels of the frame in a vertical packing layout that has: the third pixel at the top of the 2D pixel array; the first pixel below the third pixel; the fourth pixel below the first pixel; and the second pixel at the bottom of the 2D pixel array.

7. The host computer according to claim 1, wherein encoding the 2D pixel array comprises: allocating a first number of bits to encode the first pixel and the second pixel; and allocating a second number of bits to encode the third pixel and the fourth pixel, wherein the first number of bits is less than the second number of bits.

8. The host computer according to claim 1, the host computer further comprising a transceiver, wherein sending the encoded pixel data to the display device comprises wirelessly sending, via the transceiver, one or more data packets carrying the encoded pixel data.

9. A method, the method comprising: rendering, by a graphics rendering application executing on a host computer, a frame of a scene at a first resolution; generating a two-dimensional (2D) pixel array of the frame, the 2D pixel array comprising: A first pixel representing a first copy of the frame at a second resolution that is scaled down to less than the first resolution of the frame; A second pixel representing a second copy of the frame at the second resolution; A third pixel representing a first copy of a sub-region of the scene; and A fourth pixel representing a second copy of the sub-region; Encoding the 2D pixel array to obtain encoded pixel data of the frame; and Sending the encoded pixel data to a display device.

10. The method according to claim 9, wherein: The third pixel corresponds one-to-one with the original pixels of the frame representing the sub-region; and The fourth pixel corresponds one-to-one with the original pixels.

11. The method according to claim 9, wherein the sub-region is pre-determined and corresponds to the center of the scene.

12. The method according to claim 9, the method further comprising determining the sub-region at least in part based on eye tracking data received from the display device.

13. The method according to claim 9, wherein the generation of the 2D pixel array includes arranging the pixels of the frame in a vertical packing layout having: The third pixel at the top of the 2D pixel array; The first pixel below the third pixel; The fourth pixel below the first pixel; and The second pixel at the bottom of the 2D pixel array.

14. The method according to claim 9, wherein the encoding of the 2D pixel array includes: Allocating a first number of bits to encode the first pixel and the second pixel; and Allocating a second number of bits to encode the third pixel and the fourth pixel, wherein the first number of bits is less than the second number of bits.

15. The method according to claim 9, wherein the sending of the encoded pixel data to the display device includes wirelessly sending one or more data packets carrying the encoded pixel data via a transceiver of the host computer.

16. The method according to claim 9, wherein: The sub-region is rectangular, having M pixels in the horizontal dimension and N pixels in the vertical dimension; and The first copy of the frame and the second copy of the frame are each scaled down to M pixels in the horizontal dimension and N pixels in the vertical dimension.

17. A display device, the display device comprising: At least one display panel; A processor; And A memory storing computer-executable instructions that, when executed by the processor, cause the display device to: Receive encoded pixel data of a frame of a scene from a host computer; Decode the encoded pixel data to obtain a two-dimensional (2D) pixel array of the frame, the 2D pixel array including: A first pixel representing a first copy of the frame at a second resolution that is less than a first resolution at which the frame is rendered; A second pixel representing a second copy of the frame at the second resolution; a third pixel representing a first copy of a sub-region of the scene; and a fourth pixel representing a second copy of the sub-region; enlarging the first copy of the frame at least in part based on the first pixel to obtain a first enlarged copy of the frame; enlarging the second copy of the frame at least in part based on the second pixel to obtain a second enlarged copy of the frame; generating a first image at least in part based on the first enlarged copy of the frame and the third pixel, wherein a subset of the third pixel at the periphery of the sub-region is blended in the first image; generating a second image at least in part based on the second enlarged copy of the frame and the fourth pixel, wherein a subset of the fourth pixel at the periphery of the sub-region is blended in the second image; and presenting the first image and the second image on the at least one display panel.

18. The display device according to claim 17, wherein the computer-executable instructions, when executed by the processor, further cause the display device to output the first image and the second image to a frame buffer in a single write operation.

19. The display device according to claim 17, wherein the computer-executable instructions, when executed by the processor, further cause the display device to output the first image and the second image to a frame buffer in multiple write operations.

20. The display device according to claim 17, wherein the computer-executable instructions, when executed by the processor, further cause the display device to transmit eye-tracking data to the host computer for use by the host computer in determining the sub-region.

Images