Splitting rendering between a head-mounted display (HMD) and a host computer

Abstract

A rendering workload for a single frame can be divided between a head-mounted display (HMD) and a host computer that is executing an application program. To divide the rendering workload for a frame, the HMD can send head tracking data to the host computer, which can use the head tracking data to generate pixel data associated with the frame and additional data in addition to the pixel data. The additional data can include, but is not limited to, pose data, depth data, motion vector data, and / or additional pixel data. The HMD can receive the pixel data and at least some of the additional data, determine an updated pose of the HMD, and apply a re-projection adjustment to the pixel data based on the updated pose and the received additional data to obtain modified pixel data, which is used to render an image on a display panel of the HMD.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This PCT application claims priority to U.S. patent application serial number 16 / 717,940, filed on December 17, 2019, entitled “SPLIT RENDERING BETWEEN AHEAD-MOUNTED DISPLAY (HMD) AND A HOST COMPUTER,” which is hereby incorporated by reference in its entirety. Background Art

[0003] Virtual reality (VR) systems are used within the video game industry, among other things. A traditional VR system setup includes a VR headset that is physically tethered to a host computer via a wired data connection. In this conventional setup, the host computer executes a graphics-based application, such as a video game, where most (if not all) graphics rendering operations are handled by the host computer, and the VR headset simply displays pixel data received from the host computer. This setup takes advantage of the high computing power of the host computer and the low latency of the wired connection to display high-quality images on a lightweight VR headset that functions much like a "thin client" device in terms of the headset's graphics processing capabilities. However, because such VR headsets are physically connected to the host computer, the user's mobility while using the VR headset is limited. Furthermore, due to the need to connect and disconnect cables, both setting up and tearing down such VR systems is more difficult than necessary.

[0004] On the other hand, integrated (or standalone) VR headsets perform all graphics processing operations to display images without the need for a separate machine. While standalone VR headsets offer users greater mobility because they don't have to be tethered to a host computer, creating integrated VR headsets that are both comfortable and capable of rendering high-quality graphics can be challenging. For example, standalone VR headsets, tasked with performing computationally intensive, power-intensive graphics processing operations to render high-quality graphics, tend to heat up quickly, and they also tend to be bulky and / or heavy, making them uncomfortable to wear for extended periods. To mitigate these drawbacks, some standalone VR headsets trade comfort for lower-quality graphics processing components that render graphics at lower resolutions, lower dynamic ranges, and / or with a limited set of only basic textures. This makes the graphics processing operations on such headsets less computationally intensive, resulting in lighter headsets that don't get as hot and are therefore more comfortable to wear. However, users who desire to experience high-quality graphics in VR are dissatisfied with today's standalone VR headsets, which fail to strike a balance between quality and comfort.

[0005] Provided herein are technical solutions for improving and enhancing these and other systems. BRIEF DESCRIPTION OF DRAWINGS

[0006] The detailed description is described with reference to the accompanying figures. In these figures, the left-most digit(s) of the reference numbers identify the figure in which that reference number first appears. The use of the same reference numbers in different figures indicates similar or identical components or features.

[0007] Figure 1 is an illustration showing an exemplary technique for dividing a rendering workload of a frame between a head-mounted display (HMD) and a host computer, in accordance with embodiments disclosed herein.

[0008] Figure 2 is an illustration showing two exemplary timelines showing respective rendering workloads of individual frames, divided between a host computer and an HMD, in accordance with embodiments disclosed herein.

[0009] Figure 3 is a flowchart showing an exemplary process for rendering a frame by dividing a rendering workload of the frame between an HMD and a host computer, in accordance with embodiments disclosed herein.

[0010] Figure 4 is a flowchart showing an exemplary process for applying re-projection adjustments on an HMD based on motion vector data generated by a host computer, in accordance with embodiments disclosed herein.

[0011] Figure 5 is a flowchart showing an exemplary process for applying re-projection adjustments based on extra pixel data generated by an application program executing on a host computer, in accordance with embodiments disclosed herein.

[0012] Figure 6 is a flowchart showing an exemplary process for applying re-projection adjustments based on depth data generated by an application program executing on a host computer, in accordance with embodiments disclosed herein.

[0013] Figure 7 is a flowchart showing an exemplary process for an HMD to receive hand tracking data directly from a handheld controller, and to use the hand tracking data to overlay a virtual hand on a scene rendered by an application program, in accordance with embodiments disclosed herein.

[0014] Figure 8A and Figure 8B is a flowchart showing two alternative settings of a system that divides a rendering workload of a frame between an HMD and a host computer, in accordance with embodiments disclosed herein.

[0015] Figure 9 Exemplary components of a wearable device such as an HMD (e.g., a VR headset) and a host computer in which the technology disclosed herein can be implemented are shown. DETAILED DESCRIPTION

[0016] A user can wear a head-mounted display (HMD) to achieve the purpose of immersing the user in a virtual reality (VR) environment or an augmented reality (AR) environment. One or more display panels of the HMD present images based on data generated by an application (e.g., a video game). The application executes on a host computer that is communicatively coupled to the HMD, and the application generates pixel data for individual frames in a series of frames. The pixel data is sent to the HMD to present images that are viewed by the user through optics included in the HMD, causing the user to perceive the images as if the user is immersed in the VR or AR environment.

[0017] The techniques and systems described herein function to divide the rendering workload of a single frame between an HMD and a host computer, such that the host computer performs a first portion of the rendering workload and the HMD performs a second portion of the rendering workload. For a given frame, the HMD is configured to send head tracking data to the host computer, and the host computer is configured to use the head tracking data to generate pixel data for the frame as well as additional data in addition to that pixel data. This additional data can include, but is not limited to, pose data, depth data, motion vector data, parallax occlusion data, and / or additional pixel data. For example, the host computer can use the head tracking data to generate pose data that represents a predicted pose that the HMD will be in at the time that the light emitting elements of the display panel of the HMD will illuminate the frame. The host computer can additionally or alternatively instruct an application to generate depth data and / or additional pixel data based at least in part on that pose data. The host computer can also generate motion vector data based at least in part on the head tracking data and / or movement within the scene that is being rendered. Some or all of this additional data can be sent from the host computer to the HMD, and the HMD can use at least some of the received additional data to modify the pixel data, such as by applying a re-projection adjustment to the pixel data. A "re-projection" is a technique used to compensate for slight inaccuracies in the original pose prediction of the HMD and / or to compensate for an application that fails to achieve a frame rate, which has the same effect as slight inaccuracies in the initial pose prediction. For example, a re-projected frame can be generated using pixel data from an application-rendered frame by transforming (e.g., by rotation and re-projection calculations) that application-rendered frame in a manner that takes into account an updated prediction of the pose of the HMD. Thus, the modified pixel data obtained from applying a re-projection adjustment (and possibly other adjustments) can be used to present an image of a given frame on the display panel of the HMD, and this process can be iterated for a series of frames.

[0018] In some embodiments, the additional data that is generated, sent, and / or used to render a frame can vary between frames in addition to the pixel data. For example, the host computer can dynamically determine the type and / or extent of additional data to generate as part of the first portion of the rendering workload for individual frames, and / or the type and / or extent of additional data to send to the HMD. Meanwhile, the HMD can dynamically determine the type and / or extent of additional data received from the host computer to use as part of the second portion of the rendering workload for individual frames.

[0019] Dividing the rendering workload of a given frame between the host computer and the HMD allows for the implementation of a system in which the host computer and the HMD can be wirelessly connected to one another; such a system is currently not feasible for today's high-latency wireless communication protocols that rely entirely on the host computer and the HMD. In turn, dividing the rendering workload allows for a high-quality VR or AR experience to be provided on the HMD that is comfortable to wear for long periods of time, as the high computational power of the host computer can still be leveraged in the systems disclosed herein. Moreover, the HMDs disclosed herein can be and can remain physically untethered to the host computer, as compared to tethered HMDs, thereby providing greater mobility to the user, as the user is able to better move around in space while wearing the HMD without worrying about accidentally unplugging the HMD, etc. Given the user's demand for high-fidelity, high-resolution VR graphics, wireless VR systems that follow these demands will tend to experience greater latency in the transmission of data over the wireless communication link due to the greater amount of data being wirelessly transmitted. This means that, as compared to pose predictions for conventional physically tethered HMDs that can leverage the higher data transmission rates of a wired connection, the pose predictions of the HMD that are used by the application to render a given frame are made further in advance in the systems disclosed herein. Pose predictions made further in advance of the illumination time of a frame, as compared to pose predictions for physically tethered HMDs that are made later in time, means that there is more error in the pose predictions, which in turn means that the HMDs disclosed herein are tasked with performing computationally intensive graphics processing operations in order to modify the pixel data received from the host computer (e.g., to correct for errors in the pixel data received from the host computer) so that the appropriate image is displayed on the HMD. Generally, the HMDs with the additional data received from the host computer can better account for the relatively lower data transmission rates over the wireless communication link between the host computer and the HMD in order to modify the received pixel data in a manner that improves the quality of the resulting image presented on the display panel of the HMD. Moreover, the rendering techniques and systems described herein can allow for different rendering frequencies (or frame rates) on each of the host computer and the HMD.

[0020] Accordingly, the disclosed HMD is configured to perform a portion of the rendering workload for a given frame, which allows data to be transmitted wirelessly between the host computer and the HMD, despite the relatively high latency of the wireless connection as compared to the relatively low latency wired connection of today's HMDs. The HMD can compensate for the higher latency of the wireless communication link using HMD on-board graphics processing logic to correct for errors in the data generated by the host computer. Moreover, such on-board graphics processing logic allows the HMD to be used as a standalone device (perhaps in limited use scenarios). For example, the HMDs disclosed herein can be used in standalone mode to play video games that render more basic graphics in their images, requiring less computationally intensive graphics processing operations to render frames. As another example, the HMDs disclosed herein can be used in standalone mode to play back movies and / or video clips on the HMD, all independent of a host computer. However, when a user of the HMDs disclosed herein wishes to play a video game with more rich graphics, the user can operate the HMD in connected mode to connect the HMD to a host computer through a wired or wireless communication link, thereby leveraging the additional graphics processing capabilities of the host computer. By leveraging the additional power capacity of the host computer (e.g., so the HMD does not run out of battery power), a user who wishes to play a video game with more rich graphics for an extended period of time can still use a wired communication link. For example, the user can obtain the high-fidelity graphics experience provided by the connected host computer as well as the enhanced mobility enabled by the available wireless connection between the host computer and the HMD as compared to today's integrated systems.

[0021] Non-transitory computer-readable media storing computer-executable instructions for implementing the techniques and processes disclosed herein are also disclosed herein. Although the techniques and systems disclosed herein are discussed in the context of video game applications, and VR game applications in particular, by way of example, it should be understood that the techniques and systems described herein can provide benefits for other applications, including but not limited to non-VR applications (e.g., AR applications) and / or non-game applications (such as industrial machine applications, defense applications, robotics applications, etc.).

[0022] Figure 1 FIG. 1 is a diagram illustrating an example technique for dividing a rendering workload 100 of a frame between a head-mounted display (HMD) and a host computer. Figure 1 A head-mounted display (HMD) 102 worn by a user 104 is depicted, as well as a host computer 106. Figure 1Exemplary embodiments of a host computer 106 are shown in the form of a laptop computer 106 (1) that is carried in a backpack, for example, or a personal computer (PC) 106 (N) that may be located, for example, in the home of the user 104. However, it should be understood that these exemplary types of host computers 106 are not limiting of the present disclosure. For example, the host computer 106 may be implemented as any type and / or any number of computing devices, including, but not limited to, a PC, a laptop computer, a desktop computer, a portable digital assistant (PDA), a mobile phone, a tablet computer, a set-top box, a game console, a server computer, a wearable computer (e.g., a smartwatch, etc.), or any other electronic device that can send / receive data. The host computer 106 may be located in the same environment as the HMD 102, such as in the home of the user 104 wearing the HMD 102. Alternatively, the host computer 106 may be remotely located relative to the HMD 102, such as in the form of a server computer that is located at a remote geographic location relative to the geographic location of the HMD 102. In a remote host computer 106 implementation, the host computer 106 may be communicatively coupled to the HMD 102 via a wide area network, such as the Internet. In a local host computer 106 implementation, the host computer 106 may be co-located with the HMD 102 in an environment (e.g., a home), whereby the host computer 106 and the HMD 102 may be communicatively coupled together directly or via an intermediary network device over a local area network (LAN).

[0023] like Figure 1 As shown, for a given frame, the host computer 106 is configured to perform a first portion of the rendering workload 100(1) (e.g., a first portion of the rendering workload 100 for the given frame), and the HMD 102 is configured to perform a second portion of the rendering workload 100(2) (e.g., a second portion of the rendering workload 100 for the given frame). In this manner, the HMD 102 and the host computer 106 are communicatively coupled together and configured to work together in a collaborative manner to render the given frame by generating pixel data that is ultimately used to present a corresponding image on the display panel 108 of the HMD 102.

[0024] Figure 1The HMD 102 in examples of the present disclosure can include a single display panel 108 or multiple display panels 108, such as a left display panel and a right display panel in a pair of stereoscopic display panels. The display panel(s) 108 of the HMD 102 can be used to present a series of image frames (referred to herein as “frames”) that can be viewed by a user 104 wearing the HMD 102. It will be appreciated that the HMD 102 can include any number of display panels 108 (e.g., more than two display panels, a pair of display panels, or a single display panel). Thus, the term “display panel” as used herein in the singular can refer to one of a pair of display panels 108 of a dual-panel HMD 102, or it can refer to a single display panel 108 of an HMD 102 having any number of display panels (e.g., a single-panel HMD 102 or a multi-panel HMD 102). In a dual-panel HMD 102, a stereoscopic frame buffer can render, for example, 2160 x 1200 pixels on both display panels of the HMD 102 (e.g., 1080 x 1200 pixels on each display panel).

[0025] The display panel 108 of the HMD 102 can utilize any suitable type of display technology, such as an emissive display that utilizes light-emitting elements (e.g., light-emitting diodes (LEDs)) that emit light during the presentation of a frame on the display panel 108. As an example, the display panel 108 of the HMD 102 can include a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an inorganic light-emitting diode (ILED) display, or any other suitable type of display technology for HMD applications.

[0026] The display panel 108 of the HMD 102 can operate at any suitable refresh rate, such as a 90 hertz (Hz) refresh rate, which can be a fixed refresh rate or a variable refresh rate that dynamically varies within a range of refresh rates. The “refresh rate” of a display is the number of times per second that the display redraws the screen. If a fixed refresh rate is used, the number of frames displayed per second can be limited by the refresh rate of the display. Thus, a series of frames can be processed (e.g., rendered) and displayed as images on the display such that individual frames in the series of frames are displayed with each screen refresh. That is, to present a series of images on the display panel 108, the display panel 108 can transition from frame to frame in the series of frames at the refresh rate of the display, thereby illuminating the pixels at each screen refresh. In some embodiments, the frame rate can be throttled and / or the application can not achieve the target frame rate, and phantom frames (based on re-projection) can be inserted between frames rendered by the application.

[0027] The display system of HMD 102 can implement any suitable type of display driving scheme, such as a global flash type of display driving scheme, a rolling band type of display driving scheme, or any other suitable type of display driving scheme. In a global flash type of display driver scheme, the array of light emitting elements of the display are illuminated all at once at each screen refresh, flashing globally at the refresh rate. In a rolling band type of display driving scheme, individual subsets of the light emitting elements of the display can be independently and sequentially illuminated in a rolling band of illumination during an illumination period. These types of display driving schemes can be implemented with individually addressable light emitting elements. If the array of pixels and the array of light emitting elements on display panel 108 are arranged in rows and columns (but not necessarily with a one-to-one correspondence of light emitting elements to pixels), individual rows and / or individual columns of light emitting elements can be addressed in sequence, and / or for a rolling band type of display driving scheme, consecutive rows of individual groups of light emitting elements and / or consecutive columns of individual groups of light emitting elements can be addressed in sequence.

[0028] Generally, as used herein, "illuminating a pixel" means illuminating the light emitting element corresponding to that pixel. For example, an LCD illuminates a light emitting element with a backlight to illuminate the corresponding pixel of the display. Also, as used herein, a "subset of pixels" can include a single pixel or multiple pixels (e.g., a group of pixels). To drive display panel 108, HMD 102 can include, among other things, a display controller, display driver circuitry, and similar electronics for driving display panel 108. The display driver circuitry can be coupled to the array of light emitting elements of display panel 108 via conductive paths (such as metal traces) on a flexible printed circuit. In one example, the display controller can be communicatively coupled to the display driver circuitry and configured to provide signals, information, and / or data to the display driver circuitry. The signals, information, and / or data received by the display driver circuitry can cause the display driver circuitry to illuminate the light emitting elements in a particular manner. That is, the display controller can determine which light emitting element(s) to illuminate, when to illuminate the light emitting element(s), and the level of light output to be emitted by the light emitting element(s), and can communicate the appropriate signals, information, and / or data to the display driver circuitry in order to achieve that purpose.

[0029] In the illustrated implementation, the HMD 102 includes one or more processors 110 and a memory 112 (e.g., a computer-readable medium 112). In some implementations, the processor(s) 110 can include a central processing unit (CPU), a graphics processing unit (GPU) 114, both a CPU and a GPU 114, a microprocessor, a digital signal processor, or other processing units or components known in the art. Alternatively or additionally, the functionality described herein can be performed, at least in part, by one or more hardware logic components. For example and without limitation, illustrative types of hardware logic components that can be used include a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip system (SOC), a complex programmable logic device (CPLD), etc. In addition, each of the processors 110 can have its own local memory, which can also store program modules, program data, and / or one or more operating systems.

[0030] The memory 112 can include volatile memory and non-volatile memory implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Such memory includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, RAID storage systems, or any other medium that can be used to store the desired information and that can be accessed by a computing device. The memory 112 can be implemented as a computer-readable storage medium (“CRSM”) that can be accessed by the processor(s) 110 to execute instructions stored on the memory 112. In one basic implementation, the CRSM can include random access memory (“RAM”) and flash memory. In other implementations, the CRSM can include, but is not limited to, read-only memory (“ROM”), electrically erasable programmable read-only memory (“EEPROM”), or any other tangible medium that can be used to store the desired information and that can be accessed by the processor(s) 110.

[0031] In general, the HMD 102 may include logic (e.g., software, hardware, and / or firmware, etc.) configured to implement the techniques, functions, and / or operations described herein. The computer-readable medium 112 may include various modules, such as instructions, data storage, etc., that may be configured to execute on the processor 110 to implement the techniques, functions, and / or operations described herein. An exemplary functional module in the form of a compositor 116 is shown as stored in the computer-readable medium 112 and executable on the processor 110, but the same functionality may alternatively be implemented using hardware, firmware, or a system-on-a-chip (SOC) and / or other logic. Furthermore, additional or different functional modules may be stored in the computer-readable medium 112 and executable on the processor 110. The compositor 116 is configured to modify pixel data received from the host computer 106 as part of the second portion of the rendering workload 100(2) and output the modified pixel data to a frame buffer (e.g., a stereoscopic frame buffer) so that a corresponding image can be presented on the display panel 108 of the HMD 102.

[0032] The HMD 102 may also include a head tracking system 118 and a communication interface 120. The head tracking system 118 may utilize one or more sensors (e.g., infrared (IR) light sensors mounted on the HMD 102) and one or more tracking beacons (e.g., IR light emitters co-located in the environment with the HMD 102) to track the head motion or movement of the user 104, including head rotation. This exemplary head tracking system 118 is non-limiting, and other types of head tracking systems 118 (e.g., camera-based, inertial measurement unit (IMU)-based, etc.) may also be utilized. The head tracking system 118 is configured to generate head tracking data 122, which may be sent to the host computer 106 via the communication interface 120 during runtime when a frame is rendered.

[0033] The communication interface 120 of the HMD 102 may include wired and / or wireless components (e.g., chips, ports, etc.) for facilitating the sending and / or receiving of wired and / or wireless data to / from the host computer 106, either directly or via one or more intermediate devices (such as a wireless access point (WAP)). For example, the communication interface 120 may include a wireless unit coupled to an antenna to facilitate wireless connection with the host computer 106 and / or another device. Such a wireless unit may implement one or more of a variety of wireless technologies, such as Wi-Fi, Bluetooth, radio frequency (RF), etc. The communication interface 120 may also include one or more physical ports for facilitating wired connection with the host computer 106 and / or another device (e.g., a plug-in network device that communicates with other wireless networks).

[0034] In the illustrated implementation, the host computer 106 includes one or more processors 124 and a memory 126 (e.g., a computer-readable medium 126). In some implementations, the processor(s) 124 can include a CPU, a GPU 128, both a CPU and a GPU 128, a microprocessor, a digital signal processor, or other processing units or components known in the art. Alternatively or additionally, the functionality described herein can be performed, at least in part, by one or more hardware logic components. For example and without limitation, illustrative types of hardware logic components that can be used include FPGAs, ASICs, ASSPs, SOCs, CPLDs, and the like. In addition, each of the processor(s) 124 can have its own local memory that can also store program modules, program data, and / or one or more operating systems.

[0035] The memory 126 can include volatile memory and non-volatile memory implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Such memory includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, DVD or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, RAID storage systems, or any other medium which can be used to store the desired information and which can be accessed by the computing device. The memory 126 can be implemented as a CRSM, which can be any available physical media accessible by the processor(s) 124 to execute instructions stored in the memory 126. In one basic implementation, the CRSM can include RAM and flash memory. In other implementations, the CRSM can include, without limitation, ROM, EEPROM, or any other tangible medium that can be used to store the desired information and that can be accessed by the processor(s) 124.

[0036] In general, the host computer 106 can include logic (e.g., software, hardware, and / or firmware, etc.) configured to implement the techniques, functions, and / or operations described herein. The computer-readable medium 126 can include various modules, such as instructions, data storage, etc., that can be configured to execute on the processor(s) 124 to implement the techniques, functions, and / or operations described herein. Example functional modules in the form of application programs 130, such as a video game 130(1), and a rendering component 132 are shown as being stored in the computer-readable medium 126 and executable on the processor(s) 124. In some embodiments, the functionality of the rendering component 132 can alternatively be implemented in hardware, firmware, or as a system on a chip (SOC) and / or other logic. Further, additional or different functional modules can be stored in the computer-readable medium 126 and executable on the processor(s) 124.

[0037] The host computer 106 can also include a communication interface 134, which can include wired and / or wireless components (e.g., chips, ports, etc.) for facilitating wired and / or wireless transmission / reception of data to / from the HMD 102, either directly or via one or more intermediary devices, such as a WAP. For example, the communication interface 134 can include a wireless unit coupled to an antenna to facilitate wireless connections with the HMD 102 and / or another device. Such a wireless unit can implement one or more of a variety of wireless technologies, such as Wi-Fi, Bluetooth, RF, etc. The communication interface 134 can also include one or more physical ports for facilitating wired connections with the HMD 102 and / or another device (e.g., a plug-in network device that communicates with other wireless networks).

[0038] It should be appreciated that the HMD 102 can represent a VR headset for use in a VR system, such as a VR headset for use with a VR gaming system, in which case the video game 130(1) can represent a VR video game 130(1). However, the HMD 102 can additionally or alternatively be implemented as an AR headset for use in an AR application, or a headset that can be used for VR and / or AR applications that are not related to gaming (e.g., industrial applications). In AR, the user 104 sees virtual objects superimposed on a real-world environment, whereas in VR, the user 104 generally does not see the real-world environment, but is fully immersed in a virtual environment, as perceived via the display panel 108 and optics (e.g., lenses) of the HMD 102. It should be appreciated that in some VR systems, a pass-through image of the real-world environment of the user 104 can be displayed in conjunction with virtual images to create an augmented VR environment in the VR system, whereby the VR environment is augmented with real-world images (e.g., superimposed on the virtual world). The examples described herein primarily relate to VR-based HMDs 102, but it should be appreciated that the HMD 102 is not limited to implementation in VR applications.

[0039] In general, the application 130 executing on the host computer 106 can be a graphics-based application 130 (e.g., a video game 130(1)). The application 130 is configured to generate pixel data for a series of frames, and this pixel data is ultimately used to render corresponding images on the display panel 108 of the HMD 102. During runtime, for a given frame, the rendering component 132 can determine a predicted "illumination time" for that frame. The predicted "illumination time" for the frame represents the time at which the emissive elements of the display panel 108 of the HMD 102 will illuminate the frame. This prediction can take into account the inherent latency of the wireless communication link between the host computer 106 and the HMD 102, as well as a predicted rendering time and / or a known scan-out time of the pixels from the frame buffer. In other words, the prediction for a wireless communication link can be different than the prediction for a wired communication link. For example, for a wired communication link, the rendering component 132 can predict an illumination time a first amount of time in the future (e.g., about 22 milliseconds in the future), whereas for a wireless communication link, due to the inherent differences in latency when transmitting data over a wired connection versus a wireless connection, the rendering component 132 can predict an illumination time a second, greater amount of time in the future (e.g., about 44 milliseconds in the future).

[0040] The host computer 106 can also receive head tracking data 122 (e.g., first head tracking data 122) generated by the head tracking system 118 of the HMD 102 from the HMD 102. This head tracking data 122 can be generated and / or transmitted at any suitable frequency, such as a frequency corresponding to a target frame rate and / or refresh rate of the HMD 102, or a different (e.g., faster) frequency, such as 1000 Hz (or reading the sensors 1 time per 1 millisecond). The rendering component 132 is configured to determine, based at least in part on the head tracking data 122, a predicted pose at which the HMD 102 will be at the predicted illumination time. The rendering component 132 can then provide pose data indicative of the predicted pose to the application 130 in execution for rendering the frame (e.g., generating pixel data for the frame) based on the predicted pose, and the rendering component 132 can obtain pixel data 136 associated with the frame from the application 130. This pixel data 136 can correspond to a pixel array of the display panel 108 of the HMD 102. For example, the pixel data 136 output by the application 130 based on the pose data can include a two-dimensional array of per-pixel values (e.g., color values) for the pixel array on the display panel 108 of the HMD 102. In an illustrative example, a stereoscopic display panel 108 pair can include a 2160 x 1200 pixel array on each of the two display panels of the HMD 102 (e.g., 1080 x 1200 pixels per display panel 108). In this illustrative example, the pixel data 136 can include 2160 x 1200 pixel values (or 2,592,000 pixel values). In some embodiments, the pixel data 136 can include data for each pixel represented by a single set of color values and alpha values (e.g., one color value for a red channel, one color value for a green channel, one color value for a blue channel, and one or more values for one or more alpha channels).

[0041] In addition to the pixel data 136, the logic of the host computer 106 can also generate additional data 138, and at least some of this additional data 138 can be sent to the HMD 102 to help the HMD 102 process the second portion of the rendering workload 100(2). For example, the additional data 138 can be packaged together with the pixel data 136 and sent to the HMD 102, and at least some of the additional data 138 can be used by the logic of the HMD 102 to modify the pixel data 136 for rendering an image corresponding to the frame on the display panel 108 of the HMD 102. The additional data 138 can include, but is not limited to, pose data generated by the rendering component 132, as well as depth data, motion vector data, disparity occlusion data, and / or additional pixel data. For example, in providing pose data to the application program 130 in execution for rendering the frame, the rendering component 132 can further instruct the application program 130 to generate depth data (e.g., Z-buffer data) and / or additional pixel data (sometimes referred to herein as “out-of-bound pixel data” or “additional pixel data”) for the frame, and in response, the rendering component 132 can obtain the depth data and / or additional pixel data associated with the frame from the application program 130. Additionally or alternatively, the rendering component 132 can generate motion vector data based at least in part on head tracking data 122 received from the HMD 102. For example, the motion vector data can be generated based on a comparison of head tracking data generated at two different points in time (e.g., a comparison of head tracking data spaced apart by a few milliseconds). Some or all of the additional data 138 can be utilized by the logic of the HMD 102 (e.g., the compositor 116) to the end of modifying the pixel data 136 to correct for errors in pose prediction made by the rendering component 132 in advance, which takes into account the inherent latency of the wireless connection between the host computer 106 and the HMD 102. For example, the compositor 116 can apply re-projection adjustments based at least in part on the additional data 138 received from the host computer 106. Other adjustments made by the compositor 116 as part of the second portion of the rendering workload 100(2) can include, but are not limited to, adjustments for geometric distortion, chromatic aberration, re-projection, etc. The ways in which the additional data 138 can be used as part of the second portion of the rendering workload 100(2) are described in more detail below with reference to the following figures.

[0042] Figure 2 FIG. 1 is a diagram illustrating an example system 100 in accordance with the embodiments disclosed herein. Figure 2The example depicts three exemplary frames: frame "F," frame "F+1," and frame "F+2" relative to a first timeline 200(1) associated with a host computer 106. The first timeline 200(1) illustrates how an executing application 130 on the host computer 106 may render frames sequentially using the GPU 128 of the host computer 106. Here, as shown sequentially from left to right on the first timeline 200(1), the application 130 renders frame F as part of a first portion of the rendering workload 100(1)(a), then renders frame F+1 as part of a second portion of the rendering workload 100(1)(b), and then renders frame F+2 as part of a third portion of the rendering workload 100(1)(c). The ellipsis on the first timeline 200(1) indicates that the rendering may continue for any number of frames as the application 130 continues to execute. The first timeline 200(1) also implies that the application 130 is targeting a target frame rate (e.g., a frame rate of 90 Hz, where these vertical lines would be approximately 11.11 milliseconds apart) by having vertical lines oriented perpendicular to the horizontal timeline 200(1). Figure 2 In the example shown, application 130 executing on host computer 106 happens to achieve the target frame rate over the series of these three exemplary frames, but this may not always be the case in practice because application 130 may, in some cases (e.g., for scenes with a large number of moving objects or complex textures), take longer than the allotted time to render a given frame 202. This scenario is sometimes referred to as application 130 failing to achieve the target frame rate.

[0043] and Figure 2 102 , shows portions of the rendering workload 100(2)(a), 100(2)(b), and 100(2)(c) of the compositor 116 of the HMD 102 for respective frames. Each rendering workload 100(2) of the compositor 116 of the HMD 102 for a given frame may represent adjustments that are applied to pixel data 136 generated by an application 130 executing on the host computer 106 before the final image is presented on the display panel 108 of the HMD 102. Such adjustments may include, but are not limited to, adjustments for geometric distortion, chromatic aberration, re-projection, and the like, which are applied to the pixel data 136 received from the host computer 106 before the final image is rendered on the HMD 102. At least some of these adjustments may utilize additional data 138 received from the host computer 106, such as pose data, depth data, additional pixel data, parallax occlusion data, and / or motion vector data, as described herein. Thus, Figure 2The frames shown in the second timeline 200(2) should represent "actual" frames in the sense that they are output from the application 130, which can represent a video game application 130(1) or any other type of graphics-based application. In contrast, if the application 130 fails to achieve the target frame rate for a given frame, or if the frame rate is throttled to a rate that is slower than the refresh rate of the display panel 108 of the HMD 102, then the compositor 116 of the HMD 102 can generate a "phantom" frame (e.g., using re-projection) using previously received pixel data 136 for the previous frame based on the pose prediction for the previous frame and an updated pose prediction made by the HMD 102. In any case, the result of the partial rendering workload 100(2) is to generate modified pixel data that can be output to a frame buffer (e.g., a stereoscopic frame buffer). This distinction between "actual" frames and "phantom" frames does not imply that actual frames are not adjusted on the HMD 102, and in this sense, the frames generated on the HMD side are all effectively composited (i.e., different from the raw frames output by the application 130 executing on the host computer 106).

[0044] Figure 2 The second timeline 200(2) also shows the scan-out time 202(a), 202(b), and 202(c) for each frame, as well as the illumination time 204(a), 204(b), and 204(c) for each frame. During the scan-out time 202 for a given frame, a subset of pixel values (of modified pixel data) are scan-out to the display panel 108 via a display port (e.g., a high-definition multimedia interface (HDMI)), and during the illumination time 204 for a given frame, the light-emitting elements of the display panel 108 are illuminated to cause the pixels of the display panel 108 to be illuminated. Figure 2 An example of a global flicker type of display driving scheme is shown, which can be used with an LCD panel to emit light from the light-emitting elements of the display panel 108 at the refresh rate of the HMD 102 at the same time. In the illustrative example, if the HMD 102 operates at a 90 Hz refresh rate, then the illumination time 204 for each frame can be separated by approximately 11.11 milliseconds.

[0045] It should be appreciated that although Figure 2The respective rendering cycles of host computer 106 and HMD 102 are depicted as appearing to be synchronized (they can be synchronized), but the techniques and systems described herein do not require frame synchronization between these two devices. In general, once data (e.g., pixel data 136 and additional data 138) is received from host computer 106 and / or once HMD 102 determines that application 130 of host computer 106 can have missed a frame or that a packet can have been dropped in transmission, etc., compositor 116 of HMD 102 can begin its rendering workload 100(2) for a given frame. Due to varying conditions of the wireless communication link, processing loads on the respective devices, and / or other factors, the respective rendering cycles of host computer 106 and HMD 102 can sometimes be out of sync with respect to each other. Thus, while host computer 106 and HMD 102 are configured to work together in a cooperative manner by dividing the rendering workload for a given frame into partial workloads that are performed on the respective devices, it should be understood that these devices can operate independently of each other to perform their respective partial workloads.

[0046] The processes described herein are illustrated as a collection of blocks in logical flow graphs, which represent a sequence of operations (i.e., logical) that can be implemented in hardware, software, firmware, or a combination thereof. In the context of software, the blocks represent computer-executable instructions that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks can be combined in any order and / or

[0047] Figure 3 A flow diagram of an exemplary process 300 for rendering a frame by dividing the rendering workload 100 for the frame between HMD 102 and host computer 106 in accordance with the embodiments disclosed herein is shown. Process 300 is described with reference to the preceding figures for purposes of discussion.

[0048] At 302, the HMD 102 can send first head tracking data 122 generated by the head tracking system 118 of the HMD 102 to the host computer 106 that is communicatively coupled to the HMD 102. The manner in which the host computer 106 is communicatively coupled to the HMD 102 can vary from implementation to implementation. For implementations in which the host computer 106 is wirelessly coupled to the HMD 102, the first head tracking data 122 can be sent wirelessly from the HMD 102 to the host computer 106 (e.g., using WiFi, Bluetooth, or any suitable wireless communication protocol, including a proprietary protocol). For implementations in which the host computer 106 is coupled to the HMD 102 via a wired connection (e.g., a data cable), the first head tracking data 122 can be sent from the HMD 102 to the host computer 106 over the wired connection, such as a data cable. Moreover, if the host computer 106 is located in a remote geographic location relative to the HMD 102, the first head tracking data 122 can be sent from the HMD 102 to the host computer 106 over a wide area network, such as the Internet, at block 302.

[0049] At 304, the host computer 106 can receive the first head tracking data 122 from the HMD 102. As mentioned with respect to block 302, various ways of receiving the first head tracking data 122 can be employed depending on the implementation, such as wirelessly, over a wired connection, via a wide area network, etc. At the point in time at which the host computer 106 receives the first head tracking data 122, the host computer 106 can be executing an application 130 thereon, such as a video game 130(1), the task of which is to render a first frame of a series of frames for creating visual video game content to be displayed on the HMD 102.

[0050] At 306, the logic of the host computer 106 (e.g., the rendering component 132) can determine a predicted illumination time that represents a time at which a light emitting element of the display panel 108 of the HMD 102 will illuminate the first frame. That is, the logic of the host computer 106 can determine a time at which a photon associated with rendering an image for the first frame will actually reach the eye of the user 104. This predicted illumination time is a time in the future (e.g., about 44 milliseconds in the future), as the rendering component 132 must account for the time it takes for the application 130 to generate pixel data 136, the time it takes for the pixel data 136 to be sent from the host computer 106 to the HMD 102, and the time it takes to modify and scan out the pixel data 136 on the HMD 102 before the corresponding image is finally rendered on the display panel 108 of the HMD 102.

[0051] At 308, logic of host computer 106 (e.g., rendering component 132) can determine, based at least in part on first head tracking data 122 received at block 304, a predicted pose at which HMD 102 will be at the predicted time of illumination determined by HMD 102 at block 306. For example, head tracking system 118 of HMD 102 can be configured to track up to six degrees of freedom of HMD 102 (e.g., 3D position, roll, pitch, and yaw), which can be sent to host computer 106 as head tracking data 122 to determine a predicted pose of HMD 102 (e.g., taking into account predicted head movement that leads to a future pose of HMD 102).

[0052] At 310, logic of host computer 106 (e.g., rendering component 132) can provide pose data indicative of the predicted pose determined at block 308 to application 130 executing on host computer 106 for rendering the first frame. For example, application 130 can call a function that receives pose data from rendering component 132, and rendering component 132 can provide the requested pose data (predicted to be the target time of illumination for the first frame, and predicted based at least in part on head tracking data 122 received from HMD 102) to application 106, so that application 130 can render the first frame according to the pose data, which corresponds to a virtual camera pose for rendering the scene. In some embodiments, rendering component 132 can instruct application 130 to generate not only pixel data 136 for the frame, but also additional data, such as depth data and / or additional pixel data.

[0053] At 312, logic of host computer 106 (e.g., rendering component 132) can obtain (or receive) data associated with the first frame (e.g., pixel data 136) from application 130. As described herein, pixel data 136 can include pixel values for individual pixels in a pixel array of display panel 108 of HMD 102. As described above, in embodiments in which rendering component 132 requests that application 130 generate additional data 138 in addition to pixel data 136, at block 312, rendering component 132 can obtain (or receive) additional data 138, such as depth data (e.g., Z-buffer data) and / or additional pixel data that includes additional pixel values outside of the bounds of the pixel array of display panel 108. For example, if display panel 108 has an array of 2160 x 1200 pixels, pixel data 136 can correspond to pixel values in that 2160 x 1200 pixel array, while additional pixel data can correspond to pixels outside of the bounds of that 2160 x 1200 array. Thus, pixel data 136 and additional pixel data can constitute a larger number of pixel values, e.g., a larger array of 2400 x 1400 pixels.

[0054] At 314, logic of the host computer 106 (e.g., the rendering component 132) can compress and / or serialize data to be sent to the HMD 102 for rendering an image associated with the first frame. This data can include the pixel data 136 and any additional data 138 generated by the application 130, by the rendering component 132, and / or by any other component of the host computer 106. In contrast to post-distorted images, the compression at block 314 can be optimized for pre-distorted images. For example, in systems where the entire rendering workload is performed on the host computer, any compression of pixel data sent to the headset can utilize a compression algorithm that mitigates stereo compression artifacts. In process 300, because the pixel data 136 generated by the application 130 is "pre-distorted," meaning that adjustments such as re-projection adjustments, geometric distortion adjustments, chromatic aberration adjustments, etc. are to be applied after compression, the compression algorithm used at block 314 can not need to account for stereo compression artifacts. In some embodiments, the pixel data 136 output by the application 130 can be categorized into foreground pixels and background pixels (and potentially intermediate layer pixels), and different compression algorithms or schemes can be used for different pixel layers. That is, foreground pixels can be compressed differently (e.g., using a different compression algorithm) than background pixels that are compressed.

[0055] At 316, the host computer 106 can send the data including the pixel data 136 and the additional data 138 to the HMD 102. The additional data 138 sent at block 316 can include, but is not limited to, pose data 138(1) generated based on the first head tracking data 122 and used by the application 130 to generate the pixel data 136 of the first frame, depth data 138(2) generated by the application 130, additional pixel data 138(3) generated by the application 130, motion vector data 138(4) generated by the rendering component 132 based on the head tracking data 122 and / or based on movement of virtual objects within the rendered scene, disparity occlusion data 138(5), and / or cube map data (e.g., for fast, large-scale head movements so that the HMD 102 has other options besides rendering dark pixels with no data in them). Again, various ways of sending the pixel data 136 and the additional data 138 to the HMD 102 can be employed depending on the particular implementation, such as wirelessly, through a wired connection, via a wide area network, etc.

[0056] At 318, the HMD 102 can receive the pixel data 136 associated with the first frame, and additional data 138 in addition to the pixel data 136, from the host computer 106. Again, various ways of receiving the pixel data 136 and the additional data 138 from the host computer 106 can be employed depending on the particular implementation, such as wirelessly, through a wired connection, via a wide area network, etc. Moreover, when receiving compressed serialized data from the host computer 106 at block 318, the data can be deserialized and decompressed at block 318.

[0057] At 320, the logic of the HMD 102 (e.g., the compositor 116) can determine an updated pose that the HMD 102 will be in at the illumination time of the first frame based at least in part on second head tracking data 122 generated by the head tracking system 118 of the HMD 102, the illumination time representing a time at which the emissive elements of the display panel 108 of the HMD 102 will illuminate the first frame. Because the determination at block 320 is closer in time to the illumination time of the first frame, the pose prediction at block 320 is more accurate (e.g., has less error) than the pose prediction determined at block 308, which is further ahead in time from the illumination time. In some embodiments, the determination of the updated pose of the HMD 102 at block 320 can be based at least in part on motion vector data 138(4) received from the host computer 106 or generated locally on the HMD 102. For example, the motion vector data 138(4) generated from the head tracking data 122 can indicate a predicted head movement of the user 104, and the compositor 116 of the HMD 102 can use the motion vector data 138(4) to make an updated pose prediction of the HMD 102 with improved accuracy.

[0058] At 322, the logic of the HMD 102 (e.g., the compositor 116) can modify the pixel data 136 received from the host computer 106 to obtain modified pixel data. As shown in the sub-blocks of block 322, such modification of the pixel data 136 can include various sub-operations.

[0059] At sub-block 324, the logic of the HMD 102 (e.g., the compositor 116) can apply re-projection adjustments to the pixel data 136 to obtain modified pixel data associated with the first frame. The re-projection adjustments applied at sub-block 324 can be based at least in part on the additional data 138 received from the host computer 106. For example, a comparison between the original predicted pose determined at block 308 (and shown in pose data 138(1)) and the updated pose determined at block 320 can reveal an increment (or difference) between the compared poses, and the re-projection adjustments can include rotation calculations to compensate for the increment (e.g., by moving and / or rotating the pixel data 136 in some manner depending on the increment between the two pose determinations).

[0060] In some embodiments, the additional data 138 received from the host computer 106 includes depth data 138(2) associated with the first frame, and the re-projection adjustments applied at sub-block 324 can be based at least in part on the depth data 138(2). For example, depth data 138(2) such as from a depth buffer (or Z-buffer) can indicate occluding objects in the scene. Thus, the depth data 138(2) can be used to make adjustments for parallax for objects in the scene (e.g., a ship that is far away in world space can not move as much with the same head movement as a close-up object that will move with head movement). Knowing the depth of pixels corresponding to virtual objects in the scene helps to know how to make adjustments for such parallax during re-projection on the HMD 102 at sub-block 324. Other ways of using the depth data 138(2) on the HMD side are described in more detail below. In some embodiments, the additional data 138 received from the host computer 106 includes additional pixel data 138(3) associated with the first frame, and the re-projection adjustments applied at sub-block 324 can be based at least in part on the additional pixel data 138(3). For example, in the case where a fast / large-scale head movement occurs, at least some of the pixel values of the pixel data 136 can be replaced with additional pixel values of the additional pixel data 138(3) to render a portion of the scene corresponding to the current head movement of the user 104 as determined by the HMD 102. In some embodiments, the additional data 138 received from the host computer 106 includes motion vector data 138(4) associated with the first frame, and the re-projection adjustments applied at sub-block 324 can be based at least in part on the motion vector data 138(4).

[0061] At sub-block 326, logic of the HMD 102 (e.g., the compositor 116) can apply a geometric distortion adjustment to the pixel data 136 to obtain modified pixel data associated with the first frame at block 322. The geometric distortion adjustment can compensate for distortion of the near-eye optical subsystem (e.g., lenses and other optics) of the HMD 102. For similar reasons, at sub-block 328, logic of the HMD 102 (e.g., the compositor 116) can apply a chromatic aberration adjustment to the pixel data 136 to obtain modified pixel data associated with the first frame at block 322.

[0062] At sub-block 330, the pixel data 136 can be modified by superimposing one or more virtual hands on the scene represented by the pixel data 136 to obtain modified pixel data associated with the first frame at block 322. For example, the HMD 102 can be communicatively coupled to one or more handheld controllers that are very like the HMD 102, whose movements and positions in 3D space are tracked, and the HMD 102 can receive hand tracking data from the handheld controllers (e.g., through direct wireless data transmission from the handheld controllers to the HMD 102). The hand tracking data can be processed by logic of the HMD 102 (e.g., the compositor 116) to render virtual hands on the scene that are superimposed on the content of the scene.

[0063] At 332, logic of the HMD 102 (e.g., the compositor 116) can output the modified pixel data to a frame buffer. Also, for an HMD 102 having a pair of display panels 108, the modified pixel data can correspond to a frame representing a pair of images to be displayed on the pair of display panels 108, and can be output to a stereoscopic frame buffer accordingly.

[0064] At 334, logic of the HMD 102 can cause the first image to be rendered on the display panel 108 of the HMD 102 based on the modified pixel data output to the frame buffer at block 332. This can involve scan-out of the modified pixel data to the display panel 108 of the HMD 102, and illumination of light-emitting elements of the display panel 108 to illuminate pixels on the display panel 108.

[0065] Figure 4 A flowchart illustrating an exemplary process 400 for applying re-projection adjustments on the HMD 102 based on motion vector data 138(4) generated by the host computer 106 in accordance with the embodiments disclosed herein is shown. For purposes of discussion, the process 400 is described with reference to the preceding figures.

[0066] At 402, the HMD 102 can send head tracking data 122 to the host computer 106. This head tracking data 122 can be generated and / or sent at any suitable frequency, such as a frequency corresponding to a target frame rate of the application 130 and / or a refresh rate of the HMD 102, or a different (e.g., faster) frequency, and the head tracking data 122 can be indicative of head movement of the user 104 wearing the HMD 102. Thus, at block 402, the host computer 106 can receive multiple instances of the head tracking data 122 over time, such as a first head tracking data 122 generated at time ti, a second head tracking data 122 generated at time t2, and so on. Further, the host computer 106 can maintain a history of the head tracking data 122 it receives over time, such that it has multiple instances of head tracking data 122 available at any given time. The host computer 106 can discard head tracking data 122 older than a predefined age to conserve memory resources, while retaining at least some past head tracking data 122.

[0067] At 404, logic of the host computer 106 (e.g., the rendering component 132) can generate motion vector data 138(4) based at least in part on the head tracking data 122 received from the HMD 102. For example, the host computer 106 can receive the first head tracking data 122, and can have received the second head tracking data 122 prior to receiving the first head tracking data 122, and can compare between the set of these head tracking data 122 to generate the motion vector data 138(4). In some embodiments, the motion vector data 138(4) can additionally or alternatively be generated from pixel data associated with multiple frames rendered by the application. For example, a pair of frames previously presented by the application 130 can be provided as input to the GPU 128 of the host computer 106, and a decoder of the GPU 128 can generate motion vector data 138(4) based at least in part on the input frames, such as by finding similarities in the pair of frames (e.g., similar color values, similar brightness values, etc.) and mapping motion vectors to corresponding locations of the similarities in the pair of frames. In some embodiments, the logic of the host computer 106 can be configured to filter out motion related to moving virtual objects in the scene, such that it leaves motion due to head movement and not motion of virtual objects in the scene.

[0068] At 406, the HMD 102 can receive motion vector data 138(4) from the host computer 106. For example, the motion vector data 138(4) can be sent (e.g., encapsulated with) as additional data 138 along with pixel data 136 for a given frame on which its image is to be presented on the HMD 102. Thus, at block 406 or at a slightly different (e.g., earlier or later) time, the HMD 102 can receive the pixel data 136 for the given frame.

[0069] At 408, logic of the HMD 102 (e.g., the compositor 116) can apply re-projection adjustments to the pixel data 136 based at least in part on the motion vector data 138(4) received at block 406 to obtain modified pixel data 410 associated with the given frame. As described herein, the motion vector data 138(4) can be considered in addition to the pose data indicating the original predicted pose and the updated pose determination of the HMD 102. Thus, the motion vector data 138(4) can augment the pose data 138(1) for predicting the pose of the HMD 102 and thereby applying re-projection adjustments (e.g., rotation calculations for moving and / or rotating the pixel data 136 in some manner). Alternatively, the motion vector data 138(4) can be applied directly to the pixel data 136, such as by converting the motion vector data 138(4) into a field of motion vectors corresponding to screen space of the scene to be rendered, and moving pixel values based on the magnitude and direction of the motion vectors corresponding to those pixel values.

[0070] Figure 5 A flowchart illustrating an exemplary process 500 for applying re-projection adjustments based on additional pixel data 138(3) generated by an application 130 executing on a host computer 106 in accordance with embodiments disclosed herein is shown. The process 500 is described with reference to the previous Figures for purposes of discussion.

[0071] At 502, the HMD 102 can send head tracking data 122 to the host computer 106. The head tracking data 122 can be generated and / or sent at any suitable frequency, such as a frequency corresponding to a target frame rate of the application 130 and / or a refresh rate of the HMD 102, or a different (e.g., faster) frequency, and the head tracking data 122 can indicate head movement of the user 104 wearing the HMD 102.

[0072] At 504, logic of host computer 106 (e.g., rendering component 132) can instruct application 130 (such as video game 130(1)) executing on host computer 106 to generate additional pixel data 138(3) in addition to pixel data 136 corresponding to an array of pixels on display panel 108 of HMD 102. For example, if display panel 108 has an array of 2160 x 1200 pixels, pixel data 136 can correspond to pixel values in that 2160 x 1200 array of pixels, and additional pixel data 138(3) can correspond to pixels outside the bounds of that 2160 x 1200 array. Thus, as an example, pixel data 136 and additional pixel data 138(3) requested at block 504 can constitute a larger number of pixel values, e.g., a larger array of 2400 x 1400 pixels. The determination to instruct application 130 to generate additional pixel data 138(3) at block 504 can be a dynamic determination for a given frame, as indicated by the sub-block of block 504.

[0073] At sub-block 506, logic of host computer 106 (e.g., rendering component 132) can determine to generate (e.g., by instructing application 130 to generate) additional pixel data 138(3) based at least in part on head tracking data 122 received at block 502 indicating that an amount of movement of HMD 102 is greater than a threshold amount of movement. For example, two instances of head tracking data 122 can be compared to determine an amount of movement of HMD 102 over a period of time (e.g., a period of time of a few milliseconds), and if the amount of movement is greater than the threshold amount of movement, rendering component 132 can determine to generate additional pixel data 138(3) for a given frame. On the other hand, if the amount of movement of HMD 102 is less than or equal to the threshold amount of movement, rendering component 132 can determine not to generate additional pixel data 138(3) for the given frame. This is illustrated graphically to the right of sub-block 506 in Figure 5 sub-block 506 in FIG. 6, where rendering component 132 is configured to instruct application 130 to render only pixel data 136 (and not any additional pixel data 138(3)) if the amount of HMD movement is at or below the threshold amount of movement, and to render both pixel data 136 and additional pixel data 138(3) if the amount of HMD movement is above the threshold amount of movement. This allows for resource savings by determining that additional pixel data 138(3) is unlikely to be used in cases of current little head movement, and reserving generation of additional pixel data 138(3) for cases of current large and / or rapid head movement.

[0074] At sub-block 508, logic of host computer 106 (e.g., rendering component 132) can determine, based at least in part on the amount of movement of HMD 102 indicated by head tracking data 122 received at block 502, a number of additional pixel values in additional pixel data 138(3) to render (e.g., by instructing application 130 to render). This is shown graphically to the right of sub-block 508 in Figure 5 sub-block 508 in FIG. 7, where rendering component 132 is configured to instruct application 130 to render a first number of additional pixel values in additional pixel data 138(3) if the HMD movement amount is a first, smaller movement amount, and to instruct application 130 to render a second, larger number of additional pixel values in additional pixel data 138(3) if the HMD movement amount is a second, larger movement amount. For example, for a lesser amount of head movement, rendering component 132 can instruct application 130 to render a total of 2300 x 1300 pixels, and for a larger amount of head movement, rendering component 132 can instruct application 130 to render a total of 2500 x 1500 pixels (i.e., a larger amount of additional pixel values for a fixed number of "on-screen" pixel values of pixel data 136). In other words, the additional pixel buffer can expand or contract based on the degree of head movement. This is again a technique to conserve resources on host computer 106 by producing an adequate amount of additional pixel data 138(3) without producing an excessive amount of additional pixel data 138(3).

[0075] At sub-block 510, logic of host computer 106 (e.g., rendering component 132) can determine to generate (e.g., by instructing application 130 to generate) particular additional pixel data 138(3) based at least in part on at least one of motion vector data 138(4) or prediction data known and / or generated by application 130. For example, if motion vector data 138(4) indicates head movement in an upward and leftward direction in a field of view that references the front side of display panel 108 of HMD 102, rendering component 132 can determine to instruct application 130 to generate additional pixel data 138(3) above the topmost pixel in pixel data 136 and to the left of the leftmost pixel of pixel data 136, and to refrain from generating additional pixel data 138(3) to the right of the rightmost pixel of pixel data 136 or below the bottommost pixel of pixel data 136, respectively, which conserves resources of host computer 106. In other words, if user 104 is not likely to move his / her head to the right and / or downward, it is not necessary to render additional pixels to the right of the rightmost pixel in the scene and below the bottommost pixel in the scene because the likelihood that user 104 needs to see those pixels is low. Similarly, if motion vector data 138(4) indicates head movement in a downward and rightward direction in a field of view that references the front side of display panel 108 of HMD 102, rendering component 132 can determine to instruct application 130 to generate additional pixel data 138(3) below the bottommost pixel in pixel data 136 and to the right of the rightmost pixel of pixel data 136, and to refrain from generating additional pixel data 138(3) to the left of the leftmost pixel of pixel data 136 or above the topmost pixel of pixel data 136, respectively, which conserves resources of host computer 106. This is illustrated graphically to the right of sub-block 510 in FIG. 5B. Figure 5

[0076] In some embodiments, at block 510, a determination of which "out-of-bounds" pixels to render can be made from prediction data obtained from application 130. For example, if rendering component 132 learns (or predicts with high likelihood) from information obtained from executing application 130 that an explosion is about to occur on the left side of the screen, rendering component 132 can predict with high likelihood a future head movement in a leftward direction (i.e., toward the explosion). Thus, at sub-block 510, for a given frame, rendering component 132 can instruct application 130 to render "out-of-bounds" pixels beyond the leftmost pixel in the scene in anticipation of a leftward head movement, such that if user 104 moves his / her head to the left as predicted to cope with seeing the explosion, the out-of-bounds pixels are available to present an image of the frame.

[0077] ​At sub-block 512, logic of host computer 106 (e.g., rendering component 132) can determine to generate pixel data 136 at a first resolution and to generate additional pixel data 138(3) at a second resolution lower than the first resolution (e.g., by instructing application 130 to do so). This is another technique to save resources, where it is expected that additional pixel data 138(3) will be used to create re-projection frames in the presence of large scale or fast head movement, otherwise the additional pixel data can not be used, and because the additional pixel data 138(3) is used during large scale or fast head movement, the lower resolution is less likely to be noticed by user 104 when the scene in front of user's 104 eyes moves. In other words, if there is unexpected head movement, it is expected that "out-of-bounds" pixels in additional pixel data 138(3) are used, otherwise discarded, meaning that head movement can occur during image presentation based on additional pixel data 138(3), and users can generally tolerate lower resolution images during head movement because their eyes are busy tracking the scene, and fine details in the scene are generally not noticed during head movement.

[0078] At 514, after rendering component 132 obtains additional pixel data 138(3) from application 130, HMD 102 can receive additional pixel data 138(3) from host computer 106. For example, this additional pixel data 138(3) can be sent (e.g., encapsulated with) the pixel data 136 for a given frame along with its image that is to be presented on HMD 102. Thus, at block 514 or at a slightly different (e.g., earlier or later) time, HMD 102 can receive pixel data 136 for a given frame.

[0079] At 516, logic of the HMD 102 (e.g., the compositor 116) can apply the re-projection adjustment to the pixel data 136 based at least in part on the additional pixel data 138(3) received at block 514 to obtain modified pixel data 518 associated with the given frame. The additional pixel data 138(3) can include additional pixel values outside the bounds of the pixel array of the display panel 108 of the HMD 102 and can be used in the re-projection by replacing at least some of the pixel values included in the pixel data 136 with at least some of the additional pixel values included in the additional pixel data 138(3). For example, if the re-projection adjustment shifts the pixel data 136 to the left, then the additional pixel data 138(3) to the right of the rightmost pixels of the pixel data 136 can be output in the modified pixel data 518. These "out-of-bounds" pixel values (e.g., pixel values beyond the left, right, top, and / or bottom edges of the pixels corresponding to the display panel 108) allow content to be displayed in the final image rather than dark pixels. Although this means that more pixels are rendered than would actually be displayed in the final image, this redundancy compensates for, for example, latency in the wireless communication link between the HMD 102 and the host computer 106. That is, the resource-conserving techniques described with reference to the process 500 can improve the performance of the computing device despite the redundancy taken into account the latency in the transmission of data.

[0080] Figure 6 A flowchart illustrating an example process 600 for applying a re-projection adjustment based on depth data 138(2) generated by an application 130 executing on a host computer 106 in accordance with embodiments disclosed herein is shown. The process 600 is described with reference to the previous figures for purposes of discussion.

[0081] At 602, the HMD 102 can send head tracking data 122 to the host computer 106. The head tracking data 122 can be generated and / or sent at any suitable frequency, such as a frequency corresponding to a target frame rate of the application 130 and / or a refresh rate of the HMD 102, or a different (e.g., faster) frequency, and the head tracking data 122 can be indicative of head movement of the user 104 wearing the HMD 102.

[0082] At 604, the rendering component 132 can instruct the application 130 to generate depth data 138(2) associated with a given frame, and the rendering component 132 can obtain the depth data from the application 130. The depth data 138(2) (or Z-buffer data) can be indicative of occluding objects in the scene, indicating their depth relative to the user's position in world space.

[0083] At 606, the HMD 102 can receive depth data 138(2) from the host computer 106. For example, the depth data 138(2) can be sent (e.g., encapsulated with) the pixel data 136 for a given frame along with its image that is to be presented on the HMD 102 as additional data 138. Thus, the HMD 102 can receive the pixel data 136 for a given frame at block 606 or at a slightly different (e.g., earlier or later) time.

[0084] At 608, the logic of the HMD 102 (e.g., the compositor 116) can apply re-projection adjustments to the pixel data 136 based at least in part on the depth data 138(2) received at block 606 to obtain modified pixel data 610 associated with the given frame. As shown in the sub-blocks of block 608, the modification of the pixel data 136 based on the depth data 138(2) can involve sub-operations.

[0085] At sub-block 612, the logic of the HMD 102 (e.g., the compositor 116) can classify a first subset of pixel values included in the pixel data 136 as foreground pixels 614 and a second subset of pixel values included in the pixel data 136 as background pixels 616 based at least in part on the depth data 138(2). At block 612, one or more intermediate layers can be used to classify pixels, such as an intermediate layer between foreground and background. Thus, pixels included in the pixel data 136 can be classified into multiple layers with any suitable granularity.

[0086] At sub-block 618, applying the re-projection adjustment at block 608 can include modifying a first subset of pixel values classified as foreground pixels 614 and refraining from modifying a second subset of pixel values classified as background pixels 616. In other words, the compositor 116 of the HMD 102 can apply different adjustments or no adjustment to a given pixel depending on the layer in which the pixel is classified (e.g., foreground vs. background). This is again a resource-conserving technique to avoid wasting computational resources in applying re-projection adjustments to background pixels 616, as slight inaccuracies in the background content will be overlooked by the user 104. This technique described in process 600 can be enabled by defining different update rates for the re-projection adjustments applied to different layers of classified pixels, as determined from the depth data 138(2). For example, background pixels corresponding to objects in the background of a scene can be updated (for re-projection) less frequently than foreground pixels corresponding to objects in the foreground of that scene. For example, the logic of the HMD 102 (e.g., the compositor 116) can perform re-projection adjustments on the background pixels 616 at 30 Hz, while performing re-projection adjustments on the foreground pixels 614 at a higher frequency, such as 90 Hz. For a frame rate of 90 Hz, this would mean that the foreground pixels 614 are modified for re-projection adjustment at every frame, while the background pixels 616 are modified for re-projection adjustment at every third frame in the series, meaning that two consecutive frames will not include a re-projection adjustment to the background pixels to conserve resources. This is based at least in part on the concept that re-projection adjustments to the background pixels 616 are expected to be smaller-scale adjustments than re-projection adjustments to the foreground pixels 614, given the same error in the original pose prediction of the HMD 102.

[0087] Figure 7 A flowchart illustrating an exemplary process 700 for the HMD 102 to receive hand tracking data directly from a handheld controller and use the hand tracking data to overlay a virtual hand on a scene rendered by an application, in accordance with implementations disclosed herein, is shown. The process 700 is described with reference to the previous figures for purposes of discussion.

[0088] At 702, the HMD 102 can receive hand tracking data 706 from at least one handheld controller 704 communicatively coupled to the HMD 102. The handheld controller 704 can represent a video game controller, such as a handheld VR controller, whose movement and / or position in 3D space is tracked much like the movement and / or position of the HMD 102 can be tracked. Instead of the handheld controller 704 sending the hand tracking data 706 to the host computer 106 for superimposing virtual hands on the scene at the host computer 106, a latency reduction measure can be taken by having the handheld controller 704 send the hand tracking data 706 directly to the HMD 102 indicating the movement and / or position of the handheld controller 704 without sending the hand tracking data 706 to the host computer 106 before the HMD 102 receives the hand tracking data 706. The hand tracking data 706 can include data generated by tracking sensors mounted on the handheld controller 704, as well as proximity sensor data, possibly generated by proximity sensors (e.g., capacitive sensors) of the handheld controller 704, that indicate the positioning / distance of a finger relative to the handheld controller 704.

[0089] At 708, logic of the HMD 102 (e.g., the compositor 116) can modify the pixel data 136 of a given frame to include one or more virtual hands 710 superimposed on the scene 712 represented by the pixel data 136 based at least in part on the hand tracking data 706 to obtain modified pixel data associated with the given frame. For example, at sub-box 714, the compositor 116 of the HMD 102 can render the virtual hands 710 by replacing the pixel values generated by the application 130 with pixel values generated by the compositor 116 to superimpose the one or more virtual hands 710 on the scene rendered by the application 130. Thus, the HMD 102 can render the virtual hands 710 on the scene 712 based on the hand tracking data 706 that the HMD 102 receives directly from the handheld controller 704. This can allow the re-projection adjustment of the rendering of the virtual hands 710 to be minimized because no pose prediction of the handheld controller 704 has to be made in advance if the data does not have to be transmitted to the host computer 106 and then from the host computer 106 to the HMD 102.

[0090] Figure 8A and Figure 8B Two alternative settings of a system that splits the rendering workload 100 of a frame between the HMD 102 and the host computer 106 are shown in accordance with the embodiments disclosed herein. Briefly referring to FIG. 7, the HMD 102 can receive hand tracking data 706 from at least one handheld controller 704 communicatively coupled to the HMD 102. The handheld controller 704 can represent a video game controller, such as a handheld VR controller, whose movement and / or position in 3D space is tracked much like the movement and / or position of the HMD 102 can be tracked. Instead of the handheld controller 704 sending the hand tracking data 706 to the host computer 106 for superimposing virtual hands on the scene at the host computer 106, a latency reduction measure can be taken by having the handheld controller 704 send the hand tracking data 706 directly to the HMD 102 indicating the movement and / or position of the handheld controller 704 without sending the hand tracking data 706 to the host computer 106 before the HMD 102 receives the hand tracking data 706. The hand tracking data 706 can include data generated by tracking sensors mounted on the handheld controller 704, as well as proximity sensor data, possibly generated by proximity sensors (e.g., capacitive sensors) of the handheld controller 704, that indicate the positioning / distance of a finger relative to the handheld controller 704. Figure 1An example implementation is where the host computer 106 is co-located in an environment with the HMD 102 worn by the user 104. For example, the host computer 106 can be located in a house of the user 104 while the user 104 is using the HMD 102 in that house, regardless of whether the host computer 106 is located in the same room as the HMD 102 or a different room. Alternatively, the host computer 106, in the form of a mobile computing device (e.g., a tablet or laptop), can be carried in a backpack on the back of the user 104, allowing for greater mobility. For example, the user 104 can be located in a park when using such a system.

[0091] Figure 8A An alternative implementation is shown where the host computer 106 represents one or more server computers located in a geographically remote location relative to the HMD 102. In this case, the HMD 102 can be communicatively coupled to the host computer 106 via an access point (AP) 800, such as a wireless AP (WAP), a base station, etc. In the illustrated example, data is exchanged (e.g., streamed) between the host computer 106 and the HMD 102 via the AP 800, such as by streaming data over the Internet.

[0092] Figure 8B Another alternative implementation is shown where the host computer 106 is communicatively coupled to the HMD 102 via an intermediary computing device 802, such as a laptop or tablet computer. Figure 8A and Figure 8B The difference between Figure 8A the AP 800 can simply act as a data routing device that does not perform rendering, while Figure 8B the intermediary computing device 802 can perform a portion of the rendering workload 100. That is, instead of splitting the rendering workload 100 between the host computer 106 and the HMD 102 into two portions, the rendering workload 100 can be divided among more than two devices, such as three devices: the host computer 106, the intermediary computing device 802, and the HMD 102. In Figure 8B the scenario of the intermediary computing device 802 can perform a first set of rendering operations to modify the pixel data 136, and the HMD 102 can perform a final set of rendering operations to modify the modified pixel data.

[0093] Figure 9Exemplary components of a wearable device such as HMD 102 (e.g., a VR headset) and host computer 106 in which the technology disclosed herein can be implemented are shown in accordance with the embodiments disclosed herein. HMD 102 can be implemented as a tethered device communicatively coupled to host computer 106 during operation, and / or as a standalone device. In either mode of operation, HMD 102 will be worn by user 104 (e.g., on the head of user 104). In some embodiments, HMD 102 can be head-mountable, such as by allowing user 104 to secure HMD 102 to his / her head using a securing mechanism (e.g., adjustable bands) sized to fit around the head of user 104. In some embodiments, HMD 102 comprises a virtual reality (VR) or augmented reality (AR) headset that includes a near-eye display. As such, the terms “wearable device,” “wearable electronic device,” “VR headset,” “AR headset,” and “head-mounted display (HMD)” can be used interchangeably herein to refer to devices such as device 102. Figure 9 It should be appreciated, however, that these types of devices are merely examples of HMD 102, and it should be appreciated that HMD 102 can be implemented in various other form factors. It should also be appreciated that some or all of the components shown can be implemented on HMD 102. Thus, in some embodiments, a subset of the components shown as being implemented in HMD 102 can be implemented on host computer 106 or another computing device separate from HMD 102. Figure 9

[0094] In the particular implementation shown, HMD 102 includes processor 110 described above (which can include one or more GPUs 114), as well as memory 112 storing compositor 116 executable by processor 110, display panel 108, head tracking system 118, and communication interface 120.

[0095] ​Additional functional modules are shown as being stored in the computer-readable media 112 and executable on the processor 110, although the same functionality can alternatively be implemented in hardware, firmware, or SOC and / or other logic. For example, for the benefit of other modules, an operating system module 900 can be configured to manage hardware located within and coupled to the HMD 102. In addition to this, in some instances, the HMD 102 can include one or more application programs 902 stored in the memory 112 or otherwise accessible by the HMD 102. For example, the application programs 902 can include, but are not limited to, a video game application (e.g., a basic video game with processing graphics of lesser computational load), a video playback application (e.g., an application that accesses a library of video content stored in the HMD 102 and / or in the cloud), and the like. The HMD 102 can include any number or type of application programs 902 and is not limited to the specific examples described herein.

[0096] In general, the HMD 102 has input devices 904 and output devices 906. The input devices 904 can include control buttons. In some implementations, one or more microphones can be used as input devices 904 to receive audio input, such as user voice input. In some implementations, one or more cameras or other types of sensors (e.g., inertial measurement units (IMUs)) can be used as input devices 904 to receive pose input, such as user 104 hand and / or head movements. In some embodiments, additional input devices 904 can be provided in the form of a keyboard, keypad, mouse, touch screen, joystick, etc. In other embodiments, the HMD 102 can omit a keyboard, keypad, or other similar form of mechanical input. Instead, the HMD 102 can be implemented with a relatively simple form of input devices 904, a network interface (wired or wireless based), power, and processing / storage capabilities. For example, a limited set of one or more input components (e.g., a dedicated button for initiating configuration, power on / off, etc.) can be employed so that the HMD 102 can be used thereafter. In one implementation, the input devices 904 can include control mechanisms such as a base volume control button for increasing / decreasing volume, as well as power and reset buttons.

[0097] The output devices 906 can include the display panel 108, which can include one or more display panels 108 (e.g., a pair of stereoscopic display panels 108), as described herein. The output devices 906 can also include, but are not limited to, light emitting elements (e.g., LEDs), vibrators for generating haptics, speakers (e.g., earphones), etc. There can also be simple light emitting elements (e.g., LEDs) to indicate status, such as when powered on, for example.

[0098] The HMD 102 can also include a communication interface 120 including, but not limited to, a wireless unit 908 coupled to an antenna 910 to facilitate wireless connection with a network and / or with a second device, such as the host computer 106. The wireless unit 908 can implement one or more of various wireless technologies, such as Wi-Fi, Bluetooth, radio frequency (RF), etc. It is appreciated that the HMD 102 can also include physical ports to facilitate wired connection with a network and / or with a second device, such as the host computer 106.

[0099] The HMD 102 can also include an optical subsystem 912 that directs light from the display panel 108 to the user’s eye using one or more optical elements. The optical subsystem 912 can include different types of different optical elements and combinations thereof, including but not limited to, such as apertures, lenses (e.g., Fresnel lenses, convex lenses, concave lenses, etc.), filters, etc. In some embodiments, one or more of the optical elements in the optical subsystem 912 can have one or more coatings, such as an anti-reflective coating. Magnification of the image light by the optical subsystem 912 allows the display panel 108 to be physically smaller, lighter in weight, and consume less power compared to larger displays. Additionally, magnification of the image light can increase the field of view (FOV) of the displayed content (e.g., images). For example, the FOV of the displayed content is such that the displayed content is presented using almost all (e.g., 120 to 150 degrees diagonal) and in some cases all of the user’s FOV. AR applications can have a narrower FOV (e.g., about 40 degree FOV). The optical subsystem 912 can be designed to correct for one or more optical errors, such as but not limited to, barrel distortion, pincushion distortion, longitudinal chromatic aberration, transverse chromatic aberration, spherical aberration, coma, field curvature, astigmatism, etc. In some embodiments, the content provided to the display panel 108 for display is pre-distorted (e.g., pre-distorted by the applied geometric distortion adjustments and / or chromatic aberration adjustments described herein) and the optical subsystem 912 corrects for this distortion when it receives the image light generated based on the content from the display panel 108.

[0100] The HMD 102 can also include one or more sensors 914, such as sensors to generate motion, position, and orientation data. These sensors 914 can be or include gyroscopes, accelerometers, magnetometers, cameras, color sensors, or other motion, position, and orientation sensors. The sensors 914 can also include subparts of sensors, such as a series of active or passive markers that can be externally observed by a camera or color sensor in order to generate motion, position, and orientation data. For example, a VR headset can include a plurality of markers on its exterior, such as reflectors or lights (e.g., infrared or visible lights), that can provide one or more reference points for software to interpret in order to generate motion, position, and orientation data when observed by an external camera or illuminated by light (e.g., infrared or visible light). The HMD 102 can include light sensors that are sensitive to light (e.g., infrared or visible light) projected or scattered into the environment of the HMD 102 by a base station.

[0101] In one example, the sensors 914 can include an inertial measurement unit (IMU) 916. The IMU 916 can be an electronic device that generates calibration data based on measurement signals received from accelerometers, gyroscopes, magnetometers, and / or other sensors or some combination thereof adapted to detect motion, correct for errors associated with the IMU 916. Based on the measurement signals, such motion-based sensors, such as the IMU 916, can generate calibration data that indicates an estimated position of the HMD 102 relative to an initial position of the HMD 102. For example, multiple accelerometers can measure translational motion (forward / back, up / down, left / right) and multiple gyroscopes can measure rotational motion (e.g., pitch, yaw, and roll). The IMU 916 may, for example, sample the measurement signals rapidly and compute an estimated position of the HMD 102 from the sampled data. For example, the IMU 916 can integrate the measurement signals received from the accelerometers over time to estimate a velocity vector and integrate the velocity vector over time to determine an estimated position of a reference point on the HMD 102. A reference point is a point that can be used to describe the position of the HMD 102. While a reference point can generally be defined as a point in space, in various implementations, the reference point is defined as a point within the HMD 102 (e.g., the center of the IMU 916). Alternatively, the IMU 916 provides the sampled measurement signals to an external console (or other computing device) that determines the calibration data.

[0102] The sensors 914 can operate at a relatively high frequency in order to provide sensor data at a high rate. For example, the sensor data can be generated at a rate of 1000 Hz (or one sensor reading taken every 1 millisecond). As such, one thousand readings are taken per second. When the sensors generate this much data at this rate (or at a greater rate), the data set for predicting motion is quite large, even for a relatively short period of time of about tens of milliseconds.

[0103] As mentioned, in some embodiments, the sensors 914 can include light sensors that are sensitive to light emitted by base stations in the environment of the HMD 102 for the purpose of tracking the position and / or orientation, pose, etc. of the HMD 102 in 3D space. The calculation of the position and / or orientation can be based on the timing characteristics of the light pulses and the presence or absence of light detected by the sensors 914.

[0104] The HMD 102 can also include an eye tracking system 918 that generates eye tracking data. The eye tracking system 918 can include, without limitation, a camera or other optical sensor inside the HMD 102 to capture image data (or information) of the user's eyes, and the eye tracking system 918 can use the captured data / information to determine a motion vector, an interpupillary distance, an interocular distance, a three-dimensional (3D) position of each eye relative to the HMD 102 including the magnitude of the twist and rotation (i.e., roll, pitch, and yaw) of each eye, and a gaze direction. In one example, infrared light is emitted within the HMD 102 and reflected from each eye. The reflected light is received or detected by a camera of the eye tracking system 918 and analyzed to extract the eye rotation from the changes in the infrared light reflected from each eye. The eye tracking system 918 can use many methods for tracking the eyes of the user 104. Thus, the eye tracking system 918 can track up to six degrees of freedom of each eye (i.e., 3D position, roll, pitch, and yaw), and at least a subset of the quantities tracked from both eyes of the user 104 can be combined to estimate a gaze point (i.e., a 3D positioning of where the user is looking or a location in a virtual scene), which can be mapped to a location on the display panel 108 for predicting where the user 104 will be looking in terms of individual subsets (e.g., rows) or groups of contiguous subsets (e.g., groups of contiguous rows) of pixels of the display panel 108. For example, the eye tracking system 918 can integrate information from past measurements, measurements that identify the position of the head of the user 104, and 3D information that describes the scene being presented by the display panel 108. Thus, information for the position and orientation of the eyes of the user 104 is used to determine a gaze point in a virtual scene being presented by the HMD 102 that the user 104 is looking at and map that gaze point to a location on the display panel 108 of the HMD 102.

[0105] The HMD 102 can also include the head tracking system 118 described above. The head tracking system 118 can utilize one or more of the sensors 914 to track head motion (including head rotation) of the user 104, as described above. For example, the head tracking system 118 can track up to six degrees of freedom of the HMD 102 (i.e., 3D position, roll, pitch, and yaw). These calculations can be made at each frame in a series of frames so that the application 130 can determine how to render the scene in the next frame according to the head position and orientation. In some embodiments, the head tracking system 118 is configured to generate head tracking data 122 that can be used to predict future poses (positions and / or orientations) of the HMD 102 based on current and / or past data and / or based on known / implicit scan-out latencies of the pixels in various subsets of the display system. This is because the application 130 is required to render a frame before the user 104 actually sees the light (and thus the image) on the display 108. Thus, the next frame can be rendered based on this future prediction of the head position and / or orientation made at an earlier point in time. The rotational data provided by the head tracking system 118 can be used to determine both a direction of rotation of the HMD 102 and an amount of rotation of the HMD 102 in any suitable unit of measurement. For example, the direction of rotation can be simplified and output according to positive or negative horizontal directions and positive or negative vertical directions corresponding to left, right, up, and down. The amount of rotation can be in terms of degrees, radians, etc. An angular velocity can be calculated to determine a rate of rotation of the HMD 102.

[0106] In the particular implementation shown, the host computer includes the processor 124 described above (which can include one or more GPUs 128), as well as a memory 126 storing the application 130 and the rendering component 132 executable by the processor 128, and a communication interface 134.

[0107] The memory 126 can also include an operating system 920 configured to manage hardware within and coupled to the host computer 106 to facilitate other modules. The host computer 106 can also have a video game client 922 installed in the memory 126. The video game client 922 can represent an executable client application configured to launch and execute programs such as video games (or video game programs). In other words, the video game client 922 can include game software that can be used to play video games on a system including the HMD 102 and the host computer 106. After installing the video game client 922, the host computer 106 can then have the ability to receive (e.g., download, stream, etc.) video games from a remote system over a computer network (e.g., the Internet) and execute these video games via the video game client 922. Any type of content distribution model can be used for this purpose, such as a direct purchase model where video games can be purchased individually to download and execute on the host computer 106, a subscription-based model, a content distribution model where video games are rented or leased for a period of time, etc. Thus, the host computer 106 can include one or more video games within a video game library 924, such as the video game 130(1). These video games can be retrieved and executed by loading the video game client 922. In one example, the user 104 can select one of a plurality of video games that they have purchased and downloaded to the video game library 924 by loading the video game client 922 and selecting the video game 130(1) to start executing the video game 130(1). The video game client 922 can allow the user to log in to a video game service using credentials (e.g., a user account, a password, etc.).

[0108] The host computer 106 can also include a communication interface 134 including, but not limited to, a wireless unit 926 coupled to an antenna 928 to facilitate wireless connection with a network and / or with a second device, such as the HMD 102. The wireless unit 926 can implement one or more of a variety of wireless technologies, such as Wi-Fi, Bluetooth, radio frequency (RF), etc. It should be understood that the host computer 106 can also include physical ports to facilitate wired connection with a network and / or with a second device, such as the HMD 102.

[0109] In general, the host computer 106 has input device 930 and output device 932. The input device 930 can be a keyboard, a keypad, a mouse, a touch screen, a joystick, control buttons, a microphone, a camera, etc. The output device 932 can include, but is not limited to, a display, a light-emitting element (e.g., an LED), a vibrator to generate a haptic sensation, a speaker (e.g., headphones), etc.

[0110] Although the subject matter has been described in language specific to structural features, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features described. Rather, the specific features are disclosed as illustrative forms of implementing the claims.

Claims

1. A head-mounted display (HMD), comprising: one or more display panels having an array of light emitting elements; Head tracking system; processor; and a memory storing computer-executable instructions that, when executed by the processor, cause the HMD to: sending first head tracking data generated by the head tracking system to a host computer communicatively coupled to the HMD; receiving, from the host computer and based at least in part on the first head tracking data: pixel data associated with a first frame, the first frame being rendered at a frame rate by an application executing on the host computer; as well as Additional data in addition to the pixel data, the additional data including: pose data indicating a predicted pose of the HMD, the pose data being used by the application to generate the pixel data; and depth data associated with the first frame; determining, based at least in part on second head tracking data generated by the head tracking system, an updated pose that the HMD will be in at the time the light emitting element will illuminate the first frame; classifying the pixel data into foreground pixels and background pixels based at least in part on the depth data; applying a reprojection adjustment to the pixel data to obtain modified pixel data associated with the first frame based at least in part on the depth data and a comparison between the predicted pose and the updated pose, wherein applying the reprojection adjustment to the pixel data based at least in part on a comparison between the predicted pose and the updated pose comprises compensating for differences between the predicted pose and the updated pose by shifting and / or rotating the pixel data, and Wherein applying the reprojection adjustment to the pixel data based at least in part on the depth data comprises: modifying the foreground pixels based at least in part on a first rate at which the reprojection adjustments are applied to the foreground pixels; and modifying the background pixels based at least in part on a second rate at which the reprojection adjustments are applied to the background pixels, the second rate being less than the first rate and less than the frame rate; and A first image is presented on the one or more display panels based at least in part on the modified pixel data.

2. The HMD according to claim 1, wherein: The host computer is wirelessly coupled to the HMD; The first head tracking data is wirelessly transmitted to the host computer; and The pixel data and the additional data are received wirelessly from the host computer.

3. The HMD according to claim 1, wherein: The computer-executable instructions, when executed by the processor, further cause the HMD to: receiving second pixel data and second depth data associated with a second frame from the host computer; classifying the second pixel data into second foreground pixels and second background pixels based at least in part on the second depth data, Applying a second re-projection adjustment to the second pixel data based at least in part on the second depth data to obtain second modified pixel data associated with the second frame, wherein applying the second re-projection adjustment to the second pixel data based at least in part on the second depth data comprises: modifying the second foreground pixel based at least in part on the first rate; and refraining from modifying the background pixels based at least in part on the second rate; and presenting a second image on the one or more display panels based at least in part on the second modified pixel data.

4. The HMD of claim 1 , wherein the computer-executable instructions, when executed by the processor, further cause the HMD to: receiving hand tracking data from at least one handheld controller communicatively coupled to the HMD; and Based at least in part on the hand tracking data, the pixel data is modified to include one or more virtual hands superimposed on a scene represented by the pixel data to obtain the modified pixel data associated with the first frame.

5. The HMD according to claim 1, wherein: Classifying the pixel data based at least in part on the depth data includes classifying the pixel data into the foreground pixels, the background pixels, and intermediate layer pixels located between the foreground pixels and the background pixels; as well as Applying the reprojection adjustment to the pixel data based at least in part on the depth data also includes modifying the intermediate layer pixels based at least in part on a third rate, and the reprojection adjustment is applied to the intermediate layer pixels at the third rate, which is different from the first rate and the second rate.

6. The HMD according to claim 1, wherein: The computer-executable instructions, when executed by the processor, further cause the HMD to: outputting the modified pixel data to a frame buffer, Wherein, presenting the first image on the one or more display panels based at least in part on the modified pixel data comprises: scanning out the modified pixel data to the one or more display panels; and The light emitting elements illuminate the one or more display panels.

7. A method implemented by a head-mounted display (HMD), the HMD comprising one or more display panels having an array of light-emitting elements, the method comprising: sending first head tracking data generated by a head tracking system of the HMD to a host computer; receiving, from the host computer and based at least in part on the first head tracking data: pixel data associated with a first frame, the first frame being rendered at a frame rate by an application executing on the host computer; as well as Additional data, including: pose data indicating a predicted pose of the HMD, the pose data being used by the application to generate the pixel data; and depth data associated with the first frame; determining, based at least in part on second head tracking data generated by the head tracking system, an updated pose that the HMD will be in at the time the light emitting element will illuminate the first frame; classifying the pixel data into foreground pixels and background pixels based at least in part on the depth data; applying a reprojection adjustment to the pixel data to obtain modified pixel data associated with the first frame based at least in part on the depth data and a comparison between the predicted pose and the updated pose, wherein applying the reprojection adjustment to the pixel data based at least in part on a comparison between the predicted pose and the updated pose comprises compensating for differences between the predicted pose and the updated pose by shifting and / or rotating the pixel data, and Wherein applying the reprojection adjustment to the pixel data based at least in part on the depth data comprises: modifying the foreground pixels based at least in part on a first rate at which the reprojection adjustments are applied to the foreground pixels; and modifying the background pixels based at least in part on a second rate at which the reprojection adjustments are applied to the background pixels, the second rate being less than the first rate and less than the frame rate; and A first image is presented on the one or more display panels based at least in part on the modified pixel data.

8. The method according to claim 7, wherein: The host computer is wirelessly coupled to the HMD; The first head tracking data is wirelessly transmitted to the host computer; and The pixel data and the additional data are received wirelessly from the host computer.

9. The method according to claim 7, further comprising: receiving second pixel data and second depth data associated with a second frame from the host computer; classifying the second pixel data into second foreground pixels and second background pixels based at least in part on the second depth data, Applying a second re-projection adjustment to the second pixel data based at least in part on the second depth data to obtain second modified pixel data associated with the second frame, wherein applying the second re-projection adjustment to the second pixel data comprises: modifying the second foreground pixel based at least in part on the first rate; and refraining from modifying the second background pixels based at least in part on the second rate; and presenting a second image on the one or more display panels based at least in part on the second modified pixel data.

10. The method according to claim 7, further comprising: receiving hand tracking data from at least one handheld controller communicatively coupled to the HMD; as well as Based at least in part on the hand tracking data, the pixel data is modified to include one or more virtual hands superimposed on a scene represented by the pixel data to obtain the modified pixel data associated with the first frame.

11. The method according to claim 7, wherein: Classifying the pixel data based at least in part on the depth data includes classifying the pixel data into the foreground pixels, the background pixels, and intermediate layer pixels located between the foreground pixels and the background pixels; as well as Applying the reprojection adjustment to the pixel data based at least in part on the depth data also includes modifying the intermediate layer pixels based at least in part on a third rate, and the reprojection adjustment is applied to the intermediate layer pixels at the third rate, which is different from the first rate and the second rate.

12. A method implemented by a head-mounted display (HMD), wherein the head-mounted display includes one or more display panels, the method comprising: sending first head tracking data generated by a head tracking system of the HMD to a host computer; receiving, from the host computer and based at least in part on the first head tracking data, gesture data indicative of a predicted gesture; receiving pixel data and depth data associated with a first frame from the host computer, the first frame being rendered at a frame rate by an application executing on the host computer, wherein receiving the pixel data associated with the first frame is based at least in part on the first head tracking data; determining an update pose based at least in part on second head tracking data generated by the head tracking system; classifying the pixel data into foreground pixels and background pixels based at least in part on the depth data; applying a reprojection adjustment to the pixel data to obtain modified pixel data associated with the first frame based at least in part on the depth data and a comparison between the predicted pose of the HMD used by the application to generate the pixel data and the updated pose that the HMD will be in when light emitting elements of the one or more display panels will illuminate the first frame, wherein applying the reprojection adjustment to the pixel data based at least in part on a comparison between the predicted pose and the updated pose comprises compensating for differences between the predicted pose and the updated pose by shifting and / or rotating the pixel data, and Wherein applying the reprojection adjustment to the pixel data based at least in part on the depth data comprises: modifying the foreground pixels based at least in part on a first rate at which the reprojection adjustments are applied to the foreground pixels; and modifying the background pixels based at least in part on a second rate at which the reprojection adjustments are applied to the background pixels, the second rate being less than the first rate and less than the frame rate; and A first image is presented on the one or more display panels based at least in part on the modified pixel data.

13. The method according to claim 12, wherein: The pixel data and the depth data are received wirelessly from the host computer.

14. The method according to claim 12, further comprising: receiving second pixel data and second depth data associated with a second frame from the host computer; classifying the second pixel data into second foreground pixels and second background pixels based at least in part on the second depth data, Applying a second re-projection adjustment to the second pixel data based at least in part on the second depth data to obtain second modified pixel data associated with the second frame, wherein applying the second re-projection adjustment to the second pixel data comprises: modifying the second foreground pixel based at least in part on the first rate; and refraining from modifying the second background pixels based at least in part on the second rate; and presenting a second image on the one or more display panels based at least in part on the second modified pixel data.

15. The method according to claim 12, further comprising: receiving hand tracking data from at least one handheld controller communicatively coupled to the HMD; as well as Based at least in part on the hand tracking data, the pixel data is modified to include one or more virtual hands superimposed on a scene represented by the pixel data to obtain the modified pixel data associated with the first frame.

16. The method according to claim 12, wherein: The applications include video games.

17. The method of claim 12, wherein: Classifying based at least in part on the depth data includes classifying the pixel data into the foreground pixels, the background pixels, and intermediate layer pixels located between the foreground pixels and the background pixels; as well as Applying the reprojection adjustment to the pixel data based at least in part on the depth data also includes modifying the intermediate layer pixels based at least in part on a third rate, and the reprojection adjustment is applied to the intermediate layer pixels at the third rate, which is different from the first rate and the second rate.

18. The method according to claim 12, further comprising: outputting the modified pixel data to a frame buffer, Wherein, presenting the first image on the one or more display panels based at least in part on the modified pixel data comprises: scanning out the modified pixel data to the one or more display panels; and The light emitting elements illuminate the one or more display panels.

Images