Motion smoothing in distributed systems

Abstract

This paper describes a motion smoothing technique for displays or display systems, such as head-mounted displays (HMDs), that takes into account the motion of moving or animated objects in a way that reduces jitter. The display system may be separate from, but communicatively coupled to, a host computer, where graphics-based applications, such as video games, are outputting frames for rendering on the display system. The host computer can generate motion vectors that represent compressed pixel data transmitted to the display system. These motion vectors can be used by the display system to modify the pixel data of the frame. The modified pixel data of the frame is then "motion smoothed" for rendering on the display system in a way that reduces jitter in moving or animated objects.

Description

[0001] Cross-reference to related applications

[0002] This PCT application claims priority to co-pending and co-assigned U.S. Patent Application Serial No. 16 / 792,080, filed on February 14, 2020, entitled “MOTION SMOOTHING IN ADISTRIBUTED SYSTEM,” the entire contents of which are incorporated herein by reference. Background Technology

[0003] Virtual reality (VR) systems are used within the video game industry and other applications. VR systems can be configured as integrated (or standalone) VR headsets or distributed systems. In a distributed setup, the VR headset can be physically tethered or wirelessly connected to a host computer. In this distributed setup, the host computer typically executes graphics-based applications, such as video games, that output frames, and the VR headset displays these frames as they are streamed from the host computer. This type of setup leverages the host computer's high computing power to display high-quality images on a lightweight VR headset that functions much like a "thin client" device. Furthermore, displays used in VR systems (such as those embedded in VR headsets) typically operate at the lowest refresh rate suitable for VR applications. For example, 90 Hz is a common refresh rate for VR displays. In "real-time rendering" scenarios, graphics-based applications output frames for rendering at a frame rate that matches the display's refresh rate. In this scenario, assuming frames are transmitted from the host computer to the VR headset at a sufficient data transfer rate, new frames output by the application (referred to as "actual frames" in this document) can be displayed with each screen refresh. This real-time rendering scenario is often referred to as the application "reaching frame rate".

[0004] In reality, applications don't always reach the frame rate for various reasons. For example, an application may intermittently lose frames, and / or it may temporarily output frames at a slower rate (e.g., 45 frames per second when the ideal frame rate is 90 frames per second). Furthermore, in distributed systems, network congestion can cause latency in the rate at which data is transmitted from the host computer to the VR headset. In these cases, a technique known as "rotation-only reprojection" can be used to replace lost frames with reprojected frames in a way that takes into account the user's head rotation, making it appear from the user's perspective that the lost frames are not present. Without reprojection, for example, insufficient frame rate from the application or frame latency reaching the VR headset can cause stuttering or lag in the game. In VR applications, where the user is fully immersed in the virtual environment, if frames are lost and there is no reprojection to compensate for them, the user may experience nausea. Therefore, reprojection is a technique that allows for a better user experience when frames are lost. Consider the following example: an application is outputting frames at half the ideal frame rate (e.g., 45 frames per second when the ideal frame rate is 90 frames per second). In this example, every other frame, pixel data from the most recently rendered actual frame is reprojected to create a reprojected frame that transforms the scene (e.g., calculated through rotation and reprojection) to match the reprojected scene with the user's current head orientation. This makes it appear from the user's perspective that the scene is moving in the expected way, taking into account the user's head rotation, even when the reprojected frame is used to compensate for lost frames.

[0005] While rotation-only reprojection prevents stuttering or lag in gameplay, it introduces its own set of visual artifacts during head rotation, at least in VR systems using low-persistence displays (e.g., displays illuminated for only a fraction of a frame). For example, while rotation-only reprojection takes head rotation into account, it doesn't account for virtual objects moving or animating within the scene between frames. This can result in a visual artifact known as "jitter" regarding moving or animated objects. Jitter causes the user to perceive a "double-shadow effect," where moving objects (e.g., a bullet or ball moving across the screen) appear to wobble frame-by-frame between two locations (or separate from themselves). Therefore, any moving or animated object in the scene will jitter when the user rotates their head while using reprojection.

[0006] This article provides technical solutions for improving and enhancing these and other systems. Attached Figure Description

[0007] The accompanying drawings provide a detailed description. In these drawings, the leftmost digit of the reference numeral indicates the drawing in which that reference numeral first appears. The same reference numerals are used in different drawings to indicate similar or identical parts or features.

[0008] Figure 1 This is a flowchart of an exemplary process according to an embodiment disclosed herein, which is used to render a frame on the HMD when an application executing on a host computer communicatively coupled to the head-mounted display (HMD) outputs a frame. Figure 1 This demonstrates a motion smoothing technique that can be implemented during reprojection to account for moving or animated objects in the scene.

[0009] Figure 2 This is a diagram illustrating two exemplary timelines according to the embodiments disclosed herein, showing the corresponding rendering workloads of the host computer and the HMD.

[0010] Figure 3 This is an illustration of an exemplary motion smoothing technique that takes into account moving or animated objects in a scene when reprojection is used to render frames on a display such as an HMD.

[0011] Figure 4 This is an illustration showing an exemplary rendering mesh and how it can be used in motion smoothing of a reprojected frame.

[0012] Figure 5 This is a flowchart of an exemplary process according to an embodiment disclosed herein, which is used to generate reprojected frames using motion vectors generated by a graphics processing unit (GPU) as part of a motion smoothing technique.

[0013] Figure 6 This is a flowchart of an exemplary process according to an embodiment disclosed herein, which is used to align pixel data of a previously rendered frame before inputting pixel data into a GPU for motion vector estimation.

[0014] Figure 7 This is a flowchart of an exemplary process according to an embodiment disclosed herein, which excludes some pixel data from previously rendered frames and provides the remaining pixel data as input to the GPU for motion vector estimation.

[0015] Figure 8 This is a flowchart of an exemplary process according to an embodiment disclosed herein, which is used to align pixel data of a previously rendered frame, exclude panel mask portions of the pixel data of the previously rendered frame, and provide the remaining portion of the pixel data as input to the GPU for motion vector estimation.

[0016] Figure 9This is a flowchart of an exemplary process according to an embodiment disclosed herein, which is used to threshold motion vectors before they are used to generate reprojected frames as part of a motion smoothing technique.

[0017] Figure 10 This is a flowchart of an exemplary process according to an embodiment disclosed herein, which is used to attenuate motion vectors output from the GPU before they are used to generate reprojected frames as part of a motion smoothing technique.

[0018] Figure 11 This is a flowchart of an exemplary process according to an embodiment disclosed herein, which is used to generate a decay texture that can be used to decay these motion vectors before the motion vectors output from the GPU are used to generate reprojected frames as part of a motion smoothing technique.

[0019] Figure 12 This is a flowchart of an exemplary process according to an embodiment disclosed herein, which is used to zero out motion vectors corresponding to regions with little color change before the resulting set of motion vectors is used to generate a reprojected frame as part of a motion smoothing technique.

[0020] Figure 13 This is a flowchart of an exemplary process according to an embodiment disclosed herein, which is used to “clean up” the motion vector field using one or more filters before the resulting set of motion vectors is used to generate a reprojected frame as part of a motion smoothing technique.

[0021] Figure 14 This is a flowchart of an exemplary process according to an embodiment disclosed herein, which is used to rotate a previously rendered frame before inputting pixel data to the GPU for motion vector estimation.

[0022] Figure 15 This is a flowchart of an exemplary process according to an embodiment disclosed herein, which is used to select between a motion vector array generated based on luminance data and chrominance data of a previously rendered frame.

[0023] Figure 16 This is a flowchart of an exemplary process according to an embodiment disclosed herein, which is used to obtain multiple motion vector arrays, determine the differences between the arrays, and generate a final motion vector array for motion smoothing based on the determined differences.

[0024] Figure 17 This is a flowchart of an exemplary process according to an embodiment disclosed herein, which is used to obtain multiple motion vector arrays for different portions of an image region at different resolutions.

[0025] Figure 18A and Figure 18B Two alternative configurations of a system utilizing an HMD and a host computer according to the embodiments disclosed herein are shown.

[0026] Figure 19 Exemplary components of a wearable device (such as a VR headset) in which the techniques disclosed herein can be implemented are shown.

[0027] Figure 20 Exemplary components of a host computer in which the techniques disclosed herein can be implemented are shown. Detailed Implementation

[0028] This document describes motion smoothing techniques that take into account moving or animated objects in a scene when rendering frames on a display of a distributed system. The motion smoothing techniques described herein mitigate unwanted visual artifacts related to moving or animated objects, such as the jitter artifacts described earlier for moving or animated objects. Head-mounted displays (HMDs) are an exemplary type of display or display system that can implement the disclosed motion smoothing techniques when rendering images on a display. Users can wear HMDs to achieve the purpose of immersing themselves in a virtual reality (VR) or augmented reality (AR) environment. The display or display system (such as an HMD) may be separate from a host computer but communicatively coupled to it. An application (e.g., a video game) executes on the host computer and generates pixel data for each frame in a series of frames. The pixel data is sent frame-by-frame to the HMD, and one or more display panels of the HMD render images based on the received pixel data. Users view these images through optics included in the HMD, thereby perceiving the images as if they were immersed in a VR or AR environment.

[0029] As described above, display systems (such as HMDs) can utilize a technique called "reprojection" to compensate for applications failing to reach the frame rate. For example, reprojected frames can be rendered between actual frames to achieve the desired frame rate, and each reprojected frame can be generated using pixel data from recently rendered actual frames (e.g., the most recently rendered actual frame) output by an application running on the host computer. In the reprojected frame, the scene rendered in the previous actual frame is transformed in a way that takes into account the user's head rotation (e.g., through rotation and reprojection calculations).

[0030] This paper describes motion smoothing techniques for modifying pixel data of frames in a distributed system to further consider the motion of objects moving or animated frame by frame. The motion smoothing techniques described herein use computer vision algorithms to estimate the motion (e.g., direction and magnitude) of objects across multiple frames in the form of motion vectors. For example, a motion vector array can be generated by a host computer as a result (e.g., as a byproduct) of compressing pixel data before it is transmitted to a display system (such as an HMD). Thus, the motion vector array can represent the compressed pixel data, at least for some frames in a series of frames. In an exemplary example, a video encoder (e.g., a video encoding chip) of a graphics processing unit (GPU) on a host computer can analyze pixel data from multiple previously rendered frames to generate a motion vector array representing the compressed pixel data of a given frame. The motion vectors can be sent to the HMD, and the HMD can use the motion vectors to modify the decompressed pixel data, in addition to using the motion vectors to obtain the decompressed pixel data, to consider the movement or animation of objects. In other words, motion vectors received from the host computer can be used on the HMD to extrapolate where the object should be located in the frame to be rendered (e.g., reprojected), thus reducing jitter of moving objects.

[0031] In an exemplary motion smoothing process, associated pixel data can be compressed on a host computer. A video encoder on the host computer's graphics processing unit (GPU) can be used to compress the pixel data. As a result of compressing the pixel data, the video encoder can generate a motion vector array representing the compressed pixel data. The host computer can send this motion vector array to a display system, such as an HMD. In some embodiments, the host computer can be wirelessly coupled to the HMD, but a wired connection is also envisioned herein. The HMD can receive the motion vector array from the host computer. The HMD can decompress the compressed first pixel data to obtain the pixel data (or use a decompression or decoding algorithm to obtain an approximation of the pixel data). The HMD can further modify the pixel data based at least in part on the motion vector array received from the host computer to obtain modified pixel data. This modified pixel data is "motion smoothed" to mitigate (if not eliminate) any jitter with respect to moving or animated objects in the scene. The motion-smoothed frame can then be rendered by presenting an image on the HMD's display panel based at least in part on the modified pixel data.

[0032] The motion smoothing technique described herein provides a more realistic, higher-fidelity viewing experience, where objects move or animate within the scene as expected, even when frames are lost. As mentioned above, frames can be lost when an application fails to meet the frame rate, possibly due to network congestion or other reasons. Therefore, the motion smoothing technique described herein compensates for these events by maintaining a high-fidelity user experience in distributed systems when frames can sometimes be lost. This motion smoothing technique also compensates for the inherent latency in frame transmission, even in real-time rendered scenes when no frames are lost (i.e., when frames are received in time at the HMD). Furthermore, in wireless implementations, the technique described herein enables motion smoothing correction in a wireless HMD without adding additional components to the HMD, compared to many integrated (or standalone) headsets on the market today, allowing for lighter headsets that don't get too hot and are more comfortable to wear. Moreover, the technique and system described herein can utilize pre-generated motion vectors to compress and transmit data from the host computer to the HMD, meaning there are no additional transmission costs when using motion vectors to smooth the reprojected frames on the HMD.

[0033] It should be understood that the HMD can extrapolate to future frames (e.g., reprojected frames) using an array of motion vectors received from the host computer, because modified second pixel data is output to that frame buffer on the HMD after the first pixel data is output to the frame buffer. In this way, in the example described herein, the first frame is rendered before the reprojected frame. This extrapolation technique contrasts with interpolation between frames, and it should be understood that the techniques and systems described herein involve extrapolating to future frames using motion vectors received from the host computer, rather than using motion vectors for interpolation between frames.

[0034] This document also discloses systems, such as display systems (e.g., HMDs) configured to implement the techniques and processes disclosed herein, and systems storing non-transitory computer-readable media for implementing the techniques and processes disclosed herein. Although the techniques and systems disclosed herein are discussed by way of example in the context of video game applications, particularly VR game applications, it should be understood that the techniques and systems described herein can benefit other applications, including but not limited to non-VR applications (e.g., AR applications) and / or non-gaming applications (such as industrial machine applications, defense applications, robotics applications, etc.). Furthermore, although HMDs are provided as an example of display systems for displaying images, it should be understood that other types of display systems can benefit from the motion smoothing techniques described herein, such as handheld display devices streaming video content from host computers, relatively large wall-mounted or billboard display systems, etc.

[0035] The processes described herein are illustrated as a collection of boxes in a logic flowchart, representing a series of operations (i.e., logic) that can be implemented in hardware, software, firmware, or a combination thereof. In a software environment, these boxes represent computer-executable instructions that, when executed by one or more processors, perform the enumerated operations. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific abstract data type. The order in which the operations are described is not intended to be construed as limiting, and any number of the described boxes can be combined in any order and / or in parallel to implement these processes.

[0036] Figure 1 This is a flowchart of an exemplary process 100 according to an embodiment disclosed herein, which is used to render frames on HMD 102 when an application executing on a host computer 106 communicatively coupled to a head-mounted display (HMD) 102 (worn by a user 104) outputs a frame. Figure 1 This demonstrates a motion smoothing technique that can be implemented during reprojection to account for moving or animated objects in the scene.

[0037] exist Figure 1 At the top, HMD 102 is shown being worn by user 104, and host computer 106 is shown being communicatively coupled to HMD 102. Host computer 106 may be implemented as any type of computing device and / or any number of computing devices, including but not limited to personal computers (PCs), laptop computers, desktop computers, portable digital assistants (PDAs), mobile phones, tablet computers, set-top boxes, game consoles, server computers, wearable computers (e.g., smartwatches, etc.) or any other electronic device capable of sending / receiving data.

[0038] Host computer 106 and HMD 102 collectively represent a distributed system for executing applications (e.g., video games) and rendering associated images on a display. In some embodiments, host computer 106 may coexist with HMD 102 in the same environment, such as the home of user 104 wearing HMD 102. Alternatively, host computer 106 may be remotely located relative to HMD 102, such as a host computer 106 in the form of a server computer located at a remote geographical location relative to HMD 102. In a remote host computer 106 embodiment, host computer 106 may be communicatively coupled to HMD 102 via a wide area network (such as the Internet). In a local host computer 106 embodiment, host computer 106 may coexist with HMD 102 in an environment (e.g., a home), whereby host computer 106 and HMD 102 may be directly communicatively coupled together, or communicatively coupled together via a local area network (LAN) through an intermediate network device.

[0039] HMD 102 and host computer 106 are communicatively coupled together and configured to work collaboratively to render a given frame by generating pixel data for presenting a corresponding image on the display panel of HMD 102. Host computer 106 and HMD 102 may be communicatively coupled together wirelessly and / or via a wired connection. For example, devices 102 / 106 may exchange data using Wi-Fi, Bluetooth, radio frequency (RF), and / or any other suitable wireless protocol. Additionally or alternatively, devices 102 / 106 may include one or more physical ports for wired connections that facilitate data transfer between these devices.

[0040] Although this document presents HMD 102 as an example of a "display system" capable of implementing the disclosed motion smoothing techniques, it should be understood that other types and / or specific implementations of "display systems" can implement the motion smoothing techniques described herein. Therefore, it should be understood that any reference to "HMD" herein can be replaced by the terms "display" or "display system," even though it should be understood that HMD is merely an exemplary type of display or display system used to implement the techniques described herein. In some examples, HMD 102 may represent a VR headset for use in VR systems (such as for use with VR gaming systems). However, HMD 102 may additionally or alternatively be implemented as an AR headset for use in AR applications, or as a headset for VR and / or AR applications not related to gaming (e.g., industrial applications). In AR, user 104 sees virtual objects superimposed on the real-world environment, while in VR, user 104 typically does not see the real-world environment but is fully immersed in a virtual environment, as perceived via the display panel and optics (e.g., lenses) of HMD 102. It should be understood that in some VR systems, virtual images can be combined to display a passthrough image of the user's real-world environment 104 to create an enhanced VR environment in the VR system, thereby utilizing real-world images for enhancement (e.g., overlaid on the virtual world). The examples described herein primarily relate to a VR-based HMD 102, but it should be understood that the HMD 102 is not limited to implementations in VR applications.

[0041] Furthermore, the HMD 102 may include a single display panel or multiple display panels, such as the left and right display panels in a stereoscopic display panel pair. One or more display panels of the HMD 102 may be used to present a series of image frames (referred to herein as “frames”) that can be viewed by a user 104 wearing the HMD 102. It should be understood that the HMD 102 may include any number of display panels (e.g., more than two display panels, a pair of display panels, or a single display panel). Thus, the term “display panel” as used herein in the singular may refer to one of the display panels in a pair of display panels of a dual-panel HMD 102, or it may refer to a single display panel of an HMD 102 having any number of display panels (e.g., a single-panel HMD 102 or a multi-panel HMD 102). In a dual-panel HMD 102, a stereoscopic frame buffer may render, for example, 2160 × 1200 pixels on both display panels of the HMD 102 (e.g., 1080 × 1200 pixels on each display panel).

[0042] Additionally, the display panel of the HMD 102 can utilize any suitable type of display technology, such as an emissive display that uses light-emitting elements (e.g., light-emitting diodes (LEDs)) to present light emitted during frames on the display panel. As an example, the display panel of the HMD 102 may include a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an inorganic light-emitting diode (ILED) display, or any other suitable type of display technology for HMD applications.

[0043] The HMD 102's display panel can operate at any suitable refresh rate (such as a 90 Hz refresh rate), which can be a fixed refresh rate or a variable refresh rate that changes dynamically within a refresh rate range. A display's "refresh rate" is the number of times the display redraws the screen per second. If a fixed refresh rate is used, the number of frames displayed per second can be limited by the display's refresh rate. Therefore, a series of frames can be processed (e.g., rendered) and displayed as images on the display, such that individual frames in the series are displayed with each screen refresh. That is, in order to present a series of images on the HMD 102, the HMD 102's display panel can transition frame-by-frame within the series of frames at the display's refresh rate, thereby illuminating these pixels with each screen refresh. In some implementations, the frame rate may be suppressed and / or the application may not be able to reach the target frame rate, and / or network congestion may cause delays in data transmission. In these scenarios, reprojected frames (sometimes referred to herein as "phantom frames") can be inserted between the frames rendered by the application (sometimes referred to herein as "real frames").

[0044] Typically, the application running on host computer 106 can be a graphics-based application (e.g., a video game). This application is configured to output a series of frames that can be used to render an image on the display panel of HMD 102. For example, the application can generate pixel data for this series of frames, and this pixel data can be used to render the corresponding image on the display panel of HMD 102. In some implementations, off-screen rendering is used so that frames can be rendered to a target before being rendered on the display panel of HMD 100. Therefore, as used herein, "rendering" can include rendering to a target other than the display itself before rendering on the display itself, and / or rendering on the display (e.g., without off-screen rendering to a different target, or after off-screen rendering to a different target).

[0045] Referring now to process 100, at 112, the logic (e.g., software, hardware, and / or firmware, etc.) of host computer 106 may obtain (or receive) pixel data for the first frame of a series of frames from an application executing on host computer 106. This pixel data may include pixel values ​​for individual pixels in a pixel array for the display panel of HMD 102. In some embodiments, the pixel data obtained from the executing application may include a two-dimensional array of per-pixel values ​​(e.g., color values). In some embodiments, the pixel data may also include additional data or metadata, such as depth values, brightness values, etc. In some embodiments, the pixel data may include data for each pixel represented by a single set of color values ​​and alpha values ​​(e.g., a red channel corresponding to a color value, a green channel corresponding to a color value, a blue channel corresponding to a color value, and one or more alpha channels corresponding to one or more values).

[0046] In some implementations, host computer 106 may first receive head tracking data from HMD 102, which can be used to determine the predicted pose of HMD 102 for generating pixel data for a frame. That is, host computer 106 may use the head tracking data received from HMD 102 to generate pose data indicating the predicted pose of HMD 102 at the time when the light-emitting elements of HMD 102's display panel will illuminate a frame to be rendered by the application. For example, the head tracking system of HMD 102 may be configured to track up to six degrees of freedom of HMD 102 (e.g., three-dimensional (3D) position, roll, pitch, and yaw), which may be sent as head tracking data to host computer 106 to determine the predicted pose of HMD 102 (e.g., considering predicted head movements that would lead to the future pose of HMD 102). Therefore, the pose data indicating the predicted pose can be provided as input to the application for rendering a frame, and the application can output pixel data based on the pose data. For example, an application can invoke a function to receive pose data, and the requested pose data (predicted as the target illumination time of a frame) can be provided to the application so that the application can render a frame based on the pose data, which corresponds to the virtual camera pose used to render the scene.

[0047] In some implementations, the application may be further instructed to generate depth data (e.g., Z-buffer data) and / or additional pixel data (sometimes referred to herein as “out-of-bounds pixel data” or “additional pixel data”), and in response, at box 112, in addition to the aforementioned pixel data, the logic of host computer 106 may obtain depth data and / or additional pixel data associated with the frame from the application. Depth data (such as depth data from a depth buffer (or Z-buffer) output by the application) may indicate occluded objects in the scene. Therefore, depth data can be used to adjust for parallax of objects in the scene (e.g., a ship far away in world space may not move with the same head movement as a close-up object moving with its head). Knowing the depth of the pixels corresponding to virtual objects in the scene helps in understanding how to adjust for such parallax during reprojection on HMD 102. Knowing depth information also allows the scene to be distorted not only based on the rotation of HMD 102 but also on the translation of HMD 102 in space. In the absence of any depth data, the system may assume an average depth of approximately 2 meters. However, when performing reprojection on the HMD 102, any suitable depth resolution that provides a better estimate of the average depth in the scene may be beneficial, and the higher the resolution of the depth data that can be included in the available bandwidth, the better the reprojection adjustments that can be made on the HMD 102. That is, there may be benefits to transmitting lower resolution depth data, such as lower latency. Including depth data may be beneficial as long as the resolution of the depth data is suitable for improving the reprojection adjustments on the HMD 102.

[0048] The additional pixel data output by the application can include extra pixel values ​​outside the boundaries of the pixel array of the HMD 102's display panel. For example, if the HMD 102's display panel has a 2160×1200 pixel array, the pixel data obtained at frame 112 can correspond to the pixel values ​​within that 2160×1200 pixel array, while the additional pixel data can correspond to pixels outside the boundaries of that 2160×1200 array. Therefore, the pixel data and the additional pixel data can constitute a larger number of pixel values, such as a larger array of 2400×1400 pixels. The ability to render in a view larger than the target allows panel masking to be applied around the perimeter of the display area without any substantial reduction in the field of view (FOV).

[0049] At 114, the host computer 106 logically compresses the pixel data obtained from the application at box 112. An exemplary reason for compressing the pixel data at box 114 is due to bandwidth limitations of the data connection between the host computer 106 and the HMD 102. For example, there may not typically be sufficient bandwidth to send uncompressed pixel data (e.g., files) within the amount of time allocated for displaying frames at the desired frame rate (e.g., 90Hz or higher). Although compression introduces some amount of latency (compared to uncompressed pixel data) in the end-to-end graphics pipeline, this additional time for compressing the pixel data can likely be taken into account in pose prediction, and adjustments can be made on the HMD 102 to account for the latency introduced by compression.

[0050] In some implementations, the compression performed at box 114 involves changing the format of the pixel data. For example, the compression at box 114 may utilize video compression standards such as High Efficiency Video Coding (HEVC) and / or extensions of HEVC, which are sometimes referred to as H.265 and / or MPEG-H Part 2. HEVC is an example of a video compression standard that can be utilized at box 114, but it should be understood that other suitable video compression standards may be used to compress the pixel data at box 114. HEVC uses motion estimation to compress the pixel data with the goal of transmitting less data than the uncompressed pixel data in a system with bandwidth limitations, but where the compressed data is sufficiently close to the original uncompressed data at the receiving end (e.g., at HMD 102). As part of the motion estimation utilized in the compression operation at box 114, a motion vector array may be generated, as indicated in sub-box 116. Typically, a "motion vector" is a two-dimensional (2D) arrow with X and Y components for direction, and magnitude (typically represented by the length of the 2D arrow). The magnitude of the motion vector can be specified using any suitable unit of measurement, such as the number of pixels in the X and Y component directions. In some examples, the video encoder (e.g., a video encoding chip) of the host computer 106's GPU can generate a motion vector array at sub-box 116 based on first pixel data obtained at box 112 and pixel data based on one or more other frames (e.g., previously rendered frames). To generate the motion vector array at sub-box 116, the video encoder can compare each pixel value (e.g., luminance value) between the pixel data of each frame it is responsible for analyzing according to a compression algorithm. Additionally or alternatively, the video encoder can compare macroblocks (e.g., 16-pixel blocks (i.e., 4×4 pixel macroblocks), 64-pixel blocks (i.e., 8×8 pixel macroblocks), 4096-pixel blocks (i.e., 64×64 pixel macroblocks, etc.) between the pixel data of each frame it is responsible for analyzing according to a compression algorithm. As part of the compression operation, the GPU of the host computer 106 can compare portions of the pixel data between a pair of frames at any suitable resolution.

[0051] In many video compression algorithms, including HEVC, different types of frames may exist, such as I-frames, P-frames, and B-frames. I-frames can exist independently because they contain enough data to reconstruct the frame during decompression. Therefore, compared to P-frames and B-frames, I-frames contain a relatively large amount of data and are the most difficult to compress. P-frames use data from previous frames (e.g., reference frames) to compress, then decompress, and reconstruct the original pixel data. P-frames are relatively small compared to I-frames because they encode differences from earlier frames, such as in the form of motion vectors. The "B" in B-frame stands for bidirectional, meaning that B-frames use both previous and future frames to compress, then decompress, and reconstruct the original pixel data. Because I-frames are relatively large, instances of transmitting I-frames may be reduced or minimized in this disclosure. However, transmitting I-frames may be beneficial in certain situations, such as if the wireless connection between host computer 106 and HMD 102 is temporarily lost and subsequently re-established. In any case, a typical sequence of frames may begin with an I-frame followed by a series of P-frames. Compression of the P-frame at box 114 may involve encoding the differences from the reference frame based on the positions that the blocks of the image have moved to in the new frame relative to their positions in the reference frame (e.g., I-frame and / or preceding P-frames, etc.). These differences may be represented by motion vectors generated at sub-box 116.

[0052] In addition, at point 114, if referenced Figure 6 As described in more detail in process 600, the logic of host computer 106 may align (or otherwise modify) these frames before providing the pixel data of the frames used for compression as input to the GPU of host computer 106. This is done because of the movement of HMD 102 between frames, which may cause objects (stationary and moving objects) to move between positions during the rendering of multiple consecutive frames. By aligning one frame with another, or vice versa, or by adjusting two frames to align them with each other, the pixel data representing a particular static object in the scene can be moved to approximately the same position between the two frames, so that the GPU's video encoder does not mistake the pixel data corresponding to a static object for a moving object. The alignment performed at box 114 may include a rotational reprojection modification of one or both of these frames (e.g., the pixel data of the frame) to reduce the increment (or difference) in the motion vector due to head movement. As described herein, this reduction in the increment (or difference) in the motion vector can provide the additional benefit of reduced bandwidth when transferring the motion vector from host computer 106 to HMD 102. Therefore, when generating motion vectors at sub-box 116, modified pixel data can be provided as input to the GPU for compression at box 114.

[0053] At 118, host computer 106 may send data 110 to HMD 102, which may include compressed pixel data 110(1). As described above with reference to P-frames, most of the time the compressed pixel data 110(1) may be sent in the form of an array of motion vectors 110(2) generated at subframe 116. That is, for a series of P-frames, the array of motion vectors 110(2) may represent the compressed pixel data. However, when sending I-frames to HMD 102, the compressed pixel data 110(1) may not include motion vectors. The data 110 sent at frame 118 may also include pose data 110(3) used by the application to generate the pixel data 110(1) of the frame. In some implementations, the pose data 110 (3) transmitted at box 118 may omit the data 110 transmitted at box 118, and / or the data 110 may include additional data such as depth data 110 (4), additional pixel data (e.g., outside the boundaries of the display panel of HMD 102), parallax occlusion data, and / or cube map data (e.g., for rapid, large-scale head movements, giving HMD 102 options other than displaying dark pixels that do not have any data). The data 110 may be transmitted to HMD 102 at box 118 in various ways, depending on the implementation, such as wirelessly, via a wired connection, via a wide area network, etc. In some implementations, some or all of the data 110 may be transmitted in-band or out-of-band along with compressed pixel data 110 (1) (such as motion vector 110 (2), as part of an encoded data stream). For example, pose data 110(3), depth data 110(4) and / or additional data may be transmitted out of band from compressed pixel data 110(1) (such as motion vector 110(2)).

[0054] At 120, HMD 102 may receive data 110 from host computer 106, including but not limited to compressed pixel data 110(1) (such as motion vector 110(2) (which represents compressed pixel data when received)), pose data 110(3) and / or depth data 110(4), etc.

[0055] At 122, the logic of HMD 102 (e.g., software, hardware, and / or firmware, etc.) can decompress the compressed pixel data 110(1), such as by using motion vector 110(2) and a reference frame (e.g., pixel data 110(1) of the previous frame) to obtain the pixel data of the frame output by the application, or at least an approximation of the original pixel data. The decompression of the compressed pixel data at box 122 can be performed using a decompression or decoding algorithm, such as the HEVC algorithm. Furthermore, as shown in subbox 124, for example, at least where motion vector 110(2) represents compressed pixel data 110(1) (e.g., for a P-frame), the logic of HMD 102 can extract motion vector 110(2) from a data stream (e.g., from an encoded data stream (e.g., an HEVC stream)). The decompressed pixel data 110(1), motion vector 110(2), and possible additional data (e.g., pose data 110(3), depth data 110(4), etc.) may be cached in the memory of HMD 102, so that this data can be accessed later, such as for rendering reprojected frames, and / or decompressed in the future. It should be understood that frames 120-124 may be iterated as frames are continuously received from host computer 106. It should be understood that data 110 may be held in the memory of HMD 102 (e.g., cached) for a period of time, and then data 110 may be discarded to make room for storing future data 110.

[0056] At 126, the logic of HMD 102 can retrieve the most recently decompressed data 110 (such as decompressed pixel data 110(1)) for rendering an image on the display panel of HMD 102. For example, when data 110 is received at frame 120 and decompressed at frame 122, the retrieval operation at frame 126 can determine that the cached pixel data 110(1) stored in the memory of HMD 102 represents the most recently decompressed pixel data available for use by HMD 10, and the logic of HMD 102 can retrieve the most recently decompressed data from memory, such as pixel data 110(1) obtained from decompressing the most recently received motion vector 110(2). In an exemplary example, if HMD 102 operates at a refresh rate of 90Hz, a frame will be displayed approximately every 11.11 milliseconds. In this case, if HMD 102 receives and decompresses new data within the last 11.11 milliseconds since the last frame was displayed, the new, decompressed pixel data 110(1) can be used to render the next frame. On the other hand, if HMD 102 does not receive and decompress new data within the last 11.11 milliseconds since the last frame was displayed, the pixel data 110(1) from the most recently rendered frame can be used to render the next frame.

[0057] At 128, the logic of HMD 102 may modify the pixel data of the frame at least in part based on the predicted pose of HMD 102 to obtain the modified pixel data of the frame. For example, the logic of HMD 102 may apply adjustments to the pixel data based on a comparison between the original predicted pose of HMD 102 used by the application to generate the pixel data retrieved at box 126 and an updated pose predicted by the logic of HMD 102 at a lighting time closer to the frame. This comparison may reveal the increment (or difference) between the original pose prediction when the application renders the frame and the updated pose prediction at HMD 102 before the frame is rendered on the display panel, and the adjustments applied at box 118 may include rotation calculations to compensate for this increment (e.g., by moving and / or rotating the pixel data in a certain way, depending on the increment between the two pose determinations).

[0058] Furthermore, at 128, the logic of HMD 102 can also modify the pixel data of the frame at least in part based on the array of motion vectors 110(2) extracted at box 124 and retrieved at box 126. As a result of the modification at box 128, modified pixel data of the reprojected frame is obtained. Although this disclosure distinguishes between “actual” frames and reprojected (or “phantom” frames), this distinction does not imply that actual frames are not often adjusted / modified at box 128 before being presented on HMD 102. That is, since modification operations can be performed on both “actual” frames and “phantom” frames at box 128, the frame presented on the HMD side can be considered synthetic (i.e., different from the original frame output by the application executed on host computer 106). In this sense, each frame modified at box 128 can be considered a “reprojected” frame as used herein. Whether the pixel data is modified, and / or the extent of modification at box 128, depends on the operational performance of the system. For example, if everything goes as expected in a real-time rendering scenario, where each frame output by the application is received in time at HMD 102, the increment between the original pose prediction and the updated pose prediction can be close to zero (if not zero), and in this case, the modification applied at box 128 (if any) may have little effect on the final output. However, in scenarios where the application fails to reach the frame rate, or where frames arrive late or are lost in transit due to network congestion, the modification applied to pixel data 110(1) at box 128 is significant for compensating for the larger increment between the original pose prediction and the updated pose prediction. As will be described in more detail below, the pixel data of the reprojected frame modified based on motion vector 110(2) can be utilized with a rendering mesh. For example, a rendering mesh can be generated for the reprojected frame. The rendering mesh may include a checkerboard-shaped mesh with multiple vertices, and the vertices of the rendering mesh may be moved to different positions within the rendering mesh as moved vertices (e.g., by moving the vertex (i) in the direction of the non-zero motion vector 110(2) and (ii) by an amount corresponding to the magnitude of the non-zero motion vector 110(2)). Therefore, modifying pixel data at box 128 may be done based on the moved vertices of the rendering mesh, such as by moving the pixel values ​​of the pixel data retrieved at box 126 according to the moved vertices, to obtain modified pixel data for the reprojected frame. For example, based on the moved vertices within the rendering mesh, the pixel values ​​may be moved 4 pixels to the left and 4 pixels up to a new position within the modified pixel data. In some embodiments, the motion vector 110(2) may be modified before the modified motion vector 110(2) is applied to the pixel data.For example, as described in more detail below, a filter (e.g., an N×N scalar median filter, an M×M average-maximum blur filter, etc.) can be applied to the motion vector 110(2) array to obtain a modified motion vector array, and the modified (e.g., filtered) motion vector array can be used at box 128 to modify pixel data.

[0059] Furthermore, at 128, the logic of HMD 102 can also modify the pixel data of the frame, at least in part, based on the depth data 110(4) retrieved at box 126. The depth data 110(4) can be used at box 128 to adjust for the parallax of objects in the scene (e.g., a ship far away in world space may not move with the same head movement as a close-up object). Knowing the depth of the pixels corresponding to virtual objects in the scene helps in understanding how to adjust for this parallax during reprojection on HMD 102. Knowing the depth information also allows the scene to be distorted not only based on the rotation of HMD 102 but also based on the translation of HMD 102 in space. If the depth data 110(4) is not received from host computer 106, the logic of HMD 102 can assume an average depth of approximately 2 meters.

[0060] At 130, an image can be rendered on the display panel of HMD 102 at least in part based on modified pixel data. For example, the logic of HMD 102 can output modified pixel data to a frame buffer, and the image can be rendered on the display panel of HMD 102 based on the modified pixel data output to the frame buffer. This may involve scanning the modified pixel data output to the display panel of HMD 102 and illuminating the light-emitting elements of the display panel to illuminate the pixels on the display panel. For HMD 102 having a pair of display panels, the modified pixel data may correspond to frames representing a pair of images to be displayed on the pair of display panels, and may be output to a stereo frame buffer and scanned output accordingly. By modifying the pixel data according to motion vector 110(2), the resulting image corresponding to the reprojected frame is "motion smoothed".

[0061] Figure 2 The illustration shows two exemplary timelines 200(1) and 200(2) according to the embodiments disclosed herein, which show the corresponding rendering workloads of host computer 106 and HMD 102. Figure 2The example depicts three exemplary frames relative to a first timeline 200(1) associated with host computer 106: frame "F", frame "F+1", and frame "F+2". This first timeline 200(1) illustrates how an application executing on host computer 106 can sequentially render frames using the GPU of host computer 106. Here, the application renders frame F sequentially from left to right on the first timeline 200(1), then frame F+1, and then frame F+2. The ellipses on the first timeline 200(1) indicate that the rendering can continue to render any number of frames as the application continues to execute. The first timeline 200(1) also implies that the application is targeting a target frame rate (e.g., a frame rate of 90 Hz, where these vertical lines would be separated by approximately 11.11 milliseconds) via vertical lines perpendicular to the orientation of the horizontal timeline 200(1). Figure 2 In the example, the application running on host computer 106 happens to reach the target frame rate on frames F and F+1, but fails to reach the target frame rate for frame F+2. For example, this could be a situation where the scene in frame F+2 includes a large number of moving objects or complex textures, and due to this complexity and / or for other reasons, the application takes longer than the allocated time to render frame F+2. Host computer 106 may receive head tracking data 208 about the movement of HMD 102 to determine the predicted pose of HMD 102 for each frame on rendering timeline 200(1).

[0062] Figure 2The second timeline 200(2) associated with HMD 102 shows the rendering workloads 202(a), 202(b), and 202(c) of HMD 102 for each frame. The respective rendering workloads 202 of HMD 102 for a given frame can represent adjustments applied to pixel data 110(1) before the final image is rendered on the display panel of HMD 102. Such adjustments may include, but are not limited to, adjustments for geometric distortion, chromatic aberration, head movement, etc., which are applied to pixel data 110(1) before the final image is rendered on HMD 102. For frames F and F+1, these adjustments are applied to pixel data generated by an application running on host computer 106, and at least some of these adjustments can utilize the pose data 110(3) received from host computer 106 by taking into account the increment between the original predicted pose of HMD 102 and the updated pose prediction of HMD 102. Therefore, frames F and F+1 on the second timeline 200(2) represent modified versions of the “actual” frames, as they are modified versions of the pixel data 110(1) output from the application in real time in a real-time rendering scenario. In contrast, to render frame F+2, HMD 102 can use previously received pixel data 110(1) of the preceding frame (e.g., pixel data 110(1) of frame F+1) to generate a reprojected (or “phantom”) frame based on the pose prediction and updated pose prediction of the preceding frame made by HMD 102. Furthermore, as described herein, rendering workload 202(c) may include modifying the pixel data from frame F+1 to “motion-smoothed” frame F+2 based on the motion vector 110(2) received along with the data 110 of frame F+1. In any case, the result of rendering workload 202 is the generation of modified pixel data that can be output to a frame buffer (e.g., a stereo frame buffer). This distinction between the “actual” frame and the “phantom” frame in this article does not imply that the actual frame is not adjusted on HMD 102, and in this sense, all frames generated on the HMD side are effectively synthesized (i.e., different from the original frames output by the application running on host computer 106).

[0063] Figure 2 The second timeline 200(2) also shows the scan output times 204(a), 204(b), and 204(c) for each frame, and the illumination times 206(a), 206(b), and 206(c) for each frame. During the scan output time 204 of a given frame, a subset of pixel values ​​(of modified pixel data) is scanned and output to the display panel of HMD 102 via a display port (e.g., a High Definition Multimedia Interface (HDMI)), and during the illumination time 206 of that given frame, the light-emitting elements of the display panel of HMD 102 are illuminated so that the pixels of the display panel are illuminated. Figure 2 An example of a global flicker type display driving scheme is shown, which can be used with an LCD panel to simultaneously emit light from the light-emitting elements of the display panel at the refresh rate of the HMD 102. In the illustrative example, if the HMD 102 operates at a refresh rate of 90Hz, the illumination time 206 for each frame can be separated by approximately 11.11 milliseconds.

[0064] Figure 3 This is an illustration of an exemplary motion smoothing technique that takes into account moving or animated objects in a scene when reprojection is used to render frames on a display such as HMD 102. Figure 3 The example depicts three exemplary frames 300(1), 300(2), and 300(3) that will be rendered sequentially as a series of frames 300. Figure 3 In the example, object 302 is shown moving through the scene in a leftward direction (i.e., from right to left) during frames 300(1)-300(3). The dashed outline of object 302 in frames 300(2) and 300(3) indicates the position of object 302 in the previously rendered frame 300. Here, frame 300(1) is rendered first, then frame 300(2), and then frame 300(3).

[0065] At least some of the frames 300 in this series can be “actual” frames because they are output from an application (such as a video game application, or any other type of graphics-based application that has enough time to render the corresponding image on the HMD 102 based on the pixel data of the frames 300). The application can be executed in a graphics pipeline that outputs pixel data 110(1) to a frame buffer to render the individual frames 300.

[0066] During operation, the head tracking module of HMD 102 generates data about the position and orientation (pose) of HMD 102, which is provided to the host computer 106 executing the application to inform the application about how to render the next frame 300 in the series of frames 300 based on the head pose of user 104. This allows the application to output pixel data 110(1) to render an image on HMD 102, making user 104 believe that he / she is viewing a virtual environment that includes objects (both static and moving objects, such as moving object 302). If the application reaches the frame rate, both static and moving objects are perceived as moving within the scene in the expected manner along with the head movement of user 104. The motion smoothing techniques described herein are a way to compensate for the application's failure to reach the frame rate, enabling a similar visual perception of moving objects.

[0067] exist Figure 3In the example, the first frame 300(1) may represent the first “actual” frame received from the application, the second frame 300(2) may represent the second “actual” frame received from the application and rendered after the first frame 300(1), and the third frame 300(3) may represent the reprojected frame generated from the pixel data 110(1)(2) associated with the second frame 300(2). Therefore, Figure 3 The “third frame 300(3)” in this paper is sometimes referred to as “reprojected frame 300(3)”. Figure 3 In the example, in order to compress the second frame 300(2), the logic of the host computer 106 can provide the first pixel data 110(1)(1) associated with the first frame 300(1) and the second pixel data 110(1)(2) associated with the second frame 300(2) as inputs to the graphics processing unit (GPU) 304 of the host computer 106.

[0068] The video encoder (e.g., a video encoding chip) of GPU 304 can generate an array of motion vectors 110(2)(2) based on the first pixel data 110(1)(1) and the second pixel data 110(1)(2) input to GPU 304. To generate the array of motion vectors 110(2)(2), the video encoder of GPU 304 can compare each pixel value (e.g., a luminance value) between the pixel data 110(1) of each frame 300 provided as input. Additionally or alternatively, the video encoder of GPU 304 can compare macroblocks (e.g., blocks of 16 pixels (i.e., 4×4 pixel macroblocks), blocks of 64 pixels (i.e., 8×8 pixel macroblocks)) between the pixel data 110(1) of each frame 300 provided as input. In this way, GPU 304 can compare portions of the pixel data 110(1) between a pair of frames 300 at any suitable resolution. In some implementations, the input pixel data 110(1)(1) and 110(1)(2) are downsampled to a lower resolution so that the downsampled frame is input to GPU 304. In some implementations, the motion vector 110(2)(2) is generated according to any suitable compression algorithm, such as HEVC.

[0069] The array of motion vectors 110(2)(2) output from GPU 304 is sometimes referred to herein as a “motion vector field”. This motion vector field 110(2)(2) can also be output at any suitable resolution and / or downsampled / upsampled to any suitable resolution. For example, the array of motion vectors 110(2)(2) may include a single motion vector for each pixel, a single motion vector for each group of pixels (e.g., a motion vector for a 4×4 macroblock, an 8×8 macroblock, a pixel patch of arbitrary shape, etc.), or even a single motion vector for all pixels of a given frame 300.

[0070] The video encoder based on GPU 304 compares the input pixel data 110(1) and if a portion of the second frame 300(2) is similar to a portion of the first frame 300(1) (e.g., within the latter's threshold brightness value), and if the similar portions in each frame 300 are offset by a certain distance (e.g., the number of pixels in the X and Y component directions), this can be represented by motion vectors included in the array of motion vectors 110(2)(2). Consider the following example: the pixel values ​​corresponding to object 302 in frames 300(1) and 300(2) are determined by the video encoder of GPU 304 to be similar (e.g., based on the matching portions of the pixel values ​​in pixel data 110(1) that satisfy some similarity metric). The motion vector of object 302 may have a direction pointing in the same direction as the direction of movement of object 302, or the motion vector may point in a direction opposite to the direction of movement of object 302. In other words, the motion vector can point in a direction that again indicates the position of object 302 in the previous frame 300(1), which is offset from the position of object 302 in the subsequent frame 300(2). Therefore, the motion vectors in array 110(2)(2) provide the offset from the coordinates in the second frame 300(2) to the coordinates in the first frame 300(1). This offset describes the transformation of the image of a moving or animated object (such as object 302) from the image in the first frame 300(1) to the image in the second frame 300(2).

[0071] As mentioned above, Figure 3 The third frame 300(3) in the image can represent a reprojected frame, meaning that the pixel data 110(1)(3) of the third frame 300(3) can be derived from the pixel data 110(1)(2) associated with the previously rendered actual frame (in this case, the second frame 300(2)). In other words, the pixel data 110(1)(3) of the reprojected frame 300(3) is not received in real time from the host computer 106, but is generated from the pixel data 110(1) of the frame generated by the application and is used to “fill” the gaps of lost frames when the application does not reach the frame rate and / or when network congestion occurs. In this case, Figure 3In the example, the pixel data 110(1)(3) of the reprojected frame 300(3) is generated from the pixel data 110(1)(2) associated with the second frame 300(2), since the second frame 300(2) is the most recently rendered frame 300 before the reprojected frame 300(3). In some implementations, the second pixel data 110(1)(2) associated with the second frame 300(2) may be calculated and modified using rotation and reprojection transformations to generate the third pixel data 110(1)(3) of the reprojected frame 300(3), which effectively rotates, translates and / or otherwise moves the scene rendered in the second frame 300(2), such that rotation of the HMD 102 since the second frame 300(2) was rendered is taken into account. For example, user 104 may start rotating his / her head from the time the second frame 300(2) is rendered, taking into account the head movement when generating the third pixel data 110(1)(3) of the reprojected frame 300(3), so that the scene is presented according to the head movement.

[0072] Figure 3 The motion smoothing technique then modifies the third pixel data 110(1)(3) based at least in part on the array of motion vectors 110(2)(2) associated with the second frame 300(2) received from the host computer 106 to obtain the modified third pixel data 110(1)(3)' of the reprojected frame 300(3). In some embodiments, this modification of the third pixel data 110(1)(3) to the modified third pixel data 110(1)(3)' includes moving the pixel values ​​of the third pixel data 110(1)(3) to different positions based on non-zero motion vectors 110(2)(2) corresponding to a particular pixel or group of pixels. This movement may be in one direction and by a certain amount (e.g., moving multiple pixels in the horizontal (+ / -) and vertical (+ / -) directions). The reprojected frame 300(3) is then rendered on a display (e.g., on the display panel of HMD 102) based at least in part on the modified third pixel data 110(1)(3)'. Therefore, the reprojected frame 300(3), which has been modified based on motion vector 110(2)(2) (which has been generated from pixel data 110 of previously received (and / or previously rendered) actual frames 300(1) and 300(2) as a result of compressing the pixel data 110(1)(2) of the second frame 300(2) (e.g., a byproduct)), is "motion smoothed" to render object 302 at the expected position.

[0073] Figure 4This is an illustration of an exemplary rendering mesh 400 and how the rendering mesh 400 can be used in motion smoothing of a reprojected frame. For example, the rendering mesh 400 can be modified by the HMD 102 based on an array of motion vectors 110(2) output by the GPU 304 of the host computer 106. Figure 3 The third pixel data 110(1)(3) of the reprojected frame 300(3). In this example, the logic of HMD 102 can generate a rendering mesh 400, and the vertices 402 of the rendering mesh 400 can move according to the motion vector array 110(2). For example, the vertices 402 can move in the direction of the non-zero motion vector 404 and by an amount corresponding to the magnitude of the non-zero motion vector 404. For example, the third pixel data 110(1)(3) of the reprojected frame 300(3). Figure 3 The leftward-moving object 302 shown is an example. Motion vector 404 can be applied in the environment of the rendering mesh 400 to move vertex 402 a specific number of pixels (corresponding to the magnitude of motion vector 404) in the left (or negative X) direction.

[0074] The rendered mesh 400 is shown as a checkerboard grid with multiple vertices 402(1), 402(2), ..., 402(N) (collectively referred to as 402). The checkerboard pattern of the rendered mesh 400 can be any suitable geometric pattern. Although Figure 4 An exemplary rendering mesh 400 is shown as a repeating pattern of triangles 406, but any suitable geometry can be used for the rendering mesh 400, including but not limited to squares (sometimes called "quadrilaterals"), hexagons (e.g., for honeycomb patterns), etc. In this example, a diagonal line from the bottom left corner of the square (or quadrilateral) to the top right corner of the square (or quadrilateral) is used to create the rendering mesh 400 of repeating triangles 406 with a specific orientation. This can be replaced by using a diagonal line from the top left corner of the square (or quadrilateral) to the bottom right corner of the square (or quadrilateral). Figure 4 The orientations shown are used to divide a square (or quadrilateral) into triangles 406, thereby creating different orientations for the rendering mesh 400. In some embodiments, a mixture of these different orientations can also be used for a single rendering mesh 400, such as dividing every other square by using a diagonal from the lower left corner to the upper right corner of the square (or quadrilateral), and dividing between squares by using a diagonal from the upper left corner to the lower right corner of the square (or quadrilateral). In some embodiments, the logic of HMD 102 can be configured to dynamically determine which of these multiple orientations is used to generate the rendering mesh 400 based on the motion vector field 110(2) received from the host computer 106. This may be done to select the optimal orientation for the geometry (e.g., triangle 406) in the rendering mesh 400, thereby producing a motion-smoothed image that appears smoothest.

[0075] The rendering mesh 400 can also be generated at any suitable resolution. For example, the highest resolution rendering mesh 400 might be two adjacent triangles 406 per pixel, where each square (or quadrilateral) maps to a single pixel. A lower resolution might be two adjacent triangles 406 per group of pixels (such as a group of 16 pixels). Alternatively, pixels can be mapped to vertices 402 of the rendering mesh 400 at any suitable resolution. For example, each vertex 402 might be associated with a single pixel at the highest resolution, or each vertex 402 might be associated with a group of pixels (such as a group of 16 pixels) at a lower resolution. In some embodiments, the resolution of the rendering mesh 400 is the same as the resolution of the motion vector 110(2) array, such that a single motion vector 404 in the motion vector 110(2) array maps to a vertex 402 or a square (or quadrilateral) (e.g., two adjacent triangles 406). The resolution matching between the rendering mesh 400 and the motion vector 110(2) array can be achieved in various ways, such as by requesting a motion vector 110(2) array of a specific resolution matching the resolution of the rendering mesh 400 from the GPU 304, by downsampling or upsampling the motion vector 110(2) array to match the resolution of the rendering mesh 400, or by generating the rendering mesh 400 at a resolution matching the resolution of the motion vector 110(2) array received from the host computer 106.

[0076] Figure 4 An example is shown where four non-zero motion vectors 404(1), 404(2), 404(3), and 404(4) correspond to four vertices 402 of the rendered mesh 400. These four motion vectors 404(1)-(4) may correspond to GPU 304 based on... Figure 3 The motion vectors detected by the moving object 302 depicted in the figure. Therefore, although the exemplary motion vectors 404(1)-(4) may point to the left direction corresponding to the directional motion of the object 302, as described above, this directionality may be opposite to the directionality of the directional motion of the object 302 (e.g., to the right). The directionality of the motion vector 404 may be taken into account in the motion smoothing algorithm to modify the pixel data 110(1) in the desired direction. Consider the basic example: where the field of motion vector 110(2) includes four exemplary motion vectors 404(1)-(4), and all remaining motion vectors in the array of motion vectors 110(2) are zero vectors. In this example, the pixel data 110(1)(3) of the reprojected frame 300(3) can be modified based on the non-zero motion vectors 404(1)-(4), specifically by moving the vertex 402 corresponding to the non-zero motion vectors 404(1)-(4) to different positions within the rendering mesh 400 as moved vertices 408(1), 408(2), 408(3), and 408(4). Figure 4 (As shown at the bottom). Figure 4 The bottom shows the rendered mesh 400 after motion smoothing has been applied, where the moved vertices 408(1)-(4) are located in different positions within the rendered mesh 400 compared to the position of vertex 402 before motion smoothing. When motion vectors 404(1)-(4) are applied in this manner, the moved vertices 408(1)-(4) cause distortion in one or more portions of the rendered mesh 400, such as by stretching or distorting specific geometries (e.g., triangle 406) within the rendered mesh 400. Figure 4 In the example, some triangles in triangle 406 are stretched to create stretched triangle 410 due to smooth motion, as in Figure 4 As shown at the bottom. Pixel values ​​corresponding to the moved vertices 408(1)-(4) within the rendering mesh 200 are rendered at the locations corresponding to the moved vertices 408(1)-(4). Pixel positions between the moved vertices 206(1)-(4) and the unmoved vertices 402 can be blended (e.g., by applying gradients, such as by interpolating the pixel values ​​between the moved vertices 408(1)-(4) and the unmoved vertices 402). In some implementations, a depth buffer can be used to determine the final set of pixel values ​​for the frame buffer that outputs the modified pixel data 110(1)(3)′ to the reprojected frame 300(3). That is, as a result of applying motion vectors 404(1)-(4) to the rendering mesh 400, multiple pixel values ​​may exist at the locations corresponding to the moved vertices 408(1)-(4) in the image. In this case, any pixel value associated with the "nearer" (smaller) depth value can be rendered, instead of rendering another pixel value associated with the "farther" (larger) depth value at that location.

[0077] Figure 5 This is a flowchart of an exemplary process 500 according to an embodiment disclosed herein, which is used to generate a reprojected frame using motion vectors generated by a graphics processing unit (GPU) as part of a motion smoothing technique. For purposes of discussion, process 500 is described with reference to the preceding figures.

[0078] At 502, the logic of host computer 106 may provide pixel data 110(1) associated with a previously rendered frame 300 as input to GPU 304. For example, before transmitting compressed pixel data 110(1), pixel data 110(1) associated with two recently rendered frames 300 may be provided as input to GPU 304 as part of a compression algorithm for compressing pixel data 110(1). These frames 300 may be actual frames received from an application (e.g., a video game application), such as a first frame 300(1) rendered previously, and a second frame 300(2) rendered after the first frame 300(1). Therefore, the second frame 300(2) may represent the most recently rendered frame output by an application running on the host computer 106, and the first frame 300(1) and the second frame 300(2) may have been output consecutively by the application in a series of frames 300, although the pixel data 110(1) provided as input at box 502 does not need to be the pixel data 110(1) of consecutively rendered frames. For example, an intermediate frame 300 may be rendered between the first frame 300(1) and the second frame 300(2), and the pixel data 110(1) provided as input at box 502 may relate to the first frame 300(1) and the second frame 300(2).

[0079] At 504, an array of motion vectors 110(2) can be received from GPU 304. The array of motion vectors 110(2) received at box 504 may have been generated by the video encoder of GPU 304 based at least in part on first pixel data 110(1)(1) associated with the first frame 300(1) and second pixel data 110(1)(2) associated with the second frame 300(2) (e.g., based on a comparison between the first pixel data 110(1)(1) and the second pixel data 110(1)(2)). The video encoder of GPU 304 may be configured to use appropriate computer vision algorithms and / or video coding algorithms (e.g., HEVC) to find similarity between pixel values ​​(or groups of pixel values), such as by determining whether the difference between the compared pixel values ​​is less than a threshold difference. Any pixel value within such a similarity metric can be considered as matching pixel data 110(1) between the two frames 300.

[0080] At 506, the logic of HMD 102 may generate, at least in part, the third pixel data 110(1)(3) of the reprojected frame 300(3) based on the second pixel data 110(1)(2) of the second frame 300(2). In this case, the second frame 300(2) represents the frame rendered immediately preceding the reprojected frame 300(3). For example, between boxes 504 and 506, HMD 102 may have received the compressed second pixel data 110(1)(2) of the second frame 300(2), decompressed the pixel data 110(1)(2), modified the pixel data 110(1)(2) to take into account the updated pose prediction of HMD 102, and rendered the image of the second frame 300(2) on HMD 102, and at box 506, HMD 102 is preparing to render the reprojected frame 300(3) as the next frame.

[0081] At 508, the logic of HMD 102 may modify the third pixel data 110(1)(3) based at least in part on the motion vector array 110(2)(2) to obtain the modified third pixel data 110(1)(3)' of the reprojected frame 300(3). As shown in subframes 510 and 512, the modification of the pixel data 110(1)(3) of the reprojected frame 300(3) may be made using the rendering mesh 400.

[0082] Therefore, at 510, the logic of HMD 102 can generate a rendering mesh 400 of the reprojected frame 300(3). The rendering mesh 400 may include a checkerboard mesh with multiple vertices 402. In some embodiments, the resolution of the rendering mesh 400 may match the resolution of the motion vector field 110(2)(2) array, such that there is a one-to-one correspondence between the motion vector 404 and the “elements” of the rendering mesh 400 (e.g., elements such as vertices 402 of the rendering mesh 400, squares (or quadrilaterals) of the rendering mesh 400). Obtaining a matching resolution between the motion vector field 110(2)(2) and the rendering mesh 400 may include any of the techniques described herein, such as requesting the GPU 304 to output the motion vector field 110(2)(2) at a specific resolution, downsampling or upsampling the resolution of the motion vector field 110(2)(2), and / or generating the rendering mesh 400 at a resolution matching the resolution of the motion vector field 110(2)(2) output by the GPU 304.

[0083] At 512, the logic can move vertex 402 (among multiple vertices 402) of the rendering mesh 400 to a different position within the rendering mesh 400 as moved vertex 408. Vertex 402 can be moved (i) in the direction of a non-zero motion vector 404 and (ii) by an amount corresponding to the magnitude of the non-zero motion vector 404. Thus, modifying the third pixel data 110(1)(3) at box 508 can be based on the moved vertex 408 of the rendering mesh 400, such as by moving the pixel value of the third pixel data 110(1)(3) according to the moved vertex 408 to obtain the modified pixel data 110(1)(3)' of the reprojected frame 300(3). For example, based on the moved vertex 408 within the rendering mesh 400, the pixel value of the third pixel data 110(1)(3) might be moved 4 pixels to the left and 4 pixels up to a new position within the modified pixel data 110(1)(3)'.

[0084] In some implementations, multiple motion vector fields 110(2) may be received at box 504 based on different sets of previously rendered frames 300 input at box 502, and additional motion-related parameters may be determined based on the multiple motion vector fields 110(2) for use in motion smoothing of the reprojected frames at box 508. For example, in addition to the first two frames 300(1) and 300(2) that generate the array of motion vectors 110(2)(2) received at box 504, in some implementations, Figure 5 The algorithm can backtrack one or more additional frames, such as by providing the GPU 304 with the first frame 300(1) and the “zero” frame 300(0) rendered before the first frame 300(1) as input and receiving additional arrays of motion vectors 110(2) based on the different pairs of the input frames 300. The multiple arrays of motion vectors can then be compared to determine motion-related parameters, such as the acceleration of an object moving frame by frame, and these motion-related parameters can be applied at box 508, such as by modifying (e.g., increasing / decreasing) the magnitude of the final motion vector 404 applied to the third pixel data 110(1)(3) to more or less adjust the pixel values ​​according to motion smoothing.

[0085] At 514, the logic of HMD 102 may render the reprojected frame 300(3) on the display (e.g., on the display panel of HMD 102) based at least in part on the modified third pixel data 110(1)(3). The resulting reprojected frame 300(3) rendered at box 514 is “motion smoothed” by modifying the third pixel data 110(1)(3) according to the motion vector 110(2) received from GPU 304. It should be understood that GPU 304 may represent one or more GPUs 304. For example, a given frame 300 may be rendered on the stereoscopic display panel of HMD 102 using multiple GPUs 304, and the pixel data 110(1) of frame 300 input to these GPUs 304 may be divided accordingly (e.g., the left half of pixel data 110(1) may be provided as input to a first GPU 304, and the right half of pixel data 110(1) may be provided as input to a second GPU 304).

[0086] Figure 6 This is a flowchart of an exemplary process 600 according to an embodiment disclosed herein, which is used to align pixel data of a previously rendered frame before inputting pixel data to a GPU for motion vector estimation. For purposes of discussion, process 600 is described with reference to the preceding figures. Furthermore, as... Figure 5 and Figure 6 As shown in the external reference "A" in the document, process 600 can be represented in... Figure 5 The operation performed before the operation at box 504, and in some implementations, process 500 may continue the operation at boxes 504 and 514.

[0087] At 602, logic (e.g., the logic of host computer 106) can determine, based on rotation data received from HMD 102, that HMD 102 has rotated from a first direction to a second direction between frames 300 in which previously rendered pixel data 110(1) will be provided as input to GPU 304. For example, user 104 may have rotated his / her head to the right between time t1 and time t2, which may correspond to the times of rendering the first frame 300(1) and the second frame 300(2).

[0088] At 604, those frames 300 can be aligned before the pixel data 110(1) of the previously rendered frame 300 is provided as input to the GPU 304 of the host computer 106. This is because the HMD 102 moves between frames 300, which may cause objects (static and moving objects) to move between positions during the rendering of the two frames 300(1) and 300(2). By aligning one frame 300(1) with another frame 300(2), or vice versa, the pixel data 110(1) representing a specific static object in the scene can be moved to approximately the same position between the two frames 300(1) and 300(2), so that the video encoder of the GPU 304 will not mistake the pixel data 110(1) corresponding to the static object for a moving object. The alignment at frame 604 may include at least one of the following: (i) modifying the first pixel data 110(1)(1) (associated with the first frame 300(1)) to obtain modified first pixel data 610(1)(1) that aligns the scene in the first frame 300(1) with the second direction of the HMD 102 when rendering the second frame 300(2), or (ii) modifying the second pixel data 110(1)(2) (associated with the second frame 300(2)) to obtain modified second pixel data that aligns the scene in the second frame 300(2) with the first direction of the HMD 102 when rendering the first frame 300(1). Figure 6 The bottom diagram illustrates the former scenario, where the first pixel data 110(1)(1) is modified to obtain modified first pixel data 610(1)(1) that aligns the scene in the first frame 300(1) with the second direction of the HMD 102. However, it should be understood that either the first pixel data 110(1)(1) or the second pixel data 110(1)(2) may be modified for alignment purposes. Alignment at box 604 may include a rotational reprojection modification of one or both of the pixel data of the frame to reduce the increment in the motion vector 110(2) due to head movement. As described herein, this reduction in the increment in the motion vector 110(2) can provide the additional benefit of reduced bandwidth when transmitting the motion vector 110(2) from the host computer 106 to the HMD 102.

[0089] At 606, the logic of the host computer 106 can provide the GPU 304 with modified pixel data of one frame of the previously rendered frame 300 and the original pixel data 110(1) of another frame 300 as input. Figure 6The example shown depicts providing modified first pixel data 610(1)(1) and original second pixel data 110(1)(2) as input to GPU 304. As mentioned, process 600 can continue from box 606 to box 504 of process 500 (as shown in external reference "A"). Therefore, at box 504, the motion vector 110(2) array can be received from GPU 304 based on the pixel data provided as input to GPU 304 at box 606, and the following can be performed. Figure 5 The remaining operations of the motion smoothing algorithm are used to render the motion-smoothed reprojected frame 300(3).

[0090] Figure 7 This is a flowchart of an exemplary process 700 according to an embodiment disclosed herein, which excludes some pixel data from previously rendered frames and provides the remaining pixel data as input to the GPU for motion vector estimation. For discussion purposes, process 700 is described with reference to the preceding figures. Furthermore, as... Figure 5 and Figure 7 As shown in the external reference "A" in the page, process 700 can be represented in Figure 5 The operation performed before the operation at box 504, and in some implementations, process 500 may continue the operations at boxes 504-514.

[0091] At 702, logic (e.g., the logic of host computer 106) can determine, based on rotation data received from HMD 102, that HMD 102 has rotated from a first direction to a second direction between rendering previously rendered frames 300 that will be provided as input to GPU 304 (e.g., for pixel data compression). The previously rendered frames 300 can be... Figure 3 The first frame 300(1) and the second frame 300(2) are shown.

[0092] At 704, logic (e.g., the logic of host computer 106) may provide a portion of the available pixel data 110(1) as input to GPU 304. The portion of the available pixel data 110(1) provided as input to GPU 304 may include, for example, a portion 706(1) of the first pixel data 110(1)(1) and a portion 706(2) of the second pixel data 110(1)(2), each corresponding to pixels outside a subset of pixels at one or more edges of the display panel of HMD 102. For example, as Figure 7 As shown, the portion 706 of pixel data 110(1) provided as input to GPU 304 does not include the remaining portion of pixel data 110(1) at the left and right edges of the display panel (in Figure 7(Shown in black in the image). In other words, the pixel data 110(1) at the left and right edges of the display panel is not provided to the GPU 304, so the GPU 304 focuses its motion estimation work on the central portion 706 of the pixel data 110(1) between the left and right edges of the display panel. For HMD 102 rotation in the positive and / or negative vertical directions, the excluded portion of the pixel data 110(1) can be the top and bottom edges of the image. In any case, excluding the portion of the pixel data 110(1) at the image edges and providing only the remaining portion 706 of the pixel data 110(1) to the GPU 304 will effectively result in zero motion vectors 404 at one or more edges of the image, which may be useful in cases where extraneous visual artifacts appear near the image edges, which may result in outlier motion vectors 404 that do not belong to the features of the moving object 302. As mentioned, process 700 can continue from block 704 to block 504 of process 500, wherein a motion vector 110(2) array is received from GPU 304 based on the pixel data provided as input to GPU 304 at block 704, and can be performed Figure 5 The motion smoothing algorithm is used to render the motion-smoothed reprojected frame 300(3).

[0093] Figure 8 This is a flowchart of an exemplary process 800 according to an embodiment disclosed herein, which aligns pixel data of a previously rendered frame, excludes panel mask portions of the pixel data of the previously rendered frame, and provides the remaining portion of the pixel data as input to the GPU for motion vector estimation. For purposes of discussion, process 800 is described with reference to the preceding figures. Furthermore, as... Figure 5 and Figure 8 As shown in the external reference "A" in the page, process 800 can be represented in Figure 5 The operation performed before the operation at box 504, and in some implementations, process 500 may continue the operations at boxes 504-514.

[0094] At 802, logic (e.g., the logic of host computer 106) can determine, based on rotation data received from HMD 102, that HMD 102 has rotated from a first direction to a second direction between previously rendered frames 300 that will be provided as input to GPU 304. The previously rendered frames can be... Figure 3 The first frame 300(1) and the second frame 300(2) are shown.

[0095] At 804, logic (e.g., the logic of host computer 106) may modify the first pixel data 110(1)(1) associated with the first frame 300(1) to obtain modified first pixel data 808(1) that aligns the scene in the first frame 300(1) with a second direction of HMD 102. Figure 8 In the example, a portion of the first pixel data 110(1)(1) represents data corresponding to the panel mask (displayed in black) surrounding the display panel of the HMD 102 for each frame. Therefore, the panel mask portion 814(1) of the modified first pixel data 808(1) (i.e., in Figure 8 The portion shown in black in the middle represents the data in the modified first pixel data 808(1) corresponding to the panel mask. As mentioned, the application can be instructed to generate additional pixel data corresponding to the panel mask when rendering the frame. In this way, the FOV is not actually reduced when the panel mask is applied.

[0096] At 806, logic (e.g., the logic of host computer 106) can modify the second pixel data 110(1)(2) associated with the second frame 300(2) to obtain modified second pixel data 808(2) that aligns the scene in the second frame with the first direction of HMD 102. Similarly, a portion of the second pixel data 110(1)(2) represents data corresponding to the panel mask (displayed in black) surrounding the display panel of HMD 102 for each frame. Therefore, the panel mask portion 814(2) of the modified second pixel data 808(2) (i.e., in...) Figure 8 The portion shown in black in the middle represents the data corresponding to the panel mask in the modified second pixel data 808(2).

[0097] At 810, logic (e.g., the logic of host computer 106) can combine the panel mask portion 814(1) of the modified first pixel data 808(1) with the panel mask portion 814(2) of the modified second pixel data 808(2) to determine a subset of pixels having common values ​​corresponding to panel mask 816. This can be considered as various Venn diagrams, where the subset of pixels corresponding to panel mask 816 is a combined version of the modified panel mask in the modified first pixel data 808(1) and the modified second pixel data 808(2).

[0098] At 812, logic (e.g., the logic of host computer 106) may provide GPU 304 with specific portions (e.g., the central portion) of the first pixel data 110(1)(1) and specific portions (e.g., the central portion) of the second pixel data 110(1)(2), which correspond to pixels other than the subset of pixels corresponding to panel mask 816. This allows GPU 304 to ignore motion estimation for the portion of pixel data 110(1) covered by the pixels of panel mask 816 in each previously rendered frame, and GPU 304 may focus its motion estimation work on the center, i.e., the non-panel mask portion of pixel data 110(1) in each frame 300. As mentioned, process 800 may continue from box 812 to box 504 of process 500, wherein an array of motion vectors 110(2) is received from GPU 304 based on the pixel data provided as input to GPU 304 at box 812, and may be performed Figure 5 The motion smoothing algorithm is used to render the motion-smoothed reprojected frame 300(3).

[0099] Figure 9 This is a flowchart of an exemplary process 900 according to an embodiment disclosed herein, which is used to threshold motion vectors output from the GPU before they are used to generate reprojected frames as part of a motion smoothing technique. Process 900 is described with reference to the preceding figures for discussion purposes.

[0100] At 902, logic (e.g., logic of host computer 106) may provide pixel data 110(1) associated with the previously rendered frame 300 as input to GPU 304. The operations performed at box 902 may be similar to those described with respect to process 500 at box 502.

[0101] At 904, logic (e.g., the logic of host computer 106) may receive an array of motion vectors 110(2) from GPU 304. The operations performed at box 904 may be similar to those described with respect to process 500 at box 504.

[0102] At 906, after receiving motion vector 110(2) from host computer 106, logic (e.g., the logic of HMD 102) can compare the magnitude (or length) of each motion vector in the array of motion vectors 110(2) with a first threshold magnitude to determine a first subset of motion vectors 110(2) with magnitudes greater than the first threshold magnitude. The first threshold magnitude can be used to mitigate the impact of this high-magnitude outlier subset of motion vectors 110(2), which may represent changes in movement or animation between frames that do not belong to virtual object 302. Motion vectors with abnormally high magnitudes may occur for various reasons. GPU 304 may find that the pixel at the top right corner of the second frame 300(2) is sufficiently similar to the pixel at the bottom left corner of the first frame 300(1) and may output a resulting motion vector 404 with a relatively high magnitude compared to other motion vectors 404 in the array, even if the motion vector 404 does not represent movement or animation of virtual object 302 between frames 300. In some cases, video games allow users to teleport to different locations where the scene changes drastically between frames 300, resulting in large motion vectors 404 being generated by the video encoder of the GPU 304. In these and other cases, thresholding these large motion vectors 404 may be useful.

[0103] At 907, when multiple previously rendered frames 300 include a first frame 300(1) and a second frame 300(2), logic (e.g., the logic of HMD 102) can determine the time period (or gap) between a first time when the first frame 300(1) is rendered and a second time when the second frame 300(2) is rendered.

[0104] At 909, the first threshold value used in the comparison in box 906 can be selected at least in part based on the time interval between the first and second times. In an illustrative example, the first threshold value, measured in degrees of movement from the user's viewpoint per unit time, is 6 degrees of movement every 11.11 ms between the two frames 300(1) and 300(2). Therefore, the longer the time interval between the two frames 300(1) and 300(2), the larger the first threshold value, and thus the longer the allowed motion vector 110(2) (the magnitude of the motion vector). This is because once your motion exceeds a certain speed, motion smoothing may not be as effective and can actually lead to undesirable visual artifacts (e.g., by making some parts of the scene appear "blurred").

[0105] At 908, logic (e.g., the logic of HMD 102) may reduce the magnitudes of a first subset of motion vectors 110(2) determined to have magnitudes greater than a first threshold magnitude, such that the magnitudes are limited to within the first threshold magnitude. This creates a first modified array of motion vectors 110(2). In other words, for those motion vectors 110(2) exceeding the first threshold magnitude, the logic is configured to limit the magnitudes of those motion vectors 110(2) to within the first threshold magnitude, such that the first modified array of motion vectors 110(2) includes magnitudes equal to or less than the first threshold magnitude and excludes any magnitudes greater than the first threshold magnitude. In some embodiments, instead of limiting the magnitudes of motion vectors 110(2) to within the first threshold magnitude and using all non-zero motion vectors for motion smoothing, the logic may discard a first subset of motion vectors 110(2) having magnitudes that reach or exceed the first threshold magnitude, such that these motion vectors are not used for motion smoothing at all.

[0106] At 910, logic (e.g., the logic of HMD 102) compares the magnitude (or length) of each motion vector in the array of motion vectors 110(2) with a second threshold magnitude to determine a second subset of motion vectors 110(2) with magnitudes less than the second threshold magnitude. The second threshold magnitude can be used to mitigate the effects of this low-magnitude outlier subset of motion vectors 404, which can represent variations between frames typically caused by constant head movement of the user and / or head tracking, which, while precise, is not precise to the point of absolute zero motion. Therefore, the output of GPU 304 rarely provides zero-length motion vectors 404. Instead, the output of GPU 304 is more commonly characterized by significant noise due to ambient head movement and / or tracking jitter, as described above. In other words, pixels between two consecutive frames rarely match 100%. In the illustrative example, the second threshold magnitude, measured in motion pixels, is a threshold of 2 motion pixels. In this example, any motion vector with a magnitude (or length) less than 2 motion pixels is considered a low-magnitude outlier vector.

[0107] At 912, the logic (e.g., the logic of HMD 102) can be reduced to zero for the magnitudes of a second subset of motion vectors 110(2) that have magnitudes less than a second threshold magnitude (e.g., low-magnitude outlier motion vectors with a length less than 2 motion pixels). This creates a second modified array of motion vectors 110(2) that does not include any motion vectors with magnitudes less than the second threshold magnitude. This is sometimes referred to herein as applying a small dead zone to motion vectors 110(2).

[0108] At 914, logic (e.g., the logic of HMD 102) may modify the pixel data 110(1)(3) of the reprojected frame based on a second modified array of motion vectors 110(2), the magnitudes of which are constrained to within a first threshold magnitude and there are no motion vectors 404 with magnitudes less than a second threshold. In some embodiments, such as when a first subset of motion vectors is discarded instead of constrained to within the first threshold magnitude, a subset of motion vectors 110(2) other than the discarded first subset of motion vectors 110(2) is used to modify the pixel data 110(1)(3) of the reprojected frame at block 914. Modification of the pixel data 110(1)(3) may include any techniques described herein (e.g., those described with reference to blocks 508-512 of process 500).

[0109] At 916, logic (e.g., the logic of HMD 102) may render the reprojected frame 300(3) on the display (e.g., on the display panel of HMD 102) based at least in part on the modified pixel data 110(1)(3)' of the reprojected frame 300(3). The operation performed at box 916 may be similar to those described with respect to process 500 in box 514.

[0110] Figure 10 This is a flowchart of an exemplary process 1000 according to an embodiment disclosed herein, which is used to attenuate motion vectors output from the GPU before they are used to generate reprojected frames as part of a motion smoothing technique. For purposes of discussion, process 1000 is described with reference to the preceding figures.

[0111] At 1002, logic (e.g., logic of host computer 106) may provide pixel data 110(1) associated with the previously rendered frame 300 as input to GPU 304. The operations performed at box 1002 may be similar to those described with respect to process 500 at box 502.

[0112] At 1002, logic (e.g., the logic of host computer 106) may receive an array of motion vectors 110(2) from GPU 304. The operations performed at box 1004 may be similar to those described with respect to process 500 at box 504.

[0113] At 1006, logic (e.g., the logic of HMD 102, after receiving motion vectors 110(2) from host computer 106) can generate an attenuation texture overlaid on the array of motion vectors 110(2) to attenuate (e.g., shorten, reduce, etc.) the magnitude of each motion vector 110(2) (e.g., if the confidence level of the quality of those motion vectors 110(2) is low). The attenuation texture can be generated at any suitable resolution such that the attenuation texture comprises multiple texels (e.g., a texel grid). The resolution of the attenuation texture can be a lower resolution than the resolution of the array of motion vectors 110(2), such that multiple motion vectors are within a single texel of the attenuation texture. In one example, the attenuation texture may have a resolution such that the texels at the center of the user's field of view (FOV) are approximately 6 degrees by 6 degrees (horizontal and vertical). The resolution (horizontal and vertical) of each texel may be approximate because the degree of each texel may not be constant in the attenuation texture. For example, the FOV of the HMD 102 can span an area of ​​approximately 100 degrees by 100 degrees (e.g., plus or minus approximately 10 degrees). The attenuation texture can be generated based on a non-linear projection matrix, such that the resolution varies according to each texel of the attenuation texture, but the resolution provides approximately 6-10 degrees × 6-10 degrees of texels. Furthermore, an attenuation value can be assigned to each texel of the attenuation texture (e.g., in the range of 0.0 and 1.0). The attenuation value assigned to a given texel controls the amount by which the magnitude of the motion vector 404 in that texel decreases (shortens). For example, an attenuation value of 1.0 can correspond to no attenuation; therefore, applying an attenuation value of 1.0 to the motion vector 404 means that the motion vector 404 remains unchanged in terms of its magnitude (e.g., does not decrease, shorten, etc.). However, an attenuation value of 0.0 can correspond to full attenuation; therefore, applying an attenuation value of 0.0 to the motion vector 404 means that the motion vector 404 decreases to zero in terms of its magnitude. Attenuation values ​​between 0.0 and 1.0 can correspond to partial attenuation. Therefore, applying an attenuation value of, for example, 0.5 to motion vector 404 means that the motion vector is reduced to 50% (half) of its original length (magnitude). Thus, attenuation textures can reduce (shorten) the magnitude of motion vector 404, but cannot increase (lengthen) the magnitude of motion vector 404.

[0114] Figure 10An exemplary attenuation texture 1007 is shown, which may have been generated at box 1006 based on a sudden change in brightness (or color) at the bottom of the screen between two consecutively rendered frames 300. For example, while playing a video game, a user of HMD 102 may select a button on a handheld controller, which causes a large blue grid to appear in the lower half of the screen in the next rendered frame 300. In this exemplary scene, when processing this frame 300 and the previously rendered frame 300 as input, GPU 304 may find a pixel rendered at the top of the scene in the sky whose brightness value best matches the brightness value of the pixel in the large blue grid in the lower half of the screen. This causes GPU 304 to output a motion vector field 110(2), in which the motion vector 404 at the bottom of the scene is suddenly very large in magnitude and does not represent actual motion in the scene. In the exemplary attenuation texture 1007, white texels may represent an attenuation value of 1.0, while black texels may represent an attenuation value of 0.0, and the hue of gray texels represents the attenuation value between 0.0 and 1.0 (excluding the end values).

[0115] At 1008, logic (e.g., the logic of HMD 102) can use the decay texture generated at box 1006 to reduce (e.g., shrink) the magnitude of each motion vector 404 in the motion vector 110(2) array (e.g., using the decay value of the decay texture to reduce the magnitude of motion vector 110(2) by the amount corresponding to that decay value). This creates a modified motion vector 110(2) array, assuming that at least one decay value in the decay texture is less than 1.0. The aim is to shrink motion vectors 404 that change by more than a threshold amount between frames 300, which likely indicates that these motion vectors 404 do not represent actual motion within the scene and should not be mistaken for representing motion. The decay texture is used to find areas of screen area where motion vectors are extremely large (in terms of their magnitude) in a single frame, indicating that these motion vectors cannot be trusted to represent actual motion, or, if they do represent actual motion in the scene, that motion is too fast for motion smoothing to have a noticeable effect anyway, and it is better to use zero motion vectors rather than trying to use these motion vectors for motion smoothing. In reality, whenever there is motion between frames, the motion is usually not significantly different from the previous frame because the refresh rate of a display is typically around 90Hz, and it is difficult for anything to make a very large movement in that short period between frames. Therefore, these maximal motion vectors 404, attenuated by the decaying texture, do not usually represent actual motion.

[0116] As shown in subframe 1009, in some embodiments, using an attenuation texture at frame 1008 to reduce the magnitude of the motion vector may include applying a minimum N×N filter to the attenuation texture before using the attenuation texture to reduce the magnitude of the motion vector. Here, N can be any suitable number. N=3 uses a minimum 3×3 filter (i.e., a 3×3 texel block). Using N=3 as an example, when applying the minimum 3×3 filter to the attenuation texture at frame 1009, the logic (e.g., the logic of HMD 102) may, based on each texture, obtain the minimum of the nine attenuation values ​​in each 3×3 texel block centered on the corresponding texel (e.g., a texel block including the corresponding texel and its eight adjacent texels) from the attenuation texture 1007. Applying the minimum N×N filter at box 1009 typically results in more motion vectors 404 being zeroed out because any corresponding texel of an attenuation texture assigned a non-zero attenuation value, but whose neighboring texels are assigned zero attenuation values, will apply the zero attenuation value to the corresponding motion vector of that texel, thereby reducing the motion vector in that texel to zero, even though the corresponding texel was assigned a non-zero attenuation value in attenuation texture 1007. In other words, if the neighboring texels of a given texel of an attenuation texture are assigned zero attenuation values, meaning that the neighboring texels have a large number of high-value motion vectors, then the motion vector 404 in that given texel is zeroed out. Figure 10 An exemplary attenuation texture 1011 is shown, which is the result of applying a minimum N×N filter to attenuation texture 1007. The resulting attenuation texture 1011 includes more black texels than attenuation texture 1007, which means that more motion vectors are zeroed out using the resulting attenuation texture 1011.

[0117] At 1010, logic (e.g., the logic of HMD 102) may modify the pixel data 110(1)(3) of the reprojected frame based on a modified array of motion vectors 110(2), which may have some amount of attenuation for attenuation values ​​less than 1.0 in the attenuation texture. Modification of pixel data 110(1)(3) may include any techniques described herein (e.g., those described in boxes 508-512 of reference process 500).

[0118] At 1012, logic (e.g., the logic of HMD 102) may render the reprojected frame 300(3) on the display (e.g., on the display panel of HMD 102) based at least in part on the modified pixel data 110(1)(3)' of the reprojected frame 300(3). The operation performed at box 1012 may be similar to those described with respect to box 514 of process 500.

[0119] Figure 11This is a flowchart of an exemplary process 1100 according to an embodiment disclosed herein, which is used to generate a decay texture that can be used to attenuate these motion vectors before the motion vectors output from the GPU are used to generate reprojected frames as part of a motion smoothing technique. Therefore, process 1100 may represent a subprocess that can be performed at block 1006 of process 1000 to generate the decay texture. For purposes of discussion, process 1100 is described with reference to the preceding figures.

[0120] At 1102, after receiving the motion vector 110(2) array from the host computer 106, the logic of the HMD 102 may determine a set of incremental vectors for each texel of the attenuation texture based at least in part on a plurality of previously obtained motion vector 110(2) arrays. For example, the history of the motion vector 110(2) arrays may be maintained while processing a series of frames, and the last two motion vector 110(2) arrays (e.g., the two most recent motion vector 110(2) arrays) may be compared at frame 1102 to determine the set of incremental vectors (e.g., difference vectors indicating the difference (e.g., the difference in magnitude) between a pair of corresponding motion vectors in the two arrays).

[0121] At box 1104, logic (e.g., the logic of HMD 102) can calculate the percentage of increment vectors in each texel of the decaying texture that have an increment (e.g., a magnitude) greater than a threshold value. For example, at box 1104, this logic can evaluate the increment vectors in a given texel of the decaying texture, identify “outlier” vectors in the texel as those increment vectors that have an increment (e.g., a magnitude) greater than a threshold value, and calculate the percentage of outlier vectors in the texel. Any suitable threshold can be used for the purpose of identifying “outlier” vectors at box 1104, such as a threshold of approximately 3 degrees of movement / motion (from the user’s perspective) per frame (or every 11.11 milliseconds). In other words, the threshold used at box 1104 can be based on the time period (or gap) between the first time when the first frame 300(1) is rendered and the second time when the second frame 300(2) is rendered, as referenced above. Figure 9 The process is described in blocks 907 and 909 of 900.

[0122] At 1106, logic (e.g., the logic of HMD 102) can determine for each texel of the decaying texture whether the percentage of increment vectors in the texel that are “outlier” increment vectors reaches or exceeds a threshold percentage, such as 25%. At box 1106, if the percentage of increment vectors in the texel that are “outlier” increment vectors reaches or exceeds that threshold percentage, then process 1100 can follow a “yes” route from box 1106 to box 1108.

[0123] At 1108, logic (e.g., the logic of HMD 102) can calculate a decay value of zero for texels with a number of “outlier” increment vectors that reach or exceed a threshold percentage. In other words, if a certain amount (e.g., 25% or more) of the increment vectors of a given texel is greater than the maximum length, the logic can calculate a decay value of zero for that texel. As described in process 1000, applying a decay value of zero to the motion vectors 404 within a given texel reduces the magnitude of those motion vectors 404 to zero (e.g., completely decays the motion vectors within that texel).

[0124] At 1106, for texels having a percentage less than a threshold value for the “outlier” increment vector, process 1100 may follow a “No” route from box 1106 to box 1110, where the logic may calculate the attenuation value for those texels as at least in part based on the percentage calculated at box 1104. For example, if the percentage of increment vectors having a magnitude greater than a threshold value for a given texel is less than a threshold percentage (e.g., 25%), the attenuation value may be set to be linearly mapped to a value calculated as a percentage within a range of values ​​less than the threshold percentage (e.g., 0%–25%). The attenuation value calculated at box 1110 can effectively (when the attenuation texture is applied to motion vector field 110(2)) reduce noise in the remaining vectors that are not zeroed out.

[0125] At box 1112, the calculated attenuation value can be compared with the existing attenuation value of the texel based on the previously rendered frame 300. At box 1112, if the attenuation value is less than the existing value, process 1100 can follow a "yes" route from box 1112 to box 1114, where the attenuation value of the texel is set to the attenuation value calculated at box 1108 or box 1110. For example, if the texel that decayed the texture in the previous frame did not decay, the existing attenuation value might be equal to 1.0, and if the new attenuation value calculated at box 1108 or box 1110 is less than 1.0, then in this example, the attenuation value is set to the newly calculated attenuation value to attenuate the motion vector in the given texel.

[0126] At 1112, if the calculated attenuation value is not less than the existing value of the texel in question, then process 1100 may follow a "No" route from box 1112 to box 1116, where the existing attenuation value increases over time toward the maximum attenuation value corresponding to no attenuation (e.g., 1.0). For example, due to unstable motion vectors that can change between frames due to certain conditions, the new attenuation value may be "mixed" with the previous (old) attenuation value of the texel of the attenuating texture, meaning that the attenuation value increases incrementally to no attenuation over a period of time (e.g., within 1 second), assuming that the calculated attenuation value is never less than the existing attenuation value of the texel during that period of time.

[0127] Figure 12This is a flowchart of an exemplary process 1200 according to an embodiment disclosed herein, which is used to zero out motion vectors corresponding to regions with little color change before the resulting set of motion vectors is used to generate a reprojected frame as part of a motion smoothing technique. For purposes of discussion, process 1200 is described with reference to the preceding figures.

[0128] At 1202, logic (e.g., logic of host computer 106) may provide pixel data 110(1) associated with the previously rendered frame 300 as input to GPU 304. The operations performed at box 1202 may be similar to those described with respect to box 502 of process 500.

[0129] At 1204, logic (e.g., the logic of host computer 106) may receive an array of motion vectors 110(2) from GPU 304. The operations performed at box 1204 may be similar to those described with respect to box 504 of process 500.

[0130] At 1206, logic (e.g., the logic of HMD 102, after receiving motion vectors 110(2) from host computer 106) can compare color pixel values ​​(in terms of RGB vectors) between multiple previously rendered frames 300 input to GPU 304 to determine a color difference value for each motion vector in the array of motion vectors 110(2). These color difference values ​​can be compared with a threshold color difference to determine a subset of motion vectors 110(2) that are associated with little color change between frames because their color difference values ​​are less than the threshold color difference. In some implementations, this logic treats RGB pixel values ​​as vectors. The corresponding RGB vectors between previously rendered frames 300 are then compared to determine color differences (e.g., incremental vectors). Since each motion vector 404 in the array of motion vectors 110(2) may correspond to a pixel block (e.g., an 8×8 pixel block), this color comparison can be performed on each motion vector by comparing some or all of the RGB pixel values ​​in the 8×8 pixel block between frames 300. In some implementations, logic (e.g., the logic of HMD 102) compares the color values ​​of every row and every column of pixels in each 8×8 pixel block corresponding to a single motion vector 404 to determine the amount of color change (e.g., color difference value) between frames 300, which is associated with a particular motion vector 404. Therefore, color comparison between frames may involve comparing a portion (e.g., 25%) of the pixels in the corresponding pixel block relative to a given motion vector 110(2) to save processing resources.

[0131] At 1208, logic (e.g., the logic of HMD 102) reduces the magnitude of a subset of motion vectors 110(2) where the color difference value is less than a threshold color difference to zero length / magnitude. This creates a modified array of motion vectors 110(2) that excludes any motion vectors (sometimes referred to as zero motion color difference) in scene areas where there is little color change between previously rendered frames. For example, this helps prevent text edges from appearing blurry or curved when it is in front of a solid color background. This also helps prevent visually distracting artifacts whenever lights or other user interface elements in the scene gradually brighten or darken.

[0132] At 1210, logic (e.g., the logic of HMD 102) may modify the pixel data 110(1)(3) of the reprojected frame based on a modified array of motion vectors 110(2), which no longer includes any motion vectors 404 in areas where the colors are actually the same between previously rendered frames 300. Modification of pixel data 110(1)(3) may include any techniques described herein (e.g., those described in boxes 508-512 of reference process 500).

[0133] At 1212, logic (e.g., the logic of HMD 102) may render the reprojected frame 300(3) on the display (e.g., on the display panel of HMD 102) based at least in part on the modified pixel data 110(1)(3)' of the reprojected frame 300(3). The operation performed at box 1212 may be similar to those described with respect to box 514 of process 500.

[0134] Figure 13 This is a flowchart of an exemplary process 1300 according to an embodiment disclosed herein, which is used to “clean up” the motion vector field using one or more filters before the resulting set of motion vectors is used to generate a reprojected frame as part of a motion smoothing technique. For purposes of discussion, process 1300 is described with reference to the preceding figures.

[0135] At 1302, logic (e.g., logic of host computer 106) may provide pixel data 110(1) associated with the previously rendered frame 300 as input to GPU 304. The operations performed at box 1302 may be similar to those described with respect to box 502 of process 500.

[0136] At 1304, logic (e.g., the logic of host computer 106) may receive an array of motion vectors 110(2) from GPU 304. The operations performed at box 1304 may be similar to those described with respect to process 500 at box 504.

[0137] At 1306, logic (e.g., the logic of HMD 102, after receiving motion vector 110(2) from host computer 106) can apply an N×N scalar median filter to the motion vector 110(2) array to obtain a first modified motion vector 110(2) array. Here, N can be any suitable number, such as N=5, which will apply a 5×5 scalar median filter. In the example of N=5, the 5×5 scalar median filter is applied to each motion vector in the motion vector 110(2) array by looking at a 5×5 set of motion vectors 110(2) surrounding the selected motion vector. For the selected motion vector, there are 25 motion vectors in the 5×5 region surrounding the selected vector. The scalar median filter applied to the selected motion vector calculates the median x-component value (out of 25 x-component values) and the median y-component value (out of 25 y-component values), and then combines the median x-component value and the median y-component value to obtain a scalar median vector, which can be used to replace the selected motion vector. The process iterates over each vector in the array of motion vectors 110(2). Again, any suitable value of N can be used, and N=5 is merely an example. Applying a scalar median filter at box 1306 reduces noise in the motion vector field 110(2) and helps remove outlier motion vectors. Using a scalar median filter at box 1306 is also computationally faster than applying more complex filters, such as vector median filters, which would rely on a more computationally dense and complex distance algorithm.

[0138] At 1308, logic (e.g., the logic of HMD 102) applies an M×M average-maximum blur filter to the first modified motion vector 110(2) array to obtain a second modified motion vector 110(2) array. Here, M can be any suitable number, such as M=3, which will apply a 3×3 average-maximum blur filter. This average-maximum blur filter independently applies two separate filters (average vector filter and maximum length vector filter), and then takes the vector average of the resulting vectors from the two independently applied filters. In the example using M=3, the 3×3 average-maximum blur filter is applied to each motion vector in the motion vector 110(2) array by looking at a 3×3 set of motion vectors 110(2) around the selected motion vector. For the selected motion vector, there are 9 motion vectors in the 3×3 region around the selected vector. First, an average vector filter is applied to the selected motion vector, the average x-component value (in 9 x-component values) and the average y-component value (in 9 y-component values) are calculated, and then the average x-component value and the average y-component value are combined to obtain the average vector. Next, the maximum length vector filter is applied independently to the selected motion vectors to determine the longest vector (out of 9 motion vectors). The resulting vector is then determined by taking the vector average of the average vector and the longest vector, and this resulting vector can be used to replace the selected motion vector. This process is iterated for each vector in the array of motion vectors 110(2). Again, any suitable value of M can be used, and M=3 is only an example. Applying the mean-maximum blur filter at box 1308 provides a smooth motion vector field that eliminates discontinuities in subregions of the motion vector group. In some embodiments, applying the mean-maximum blur filter at box 1308 is a final “clean-up” step of the motion vector field 110(2) before motion smoothing, as described herein. For example, these various operations can be performed before applying the mean-maximum blur filter if used in combination with other techniques described herein, such as thresholding motion vectors, using attenuated textures, zeroing vectors in areas with little color difference, etc.

[0139] At 1310, logic (e.g., the logic of HMD 102) may modify the pixel data 110(1)(3) of the reprojected frame based on a modified motion vector array 110(2) having filtered motion vectors. Modification of pixel data 110(1)(3) may include any techniques described herein (e.g., those described in blocks 508-512 of reference process 500).

[0140] At 1312, logic (e.g., the logic of HMD 102) may render the reprojected frame 300(3) on the display (e.g., on the display panel of HMD 102) based at least in part on the modified pixel data 110(1)(3)' of the reprojected frame 300(3). The operation performed at box 1312 may be similar to those described with respect to process 500 in box 514.

[0141] It should be understood that, Figures 9 to 13 The various techniques described in the process (various techniques for modifying the motion vector field output by the GPU 304) can be used in any combination and / or in any order. For example, refer to Figure 9 The described techniques are comparable to those in the reference section. Figure 10 and Figure 11 The described techniques are used in combination, and some or all of these techniques may be used in conjunction with the references. Figure 12 The described techniques are used in combination, and some or all of these techniques may be used in conjunction with the references. Figure 13 The described techniques are used in combination. That is, robust techniques for obtaining optimized motion vector fields can be utilized using reference... Figures 9 to 13 All techniques described herein work together to robustly modify the motion vector field. Furthermore, it should be understood that in a distributed system including a host computer 106 and a communicatively coupled HMD 102, in Figures 9 to 13 Some or all of the techniques described in the process (various techniques for modifying the motion vector field output by GPU 304) may be executed on host computer 106. For example, host computer 106 may be configured to threshold the motion vectors (e.g., using a decaying texture) and / or apply filters to the motion vectors for motion smoothing before sending the modified motion vector array to HMD 102. In this scenario, the modified motion vector array 110(2) may be sent out-of-band from the compressed pixel data 110(1), and / or zero motion vector run-length encoding may be performed in the modified motion vector array 110(2) to compress the motion vector data before transmitting the motion vectors to HMD 102.

[0142] Figure 14 This is a flowchart of an exemplary process 1400 according to an embodiment disclosed herein, which is used to rotate a previously rendered frame before inputting pixel data to the GPU for motion vector estimation. For purposes of discussion, process 1400 is described with reference to the preceding figures.

[0143] At 1402, logic (e.g., logic of host computer 106) can rotate pixel data 110 (1) by a certain amount from the first frame 300 (1) of a plurality of previously rendered frames 300 provided as input to GPU 304 to obtain a first rotated frame 1404 (1).

[0144] At 1406, logic (e.g., logic of host computer 106) can rotate the second frame 300(2) of the plurality of previously rendered frames 300 by a certain amount of rotation to obtain a second rotated frame 1404(2).

[0145] At 1408, logic (e.g., logic of host computer 106) can provide the pixel data of rotated frames 1404(1) and 1404(2) as input to GPU 304. Process 1400 can continue from box 1408 to box 504 of process 500 (as shown in external reference "A"). Thus, at box 504, the motion vector 110(2) array can be received from GPU 304 based on the pixel data provided as input to GPU 304 at box 1408, and the following can be performed. Figure 5 The remaining operations of the motion smoothing algorithm are used to render the motion-smoothed reprojected frame 300(3).

[0146] Process 1400 mitigates the effects of horizontal and / or vertical building edges in the scene to avoid generating “strange” motion vectors that do not truly represent moving or animated objects. Multiple previously rendered frames 300(1) and 300(2) can be rotated by the same amount at frames 1402 and 1406, or each frame 300(1) and 300(2) can be rotated by different amounts at frames 1402 and 1406, respectively. In some implementations, the rotation amount for each frame at frames 1402 and 1406 is predefined (e.g., rotating input frames 300(1) and 300(2) by 45 degrees). Rotating frames 300(1) and 300(2) by 45 degrees makes the horizontal and vertical edges in the scene diagonally aligned, thus preventing the video encoder of GPU 304 from generating as many “strange” motion vectors. In some implementations, the rotation amount for each frame at boxes 1402 and 1406 can be a random rotation amount for each frame 300. This may temporarily hide any edges that might be aligned with 45 degrees.

[0147] Figure 15 This is a flowchart of an exemplary process 1500 according to an embodiment disclosed herein, which is used to select between a motion vector array generated based on luminance data and chrominance data of a previously rendered frame. For purposes of discussion, process 1500 is described with reference to the preceding figures. Furthermore, as... Figure 5 and Figure 15As shown in the external reference "B" in the document, process 1500 can be represented in... Figure 5 The operation performed before the operation at box 506, and in some implementations, process 500 may continue the operations of boxes 506-514.

[0148] At 1502, logic (e.g., logic of host computer 106) may provide luminance data associated with a previously rendered frame 300 as input to GPU 304. For example, first pixel data 110(1)(1) associated with the first frame 300(1) may include first luminance data, first chrominance data, and / or additional types of data. Similarly, second pixel data 110(1)(2) associated with the second frame 300(2) may include similar types of data, such as second luminance data, second chrominance data, etc. Therefore, at box 1502, first luminance data and second luminance data may be provided as input to GPU 304.

[0149] At 1504, logic (e.g., the logic of host computer 106) may receive from GPU 304 an array of first motion vectors 110(2) generated by the video encoder of GPU 304 based at least in part on first luminance data and second luminance data.

[0150] At 1506, logic (e.g., logic of host computer 106) can provide GPU 304 with first chroma data associated with the first frame 300(1) and second chroma data associated with the second frame 300(2) as input.

[0151] At 1508, logic (e.g., the logic of host computer 106) may receive from GPU 304 an array of second motion vectors 110(2) generated by the video encoder of GPU 304 based at least in part on the first chroma data and the second chroma data.

[0152] At 1510, logic (e.g., the logic of host computer 106, or the logic of HMD 102 after receiving multiple arrays of motion vectors 110(2)) may select one of the first or second arrays of motion vectors 110(2) as the selected motion vector array based on analysis of the first and second arrays of motion vectors 110(2). For example, as an example, the selection algorithm may select the most unstable motion vector based on the magnitude and / or direction of the motion vectors 110(2) in each field of motion vectors 110(3). For example, when a bright flash (e.g., a virtual explosion) occurs in the second frame 300(2), the luminance data may cause the GPU 304 to output a motion vector 110(2) with a magnitude exceeding a certain threshold value, while the chroma data of the same previously rendered frame 300 may cause the GPU 304 to output a motion vector 110(2) that does not include such motion vectors with extremely large values. In this case, at box 1510, an array of motion vectors 110(2) generated from chroma data can be selected. This is merely one example of a selection algorithm, and other algorithms may be employed. As described above, process 1500 may continue from box 1510 to box 506 of process 500, where the pixel data 110(1)(3) of the reprojected frame 300(3) is modified based on the selected array of motion vectors 110(2) from box 1510.

[0153] Figure 16 This is a flowchart of an exemplary process 1600 according to an embodiment disclosed herein, which is used to obtain multiple motion vector arrays, determine the differences between the arrays, and generate a final motion vector array for motion smoothing based on the determined differences. For discussion purposes, process 1600 is described with reference to the preceding figures. Furthermore, as... Figure 5 and Figure 16 As shown in the external reference "B" in the document, process 1600 can be represented in... Figure 5 The operation performed before the operation at box 506, and in some implementations, process 500 may continue the operations of boxes 506-514.

[0154] At 1602, logic (e.g., logic of host computer 106) may provide the first pixel data 110(1)(1) associated with the previously rendered frame 300 as input to GPU 304. The operations performed at box 1602 may be similar to those described with respect to box 502 of process 500.

[0155] At 1604, logic (e.g., logic of host computer 106) may receive an array of first motion vectors 110(2) from GPU 304 based on first pixel data 110(1)(1) provided as input at box 1602.

[0156] At 1606, logic (e.g., the logic of host computer 106) can provide the second pixel data 110(1)(2) associated with the previously rendered frame 300 as input to GPU 304.

[0157] At 1608, logic (e.g., the logic of host computer 106) may receive the array of second motion vectors 110(2) from GPU 304 at least in part based on the second pixel data 110(1)(2).

[0158] At 1610, logic (e.g., the logic of host computer 106, or the logic of HMD 102 after receiving the motion vector array 110(2) from host computer 106) can determine the difference in direction and / or magnitude of the motion vectors 110(2) between the first array 110(2) and the second array 110(2). For example, a comparison can be made between motion vector 404 in the first array 110(2) and a corresponding motion vector 404 in the second array 110(2) to determine whether these motion vectors differ in one or both of their direction or magnitude, and if so, to determine the amount of difference between these motion vectors.

[0159] At 1612, logic (e.g., the logic of host computer 106, or the logic of HMD 102) may generate a final motion vector 110(2) array based at least in part on the difference between the first motion vector 110(2) array or the second motion vector 110(2) array, as determined at block 1610. For example, the final motion vector 110(2) array may represent a set of motion vectors generated by attenuating a single motion vector in the first array 110(2) or the second array 110(2) or by attenuating an average motion vector representing the average of the corresponding motion vectors in each array 110(2). As described above, process 1600 may continue from block 1612 to block 506 of process 500, wherein the pixel data 110(1)(3) of the reprojected frame 300(3) is modified based on the final motion vector 110(2) array generated at block 1612.

[0160] As an illustrative example of how process 1600 can be used, consider the following example: where a first motion vector 110(2) array is received at frame 1604 based on pixel data 110(1) of input frames 300(1) and 300(2) in their original (“y-up”) orientation, and a second motion vector 110(2) array is received at frame 1608 based on pixel data 110(1) of input frames 300(1) and 300(2) that have been rotated 45 degrees. Reference Figure 14The concept of rotating an input image frame 300 is described. Therefore, in this illustrative example, pixel data 110(1) of the first frame 300(1) and the second frame 300(2) can be provided as input at box 1602 to generate a first motion vector 110(2) array, and pixel data 110(1) of the first rotated frame 1404(1) and the second rotated frame 1404(2) can be provided as input at box 1606 to generate a second motion vector 110(2) array. In this illustrative example, the two motion vector 110(2) arrays can be realigned (e.g., by rotating the second array 110(2) by 45 degrees in opposite directions), and mismatches are determined by comparing the angular difference (direction) and / or length difference (magnitude) between corresponding vectors in both arrays 110(2). Actual motion (e.g., a moving or animated object in a scene) may generate very similar motion vector arrays for rotating the input frame 300, while incorrectly detected motion may generate very different motion vector arrays. In this exemplary example, the final array of motion vectors 110(2) can be generated at box 1612 by attenuating the length of the average of the two corresponding vectors in each array 110(2) by the dot product of the two normalized vectors. In some embodiments, attenuation textures (such as reference textures) can be utilized at box 1612. Figure 10 and Figure 11 The attenuation texture is used to generate the final array. For example, a set of incremental vectors determined at block 1102 of process 1100 may be based on the increment between the first motion vector 110(2) array received at block 1604 and the second motion vector 110(2) array received at block 1608, and the resulting attenuation texture may be used at block 1612 to attenuate the first array 110(2), the second array 110(2), or a combination of vectors between the two arrays 110(2) (e.g., average value).

[0161] As another illustrative example of how process 1600 can be used, consider the following example: a first mip level (mipmap level) is generated for an input texture of multiple previously rendered frames 300, and the corresponding pixel data 110(1) of the first mip level is provided as input to GPU 304 at box 1602 to generate a first motion vector 110(2) array. Simultaneously, a second mip level can be generated for an input texture of multiple previously rendered frames 300, and the corresponding pixel data 110(1) of the second mip level can be provided as input to GPU 304 at box 1606 to generate a second motion vector 110(2) array. This can be done even for any number of mip levels to generate any number of corresponding motion vector arrays. In some embodiments, three or four mip levels of the input texture can be generated such that three or four motion vector 110(2) arrays are received before box 1610. In some embodiments, each mip level has half the width and height of the previous mip level (e.g., 25% of the area). At box 1610, in this exemplary example, the differences between pairs of arrays 110(2) generated from different mip levels can be determined, and at box 1612, the final array of motion vectors 110(2) can be generated based on the determined differences. For example, this logic can detect anomalies across multiple arrays of motion vectors 110(2) for different mip levels and can attenuate the anomalous motion vectors (e.g., reduce them to zero). This example of using different mip levels of input frame 300 to generate different arrays of motion vectors 110(2) is helpful in situations where there is a large area of ​​repeating pattern in the scene (e.g., wallpaper); and the video encoder of GPU 304 may additionally generate a large number of high-value motion vectors, assuming a large amount of motion due to the frequency of the repeating pattern relative to the motion vector resolution, even if there are no moving or animated objects. Therefore, when generating arrays of motion vectors 110(2) for different mip levels, process 1600 helps to detect anomalies at different frequencies of the repeating pattern, and when anomalies are detected, attenuation textures can be used to attenuate (e.g., reduce) those motion vectors 110(2) to zero.

[0162] As another example of how process 1600 can be used, consider the following example: where multiple arrays 110(2) are obtained at different resolutions, and at block 1610, differences are determined by comparing a single motion vector in the first array 110(2) with multiple corresponding motion vectors in the second array 110(2) (e.g., the average of the multiple corresponding motion vectors in the second array 110(2) in terms of direction and / or magnitude). Using arrays of motion vectors 110(2) at different resolutions can produce differences that can be used to generate the final array of motion vectors at block 1612.

[0163] Figure 17 This is a flowchart of an exemplary process 1700 according to an embodiment disclosed herein, which is used to obtain multiple motion vector arrays for different portions of an image region at different resolutions. For discussion purposes, process 1700 is described with reference to the preceding figures. Furthermore, as... Figure 5 and Figure 17 As shown in the external page reference "B", process 1700 can be represented in Figure 5 The operation performed before the operation at box 506, and in some implementations, process 500 may continue the operations of boxes 506-514.

[0164] At 1702, logic (e.g., logic of host computer 106) may provide pixel data 110(1) associated with the previously rendered frame 300 as input to GPU 304. The operations performed at box 1702 may be similar to those described with respect to process 500 at box 502.

[0165] At 1704, logic (e.g., logic of host computer 106) may receive a first motion vector 110(2)(1) array from GPU 304, the first motion vector 110(2)(1) array being received at a first resolution or otherwise upsampled or downsampled to a first resolution.

[0166] At 1706, logic (e.g., the logic of host computer 106) may detect the absence of motion in a first portion 1705(A) of an image region 1705 spanning the display, at least in part, based on the first motion vector array 110(2)(1), and may detect motion of object 302 in a second portion 1705(B) of the image region 1705, at least in part, based on the first motion vector array 110(2)(1). For example, non-zero motion vectors may be detected in the right half of the image region 1705, while the left half of the image region 1705 may contain zero-value motion vectors that do not have any non-zero motion vectors.

[0167] At 1708, logic (e.g., logic of host computer 106) may provide the first portion of the first pixel data 110(1)(1) (associated with the first frame 300(1)) and the first portion of the second pixel data 110(1)(2) (associated with the second frame 300(2)) as input to GPU 304. The first portion of the first pixel data 110(1)(1) and the first portion of the second pixel data 110(1)(2) may each correspond to the first portion 1705(A) of the image region 1705 where no motion is detected at frame 1706.

[0168] At 1710, logic (e.g., logic of host computer 106) may receive from GPU 304 a second motion vector array 110(2)(2) generated by the video encoder of GPU 304 based at least in part on a first portion of first pixel data 110(1)(1) and a first portion of second pixel data 110(1)(2). This second motion vector array 110(2)(2) may be generated at a second resolution higher than the first resolution of the first motion vector array 110(2)(1). Obtaining a higher resolution motion vector field 110(2)(2) for a first portion 1705(A) of an image region 1705 in which no motion is detected may be based on the concept that a higher resolution motion vector field 110(2)(2) facilitates the detection of small-scale movements in the first portion 1705(A) of the image region 1705 in which no large-scale movement is detected.

[0169] At 1712, logic (e.g., the logic of host computer 106) may provide the second portion of the first pixel data 110(1)(1) (associated with the first frame 300(1)) and the second portion of the second pixel data 110(1)(2) (associated with the second frame 300(2)) as input to GPU 304. The second portion of the first pixel data 110(1)(1) and the second portion of the second pixel data 110(1)(2) may each correspond to the second portion 1705(B) of the image region 1705 in which motion of object 302 is detected at frame 1706.

[0170] At 1714, logic (e.g., logic of host computer 106) may receive from GPU 304 a third motion vector array 110(2)(3) generated by the video encoder of GPU 304 based at least in part on the second portion of the first pixel data 110(1)(1) and the second portion of the second pixel data 110(1)(2). This third motion vector array 110(2)(3) may be generated at a third resolution lower than the second resolution of the second motion vector array 110(2)(2). This relatively lower resolution motion vector field 110(2)(3) for the second portion 1705(B) of the image region 1705 where motion was detected may be based on the concept that a higher resolution motion vector field may not be necessary for detecting movement in the second portion 1705(B) of the image region 1705 where large-scale movement has already been detected at the first resolution of the first motion vector field 110(2)(1).

[0171] At 1716, logic (e.g., the logic of host computer 106, or the logic of HMD 102 after receiving the motion vector array 110(2) from host computer 106) can perform motion smoothing using the second motion vector array 110(2)(2) and the third motion vector array 110(2)(3). Process 1700 can continue from box 1716 to box 506 of process 500, wherein the pixel data 110(1)(3) of the reprojected frame 300(3) is modified based on the motion vector array 110(2) from box 1716. In one example, motion smoothing using multiple motion vector arrays at box 1716 can include the operations described with reference to box 1612 of process 1600.

[0172] Figure 18A and Figure 18B Two alternative configurations of a system utilizing HMD 102 and host computer 106 according to the embodiments disclosed herein are shown. For example... Figure 1 As shown, an exemplary embodiment involves the host computer 106 and the HMD 102 worn by the user 104 co-located in the environment. For example, the host computer 106 may be located in the user 104's house, while the user 104 is using the HMD 102 in that house, regardless of whether the host computer 106 is located in the same room as the HMD 102 or in a different room. Alternatively, the host computer 106, in the form of a mobile computing device (e.g., a tablet or laptop), may be carried in a backpack on the user 104's back, allowing for greater mobility. For example, when using such a system, the user 104 may be in a park.

[0173] Figure 18A An alternative embodiment is shown, in which host computer 106 represents one or more server computers located at a geographically remote location relative to HMD 102. In this case, HMD 102 may be communicatively coupled to host computer 106 via access point (AP) 1800 (such as a wireless AP (WAP), base station, etc.). In the exemplary example, data is exchanged (e.g., streaming) between host computer 106 and HMD 102 via AP 1800, such as by streaming data over the Internet.

[0174] Figure 18B Another alternative embodiment is shown, in which host computer 106 is communicatively coupled to HMD 102 via intermediate computing device 1802 such as a laptop computer or tablet computer. Figure 18A and Figure 18B The difference is Figure 18A The AP 1800 in the system can simply act as a data routing device that does not perform rendering, while Figure 18BThe intermediate computing device 1802 can perform a portion of the rendering workload. That is, instead of splitting the rendering workload between the host computer 106 and the HMD 102, the rendering workload can be divided among more than two devices (such as three devices): the host computer 106, the intermediate computing device 1802, and the HMD 102. Figure 18B In the scenario, host computer 106 can generate pixel data 110(1), as described herein, intermediate computing device 1802 can perform a first set of rendering operations to modify pixel data 110(1), and HMD 102 can perform a final set of rendering operations to modify the modified pixel data.

[0175] Figure 19 Exemplary components of an HMD 1900 (such as an embeddable VR headset) according to embodiments disclosed herein are shown. HMD 102 can be implemented as a device to be worn by user 104 (e.g., worn on the head of user 104). In some embodiments, HMD 102 can be head-worn, such as by allowing user 104 to secure HMD 102 to his / her head using a fixation mechanism (e.g., an adjustable strap) sized to fit around user 104's head. In some embodiments, HMD 102 includes a virtual reality (VR) or augmented reality (AR) headset that includes a near-eye or near-eye display. Thus, the terms "wearable device," "wearable electronic device," "VR headset," "AR headset," and "head-mounted display (HMD)" may be used interchangeably herein to refer to device 102. However, it should be understood that these types of devices are merely examples of HMD 102, and it should be understood that HMD 102 can be implemented in a variety of other form factors.

[0176] In the illustrated embodiment, HMD 102 includes one or more processors 1900 and memory 1902 (e.g., computer-readable medium 1902). In some embodiments, processor 1900 may include a central processing unit (CPU), GPU 1900(1), both CPU and GPU 1900(1), a microprocessor, a digital signal processor, or other processing units or components known in the art. Alternatively or otherwise, the functions described herein may be performed at least in part by one or more hardware logic components. Examples, but not limited to, exemplary types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SOCs), complex programmable logic devices (CPLDs), etc. Additionally, each processor in processor 1900 may have its own local memory, which may also store program modules, program data, and / or one or more operating systems.

[0177] Memory 1902 may include volatile and non-volatile memory, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Such memory includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage devices, magnetic tape, magnetic tape, disk storage devices or other magnetic storage devices, RAID storage systems, or any other medium that can be used to store desired information and is accessible by a computing device. Memory 1902 may be implemented as a computer-readable storage medium (“CRSM”), which may be any available physical medium accessible to processor 1900 to execute instructions stored on memory 1902. In one basic embodiment, the CRSM may include random access memory (“RAM”) and flash memory. In other embodiments, the CRSM may include, but is not limited to, read-only memory (“ROM”), electrically erasable programmable read-only memory (“EEPROM”), or any other tangible medium that can be used to store desired information and is accessible by processor 1900.

[0178] Generally, HMD 102 may include logic (e.g., software, hardware, and / or firmware, etc.) configured to implement the techniques, functions, and / or operations described herein. Computer-readable medium 1902 is shown to include various modules, such as instructions, data storage, etc., which may be configured to execute on processor 1900 to implement the techniques, functions, and / or operations described herein. Some exemplary functional modules are shown to be stored in computer-readable medium 1902 and executable on processor 1900; however, the same functionality may alternatively be implemented using hardware, firmware, or system-on-a-chip (SoC) and / or other logic.

[0179] For the benefit of other modules, the operating system module 1904 may be configured to manage hardware located within and coupled to the HMD 102. Additionally, in some instances, the HMD 102 may include one or more applications 1906 stored in memory 1902 or otherwise accessible by the HMD 102. In this specific implementation, application 1906 includes a game application 1908. However, the HMD 102 may include any number or type of applications and is not limited to the specific examples shown herein. A compositor 1910, combined with other logic of the HMD 102, may be configured to perform the motion smoothing techniques described herein to render a motion-smoothed reprojected frame 300.

[0180] Generally, HMD 102 has an input device 1912 and an output device 1914. Input device 1912 may include control buttons. In some embodiments, one or more microphones may be used as input device 1912 to receive audio input, such as user voice input. In some embodiments, one or more cameras or other types of sensors (e.g., inertial measurement units (IMUs)) may be used as input device 1912 to receive gesture input, such as hand and / or head movements of user 104. In some embodiments, additional input device 1912 may be provided in the form of a keyboard, keypad, mouse, touchscreen, joystick, etc. In other embodiments, HMD 102 may omit a keyboard, keypad, or other similar mechanical input. Instead, HMD 102 may be implemented with a relatively simple form of input device 1912, a (wireless or wired) network interface, power, and processing / storage capacity. For example, a limited set of one or more input components (e.g., dedicated buttons for initiating configuration, power on / off, etc.) may be employed to make HMD 102 usable afterwards. In one implementation, the input device 1912 may include control mechanisms such as basic volume control buttons for increasing / decreasing volume, as well as power and reset buttons.

[0181] Output device 1914 may include display 1916, which may include one or more display panels (e.g., a stereoscopic display panel pair). Output device 1914 may also include, but is not limited to, light-emitting elements (e.g., LEDs), vibrators for generating tactile sensations, speakers (e.g., headphones), etc. Simple light-emitting elements (e.g., LEDs) may also be present to indicate status, such as, for example, when powered on.

[0182] HMD 102 may also include a wireless unit 1918 coupled to antenna 1920 to facilitate wireless connectivity with a network. Wireless unit 1918 may implement one or more of various wireless technologies, such as Wi-Fi, Bluetooth, radio frequency (RF), etc. It should be understood that HMD 102 may also include a physical port to facilitate wired connections with a network, connected peripherals (including host computer 106, such as PCs, game consoles, etc.), or plug-in network devices communicating with other wireless networks.

[0183] HMD 102 may also include an optical subsystem 1922 that uses one or more optical elements to direct light from the electronic display 1916 to the user's eyes. The optical subsystem 1922 may include different types of optical elements and combinations thereof, including but not limited to apertures, lenses (e.g., Fresnel lenses, convex lenses, concave lenses, etc.), filters, etc. In some embodiments, one or more optical elements in the optical subsystem 1922 may have one or more coatings, such as anti-reflective coatings. The amplification of image light by the optical subsystem 1922 allows the electronic display 1916 to be physically smaller, lighter, and consume less power compared to larger displays. Additionally, the amplification of image light increases the field of view (FOV) of the displayed content (e.g., images). For example, the FOV of the displayed content allows the displayed content to be presented using almost the entire user's FOV (e.g., 120 to 150 degrees diagonally) and, in some cases, the entire user's FOV. AR applications may have a narrower FOV (e.g., approximately 40 degrees FOV). The optical subsystem 1922 may be designed to correct one or more optical errors, such as, but not limited to, barrel distortion, pincushion distortion, longitudinal chromatic aberration, lateral chromatic aberration, spherical aberration, coma, image plane curvature, astigmatism, etc. In some embodiments, the content provided to the electronic display 1916 for display is pre-distorted, and the optical subsystem 1922 corrects this distortion when it receives content-based image light from the electronic display 1916.

[0184] HMD 102 may also include one or more sensors 1924, such as sensors for generating motion, position, and orientation data. These sensors 1924 may be or include gyroscopes, accelerometers, magnetometers, cameras, color sensors, or other motion, position, and orientation sensors. Sensors 1924 may also include sub-parts of the sensor, such as a series of active or passive markers that can be observed externally by a camera or color sensor to generate motion, position, and orientation data. For example, a VR headset may include multiple markers, such as reflectors or lights (e.g., infrared or visible light), on its exterior that, when observed by an external camera or illuminated by light (e.g., infrared or visible light), provide one or more reference points for software interpretation to generate motion, position, and orientation data. HMD 102 may include a light sensor sensitive to light (e.g., infrared or visible light) in the environment projected or diffused into HMD 102 by a base station.

[0185] In one example, sensor 1924 may include an inertial measurement unit (IMU) 1926. IMU 1926 may be an electronic device that generates calibration data based on measurement signals received from accelerometers, gyroscopes, magnetometers, and / or other sensors, or some combination thereof, suitable for detecting motion and correcting errors associated with IMU 1926. Based on the measurement signals, such motion-based sensors, such as IMU 1926, can generate calibration data indicating an estimated position of HMD 102 relative to its initial position. For example, multiple accelerometers may measure translational motion (forward / backward, up / down, left / right), and multiple gyroscopes may measure rotational motion (e.g., pitch, yaw, and roll). IMU 1926 may, for example, rapidly sample the measurement signals and calculate the estimated position of HMD 102 based on the sampled data. For example, the IMU 1926 can integrate the measurement signal received from the accelerometer over time to estimate the velocity vector, and integrate the velocity vector over time to determine the estimated location of a reference point on the HMD 102. The reference point is a point that can be used to describe the location of the HMD 1900. While a reference point can generally be defined as a point in space, in various embodiments, the reference point is defined as a point within the HMD 102 (e.g., the center of the IMU 1926). Alternatively, the IMU 1926 provides the sampled measurement signal to an external console (or other computing device) to determine calibration data.

[0186] The sensor 1924 can operate at a relatively high frequency to provide sensor data at a high rate. For example, sensor data can be generated at a rate of 1000 Hz (or one sensor reading per 1 millisecond). This results in one thousand readings per second. When the sensor generates so much data at this rate (or even higher), the dataset used to predict motion becomes quite large, even within a relatively short time period of about tens of milliseconds.

[0187] As mentioned, in some embodiments, sensor 1924 may include optical sensors that are sensitive to light emitted by a base station in the environment of HMD 102 to track the position and / or orientation, attitude, etc. of HMD 102 in 3D space. The calculation of position and / or orientation may be based on the timing characteristics of the light pulses and the presence or absence of light detected by sensor 1924.

[0188] HMD 102 may also include an eye-tracking module 1928. A camera or other optical sensor within HMD 102 can capture image information of the user's eyes, and eye-tracking module 1928 can use the captured information to determine interpupillary distance, interocular distance, the three-dimensional (3D) position of each eye relative to HMD 102 (e.g., for distortion correction purposes), which includes the magnitude of torsion and rotation (i.e., roll, pitch, and yaw) of each eye, and the direction of gaze. In one example, infrared light is emitted within HMD 102 and reflected from each eye. The reflected light is received or detected by a camera in HMD 102 and analyzed to extract eye rotation from changes in the infrared light reflected from each eye. Eye-tracking module 1928 can use a variety of methods for tracking the eyes of user 104. Therefore, the eye-tracking module 1928 can track up to six degrees of freedom for each eye (i.e., 3D position, roll, pitch, and yaw), and at least a subset of the quantities tracked from the user 104's eyes can be combined to estimate the gaze point (i.e., the 3D location of where the user is looking or their position in the virtual scene). For example, the eye-tracking module 1928 can integrate information from past measurements, measurements identifying the position of the user 104's head, and 3D information describing the scene presented by the electronic display 1916. Thus, information regarding the position and orientation of the user 104's eyes is used to determine the gaze point that the user 104 is looking at in the virtual scene presented by the HMD 102.

[0189] HMD 102 may also include a head tracking module 1930. The head tracking module 1930 may utilize one or more of the sensors 1924 to track the head movements (including head rotation) of user 104, as described above. For example, the head tracking module 1930 may track up to six degrees of freedom of HMD 102 (i.e., 3D position, roll, pitch, and yaw). These calculations may be performed at each of a series of frames 300, allowing an application (e.g., a video game) to determine how to render the scene in the next frame 300 and / or allowing compositor 1910 to determine how to render the reprojected frame 300 based on head position and orientation. In some embodiments, the head tracking module 1930 is configured to predict the future position and / or orientation of HMD 102 based on current and / or past data. This is because it requires the application to render frame 300 before user 104 actually sees the light on display 1916 (and therefore sees the image). Therefore, the next frame 300 can be rendered based on this future prediction of the head position and / or orientation made at an earlier point in time (such as approximately 25 to 30 milliseconds before rendering frame 300). In a distributed system where the host computer 106 is communicatively (e.g., wirelessly) coupled to the HMD 102, the future prediction of the head pose can be made 30 milliseconds or longer before the illumination time of frame 300 to account for network latency, compression operations, etc. Rotation data provided by the head tracking module 1930 can be used to determine both the rotation direction of the HMD 102 and the amount of rotation of the HMD 102 in any suitable unit of measurement. For example, the rotation direction can be simplified and output based on positive or negative horizontal directions corresponding to left, right, up, and down, and positive or negative vertical directions. The amount of rotation can be in terms of angles, radians, etc. Angular velocity can be calculated to determine the rotation rate of the HMD 102.

[0190] Figure 20 Exemplary components of a host computer 106, in which the techniques disclosed herein are implemented, are shown. In the illustrated embodiment, the host computer 106 includes one or more processors 2000 and memory 2002 (e.g., computer-readable medium 2002). In some embodiments, processor 2000 may include a CPU, GPU 304, both CPU and GPU 304, a microprocessor, a digital signal processor, or other processing units or components known in the art. Alternatively or otherwise, the functions described herein may be performed at least in part by one or more hardware logic components. Examples, but not limited to, exemplary types of hardware logic components that may be used include FPGAs, ASICs, ASSPs, SOCs, CPLDs, etc. Additionally, each processor in processor 2000 may have its own local memory, which may also store program modules, program data, and / or one or more operating systems.

[0191] Memory 2002 may include volatile and non-volatile memory, removable and non-removable media implemented using any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Such memory includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD or other optical storage devices, cassette tape, magnetic tape, disk storage devices or other magnetic storage devices, RAID storage systems, or any other medium that can be used to store desired information and is accessible by a computing device. Memory 2002 may be implemented as a CRSM, which may be any available physical medium accessible to processor 2000 to execute instructions stored on memory 2002. In one basic implementation, the CRSM may include RAM and flash memory. In other implementations, the CRSM may include, but is not limited to, ROM, EEPROM, or any other tangible medium that can be used to store desired information and is accessible by processor 2000.

[0192] Generally, host computer 106 may include logic (e.g., software, hardware, and / or firmware, etc.) configured to implement the techniques, functions, and / or operations described herein. Computer-readable medium 2002 may include various modules, such as instructions, data storage, etc., which may be configured to execute on processor 2000 to implement the techniques, functions, and / or operations described herein. Exemplary functional modules in the form of operating system 2004, video game client 2006 (including application 2008, such as a video game (or game application 2010)), and rendering component 2012 are shown stored in computer-readable medium 2002 and executable on processor 2000. In some embodiments, additional or different functional modules may be stored in computer-readable medium 2002 and executable on processor 2000.

[0193] Operating system 2004 can be configured to manage the hardware within and coupled to host computer 106 to facilitate other modules. Video game client 2006 can represent an executable client application configured to launch and execute programs such as video games (or video game programs). In other words, video game client 2006 can include game software that can be used to play video games on a system including HMD 102 and host computer 106. After installing video game client 2006, host computer 106 can then have the ability to receive (e.g., download, stream, etc.) video games from remote systems via a computer network (e.g., the Internet) and execute these video games via video game client 2006. Any type of content distribution model can be used for this purpose, such as a direct purchase model where video games can be purchased individually for download and execution on host computer 106, a subscription-based model, a content distribution model where video games are rented or leased for a period of time, etc. Therefore, host computer 106 can include one or more video games within video game library 2014. These video games can be retrieved and executed by loading video game client 2006. In one example, user 104 can select to play one of several video games they have purchased and downloaded to the video game library 2014 by loading the video game client 2006 and selecting a video game to begin playing. The video game client 2006 allows users to log in to the video game service using credentials such as user account, password, etc.

[0194] Application 2008, executing on host computer 106, may be a graphics-based application 2008 (e.g., a video game 2010). Application 2008 is configured to generate pixel data for a series of frames, which are ultimately used to render corresponding images on display panel 1916 of HMD 102. During runtime, for a given frame, rendering component 2012 can determine a predicted "illumination time" for that frame. The predicted "illumination time" represents the time it will take for the light-emitting elements of display panel 1916 of HMD 102 to illuminate that frame. This prediction may take into account the inherent latency of the wireless communication link between host computer 106 and HMD 102, as well as the predicted rendering time and / or the known scan output time of pixels from the frame buffer. In other words, the prediction for the wireless communication link may differ from the prediction for the wired communication link. For example, for a wired communication link, the rendering unit 2012 can predict the illumination time in the first time period in the future (e.g., about 22 milliseconds in the future), while for a wireless communication link, due to the inherent difference in latency when transmitting data between a wired connection and a wireless connection, the rendering unit 2012 can predict the illumination time in the second larger time period in the future (e.g., about 44 milliseconds in the future).

[0195] As described herein, host computer 106 may also receive head tracking data 208 generated by head tracking module 1930 of HMD 102 from HMD 102. This head tracking data 208 may be generated and / or transmitted at any suitable frequency, such as a frequency corresponding to the target frame rate and / or refresh rate of HMD 102, or a different (e.g., faster) frequency, such as 1000 Hz (or one sensor read per 1 millisecond). Rendering unit 2012 is configured to determine, at least in part, the predicted pose in which HMD 102 will be at the predicted illumination time based on the head tracking data 208. Rendering unit 2012 may then provide pose data indicative of the predicted pose to executing application 2008 for rendering the frame based on the predicted pose (e.g., generating pixel data for the frame), and rendering unit 2012 may obtain pixel data 110 (1) associated with frame 300 from application 2008.

[0196] The host computer 106 may also include a communication interface 2016, which includes, but is not limited to, a wireless unit 2018 coupled to an antenna 2020 to facilitate wireless connectivity with a network and / or a second device (such as HMD 102). The wireless unit 2018 may implement one or more of various wireless technologies, such as Wi-Fi, Bluetooth, radio frequency (RF), etc. It should be understood that the host computer 106 may also include physical ports to facilitate wired connectivity with a network and / or a second device (such as HMD 102).

[0197] Generally, the host computer 106 has input devices 2022 and output devices 2024. Input devices 2022 may be a keyboard, keypad, mouse, touchscreen, joystick, control buttons, microphone, camera, etc. Output devices 2024 may include, but are not limited to, a display, light-emitting elements (e.g., LEDs), vibrators that generate tactile sensations, speakers (e.g., headphones), etc. It should be understood that in some embodiments, a subset of components shown as implemented on the HMD 102 may be implemented on the host computer 106 or on another computing device separate from the HMD 102, and / or a subset of components shown as implemented on the host computer 106 may be implemented on the HMD 102 or on another computing device separate from the host computer 106.

[0198] Although the subject matter has been described in language specific to structural features, it should be understood that the subject matter defined in the appended claims is not limited to the specific features described. Rather, specific features are disclosed as exemplary forms for implementing the claims.

Claims

1. A head-mounted display (HMD), comprising: One or more display panels; processor; and The memory stores computer-executable instructions that, when executed by the processor, cause the HMD to: A host computer communicatively coupled to the HMD receives a motion vector array generated based on first pixel data and second pixel data, the first pixel data being associated with a first frame and the second pixel data being associated with a second frame rendered after the first frame, wherein the motion vector array represents compressed pixel data generated as a result of compressing the second pixel data associated with the second frame; The compressed pixel data is decompressed to obtain decompressed pixel data associated with the second frame; A third pixel data is generated from the decompressed pixel data, wherein the third pixel data is associated with a third reprojected frame that takes into account the rotation of the HMD since the second frame was rendered. A rendering mesh is generated for modifying the third pixel data of the third reprojected frame, wherein the rendering mesh comprises a checkerboard grid having geometry and vertices, the vertices or the geometry being mapped to pixels of the third pixel data; At least in part, based on the non-zero motion vectors in the motion vector array, the vertices of the rendering mesh are moved to different positions within the rendering mesh as moved vertices; The third pixel data is modified to obtain modified third pixel data, wherein the modification of the third pixel data is based at least in part on the following: The moved vertex, wherein modifying the third pixel data at least in part based on the moved vertex includes: moving the pixel value of the third pixel data according to the moved vertex of the rendering mesh to obtain the modified third pixel data; and The image is rendered on one or more display panels based at least in part on the modified third pixel data.

2. The HMD according to claim 1, wherein the vertices: In the direction of the non-zero motion vector The quantity shift corresponding to the magnitude of the non-zero motion vector.

3. The HMD of claim 1, wherein the computer-executable instructions, when executed by the processor, further cause the HMD to extract the motion vector array from the encoded data stream.

4. The HMD according to claim 1, wherein: The host computer is wirelessly coupled to the HMD; and The motion vector array is received wirelessly from the host computer.

5. The HMD of claim 1, wherein the computer-executable instructions, when executed by the processor, further cause the HMD to: At least one filter is applied to the motion vector array to obtain a modified motion vector array. The at least one filter includes at least one of a scalar median filter for noise reduction, or a mean-maximum blur filter for eliminating discontinuities in sub-regions of a motion vector group. Modifying the third pixel data includes modifying the third pixel data at least in part based on the modified motion vector array.

6. The HMD of claim 1, wherein the computer-executable instructions, when executed by the processor, further cause the HMD to: before rendering the image. Depth data is received from the host computer, the depth data representing Z-buffer data output by an application running on the host computer. in, The modification of the third pixel data is also based at least in part on the depth data.

7. The HMD of claim 1, wherein the computer-executable instructions, when executed by the processor, further cause the HMD to modify the third pixel data before: The decompressed pixel data is cached in the memory as cached pixel data; The cached pixel data represents the most recently decompressed pixel data available to the HMD; and Retrieve decompressed pixel data from the cache from the memory.

8. A method for presenting an image, comprising: The head-mounted display (HMD) receives a motion vector array generated from a host computer based on first pixel data and second pixel data, the first pixel data being associated with a first frame and the second pixel data being associated with a second frame rendered after the first frame, wherein the motion vector array represents compressed pixel data generated as a result of compressing the second pixel data associated with the second frame; The compressed pixel data is decompressed by the HMD to obtain decompressed pixel data associated with the second frame; A third pixel data is generated from the decompressed pixel data, wherein the third pixel data is associated with a third reprojected frame that takes into account the rotation of the HMD since the second frame was rendered. The HMD generates a rendering mesh for modifying the third pixel data of the third reprojected frame, wherein the rendering mesh comprises a checkerboard grid having geometry and vertices, the vertices or the geometry being mapped to pixels of the third pixel data; At least in part, based on the non-zero motion vectors in the motion vector array, the vertices of the rendering mesh are moved to different positions within the rendering mesh as moved vertices; The HMD modifies the third pixel data at least in part based on the moved vertices to obtain modified third pixel data, wherein modifying the third pixel data at least in part based on the moved vertices includes: moving the pixel values ​​of the third pixel data according to the moved vertices of the rendering mesh to obtain the modified third pixel data; and Images are rendered on one or more display panels of the HMD based at least in part on the modified third pixel data.

9. The method of claim 8, further comprising, before modifying the third pixel data: The decompressed pixel data is cached in the memory of the HMD as cached pixel data; The cached pixel data represents the most recently decompressed pixel data available to the HMD; as well as Retrieve cached, decompressed pixel data from the memory of the HMD.

10. The method of claim 8, further comprising extracting the motion vector array from the encoded data stream.

11. The method of claim 8, further comprising, before presenting the image: The HMD receives pose data from the host computer indicating the predicted pose of the HMD, and this pose data is used by an application running on the host computer to generate the second pixel data associated with the second frame. The modification of the third pixel data is also based, at least in part, on a comparison between the predicted pose and the updated pose prediction of the HMD, which is performed while the HMD is preparing to render the third reprojected frame as the next frame.

12. The method of claim 8, further comprising, before presenting the image: The HMD receives depth data from the host computer, the depth data representing Z-buffer data output by an application running on the host computer. The modification of the third pixel data is also based at least in part on the depth data.

13. The method of claim 8, further comprising: The HMD applies at least one filter to the motion vector array to obtain a modified motion vector array. The at least one filter includes at least one of a scalar median filter for noise reduction, or a mean-maximum blur filter for eliminating discontinuities in sub-regions of a motion vector group; The modification of the third pixel data includes modifying the third pixel data based on the modified motion vector array.

14. A display system, comprising: One or more display panels; processor; and The memory stores computer-executable instructions that, when executed by the processor, cause the display system to: The host computer receives a motion vector array generated based on first pixel data and second pixel data, the first pixel data being associated with a first frame and the second pixel data being associated with a second frame rendered after the first frame, wherein the motion vector array represents compressed pixel data generated as a result of compressing the second pixel data associated with the second frame; The compressed pixel data is decompressed to obtain decompressed pixel data associated with the second frame; A third pixel data is generated from the decompressed pixel data, wherein the third pixel data is associated with a third reprojected frame that takes into account the rotation of the display system since the second frame was rendered. A rendering mesh is generated for modifying the third pixel data of the third reprojected frame, wherein the rendering mesh comprises a checkerboard grid having geometry and vertices, the vertices or the geometry being mapped to pixels of the third pixel data; At least in part, based on the non-zero motion vectors in the motion vector array, the vertices of the rendering mesh are moved to different positions within the rendering mesh as moved vertices; Modifying the third pixel data at least partially based on the moved vertices to obtain modified third pixel data, wherein modifying the third pixel data at least partially based on the moved vertices includes: moving the pixel values ​​of the third pixel data according to the moved vertices of the rendered mesh to obtain the modified third pixel data; and The image is rendered on one or more display panels based at least in part on the modified third pixel data.

15. The display system of claim 14, wherein the computer-executable instructions, when executed by the processor, cause the display system to: before modifying the third pixel data: The decompressed pixel data is cached in the memory as cached pixel data; The cached pixel data represents the most recently decompressed pixel data available to the display system; and Retrieve decompressed pixel data from the cache from the memory.

16. The display system according to claim 14, wherein: The host computer is wirelessly coupled to the display system; and The motion vector array is received wirelessly from the host computer.

17. The display system of claim 14, wherein the computer-executable instructions, when executed by the processor, cause the display system to: before presenting the image. Depth data is received from the host computer, the depth data representing Z-buffer data output by an application running on the host computer. The modification of the third pixel data is also based at least in part on the depth data.

18. The display system of claim 14, wherein the computer-executable instructions, when executed by the processor, cause the display system to: At least one filter is applied to the motion vector array to obtain a modified motion vector array. The at least one filter includes at least one of a scalar median filter for noise reduction, or a mean-maximum blur filter for eliminating discontinuities in sub-regions of a motion vector group; Modifying the third pixel data includes modifying the third pixel data based on the modified motion vector array.

19. The display system of claim 14, wherein the display system comprises at least one of a virtual reality (VR) headset or an augmented reality (AR) headset.

20. The display system of claim 14, wherein the computer-executable instructions, when executed by the processor, further cause the display system to extract the motion vector array from the encoded data stream.

Images