Motion smoothing in distributed systems.

Abstract

Described herein are motion smoothing techniques for displays or display systems, such as head-mounted displays, to account for the motion of moving or moving objects in a manner that reduces judder. The display system may be separate from, but communicatively coupled to, a host computer to which a graphics-based application, such as a video game, is outputting frames for rendering on the display system. The host computer may generate motion vectors that represent compressed pixel data for transmission to the display system. The motion vectors may be used by the display system to modify the pixel data of the frames. The modified pixel data of the frames is "motion smoothed" for rendering on the display system in a manner that reduces the judder of moving or moving objects.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a PCT application claiming priority to co-pending and commonly assigned U.S. patent application Ser. No. 16 / 792,080, entitled “MOTION SMOOTHING IN A DISTRIBUTED SYSTEM,” filed Feb. 14, 2020, which is hereby incorporated by reference in its entirety. [Background technology]

[0002] Virtual reality (VR) systems are used both inside and outside the video game industry. VR systems may be set up as all-in-one (or standalone) VR headsets or as distributed systems. In a distributed setup, the VR headset may be physically tethered or wirelessly connected to a host computer. In this distributed setup, the host computer typically runs a graphics-based application, such as a video game, that outputs frames, and the VR headset displays the frames as they are streamed from the host computer. This type of setup leverages the high computing power of the host computer to display high-quality images to a lightweight VR headset that acts like a "thin client" device. Additionally, displays for VR systems, such as those built into the VR headset, typically operate at a minimum refresh rate that is suitable for VR applications. For example, 90 Hertz (Hz) is a common refresh rate for VR displays. In a "live rendering" scenario, the graphics-based application outputs frames for rendering at a frame rate that matches the refresh rate of the display. In this scenario, assuming frames are being transferred from the host computer to the VR headset at a sufficient data transfer rate, each time the screen is refreshed, a new frame (referred to herein as an "actual frame") output by the application may be displayed. Such live rendering scenarios are often referred to as an application "hitting frame rate."

[0003] In practice, applications do not always reach the frame rate for various reasons. For example, an application may intermittently drop frames and / or the application may temporarily output frames at a slower rate (e.g., 45 frames per second when the ideal frame rate is 90 frames per second). Furthermore, in distributed systems, network congestion may result in latency in the rate at which data is transferred from the host computer to the VR headset. In these situations, a technique called "rotation-only reprojection" may be used to replace missing frames with reprojected frames that take into account the rotation of the user's head, which makes it appear to the user as if there were no missing frames. For example, without reprojection, a lack of frame rate from the application or frames arriving late to the VR headset may result in stuttering or hitching in the game. In VR applications where the user is fully immersed in the virtual environment, if frames are missed and there is no reprojection to compensate for the missing frames, the user may become uncomfortable. Thus, reprojection is a technique that allows for a better user experience when frames are missed. Consider an example where an application is outputting frames at half the ideal frame rate (e.g., 45 frames per second when the ideal frame rate is 90 frames per second). In this example, every other frame can be reprojected by using the pixel data of the last rendered actual frame to create a reprojected frame that transforms the scene (e.g., through rotation and reprojection calculations) and matches the reprojected scene to the user's current head orientation. This makes the scene appear to the user to be moving in an expected manner taking into account the user's head rotation, even when reprojected frames are used to compensate for missing frames.

[0004] While rotation-only reprojection prevents stutter or hitches in the game, it creates undesirable visual artifacts of its own during head rotation, at least in VR systems using low-persistence displays (e.g., when the display is lit for a small fraction of the frame time). For example, rotation-only reprojection takes head rotation into account, but does not account for virtual objects that move or move in the scene between frames. This can create an undesirable visual artifact called "judder" for moving or moving objects. Judder causes the user to perceive a "double ghosting effect" in which a moving object (e.g., a bullet or ball moving on the screen) appears to bounce between two positions (or away from itself) between frames. Thus, any object that is moving or moving arbitrarily in the scene will judder when the user rotates their head while reprojection is in use.

[0005] Provided herein are technical solutions for improving and enhancing these and other systems. [Brief description of the drawings]

[0006] The detailed description will now be described with reference to the accompanying drawings, in which: In each figure, the leftmost digit(s) of a reference number identifies the figure in which the reference number first appears. Use of the same reference number in different figures indicates similar or identical components or features.

[0007] [Figure 1] FIG. 1 illustrates a flow diagram of an exemplary process for rendering frames on a head-mounted display (HMD) when the frames are output by an application executing on a host computer communicatively coupled to the HMD, according to embodiments disclosed herein. FIG. 1 illustrates a motion smoothing technique that may be implemented during reprojection to account for moving or moving objects in the scene. [Diagram 2]1A-1C are diagrams illustrating two exemplary timelines showing the rendering workload of a host computer and an HMD, respectively, according to embodiments disclosed herein. [Diagram 3] FIG. 13 illustrates an example motion smoothing technique to account for moving or moving objects in a scene when reprojection is being used to render a frame on a display such as an HMD. [Figure 4] FIG. 1 illustrates an example rendering mesh and how it can be used for motion smoothing of reprojected frames. [Diagram 5] FIG. 1 is a flow diagram of an example process for generating a reprojected frame using motion vectors generated by a graphics processing unit (GPU) as part of a motion smoothing technique, according to embodiments disclosed herein. [Figure 6] FIG. 1 is a flow diagram of an example process for aligning pixel data of a previously rendered frame before the pixel data is input to a GPU for motion vector estimation, according to embodiments disclosed herein. [Figure 7] FIG. 1 is a flow diagram of an example process for excluding some pixel data of a previously rendered frame and providing a remaining portion of the pixel data as input to a GPU for motion vector estimation, according to embodiments disclosed herein. [Figure 8] FIG. 10 is a flow diagram of an example process for aligning pixel data of a previously rendered frame that excludes a panel mask portion of the pixel data of the previously rendered frame and provides a remaining portion of the pixel data as input to a GPU for motion vector estimation, in accordance with an embodiment disclosed herein. [Figure 9] FIG. 1 is a flow diagram of an example process for thresholding motion vectors output from a GPU before they are used to generate a reprojected frame as part of a motion smoothing technique, according to embodiments disclosed herein. [Figure 10] FIG. 1 is a flow diagram of an example process for damping motion vectors output from a GPU before they are used to generate a reprojected frame as part of a motion smoothing technique, according to embodiments disclosed herein. [Figure 11] FIG. 1 is a flow diagram of an example process for generating a attenuation texture, according to embodiments disclosed herein, which may be used to attenuate motion vectors output from a GPU before being used to generate a reprojected frame as part of a motion smoothing technique. [Figure 12] FIG. 1 is a flow diagram of an example process for zeroing motion vectors corresponding to areas of little or no color change before the resulting set of motion vectors is used to generate a reprojection frame as part of a motion smoothing technique, according to embodiments disclosed herein. [Figure 13] FIG. 1 is a flow diagram of an example process for "cleaning up" a motion vector field using one or more filters before the resulting set of motion vectors is used to generate a reprojection frame as part of a motion smoothing technique, according to embodiments disclosed herein. [Figure 14] FIG. 1 is a flow diagram of an example process for rotating a previously rendered frame before pixel data is input to a GPU for motion vector estimation, according to embodiments disclosed herein. [Figure 15] FIG. 1 is a flow diagram of an example process for selecting between motion vector arrays generated based on luma and chroma data of a previously rendered frame, according to embodiments disclosed herein. [Figure 16] FIG. 1 is a flow diagram of an example process for obtaining multiple arrays of motion vectors, determining differences between the arrays, and generating a final motion vector array based on the determined differences for motion smoothing, in accordance with embodiments disclosed herein. [Figure 17]4 is a flow diagram of an example process for obtaining multiple motion vector arrays at different resolutions for different portions of an image region, according to embodiments disclosed herein. [Figure 18A] 1 illustrates two alternative setups of a system utilizing an HMD and a host computer according to embodiments disclosed herein. [Figure 18B] 1 illustrates two alternative setups of a system utilizing an HMD and a host computer according to embodiments disclosed herein. [Figure 19] 1 illustrates example components of a wearable device, such as a VR headset, in which the technology disclosed herein may be implemented. [Figure 20] 1 illustrates exemplary components of a host computer in which the techniques disclosed herein may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Described herein, among other things, are motion smoothing techniques for taking into account objects moving or in motion in a scene when rendering frames on a display of a distributed system. The motion smoothing techniques described herein mitigate undesirable visual artifacts related to moving or in motion objects, such as the aforementioned judder artifacts of moving or in motion objects. A head mounted display (HMD) is an exemplary type of display or display system that can implement the disclosed motion smoothing techniques when rendering images on a display. The HMD may be worn by a user for the purpose of immersing the user in a virtual reality (VR) or augmented reality (AR) environment. A display or display system such as an HMD may be separate from, but still communicatively coupled to, a host computer. An application (e.g., a video game) runs on the host computer and generates pixel data for each individual frame of a sequence of frames. The pixel data is transmitted to the HMD for each frame, and one or more display panels of the HMD render an image based on the received pixel data. These images are viewed by the user through optics included in the HMD, causing the user to perceive the images as if the user were immersed in the VR or AR environment.

[0009] As described, display systems such as HMDs can utilize a technique called "reprojection" to compensate for applications that are not reaching their frame rate. For example, reprojected frames can be rendered between real frames to achieve the ideal frame rate, with each reprojected frame being generated using pixel data from a recently rendered real frame (e.g., the last rendered real frame) output by an application running on a host computer. In a reprojected frame, the scene rendered in the previous real frame is transformed (e.g., through rotation and reprojection calculations) in a manner that accounts for the rotation of the user's head.

[0010] Described herein are motion smoothing techniques that can be used in a distributed system to modify pixel data of frames in a manner that further accounts for the motion of moving or moving objects between frames. The motion smoothing techniques described herein use computer vision algorithms to estimate the motion (e.g., direction and magnitude) of objects across multiple frames in the form of motion vectors. For example, the motion vector array may be generated by a host computer as a result (e.g., as a by-product) of compressing pixel data before the pixel data is transmitted to a display system such as an HMD. Thus, the motion vector array may represent compressed pixel data for at least some frames in a sequence of frames. In an example, a video encoder (e.g., a video encoding chip) of a graphics processing unit (GPU) on a host computer may analyze pixel data of multiple previously rendered frames to generate a motion vector array that represents compressed pixel data of a given frame. The motion vectors may be transmitted to an HMD, and in addition to using the motion vectors to obtain decompressed pixel data, the motion vectors may also be used by the HMD to modify the decompressed pixel data in a manner that accounts for moving or moving objects. In other words, the motion vectors received from the host computer can be used on the HMD to extrapolate (from the object's movement in previously rendered frames) where an object should be located in the frame to be rendered (e.g., reprojected) so that judder of the moving object is reduced.

[0011] In an exemplary motion smoothing process, the associated pixel data may be compressed on a host computer. A video encoder of a graphics processing unit (GPU) on the host computer may be used to compress the pixel data. As a result of compressing the pixel data, the video encoder may generate a motion vector array representing the compressed pixel data. The host computer may transmit the motion vector array to a display system such as an HMD. In some embodiments, the host computer may be wirelessly coupled to the HMD, although wired connections are also contemplated herein. The HMD may receive the motion vector array from the host computer. The HMD may decompress the compressed first pixel data to obtain pixel data (or an approximation of the pixel data using a decompression or decoding algorithm). The HMD may further modify the pixel data based at least in part on the motion vector array received from the host computer to obtain modified pixel data. This modified pixel data is "motion smoothed" to reduce, if not eliminate, any judder associated with moving or moving objects in the scene. A motion smoothed frame may then be rendered by presenting an image on a display panel of the HMD based at least in part on the modified pixel data.

[0012] The motion smoothing techniques described herein provide a more realistic, higher fidelity viewing experience where objects move or move within a scene in an expected manner, even when frames are dropped. As previously mentioned, frames may be dropped if an application fails to reach the frame rate, network congestion occurs, or other possible reasons. Thus, the motion smoothing techniques described herein compensate for these occurrences by maintaining a high fidelity user experience in a distributed system where frames may be dropped from time to time. The motion smoothing techniques also compensate for the inherent latency in transmitting frames, even in live rendering scenarios when frames are not dropped (i.e., when frames are received in a timely manner at the HMD). Furthermore, in wireless implementations, the techniques described herein enable motion smoothing correction in wireless HMDs without the need to add more components to the HMD, thereby enabling lightweight headsets that do not get too hot and are easy to wear, compared to many all-in-one (or standalone) headsets on the market today. Furthermore, the techniques and systems described herein can leverage already generated motion vectors to compress and transmit data from the host computer to the HMD, meaning there are no additional transportation costs when using motion vectors for motion smoothing of reprojected frames on the HMD.

[0013] It should be understood that the modified second pixel data is output to a frame buffer on the HMD after outputting the first pixel data to the frame buffer, so that the motion vector array received from the host computer can be used by the HMD to extrapolate to future frames (e.g., reprojected frames). Thus, in the example described herein, the first frame is rendered before the reprojected frame. It should be understood that this extrapolation technique can be contrasted with interpolation between frames, and that the techniques and systems described herein relate to using motion vectors received from the host computer to extrapolate to future frames, rather than using motion vectors to interpolate between frames.

[0014] Also disclosed herein are systems configured to implement the techniques and processes disclosed herein, such as systems including display systems (e.g., HMDs), as well as non-transitory computer-readable media storing computer-executable instructions for implementing the techniques and processes disclosed herein. Although the techniques and systems disclosed herein are discussed in the context of video game applications, particularly VR game applications, by way of example, it should be understood that the techniques and systems described herein may provide benefits to other applications, including, but not limited to, non-VR applications (e.g., AR applications), and / or non-game applications, such as industrial machinery applications, defense applications, robotics applications, and the like. Furthermore, while an HMD is provided as an example of a display system for displaying images, it should be understood that other types of display systems may benefit from the motion smoothing techniques described herein, such as handheld display devices that stream video content from a host computer, relatively large wall-mounted display systems, or billboard display systems.

[0015] The processes described herein are illustrated as a collection of blocks in a logic flow graph, which represent a sequence of operations that can be implemented in hardware, software, firmware, or combinations thereof (i.e., logic). In the context of software, the blocks represent computer-executable instructions that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any some of the described blocks may be combined in any order and / or in parallel to implement a process.

[0016] 1 is a flow diagram of an example process 100 for rendering frames on a head mounted display (HMD) 102 (worn by a user 104), where the frames are output by an application running on a host computer 106 that is communicatively coupled to the HMD 102, according to embodiments disclosed herein. FIG 1 illustrates a motion smoothing technique that may be implemented during reprojection to account for moving or moving objects in the scene.

[0017] 1, an HMD 102 is shown being worn by a user 104, and a host computer 106 is shown communicatively coupled to the HMD 102. For example, the host computer 106 can be implemented as any type and / or number of computing devices, including, but not limited to, a personal computer (PC), a laptop computer, a desktop computer, a portable digital assistant (PDA), a mobile phone, a tablet computer, a set-top box, a game console, a server computer, a wearable computer (e.g., a smart watch, etc.), or any other electronic device capable of transmitting and receiving data.

[0018] The host computer 106 and the HMD 102 collectively represent a distributed system for executing applications (e.g., video games) and rendering associated images on a display. In some embodiments, the host computer 106 may be collocated in the same environment as the HMD 102, such as in the household of a user 104 wearing the HMD 102. Alternatively, the host computer 106 may be located remotely with respect to the HMD 102, such as a host computer 106 in the form of a server computer located in a remote geographic location with respect to the geographic location of the HMD 102. In a remote host computer 106 implementation, the host computer 106 may be communicatively coupled to the HMD 102 via a wide area network, such as the Internet. In a local host computer 106 implementation, the host computer 106 may be collocated in an environment (e.g., a household) with the HMD 102, whereby the host computer 106 and the HMD 102 may be communicatively coupled together through a local area network (LAN), either directly or via an intermediary network device.

[0019] By being communicatively coupled together, the HMD 102 and the host computer 106 are configured to work cooperatively together to render a given frame by generating pixel data used to present a corresponding image on a display panel of the HMD 102. The host computer 106 and the HMD 102 may be communicatively coupled together via wireless and / or wired connections. For example, the devices 102 / 106 may exchange data using Wi-Fi, Bluetooth, radio frequency (RF), and / or any other suitable wireless protocol. Additionally or alternatively, the devices 102 / 106 may include one or more physical ports to facilitate a wired connection for data transfer therebetween.

[0020] Although the HMD 102 is presented herein as an exemplary "display system" that may implement the disclosed motion smoothing techniques, it should be understood that other types and / or implementations of a "display system" may implement the motion smoothing techniques described herein. Thus, it should be understood that the HMD is merely an exemplary type of display or display system for implementing the techniques described herein, but any reference to an "HMD" herein may be replaced with the term "display" or "display system." In some examples, the HMD 102 may represent a VR headset for use in a VR system, such as for use in a VR gaming system. However, the HMD 102 may additionally or alternatively be implemented as an AR headset for use in an AR application, or a headset usable in non-gaming related (e.g., industrial applications) VR and / or AR applications. In AR, the user 104 sees virtual objects superimposed on a real-world environment, whereas in VR, the user 104 typically does not see the real-world environment, but is fully immersed in the virtual environment as perceived through the display panel and optics (e.g., lenses) of the HMD 102. It should be understood that in some VR systems, a pass-through image of the user's 104 real-world environment may be displayed in conjunction with the virtual image to create an augmented VR environment within the VR system, whereby the VR environment is augmented with real-world imagery (e.g., overlaid on the virtual world). Although the examples described herein relate primarily to a VR-based HMD 102, it should be understood that the HMD 102 is not limited to implementation in VR applications.

[0021] Further, the HMD 102 may include a single display panel or multiple display panels, such as a left and right display panel of a stereo pair of display panels. One or more display panels of the HMD 102 may be used to present a sequence of image frames (referred to herein as "frames") that are viewable by a user 104 wearing the HMD 102. It should be understood that the HMD 102 may include any number of display panels (e.g., more than two display panels, a pair of display panels, or a single display panel). Thus, the term "display panel," as used herein in the singular, may refer to either display panel of a pair of display panels of a two-panel HMD 102, or a single display panel of an HMD 102 having any number of display panels (e.g., a single-panel HMD 102 or a multi-panel HMD 102). In a two-panel HMD 102, the stereo frame buffer may, for example, render 2160x1200 pixels (e.g., 1080x1200 pixels per display panel) on both display panels of the HMD 102.

[0022] Additionally, the display panel of the HMD 102 may utilize any suitable type of display technology, such as an emissive display that utilizes light emitting elements (e.g., light emitting diodes (LEDs)) to emit light during the presentation of frames on the display panel. By way of example, the display panel of the HMD 102 may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an inorganic light emitting diode (ILED) display, or any other suitable type of display technology for HMD applications.

[0023] The display panel of the HMD 102 may operate at any suitable refresh rate, such as a 90 Hertz (Hz) refresh rate, which may be a fixed refresh rate or a variable refresh rate that changes dynamically over a range of refresh rates. The "refresh rate" of a display is the number of times per second that the display redraws the screen. The number of frames displayed per second may be limited by the refresh rate of the display if the refresh rate is fixed. Thus, a series of frames may be processed (e.g., rendered) and displayed as an image on the display, such that one frame of the series is displayed with each screen update. That is, to present a series of images on the HMD 102, the display panel of the HMD 102 may transition between frames at the refresh rate of the display within the series of frames, lighting up pixels with each screen refresh. In some embodiments, the frame rate may be throttled and / or the application may be unable to reach the target frame rate and / or network congestion may introduce latency into the data transmission. In these scenarios, reprojected frames (sometimes referred to herein as "phantom frames") may be inserted between application-rendered frames (sometimes referred to herein as "actual frames").

[0024] In general, the application executing on the host computer 106 may be a graphics-based application (e.g., a video game). The application is configured to output a series of frames that may be used to present images on a display panel of the HMD 102. For example, the application may generate pixel data for the series of frames, which is used to present corresponding images on the display panel of the HMD 102. In some embodiments, off-screen rendering is utilized such that the frames may be rendered to a target before being rendered on the display panel of the HMD 100. Thus, "rendering," as used herein, may include rendering to a target other than the display and before rendering to the display itself and / or rendering to the display (e.g., without or after off-screen rendering to a different target).

[0025] Referring now to process 100, at 112, logic (e.g., software, hardware, and / or firmware, etc.) of host computer 106 may obtain (or receive) pixel data for a first frame of a series of frames from an application running on host computer 106. The pixel data may include pixel values ​​for individual pixels in a pixel array of a display panel of HMD 102. The pixel data obtained from the running application may in some embodiments include a two-dimensional array of per-pixel values ​​(e.g., color values). In some embodiments, the pixel data further includes additional data or metadata, such as depth values, luminance values, etc. In some embodiments, the pixel data may include data for each pixel represented by a single set of color and alpha values ​​(e.g., one color value for a red channel, one color value for a green channel, one color value for a blue channel, and one or more values ​​for one or more alpha channels).

[0026] In some embodiments, the host computer 106 may first receive head tracking data from the HMD 102, which may be used to determine a predicted pose of the HMD 102 for generating pixel data for a frame. That is, the host computer 106 may use the head tracking data received from the HMD 102 to generate pose data indicative of a predicted pose the HMD 102 will be in at a time when light emitting elements of a display panel of the HMD 102 are illuminated for a frame to be rendered by an application. For example, the head tracking system of the HMD 102 may be configured to track up to six degrees of freedom (e.g., three-dimensional (3D) position, roll, pitch, and yaw) of the HMD 102, which may be transmitted as head tracking data to the host computer 106 to determine a predicted pose of the HMD 102 (e.g., taking into account predicted head movement due to a future pose of the HMD 102). Thus, the pose data indicative of the predicted pose may be provided as an input to an application for rendering a frame, and the application may output pixel data based on the pose data. For example, an application may call a function to receive pose data, and the requested pose data (predicted to the target lighting time of the frame) may be provided to the application so that the application can render the frame in accordance with the pose data, which corresponds to the pose of a virtual camera used to render the scene.

[0027] In some embodiments, the application may be further instructed to generate depth data (e.g., Z-buffer data) for the frame and / or extra pixel data (sometimes referred to herein as “out-of-bounds pixel data” or “additional pixel data”), in response to which, at block 112, logic in the host computer 106 may obtain the depth data and / or extra pixel data associated with the frame from the application in addition to the pixel data described above. Depth data, such as from a depth buffer (or Z-buffer) output by the application, may indicate occluded objects in the scene. Thus, the depth data may be used, among other things, to adjust the parallax of objects in the scene (e.g., a ship far away in world space may not move with head movement as much as a close-up object moves with the same head movement). Knowing the depth of pixels corresponding to virtual objects in the scene is useful in knowing how to adjust such parallax during reprojection onto the HMD 102. Knowing the depth information also allows for warping the scene not only based on the rotation of the HMD 102, but also based on the translation of the HMD 102 in space. Without any depth data, the system may assume an average depth of about 2 meters. However, any suitable depth resolution that provides a better estimate of the average depth in the scene may be beneficial in performing reprojection on the HMD 102, and the higher the depth resolution that can be contained in the available bandwidth, the better the reprojection adjustments can be made on the HMD 102. However, there may be advantages, such as lower latency, in transmitting lower resolution depth data. As long as the resolution of the depth data is suitable to improve the reprojection adjustments on the HMD 102, it may be beneficial to include the depth data.

[0028] The extra pixel data output by the application may include extra pixel values ​​outside the boundaries of the array of pixels of the display panel of the HMD 102. For example, if the display panel of the HMD 102 has an array of 2160×1200 pixels, the pixel data acquired in block 112 may correspond to pixel values ​​within the 2160×1200 array of pixels, while the extra pixel data may correspond to pixels that are outside the boundaries of the 2160×1200 array. Thus, the pixel data and the extra pixel data may constitute a greater number of pixel values, such as a larger array of 2400×1400 pixels. The ability to render with a view larger than the target may allow for the application of a panel mask around the periphery of the display area without any effective reduction in the field of view (FOV).

[0029] At 114, logic on the host computer 106 may compress the pixel data obtained from the application at block 112. An exemplary reason for compressing the pixel data at block 114 is due to bandwidth limitations of the data connection between the host computer 106 and the HMD 102. For example, there is often not enough bandwidth to transmit uncompressed pixel data (e.g., a file) in the amount of time allotted to display frames at a desired frame rate (e.g., a frame rate of 90 Hz or higher). Although compression introduces some latency into the end-to-end graphics pipeline (compared to not compressing the pixel data), this additional time to compress the pixel data may be factored into the pose prediction, and adjustments may be made on the HMD 102 to account for the latency introduced due to compression.

[0030] In some embodiments, the compression performed in block 114 involves changing the format of the pixel data. For example, the compression in block 114 may utilize a video compression standard such as High Efficiency Video Coding (HEVC) and / or an extension of HEVC, sometimes referred to as H.265 and / or MPEG-H Part 2. While HEVC is an example of a video compression standard that may be utilized in block 114, it should be understood that other suitable video compression standards may be utilized to compress the pixel data in block 114. HEVC utilizes motion estimation to compress the pixel data, with the goal of transmitting less data than uncompressed pixel data in systems where bandwidth constraints exist, and that the compressed data is sufficient to approximate the original uncompressed data at the receiving end (e.g., at the HMD 102). As part of the motion estimation utilized in the compression operation in block 114, a motion vector array may be generated, which is illustrated in sub-block 116. Generally, a "motion vector" is a two-dimensional (2D) arrow having X and Y components for direction, and a magnitude (usually represented by the length of the 2D arrow). The magnitude of the motion vector may be specified in any suitable units of measure, such as the number of pixels in both the X and Y component directions. In some examples, a video encoder (e.g., a video encoding chip) of a GPU of the host computer 106 may generate a motion vector array in sub-block 116 based on the first pixel data obtained in block 112 and based on pixel data of one or more other frames (e.g., previously rendered frames). To generate the motion vector array in sub-block 116, the video encoder may compare pixel-by-pixel values ​​(e.g., luminance values) among the pixel data of each frame that the video encoder is tasked with analyzing according to a compression algorithm.Additionally or alternatively, the video encoder may compare macroblocks (e.g., blocks of 16 pixels (i.e., 4x4 pixel macroblocks), blocks of 64 pixels (i.e., 8x8 pixel macroblocks), blocks of 4096 pixels (i.e., 64x64 pixel macroblocks, etc.)) between pixel data of each frame that the video encoder is tasked with analyzing according to the compression algorithm. The GPU of the host computer 106 may compare portions of pixel data between a pair of frames at any suitable resolution as part of the compression operation.

[0031] In many video compression algorithms, including HEVC, there may be different types of frames, such as I-frames, P-frames, and B-frames. I-frames can be self-sustaining in the sense that they contain enough data to reconstruct a frame during decompression. For this reason, I-frames contain a relatively large amount of data and are the least compressible compared to P-frames and B-frames. P-frames use data from a previous frame (e.g., a reference frame) to compress and then decompress and reconstruct the original pixel data. P-frames are relatively small compared to I-frames because they encode differences from the previous frame, such as in the form of motion vectors. The "B" in B-frames stands for bidirectional, meaning that B-frames use both the preceding and future frames to compress and then decompress and reconstruct the original pixel data. Because I-frames are relatively large, instances of transmitting I-frames may be reduced or minimized in this disclosure. However, in some scenarios, it may be beneficial to transmit I-frames, such as when a wireless connection between the host computer 106 and the HMD 102 is temporarily lost and then reestablished. In either case, the common series of frames may begin with an I-frame, followed by a series of P-frames. Compressing the P-frames in block 114 may involve encoding differences from a reference frame (e.g., an I-frame, and / or a preceding P-frame, etc.) as to where blocks of the image have moved in the new frame relative to their positions in the reference frame. These differences may be represented by motion vectors generated in sub-block 116.

[0032] Additionally, at 114, as described in more detail with reference to process 600 of FIG. 6, logic in the host computer 106 may align (or otherwise modify) the frames used for compression before pixel data for those frames are provided as input to the GPU of the host computer 106. This may be done due to movement of the HMD 102 between frames, where objects (both stationary and moving objects) may be moved between positions in the course of rendering multiple consecutive frames. By aligning one frame with the other, or vice versa, or by adjusting both frames to be aligned with each other, pixel data representing a particular stationary object in the scene may be moved to approximately the same position between the two frames such that pixel data corresponding to the stationary object is not mistaken for a moving object by the video encoder of the GPU. The alignment performed at block 114 may include a rotational reprojection correction to one or both of the frames (e.g., the pixel data of the frames) to reduce the delta (or difference) in the motion vectors due to head movement. This reduction in the motion vector deltas (or differences) may provide the additional benefit of reduced bandwidth in transmitting the motion vectors from the host computer 106 to the HMD 102, as described herein. Thus, the modified pixel data may be provided as input to the GPU for compression in block 114 when generating the motion vectors in sub-block 116.

[0033] At 118, the host computer 106 may transmit data 110 to the HMD 102, which may include compressed pixel data 110(1). As described above with respect to P frames, most of the time the compressed pixel data 110(1) may be transmitted in the form of an array of motion vectors 110(2) generated in sub-block 116. That is, for a series of P frames, the motion vector array 110(2) may represent the compressed pixel data. However, when an I frame is transmitted to the HMD 102, the compressed pixel data 110(1) may not include motion vectors. The data 110 transmitted at block 118 may further include pose data 110(3) used by an application to generate the pixel data 110(1) for the frame. In some embodiments, the data 110 transmitted in block 118 may omit pose data 110(3) and / or the data 110 may include additional data such as depth data 110(4), extra pixel data (e.g., outside the boundaries of the display panel of the HMD 102), parallax occlusion data, and / or cube map data (e.g., for sudden and large head movements such that the HMD 102 has other options than presenting dark pixels with no data). The data 110 may be transmitted in block 118 to the HMD 102 in various manners depending on the implementation, such as wirelessly, via a wired connection, via a wide area network, etc. In some embodiments, some or all of the data 110 may be transmitted in-band or out-of-band as part of an encoded data stream along with the compressed pixel data 110(1), e.g., motion vectors 110(2). For example, pose data 110(3), depth data 110(4), and / or additional data may be transmitted out-of-band from the compressed pixel data 110(1), such as motion vectors 110(2).

[0034] At 120, the HMD 102 may receive data 110 from the host computer 106, including compressed pixel data 110(1), such as, but not limited to, motion vectors 110(2) (which, when received, represent the compressed pixel data), pose data 110(3), and / or depth data 110(4).

[0035] At 122, logic (e.g., software, hardware, and / or firmware, etc.) of HMD 102 may decompress compressed pixel data 110(1), such as by using motion vector 110(2) and a reference frame (e.g., pixel data 110(1) of a previous frame) to obtain pixel data of a frame output by an application, or at least an approximation of the original pixel data. Decompressing the compressed pixel data at block 122 may use a decompression or decoding algorithm, such as an HEVC algorithm. Further, as indicated by sub-block 124, for example, if at least motion vector 110(2) represents compressed pixel data 110(1) (e.g., for a P frame), logic of HMD 102 may extract motion vector 110(2) from a data stream, such as from an encoded data stream (e.g., an HEVC stream). The decompressed pixel data 110(1), motion vectors 110(2), and possibly additional data (e.g., pose data 110(3), depth data 110(4), etc.) may be cached in memory of the HMD 102 so that the additional data may be accessed at a later time, such as for use in rendering a reprojected frame and / or in decompressing future data. It is understood that blocks 120-124 may repeat as frames are continually received from the host computer 106. It is understood that the data 110 may be maintained (e.g., cached) in the memory of the HMD 102 for a period of time, after which the data 110 may be discarded to make space for storing future data 110.

[0036] At 126, logic in the HMD 102 may retrieve the most recent decompressed data 110, such as decompressed pixel data 110(1), for use in presenting an image on a display panel of the HMD 102. For example, when data 110 is received at block 120 and decompressed at block 122, the retrieval operation at block 126 may determine that cached pixel data 110(1) stored in memory of the HMD 102 represents the most recently decompressed pixel data available to the HMD 10, and logic in the HMD 102 may retrieve from memory the most recently decompressed data, e.g., pixel data 110(1) obtained from decompressing the most recently received motion vector 110(2). In an example, if the HMD 102 is operating at a refresh rate of 90 Hz, a frame will be displayed approximately once every 11.11 milliseconds. In this context, if the HMD 102 has received and decompressed new data in the last 11.11 ms since the last frame was displayed, then the new decompressed pixel data 110(1) may be used to render the next frame. On the other hand, if the HMD 102 has not received and decompressed new data in the last 11.11 ms since the last frame was displayed, then the pixel data 110(1) of the last rendered frame may be used to render the next frame.

[0037] At 128, logic in the HMD 102 may modify pixel data for the frame based at least in part on the predicted pose of the HMD 102 to obtain modified pixel data for the frame. For example, logic in the HMD 102 may apply an adjustment to the pixel data based on a comparison between the original predicted pose of the HMD 102 used by the application to generate the pixel data retrieved in block 126 and an updated pose predicted by logic in the HMD 102 at a time closer to the illumination time of the frame. This comparison may reveal a difference (or difference) between the original pose prediction at the time the application rendered the frame and the updated pose prediction in the HMD 102 before rendering the frame on the display panel, and the adjustment applied in block 118 may include a rotation calculation to compensate for this difference (e.g., by shifting and / or rotating the pixel data in one direction or another depending on the difference between the two pose determinations).

[0038] Additionally, at 128, logic of the HMD 102 may also modify pixel data of the frame based at least in part on the array of motion vectors 110(2) extracted at block 124 and retrieved at block 126. The modification at block 128 results in modified pixel data of a reprojected frame. Although this disclosure makes a distinction between "real" frames and reprojected (or "phantom" frames), this distinction is not intended to imply that real frames are often not adjusted / modified at block 128 prior to presentation on the HMD 102. That is, the frames presented on the HMD may be considered to be synthetic (i.e., not the same as the original frames output by the application running on the host computer 106) due to the modification operations that may be performed at block 128 on both the "real" and "phantom" frames. In this sense, all frames modified at block 128 may be considered to be "reprojected" frames, as used herein. Whether and / or the extent to which pixel data is modified at block 128 depends on the operational performance of the system. For example, in a live rendering scenario where all frames output by an application are received at HMD 102 in a timely manner, if all goes as it should, the difference between the original and updated pose predictions will be close to, if not zero, in which case the corrections applied in block 128 may have little or no effect on the final output, if any. However, in a scenario where an application fails to reach a frame rate or where frames arrive late or are dropped during transmission due to network congestion, the corrections applied to pixel data 110(1) in block 128 may be significant to compensate for larger differences between the original and updated pose predictions. As described in more detail below, the correction of the pixel data of the reprojected frames based on motion vectors 110(2) may utilize a rendering mesh. For example, a rendering mesh may be generated for the reprojected frames.The rendering mesh may include a tessellation mesh having multiple vertices, and the vertices of the rendering mesh may be moved to different positions within the rendering mesh as moved vertices (e.g., by moving the vertices (i) in the direction of the non-zero motion vector 110(2) and (ii) by moving them by an amount corresponding to the magnitude of the non-zero motion vector 110(2)). Thus, modification of the pixel data in block 128 may be performed according to the moved vertices of the rendering mesh, such as by moving pixel values ​​of the pixel data obtained in block 126 to retrieve modified pixel data of the reprojection frame. For example, the pixel values ​​may be moved four pixels to the left and four pixels up to a new position within the modified pixel data according to the moved vertices in the rendering mesh. In some embodiments, the motion vector 110(2) may be modified before applying the modified motion vector 110(2) to the pixel data. For example, as described in more detail below, a filter (e.g., an N×N scalar median filter, an M×M average max blur filter, etc.) may be applied to the motion vector array 110(2) to obtain a modified array of motion vectors, and the modified (e.g., filtered) array of motion vectors may be used to modify the pixel data in block 128.

[0039] Additionally, at 128, logic in the HMD 102 may also modify pixel data of the frame based at least in part on the depth data 110(4) retrieved at block 126. The depth data 110(4) may be used at block 128 to adjust for parallax of objects in the scene, among other things (e.g., a ship far away in world space may not move with head movement as much as a close-up object moves with the same head movement). Knowing the depth of pixels corresponding to virtual objects in the scene is useful in knowing how to adjust for such parallax during reprojection onto the HMD 102. Knowing the depth information also allows for warping of the scene based on the translation of the HMD 102 in space, as well as the rotation of the HMD 102. If depth data 110(4) is not received from the host computer 106, logic in the HMD 102 may assume an average depth of about 2 meters.

[0040] At 130, an image may be presented on a display panel of the HMD 102 based at least in part on the modified pixel data. For example, logic of the HMD 102 may output the modified pixel data to a frame buffer and cause the display panel of the HMD 102 to present an image based on the modified pixel data output to the frame buffer. This may involve scanning the modified pixel data out to the display panel of the HMD 102 and illuminating light emitting elements of the display panel to illuminate pixels on the display panel. For an HMD 102 having a pair of display panels, this modified pixel data may correspond to a frame representing a pair of images to be displayed on the pair of display panels and may be output to a stereo frame buffer and scanned out accordingly. The resulting images corresponding to the reprojected frames are "motion smoothed" by modification of the pixel data according to the motion vectors 110(2).

[0041] FIG. 2 is a diagram illustrating two exemplary timelines 200(1) and 200(2) showing the respective rendering workloads of the host computer 106 and the HMD 102, according to embodiments disclosed herein. The example of FIG. 2 depicts three exemplary frames, frame "F", frame "F+1", and frame "F+2", for a first timeline 200(1) associated with the host computer 106. This first timeline 200(1) illustrates how frames may be rendered serially by an application running on the host computer 106 using the GPU of the host computer 106. Here, the application renders frame F, then frame F+1, then frame F+2, in order, from left to right on the first timeline 200(1). The oval on the first timeline 200(1) indicates that this may continue for any number of frames as the application continues to run. The first timeline 200(1) also implies, by the vertical lines oriented orthogonally to the horizontal timeline 200(1), that the application is targeting a target frame rate (e.g., a 90 Hz frame rate with the vertical lines separated by approximately 11.11 milliseconds). In the example of FIG. 2, the application running on the host computer 106 happens to reach the target frame rate on frames F and F+1, but the application does not reach the frame target frame rate for frame F+2. For example, the scene for frame F+2 may include many moving objects or complex textures, and due to these complexities and / or other reasons, the application takes longer than the allotted time to render frame F+2. The host computer 106 may receive head tracking data 208 regarding the movement of the HMD 102 to determine a predicted pose of the HMD 102 for rendering each frame on the timeline 200(1).

[0042] The second timeline 200(2) of FIG. 2 associated with the HMD 102 illustrates the rendering workload 202(a), 202(b), and 202(c) of the HMD 102 for individual frames. The individual rendering workload 202 of the HMD 102 for a given frame may represent adjustments applied to the pixel data 110(1) before the final image is presented on the display panel of the HMD 102. Such adjustments may include, but are not limited to, adjustments for geometric distortion, chromatic aberration, head movement, and the like, that are applied to the pixel data 110(1) before rendering the final image on the HMD 102. For frames F and F+1, these adjustments are applied to the pixel data generated by an application executing on the host computer 106, and at least some of these adjustments may utilize pose data 110(3) received from the host computer 106 by considering the difference between the original predicted pose of the HMD 102 and a prediction of the updated pose of the HMD 102. Thus, frames F and F+1 of the second timeline 200(2) are meant to represent modified versions of “real” frames in the sense that they are modified versions of pixel data 110(1) output from an application in real time during a live rendering situation. In contrast, to render frame F+2, the HMD 102 may use previously received pixel data 110(1) for a preceding frame (e.g., pixel data 110(1) for frame F+1) to generate a reprojected (or “phantom”) frame based on the pose prediction of the preceding frame and the updated pose prediction made by the HMD 102. Additionally, the rendering workload 202(c) may include modifying pixel data from frame F+1 based on motion vectors 110(2) received with the data 110 for frame F+1 to “motion smooth” frame F+2, as described herein. In any event, the result of the rendering workload 202 is the generation of modified pixel data that may be output to a frame buffer (e.g., a stereo frame buffer).In this specification, this distinction between "real" and "illusory" frames is not meant to imply that real frames are not scaled on the HMD 102; in this sense, any frames generated on the HMD side are effectively synthetic (i.e., not the same as the original frames output by an application running on the host computer 106).

[0043] The second timeline 200(2) of FIG. 2 also shows scan-out times 204(a), 204(b), and 204(c) for each frame, and illumination times 206(a), 206(b), and 206(c) for each frame. During the scan-out time 204 for a given frame, a subset of pixel values ​​(of the modified pixel data) are scanned out to the display panel of the HMD 102 via a display port (e.g., High-Definition Multimedia Interface (HDMI)®), and during the illumination time 206 for a given frame, light-emitting elements of the display panel of the HMD 102 are illuminated to illuminate pixels of the display panel. FIG. 2 shows an example of a global flash type display driving scheme that may be used with an LCD panel to simultaneously emit light from the light-emitting elements of the display panel at the refresh rate of the HMD 102. In an example, if the HMD 102 is operating at a refresh rate of 90 Hz, the illumination times 206 for each frame may be separated by approximately 11.11 milliseconds.

[0044] FIG. 3 illustrates an exemplary motion smoothing technique for accounting for objects that are moving or in motion in a scene when reprojection is being used to render frames on a display such as HMD 102. The example of FIG. 3 depicts three exemplary frames 300(1), 300(2), and 300(3) that are to be rendered in sequence as a series of frames 300. In the example of FIG. 3, an object 302 is shown moving leftward (i.e., from right to left) across the scene over the course of frames 300(1)-300(3). The dotted outline of object 302 in frames 300(2) and 300(3) represents where object 302 was located in the previously rendered frame 300. Here, frame 300(1) is rendered first, then frame 300(2) is rendered second, then frame 300(3) is rendered third, and so on.

[0045] At least some of the frames 300 in the series of frames 300 may be “real” frames in the sense that they are output from an application, such as a video game application, or any other type of graphics-based application, in a time sufficient to present a corresponding image on the HMD 102 based on the pixel data of the frames 300. The application may run in a graphics pipeline that outputs pixel data 110(1) to a frame buffer for rendering each frame 300.

[0046] During execution, the head tracking module of the HMD 102 may generate data about the position and orientation (pose) of the HMD 102 that is provided to the host computer 106 executing the application to inform the application as to how to render the next frame 300 in the sequence of frames 300 according to the head pose of the user 104. This allows the application to output pixel data 110(1) for rendering images on the HMD 102 in a manner that makes the user 104 believe that he is looking around a virtual environment containing objects (both stationary objects, such as the moving object 302, and moving objects). If the application is reaching the frame rate, both stationary and moving objects are perceived to move in the scene in an expected manner with the movement of the user's 104 head. The motion smoothing technique described herein is a way to compensate for applications that have not reached the frame rate so that a similar visual perception is achieved for moving objects.

[0047] In the example of Figure 3, a first frame 300(1) may represent a first "actual" frame received from an application, a second frame 300(2) may represent a second "actual" frame received from an application and rendered after the first frame 300(1), and a third frame 300(3) may represent a reprojected frame generated from pixel data 110(1)(2) associated with the second frame 300(2). Thus, the "third frame 300(3)" in Figure 3 may also be referred to herein as the "reprojected frame 300(3)." In the example of FIG. 3, to compress the second frame 300(2) (frame 2), logic of the host computer 106 may provide as input to a graphics processing unit (GPU) 304 of the host computer 106 first pixel data 110(1)(1) associated with the first frame 300(1) and second pixel data 110(1)(2) associated with the second frame 300(2).

[0048] A video encoder (e.g., a video encoding chip) of the GPU 304 may generate an array of motion vectors 110(2)(2) based on the first pixel data 110(1)(1) and the second pixel data 110(1)(2) input to the GPU 304. To generate the array of motion vectors 110(2)(2), the video encoder of the GPU 304 may compare pixel-by-pixel values ​​(e.g., luminance values) between the pixel data 110(1) of each frame 300 provided as input. Additionally or alternatively, the video encoder of the GPU 304 may compare macroblocks (e.g., blocks of 16 pixels (i.e., 4×4 pixel macroblocks), blocks of 64 pixels (i.e., 8×8 pixel macroblocks)) between the pixel data 110(1) of each frame 300 provided as input. In this manner, the GPU 304 may compare portions of pixel data 110(1) between a pair of frames 300 at any suitable resolution. In some embodiments, the input pixel data 110(1)(1) and 110(1)(2) are downsampled to a lower resolution to input downsampled frames to the GPU 304. In some embodiments, the motion vectors 110(2)(2) are generated according to any suitable compression algorithm, such as the HEVC algorithm.

[0049] The array of motion vectors 110(2)(2) output from the GPU 304 may be referred to herein as a "motion vector field." This motion vector field 110(2)(2) may also be output at any suitable resolution and / or downsampled / upsampled. For example, the motion vector array 110(2)(2) may include a single motion vector per pixel, a single motion vector per group of pixels (e.g., one motion vector for a 4×4 macroblock, an 8×8 macroblock, an arbitrarily shaped patch of pixels, etc.), or a single motion vector for all pixels of a given frame 300.

[0050] If a portion of a second frame 300(2) is similar to a portion of a first frame 300(1) (e.g., within a threshold luminance value) based on a comparison of input pixel data 110(1) by the video encoder of GPU 304, and if the similar portions in each frame 300 are offset by a distance (e.g., a number of pixels in the X and Y component directions), this can be represented by a motion vector included in the array of motion vectors 110(2)(2). Consider an example in which pixel values ​​corresponding to an object 302 in frames 300(1) and 300(2) are determined by the video encoder of GPU 304 to be similar (e.g., matching portions that satisfy some similarity metric based on pixel values ​​in pixel data 110(1)). The motion vector of this object 302 may have a direction that points in the same direction as the movement of the object 302, or the motion vector may point in the opposite direction to the direction of movement of the object 302. In other words, the motion vector may point in a direction that references a position of object 302 in a previous frame 300(1) offset from a position of object 302 in a subsequent frame 300(2). Thus, the motion vectors in array 110(2)(2) provide an offset from coordinates in the second frame 300(2) to coordinates in the first frame 300(1). The offset describes a transformation from an image in the first frame 300(1) to an image in the second frame 300(2) with respect to moving or displacing an object, such as object 302.

[0051] As mentioned above, the third frame 300(3) of FIG. 3 may represent a reprojected frame, meaning that the pixel data 110(1)(3) for the third frame 300(3) may be derived from pixel data 110(1)(2) associated with a previously rendered actual frame (in this case, the second frame 300(2)). In other words, the pixel data 110(1)(3) for the reprojected frame 300(3) is not received in real time from the host computer 106, but is generated from the pixel data 110(1) of an application-generated frame and is used to “fill in” gaps of missing frames when the application is not reaching its frame rate and / or when network congestion occurs. 3, pixel data 110(1)(3) for reprojected frame 300(3) is generated from pixel data 110(1)(2) associated with second frame 300(2) because second frame 300(2) was the last frame 300 rendered before reprojected frame 300(3). In some embodiments, a rotation and reprojection transform may be calculated and used to modify second pixel data 110(1)(2) associated with second frame 300(2) to generate third pixel data 110(1)(3) for reprojected frame 300(3), effectively rotating, translating, and / or otherwise moving the scene rendered in second frame 300(2) in a manner that accounts for the rotation of HMD 102 since the second frame 300(2) was rendered. For example, the user 104 may rotate his or her head from the time the second frame 300(2) was rendered, and this is taken into account in generating the third pixel data 110(1)(3) for the reprojected frame 300(3) such that the scene is presented according to this head movement.

[0052] The motion smoothing technique of FIG. 3 then modifies the third pixel data 110(1)(3) based at least in part on the array of motion vectors 110(2)(2) associated with the second frame 300(2) and received from the host computer 106 to obtain modified third pixel data 110(1)(3)' for the reprojected frame 300(3). In some embodiments, this modification of the third pixel data 110(1)(3) into modified third pixel data 110(1)(3) includes moving pixel values ​​of the third pixel data 110(1)(3) to different positions based on the non-zero motion vectors 110(2)(2) corresponding to particular pixels or groups of pixels. The movement may be in a fixed direction, and a fixed amount (e.g., moving some pixels in the horizontal (+ / -) and vertical (+ / -) directions). The reprojected frame 300(3) is then rendered on a display (e.g., a display panel of the HMD 102) based at least in part on the modified third pixel data 110(1)(3)'. The reprojected frame 300(3) is thus modified based on the motion vectors 110(2)(2) (the motion vectors 110(2)(2) were generated from the pixel data 110 of the previously received (and / or previously rendered) actual frames 300(1) and 300(2) as a result (e.g., a by-product) of compressing the pixel data 110(1)(2) of the second frame 300(2)) and is "motion smoothed" to render the object 302 in an expected position.

[0053] 4 illustrates an example rendering mesh 400 and how the rendering mesh 400 can be used for motion smoothing of a reprojected frame. For example, the rendering mesh 400 can be used by the HMD 102 to modify the third pixel data 110(1)(3) for the reprojected frame 300(3) of FIG. 3 based on an array of motion vectors 110(2) output by the GPU 304 of the host computer 106. In this example, logic in the HMD 102 can generate the rendering mesh 400, and the vertices 402 of the rendering mesh 400 can be moved according to the array of motion vectors 110(2). For example, the vertices 402 can be moved in the direction of the non-zero motion vector 404 and by an amount corresponding to the magnitude of the non-zero motion vector 404. For example, taking object 302 moving left as shown in FIG. 3, motion vector 404 may be applied in the context of rendering mesh 400 to move vertex 402 a certain number of pixels (corresponding to the magnitude of motion vector 404) leftward (or in the negative X direction).

[0054] Render mesh 400 is shown as a tessellation mesh having a number of vertices 402(1), 402(2), ..., 402(N) (collectively 402). The tessellation of render mesh 400 may be any suitable geometric pattern. Although the example render mesh 400 of FIG. 4 is shown as a repeating pattern of triangles 406, any suitable geometric shape may be used for render mesh 400, including, but not limited to, squares (sometimes referred to as "quads"), hexagons (e.g., for a honeycomb pattern), and the like. In this example, a diagonal line from the lower left corner of a square (or quadrangle) to the upper right corner of a square (or quadrangle) is used to create render mesh 400 of repeating triangles 406 with a particular orientation. Different orientations of the rendering mesh 400 may be created by dividing a square (or rectangle) into triangles 406 using a diagonal from the top left corner of the square (or rectangle) to the bottom right corner of the square (or rectangle) instead of the orientations shown in FIG. 4. In some embodiments, a mixture of these different orientations may also be used for a single rendering mesh 400, such as by dividing all other squares using a diagonal from the bottom left corner of the square (or rectangle) to the top right corner of the square (or rectangle), dividing the squares in between using a diagonal from the top left corner of the square (or rectangle) to the bottom right corner of the square (or rectangle), etc. In some embodiments, logic in the HMD 102 may be configured to dynamically determine which of these multiple orientations to use to generate the rendering mesh 400 based on the motion vector field 110(2) received from the host computer 106. This may be done to select the best orientation for the geometric shapes (eg, triangle 406) in rendering mesh 400 that will result in the smoothest looking, motion-smoothed image.

[0055] The rendering mesh 400 may also be generated at any suitable resolution. For example, the highest resolution rendering mesh 400 may be two adjacent triangles 406 per pixel, with each square (or rectangle) mapping to a single pixel. A lower resolution may be two adjacent triangles 406 per group of pixels, such as a group of 16 pixels. Alternatively, pixels may be mapped to the vertices 402 of the rendering mesh 400 at any suitable resolution. For example, each vertex 402 may be associated with a single pixel at the highest resolution, or each vertex 402 may be associated with a group of pixels, such as a group of 16 pixels, at a lower resolution. In some embodiments, the resolution of the rendering mesh 400 is the same resolution as the resolution of the array of motion vectors 110(2), such that a single motion vector 404 in the array of motion vectors 110(2) maps to a vertex 402 or square (or rectangle) (e.g., two adjacent triangles 406). Achieving a matching resolution between rendering mesh 400 and motion vector array 110(2) may be accomplished in various ways, such as, for example, by requesting motion vector array 110(2) at a particular resolution from GPU 304 that matches the resolution of rendering mesh 400, by downsampling or upsampling motion vector array 110(2) to match the resolution as rendering mesh 400, or by generating rendering mesh 400 at a resolution that matches the resolution of motion vector array 110(2) received from host computer 106.

[0056] FIG. 4 illustrates an example in which four non-zero motion vectors 404(1), 404(2), 404(3), and 404(4) correspond to four vertices 402 of a rendering mesh 400. These four motion vectors 404(1)-(4) may correspond to motion vectors detected by the GPU 304 based on the moving object 302 depicted in FIG. 3. Thus, the exemplary motion vectors 404(1)-(4) may point in a leftward direction corresponding to the directional movement of the object 302, but as described above, the directionality may be opposite (e.g., rightward) to the direction of the directional movement of the object 302. The directionality of the motion vectors 404 may be taken into account in a motion smoothing algorithm to modify the pixel data 110(1) to a desired direction. Consider a basic example in which the field of motion vectors 110(2) includes four exemplary motion vectors 404(1)-(4), and all of the remaining motion vectors in the array of motion vectors 110(2) are zero vectors. In this example, pixel data 110(1)(3) of reprojected frame 300(3) may be modified based on non-zero motion vectors 404(1)-(4) by moving vertices 402 corresponding to non-zero motion vectors 404(1)-(4) to different positions within rendering mesh 400 as vertices 408(1), 408(2), 408(3), and 408(4) (shown in the lower part of FIG. 4) move. The lower part of FIG. 4 shows rendering mesh 400 after motion smoothing has been applied, with moved vertices 408(1)-(4) now in different positions within rendering mesh 400 compared to the positions of vertices 402 before motion smoothing. When motion vectors 404(1)-(4) are applied in this manner, the moved vertices 408(1)-(4) distort one or more portions of rendering mesh 400, such as by stretching or distorting certain of the geometric shapes (e.g., triangle 406) within rendering mesh 400. In the example of FIG. 4, some of triangles 406 are stretched as a result of the motion smoothing to create stretched triangles 410, as shown in the lower portion of FIG.Pixel values ​​(of pixel data 110(1)(3) of reprojected frame 300(3)) corresponding to moved vertices 408(1)-(4) are rendered at different pixel locations corresponding to the positions of moved vertices 408(1)-(4) in rendering mesh 200. Pixel locations between moved vertices 206(1)-(4) and non-moved vertices 402 may be blended (e.g., by applying a gradient, such as by interpolating pixel values ​​between moved vertices 408(1)-(4) and non-moved vertices 402). In some embodiments, a depth buffer may be utilized to determine the final set of pixel values ​​that are output to a frame buffer for modified pixel data 110(1)(3)' of reprojected frame 300(3). That is, application of motion vectors 404(1)-(4) to rendering mesh 400 may result in multiple pixel values ​​at locations in the image that correspond to moved vertices 408(1)-(4). In this case, any of the pixel values ​​associated with the "closer" (smaller) depth values ​​may be rendered instead of rendering another pixel value at that location associated with the "farther" (larger) depth value.

[0057] 5 is a flow diagram of an example process 500 for generating a reprojected frame using motion vectors generated by a graphics processing unit (GPU) as part of a motion smoothing technique according to embodiments disclosed herein. For purposes of discussion, process 500 is described with reference to the previous figures.

[0058] At 502, logic in the host computer 106 may provide pixel data 110(1) associated with previously rendered frames 300 as input to the GPU 304. For example, pixel data 110(1) associated with the two most recently rendered frames 300 may be provided as input to the GPU 304 as part of a compression algorithm for compressing the pixel data 110(1) prior to transmission of the compressed pixel data 110(1). These frames 300 may be actual frames received from an application (e.g., a video game application), such as a first frame 300(1) previously rendered and a second frame 300(2) rendered after the first frame 300(1). Thus, the second frame 300(2) may represent the last rendered frame output by an application executing on the host computer 106, and the first frame 300(1) and the second frame 300(2) may be output consecutively by the application in the series of frames 300, but the pixel data 110(1) provided as input in block 502 need not be the pixel data 110(1) of consecutively rendered frames. For example, an intermediate frame 300 may be rendered between the first frame 300(1) and the second frame 300(2), and the pixel data 110(1) provided as input in block 502 may relate to the first frame 300(1) and the second frame 300(2).

[0059] At 504, a motion vector array 110(2) may be received from the GPU 304. The array of motion vectors 110(2) received at block 504 may be generated by a video encoder of the GPU 304 based at least in part on the first pixel data 110(1)(1) associated with the first frame 300(1) and the second pixel data 110(1)(2) associated with the second frame 300(2) (e.g., based on a comparison between the first pixel data 110(1)(1) and the second pixel data 110(1)(2)). The video encoder of the GPU 304 may be configured to use a suitable computer vision and / or video encoding algorithm (e.g., HEVC) that looks for similarity between pixel values ​​(or groups of pixel values), such as, for example, by determining whether a difference between the compared pixel values ​​is less than a threshold difference. Any within such a similarity metric may be considered to be matching pixel data 110(1) between the two frames 300.

[0060] At 506, logic in the HMD 102 may generate third pixel data 110(1)(3) for the reprojected frame 300(3) based at least in part on the second pixel data 110(1)(2) for the second frame 300(2), where the second frame 300(2) represents the rendered frame immediately prior to the reprojected frame 300(3). For example, between blocks 504 and 506, the HMD 102 may receive the compressed second pixel data 110(1)(2) for the second frame 300(2), decompress the pixel data 110(1)(2), modify the pixel data 110(1)(2) to account for the updated pose prediction of the HMD 102, and present an image on the HMD 102 for the second frame 300(2), where in block 506 the HMD 102 is preparing to render the reprojected frame 300(3) as the next frame.

[0061] At 508, logic of HMD 102 may modify third pixel data 110(1)(3) based at least in part on the array of motion vectors 110(2)(2) to obtain modified third pixel data 110(1)(3)' of reprojected frame 300(3). As indicated by sub-blocks 510 and 512, the modification of pixel data 110(1)(3) of reprojected frame 300(3) may utilize rendering mesh 400.

[0062] Thus, at 510, logic of HMD 102 may generate rendering mesh 400 for reprojected frame 300(3). Rendering mesh 400 may include a tessellation mesh having multiple vertices 402. In some embodiments, the resolution of rendering mesh 400 may match the resolution of motion vector array 110(2)(2) such that there is a one-to-one correspondence between motion vectors 404 and “elements” of rendering mesh 400 (e.g., vertices 402 of rendering mesh 400, squares (or rectangles) of rendering mesh 400, etc.). Obtaining a matching resolution between motion vector field 110(2)(2) and rendering mesh 400 may include any of the techniques described herein, such as requesting that GPU 304 output motion vector field 110(2)(2) at a particular resolution, downsampling or upsampling the resolution of motion vector field 110(2)(2), and / or generating rendering mesh 400 at a resolution that matches the resolution of motion vector field 110(2)(2) output by GPU 304.

[0063] At 512, the logic may move vertices 402 (of the plurality of vertices 402) of rendering mesh 400 to different positions within rendering mesh 400 as vertices 408 move. Vertices 402 may be moved (i) in the direction of non-zero motion vector 404 and (ii) by an amount corresponding to the magnitude of non-zero motion vector 404. Thus, modification of third pixel data 110(1)(3) in block 508 may be performed according to the moved vertices 408 of rendering mesh 400, for example, by moving pixel values ​​of third pixel data 110(1)(3) according to the moved vertices 408 to obtain modified third pixel data 110(1)(3)′ of reprojected frame 300(3). For example, the pixel value of the third pixel data 110(1)(3) may be moved four pixels to the left and four pixels upward to a new location in the modified third pixel data 110(1)(3) according to the moved vertex 408 in the rendering mesh 400.

[0064] In some embodiments, multiple motion vector fields 110(2) may be received at block 504 based on a different set of previously rendered frames 300 input at block 502, and additional motion-related parameters may be determined based on the multiple motion vector fields 110(2) for use in motion smoothing of the reprojected frame at block 508. For example, in addition to the previous two frames 300(1) and 300(2) resulting in the array of motion vectors 110(2)(2) received at block 504, the algorithm of FIG. 5 may, in some embodiments, return one or more additional frames, such as by providing as input to the GPU 304 the first frame 300(1) and a “zeroth” frame 300(0) rendered before the first frame 300(1) and receiving an array of additional motion vectors 110(2) based on that different pair of input frames 300. The multiple arrays of motion vectors may then be compared to determine motion-related parameters, such as the acceleration of objects moving between frames, and these motion-related parameters may be applied in block 508, such as by modifying (e.g., increasing / decreasing) the magnitude of the final motion vector 404 applied to the third pixel data 110(1)(3) to move pixel values ​​more or less with respect to the motion smoothing adjustment.

[0065] At 514, logic of the HMD 102 may render the reprojected frame 300(3) on a display (e.g., on a display panel of the HMD 102) based at least in part on the modified third pixel data 110(1)(3)′. The resulting reprojected frame 300(3) rendered at block 514 is “motion smoothed” by modifying the third pixel data 110(1)(3) according to the motion vector 110(2) received from the GPU 304. It should be understood that the GPU 304 may represent one or more GPUs 304. For example, multiple GPUs 304 may be utilized to render a given frame 300 on a stereo display panel of the HMD 102, and the pixel data 110(1) for frame 300 input to these GPUs 304 may be split accordingly (e.g., the left half of the pixel data 110(1) may be provided as input to a first GPU 304, and the right half of the pixel data 110(1) may be provided as input to a second GPU 304).

[0066] 6 is a flow diagram of an example process 600 for aligning pixel data of a previously rendered frame before the pixel data is input to a GPU for motion vector estimation, according to embodiments disclosed herein. For purposes of discussion, the process 600 is described with reference to previous figures. Additionally, as indicated by off-page reference "A" in FIGS. 5 and 6, the process 600 may represent operations performed prior to the operations at block 504 of FIG. 5, and the process 500 may, in some embodiments, continue with the operations of blocks 504-514.

[0067] At 602, logic (e.g., logic of host computer 106) may determine, based on rotation data received from HMD 102, that HMD 102 rotated from a first orientation to a second orientation while rendering a previously rendered frame 300, whose pixel data 110(1) is provided as an input to GPU 304. For example, user 104 may rotate their head to the right between time t1 and time t2, which may correspond to the times of rendering first frame 300(1) and second frame 300(2).

[0068] At 604, previously rendered frames 300 may be aligned before pixel data 110(1) for those frames 300 are provided as input to the GPU 304 of the host computer 106. This is done because of the movement of the HMD 102 between the frames 300, which may cause objects (both stationary and moving objects) to move between positions in the course of rendering the two frames 300(1) and 300(2). By aligning one frame 300(1) with the other frame 300(2), or vice versa, pixel data 110(1) representing a particular stationary object in the scene may be moved to approximately the same position between the two frames 300(1) and 300(2) such that the pixel data 110(1) corresponding to the stationary object is not mistaken for a moving object by the video encoder of the GPU 304. The alignment in block 604 may include modifying at least one of: (i) first pixel data 110(1)(1) (associated with the first frame 300(1)) to obtain modified first pixel data 610(1)(1) that aligns the scene in the first frame 300(1) with the second orientation of the HMD 102 at the time of rendering the second frame 300(2), or (ii) second pixel data 110(1)(2) (associated with the second frame 300(2)) to obtain modified second pixel data that aligns the scene in the second frame 300(2) with the first orientation of the HMD 102 at the time of rendering the first frame 300(1). 6 illustrates the former case, where first pixel data 110(1)(1) is modified to obtain modified first pixel data 610(1)(1) that aligns the scene in first frame 300(1) to the second orientation of HMD 102. However, it should be understood that either the first pixel data 110(1)(1) or the second pixel data 110(1)(2) may be modified for alignment purposes.The alignment in block 604 may include a rotational reprojection correction to one or both of the pixel data of the frames to reduce the difference in motion vector 110(2) due to head movement. This reduction in the difference in motion vector 110(2) may provide the additional benefit of reduced bandwidth in transmitting motion vector 110(2) from host computer 106 to HMD 102, as described herein.

[0069] At 606, logic of the host computer 106 may provide modified pixel data of one of the previously rendered frames 300 and original pixel data 110(1) of the other frame 300 as input to the GPU 304. The example shown in the diagram of FIG. 6 depicts providing modified first pixel data 610(1)(1) and original second pixel data 110(1)(2) as input to the GPU 304. As noted, process 600 may continue from block 606 of process 500 to block 504 (as indicated by off-page reference "A"). Thus, based on the pixel data provided as input to the GPU 304 at block 606, it may receive motion vector array 110(2) from the GPU 304 at block 504 and perform the remaining operations of the motion smoothing algorithm of FIG. 5 to render a motion smoothed reprojected frame 300(3).

[0070] 7 is a flow diagram of an example process 700 for filtering out some pixel data of a previously rendered frame and providing a remaining portion of the pixel data as input to a GPU for motion vector estimation, according to embodiments disclosed herein. For purposes of discussion, process 700 is described with reference to previous figures. Additionally, as indicated by off-page reference "A" in FIG. 5 and FIG. 7, process 700 may represent operations performed prior to the operations at block 504 of FIG. 5, and process 500 may, in some embodiments, continue with the operations of blocks 504-514.

[0071] At 702, logic (e.g., logic of host computer 106) may determine, based on rotation data received from HMD 102, that HMD 102 rotated from a first orientation to a second orientation while rendering a previously rendered frame 300 (e.g., for compression of pixel data) to be provided as input to GPU 304. The previously rendered frame 300 may be the first frame 300(1) and the second frame 300(2) shown in FIG.

[0072] At 704, logic (e.g., logic of host computer 106) may provide a portion of available pixel data 110(1) as input to GPU 304. The portion of available pixel data 110(1) provided as input to GPU 304 may include, for example, a portion 706(1) of first pixel data 110(1)(1) and a portion 706(2) of second pixel data 110(1)(2), each corresponding to pixels other than a subset of pixels at one or more edges of a display panel of HMD 102. For example, as shown in FIG. 7, portion 706 of pixel data 110(1) provided as input to GPU 304 excludes remaining portions of pixel data 110(1) at the left and right edges of the display panel (shown in black in FIG. 7). In other words, pixel data 110(1) at the left and right edges of the display panel are not provided to GPU 304 such that GPU 304 focuses motion estimation efforts on a central portion 706 of pixel data 110(1) between the left and right edges of the display panel. Due to rotation of HMD 102 in the positive and / or negative vertical directions, the excluded portions of pixel data 110(1) may be the top and bottom edges of the image. In any event, excluding a portion of pixel data 110(1) at the edges of the image and providing the remaining portions 706 of pixel data 110(1) exclusively to GPU 304 may result in effectively zero motion vectors 404 at the edges of the image, which may be useful in situations where unwanted visual artifacts may manifest near the edges of the image, resulting in outlier motion vectors 404 that are not characteristic of an otherwise moving object 302. As noted, process 700 may continue from block 704 of process 500 to block 504, where motion vector array 110(2) is received from GPU 304 based on pixel data provided as input to GPU 304 in block 704, and the motion smoothing algorithm of FIG. 5 may be performed to render motion-smoothed reprojected frame 300(3).

[0073] 8 is a flow diagram of an example process 800 for aligning pixel data of a previously rendered frame that excludes a panel mask portion of the pixel data of the previously rendered frame and provides a remaining portion of the pixel data as input to a GPU for motion vector estimation, according to embodiments disclosed herein. For purposes of discussion, process 800 is described with reference to previous figures. Additionally, as indicated by off-page reference "A" in FIG. 5 and FIG. 8, process 800 may represent operations performed prior to the operations at block 504 of FIG. 5, and process 500 may, in some embodiments, continue with the operations of blocks 504-514.

[0074] At 802, logic (e.g., logic of host computer 106) may determine, based on rotation data received from HMD 102, that HMD 102 rotated from a first orientation to a second orientation while rendering a previously rendered frame 300, which will be provided as input to GPU 304. The previously rendered frames may be first frame 300(1) and second frame 300(2) shown in FIG.

[0075] At 804, logic (e.g., logic of host computer 106) may modify first pixel data 110(1)(1) associated with first frame 300(1) to obtain modified first pixel data 808(1) that aligns the scene of first frame 300(1) with the second orientation of HMD 102. In the example of FIG. 8, a portion of first pixel data 110(1)(1) represents data corresponding to a panel mask (shown in black) rendered at the periphery of the display panel of HMD 102 for each frame. Thus, panel mask portion 814(1) of modified first pixel data 808(1) (i.e., the portion shown in black in FIG. 8) represents data corresponding to the panel mask of modified first pixel data 808(1). As noted, the application may be instructed to generate extra pixel data when rendering a frame, the extra pixel data corresponding to the panel mask. In this manner, there is no effective reduction in FOV when applying the panel mask.

[0076] At 806, logic (e.g., logic of host computer 106) may modify second pixel data 110(1)(2) associated with second frame 300(2) to obtain modified second pixel data 808(2) that aligns the scene in the second frame with the first orientation of HMD 102. Again, a portion of second pixel data 110(1)(2) represents data for each frame that corresponds to a panel mask (shown in black) rendered at the periphery of the display panel of HMD 102. Thus, panel mask portion 814(2) of modified second pixel data 808(2) (i.e., the black portion in FIG. 8 ) represents data that corresponds to the panel mask of modified second pixel data 808(2).

[0077] At 810, logic (e.g., logic of host computer 106) may combine panel mask portion 814(1) of modified first pixel data 808(1) with panel mask portion 814(2) of modified second pixel data 808(2) to determine a subset of pixels having a common value that corresponds to panel mask 816. This can be thought of as a kind of Venn diagram, where the subset of pixels that corresponds to panel mask 816 is a combined version of the modified panel masks in modified first pixel data 808(1) and modified second pixel data 808(2).

[0078] At 812, logic (e.g., logic of host computer 106) may provide as input to GPU 304 a particular portion (e.g., a central portion) of first pixel data 110(1)(1) and a particular portion (e.g., a central portion) of second pixel data 110(1)(2), each of which corresponds to pixels other than the subset of pixels corresponding to panel mask 816. This allows GPU 304 to ignore motion estimation of portions of pixel data 110(1) in each previously rendered frame that are covered by panel mask 816 pixels, and GPU 304 can focus its motion estimation efforts on the central, non-panel mask portions of pixel data 110(1) of each frame 300. As noted, process 800 may continue from block 812 of process 500 to block 504, where motion vector array 110(2) is received from GPU 304 based on pixel data provided as input to GPU 304 in block 812, and the motion smoothing algorithm of FIG. 5 may be performed to render motion-smoothed reprojected frame 300(3).

[0079] 9 is a flow diagram of an example process 900 for thresholding motion vectors output from a GPU before being used to generate a reprojected frame as part of a motion smoothing technique, according to an embodiment disclosed herein. For purposes of discussion, process 900 is described with reference to the previous figures.

[0080] At 902, logic (e.g., logic of host computer 106) may provide pixel data 110(1) associated with a previously rendered frame 300 as input to GPU 304. The operations performed at block 902 may be similar to the operations described with respect to block 502 of process 500.

[0081] At 904, logic (e.g., logic of the host computer 106) may receive the motion vector array 110(2) from the GPU 304. The operations performed at block 904 may be similar to the operations described with respect to block 504 of process 500.

[0082] At 906, logic (e.g., logic of the HMD 102 after receiving the motion vectors 110(2) from the host computer 106) may compare the magnitude (or length) of each individual motion vector in the array of motion vectors 110(2) to a first threshold magnitude to determine a first subset of motion vectors 110(2) having a magnitude greater than the first threshold magnitude. The first threshold magnitude may be utilized to mitigate the effect of this subset of high magnitude outlier motion vectors 110(2) that may represent changes between frames that are something other than movement or motion of the virtual object 302. Abnormally high magnitude motion vectors may arise for a variety of reasons. The GPU 304 may find that the top right pixel of the second frame 300(2) is sufficiently similar to the bottom left pixel of the first frame 300(1) and may output a resulting motion vector 404 with a relatively high magnitude compared to other motion vectors 404 in the array, even though this motion vector 404 does not represent movement or motion of the virtual object 302 between frames 300. In some cases, a video game may allow a user to teleport to a different position where the scene changes drastically between frames 300, causing large motion vectors 404 to be generated by the video encoder of the GPU 304. In these and other cases, it may be useful to threshold these large motion vectors 404.

[0083] At 907, if the multiple previously rendered frames 300 include a first frame 300(1) and a second frame 300(2), logic (e.g., logic of the HMD 102) may determine a period (or interval) between a first time when the first frame 300(1) was rendered and a second time when the second frame 300(2) was rendered.

[0084] At 909, a first threshold magnitude used in the comparison at block 906 may be selected based at least in part on the time period between the first time and the second time. In the example, the first threshold magnitude, measured in degrees of movement from the user's viewpoint per unit time, is 6 degrees of movement per 11.11 milliseconds of time between the two frames 300(1) and 300(2). Thus, the longer the time between the two frames 300(1) and 300(2), the larger the first threshold magnitude will be, and thus the motion vector 110(2) will be as long as allowed. This is because, beyond a certain rate of motion, motion smoothing may not be as effective and may actually cause deleterious visual artifacts (e.g., by making parts of the scene appear to "swimming").

[0085] At 908, logic (e.g., logic of HMD 102) may reduce the magnitude of a first subset of motion vectors 110(2) determined to have a magnitude greater than the first threshold magnitude, such that the magnitudes are limited at the first threshold magnitude. This creates a first modified motion vector array 110(2). In other words, for those motion vectors 110(2) that exceed the first threshold magnitude, the logic is configured to limit (or restrict) the magnitudes of those motion vectors 110(2) to the first threshold magnitude, such that the first modified array of motion vectors 110(2) includes magnitudes less than or equal to the first threshold magnitude, and does not include magnitudes greater than the first threshold magnitude. In some embodiments, instead of limiting the magnitudes of motion vectors 110(2) at the first threshold magnitude and using all of the non-zero motion vectors for motion smoothing, the logic may discard the first subset of motion vectors 110(2) that have magnitudes that meet or exceed the first threshold magnitude, such that they are not used in the motion smoothing at all.

[0086] At 910, logic (e.g., logic of HMD 102) may compare the magnitude (or length) of each individual motion vector in the array of motion vectors 110(2) to a second threshold magnitude to determine a second subset of motion vectors 110(2) having a magnitude less than the second threshold magnitude. The second threshold magnitude may be utilized to mitigate the effect of this subset of low magnitude outlier motion vectors 404, which may represent frame-to-frame variations often caused by a user's constant head movement and / or head tracking that is accurate but not accurate to absolute zero degrees of movement. Because of this, the output of GPU 304 rarely provides zero-length motion vectors 404. Rather, as discussed above, it is more common for the output of GPU 304 to have a significant amount of noise due to ambient head movement and / or tracking jitter. In other words, pixels between two consecutive frames rarely match 100%. In the example, the second threshold magnitude measured in pixels of motion is a threshold of two pixels of motion. In this example, any motion vector that has a motion magnitude (or length) of less than 2 pixels is considered to be a low magnitude outlier vector.

[0087] At 912, logic (e.g., logic of HMD 102) may reduce the magnitude of a second subset of motion vectors 110(2) determined to have a magnitude less than a second threshold (e.g., low magnitude outlier motion vectors having a motion length less than 2 pixels) to a length / magnitude of zero. This creates a second modified motion vector array 110(2) that does not include any motion vectors having a magnitude less than the second threshold magnitude. This is sometimes referred to herein as applying a small dead zone to motion vectors 110(2).

[0088] At 914, logic (e.g., logic of HMD 102) may modify pixel data 110(1)(3) of the reprojected frame based on the second modified array of motion vectors 110(2) having magnitudes bounded by the first threshold magnitude and without any motion vectors 404 having magnitudes less than the second threshold. In some embodiments, such as when the first subset of motion vectors is discarded instead of being bounded by the first threshold magnitude, at block 914, the remaining subset of motion vectors 110(2) other than the first subset of discarded motion vectors 110(2) is used to modify pixel data 110(1)(3) of the reprojected frame. The modification of pixel data 110(1)(3) may include any of the techniques described herein (e.g., those described with reference to blocks 508-512 of process 500).

[0089] At 916, logic (e.g., logic of HMD 102) may render reprojected frame 300(3) on a display (e.g., on a display panel of HMD 102) based at least in part on the modified pixel data 110(1)(3)′ of reprojected frame 300(3). The operations performed at block 916 may be similar to the operations described with respect to block 514 of process 500.

[0090] 10 is a flow diagram of an example process 1000 for damping motion vectors output from a GPU before they are used to generate a reprojected frame as part of a motion smoothing technique, according to an embodiment disclosed herein. For purposes of discussion, the process 1000 is described with reference to the previous figures.

[0091] At 1002, logic (e.g., logic of host computer 106) may provide pixel data 110(1) associated with a previously rendered frame 300 as input to GPU 304. The operations performed at block 1002 may be similar to the operations described with respect to block 502 of process 500.

[0092] At 1002, logic (e.g., logic of the host computer 106) may receive the motion vector array 110(2) from the GPU 304. The operations performed at block 1004 may be similar to the operations described with respect to block 504 of process 500.

[0093] At 1006, logic (e.g., logic of the HMD 102 after receiving the motion vectors 110(2) from the host computer 106) may generate an attenuation texture overlaid on the array of motion vectors 110(2) for the purpose of attenuating (e.g., shortening, reducing, etc.) the magnitude of individual ones of the motion vectors 110(2) (e.g., when there is low confidence in the quality of those motion vectors 110(2)). The attenuation texture may be generated at any suitable resolution such that the attenuation texture includes multiple texels (e.g., a grid of texels). The resolution of the attenuation texture may be a lower resolution than the resolution of the motion vector array 110(2) such that multiple motion vectors are within each texel of the attenuation texture. In an example, the attenuation texture may have a resolution such that a texel in the center of the user's field of view (FOV) is approximately 6 degrees by 6 degrees (horizontal and vertical). The resolution per texel (horizontal and vertical) may be approximate, as the number of degrees per texel may not be constant throughout the attenuation texture. For example, the FOV of the HMD 102 may span approximately 100 degrees by 100 degrees (e.g., plus or minus approximately 10 degrees) in area. The attenuation texture may be generated based on a non-linear projection matrix to provide texels with a resolution of approximately 6-10 degrees by 6-10 degrees, although the resolution varies from texel to texel in the attenuation texture. Additionally, each texel in the attenuation texture may be assigned an attenuation value (e.g., in the range of 0.0 to 1.0). The attenuation value assigned to a given texel controls the amount by which the magnitude of the motion vector 404 at that texel is reduced (shortened). For example, an attenuation value of 1.0 may correspond to no attenuation, so applying an attenuation value of 1.0 to a motion vector 404 means that the motion vector 404 is left unchanged (e.g., not reduced, not shortened, etc.) in terms of its magnitude. However, an attenuation value of 0.0 may correspond to complete attenuation, so applying an attenuation value of 0.0 to a motion vector 404 means that the motion vector 404 is reduced to zero in terms of its magnitude.Attenuation values ​​between 0.0 and 1.0 may correspond to partial attenuation, so for example, applying an attenuation value of 0.5 to a motion vector 404 means that the motion vector is reduced to 50% (half) of its original length (magnitude). Thus, the attenuation texture can reduce (shorten) the magnitude of the motion vector 404, but cannot increase (lengthen) the magnitude of the motion vector 404.

[0094] FIG. 10 illustrates an exemplary attenuation texture 1007 that may be generated in block 1006 based on an abrupt change in luminance (or color) at the bottom of the screen between two successively rendered frames 300. For example, while playing a video game, a user of the HMD 102 may select a button on the handheld controller that causes a large blue mesh to appear in the bottom half of the screen on the next rendered frame 300. In this exemplary scenario, the GPU 304, as it processes this frame 300 and the previously rendered frame 300 as inputs, may find an empty pixel rendered at the top of the scene that has a luminance value that most closely matches the luminance value of a pixel in the large blue mesh in the bottom half of the screen. This results in the GPU 304 outputting a motion vector field 110(2) with motion vectors 404 for the bottom of the scene that are suddenly very large in magnitude and do not represent actual movement in the scene. In the example attenuation texture 1007, white texels may represent an attenuation value of 1.0, while black texels may represent an attenuation value of 0.0, and shades of gray texels may represent attenuation values ​​between 0.0 and 1.0.

[0095] At 1008, logic (e.g., logic of HMD 102) may use the attenuation texture generated in block 1006 to reduce (e.g., scale down) the magnitude of each individual motion vector 404 in the array of motion vectors 110(2) (e.g., use the attenuation value of the attenuation texture to reduce the magnitude of the motion vector 110(2) by an amount corresponding to the attenuation value). This creates a modified motion vector array 110(2) assuming at least one attenuation value in the attenuation texture that is less than 1.0. The purpose is to scale down motion vectors 404 that have changed between frames 300 by more than a threshold amount, which is best illustrated as indicating that these motion vectors 404 do not represent actual motion in the scene and should not be mistaken for representing motion. The attenuation texture is used to find areas of the screen where the motion vectors in those areas are extreme (in terms of their magnitude) within a single frame, indicating that the motion vectors are not reliable to represent real motion, or, if they represent real motion in the scene, that such motion is too fast for motion smoothing to have a noticeable effect anyway, and it is preferable to have a zero motion vector rather than attempting motion smoothing using those motion vectors. Realistically, if there is motion between frames, the motion is usually not that different from the previous frame, since the refresh rate of the display is usually around 90Hz, and it is difficult for something to move significantly in that short time between frames. Thus, in many cases, these extreme motion vectors 404 that have been attenuated by the attenuation texture do not represent real motion.

[0096] As indicated by sub-block 1009, in some embodiments, using the attenuation texture to reduce the magnitude of the motion vector in block 1008 may include applying a minimum N×N filter to the attenuation texture before using the attenuation texture to reduce the magnitude of the motion vector, where N may be any suitable number. N=3 uses a minimum 3×3 filter (i.e., a 3×3 block of texels). Using the example of N=3, in applying the minimum 3×3 filter to the attenuation texture in block 1009, logic (e.g., logic of HMD 102) may obtain, on a texel-by-texel basis, from the attenuation texture 1007, the minimum of nine attenuation values ​​in each 3×3 block of texels centered on a corresponding texel (e.g., a block of texels that includes the corresponding texel and its eight neighboring texels). Applying this minimum N×N filter in block 1009 will result in more motion vectors 404 being zeroed because any corresponding texel in the attenuation texture that is normally assigned a non-zero attenuation value but has a neighboring texel assigned a zero attenuation value will have a zero attenuation value applied to its corresponding motion vector, thereby reducing the motion vector of the corresponding texel to zero, even though that corresponding texel is assigned a non-zero attenuation value in the attenuation texture 1007. In other words, a motion vector 404 in a given texel of the attenuation texture will be zeroed if a neighboring texel is assigned a zero attenuation value, meaning that the neighboring texel has many high magnitude motion vectors. FIG. 10 shows an exemplary attenuation texture 1011 that is the result of applying the minimum N×N filter to the attenuation texture 1007. The resulting attenuation texture 1011 contains more black texels than the attenuation texture 1007, meaning that more motion vectors are zeroed using the resulting attenuation texture 1011.

[0097] At 1010, logic (e.g., logic of HMD 102) may modify pixel data 110(1)(3) of the reprojected frame based on modified motion vector array 110(2), possibly with some magnitudes attenuated for attenuation values ​​less than 1.0 in the attenuation texture. The modification of pixel data 110(1)(3) may include any of the techniques described herein (e.g., those described with reference to blocks 508-512 of process 500).

[0098] At 1012, logic (e.g., logic of HMD 102) may render reprojected frame 300(3) on a display (e.g., on a display panel of HMD 102) based at least in part on the modified pixel data 110(1)(3)′ of reprojected frame 300(3). The operations performed at block 1012 may be similar to the operations described with respect to block 514 of process 500.

[0099] 11 is a flow diagram of an example process 1100 for generating a decay texture, according to an embodiment disclosed herein, that may be used to decay motion vectors output from a GPU before being used to generate a reprojected frame as part of a motion smoothing technique. Thus, process 1100 may represent a sub-process of operations that may be performed to generate a decay texture in block 1006 of process 1000. For purposes of discussion, process 1100 is described with reference to the previous figures.

[0100] At 1102, after receiving motion vector array 110(2) from host computer 106, logic in HMD 102 may determine a set of difference vectors for each texel of the attenuation texture based at least in part on multiple previously obtained arrays of motion vectors 110(2). For example, a history of the array of motion vectors 110(2) may be maintained as a series of frames are processed, and the last two arrays of motion vectors 110(2) (e.g., the last two arrays of motion vectors 110(2)) may be compared to determine a set of difference vectors in block 1102 (e.g., difference vectors indicative of a difference (e.g., in terms of magnitude) between corresponding pairs of motion vectors in the two arrays).

[0101] At 1104, logic (e.g., logic of HMD 102) may calculate a percentage of difference vectors in each texel of the attenuation texture that have a magnitude greater than a threshold magnitude. For example, at block 1104, logic may evaluate the difference vectors in a given texel of the attenuation texture, identify "outlier" vectors in the texel as those difference vectors that have a difference (e.g., magnitude) greater than a threshold magnitude, and calculate a percentage of outlier vectors in the texel. For purposes of identifying "outlier" vectors at block 1104, any suitable threshold may be used, for example, a threshold of approximately 3 degrees of movement / motion (from the user's perspective) per frame (or per 11.11 milliseconds of time). In other words, the threshold used at block 1104 may be based on a time period (or interval) between a first time when the first frame 300(1) was rendered and a second time when the second frame 300(2) was rendered, as described above with reference to blocks 907 and 909 of process 900 of FIG. 9.

[0102] At 1106, logic (e.g., logic of HMD 102) may determine, for each texel of the attenuation texture, whether the percentage of difference vectors within the texel that are “outlier” difference vectors meets or exceeds a threshold percentage, such as 25%. If at block 1106 the percentage of difference vectors within the texel that are “outlier” difference vectors meets or exceeds the threshold percentage, process 1100 may follow the “yes” path from block 1106 to block 1108.

[0103] At 1108, logic (e.g., logic of HMD 102) may calculate a damping value as zero for texels that have some "outlier" difference vectors that meet or exceed a threshold percentage. In other words, the logic may calculate a damping value for a given texel as zero if a certain amount (e.g., 25% or more) of the difference vectors are greater than the maximum length. As described in process 1000, applying a damping value of zero to the motion vectors 404 in a given texel reduces the magnitude of those motion vectors 404 to zero (e.g., to fully dampen the motion vectors in that texel).

[0104] For texels having less than a threshold percentage of "outlier" difference vectors at 1106, process 1100 may take the "No" path from block 1106 to block 1110, where logic may calculate attenuation values ​​for those texels as attenuation values ​​that are based at least in part on the percentages calculated at block 1104. For example, if the percentage of difference vectors having magnitudes greater than the threshold magnitude for a given texel is less than the threshold percentage (e.g., 25%), the attenuation value may be set as a value that linearly maps to the calculated percentage within a range of values ​​below the threshold percentage (e.g., within a range of 0% to 25%). The attenuation value calculated at block 1110 is effective (when an attenuation texture is applied to motion vector field 110(2)) to reduce noise in the remaining non-zeroed vectors.

[0105] At 1112, the calculated attenuation value may be compared to an existing attenuation value for that texel based on the previously rendered frame 300. If the attenuation value at block 1112 is less than the existing value, process 1100 may take the "Yes" path from block 1112 to block 1114, where the attenuation value for the texel is set to the attenuation value calculated at block 1108 or 1110. For example, if there was no attenuation for that texel in the attenuation texture in the previous frame, the existing attenuation value may be equal to 1.0, and if the new attenuation value calculated at block 1108 or 1110 is less than 1.0, in this example, the attenuation value is set to the newly calculated attenuation value for attenuating motion vectors in the given texture.

[0106] If, at 1112, the calculated attenuation value is greater than or equal to the existing value for the texel in question, process 1100 may follow the "No" path from block 1112 to block 1116, where the existing attenuation value is incremented over time toward a maximum attenuation value (e.g., 1.0), which corresponds to no attenuation. For example, because some situations can cause erratic motion vectors that change between frames, the new attenuation value may be "blended" on top of the texel's previous (old) attenuation value in the attenuation texture, meaning that the attenuation value is incrementally increased over a period of time (e.g., over a period of one second) until there is no attenuation, assuming that the calculated attenuation value is never less than the texel's existing attenuation value over that period of time.

[0107] 12 is a flow diagram of an example process 1200 for zeroing motion vectors corresponding to regions with little or no color change before the resulting set of motion vectors is used to generate a reprojection frame as part of a motion smoothing technique, according to an embodiment disclosed herein. For purposes of discussion, process 1200 is described with reference to the previous figures.

[0108] At 1202, logic (e.g., logic of host computer 106) may provide pixel data 110(1) associated with a previously rendered frame 300 as input to GPU 304. The operations performed at block 1202 may be similar to the operations described with respect to block 502 of process 500.

[0109] At 1204, logic (e.g., logic of the host computer 106) may receive the motion vector array 110(2) from the GPU 304. The operations performed at block 1204 may be similar to the operations described with respect to block 504 of process 500.

[0110] At 1206, logic (e.g., logic of the HMD 102 after receiving the motion vectors 110(2) from the host computer 106) may compare color pixel values ​​(in terms of RGB vectors) among multiple previously rendered frames 300 input to the GPU 304 to determine a color difference value for each motion vector in the array of motion vectors 110(2). These color difference values ​​may be compared to a threshold color difference to determine a subset of motion vectors 110(2) whose color difference values ​​are less than the threshold color difference and therefore associated with little or no color change between frames. In some embodiments, the logic treats the RGB pixel values ​​as vectors. It then compares corresponding RGB vectors among previously rendered frames 300 to determine a color difference (e.g., a difference vector). Because each motion vector 404 in the array of motion vectors 110(2) may correspond to a block of pixels (e.g., a block of 8×8 pixels), such color comparisons may be performed on a motion vector basis by comparing some or all of the RGB pixel values ​​in the block of 8×8 pixels between the frames 300. In some embodiments, logic (e.g., logic of HMD 102) compares color values ​​of every other row and every other column of pixels in each 8×8 block of pixels corresponding to a single motion vector 404 to determine an amount of color change (e.g., color difference values) between frames 300, the amount of color change associated with a particular motion vector 404. Thus, for a block of pixels corresponding to a given motion vector 110(2), the color comparison between frames may involve comparing a fraction (e.g., 25%) of pixels in the corresponding block of pixels to conserve processing resources.

[0111] At 1208, logic (e.g., logic of HMD 102) may reduce the magnitude of a subset of motion vectors 110(2) whose chrominance values ​​are less than the threshold chrominance to a length / magnitude of zero. This creates a modified motion vector array 110(2) that does not include any motion vectors in areas of the scene that had little or no color change between previously rendered frames (sometimes referred to as zero motion chrominance). For example, this may help prevent text from appearing blurry or twisted around edges when in front of a solid color background. It may also help prevent visually disturbing artifacts where lights or other user interface elements in a scene appear to gradually lighten or darken.

[0112] At 1210, logic (e.g., logic of HMD 102) may modify pixel data 110(1)(3) for the reprojected frame based on the array of modified motion vectors 110(2), where modified motion vectors 110(2) no longer include any motion vectors 404 in areas where colors are effectively the same between previously rendered frames 300. The modification of pixel data 110(1)(3) may include any of the techniques described herein (e.g., those described with reference to blocks 508-512 of process 500).

[0113] At 1212, logic (e.g., logic of HMD 102) may render reprojected frame 300(3) on a display (e.g., on a display panel of HMD 102) based at least in part on the modified pixel data 110(1)(3)′ of reprojected frame 300(3). The operations performed at block 1212 may be similar to the operations described with respect to block 514 of process 500.

[0114] 13 is a flow diagram of an example process 1300 for "cleaning up" a motion vector field using one or more filters before the resulting set of motion vectors is used to generate a reprojection frame as part of a motion smoothing technique, according to an embodiment disclosed herein. For purposes of discussion, process 1300 is described with reference to the previous figures.

[0115] At 1302, logic (e.g., logic of host computer 106) may provide pixel data 110(1) associated with a previously rendered frame 300 as input to GPU 304. The operations performed at block 1302 may be similar to the operations described with respect to block 502 of process 500.

[0116] At 1304, logic (e.g., logic of the host computer 106) may receive the motion vector array 110(2) from the GPU 304. The operations performed at block 1304 may be similar to the operations described with respect to block 504 of process 500.

[0117] At 1306, logic (e.g., logic of HMD 102 after receiving motion vectors 110(2) from host computer 106) may apply an N×N scalar median filter to the array of motion vectors 110(2) to obtain a first modified array of motion vectors 110(2), where N may be any suitable number, such as N=5, which applies a 5×5 scalar median filter. In the example of N=5, the 5×5 scalar median filter is applied to each motion vector in the array of motion vectors 110(2) by looking at the set of 5×5 motion vectors 110(2) that surround the selected motion vector. For a selected motion vector, the 5×5 region surrounding the selected vector has 25 motion vectors. The scalar median filter applied to the selected motion vector may calculate a median x component value (of 25 x component values) and a median y component value (of 25 y component values), then combine the median x and y component values ​​to obtain a scalar median vector, and the selected motion vector may be replaced with the scalar median vector. This process is repeated for each vector in the array of motion vectors 110(2). Again, any suitable value for N may be used, with N=5 being merely an example. Applying the scalar median filter in block 1306 helps to reduce noise in the motion vector field 110(2) and remove outlier motion vectors. Using a scalar median filter in block 1306 is also computationally faster than applying a more complex filter, such as a vector median filter, which relies on more complex distance algorithms that are more computationally intensive.

[0118] At 1308, logic (e.g., logic of HMD 102) may apply an M×M average max blur filter to the first modified array of motion vectors 110(2) to obtain a second modified array of motion vectors 110(2), where M may be any suitable number, such as M=3, which applies a 3×3 average max blur filter. The average max blur filter applies two separate filters independently (an average vector filter and a max length vector filter) and then obtains a vector average of the vectors resulting from these two independently applied filters. In an example using M=3, the 3×3 average max blur filter is applied to each motion vector in the array of motion vectors 110(2) by looking at a 3×3 set of motion vectors 110(2) surrounding the selected motion vector. For a selected motion vector, the 3×3 region surrounding the selected vector has 9 motion vectors. First, an average vector filter is applied to the selected motion vectors to calculate an average x component value (of nine x component values) and an average y component value (of nine y component values), and then the average x and y component values ​​are combined to obtain an average vector. Next, a maximum length vector filter is applied independently to the selected motion vectors determining the longest vector (of nine motion vectors). The resulting vector may then be determined by taking a vector average of the average vector and the longest vector, and the selected motion vector may be replaced with the resulting vector. This process is repeated for each vector in the array of motion vectors 110(2). Again, any suitable value for M may be used, with M=3 being merely an example. Applying the average maximum blur filter in block 1308 provides a smoothed motion vector field that removes discontinuities within subareas of the group of motion vectors. In some embodiments, applying the average maximum blur filter in block 1308 is a final "clean up" step on the motion vector field 110(2) before the motion vector field 110(2) is used for motion smoothing, as described herein.When used in combination with other techniques described herein, such as thresholding motion vectors, using attenuating textures, zeroing vectors in areas with little or no color difference, these various operations may be performed before applying the average maximum blur filter.

[0119] At 1310, logic (e.g., logic of HMD 102) may modify pixel data 110(1)(3) of the reprojected frame based on motion vector array 110(2) modified with the filtered motion vector. The modification of pixel data 110(1)(3) may include any of the techniques described herein (e.g., those described with reference to blocks 508-512 of process 500).

[0120] At 1312, logic (e.g., logic of HMD 102) may render reprojected frame 300(3) on a display (e.g., on a display panel of HMD 102) based at least in part on the modified pixel data 110(1)(3)′ of reprojected frame 300(3). The operations performed at block 1312 may be similar to the operations described with respect to block 514 of process 500.

[0121] It should be understood that the various techniques described in the processes of Figures 9-13 (various techniques for modifying the motion vector field output by the GPU 304) may be used in any combination and / or in any order. For example, the techniques described with reference to Figure 9 may be used in combination with the techniques described with reference to Figures 10 and 11, some or all of which may be used in combination with the techniques described with reference to Figure 12, and some or all of which may be used in combination with the techniques described with reference to Figure 13. That is, a robust technique for obtaining an optimized motion vector field may utilize all of the techniques described together with reference to Figures 9-13 in robustly modifying the motion vector field. Furthermore, it should be understood that in a distributed system including a host computer 106 and a communicatively coupled HMD 102, some or all of the various techniques described in the processes of Figures 9-13 (various techniques for modifying the motion vector field output by the GPU 304) may be executed on the host computer 106. For example, host computer 106 may be configured to threshold the motion vectors (e.g., using an attenuation texture) and / or apply a filter to the motion vectors before transmitting the modified motion vector array to HMD 102 for use in motion smoothing. In this scenario, modified motion vector array 110(2) may be transmitted out-of-band from compressed pixel data 110(1) and / or zero motion vectors in modified motion vector array 110(2) may be run-length encoded to compress the motion vector data before transmitting the motion vectors to HMD 102.

[0122] 14 is a flow diagram of an example process 1400 for rotating a previously rendered frame before pixel data is input to a GPU for motion vector estimation, according to embodiments disclosed herein. For purposes of discussion, the process 1400 is described with reference to the previous figures.

[0123] At 1402, logic (e.g., logic of host computer 106) may rotate a first frame 300(1) of multiple previously rendered frames 300, whose pixel data 110(1) will be provided as input to GPU 304, by a rotation amount to obtain a first rotated frame 1404(1).

[0124] At 1406, logic (e.g., logic of the host computer 106) may rotate a second frame 300(2) of the multiple previously rendered frames 300 by a rotation amount to obtain a second rotated frame 1404(2).

[0125] At 1408, logic (e.g., logic of host computer 106) may provide pixel data of rotated frames 1404(1) and 1404(2) as input to GPU 304. Process 1400 may continue from block 1408 to block 504 of process 500 (as indicated by off-page reference "A"). Thus, based on the pixel data provided as input to GPU 304 at block 1408, it may receive motion vector array 110(2) from GPU 304 at block 504 and perform the remaining operations of the motion smoothing algorithm of FIG. 5 to render motion smoothed reprojected frame 300(3).

[0126] Process 1400 may mitigate the effect of horizontal and / or vertical architectural edges in a scene from causing “weird” motion vectors that are not truly representative of a moving or moving object. Multiple previously rendered frames 300(1) and 300(2) may be rotated by the same amount of rotation in blocks 1402 and 1406, or each frame 300(1) and 300(2) may be rotated by a different amount of rotation in blocks 1402 and 1406, respectively. In some embodiments, the amount of rotation for each frame in blocks 1402 and 1406 is predefined (e.g., rotating input frame 300(1) and input frame 300(2) by 45 degrees). By rotating frames 300(1) and 300(2) by 45 degrees, horizontal and vertical edges in a scene may be oriented diagonally, so that these diagonal edges may not cause as many “weird” motion vectors to be generated by the video encoder of GPU 304. In some embodiments, the amount of rotation for each frame in blocks 1402 and 1406 may be a random amount of rotation for each frame 300. This may temporarily hide any edges that may be aligned at 45 degrees.

[0127] 15 is a flow diagram of an example process 1500 for selecting between motion vector arrays generated based on luma and chroma data of a previously rendered frame, according to embodiments disclosed herein. For purposes of discussion, process 1500 is described with reference to previous figures. Additionally, as indicated by off-page reference "B" in FIG. 5 and FIG. 15, process 1500 may represent operations performed prior to the operations at block 506 of FIG. 5, and process 500 may, in some embodiments, continue with the operations of blocks 506-514.

[0128] At 1502, logic (e.g., logic of the host computer 106) may provide luma data associated with a previously rendered frame 300 as input to the GPU 304. For example, first pixel data 110(1)(1) associated with a first frame 300(1) may include first luma data, first chroma data, and / or additional types of data. Similarly, second pixel data 110(1)(2) associated with a second frame 300(2) may include a similar type of data, such as second luma data, second chroma data, etc. Thus, at block 1502, the first luma data and the second luma data may be provided as input to the GPU 304.

[0129] At 1504, logic (e.g., logic of the host computer 106) may receive from the GPU 304 a first motion vector array 110(2) generated by a video encoder of the GPU 304 based at least in part on the first luma data and the second luma data.

[0130] At 1506, logic (e.g., logic of the host computer 106) may provide first chroma data associated with the first frame 300(1) and second chroma data associated with the second frame 300(2) as input to the GPU 304.

[0131] At 1508, logic (e.g., logic of the host computer 106) may receive from the GPU 304 a second motion vector array 110(2) generated by the video encoder of the GPU 304 based at least in part on the first chroma data and the second chroma data.

[0132] At 1510, logic (e.g., logic of host computer 106 or logic of HMD 102 after receiving the multiple arrays of motion vectors 110(2)) may select one of the first array of motion vectors 110(2) or the second array of motion vectors 110(2) based on an analysis of the first array of motion vectors 110(2) and the second array of motion vectors 110(2). For example, the selection algorithm may select the least erratic, as an example, based on the magnitude and / or direction of motion vector 110(2) in each field of motion vectors 110(3). For example, when a bright flashing light (e.g., a virtual explosion) occurs in second frame 300(2), the luma data may cause GPU 304 to output motion vector 110(2) with a magnitude that exceeds some threshold magnitude, while chroma data of the same previously rendered frame 300 may cause GPU 304 to output motion vector 110(2) that does not include such motion vectors with extremely large magnitude. In this case, in block 1510, an array of motion vectors 110(2) generated from the chroma data may be selected. This is just one example of a selection algorithm, and others may be used. As noted, process 1500 may continue from block 1510 to block 506 of process 500, where pixel data 110(1)(3) of reprojected frame 300(3) is modified based on the selected array of motion vectors 110(2) from block 1510.

[0133] 16 is a flow diagram of an example process 1600 for obtaining multiple arrays of motion vectors, determining differences between the arrays, and generating a final motion vector array based on the determined differences for motion smoothing, according to embodiments disclosed herein. For purposes of discussion, the process 1600 is described with reference to previous figures. Additionally, as indicated by off-page reference "B" in FIG. 5 and FIG. 16, the process 1600 may represent operations performed prior to the operations at block 506 of FIG. 5, and the process 500 may, in some embodiments, continue with the operations of blocks 506-514.

[0134] At 1602, logic (e.g., logic of the host computer 106) may provide first pixel data 110(1)(1) associated with a previously rendered frame 300 as input to the GPU 304. The operations performed at block 1602 may be similar to the operations described with respect to block 502 of process 500.

[0135] At 1604, logic (e.g., logic of the host computer 106) may receive a first motion vector array 110(2) from the GPU 304 based on the first pixel data 110(1)(1) provided as input at block 1602.

[0136] At 1606, logic (e.g., logic of the host computer 106) may provide as input to the GPU 304 the second pixel data 110(1)(2) associated with the previously rendered frame 300.

[0137] At 1608, logic (e.g., logic of the host computer 106) may receive from the GPU 304 a second motion vector array 110(2) based at least in part on the second pixel data 110(1)(2).

[0138] At 1610, logic (e.g., logic of host computer 106, or logic of HMD 102 after receiving the array of motion vectors 110(2) from host computer 106) may determine differences in the direction and / or magnitude of motion vectors 110(2) between first array 110(2) and second array 110(2). For example, a comparison may be made between the motion vectors 404 of first array 110(2) and corresponding motion vectors 404 of second array 110(2) to determine whether the motion vectors differ with respect to either or both the direction or magnitude of the motion vectors, and if so, the amount by which they differ.

[0139] At 1612, logic (e.g., logic of host computer 106 or logic of HMD 102) may generate a final array of motion vectors 110(2) based at least in part on the difference between the first array of motion vectors 110(2) and the second array of motion vectors 110(2) as determined at block 1610. For example, the final array of motion vectors 110(2) may represent a set of motion vectors resulting from attenuating individual ones of the motion vectors in the first array 110(2) or the second array 110(2), or from attenuating an average motion vector representing an average of corresponding motion vectors in each array 110(2). As noted, process 1600 may continue from block 1612 to block 506 of process 500 where pixel data 110(1)(3) of reprojected frame 300(3) is modified based on the final array of motion vectors 110(2) generated at block 1612.

[0140] As an example of how process 1600 may be used, consider an example in which a first motion vector array 110(2) is received at block 1604 based on pixel data 110(1) of input frames 300(1) and 300(2) in their original ("y up") orientation, and a second motion vector array 110(2) is received at block 1608 based on pixel data 110(1) of input frames 300(1) and 300(2) rotated by 45 degrees. This concept of rotating input image frame 300 was described with reference to FIG. 14. Thus, in this example, pixel data 110(1) of the first frame 300(1) and the second frame 300(2) may be provided as input in block 1602 to generate a first motion vector array 110(2), and pixel data 110(1) of the first rotated frame 1404(1) and the second rotated frame 1404(2) may be provided as input in block 1606 to generate a second motion vector array 110(2). In this example, the two motion vector arrays 110(2) may be realigned (e.g., by rotating the second array 110(2) 45 degrees in the opposite direction) and then compared to determine discrepancies by looking at the angle difference (direction) and / or length difference (magnitude) between corresponding vectors in both arrays 110(2). It may be the case that actual motion (e.g., objects moving or moving in the scene) may generate a very similar motion vector array for the rotated input frame 300, while erroneously detected motion may generate a very different motion vector array. In this example, the final motion vector array 110(2) may be generated in block 1612 by attenuating the average length of two corresponding vectors in each array 110(2) by the dot product of the two normalized vectors. In some embodiments, a attenuation texture such as that described with reference to Figures 10 and 11 may be utilized in block 1612 to generate the final array.For example, the set of difference vectors determined in block 1102 of process 1100 may be based on the differences between a first array of motion vectors 110(2) received in block 1604 and a second array of motion vectors 110(2) received in block 1608, and the resulting attenuation texture may be used in block 1612 to attenuate the first array 110(2), the second array 110(2), or a combination (e.g., an average) of vectors between the two arrays 110(2).

[0141] As another example of how process 1600 may be used, consider an example in which a first mip level (mipmap level) is generated for an input texture of a plurality of previously rendered frames 300, and corresponding pixel data 110(1) for this first mip level is provided as an input to GPU 304 in block 1602 to generate a first array of motion vectors 110(2). Meanwhile, a second mip level may be generated for an input texture of a plurality of previously rendered frames 300, and corresponding pixel data 110(1) for this second mip level may be provided as an input to GPU 304 in block 1606 to generate a second motion vector array 110(2). This may be done for any number of mip levels generating a corresponding array of any number of motion vectors. In some embodiments, three or four mip levels of the input texture may be generated such that three or four motion vector arrays 110(2) are received prior to block 1610. In some embodiments, each mip level is half the width and height (e.g., 25% of the area) of the previous mip level. In this example, at block 1610, a difference between pairs of arrays 110(2) generated from different mip levels may be determined, and at block 1612, a final motion vector array 110(2) may be generated based on the determined difference. For example, logic may detect anomalies across multiple motion vector arrays 110(2) for different mip levels and attenuate (e.g., reduce to zero) the anomalous motion vectors. This example of generating different motion vector arrays 110(2) using different mip levels of the input frame 300 may be useful in situations where there are large areas of repeating patterns (e.g., wallpaper) in the scene and the video encoder of the GPU 304 may generate many high magnitude motion vectors considering that there is a lot of motion due to the frequency of the repeating patterns relative to the motion vector resolution, even though there are no moving or moving objects.Thus, when process 1600 is used to generate an array of motion vectors 110(2) for different mip levels, it can be useful for detecting anomalies across different frequencies of repeating patterns, and once detected, the attenuation texture can be used to attenuate (e.g., zero out) those motion vectors 110(2).

[0142] As yet another example of how process 1600 may be used, consider an example in which multiple arrays 110(2) are acquired at different resolutions and a difference is determined in block 1610 by comparing a single motion vector in a first array 110(2) to multiple corresponding motion vectors in a second array 110(2) (e.g., by comparing to an average with respect to direction and / or magnitude of the multiple corresponding motion vectors in the second array 110(2)). Using motion vector arrays 110(2) at different resolutions may result in differences that are useful for generating a final array of motion vectors in block 1612.

[0143] 17 is a flow diagram of an example process 1700 for obtaining multiple motion vector arrays at different resolutions for different portions of an image region, according to embodiments disclosed herein. For purposes of discussion, the process 1700 is described with reference to previous figures. Additionally, as indicated by off-page reference "B" in FIG. 5 and FIG. 17, the process 1700 may represent operations performed prior to the operations at block 506 in FIG. 5, and the process 500 may, in some embodiments, continue with the operations of blocks 506-514.

[0144] At 1702, logic (e.g., logic of host computer 106) may provide pixel data 110(1) associated with a previously rendered frame 300 as input to GPU 304. The operations performed at block 1702 may be similar to the operations described with respect to block 502 of process 500.

[0145] At 1704, logic (e.g., logic of the host computer 106) may receive a first motion vector array 110(2)(1) from the GPU 304, where the first motion vector array 110(2)(1) is received at a first resolution or is otherwise upsampled or downsampled to the first resolution.

[0146] At 1706, logic (e.g., logic of host computer 106) may detect an absence of motion in a first portion 1705(A) of image area 1705 across the display based at least in part on the first array of motion vectors 110(2)(1), and may detect motion of object 302 in a second portion 1705(B) of image area 1705 based at least in part on the first array of motion vectors 110(2)(1). For example, a non-zero motion vector may be detected in a right half of image area 1705, while a left half of image area 1705 may include zero-valued motion vectors that do not include any non-zero motion vectors.

[0147] At 1708, logic (e.g., logic of host computer 106) may provide a first portion of first pixel data 110(1)(1) (associated with first frame 300(1)) and a first portion of second pixel data 110(1)(2) (associated with second frame 300(2)) as input to GPU 304. The first portion of first pixel data 110(1)(1) and the first portion of second pixel data 110(1)(2) may each correspond to a first portion 1705(A) of image region 1705 detected as having no motion in block 1706.

[0148] At 1710, logic (e.g., logic of host computer 106) may receive from GPU 304 a second motion vector array 110(2)(2) generated by a video encoder of GPU 304 based at least in part on a first portion of first pixel data 110(1)(1) and a first portion of second pixel data 110(1)(2). This second array of motion vectors 110(2)(2) may be generated at a second resolution that is higher than the first resolution of the first array of motion vectors 110(2)(1). Obtaining a higher resolution motion vector field 110(2)(2) for the first portion 1705(A) of image region 1705 in which an absence of motion was detected may be based on the idea that the motion vector field 110(2)(2) at the higher resolution may be useful for detecting small scale movements in the first portion 1705(A) of image region 1705 in which no large scale movements were detected.

[0149] At 1712, logic (e.g., logic of host computer 106) may provide as input to GPU 304 a second portion of first pixel data 110(1)(1) (associated with first frame 300(1)) and a second portion of second pixel data 110(1)(2) (associated with second frame 300(2)). The second portion of first pixel data 110(1)(1) and the second portion of second pixel data 110(1)(2) may each correspond to a second portion 1705(B) of image region 1705 in which movement of object 302 was detected at block 1706.

[0150] At 1714, logic (e.g., logic of the host computer 106) may receive from the GPU 304 a third motion vector array 110(2)(3) generated by a video encoder of the GPU 304 based at least in part on the second portion of the first pixel data 110(1)(1) and the second portion of the second pixel data 110(1)(2). This third motion vector array 110(2)(3) may be generated at a third resolution that is a lower resolution than the second resolution of the second motion vector array 110(2)(2). This relatively low resolution motion vector field 110(2)(3) for the second portion 1705(B) of the image region 1705 in which motion has been detected may be based on the idea that a higher resolution motion vector field may not be needed to detect movement in the second portion 1705(B) of the image region 1705 in which large scale movement has already been detected in the first resolution of the first motion vector field 110(2)(1).

[0151] At 1716, logic (e.g., logic of host computer 106 or logic of HMD 102 after receiving motion vector array 110(2) from host computer 106) may use the second array of motion vectors 110(2)(2) and the third array of motion vectors 110(2)(3) for motion smoothing. Process 1700 may continue from block 1716 to block 506 of process 500, where pixel data 110(1)(3) of reprojected frame 300(3) is modified based on the motion vectors 110(2) from block 1716. In an example, using multiple arrays of motion vectors at block 1716 for motion smoothing may include operations described with reference to block 1612 of process 1600.

[0152] 18A and 18B show two alternative setups of a system utilizing an HMD 102 and a host computer 106 according to embodiments disclosed herein. As depicted in FIG. 1, an exemplary implementation has the host computer 106 collocated in an environment with the HMD 102 worn by the user 104. For example, the host computer 106 may be located in the house of the user 104 while the user 104 uses the HMD 102 in the house, whether the host computer 106 is located in the same room as the HMD 102 or a different room. Alternatively, the host computer 106 in the form of a mobile computing device (e.g., tablet or laptop) may be carried in a backpack on the back of the user 104, thereby allowing for greater mobility. For example, the user 104 may be located in a public park while using such a system.

[0153] 18A illustrates an alternative implementation in which the host computer 106 represents one or more server computers located geographically remotely relative to the HMD 102. In this case, the HMD 102 may be communicatively coupled to the host computer 106 via an access point (AP) 1800, such as a wireless AP (WAP), base station, or the like. In an example, data is exchanged (e.g., streamed) between the host computer 106 and the HMD 102 via the AP 1800, such as by streaming the data over the Internet.

[0154] FIG. 18B illustrates yet another alternative implementation in which the host computer 106 is communicatively coupled to the HMD 102 via an intermediate computing device 1802, such as a laptop or tablet computer. The difference between FIG. 18A and FIG. 18B is that the AP 1800 of FIG. 18A may simply act as a data routing device that does not perform rendering, while the intermediate computing device 1802 of FIG. 18B may perform a portion of the rendering workload. That is, instead of bifurcating the rendering workload between the host computer 106 and the HMD 102, the rendering workload may be partitioned among three or more devices, such as three devices: the host computer 106, the intermediate computing device 1802, and the HMD 102. In the scenario of FIG. 18B, the host computer 106 may generate pixel data 110(1) as described herein, the intermediate computing device 1802 may perform a first set of rendering operations to modify the pixel data 110(1), and the HMD 102 may perform a final set of rendering operations to modify the modified pixel data.

[0155] FIG. 19 illustrates exemplary components of an HMD 1900, according to an embodiment disclosed herein, in which, for example, a VR headset may be embedded. The HMD 102 may be implemented as a device to be worn by the user 104 (e.g., on the head of the user 104). In some embodiments, the HMD 102 may be head-mountable, such as by allowing the user 104 to fasten the HMD 102 to the user's head using a fastening mechanism (e.g., an adjustable band) sized to fit around the head of the user 104. In some embodiments, the HMD 102 includes a virtual reality (VR) or augmented reality (AR) headset that includes a near-eye or near-to-eye display. Thus, the terms "wearable device," "wearable electronic device," "VR headset," "AR headset," and "head mounted display (HMD)" may be used interchangeably herein to refer to the device 102. However, it should be understood that these types of devices are merely examples of the HMD 102, and it should be understood that the HMD 102 may be implemented in a variety of other form factors.

[0156] In the illustrated implementation, the HMD 102 includes one or more processors 1900 and memory 1902 (e.g., computer-readable medium 1902). In some implementations, the processor 1900 may include a central processing unit (CPU), a GPU 1900(1), both a CPU and a GPU 1900(1), a microprocessor, a digital signal processor, or other processing units or components known in the art. Alternatively or additionally, the functionality described herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like. In addition, each of the processors 1900 may have its own local memory, which may also store program modules, program data, and / or one or more operating systems.

[0157] The memory 1902 may include volatile and non-volatile memory, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Such memory may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVDs) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, RAID storage systems, or any other medium that may be used to store the desired information and that may be accessed by a computing device. The memory 1902 may be implemented as a computer-readable storage medium ("CRSM"), which may be any available physical medium that is accessible by the processor 1900 to execute instructions stored on the memory 1902. In one basic implementation, the CRSM may include random access memory ("RAM") and flash memory. In other implementations, the CRSM may include, but is not limited to, read-only memory ("ROM"), electrically erasable programmable read-only memory ("EEPROM"), or any other tangible medium that can be used to store desired information and that can be accessed by the processor 1900.

[0158] In general, the HMD 102 may include logic (e.g., software, hardware, and / or firmware, etc.) configured to implement the techniques, functions, and / or operations described herein. The computer-readable medium 1902 is shown as including various modules, such as instructions, data stores, etc., that may be configured to execute on the processor 1900 to perform the techniques, functions, and / or operations described herein. Although several exemplary functional modules are shown as stored on the computer-readable medium 1902 and executable on the processor 1900, the same functions may alternatively be implemented in hardware, firmware, or as a system-on-chip (SOC) and / or other logic.

[0159] The operating system module 1904 may be configured to manage the hardware coupled within the HMD 102 for the convenience of the other modules. Additionally, in some instances, the HMD 102 may include one or more applications 1906 stored in the memory 1902 or otherwise accessible to the HMD 102. In this implementation, the applications 1906 include a gaming application 1908. However, the HMD 102 may include any number or type of applications and is not limited to the particular examples shown here. The compositor 1910, in combination with other logic of the HMD 102, may be configured to perform the motion smoothing techniques described herein to render the motion smoothed reprojected frames 300.

[0160] Generally, the HMD 102 has input devices 1912 and output devices 1914. The input devices 1912 may include control buttons. In some implementations, one or more microphones may function as the input devices 1912 for receiving audio input, such as user voice input. In some implementations, one or more cameras or other types of sensors (e.g., inertial measurement units (IMUs)) may function as the input devices 1912 for receiving gestural input, such as hand and / or head movements of the user 104. In some embodiments, additional input devices 1912 may be provided in the form of a keyboard, keypad, mouse, touch screen, joystick, etc. In other embodiments, the HMD 102 may omit a keyboard, keypad, or other similar form of mechanical input. Instead, the HMD 102 may be implemented with a relatively simple form of input device 1912, a network interface (wireless or wired based), a power source, and processing / memory capabilities. For example, a limited set of one or more input components (e.g., dedicated buttons for initiating configuration, powering on / off, etc.) may be used and the HMD 102 may thereafter be used. In one implementation, the input device 1912 may include controls such as basic volume control buttons for increasing / decreasing the volume, as well as power and reset buttons.

[0161] The output device(s) 1914 may include a display 1916, which may include one or more display panels (e.g., a stereo pair of display panels). The output device(s) 1914 may further include, without limitation, light emitting elements (e.g., LEDs), a vibrator for producing tactile sensations, and / or a speaker (e.g., headphones), etc. There may also be simple light emitting elements (e.g., LEDs) to indicate a state, such as when the power is on.

[0162] The HMD 102 may further include a wireless unit 1918 coupled to an antenna 1920 to facilitate wireless connection to a network. The wireless unit 1918 may implement one or more of a variety of wireless technologies, such as Wi-Fi, Bluetooth, radio frequency (RF), etc. It should be appreciated that the HMD 102 may further include a physical port to facilitate a wired connection to a plug-in network device that communicates with the network, connected peripherals (including the host computer 106, such as a PC, game console, etc.), or other wireless networks.

[0163] The HMD 102 may further include an optical subsystem 1922 that directs light from the electronic display 1916 to the user's eyes using one or more optical elements. The optical subsystem 1922 may include various types and combinations of different optical elements, including, but not limited to, apertures, lenses (e.g., Fresnel lenses, convex lenses, concave lenses, etc.), filters, etc. In some embodiments, one or more optical elements in the optical subsystem 1922 may have one or more coatings, such as an anti-reflective coating. The magnification of the image light by the optical subsystem 1922 allows the electronic display 1916 to be physically smaller, lighter, and less power-hungry than larger displays. In addition, the magnification of the image light may increase the field of view (FOV) of the displayed content (e.g., images). For example, the FOV of the displayed content is such that the displayed content is presented using nearly all (e.g., 120-150 degrees diagonal), and possibly all, of the user's FOV. AR applications may have a narrower FOV (e.g., FOV of about 40 degrees). The optical subsystem 1922 may be designed to correct one or more optical errors such as, but not limited to, barrel distortion, pincushion distortion, longitudinal chromatic aberration, transverse aberration, spherical aberration, field curvature, astigmatism, etc. In some embodiments, content provided to the electronic display 1916 for display is pre-distorted, and the optical subsystem 1922 corrects for that distortion when receiving image light from the electronic display 1916 generated based on the content.

[0164] The HMD 102 may further include one or more sensors 1924, such as sensors used to generate motion, position, and orientation data. These sensors 1924 may be or include gyroscopes, accelerometers, magnetometers, video cameras, color sensors, or other motion, position, and orientation sensors. The sensors 1924 may also include sub-portions of sensors, such as a set of active or passive markers that may be viewed externally by a camera or color sensor to generate motion, position, and orientation data. For example, a VR headset may include multiple markers on its exterior, such as reflectors or lights (e.g., infrared or visible light), that when viewed by an external camera or illuminated with light (e.g., infrared or visible light), may provide one or more reference points for interpretation by the software to generate motion, position, and orientation data. The HMD 102 may include a light sensor that is sensitive to light (e.g., infrared or visible light) projected or scattered by a base station in the environment of the HMD 102.

[0165] In one example, the sensors 1924 may include an inertial measurement unit (IMU) 1926. The IMU 1926 may be an electronic device that generates calibration data based on measurement signals received from accelerometers, gyroscopes, magnetometers, and / or other sensors suitable for detecting motion, correcting errors associated with the IMU 1926, or some combination thereof. Based on the measurement signals, such motion-based sensors, such as the IMU 1926, may generate calibration data indicative of an estimated position of the HMD 102 relative to an initial position of the HMD 102. For example, multiple accelerometers may measure translational motion (forward / backward, up / down, left / right) and multiple gyroscopes may measure rotational motion (e.g., pitch, yaw, and roll). The IMU 1926 may, for example, rapidly sample the measurement signals and calculate an estimated position of the HMD 102 from the sampled data. For example, the IMU 1926 may integrate measurement signals received from an accelerometer over time to estimate a velocity vector, and integrate the velocity vector over time to determine an estimated position of a reference point on the HMD 102. A reference point is a point that may be used to describe the position of the HMD 1900. A reference point may generally be defined as a point in space, but in various embodiments, the reference point is defined as a point within the HMD 102 (e.g., the center of the IMU 1926). Alternatively, the IMU 1926 provides sampled measurement signals to an external console (or other computing device), which determines the calibration data.

[0166] The sensor 1924 may operate at a relatively high frequency to provide sensor data at a high rate. For example, the sensor data may be generated at a rate of 1000 Hz (or one sensor reading every millisecond). In this manner, 1000 readings are generated per second. If the sensor generates this much data at this rate (or even faster), the data set used to predict movement becomes very large, even over a relatively short period of time, such as tens of milliseconds.

[0167] As mentioned, in some embodiments, the sensor 1924 may include a light sensor that is sensitive to light emitted by a base station in the environment of the HMD 102 for purposes of tracking the position and / or orientation, posture, etc. of the HMD 102 in 3D space. Calculations of position and / or orientation may be based on timing characteristics of the light pulses and the presence or absence of light detected by the sensor 1924.

[0168] The HMD 102 may further include an eye tracking module 1928. A camera or other optical sensor inside the HMD 102 may capture image information of the user's eyes, and the eye tracking module 1928 may use the captured information to determine the interpupillary distance, the three-dimensional (3D) position of each eye relative to the HMD 102, including the magnitude of twist and rotation (i.e., roll, pitch, and yaw) and the gaze direction of each eye (e.g., for distortion adjustment purposes). In one example, infrared light is emitted within the HMD 102 and reflected from each eye. The reflected light is received or detected by a camera of the HMD 102 and analyzed to extract eye rotation from changes in the infrared light reflected by each eye. Many methods for tracking the eyes of the user 104 may be used by the eye tracking module 1928. Thus, the eye tracking module 1928 may track up to six degrees of freedom (i.e., 3D position, roll, pitch, and yaw) of each eye, and at least a subset of the tracked quantities may be combined from the two eyes of the user 104 to estimate a point of gaze (i.e., a 3D location or position within the virtual scene where the user is looking). For example, the eye tracking module 1928 may integrate information from past measurements, measurements identifying the position of the user 104's head, and 3D information describing the scene presented by the electronic display 1916. Thus, the position and orientation information of the user 104's eyes is used to determine a point of gaze within the virtual scene presented by the HMD 102 where the user 104 is looking.

[0169] The HMD 102 may further include a head tracking module 1930. The head tracking module 1930 may utilize one or more of the sensors 1924 to track head movements, including head rotation, of the user 104, as described above. For example, the head tracking module 1930 may track up to six degrees of freedom (i.e., 3D position, roll, pitch, and yaw) of the HMD 102. These calculations may be performed at every frame 300 of a sequence of frames 300, so that an application (e.g., a video game) may determine how to render a scene in the next frame 300 and / or the compositor 1910 may determine how to render a reprojected frame 300 depending on the head position and orientation. In some embodiments, the head tracking module 1930 is configured to predict the future position and / or orientation of the HMD 102 based on current and / or past data. This is because the application is required to render a frame 300 before the user 104 actually sees the light (and therefore the image) on the display 1916. Thus, the next frame 300 may be rendered based on this future prediction of head position and / or orientation made at an earlier time, such as approximately 25-30 milliseconds (ms) prior to rendering the frame 300. In a distributed system in which the host computer 106 is communicatively (e.g., wirelessly) coupled to the HMD 102, the future prediction of head pose may be made 30 ms or more prior to the illumination time of the frame 300 to account for network latency, compression operations, etc. The rotational data provided by the head tracking module 1930 may be used to determine both the direction of rotation of the HMD 102 and the amount of rotation of the HMD 102, in any suitable units of measurement. For example, the direction of rotation may be output simply in terms of positive or negative horizontal and positive or negative vertical directions corresponding to left, right, up, and down. The amount of rotation may be in units of degrees, radians, etc. An angular velocity may be calculated to determine the rate of rotation of the HMD 102.

[0170] FIG. 20 illustrates exemplary components of a host computer 106 in which the techniques disclosed herein may be implemented. In the illustrated implementation, the host computer 106 includes one or more processors 2000 and memory 2002 (e.g., computer-readable medium 2002). In some implementations, the processor 2000 may include a CPU, a GPU 304, both a CPU and a GPU 304, a microprocessor, a digital signal processor, or other processing units or components known in the art. Alternatively or additionally, the functionality described herein may be performed, at least in part, by one or more hardware logic components. For example, but not limited to, exemplary types of hardware logic components that may be used include FPGAs, ASICs, ASSPs, SOCs, CPLDs, and the like. In addition, each of the processors 2000 may have its own local memory, which may also store program modules, program data, and / or one or more operating systems.

[0171] The memory 2002 may include volatile and non-volatile memory, removable and non-removable media implemented in any method or technology for storing information, such as computer readable instructions, data structures, program modules, or other data. Such memory includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, DVD, or other optical memory, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, RAID storage systems, or any other medium that can be used to store the desired information and that can be accessed by a computing device. The memory 2002 may be implemented as a CRSM, which may be any available physical medium that is accessible by the processor 2000 to execute the instructions stored on the memory 2002. In one basic implementation, the CRSM may include RAM and flash memory. In other implementations, the CRSM may include, but is not limited to, ROM, EEPROM, or any other tangible medium that can be used to store the desired information and that can be accessed by the processor 2000.

[0172] In general, the host computer 106 may include logic (e.g., software, hardware, and / or firmware, etc.) configured to implement the techniques, functions, and / or operations described herein. The computer-readable medium 2002 is shown as including various modules, such as instructions, data stores, etc., that may be configured to execute on the processor 2000 to perform the techniques, functions, and / or operations described herein. Exemplary functional modules in the form of an operating system 2004, a video game client 2006 including an application 2008, such as a video game (or game app 2010), and a rendering component 2012 are shown stored on the computer-readable medium 2002 and executable on the processor 2000. In some embodiments, additional or different functional modules may be stored on the computer-readable medium 2002 and executable on the processor 2000.

[0173] The operating system 2004 may be configured to manage hardware in and coupled to the host computer 106 for the benefit of the other modules. The video game client 2006 may represent an executable client application configured to launch and execute a program, such as a video game (or video game program). In other words, the video game client 2006 may include game software usable to play a video game on a system including the HMD 102 and the host computer 106. With the video game client 2006 installed, the host computer 106 may have the capability to receive (e.g., download, stream, etc.) a video game from a remote system over a computer network (e.g., the Internet) and execute the video game via the video game client 2006. To this end, any type of content distribution model may be utilized, such as a direct purchase model in which a video game may be purchased separately for download and execution on the host computer 106, a subscription-based model, a content distribution model in which a video game is rented or leased for a period of time, etc. The host computer 106 may thus include one or more video games in the video game library 2014. These video games may be retrieved and run by loading the video game client 2006. In an example, a user 104 may select to play one of a number of video games that they have purchased and downloaded to the video game library 2014 by loading the video game client 2006, selecting the video game, and beginning execution of the video game. The video game client 2006 may enable a user to log into the video game service using credentials (e.g., user account, password, etc.).

[0174] The application 2008 executing on the host computer 106 may be a graphics-based application 2008 (e.g., a video game 2010). The application 2008 is configured to generate pixel data for a series of frames, which are ultimately used to present a corresponding image on the display panel 1916 of the HMD 102. During runtime, for a given frame, the rendering component 2012 may determine a predicted "light time" for the frame. This predicted "light time" for the frame represents the time that light emitting elements of the display panel 1916 of the HMD 102 will light up for the frame. This prediction may take into account, among other things, the inherent latency of the wireless communication link between the host computer 106 and the HMD 102, as well as the predicted rendering time and / or known scan-out time of pixels from the frame buffer. In other words, the prediction for the wireless communication link may differ from the prediction for the wired communication link. For example, the rendering component 2012 may predict the turn-on time a first amount of time in the future for a wired communication link (e.g., approximately 22 milliseconds in the future), while the rendering component 2012 may predict the turn-on time a second, larger amount of time in the future for a wireless communication link (e.g., approximately 44 milliseconds in the future) due to the inherent difference in latency when transferring data over a wired connection versus a wireless connection.

[0175] As described herein, the host computer 106 may also receive head tracking data 208 from the HMD 102 generated by the head tracking module 1930 of the HMD 102. This head tracking data 208 may be generated and / or transmitted at any suitable frequency, such as a frequency corresponding to a target frame rate and / or refresh rate of the HMD 102, or at a different (e.g., faster) frequency, such as 1000 Hz (or one sensor reading every millisecond). The rendering component 2012 is configured to determine, based at least in part on the head tracking data 208, a predicted pose that the HMD 102 will be in at the predicted illumination time. The rendering component 2012 may then provide pose data indicative of the predicted pose to the executing application 2008, and the rendering component 2012 may obtain pixel data 110(1) associated with the frame 300 from the application 2008, in order to render a frame (e.g., generate pixel data for the frame) based on the predicted pose.

[0176] The host computer 106 may further include a communication interface 2016, including, but not limited to, a wireless unit 2018 coupled to an antenna 2020 to facilitate wireless connection to a network and / or a second device, such as the HMD 102. The wireless unit 2018 may implement one or more of a variety of wireless technologies, such as Wi-Fi, Bluetooth, radio frequency (RF), etc. It should be appreciated that the host computer 106 may further include a physical port to facilitate a wired connection to a network and / or a second device, such as the HMD 102.

[0177] Generally, the host computer 106 has input devices 2022 and output devices 2024. The input devices 2022 may include a keyboard, keypad, mouse, touch screen, joystick, control buttons, microphone, camera, etc. The output devices 2024 may include, without limitation, a display, light emitting elements (e.g., LEDs), a vibrator for producing haptic sensations, speakers (e.g., headphones), etc. It should be understood that in some embodiments, a subset of the components shown as being implemented on the HMD 102 may be implemented on the host computer 106 or another computing device separate from the HMD 102 and / or a subset of the components shown as being implemented on the host computer 106 may be implemented on another computing device separate from the HMD 102 or the host computer 106.

[0178] Although the present subject matter has been described in language specific to structural features, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features described. Rather, the specific features are disclosed as example forms of implementing the claims. The inventions described in the claims at the time of filing of this application are set forth below. [1] A head mounted display (HMD), one or more display panels; A processor; and a memory storing computer-executable instructions that, when executed by the processor, cause the HMD to: receiving, from a host computer communicatively coupled to the HMD, an array of motion vectors representing compressed pixel data; decompressing the compressed pixel data to obtain pixel data; modifying the pixel data to obtain modified pixel data, the pixel data comprising: a predicted pose of the HMD; and the motion vector array being modified at least in part based on the motion vector array; and presenting an image on the one or more display panels based at least in part on the modified pixel data. [2] The computer-executable instructions, when executed by the processor, cause the HMD to: generating a rendering mesh comprising a tessellation mesh having a plurality of vertices; moving a vertex of the plurality of vertices to a different position within the rendering mesh as a moved vertex based at least in part on a non-zero motion vector among the motion vector array; The HMD of [1], wherein modifying the pixel data based at least in part on the motion vector array includes moving pixel values ​​of the pixel data according to the moved vertices of the rendering mesh to obtain the modified pixel data. [3] The vertex is in the direction of the non-zero motion vector, and The HMD of [2], wherein the HMD is moved by an amount corresponding to the magnitude of the non-zero motion vector. [4] The HMD of [1], wherein the computer-executable instructions, when executed by the processor, further cause the HMD to extract the motion vector array from an encoded data stream. [5] the host computer is wirelessly coupled to the HMD; The HMD described in [1], wherein the motion vector array is received wirelessly from the host computer. [6] The computer-executable instructions, when executed by the processor, cause the HMD to: and applying a filter to the motion vector array to obtain a modified motion vector array. The HMD of [1], wherein modifying the pixel data based at least in part on the motion vector array includes modifying the pixel data based at least in part on the modified motion vector array. [7] 1. A method comprising: receiving, by a head mounted display (HMD), from a host computer, a motion vector array representing the compressed pixel data; decompressing, by the HMD, the compressed pixel data to obtain pixel data; modifying, by the HMD, the pixel data based at least in part on the motion vector array to obtain modified pixel data; and presenting an image on one or more display panels of the HMD based at least in part on the modified pixel data. [8] prior to said modifying said pixel data; caching the pixel data in a memory of the HMD as cached pixel data; determining that the cached pixel data represents the most recently decompressed pixel data available to the HMD; The method of claim 7, further comprising retrieving the cached pixel data from the memory of the HMD. [9] generating, by the HMD, a rendering mesh including a tessellation mesh having a plurality of vertices; and moving a vertex of the plurality of vertices to a different position within the rendering mesh as a moved vertex based at least in part on a non-zero motion vector among the motion vector array; The method of any one of claims 1 to 7, wherein the modifying of the pixel data based at least in part on the motion vector array comprises moving pixel values ​​of the pixel data according to the moved vertices of the rendering mesh to obtain the modified pixel data.

[10] The method of [7], further comprising extracting the motion vector array from the encoded data stream.

[11] Prior to said presenting of said image, receiving, by the HMD, from the host computer, pose data indicative of a predicted pose of the HMD used to generate the pixel data by an application executing on the host computer; The method of any one of claims 1 to 7, wherein the modification of the pixel data is further based at least in part on a comparison between the predicted pose of the HMD and an updated pose prediction.

[12] Prior to said presenting of said image, receiving, by the HMD, from the host computer, depth data representative of Z-buffer data output by an application executing on the host computer; The method of any one of claims 1 to 7, wherein the modification of the pixel data is further based at least in part on the depth data.

[13] applying, by the HMD, a filter to the motion vector array to obtain a modified motion vector array; The method of any one of claims 1 to 7, wherein the modifying of the pixel data based at least in part on the motion vector array comprises modifying the pixel data based on the modified motion vector array.

[14] 1. A display system comprising: one or more display panels; A processor; and a memory storing computer-executable instructions that, when executed by the processor, cause the display system to: receiving from a host computer a motion vector array representing compressed pixel data; decompressing the compressed first pixel data to obtain pixel data; modifying the pixel data based at least in part on the motion vector array to obtain modified pixel data; and presenting an image on the one or more display panels based at least in part on the modified pixel data.

[15] The computer-executable instructions, when executed by the processor, cause the display system to, before modifying the pixel data: caching the pixel data in the memory as cached pixel data; determining that the cached pixel data represents the most recently decompressed pixel data available to the HMD; and retrieving the cached pixel data from the memory.

[16] The computer-executable instructions, when executed by the processor, cause the display system to: generating a rendering mesh comprising a tessellation mesh having a plurality of vertices; moving vertices of the plurality of vertices to different positions within the rendering mesh as moved vertices based at least in part on non-zero motion vectors among the motion vector array; The display system of claim 14, wherein modifying the pixel data based at least in part on the motion vector array includes moving pixel values ​​of the pixel data according to the moved vertices of the rendering mesh to obtain the modified pixel data.

[17] the host computer is wirelessly coupled to the display system; The display system of

[14] , wherein the motion vector array is received wirelessly from the host computer.

[18] The computer-executable instructions, when executed by the processor, cause the display system, prior to presenting the image, to: receiving, from the host computer, depth data representative of Z-buffer data output by an application executing on the host computer; The display system of

[14] , wherein modifying the pixel data is further based at least in part on the depth data.

[19] The computer-executable instructions, when executed by the processor, cause the display system to: applying a filter to the motion vector array to obtain a modified motion vector array; 14. The display system of claim 13, wherein modifying the pixel data based at least in part on the motion vector array comprises modifying the pixel data based on the modified motion vector array.

[20] The display system of

[14] , wherein the display system includes at least one of a virtual reality (VR) headset or an augmented reality (AR) headset.

Claims

1. A head mounted display (HMD), one or more display panels; A processor; and a memory storing computer-executable instructions that, when executed by the processor, cause the HMD to: receiving, from a host computer communicatively coupled to the HMD, an array of motion vectors representing compressed pixel data; decompressing the compressed pixel data to obtain pixel data; generating a rendering mesh, wherein the rendering mesh comprises a tessellation mesh; moving vertices of the rendering mesh to different positions within the rendering mesh as moved vertices based at least in part on non-zero motion vectors among the array of motion vectors; modifying the pixel data to obtain modified pixel data, the pixel data comprising: A predicted pose of the HMD; and obtaining pixel data that is modified at least in part based on the moved vertices, wherein modifying the pixel data based at least in part on the motion vector array comprises moving pixel values ​​of the pixel data according to the moved vertices of the rendering mesh to obtain the modified pixel data; and presenting an image on the one or more display panels based at least in part on the modified pixel data.

2. The vertex is in the direction of the non-zero motion vector, and The HMD of claim 1 , wherein the motion vector is moved by an amount corresponding to the magnitude of the non-zero motion vector.

3. The HMD of claim 1 , wherein the computer-executable instructions, when executed by the processor, further cause the HMD to extract the motion vector array from an encoded data stream.

4. the host computer is wirelessly coupled to the HMD; The HMD of claim 1 , wherein the motion vector array is received wirelessly from the host computer.

5. The computer-executable instructions, when executed by the processor, cause the HMD to: and applying a filter to the motion vector array to obtain a modified motion vector array. The HMD of claim 1 , wherein modifying the pixel data comprises modifying the pixel data based at least in part on the modified motion vector array.

6. 1. A method comprising: receiving, by a head mounted display (HMD), from a host computer, a motion vector array representing the compressed pixel data; decompressing the compressed pixel data by the HMD to obtain pixel data; generating a rendering mesh by the HMD, wherein the rendering mesh includes a tessellation mesh; moving vertices of the rendering mesh to different positions within the rendering mesh as moved vertices based at least in part on non-zero motion vectors among the array of motion vectors; modifying, by the HMD, the pixel data based at least in part on the moved vertices to obtain modified pixel data, wherein the modifying of the pixel data based at least in part on the moved vertices includes moving pixel values ​​of the pixel data according to the moved vertices of the rendering mesh to obtain the modified pixel data. and presenting an image on one or more display panels of the HMD based at least in part on the modified pixel data.

7. prior to said modifying said pixel data; caching the pixel data in a memory of the HMD as cached pixel data; determining that the cached pixel data represents the last decompressed pixel data available to the HMD; The method of claim 6 , further comprising retrieving the cached pixel data from the memory of the HMD.

8. The method of claim 6 further comprising extracting the motion vector array from an encoded data stream.

9. Prior to said presenting of said image, receiving, by the HMD, from the host computer, pose data indicative of a predicted pose of the HMD used to generate the pixel data by an application executing on the host computer; The method of claim 6 , wherein the modifying of the pixel data is further based at least in part on a comparison between the predicted pose of the HMD and an updated pose prediction.

10. Prior to said presenting of said image, receiving, by the HMD, from the host computer, depth data representative of Z-buffer data output by an application executing on the host computer; The method of claim 6 , wherein the modifying of the pixel data is further based at least in part on the depth data.

11. applying, by the HMD, a filter to the motion vector array to obtain a modified motion vector array; The method of claim 6 , wherein the modifying the pixel data comprises modifying the pixel data based on the modified motion vector array.

12. 1. A display system comprising: one or more display panels; A processor; and a memory storing computer-executable instructions that, when executed by the processor, cause the display system to: receiving from a host computer a motion vector array representing compressed pixel data; decompressing the compressed pixel data to obtain pixel data; generating a rendering mesh, wherein the rendering mesh comprises a tessellation mesh; moving vertices of the rendering mesh to different positions within the rendering mesh as moved vertices based at least in part on non-zero motion vectors among the array of motion vectors; modifying the pixel data based at least in part on the moved vertices to obtain modified pixel data, wherein the modifying the pixel data based at least in part on the moved vertices includes moving pixel values ​​of the pixel data according to the moved vertices of the rendering mesh to obtain the modified pixel data. presenting an image on the one or more display panels based at least in part on the modified pixel data.

13. The computer-executable instructions, when executed by the processor, cause the display system to, before modifying the pixel data: caching the pixel data in the memory as cached pixel data; determining that the cached pixel data represents the most recently decompressed pixel data available to the display system; and retrieving the cached pixel data from the memory.

14. the host computer is wirelessly coupled to the display system; The display system of claim 12 wherein the motion vector array is received wirelessly from the host computer.

15. The computer-executable instructions, when executed by the processor, cause the display system, prior to presenting the image, to: receiving, from the host computer, depth data representative of Z-buffer data output by an application executing on the host computer; The display system of claim 12 , wherein modifying the pixel data is further based at least in part on the depth data.

16. The computer-executable instructions, when executed by the processor, cause the display system to: applying a filter to the motion vector array to obtain a modified motion vector array; 13. The display system of claim 12, wherein said modifying said pixel data comprises modifying said pixel data based on said modified motion vector array.

17. 13. The display system of claim 12, wherein the display system comprises at least one of a virtual reality (VR) headset or an augmented reality (AR) headset.

18. The display system of claim 12, wherein the computer-executable instructions, when executed by the processor, further cause the display system to extract the motion vector array from an encoded data stream.

19. The computer-executable instructions, when executed by the processor, cause the HMD to, prior to presenting the image: receiving, from the host computer, depth data representative of Z-buffer data output by an application executing on the host computer; The HMD of claim 1 , wherein modifying the pixel data is further based at least in part on the depth data.

20. The computer-executable instructions, when executed by the processor, cause the HMD to, before modifying the pixel data: caching the pixel data in the memory as cached pixel data; determining that the cached pixel data represents the last decompressed pixel data available to the HMD; and retrieving the cached pixel data from the memory.

Images