Motion compensation of player input via expected motion vectors

Abstract

The present invention discloses player input motion compensation through expected motion vectors. The system and method of the present invention incorporates a client device that uses a lookup table from a remote server to match, label and sum motion vectors. When the remote server sends the encoded video frames to the client, the client decodes them and then applies the summed motion vectors to estimate motion. The technology also generates motion vectors based on predetermined criteria and sends the motion vectors and invalidators to the client for caching. The server instructs the client to receive input and use the input to match the cached motion vector or invalidator. The technology also caches duplicate motion vectors and sends a previously generated motion vector library to the client. The server instructs the client to calculate the motion estimation result and instructs the client to update the stored motion vector library so that the client applies the stored library to initiate motion before receiving actual motion vector data.

Description

[0001] This invention patent application is a divisional application of the invention patent application with international application number PCT / US2018 / 028620, international application date April 20, 2018, application number 201880041720.9 entering the Chinese national phase, and name “Player Input Motion Compensation via Expected Motion Vectors”.

[0002] Related Applications

[0003] This application claims the benefit of the following U.S. Provisional Applications: 62 / 488,526 filed April 21, 2017, 62 / 634,464 filed February 23, 2018, 62 / 640,945 filed March 9, 2018, and 62 / 644,164 filed March 16, 2018. Background Art

[0004] Remote gaming applications, where a server-side game is controlled by a client-side player, have attempted to use existing or customized encoders to encode video output from a three-dimensional (3D) graphics engine in real time. However, the interactivity of video games, especially the player feedback loop between video output and player input, makes game video streaming more sensitive to latency than conventional video streaming. Existing video encoding methods can sacrifice computing power to reduce encoding time, with little alternative. A new approach that integrates the encoding process into the video rendering process can provide a significant reduction in encoding time while also reducing computing power, thereby improving the quality of the encoded video and maintaining the original bitstream data format to maintain interoperability with existing hardware devices.

[0005] Unlike regular video playback, video games have a unique player input for the video feedback loop. Players are very sensitive to the delay between input and video output. High latency in such a player input feedback loop is a huge stumbling block for video game streaming in applications where server-hosted video game instances are controlled by remote players. Any process that can reduce the time between input and feedback will directly improve the user experience.

[0006] Client hardware in a game streaming environment can have varying levels of computing performance, but dedicated H.264 hardware decoders are increasingly common, even in mobile and other low-power devices. Hardware decoders are good at performing small amounts of computation, such as motion compensation, which is often performed regularly according to the H.264 encoding standard. Regardless of the overall computing power of the client, it is possible to take advantage of dedicated decoding hardware to provide a better player experience in a game streaming environment.

[0007] In local, non-streaming rendering applications, the game engine can add a few frames of latency between player input and video feedback. In game streaming applications, additional latency is introduced into the player input-feedback loop because the player input must be transmitted over the network to a remote server and the video output must be encoded, transmitted, and decoded before the player receives feedback. For some player inputs, the client can estimate the result of the video feedback by performing motion compensation immediately, thereby reducing network latency.

[0008] Player input motion compensation is basically a technique that moves groups of pixels to reduce input-feedback latency at the expense of some graphical accuracy when a video game is running on a server and controlled remotely by a client connected to the network. The technique excels at reducing player feedback latency for input, resulting in coherent motion vectors, such as rotation of the player's view in a first-person game.

[0009] In video games, the player scene is defined as the current game state, which is the result of previous player actions, inputs, and decisions. The client in the game streaming system is simple for the player scene, that is, the client only receives the video output without receiving any game state information that will determine the results of a specific player input. There are a wide range of inputs that produce unique but predictable motion results based on the game state information. These inputs will benefit from the reduction of player-feedback delays, but because the client will not have player scene information, these inputs cannot be pre-cached on the client to compensate for conventional player input motion. In addition, the player scene replacement space may be too small to pre-generate motion vectors for methods such as caching repeated motion vectors. These systems and methods are described in U.S. Provisional Application Nos. 62 / 488,526, 62 / 634,464, and 62 / 640,945; the entire contents of these three applications are incorporated into this application. The game server can compensate by generating expected motion vectors and updating the client's input motion compensation cache when the player scene changes. This allows clients to use player input motion compensation techniques for a limited set of scene-dependent inputs, resulting in reduced input feedback latency.

[0010] U.S. Patent No. 9,661,351 (“the '351 patent”) discloses a system and method for skipping frames during transmission from a server to a client device, wherein, in response to detecting a skipped frame, the client device generates a predicted frame that replaces the skipped frame in a compressed video stream, the predicted frame being generated based on delta information extended from one or more previous frames decoded by the client device. For the technique, client-side frame prediction of one or more reconstructed or predicted frames following the skipped frame is performed based on data (e.g., motion vectors, residuals, etc.) of one or more previous frames. The technique also prioritizes bit allocation and / or sub-feature encoding. A coded network abstraction layer unit (NALU) may be partitioned into (1) motion vectors and (2) residuals. A device may prioritize sending only minimal coded data instead of actually skipping a frame. For example, if motion has been prioritized, only motion vectors may be sent. The present invention is superior to the techniques of the '351 patent at least because the '351 patent does not disclose a client device that uses a lookup table sent from a server to match user input to motion vectors and labels and sums those motion vectors. The '351 patent also does not disclose applying those summed motion vectors to decoded frames to estimate motion in those frames. The present invention is also advantageous because it reduces input-feedback latency, which is significantly reduced by using player input motion compensation rather than waiting for the server to return output video.

[0011] U.S. Patent No. 8,678,929 (the '929 patent) relates to the operation of a networked, interactive gaming system. The disclosed method is intended to reduce network lag by determining a route-corrected position value of a shared global object by bidirectional blending. The "blending" discussed in the patent includes the step of sending a network simulation of the results of a computation of a local player's gesture on a local console. The console blends the local emulation and the network emulation. The method also blends the shared global object using the blended gesture of the local player to determine a position value for the shared global object at the local console. The present invention is also superior to the techniques of the '929 patent, at least because the '929 patent does not disclose a client device that uses a lookup table sent from a server to match user inputs to motion vectors and labels and sums these motion vectors. The '929 patent also does not disclose applying those summed motion vectors to decoded frames to estimate motion in those frames. The present invention is also advantageous because the present invention does not require the presence of a client with extremely high processing power to reduce input-feedback delay, which is significantly reduced by using player input motion compensation rather than waiting for the server to return output video.

[0012] U.S. Patent No. 8,069,258 (the "'258 Patent") is directed to using local frame processing to reduce apparent network lag in a multiplayer simulation. The method described includes intercepting input from a local user, determining state data of remote objects in the network simulation from previous game frames, and determining interactions of indeterminate objects from a multi-game system that is part of the networked simulation. The interaction data is used together with the state data and the local input to generate a local simulation of a video frame. In this way, for a single frame, the local simulation and the network simulation can run asynchronously, with each frame corresponding to a single time phase within the game. This allows the local simulation to be updated in real time during a networked game while maintaining (substantial) synchronization with the network. The present invention is also superior to the techniques of the '258 Patent, at least because the '258 Patent does not disclose a client device that uses a lookup table sent from a server to match user inputs to motion vectors and labels and sums these motion vectors. The '929 Patent also does not disclose applying those summed motion vectors to decoded frames to estimate motion in those frames. The present invention is also advantageous because it does not require the presence of a client with extremely high processing power to reduce input-feedback latency, which is significantly reduced by using player input motion compensation rather than waiting for the server to return output video.

[0013] U.S. Patent No. 9,665,334B2 ("the '334 patent") discloses a system and method for a rendering protocol that applies multiple processes and a compositor to render a combined image on a display. The technique operates as follows: When a server provides a game screen to several client devices simultaneously, the computational load of the rendering process on the server becomes heavy, for example, for game content that requires a fast response. That is, the number of client devices to which the server can provide a screen is limited by the rendering performance of the server and the required response. In contrast, when each client device is controlled to perform a process that can share the rendering process between the server and the client device through general rendering performance, the screen can be provided to more client devices. Moreover, generally, the game screen rendered without applying texture mapping has a high compression efficiency and can be sent via a network such as the Internet with a smaller bandwidth. The present invention is superior to the techniques discussed in the '334 patent, at least because the '334 patent does not disclose generating motion vectors at the server based on predefined criteria and sending the generated motion vectors and one or more invalidators to the client, which caches these motion vectors and invalidators. The '334 patent also does not disclose that the server instructs the client to receive input from the user and use the input to match cached motion vectors or invalidators, where these vectors or invalidators are used in motion compensation. The present invention is also advantageous because it reduces input-feedback latency, which is significantly reduced by using player input motion compensation instead of waiting for the server to return output video.

[0014] U.S. Patent No. 9,736,454 (the "'454 Patent") discloses systems and methods for encoding, the systems and methods comprising checking the availability of a depth block collocated with a texture block, determining a prediction method for the texture block based on the availability of the collocated depth block, and deriving a first prediction block for the texture block based on the collocated depth block. The present invention is also superior to the techniques discussed in the '454 Patent at least because the '454 Patent does not disclose generating motion vectors at a server based on predefined criteria and sending the generated motion vectors and one or more invalidators to a client, wherein the client caches these motion vectors and invalidators. The '454 Patent also does not disclose that the server instructs the client to receive input from a user and use the input to match the cached motion vectors or invalidators, wherein the vectors or invalidators are used in motion compensation. The present invention is also advantageous because the present invention reduces input-feedback latency, which is significantly reduced by using player input motion compensation instead of waiting for the server to return output video.

[0015] U.S. Patent No. 9,705,526 (the "'526 Patent") discloses systems and methods for entropy encoding in media and imaging applications. The technology discloses a system in which, as indicated, compression begins with receiving a source of image and / or video data. A lossless compression scheme is then applied. A predictor / delta calculation unit then accepts input and attempts to reduce redundancy in the input data using delta calculations between adjacent input elements. These values ​​are then encoded using a predefined statistical model in an entropy encoder to generate compressed image and / or video data. Similar to the above patents, the present invention is superior to the technology discussed in the '526 Patent at least because the '526 Patent does not disclose generating motion vectors at a server based on predefined criteria and sending the generated motion vectors and one or more invalidators to a client, which caches these motion vectors and invalidators. The '526 Patent also does not disclose that the server instructs the client to receive input from a user and use the input to match the cached motion vectors or invalidators, where these vectors or invalidators are used in motion compensation. The present invention is also advantageous because it reduces input-feedback latency, which is significantly reduced by using player input motion compensation rather than waiting for the server to return output video.

[0016] U.S. Patent No. 8,873,636 B2 (the "'636 Patent") is directed to a motion image distribution server that provides encoded image data to a user computer running a local instance of a game (e.g., a server running an online game). To perform the process, in relevant detail, a CPU of the user computer specifies an area to be referenced in order to decode a motion vector associated with a selected block in a previous frame screen. This operation is performed by referencing a motion vector associated with the selected block (a vector included in pre-processing information provided by the selected block) and extracting an image of the area as a reference image. As with other references, the present invention is superior to the techniques discussed in the '636 Patent at least because the '636 Patent does not disclose generating motion vectors at a server based on predefined criteria and sending the generated motion vectors and one or more invalidators to a client, which caches the motion vectors and invalidators. The '636 Patent also does not disclose that the server instructs the client to receive input from the user and use the input to match the cached motion vectors or invalidators, where the vectors or invalidators are used in motion compensation. The present invention is also advantageous because it reduces input-feedback latency, which is significantly reduced by using player input motion compensation rather than waiting for the server to return output video.

[0017] International Patent Publication WO2009138878 A2 ("the '878 Publication") relates to processing and streaming multiple interactive applications in a centralized streaming application server while controlling the level of detail and post-filtering of each rendered object. In the system, the centralized interactive application server, in its video preprocessor, performs spatial and temporal filtering on a sequence of frames before encoding a compressed stream of audiovisual content sent to a client device, which decodes the compressed stream and displays the content. The GPU command processor of the centralized interactive application server includes a module that also calculates motion compensation estimation results for each macroblock in a target encoded frame in the video encoder. Still, the present invention is superior to the techniques discussed in the '878 Publication, at least because the '878 Publication does not disclose generating motion vectors at the server based on predefined criteria and sending the generated motion vectors and one or more invalidators to the client, which caches these motion vectors and invalidators. The '878 Publication also does not disclose that the server instructs the client to receive input from the user and use the input to match the cached motion vectors or invalidators, where these vectors or invalidators are used in motion compensation. The present invention is also advantageous because it reduces input-feedback latency, which is significantly reduced by using player input motion compensation rather than waiting for the server to return output video.

[0018] U.S. Patent No. 9,358,466 B2 (the "'466 Patent") relates to improving the performance of video games through the reuse of cached data. The system discloses scoring the cache performance of different generated video game tasks at least in part by identifying the digital assets used and determining whether the identified digital assets are present in the cache. The cache score can be calculated based on the cache reuse rate corresponding to the proportion of the digital assets of the task that are already present in the cache. Other techniques for generating the cache score may consider factors such as the overall size of the combined digital assets of the tasks that are already present in the cache and / or the overall size of the combined digital assets of the tasks that are not present in the cache. By caching data in this manner, requests for that data and uncached data become more efficient. The present invention is still superior to the techniques discussed in the '466 Patent, at least because the '466 Patent does not disclose caching repeated motion vectors, calculating motion estimation results based on input data, or updating a stored motion vector library based on input data, so that the client can use the stored motion vector library to initiate motion before receiving actual motion vector data from the server.

[0019] U.S. Patent No. 6,903,662 B2 (the "'662 Patent") relates to a configurable computer input device, and in particular, a configurable computer input device that caches input data to maintain faster response times. The system operates by checking a key press against an internal memory (or cache) to determine whether the particular key has been previously recognized. If the system has not previously communicated with the key, the system can then retrieve data corresponding to the input from the system key memory. The system will then update the system's internal memory (cache) with the key identification and its corresponding data. The system can then send the input data from the key to the host computer. However, the next time the system encounters the same key in a pressed state, the system is able to query the system's own memory (cache) rather than retrieving the same scan code from the key memory again. The present invention is superior to the techniques discussed in the '662 Patent at least because the '662 Patent does not disclose caching repeated motion vectors, calculating motion estimation results based on input data, or updating a stored motion vector library based on input data, so that the client can use the stored motion vector library to initiate motion before receiving actual motion vector data from the server.

[0020] Japanese Patent JP6129865B2 ("the '865 Patent") discloses a system and method for sending rendered game content data for a subset of path segments to a game client for caching on a local cache, so that the game content data can be used when needed during real-time game play. Again, the present invention is superior to the techniques discussed in the '865 Patent, at least because the '865 Patent does not disclose caching repeated motion vectors, calculating motion estimation results based on input data, or updating a stored motion vector library based on input data, so that the client can use the stored motion vector library to initiate motion before receiving actual motion vector data from the server.

[0021] U.S. Patent No. 9,762,919 (the "'919 patent") discloses a system and method for caching reference data in a block processing pipeline. The '919 patent discloses a cache memory (e.g., a fully associative cache) that may be implemented in a local (to the pipeline) memory such as an SRAM (static random access memory) from which portions (e.g., 64-byte memory blocks) of chroma reference data corresponding to motion vectors determined for a macroblock at an early stage in the pipeline may be pre-fetched. Chroma cache logic may maintain a cache and may extend across multiple stages of the pipeline. Fetching of the motion vector for a given macroblock passing through the pipeline may be initiated by chroma cache logic at one or more stages prior to the chroma motion compensation stage to provide time (i.e., multiple pipeline cycles) to read the corresponding memory block from memory into the cache before the corresponding memory block is needed for chroma motion compensation. However, the '919 patent is still insufficient compared to the present invention. The present invention is superior to the techniques discussed in the '919 patent at least because the '919 patent does not disclose caching repeated motion vectors, calculating motion estimation results based on input data, or updating a stored motion vector library based on input data, so that the client can use the stored motion vector library to initiate motion before receiving actual motion vector data from the server.

[0022] From the above discussion of the prior art in the art, it is apparent that there is a need in the art to improve upon existing computer technology related to the encoding of real-time gaming environments. Summary of the invention

[0023] It is therefore an object of the present invention to disclose a system and method for reducing latency through motion compensation techniques, wherein a client device uses a lookup table sent from a server to match user input to motion vectors and to label and sum these motion vectors. When a remote server sends encoded video frames to a client, the client decodes the video frames and applies the summed motion vectors to the decoded frames to estimate motion in the frames.

[0024] It is another object of the present invention to disclose a system and method in which encoded video frames are decoded without residual processing.

[0025] It is yet another object of the present invention to disclose a system and method for reducing delay through motion compensation techniques, wherein the client applies one or more smoothing functions to the summed tagged motion vectors in a queue.

[0026] It is yet another object of the present invention to disclose a system and method for reducing latency through motion compensation techniques, wherein labels associated with motion vectors at a client device are actually time sequential.

[0027] Another object of the present invention is to provide a system and method that reduces input-feedback delay by generating motion vectors at a server based on predefined criteria and transmitting the generated motion vectors and one or more invalidators to a client, which caches these motion vectors and invalidators. The server instructs the client to receive input from the user and use the input to match the cached motion vector or invalidator. Based on the comparison results, the client then applies the matched motion vector or invalidator to implement motion compensation in a graphical interface.

[0028] Another object of the present invention is to provide a system and method for reducing input-feedback delay by caching motion vectors in a lookup table.

[0029] It is yet another object of the present invention to provide a system and method for reducing input-feedback delay by associating an invalidator with one or more buffered motion vectors.

[0030] It is yet another object of the present invention to provide systems and methods that reduce input-feedback delay by instructing a client to delete one or more cached motion vectors when an input matches a cached invalidator associated with the one or more motion vectors.

[0031] Another object of the present invention is to disclose a system and method for reducing latency by caching duplicate motion vectors at a server that sends a previously generated motion vector library to a client. The client stores the motion vector library and monitors user input data. The server instructs the client to calculate motion estimation results based on the input data and instructs the client to update the stored motion vector library based on the input data, so that the client applies the stored motion vector library to initiate motion in a graphical interface before receiving the actual motion vector library from the server.

[0032] It is another object of the present invention to disclose a system and method for reducing latency by caching repeated motion vectors, wherein a server sends a scene update to a client to disable application of a stored library of motion vectors.

[0033] It is another object of the present invention to disclose a system and method for reducing delay by caching repeated motion vectors, wherein one or more scaling factors are applied to a pool of motion vectors. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The present invention will be better understood, and a more complete appreciation of the invention and its many attendant advantages will be readily obtained, by referring to the following detailed description when considered in conjunction with the accompanying drawings, in which:

[0035] Figure 1 is a block diagram exemplarily illustrating a remote gaming system in which a video game is run on a server while being controlled by input at a remote client;

[0036] Figure 2 is a flow chart exemplarily illustrating reducing input-feedback latency in a game streaming application by applying motion compensation to appropriate player input;

[0037] Figure 3 is a block diagram illustratively illustrating exemplary moments during operation of a video game streaming environment utilizing player input motion compensation;

[0038] FIG. 4 is a diagram showing an exemplary Figure 3 Figure 2 is a diagram of an exemplary macroblock during player input motion compensation;

[0039] Figure 5 is a diagram exemplarily illustrating an alternative method of applying motion vectors during player input motion compensation on a client;

[0040] Fig. 6A , 6B and 6C shows about Figure 5 An alternative method for performing player input motion compensation and exemplary macroblock blending is shown in FIG.

[0041] Figure 7 is a diagram showing the generation of expected motion vectors at runtime;

[0042] Figure 8 is a flow chart exemplarily illustrating the transmission and storage of expected motion vectors on the client side for the purpose of player input motion compensation;

[0043] Fig. 9 is a diagram illustrating an exemplary process of invalidating an expected motion vector;

[0044] Fig.10 is a diagram illustrating an exemplary method of generating a motion vector library and an exemplary duplication of the motion vector library for caching purposes;

[0045] Fig.11is a flow chart illustrating, by way of example, a process of caching, applying, and updating a motion vector library for player input motion compensation;

[0046] Fig.12 is a diagram exemplarily showing how a motion vector map may be updated; and

[0047] Fig.13 is a diagram illustrating exemplary modifications to applying multi-frame motion vectors during player input motion compensation, particularly in the context of cached motion vectors. DETAILED DESCRIPTION

[0048] In describing the preferred embodiments of the present invention shown in the accompanying drawings, specific terms will be used for the sake of clarity. However, the present invention is not intended to be limited to the specific terms so selected, and it should be understood that each specific term includes all technical equivalents that operate in a similar manner to achieve similar purposes. For illustrative purposes, several preferred embodiments of the present invention are described, and it should be understood that the present invention can be implemented in other forms not specifically shown in the accompanying drawings.

[0049] Certain types of player input are better choices for player input motion compensation. Two factors contribute to the suitability of a given input: the player's sensitivity to input-feedback delays, and the difficulty of implementing player input motion compensation without introducing significant artifacts. Each input will need to be evaluated for suitability. For example, in a first-person shooter game, the player will be very sensitive to mouse-based perspective rotation, and a delay of less than 1 second, such as 16ms, between the player input and the video output will be noticeable. However, in the same case, the perspective rotation based on a gamepad controller is usually slower, and the player may not be so sensitive to input-feedback delays. Perspective rotation can be approximated by moving the scene in the opposite direction of the rotation, but undesirable artifacts may appear along the edge of the image in the direction of rotation. For small perspective rotations, for example, adjusting the aiming of an enemy on the screen, the player may not even notice the edge artifacts. In another example, for player input motion compensation, due to the lack of player sensitivity to delays and / or inertia, accelerating the vehicle in a driving game may be a low priority, but steering and braking inputs may be a high priority, because the player will notice the input-feedback delay.

[0050] The time between receiving player input and displaying motion output is player-feedback latency. By using motion compensation, the estimated motion can provide feedback almost immediately while waiting for the server to process the player input. In this way, player-feedback latency is significantly reduced in game streaming applications. By implementing user input motion compensation in a game streaming application, motion estimation can be provided in the next available frame. In contrast, it takes several frames to send input to the server, generate output frames, and return. Player input motion compensation can also provide some benefits in conventional non-streaming game applications where the game engine and renderer may have a player-feedback latency of several frames.

[0051] The client will not have a suitable scene to track objects moving around the screen. Player input motion compensation will not work for situations where the client does not know the location of a specific macroblock or video object. For example, in a 2D platformer, a character can move around the screen from left to right. When the player presses the jump input, the client will not know where the character is, so player input motion compensation cannot be used alone to reduce input-feedback latency in this case.

[0052] Typically, motion vectors for player input motion compensation should be generated in advance. For motion such as player camera rotation, motion vectors can be calculated based on how the game weights the input. In some embodiments, the motion vector can be the input value multiplied by the sensitivity weight. For motion that cannot be directly calculated, such as animated motion, animations can be triggered during development so that motion vectors can be directly measured and stored. The measurement of motion vectors can be done through the same motion estimation techniques performed during H.264 encoding.

[0053] Figure 1 An exemplary system is shown in which a video game is controlled by a remote client. In this system, a server 100 hosts video game software 102 and a graphics engine 104 that renders video output. The video is encoded in a codec (also referred to as an encoding engine or encoder) 106 and the encoded video data 108 is sent to the remote client. The server architecture 100 can be any combination of hardware or software capable of supporting the functionality of both the graphics engine 104 and the codec 106. In a given example, the graphics engine 104 can be implemented as a GPU 110 controlled by the video game software 102 loaded into some computer readable memory 112, while the codec 106 can be implemented as a CPU 114 running video encoding software.

[0054] The remote client includes a computer system 116 capable of running a client-side codec 118 to decode the transmitted encoded video data 108 and a client application 120 to apply player input motion compensation. The client computer system 116 also includes a display controller 122 to drive display hardware 124. Input from the client-side input peripherals 126 will be converted by the client application 120 into control data 128, and the control data will be sent back to the game software 102 running on the server 100. The input from the peripherals 126 will also be used for example Figure 2 Determination of what player motion compensation to apply (if any) is shown in greater detail.

[0055] Figure 2 is a flow chart describing the steps required to perform player input motion compensation for a single input. At step 200, upon client initialization, the server sends a lookup table including the input and the motion vector associated with the input, and then at step 202, the lookup table is cached by the client. In this embodiment, the client is generic to meet the needs of multiple streaming games. In some embodiments, a game-specific client can skip steps 200 and 202 because the game-specific client already includes a lookup table for the game. In an alternative embodiment, the game-specific client can permanently store the motion vector lookup table without caching it from the server.

[0056] When the client receives player input from an input device such as a mouse or gamepad controller in step 204, the client application will check the cached motion vector lookup table for matching input in step 206. If there is no matching player input, the client will not perform any additional actions and send the input to the server without additional modification. If there is a matching player input in the cache, the client will apply player input motion compensation. Optionally, the cache may be able to change entries in the lookup table based on player input. For example, when a player presses a pause button, all player movement inputs should be disabled until the player exits the pause screen. In one embodiment, the client-controlled lookup table can have two sets of inputs, one set of inputs used in the pause menu and one set of inputs used outside the pause menu, and the two sets of inputs are preferably switched by the client whenever the player selects a pause input. In an alternative embodiment, the server can switch the contents of the cached lookup table on the client.

[0057] In step 208, when the client application receives player input for motion compensation, the client will tag the player input and the motion vector associated with the player input. In step 210, the tagged input is sent to the server. A tag is any identifier that can associate the player input to a future frame. For example, a tag can be an integer that increments each time the client receives input that will be used to perform player input motion compensation. The tag can be added as metadata to the same network packet as the player input or sent in a similar messaging mode to keep the tag information synchronized with the input information. In step 213, the client confirms whether the tagged tag is received. In step 212, when the player input is tagged and sent, the client applies the motion vectors included in the cached lookup table. These motion vectors will be applied to each incoming frame until the relevant tags are returned from the server. Figure 3 A detailed description of an exemplary method of applying these motion vectors is shown.

[0058] At step 214, when the server receives the tagged player input, the tagged player input is passed to the game, which generates an output frame. Then, at step 216, the video image is encoded. At step 218, before the encoded frame is sent back to the client, the player input tag is attached to the encoded frame. This is the same frame that was previously sent with the player input, and indicates that the output frame includes actual video feedback from the player input. Attaching tags to the encoded frame can be done by adding the tag as metadata to the same network packet as the encoded frame. At step 220, the tagged encoded frame is sent back to the client. When the client receives the tagged encoded frame, the client can associate the tag with the previous player input motion compensation. Then, at step 222, the client stops applying the previous motion compensation.

[0059] Figure 3 is an illustration of an exemplary moment during the run of a video game streaming environment utilizing player input motion compensation. When any player input is received by the client at step 300, it is compared to a motion vector lookup table at step 302. If there is a matching player input, the associated motion vector is used for player input motion compensation. At step 306, the motion vector is tagged with the unique input tag of step 304. In this example, the tag selected is the integer "1003". At step 308, the tagged motion vector is added to a queue that includes any other tagged motion vectors currently being used for player input compensation.

[0060] At step 320, the next frame arrives in the bitstream from the server. At step 318, the frame is tagged with a unique identifier, in this case the integer "1001", which represents a frame that includes the composite motion of all previous player inputs up to and including the input corresponding to the tag "1001". At step 322, the tag "1001" indicates to the client that it can stop applying motion compensation for this tagged input. Then, at step 308, the motion vector with the tag "1001" is removed from the tagged motion vector queue along with any motion vectors with earlier tags that may remain in the queue in the event that a previous packet has been lost.

[0061] At step 324, the encoded video 316 is decoded. Meanwhile, at step 310, the remaining motion vectors in the motion vector queue of step 308 are summed. A motion vector is typically a vector field with a vector for each macroblock in the image. To sum the motion vectors, the vectors are summed element-wise so that the result is a vector field with a vector for each macroblock. The sum of two vector fields is the sum of the vectors for each point in the field, so the sum of the two sets of motion vectors is the sum of the vectors for each macroblock in the image. The sum of two vectors is defined as the component-by-component sum of the components of the two vectors, which can be expressed as: {u1,u2}+{v1,v2}={u1+v1,u2+v2}. In this example, two sets of motion vectors with labels "1002" and "1003" are included in the queue; these two sets of motion vectors are summed. Labels are temporally ordered in nature, which allows the client to know the order of previously tagged player inputs. This lets the client discard tagged motion vectors up to and including return labels in the incoming frame. Additionally, the labeling described above is computationally cheaper than more complex methods.An optional smoothing function can be applied at this juncture to prevent clamping artifacts or mitigate similarly introduced artifacts.

[0062] Clamping artifacts are introduced when macroblocks move off screen and appear as a blur of pixel colors perpendicular to the edge of the screen. The exemplary smoothing function reduces the strength of outward pointing vectors as they get closer to the edge of the image. Only the outward components need to be weakened, with the y component of the outward pointing vector towards the top and bottom edges of the image and the x component of the outward pointing vector towards the left and right edges of the image. For vectors pointing towards the boundary, the vector components can be multiplied by the square of the distance from the boundary so that the vector components will approach zero as the distance from the boundary approaches zero. In this example, the outward pointing vector towards the right edge of the image will be transformed from {x,y} to {x*d*d,y}, where d is the distance from the edge. This will reduce the clamping artifacts, but in exchange for some slight image distortion near the image boundary. To the player, this distortion is far less noticeable than the clamping artifacts.

[0063] After the decoding process of the encoded video frame is completed in step 324, motion compensation is performed using the summed motion vectors in step 312. In step 314, the resulting video is output. The output includes the player input motion compensation data and will be displayed on the client. The frame of the output is the first frame that includes motion estimation for the input with the associated tag "1003". The frame of the output also includes motion estimation for the previous input with the associated tag "1002", for which the client is still waiting for the server to return the actual motion vector. The frame of the output is also the first frame that includes the actual motion vector for the previous input with the associated tag "1001", for which the client has previously estimated motion. As a result, there are three motion estimation states in the method at this time: a new estimation state for the new input; a continuing estimation state waiting for the actual result; and another estimation state that is stopped because the actual result has reached the client.

[0064] FIG. 4 shows Figure 3 Figure 2.2 shows an example macroblock during the player input motion compensation step of . The player input motion compensation step is procedurally similar to the process performed during H.264 decoding, where the decoded motion vectors are used to determine where macroblocks in the current frame were moved from in the previous frame. For player input motion compensation, the applied motion vectors will estimate future feedback for player input by moving macroblocks in the current frame before displaying the output video to the player. Figure 4A Two sets of tagged motion vectors are shown in the queue of step 400, the two sets of tagged motion vectors are from Figure 3 The two sets are summed by performing a vector summation for each macroblock in the image to create Figure 4B =A set of motion vectors in step 402. Each of the motion vectors in the set represents an estimated motion for each macroblock in the current frame. For each vector in the summed motion vectors in step 402, the motion Figure 4C In step 404, Figure 4C An exemplary macroblock shown in is shifted with its corresponding player input motion vector. Each macroblock in the exemplary image will be shifted in this manner. Figure 4C The exemplary image in FIG. 4 includes only 50 macroblocks, while a high resolution image will include thousands of macroblocks. The exemplary motion vectors shown in FIG. 4 show uniform rigid motion, but the player input motion compensation techniques described can be used with motion vectors of arbitrary complexity to describe rotations, vibrations, or other complex motions in screen space as known in the art.

[0065] Figure 5is an illustration of an alternative method of applying motion vectors during player input motion compensation on the client; in certain embodiments of this method, it is not necessary to process residuals during encoding and decoding of the video. After a matching player input is found in the lookup table, the associated motion vector is tagged at step 500 and used to Figure 2 , and player input motion compensation shown in steps 208 and 212 of . During the decoding process for the next frame, the player input motion vector is added to the decoded motion vector and injected into the motion compensation step in step 502 defined in the H.264 encoding standard. The deblocking filter defined by the H.264 encoding standard is applied in step 504, but should only be applied to blocks that have not been modified by player input motion compensation. The resulting output is shown in step 506, which includes player input motion compensation and will be displayed on the client. The output is then cached in steps 506 and 518 and becomes the previous image used in motion compensation for the next frame. Figure 3 , in which the tagged player input vector is only applied once. This means that the tagged player input motion vector does not need to be summed with all the tagged motion vectors in the queue, because the previous image shown in step 518 will include the sum of the previous tagged motion vectors. The time between receiving input and displaying video output is the input-feedback delay, which will be significantly reduced by using player input motion compensation instead of waiting for the server to return output video.

[0066] At the same time as step 500, the client will send the corresponding tagged player input to the server, such as Figure 2 As shown in steps 208 and 210. Figure 2 As shown in step 220 of FIG. 1 , the client will eventually receive the encoded video shown in step 508 with the same tag 510 in the bitstream from the server in step 512. Figure 2 220 in step 220. The tag 510 associated with the video indicates that the actual motion previously estimated by the player input motion compensation is shown in the encoded video as shown in step 508. As defined by the H.264 encoding standard, the encoded video is passed through entropy decoding at step 514 and inverse quantization and inverse transformation at step 516.

[0067] The previous player input motion compensation step, which has been performed before steps 508, 510, 512, 514 and 516 described in the previous paragraph, has shifted the macroblock, that is, the actual motion vector from the encoded frame is trying to shift. Therefore, directly applying the actual motion vector will shift the incorrect macroblock. The bitstream will include two related data: the encoded video frame and the associated tag. The encoded frame is sent for entropy decoding in step 514, while the tag was previously sent for the blending process in step 520. To compensate, the blending process in step 520 provides a correction motion vector by calculating the difference between the incoming actual motion vector and the previously used player input motion vector 500 with a matching associated tag. This difference can be calculated by adding the inverse motion vector for the associated tag and the actual motion vector, and is described in more detail in conjunction with Figure 6. In step 502, motion compensation is performed using the correction vector instead of the decoded motion vector. The rest of the decoding pipeline continues as usual through the deblocking filter in step 504, the next output video frame in step 506, and the caching of the output in step 518. These steps can continue to repeat indefinitely for each frame. Typically, the blending process of step 520 occurs after inverse quantization (step 514) and inverse transformation (step 516) because the actual motion vector is not available until this point. Once the actual motion vector is available, the blending process will use the motion vector to calculate the difference between the actual motion vector and the estimated motion vector.

[0068] Figure 6 shows an exemplary macroblock undergoing player input motion compensation and blending. At time 0ms, the player presses an input to rotate the camera view to the left. The associated player input motion compensation motion vector from the lookup table is applied to all macroblocks in the next frame (as shown in "PIMC" 600). Fig. 6A An exemplary macroblock is shown moving to the right. The results of the player input motion compensation appear in the next decoded frame, resulting in a maximum input-feedback delay of 16ms (the length of one frame) for video running at 60 frames per second. In other embodiments, for video running at 30 frames per second, the maximum input-feedback delay may be 33ms. It is therefore contemplated that by limiting the maximum input-feedback delay to the length of one frame, the present invention operates across a variety of frame rates. Figure 6BAs shown, when the tagged video is returned from the server, the tagged video includes the actual motion vector 602 encoded by the server. For this example, the encoded video is returned at time 100ms, but this will depend greatly on network latency. The actual vector 602 refers to the macroblock that has been shifted during the player input motion compensation 600, so the actual vector 602 cannot be directly applied to the existing frame. Instead, a correction vector 604 needs to be calculated by finding the difference between the actual vector 602 and the player input motion vector 600. This difference can be calculated by adding the inverse player input motion vector to the actual motion vector. Finding these vector differences refers to Figure 5 The smaller the correction vector 604, the more successful the player input motion compensation method is in estimating the actual motion vector. Figure 6C The resulting motion 608 for the exemplary macroblock shown in FIG. 6 is the same as the actual motion vector 602. This example shows how player input motion compensation can estimate video feedback for player input and display the result from 16ms to 100ms in time, whereas an unmodified system would not display any player-feedback within 116ms after receiving the player input.

[0069] During development, game developers will need to decide which motions and animations will send expected motion vectors during runtime. Unique but predictable motion vectors are the best choice for expected motion vectors. Clear examples will include animations adaptively changed by the engine, for example, animations that use kinematic equations to calculate joint angles, animations that are time-warped, or otherwise stretched or compressed. For example, when a player enters a defined wall range and jumps, a wall-climbing (ledge grab) animation will play. The wall-climbing animation stretches so that the player's hand is on the wall, but is still attached to the player's body. The animation is broadcast with a defined number of frames, placing the player on the wall. The starting point of this example is variable through some acceptable positions and orientations. Because the exact animation is unknown in advance, but is programmably generated on demand by the game engine, animation is an ideal choice for generating expected motion vectors. Because the client does not have any scene information about the player's position in the game streaming environment, it is particularly impossible for the client to know the motion vector for the animation.

[0070] The expected motion vectors will only be available in a limited scene or time range, for example, a specific camera position, a small time window, or some other specific player scenario. For each set of expected motion vectors, a corresponding invalidator needs to be generated. The invalidator can be used on the client to prevent the expected motion vector from being applied again after it is valid. In some embodiments, the invalidator can be a set of any player inputs that will change the game scene so that it is no longer feasible to play the expected motion vector. In other embodiments, the invalidator can be a time window for which the expected motion vector may be valid. In other embodiments, the invalidator can be a combination of invalid inputs and time windows. For example, the expected motion vectors generated for a wall climbing animation are only valid within a limited player position and orientation, so the invalidator will necessarily include any translation or rotation movement input. The invalidator will need to be designed and implemented during the development of the expected motion vector feature. As shown below in combination Figures 10 to 13 As described above, the expected motion vector may also be disabled or updated by an event or message sent from the server. Figures 10 to 13 Motion compensation using cached repeated motion vectors is described.

[0071] The expected motion vectors may be generated in advance, or may be generated on demand during runtime. For example, for animations with a limited number of permutations, it is possible to generate the expected motion vectors offline by triggering each permutation and recording the motion vectors. In some embodiments, for general clients, the motion vectors will be stored on the server side and then sent to the client for on-demand caching. When the motion vector is sent to the client, the motion vector is cached in a lookup table. The pre-generated expected motion vectors may be stored on the server and available as a game-readable file format that allows the server to send pre-generated motion vectors to cache in a lookup table on the client during game operation. Because there may not be a discrete number of possible animation permutations, animations generated during runtime, such as animations calculated by inverse kinematics, cannot be pre-generated. Inverse kinematics is a common method used in real-time rendering to fit animations within a set of boundary conditions. For example, a player character in a video game wants to grab a nearby wall, the boundary conditions will be defined by the position where the player's hand touches the wall, and the wall climbing animation will change accordingly through inverse kinematics. For adaptively modified animations such as these, during runtime, the game can speculatively render possible animations into off-screen motion vector images and record the expected motion vectors on demand. For example, if the player is near a grab-able wall, the game can anticipate that the player will soon grab the wall, and the game can speculatively render the wall climbing animation to generate the expected motion vectors. Animations that require adaptive changes to generate expected motion vectors during runtime will need to be identified by the developer ahead of time.

[0072] Existing game systems that describe the player's scene, such as player position tracking, scripting systems, trigger volumes, or pathfinding systems, can be used to generate events that signal when the game needs to speculatively render animations. For example, the game can track the player's proximity to a grabbable wall and send a signal to the game to speculatively render a wall climbing animation and record the expected movement vector. Certain animations like picking up a weapon, pulling a joystick, or pressing a button can be stretched or adjusted based on the player's proximity to the interaction and the player's orientation. These animations have too much displacement to make pre-generation feasible, but can still be generated at runtime (such as Figure 7 exemplarily shown). Motions that are played the same way each time can be generated and recorded offline. These motions are typically motions that occur in the same screen space at the same rate each time they are triggered. The motion vectors for these animations can be recorded offline by triggering all possible permutations of the animation and recording the motion vectors generated by the game or by generating motion vectors through more conventional motion estimation techniques such as those used in the H.264 codec. In a preferred embodiment, the above combined Figure 1 Game generated motion vectors as described to Figure 6 are used to ensure high quality motion estimation. This process can be performed at any point during development, but it is preferred to add this process as a stage of the build process or some other existing asset adjustment process, such as pre-generated texture maps and multiple levels of detail ("LOD"). Asset adjustment can include any process that compiles game assets from a human-readable resource format into a machine-readable format. For example, texture maps can be generated in advance by converting texture files created by artists into a game-usable format that includes multiple resolutions. Similarly, LODs can be generated in advance by converting model files created by artists into a game-usable format that includes multiple levels of detail. Motion vector generation can be added to an existing asset adjustment process that converts artist-generated animation formats into game-usable formats.

[0073] Figure 7 An exemplary method for generating motion vectors offline or on the fly is shown. At step 700, the animation can be rendered as an off-screen surface / image and the motion vectors can be recorded for immediate use. Only the portion that moves on the screen needs to be rendered, and other objects in the scene can be ignored. The dotted line objects shown at step 702 and the solid line objects shown at step 704 represent the positions of the animated objects in the previous frame and the current frame, respectively. At step 706, the movement from the previous frame to the current frame is captured in the form of motion vectors shown at step 708. The motion vectors can be captured from motion vectors generated by the game or captured by more conventional motion estimation techniques such as those used in the H.264 codec. In a preferred embodiment, the motion vectors are captured using the method described above in conjunction with the method described above. Figure 1The game generates motion vectors as described in Figure 6 to ensure high quality motion vectors. By repeating the process shown in steps 700 to 708, valuable expected motion vectors can be quickly calculated for several frames of a given animation until motion vectors are generated for all required frames. Not all frames in the animation need to be generated, only enough frames to be played on the client while the video stream catches up. The minimum number of frames generated will depend on the delay between sending player input and receiving the resulting video stream on the client; and the length of the generated portion of the animation should be at least as long as the delay. As described below in conjunction with Figures 10 to 13 The expected motion vector frames can be rate scaled during playback, Figures 10 to 13 Discusses the use of cached repeated motion vectors in motion estimation. If the playback of the expected motion vector frame is rate scaled, generating expected motion vectors for a portion of animation equal to the delay will result in a playback with a rate scaling of 50%. Generating motion vectors for a longer portion of animation will result in a playback with less noticeable rate scaling.

[0074] The macroblock size used for video encoding should be taken into account when re-encoding motion vectors, and a motion vector should be present for each macroblock. In a preferred embodiment, the motion vectors generated by the game are generated as motion vectors for each pixel, and converted to motion vectors for each macroblock by finding the arithmetic mean of the motion vectors for each pixel for each macroblock group.

[0075] Figure 8 The sending of expected motion vectors for the purpose of player input motion compensation and the storage on the client side are shown. During the development of video game software, events need to be configured to issue upcoming input-driven scene-sensitive animations. For example, the game developer wants to send the expected motion vector so that the motion vector is available when the player performs a wall climbing animation. The developer implements an event that will be triggered whenever the player faces a grabbable wall and is within the range of the grabbable wall. In this example, in the case where the player approaches a grabbable wall while playing the game, the exemplary event is received in step 800 "receiving an event during game play". The event type will describe whether the expected motion vector needs to be generated in the case of adaptively changing animations, and will describe whether the expected motion vector has been pre-generated offline in the case of animations that never change but are rarely played or depend on the player's scene. In the above example, the wall climbing animation is stretched based on the distance between the player and the wall, which means that the motion vector will need to be generated at runtime in step 802 "generate expected motion vectors". In another case, the motion vector may have been generated offline and read from step 804 "read pre-generated motion vectors".

[0076] An example of pre-generated motion vectors could be switching between weapons in a large arsenal. The number of possible weapon switching permutations can become so large that it is impractical to cache the entire set of generated motion vectors. Typically, if the motion vectors take up too much space in a limited cache and are not used frequently enough, these motion vectors are not the best choice for pre-caching. In step 806 "Send expected motion vectors and invalidators", the expected motion vectors are sent to the client. In step 808 "Cache expected motion vectors and invalidators", the expected motion vectors are added to the motion vector lookup table. In one embodiment, the invalidation system is functionally similar to a system that triggers the application of motion vectors in the lookup table but disables the motion vectors. When the set of motion vectors and the invalidator are received in step 808 "Cache expected motion vectors and invalidators", the invalidator will need to be registered with the invalidation system.

[0077] Combination of the above Figure 1 The player input motion compensation method described in steps 810 to 814 compares all player inputs to entries in the motion vector lookup table. In the example above, when the player enters the required input to initiate the wall climbing animation, the input will be matched to the input previously cached in the lookup table at step 810 "Match Received Player Input". If the cached expected motion vector has not been invalidated, the expected motion vector will be applied at step 812 "Apply Player Input Motion Compensation". If the matched player input would invalidate the expected motion vector, or if the expected motion vector expires after a predetermined time, or if the expected motion vector expires after being applied once, then at step 814 "Invalidate" the expected motion vector is removed from the lookup table and is no longer applied.

[0078] Fig. 9An exemplary method of a signal that can invalidate an expected set of motion vectors is shown. Invalidation is an update to a lookup table where a set of expected motion vectors is removed from the lookup table, which is preferably cached at the client. In addition to responding to update events received from the server, the update mechanism can also monitor player input and include an invalidation countdown timer. When a set of expected motion vectors is cached in the lookup table, its invalidator will likely be registered using the update function / mechanism. An invalidator is any data signal that triggers an invalidation update to the lookup table. The exemplary lookup table "Lookup Table" shown in step 900 includes three expected sets of motion vectors: an expected door animation shown in step 902 "Expected Door Animation", an expected wall climbing animation shown in step 904 "Expected Wall Climbing", and an expected shooting animation shown in step 906 "Expected Shooting Animation". The expected motion vector can be invalidated after being applied once. In this example, when the player presses an input to open a nearby door in step 908 "Open Door Input", the expected door animation motion vector shown in step 902 "Expected Door Animation" is applied. At the same time, the door opening input at step 908 "door opening input" is cancelled, and the expected door animation shown at step 902 "expected door animation" can be removed from the lookup table at step 910 "invalid after use". Similarly, other inputs can invalidate the expected motion vectors before application. In the example, the set of expected motion vectors for the wall climbing animation shown at step 904 "expected wall climbing" is only valid within a limited distance from the wall.

[0079] If the player moves away from the wall using the movement input shown at step 912 "Movement Input" before the player jumps to grab the wall (by using the jump input shown at step 914 "Jump Input"), then the expected wall climbing motion vector shown at step 904 "Expected Wall Climb" will become invalid at step 916 "Input Invalid". Other examples of invalidating inputs can be the following: the player has a grappling hook or some other movement-based weapon that will invalidate the expected motion vector; and the player presses a button to activate a specific ability. Because the invalid input is for a specific scenario, other may be obvious based on the specific implementation. The expected motion vector can also expire over time. In the example, the shooting opportunity is only displayed to the player within a three-second window. If within the three-second window, the player does not press the melee input at step 918 "Melee Input", then the motion vector of the expected shooting animation shown at step 906 "Expected Shooting Animation" will become invalid at step 920 "Timer Timeout Invalid". The expected motion vector can have multiple invalidators, and vice versa. For example, an intended wall climb may be invalidated if a move input is received or after a wall climb motion vector is used, whichever comes first.

[0080] Fig.10An exemplary method for generating a motion vector library and an exemplary repetitive motion vector library for caching purposes are shown. Since the selected motion is highly repetitive in nature, the motion will be played in the same way each time it is triggered. This enables motion vectors to be generated in advance and organized into a library. Motion vector library generation can be performed at any time during development, but it makes sense to add this process as a stage during the build process or some other asset adjustment process. In this example, a motion vector library will be generated for each available weapon in a first-person shooter. When library generation for the first weapon begins at step 1000 "Library Generation", the first weapon animation is triggered at step 1002 "Trigger Animation" and the motion vector is recorded at step 1004 "Record Motion Vector". The motion vectors can be generated by the game or generated by more conventional motion estimation techniques used in the H.264 codec. In a preferred embodiment, high-quality motion estimation is ensured using game-generated motion vectors as described in U.S. Provisional Application Nos. 62 / 488,256 and 62 / 634,464, all of which are incorporated into this application. If the recorded motion vectors are not correct or are not correctly quantized, then these motion vectors will introduce artifacts when used during player input motion compensation. Steps 1002 and 1004 are repeated until motion vectors are recorded for each highly repetitive animation in the library. Library generation begins again from step 1000 until all libraries are generated.

[0081] The example shown at step 1006, "Motion Vector Library," is a very simplified version of a repeating motion vector library for a plasma rifle weapon in a first person shooter game. The example is simplified to include two simple animations: a 2-frame animation for the feedback animation that plays when the player fires the plasma rifle, shown at step 1008, and a 4-frame animation of the rifle's swinging motion that occurs when the player walks forward, shown at step 1010. In a typical, real environment, a weapon would likely have many more animations and the animations could be much longer than four frames.

[0082] Fig.11 The process of caching, applying and updating a motion vector library for player input motion compensation is shown. When the game is initialized, at step 1100 "On startup, send repeated motion vector library", the server will send a pre-generated motion vector library to the client. At step 1102 "Cache repeated motion vectors", the client typically stores the motion vector library in memory in the form of a lookup table. In this embodiment, the client is universal to meet the requirements of multiple streaming games. In an alternative embodiment, a game-specific client can permanently store the motion vector library without caching repeated motion vectors from the server.

[0083] At this point, the client can begin monitoring player input and comparing incoming input to entries in the player input motion compensation lookup table. In step 1104 "Match received player input", when the incoming input has a matching entry in the lookup table, in step 1106 "Apply player input motion compensation", the motion vector associated with the player input is used to provide a match such as the combined Figures 1 to 10 Motion estimation is described illustratively. There may be certain situations where receiving a particular input may require changing the lookup table in step 1112 "Update Cached Duplicate Motion Vector Mapping". A pause button is an example of an input that would change the lookup table, which can disable player input such as character movement, weapon firing animations, or other game-related motion. After applying the motion vectors and optionally updating the lookup table, the client will continue to monitor player input. After the lookup table is updated in step 1112 "Update Cached Duplicate Motion Vector Mapping", the incoming player input will be compared to the updated entry in the lookup table.

[0084] At any time during the running of the game, a scene change to the player may require a change in the lookup table. For example, a player can switch weapons, requiring the cached motion library to switch from a previously held weapon to a new weapon. An exemplary embodiment can be seen for game monitoring on the server side for specific game events. These events will be configured by methods known in the art during game development, and can be sent through the game's existing messaging or event system. When the game instance running in step 1108 "receiving events during operation" receives one of these events, a message will be generated and sent to the client in step 1110 "sending scene updates". Scene updates can be sent for any game event that changes the relationship between the player input and the motion vectors included in the lookup table. Scene updates can enable or disable player input from a set of triggering motion vectors, change the motion vector associated with a given input, or otherwise add or remove the association between player input and the motion vector used for player input motion compensation. When the message arrives at the client, in step 1112 "update cached repeated motion vector mapping", the lookup table is changed according to the changed scene.

[0085] Fig.12 12 is a diagram showing how the motion vector map may be updated. The lookup table 1200 stored in the client cache 1202 is used by the client to look up which motion vectors are associated with a given player input. During game play, sometimes a change in the scene may change the behavior of a player input. For example, if the player moves forward but encounters a moving obstacle such as a wall, it may be necessary to stop applying the predicted motion vector for forward movement. When this happens, Fig.11As shown in step 1110 in , the server will send a scene update to the client. The movement blocking event 1204 is received, which sends a signal to the client to disable the association between the forward movement input 1206 and the corresponding motion vector 1208. When the player uses the forward movement input until another event from the server re-enables the association between the forward movement input and the forward movement motion vector, the forward movement motion vector will stop applying. For this example, when the player's movement becomes unblocked and a scene update is sent to the client to re-establish the association between the forward input and the forward movement motion vector in the lookup table.

[0086] In another example, the player holds a plasma rifle but switches to a shotgun. The plasma rifle motion vector library 1210 is stored in the cache along with its firing motion vector 1209, which corresponds to a specific firing input 1211. The cache also stores motion vector libraries for other weapons including shotguns 1212 and pistols 1214. When the client receives a weapon switch event 1216 from the server, the plasma rifle motion vector library 1210 is switched out of the lookup table 1200 for the shotgun motion vector library 1212. To prevent player input motion compensation from being performed improperly during a weapon switch, two events can be used in succession to first disable the plasma rifle motion vector library 1210 while the weapon switch animation is playing, and then switch the two motion vector libraries after the weapon switch animation is complete.

[0087] For longer multi-frame motion vectors, the application of the multi-frame motion vector can be extended so that the last frame of the motion vector is applied when the last frame of the actual video is received from the server. This will allow the server to catch up with the estimated motion when the client runs out of cached motion vectors. The scaling factor of the motion vector is defined as the playback speed scaling, which is exemplarily calculated as follows.

[0088]

[0089] Delay is defined as the time between the initial player input event and the actual video received on the client. The delay includes the time it takes to send the input to the server via the network, the time to process the game logic, rendering logic, GPU rendering and encoding on the server, and the network time to return the video to the player. Delay should have been continuously measured in any streaming environment. As described in U.S. Provisional Application Nos. 62 / 488,256 and 62 / 634,464, a preferred embodiment of player input motion compensation uses relevant tags to associate player input with actual video. Before the incoming player input is sent to the server, the relevant tag as a unique identifier is attached. When the video frame with the relevant tag is returned from the server, the client matches the unique identifier with the previous input. Send a signal to the client to stop motion estimation for the relevant input or revoke the previous motion estimation through a hybrid technique. The length of the cached portion of the animation or the cached animation time can be calculated by multiplying the number of frames in the cached animation by the length of each frame.

[0090] exist Fig.13 In , an example animation containing 10 frames is used for player input motion compensation in a game running at 60 frames per second with 100ms latency. When player input triggers player input motion compensation, the playback speed scale is calculated based on the cached motion vectors, as shown below.

[0091]

[0092]

[0093]

[0094] Player income is received at time 0ms. The play rate of the motion vectors is scaled by 1300 by the calculated play speed scale. The first frame of motion vectors is applied to the next available frame in the video stream from the server. The scaled motion vector frames will be interpolated to keep the animation smooth. Because the motion vector frames are scaled over multiple frames, interpolation is a method that can be used to calculate "how many" motion vectors to apply on any given frame. An exemplary embodiment may use linear interpolation based on the calculated play speed scale. For our example, the calculated play speed scale is 0.625, which will stretch the set of motion vectors over 1.6 display frames. Interpolation is a method used to calculate how many motion vectors to apply on a given frame. That is, when the set of motion vectors is stretched over multiple display frames, interpolation calculates how far down the macroblock along the motion vector. At 17ms, only a portion of the first scaled motion vector should be applied to the first display frame, which is equal to the play speed scale of 0.625. At the second display frame at 33ms, the remainder of the first scaled motion vector is applied, calculated as 1-0.625=0.375, and then the first portion of the second scaled motion vector is applied, calculated as the playback speed scale minus the remainder of the first scaled motion vector, or 0.62-0.375=0.25. At the third display frame at 50ms, the second set of scaled motion vectors is applied, moving the macroblock down the motion vector by the next 62.5%. At the fourth display frame at 67ms, the remainder of the second scaled motion vector is applied, calculated as 1-0.25-0.625=0.125, and the first portion of the third compressed motion vector is applied, calculated as the playback speed scale minus the remainder of the second scaled motion vector, 0.625-0.125=0.5. Linear interpolation continues as the scaled motion vectors are applied.

[0095] Multi-frame motion vectors can send a relevant tag for each frame of the cached animation to associate each frame of the estimated motion with the actual video in the future.

[0096] The latency will depend greatly on the network path and architecture between the client and server. This example uses a latency of 100ms, but latency can vary from tens to hundreds of milliseconds. Shorter latency will provide a better player experience, but player input motion compensation techniques can help mask the effects of high latency in some situations. After the delay 1304, the actual video 1302 is received. For servers located at the edge or physically close to the user, the latency can be as low as 30ms. For more typical server locations, it is more likely to be 100ms. The actual video maintains the length 1306 of the original animation because it is not scaled. Motion compensation techniques are applied to the actual video based on the player input.

[0097] If the client is not able to perfectly blend the previous motion compensation, the H.264 coding standard provides a redundant slice feature that can correct any time-propagated errors. Based on the H.264 profile settings, each slice will be encoded as an intra slice (I-slice) and sent on a rotating schedule at a specific frequency. Because the intra slice will not contain motion vectors, the blended motion vectors should be applied only when the actual motion vectors arrive at the p-slice. This will avoid applying blended motion vectors on macroblocks that already exist in the I-slice before the tagged frame is returned from the server.

[0098] The foregoing description and accompanying drawings should be considered as merely illustrative of the principles of the invention. The present invention is not intended to be limited by the preferred embodiments and may be implemented in various ways apparent to those of ordinary skill in the art. Those skilled in the art will readily appreciate many applications of the present invention. Therefore, it is not desired to limit the present invention to the specific examples disclosed or the exact construction and operation shown and described. On the contrary, all suitable modifications and equivalents may be adopted within the scope of the present invention.

Claims

1. A computer-implemented method for caching motion vectors, comprising: generating, at a server, one or more motion vectors, wherein the one or more motion vectors is a vector field having a vector for each macroblock in the image; sending the generated motion vectors and one or more invalidators to a client, wherein the generated motion vectors and invalidators are cached at the client, wherein the motion vectors are cached in a lookup table, the lookup table comprising one or more user inputs and one or more associated motion vectors, the one or more invalidators being associated with the one or more cached motion vectors, the invalidator being a combination of the user input and a time window that disables one of the one or more cached motion vectors; sending instructions to a client to receive input from a user, wherein the client is configured to match the input to a cached motion vector or invalidator; and Sends instructions to the client to apply matching motion vectors or invalidators to implement motion compensation in the graphics interface.

2. The method according to claim 1, further comprising the steps of: If the input matches a cached invalidator associated with one or more motion vectors, the client is instructed to delete the cached one or more motion vectors. 3 . The method according to claim 1 , wherein the motion vector is generated before the execution of the game or during the execution of the game. The method of claim 1 , wherein the motion vectors are stored at a server and sent to a client for on-demand caching. The method of claim 1 , wherein the motion vectors are game generated.

6. The method of claim 5, wherein the motion vector generated by the game is generated as a motion vector for each pixel and converted into a motion vector for each macroblock.

7. A system for caching motion vectors, wherein: Over the network, the server: generating one or more motion vectors, wherein the one or more motion vectors is a vector field having a vector for each macroblock in the image; sending the generated motion vectors and one or more invalidators to a client, wherein the generated motion vectors and invalidators are cached at the client, wherein the motion vectors are cached in a lookup table, the lookup table comprising one or more user inputs and one or more associated motion vectors, the one or more invalidators being associated with the one or more cached motion vectors, the invalidator being a combination of the user input and a time window that disables one of the one or more cached motion vectors; sending instructions to a client to receive input from a user, wherein the client is configured to match the input to a cached motion vector or invalidator; and Sends instructions to the client to apply matching motion vectors or invalidators to implement motion compensation in the graphics interface.

8. The system according to claim 7, further comprising the steps of: If the input matches a cached invalidator associated with one or more motion vectors, the client is instructed to delete the cached one or more motion vectors.

9. The system of claim 7, wherein the motion vector is generated before the execution of the game or during the execution of the game.

10. The system of claim 7, wherein the motion vectors are stored at a server and sent to a client for caching on demand.

11. The system of claim 7, wherein the motion vectors are game generated.

12. The system of claim 11, wherein the game generated motion vector is generated as a motion vector for each pixel and converted to a motion vector for each macroblock.

Images