Systems and method for player input motion compensation through anticipating motion vectors and / or capture of repetitive motion vectors

Abstract

Systems and methods for latency reduction through motion estimation and compensation incorporate a client device that uses lookup tables from a remote server to match, label, and sum motion vectors. When a remote server transmits encoded video frames to the client, the client decodes them and applies the summed motion vectors to estimate motion. The technology also generates motion vectors based on predetermined criteria and transmits them and invalidators to a client, which caches them. The server instructs the client to receive input and uses it to match cached motion vectors or invalidators. The technology also caches repetitive motion vectors and transmits previously generated motion vector libraries to a client.The server instructs the client to calculate a motion estimate and instructs the client to update the stored motion vector library, so that the client applies the stored library to initiate motion before actually receiving motion vector data.

Description

1 / 43 “SYSTEMS AND METHOD FOR MOTION COMPENSATION PLAYER INPUT THROUGH ANTICIPATION OF MOTION AND / OR CAPTURE VECTORS OF REPETITIVE MOTION VECTORS DESCRIPTIVE REPORT RELATED ORDERS

[001] This application claims the benefit of the following applications Provisional US Patents 62 / 488,526, filed April 21, 2017, 62 / 634,464, filed February 23, 2018, 62 / 640,945, filed March 9, 2018, and 62 / 644,164, filed March 16, 2018. BACKGROUND OF THE INVENTION

[002] Remote gaming applications, in which a server-side game is controlled by a client-side player, have attempted to encode the video output of a three-dimensional (3D) graphics engine in real time, using existing or custom encoders. However, the interactive nature of video games, particularly the player feedback loop between the video output and the player input, makes continuous video streaming of games much more latency-sensitive than traditional continuous video streaming. Existing video encoding methods can trade computational power, and little else, for reductions in encoding time.New methods for integrating the encoding process with the video rendering process can provide significant reductions in encoding time, while also reducing computational power, improving the quality of the encoded video, and maintaining the original bitstream data format to preserve the interoperability of existing hardware devices. Petition 870190105936, dated 10 / 18 / 2019, page 38 / 246 2 / 43

[003] Unlike regular video playback, video games have a single player input to the video feedback circuit. Players are very sensitive to latency between input and video output. High latency in this player input-feedback circuit has been a significant impediment to streaming video game applications, where a server-hosted video game instance is controlled by a remote player. Any process that can reduce the time between input and feedback will directly improve the user experience.

[004] Client hardware in a streaming game environment may have varying levels of computational performance, but dedicated H.264 hardware decoders are becoming more ubiquitous, even in mobile and other low-power devices. A hardware decoder is good at performing a small selection of computations, such as motion compensation, that are regularly performed according to the H.264 encoding standard. The strengths of dedicated decoding hardware can be exploited to provide a better player experience in a streaming game environment, regardless of the client's overall computing power.

[005] In local, non-streamed rendering applications, the game engine can add several frames of latency between player input and video feedback. In streaming game applications, additional latency is introduced to the player input-feedback cycle because the player input must travel across the network to the remote server and video output must be encoded, transmitted, and decoded before the player receives the feedback. For some player inputs, the client can estimate the results on the video feedback, performing motion compensation immediately, eliminating network latency.

[006] Player input motion compensation is, in its most basic form, a technique of shifting groups of pixels in order to sacrifice some image precision for a decrease in latency. Petition 870190105936, dated 10 / 18 / 2019, page 39 / 246 3 / 43 Input feedback is used in situations where a video game is running on a server while being remotely controlled by a networked client. This technique is good at reducing player-feedback latency for inputs that result in consistent motion vectors, such as player view rotations in first-person games.

[007] In a video game, player context is defined as the current game state, which is the result of previous player actions, inputs, and decisions. The client in a streaming game system is naive to player context; that is, the client receives only the video output and none of the game state information that will determine the outcomes of certain player inputs. There is a wide range of inputs that result in unique, but predictable, movement outcomes based on game state information. These inputs would benefit from a reduction in player-feedback latency, but cannot be pre-cached on the client for traditional user input movement compensation because the client will lack player context information. Additionally, the player context permutation space can be too exhaustive to pre-generate movement vectors for methods such as cached repetitive movement vectors.These systems and methods are described in U.S. Provisional Applications 62 / 488,526; 62 / 634,464; and 62 / 640,945; all three are incorporated herein in their entirety. The game server can compensate by generating anticipatory motion vectors and updating the client input motion compensation cache as the player context changes. This allows the client to utilize player input motion compensation techniques for a limited set of context-dependent inputs, resulting in reduced input-feedback latency.

[008] U.S. Patent 9,661,351 (“the ‘351 Patent”) discloses systems and methods for skipping a frame during transmission from a server to a client device, wherein, in response to the detection of the skipped frame, the client device generates a predicted frame, which replaces the skipped frame in the compressed video stream, the predicted frame Petition 870190105936, dated 10 / 18 / 2019, page 40 / 246 4 / 43 being generated based on delta information extending from one or more previous frames decoded by the client device. For this technology, client-side frame prediction of one or more reconstructed or predicted frames is used following a skipped frame based on data (e.g., motion vectors, residuals, etc.) from one or more preceding frames. The technology also prioritizes bit allocation and / or subfeature encoding. Network Abstraction Layer Units (NALUs) encoded could be divided into (1) motion vectors and (2) residuals. Instead of actually skipping a frame, the device can just send minimal encoding data as prioritized. For example, it could send only motion vectors if motion is prioritized.The present invention is superior to the technology of Patent '351 at least because Patent '351 does not disclose a client device that uses lookup tables transmitted from a server to match user input to motion vectors and tags and sum those motion vectors. Patent '351 also does not disclose the application of those summed motion vectors to the decoded frames to estimate the motion in those frames. The present invention is also superior because it reduces input-feedback latency, which is significantly reduced by using user input motion compensation instead of waiting for the server to return the output video.

[009] U.S. Patent 8,678,929 (“the ‘929 Patent”) is directed to the operation of a networked interactive gaming system. The disclosed methods are geared toward reducing network lag by determining course-corrected positional values ​​for a shared global object through bidirectional combination. The “combination” discussed in the patent includes the steps of network simulation sending the calculation of a posture to the local player on the local console. The console combines the local and network simulations. The methods also combine shared global objects using the combined posture of the local player to determine a positional value for the shared global object on the local console. The present invention is, again, superior to the technology of Petition 870190105936, dated 10 / 18 / 2019, page 41 / 246 5 / 43 Patent '929 is superior because Patent '929 does not disclose a client device that uses lookup tables transmitted from a server to match user input to motion vectors and tags and sum those motion vectors. Patent '929 also does not disclose the application of those summed motion vectors to the decoded frames to estimate the motion in those frames. The present invention is also superior because it does not require the presence of a client with extremely high processing power, and it significantly reduces input-feedback latency by using player input motion compensation instead of waiting for the server to return the output video.

[0010] US Patent 8,069,258 (“the Patent '258”) is directed to the use of local frame processing to reduce apparent network lag in multi-player simulations. The methods described include intercepting inputs from a local user, determining the state data of a remote object in a network simulation from a previous game frame, and determining the interactions of non-deterministic objects from multiple game systems that are part of the network simulation. That interaction data, along with the local input and state data, is used to generate a local simulation of the video frame. In this way, the local and network simulations can run asynchronously for a single frame, with each frame corresponding to a single temporal phase within the game. This allows for real-time local simulation updates during network gameplay while remaining (essentially) synchronized with the network.The present invention is, once again, superior to the technology of Patent '258, at least because Patent '258 does not disclose a client device that uses lookup tables transmitted from a server to match user input to motion vectors and tags and sum those motion vectors. Patent '929 also does not disclose the application of those motion vectors summed to the decoded frames to estimate motion in those frames. The present invention is also superior because it does not require the presence of a client with extremely high processing power, reducing input latency. Petition 870190105936, dated 10 / 18 / 2019, page 42 / 246 6 / 43 feedback, which is significantly reduced, using player input motion compensation instead of waiting for the server to return video output.

[0011] U.S. Patent 9,665,334 B2 (“the ‘334 Patent”) discloses systems and methods for rendering protocols by applying multiple processes and a compositor to render the combined graphics onto a single screen. The technology operates as follows: when the server simultaneously provides a game screen to a number of client devices, the computational load of the rendering processing on the server becomes heavy, for example, in a content-rich game that requires fast responsiveness. That is, the number of client devices to which the server can provide a screen is limited depending on their rendering performance and required responsiveness. In contrast, when each client device is controlled to perform the processing, which can be done by sharing the rendering processes between the server and the client device, a screen can be provided to more client devices.Furthermore, in general, a game screen, which is rendered without the application of texture mapping, has high compression efficiency and can be transmitted with lower bandwidth over a network such as the Internet. The present invention is superior to the technology discussed in Patent '334, at least because it does not disclose the generation of motion vectors on a server based on predetermined criteria and the transmission of the generated motion vectors and one or more invalidators to a client, which caches those motion vectors and invalidators. It also does not disclose the service instructing the client to receive input from a user and use that input to match cached motion vectors or invalidators, where those vectors or invalidators are used in motion compensation.The present invention is also superior because it reduces input-feedback latency, which is significantly reduced by using player input motion compensation instead of waiting for the server to return output video. Petition 870190105936, dated 10 / 18 / 2019, page 43 / 246 7 / 43

[0012] Patent 9,736,454 (“Patent '454”) discloses systems and methods for encoding, comprising examining the availability of a colocalized depth block with a texture block, determining a prediction method for a texture block based on the availability of a colocalized depth block, and deriving a first prediction block for the texture block based on the availability of the colocalized depth block. Again, the present invention is superior to the technology discussed in Patent '454, at least because it does not disclose the generation of motion vectors on a server based on predetermined criteria and the transmission of the generated motion vectors and one or more invalidators to a client, which caches those motion vectors and invalidators.It also does not reveal that the server instructs the client to receive input from a user and use that input to match cached motion vectors or invalidators, where those vectors or invalidators are used in motion compensation. The present invention is also superior because it reduces input-feedback latency, which is significantly reduced by using player input motion compensation instead of waiting for the server to return output video.

[0013] U.S. Patent 9,705,526 (“the ‘526 Patent”) discloses systems and methods for entropy coding in media and image applications. The technology discloses a system in which compression begins with the receipt of an image and / or video data source, as indicated. A lossless compression scheme is then applied. A predictor / delta computation unit then takes the input and attempts to reduce redundancy in the input data using a delta computation between neighboring input elements. These values ​​are then encoded using a predefined statistical model in an entropy encoder to produce the compressed image and / or video data. Similar to the above, the present invention is superior to the technology discussed in Patent '526 at least because it does not disclose the generation of motion vectors on a server based on predetermined criteria and transmission of the vectors. Petition 870190105936, dated 10 / 18 / 2019, page 44 / 246 8 / 43 motion generated and one or more invalidators to a client, which caches those motion vectors and invalidators. It does not yet reveal that the server instructs the client to receive input from a user, and uses that input to match cached motion vectors or invalidators, where those vectors or invalidators are used in motion compensation. The present invention is also superior because it reduces input-feedback latency, which is significantly reduced by using player input motion compensation instead of waiting for the server to return output video.

[0014] US Patent 8,873,636 B2 (“the ‘636 Patent”) is directed to a moving image distribution server (such as one running an online game) that provides encoded image data to the user’s PC running the local instance of the game. In order to perform this process, in relevant detail, the user’s PC CPU specifies the region to be referenced in order to decode the motion vector associated with the selected block on the screen of the preceding frame. It does this by referring to the motion vector associated with the selected block (a vector that is included in the preprocessing information is provided) and extracts the image of the region as a reference image.As is the case with other references, the present invention is superior to the technology discussed in Patent '636, at least because it does not disclose the generation of motion vectors on a server based on predetermined criteria and the transmission of the generated motion vectors and one or more invalidators to a client, which caches those motion vectors and invalidators. It also does not disclose the server instructing the client to receive input from a user and use that input to match cached motion vectors or invalidators, wherein those vectors or invalidators are used in motion compensation. The present invention is also superior because it reduces input-feedback latency, which is significantly reduced by using player input motion compensation instead of waiting for the server to return the output video.

[0015] International Patent Publication WO2009138878 A2 (“a Petition 870190105936, dated 10 / 18 / 2019, p. 45 / 246 9 / 43 Publication '878' is aimed at the continuous processing and transmission of multiple interactive applications on a centralized streaming application server, while controlling the detail levels and post-filtering of various rendered objects. In the system, a centralized interactive application server, in its video preprocessor, performs spatial and temporal filtering on the frame sequence, before encoding a compressed stream of audiovisual content to client devices, which encode the compressed stream and display the content. The GPU command processor of the centralized interactive application server includes a module that also computes a motion compensation estimate for each macroblock in the target encoded frame in the video encoder.However, the present invention remains superior to the technology discussed in Publication '878 at least because it does not disclose the generation of motion vectors on a server based on predetermined criteria and the transmission of the generated motion vectors and one or more invalidators to a client, which caches those motion vectors and invalidators. It also does not disclose the server instructing the client to receive input from a user and use that input to match cached motion vectors or invalidators, where those vectors or invalidators are used in motion compensation. The present invention is also superior because it reduces input-feedback latency, which is significantly reduced by using player input motion compensation instead of waiting for the server to return output video.

[0016] U.S. Patent 9,358,466 B2 (“the ‘466 Patent”) is directed toward improving video game performance through the reuse of cached data. The disclosed systems score cache performance of different generated video game missions by at least partly identifying the digital resources used, and determining whether the identified digital resources are in a cache. Cache scoring can be calculated based on a cache reuse rate, corresponding to a proportion of digital resources for a mission that are already in a cache. Petition 870190105936, dated 10 / 18 / 2019, page 46 / 246 10 / 43 Other techniques for generating cache scores may account for factors such as the overall size of combined digital resources for a mission that are already in a cache and / or the overall size of combined digital resources for a mission that are not already in a cache. By caching data in this way, that data, and other requests for data not stored in a cache, become more efficient. The present invention remains superior to the technology discussed in Patent '466, at least because it does not disclose caching of repetitive motion vectors, calculating a motion estimate from input data, or updating the stored motion vector library based on input data, so that a client can use the stored motion vector library to initiate motion before actually receiving motion vector data from a server.

[0017] Patent 6,903,662 B2 (“the ‘662 Patent”) is directed to a configurable computer input device, specifically one that caches input data to maintain faster response times. The system operates by checking keystrokes against an internal memory (or cache) to determine if the particular key has been previously identified. If the system has not communicated with the key before this point, the system can then retrieve data corresponding to the input from its key memory. The system will then update its internal memory (cache) with the key identity and its corresponding data. The system can then send the key input data to the host computer. However, the next time the system encounters the same key in the pressed state, it can query its own memory (cache) instead of retrieving the same scan code from the key memory again.The present invention is superior to the technology discussed in Patent '662, at least because it does not disclose caching of repetitive motion vectors, calculating a motion estimate from input data, or updating the stored motion vector library based on input data, so that a client can... Petition 870190105936, dated 10 / 18 / 2019, page 47 / 246 11 / 43 Use the stored motion vector library before actually receiving motion vector data from a server.

[0018] Japanese Patent JP6129865B2 (“Patent '865”) discloses systems and methods for transmitting rendered game content data for route segment subsets to a game client for caching in the local cache, so that game content data can be available when needed during real-time gameplay. Again, the present invention is superior to the technology discussed in Patent '865 at least because it does not disclose repetitive motion vector caching, calculating a motion estimate from input data, or updating the stored motion vector library based on input data, so that a client can use the stored motion vector library to initiate movement before actually receiving motion vector data from a server.

[0019] Patent 9,762,919 (“the '919 Patent”) discloses systems and methods for caching reference data in a block processing pipeline. The technology of Patent '919 discloses a cache memory (e.g., a fully associative cache), which can be implemented, for example, in local memory (relative to the pipeline), such as SRAM (static random access memory), to which portions of the chroma reference data (e.g., 64-byte memory blocks) corresponding to determined motion vectors for macroblocks in early pipeline stages can be prefetched from memory. Chroma cache logic can maintain the cache, and can extend over multiple pipeline stages.Searches for the motion vectors of a given macroblock passing through the pipeline can be initiated by the chroma cache logic one or more stages before the chroma motion compensation stage to provide time (i.e., multiple pipeline cycles) to read the respective memory blocks from the memory within the cache before the chroma motion compensation needs them. However, Patent '919 remains deficient compared to the present invention. The present... Petition 870190105936, dated 10 / 18 / 2019, page 48 / 246 12 / 43 The invention is superior to the technology discussed in Patent '919, at least because it does not disclose caching of repetitive motion vectors, calculating a motion estimate from input data, or updating the stored motion vector library based on input data, so that a client can use the stored motion vector library to initiate motion before actually receiving motion vector data from a server.

[0020] As is apparent from the above discussion of the state of the art in this technology, there is a need in the art for an improvement to the present computer technology related to the encoding of real-time game environments. SUMMARY OF THE INVENTION

[0021] It is therefore an object of the present invention to disclose systems and methods for reducing latency through motion compensation techniques, wherein a client device uses lookup tables transmitted from a server to match user input to motion vectors, and to label and sum those motion vectors. When a remote server transmits encoded video frames to the client, the client decodes those video frames and applies the motion vectors summed to the decoded frames to estimate motion in those frames.

[0022] Another object of the present invention is to disclose systems and methods whereby encoded video frames are decoded without residual handling.

[0023] It is yet another object of the present invention to disclose systems and methods for reducing latency through motion compensation techniques, wherein the client applies one or more smoothing functions to the motion vectors with tags summed in the queue.

[0024] It is yet another object of the present invention to disclose systems and methods for reducing latency through compensation techniques. Petition 870190105936, dated 10 / 18 / 2019, page 49 / 246 13 / 43 movement, in which the tags associated with the movement vectors on the client device are, by nature, chronological.

[0025] Another object of the invention is to provide systems and methods for reducing input-feedback latency by generating motion vectors on a server based on predetermined criteria and transmitting the generated motion vectors and one or more invalidators to a client, which caches those motion vectors and invalidators. The server instructs the client to receive input from a user and use that input to match cached motion vectors or invalidators. Based on that comparison, the client then applies the matched motion vectors or invalidators to effect motion compensation in a graphical interface.

[0026] Another object of the invention is to provide systems and methods for reducing input-feedback latency by caching motion vectors in a lookup table.

[0027] Yet another object of the invention is to provide systems and methods for reducing input-feedback latency by associating invalidators with one or more cached motion vectors.

[0028] Yet another object of the invention is to provide systems and methods for reducing input-feedback latency by instructing the client to clear one or more cached motion vectors if the input is matched with a cached invalidator associated with one or more motion vectors.

[0029] Another objective of the present invention is to disclose systems and methods for reducing latency by caching repetitive motion vectors on a server that transmits a previously generated motion vector library to a client. The client stores the motion vector library and monitors it for user input data. The server instructs the client to calculate a motion estimate from the input data and instructs the client to update the stored motion vector library based on the input data, so that the client Petition 870190105936, dated 10 / 18 / 2019, page 50 / 246 14 / 43 applies the stored motion vector library to initiate movement in a graphical interface before actually receiving motion vector data from a server.

[0030] Another object of the present invention discloses systems and methods for reducing latency by caching repetitive motion vectors, wherein the server transmits a context update to the client to disable the application of the stored motion vector library.

[0031] It is yet another object of the present invention to disclose systems and methods for reducing latency by means of caching repetitive motion vectors, wherein one or more resizing factors is applied to the motion vector library. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] A fuller appreciation of the invention and of many of the advantages arising therefrom will be easily obtained as it becomes better understood by reference to the following detailed description, when considered in connection with the accompanying drawings, in which:

[0033] FIG. 1 is a block diagram exemplarily illustrating a remote gaming system, in which an electronic game is running on a server, when controlled by input on a remote client;

[0034] FIG. 2 is an exemplary flowchart describing the reduction in input-feedback latency in continuous game streaming applications, applying motion compensation for an appropriate player input;

[0035] FIG. 3 is a block diagram exemplarily illustrating an exemplary moment during the runtime of a continuous electronic game streaming environment, which is using player input motion compensation;

[0036] FIG. 4 is a diagram exemplarily illustrating a Petition 870190105936, dated 10 / 18 / 2019, page 51 / 246 15 / 43 exemplary macroblock during player input movement compensation in FIG. 3;

[0037] FIG. 5 is a diagram exemplarily illustrating an alternative method for applying motion vectors during player input motion compensation to the client;

[0038] FIGS. 6A, 6B and 6C present an exemplary macroblock undergoing player input movement compensation and combination for the alternative method shown in FIG. 5;

[0039] FIG. 7 is a diagram illustrating runtime generation of anticipatory motion vectors;

[0040] FIG. 8 is a flowchart illustrating exemplarily the client-side transmission and storage of anticipated motion vectors for the purpose of player input motion compensation;

[0041] FIG. 9 is a diagram illustrating an exemplary process of invalidating anticipated motion vectors;

[0042] FIG. 10 is a diagram presenting an exemplary method for generating the motion vector library and an exemplary repetitive motion vector library for caching purposes;

[0043] FIG. 11 is a flowchart illustrating exemplarily the process for caching, applying, and updating motion vector libraries for player input motion compensation;

[0044] FIG. 12 is an illustrative diagram showing how the motion vector mapping can be updated; and

[0045] FIG. 13 is a diagram illustrating an exemplary modification for the application of multi-frame motion vectors during player input motion compensation, especially in the case of cached motion vectors. DETAILED DESCRIPTION OF PREFERRED ACHIEVEMENTS

[0046] In the description of the preferred embodiments of the invention illustrated Petition 870190105936, dated 10 / 18 / 2019, page 52 / 246 16 / 43 In the drawings, specific terminology will be used for clarity. However, the invention is not intended to be limited to the specific terms thus selected, and it should be understood that each specific term includes all technical equivalents that operate in a similar way to achieve a similar end. Several preferred embodiments of the invention are described for illustrative purposes, it being understood that the invention can be carried out in other forms not specifically shown in the drawings.

[0047] Certain types of player input result in better candidates for player input motion compensation. Two factors contribute to a given input suitability: the player's sensitivity to input-feedback latency, and the difficulty of implementing player input motion compensation without introducing significant artifacting. Each input will need to be evaluated for suitability. For example, in a first-person shooter game, the player will be very sensitive to mouse-based view rotation; a split-second delay, as small as 16 ms, between player input and video output will be noticeable. However, in the same situation, gamepad-based view rotation is typically slower, and players might be less sensitive to input-feedback latency. View rotation can be approximated by shifting the scene in the opposite direction of the rotation, but there may be undesirable artifacting along the edge of the image in the direction of the rotation.For small view rotations, such as aiming at an enemy on the screen, players may not even notice edge artifacting. In another example, accelerating a car in a racing game may be low priority for player input motion compensation due to the player's lack of sensitivity and / or inertia latency, but steering and braking inputs may be high priority because players will perceive input feedback latency.

[0048] The time between receiving player input and displaying movement output is the player-feedback latency. When using movement compensation, an estimated movement can provide feedback almost Petition 870190105936, dated 10 / 18 / 2019, page 53 / 246 17 / 43 immediately, while waiting for the server to process the player input. In this way, player-feedback latency is dramatically reduced in streaming game applications. By implementing player input motion compensation in a streaming game application, a motion estimate can be provided in the next available frame. In contrast, it takes several frames for the input to travel to the server, produce an output frame, and return. Player input motion compensation can also provide some benefit in traditional non-streamed game applications, where the game engine and renderer may have a player-feedback latency of a few frames.

[0049] The client will not have the appropriate context to track the movement of an object around the screen. Player input motion compensation will not be appropriate for cases where the location of specific macroblocks or video objects is unknowable to the client. For example, in a 2D platform game, the character may move around the screen from left to right. The client will not know where the character is located when the player presses the input to jump; therefore, player input motion compensation alone cannot be used in this case to reduce input-feedback latency.

[0050] In general, motion vectors for player input motion compensation should be generated ahead of time. For motion such as player camera rotation, motion vectors can be calculated based on how the game weights the input. In certain realizations, motion vectors can be the input value multiplied by the sensitivity weight. For motion that cannot be directly calculated, such as animated motion, animation can be triggered during development so that motion vectors can be directly measured and stored. Measuring motion vectors can be performed using the same motion estimation techniques performed during H.264 encoding.

[0051] FIG. 1 illustrates an example system, in which a game Petition 870190105936, dated 10 / 18 / 2019, page 54 / 246 18 / 43 electronic is controlled by a remote client. In this system, a server 100 hosts the electronic game software 102 and a graphics engine 104, which renders the video output. The video is encoded in a codec (also referred to as an encoding engine or encoder) 106, and the encoded video data 108 is transmitted to a remote client. The server architecture 100 can be any combination of hardware or software that can support the functions of both the graphics engine 104 and the codec 106. In the given example, the graphics engine 104 can be implemented as, for example, a GPU 110 controlled by the electronic game software 102 loaded into some computer-readable memory 112, while the codec 106 can be implemented as a CPU 114 running video encoding software.

[0052] The remote client consists of a computer system 116 capable of running a client-side codec 118 to decode the transmitted encoded video data 108 and a client application 120 to apply player input motion compensation. The client computer system 116 also contains a display controller 122 to control the screen hardware 124. The input from the client-side input peripherals 126 will be converted by the client application 120 into control data 128, which is transmitted back to the game software 102 running on the server 100. The input from the peripherals 126 will also be used to determine which, if any, player input motion compensation to apply, as illustrated in greater detail by FIG. 2.

[0053] FIG. 2 is a flowchart describing the steps required to perform player input movement compensation for a single input. When the client initializes, the server sends a lookup table containing inputs and their associated movement vectors in step 200, which is then cached by the client in step 202. In this implementation, the client is generalized to serve the needs of multiple streaming games. In certain embodiments, a game-specific client may skip steps 200 and 202 because it already contains the lookup table for the game. In an alternative implementation, a Petition 870190105936, dated 10 / 18 / 2019, page 55 / 246 19 / 43 The game-specific client can permanently store the movement vector lookup table, without needing to cache it from the server.

[0054] When the client receives player input from an input device such as a mouse or gamepad controller in step 204, the client application will check the cached motion vector lookup table for matching inputs in step 206. If there is no matching player input, the client will take no further action and send the input to the server without further modification. If there is a matching player input in the cache, the client will apply player input motion compensation. Optionally, the cache may be able to change entries in the lookup table based on the player input. For example, when the player presses the pause button, all player motion inputs should be disabled until the player exits the pause screen.In one implementation, the client-controlled lookup table might have two sets of inputs, one for use in the pause menu and one for use outside the pause menu, which are swapped, preferably by the client, whenever the player selects the pause input. In an alternative implementation, the server might cache the contents of the lookup table on the client.

[0055] When the client application receives player input for movement compensation, the client will add a tag to the player input and its associated movement vectors in step 208. The tagged input is sent to the server in step 210. The tag is any identifier that can correlate player input to a future frame. For example, the tag can be an integer that is incremented each time the client receives input that will be used to perform movement compensation from player input. The tag can be added as metadata in the same network packet as the player input or sent in a similar message pattern that keeps the tag information synchronized with the input information. The client confirms whether a tag is received in step 213 or not. While the player input is being tagged and sent, the client Petition 870190105936, dated 10 / 18 / 2019, page 56 / 246 Step 20 / 43 applies the motion vectors contained in the cached lookup table from step 212. These motion vectors will be applied to each received frame until the correlated tag is returned from the server. A detailed description of an example method for applying these motion vectors is illustrated in FIG. 3.

[0056] When the server receives labeled player input, the labeled player input is passed to the game, which generates an output frame in step 214. The video image is then encoded in step 216. Before the encoded frame is sent back to the client, the player input tag is attached to the encoded frame in step 218. This is the same tag that was previously sent with the player input and means that the output frame contains the actual video feedback from the player input. Attaching a tag to the encoded frame can be done by adding the tag as metadata to the same network packet as the encoded frame. The labeled encoded frame is sent back to the client in step 220. When the client receives an encoded frame with a tag, the client can correlate the tag to a previous player input motion offset. The client then stops applying the previous motion offset in step 222.

[0057] FIG. 3 is an illustration of an exemplary moment during the runtime of a continuous video game streaming environment, which is using player input motion compensation. When the client receives any player input in step 300, it will be compared to the motion vector lookup table in step 302. If there is a matching player input, the associated motion vectors will be used for player input motion compensation. The motion vectors are labeled in step 306 with a unique input tag in step 304. In this example, the chosen tag is the integer “1003”. The labeled motion vectors are added to a queue in step 308, containing any other currently labeled motion vectors being used for player input motion compensation. Petition 870190105936, dated 10 / 18 / 2019, page 57 / 246 21 / 43

[0058] The next frame arrives in the bitstream from the server at step 320. This frame is labeled with a unique identifier at step 318, in this case, the integer “1001”, which indicates that the frame contains the resulting motion from all previous player inputs up to and including the input corresponding to the tag “1001”. The tag “1001” indicates to the client that it can stop applying motion compensation at step 322 for this labeled input. The motion vectors with the tag “1001” are then removed from the motion vector queue labeled at step 308 along with any motion vectors with previous tags that may remain in the queue in case previous packets were lost.

[0059] The encoded video 316 is decoded in step 324. Meanwhile, the remaining motion vectors in the motion vector queue in step 308 are summed in step 310. Motion vectors are typically vector fields with one vector for each macroblock in the image. To sum the motion vectors, the vectors are summed in terms of elements, so the result is a vector field with one vector for each macroblock. The sum of two vector fields is the vector sum of the vectors for each point in the field, so the sum of two sets of motion vectors is the vector sum for each macroblock in the image. The sum of two vectors is defined as the component sum of their components, which can be represented as {u1, u2} + {vi, V2} = {ui + vi, u2 + V2}. In this example, two sets of motion vectors with tags “1002” and “1003” are contained in the queue; These two sets of motion vectors are added together.Tags are, by nature, chronological, which allows the client to know the order of previously labeled player inputs. This allows the client to discard movement vectors labeled up to and including the return tag in a received frame. Furthermore, the tagging discussed above is computationally cheaper than more complex methods. Optional smoothing functions can be applied at this point to prevent artifacting by locking or to mitigate similar introduced artifacting.

[0060] Artifacting by locking is introduced when macroblocks Petition 870190105936, dated 10 / 18 / 2019, p. 58 / 246 22 / 43 are moved off-screen and manifest as a pixel color blur perpendicular to the screen edge. An example smoothing function would reduce the strength of outward-pointing motion vectors as they approach the image edge. Only the outward-pointing directional component needs to be weakened: the y-component for outward-pointing vectors towards the top and bottom edges of the image, and the x-component for outward-pointing vectors towards the left and right edges of the image. For vectors pointing towards the edge, the vector component could be multiplied by the square of the distance from the edge, so that as the distance from the edge approaches zero, the vector component approaches zero. In this example, the outward-pointing vector towards the right edge of the image will be transformed from {x,y} to {x*d*d,y}, where d is the distance from the edge.This will mitigate artifacting caused by crashing, in exchange for some slight distortion in the image around the edge of the image. The distortion is much less noticeable to the player than artifacting caused by crashing.

[0061] After the decoding process is completed in step 324 on the encoded video frame, the summed motion vectors are used in motion compensation in step 312. The resulting video is output in step 314. This output contains the motion compensation data from the player input and will be displayed on the client. This output frame is the first frame containing a motion estimate for the input with correlation tag “1003”. This output frame also contains a motion estimate for the previous inputs with correlation tag “1002”, for which the client is still waiting for the server to return actual motion vectors. This output frame is also the first frame to contain the actual motion vectors for a previous input with correlation tag “1001”, for which the client previously estimated the motion.As a consequence, three states of motion estimation exist at this point in the method: a new estimation state for new inputs; a continued estimation state, which is awaiting actual results; and another estimation state, which is stalled because the actual results have arrived. Petition 870190105936, dated 10 / 18 / 2019, p. 59 / 246 23 / 43 customer.

[0062] FIG. 4 is a diagram illustrating an exemplary macroblock during the player input motion compensation step of FIG. 3. The player input motion compensation step is procedurally similar to the process performed during H.264 decoding, where the decoded motion vectors are used to determine where macroblocks in the current frame moved from in the previous frame. For player input motion compensation, the applied motion vectors will estimate future feedback for a player input, moving macroblocks in the current frame before displaying the output video to the player. FIG. 4A shows two sets of labeled motion vectors in the row at step 400 of the exemplary labeled motion vectors shown at step 308 in FIG. 3. These two sets are summed, taking the vector sum for each macroblock in the image, to create a set of motion vectors at step 402 in FIG. 4B.Each of the motion vectors in this set represents the estimated motion for each macroblock in the current frame. For each vector in the summed motion vectors in step 402, a corresponding macroblock in the example image of FIG. 4C will be swapped. An example macroblock is shown in FIG. 4C being swapped for its corresponding player input motion vector in step 404. Each macroblock in the example image will be swapped in this way. The example image in FIG. 4C contains only 50 macroblocks, but a high-definition image will contain thousands of macroblocks. The example motion vectors shown in FIG. 4 show uniform rigid motion, but the player input motion compensation technique described can be used with motion vectors of arbitrary complexity to describe rotations, vibrations, or other complex motions in screen space that are known in the art.

[0063] FIG. 5 is an illustration of an alternative method for applying motion vectors during player input motion compensation in the client. In certain embodiments of this method, it is not necessary Petition 870190105936, dated 10 / 18 / 2019, page 60 / 246 24 / 43 Handling residuals during video encoding and decoding. After a matching player input is found in the lookup table, the associated motion vectors are labeled in step 500 and used for player input motion compensation, as shown in steps 208 and 212 in FIG. 2. During the decoding process in the following frame, the player input motion vectors are added to the decoded motion vectors and injected into the motion compensation step in step 502, as defined in the H.264 encoding standard. The unlock filter, as defined by the H.264 decoding standard, is applied in step 504, but should only be applied to blocks that were not modified by the player input motion compensation. The resulting output, shown in step 506, contains the player input motion compensation and will be displayed on the client.The output is subsequently stored in step 506 and, in step 518, becomes the previous image for use in motion compensation in the following frame. Unlike the implementation in FIG. 3, in this implementation, the labeled player input motion vectors are only applied once. This means that they do not need to be summed with all the labeled motion vectors in the queue, because the previous image, shown in step 518, will contain the sum of previous labeled motion vectors. The time between receiving the input and displaying the video output is the input-feedback latency, which will have been significantly reduced by using player input motion compensation, instead of waiting for the server to return output video.

[0064] At the same time as step 500, the client will send the corresponding labeled player input to the server, as shown in steps 208 and 210 in FIG. 2. The client will eventually receive encoded video, as shown in step 508, with the same tag 510 in the server's bitstream in step 512, as shown in step 220 in FIG. 2. The tag 510 associated with the video means that the actual motion previously estimated by the player input motion compensation is represented in the encoded video, as shown in step 508. The encoded video passes through Petition 870190105936, dated 10 / 18 / 2019, page 61 / 246 25 / 43 entropy decoding in step 514 and inverse quantization and inverse transformation in step 516, as defined by the H.264 encoding standard.

[0065] The previous player input motion compensation step, which was performed before steps 508, 510, 512, 514, and 516 described in the preceding paragraphs, has already changed the macroblocks that the encoded frame's actual motion vectors are attempting to change. Therefore, applying the actual motion vectors directly will change incorrect macroblocks. The bitstream will contain two related pieces of data: the encoded video frame and a correlation tag. The encoded frame is sent to entropy decoding in step 514, while the tag is sent forward to the combination process in step 520. To compensate, the combination process in step 520 provides motion vector correction by calculating the difference between the input actual motion vectors and the previously used player input motion vectors 500 with a corresponding correlation tag.The difference can be calculated by adding the inverse motion vectors for the tag correlation to the actual motion vectors and is described in greater detail in connection with FIG. 6. The correction vectors are used in place of the decoded motion vectors for motion compensation in step 502. The remainder of the decoding pipeline continues as usual, with the unlocking filter in step 504, the next output video frame in step 506, and output buffering in step 518. These steps can continue to repeat for each frame indefinitely. In general, the combination process in step 520 occurs after inverse quantization (step 514) and inverse transformation (step 516), since the actual motion vectors will not be available until this point. Once the actual motion vectors are available, the combination process will use them to calculate the difference between the actual and estimated motion vectors.

[0066] FIG. 6 illustrates an exemplary macroblock undergoing player input movement compensation and combination. At time 0 Petition 870190105936, dated 10 / 18 / 2019, page 62 / 246 At 26 / 43 ms, the player presses the input to rotate the camera view to the left. The associated player input motion compensation vectors, shown as “PIMC” 600, from the lookup table are applied to all macroblocks in the following frame. FIG. 6A shows an exemplary macroblock being shifted to the right. The player input motion compensation results appear in the following decoded frame, resulting in a maximum input-feedback latency of 16 ms, the length of one frame, for video running at 60 frames per second. In other embodiments, the maximum input-feedback latency may be 33 ms for video running at 30 frames per second. It is thus contemplated that the invention operates across a range of frame frequencies, limiting the maximum input-feedback latency to the length of one frame.When the labeled video returns from the server, it contains the server-encoded fact vectors 602, as shown in FIG. 6B. For this example, the encoded video returns in 100 ms, but this will be heavily dependent on network latency. The fact vectors 602 refer to macroblocks, which have already been changed during the player input motion compensation 600, so the fact vectors 602 cannot be applied directly to the existing frame. Instead, the correction vectors 604 need to be calculated by finding the difference between the fact vectors 602 and the player input motion vectors 600. The difference can be calculated by summing the inverse player input motion vectors to the fact motion vectors. Finding these vector differences is referred to as combining in step 524 in FIG. 5.The smaller the correction vector 604, the more successful the player input motion compensation method was in estimating the actual motion vectors. The resulting motion 608 shown in FIG. 6C for the example macroblock is the same as the actual motion vector 602. This example illustrates how player input motion compensation can estimate video feedback for a player input and show the result for times between 16 ms and 100 ms, while an unmodified system. Petition 870190105936, dated 10 / 18 / 2019, page 63 / 246 27 / 43 would not display any player feedback until 116 ms after the player input was received.

[0067] During development, game developers will need to decide which movements and animations will send anticipatory motion vectors during runtime. Unique, but predictable, motion vectors are the best candidates for anticipatory motion vectors. A prime example would include animations that are adaptively altered by the engine, such as animations that use kinematic equations to calculate join angles, animations that are time-warped, or animations that are otherwise stretched or compressed. For example, an edge grab animation plays when a player enters the range of a defined edge and jumps. The edge grab animation is stretched so that the player's hands are on the edge but are still attached to the player's body. The animation plays over a defined number of frames, placing the player over the edge.This starting point, in this example, is variable, with a range of acceptable locations and orientations. This edge-grabbing animation is a good candidate for generating anticipatory motion vectors because the subsequent animation is not known ahead of time but is generated programmatically by the game engine on demand. The client specifically cannot know the motion vectors for this animation, since the client has no contextual information about the player's location in a streaming game environment.

[0068] Anticipatory movement vectors will only be useful within a limited contextual or temporal scope, such as a specific camera location, a small window in time, or some other player-specific context. For each set of anticipatory movement vectors, a corresponding invalidator needs to be generated. The invalidator can be used on the client side to prevent anticipatory movement vectors from being applied after they have been validated. In certain realizations, an invalidator can be a set of any player inputs that would change the game context, such that playing the movement vectors Petition 870190105936, dated 10 / 18 / 2019, page 64 / 246 28 / 43 anticipated is no longer feasible. In other embodiments, an invalidator might be a time window for which anticipated motion vectors can be valid. In still other embodiments, an invalidator might be a combination of invalidating inputs and a time window. For example, the anticipated motion vectors generated for an edge-grabbing animation are only valid within a limited player location and orientation, and as such, an invalidator would necessarily include any translational or rotational motion input. Invalidators will need to be designed and implemented during the development of the anticipated motion vector feature. Anticipated motion vectors can also be disabled or updated by events or messages sent from the server, as described below, in connection with FIGS. 10 to 13, which discuss the use of cached repetitive motion vectors for motion compensation.

[0069] Anticipatory motion vectors can be generated ahead of time or they can be generated as needed during runtime. For animations that have a limited number of permutations, for example, anticipatory motion vectors can be generated offline, triggering each permutation and logging the motion vectors. In certain embodiments, for a generalized client, motion vectors would be stored server-side and then sent to the client to be cached on demand. When motion vectors are sent to the client, they are cached in the lookup table. Pre-generated anticipatory motion vectors can be stored on the server and will be available as an in-game readable file format, allowing the server to send pre-generated motion vectors to be cached in the lookup table on the client during game runtime.Animations that are generated during runtime, such as animations calculated using inverse kinematics, cannot be pre-generated because there may not be a distinct number of possible animation permutations. Inverse kinematics is a commonly used method in... Petition 870190105936, dated 10 / 18 / 2019, page 65 / 246 29 / 43 Real-time rendering, to fit an animation within a set of boundary conditions. For example, a player character in a video game wants to grab a nearby ledge; the boundary conditions will be defined by the locations where the player's hands hit the ledge, and the ledge-grabbing animation will be altered accordingly through inverse kinematics. For adaptively altered animations such as these, the game can speculatively render possible animations in an off-screen motion vector image during runtime and record the anticipated motion vectors as needed. For example, if the player is near a grabable ledge, the game can anticipate that the player will soon grab the ledge, and the game can speculatively render the ledge-grabbing animation to generate anticipated motion vectors.Adaptive animations, which require anticipatory motion vectors to be generated at runtime, need to be identified by a developer ahead of time.

[0070] Existing game systems that describe player context, such as player location tracking, scripting systems, trigger volumes, or pathfinding systems, can be used to generate an event that will signal when the game needs to speculatively render an animation. For example, a game could track the player's proximity to a grabable edge and signal the game to speculatively render the edge-grabbing animation and record the anticipated movement vectors. Certain animations, such as picking up a weapon, pulling a lever, or pushing a button, could be stretched or adjusted based on the player's proximity to the interaction and orientation. These animations have many permutations to make pre-generation practical, but could also be generated during runtime, as shown exemplarily in FIG. 7.Movements that reproduce in the same way every time can be generated and recorded offline. These are typically movements that occur in the same screen space, at the same rate, every time they are recorded. Petition 870190105936, dated 10 / 18 / 2019, page 66 / 246 30 / 43 triggered. Motion vectors for these animations can be recorded offline by triggering all possible permutations of the animation and recording the motion vectors generated by the game, or by generating motion vectors through more traditional motion estimation techniques, such as those used in the H.264 codec. In a preferred embodiment, game-generated motion vectors, as described above in connection with FIGS. 1 to 6, are used to ensure high-quality motion estimates. This process can occur at any point during development, but it is preferable to add this process as a stage during the build process or some other existing feature conditioning process, such as pre-generation of MIP maps and level of detail (“LODs”). Feature conditioning can include any process that compiles game assets from human-readable source formats into machine-readable formats.For example, MIP maps can be generated ahead of time by converting artist-created texture files into game-ready formats containing multiple resolutions. Similarly, LODs can be generated ahead of time by converting artist-created model files into game-ready formats containing multiple levels of detail. Motion vector generation can be added to an existing feature conditioning process that converts artist-generated animation formats into game-ready formats.

[0071] FIG. 7 illustrates an exemplary method for generating motion vectors in offline or runtime scenarios. Animations can be rendered to an off-screen surface / image, step 700, and the motion vectors can be recorded for immediate use. Only the moving portions of the screen need to be rendered; other objects in the scene can be ignored. The dashed object, shown in step 702, and the solid object, shown in step 704, represent the position of an animated object in a previous frame and current frame, respectively. The movement from the previous frame to the current frame will be captured in step 706, in the form of vectors. Petition 870190105936, dated 10 / 18 / 2019, page 67 / 246 31 / 43 motion, shown in step 708. Motion vectors can be captured from game-generated motion vectors or captured through more traditional motion estimation techniques, such as those used in the H.264 codec. In a preferred embodiment, game-generated motion vectors, as described above in connection with FIGS. 1 to 6, are used to ensure high-quality motion vectors. The value of several frames of anticipated motion vectors can be quickly calculated for a given animation by repeating the process illustrated by steps 700 to 708 until motion vectors have been generated for all required frames. Not all frames in the animation need to be generated, only enough needs to be generated to play back on the client as the video stream catches up.The minimum number of frames generated will depend on the delay between sending player input and receiving the resulting video stream on the client; and the length of the generated portion of an animation must be at least as long as the delay. Anticipated motion vector frames can be resized at the rate during playback, as described below in connection with FIGS. 10 to 13, which discuss the use of cached repetitive motion vectors in motion estimation. If playback of anticipated motion vector frames is resized at the rate, generating anticipated motion vectors for a portion of an animation equal to the delay will result in 50% playback rate scaling. Generating motion vectors for a longer portion of the animation will result in playback that suffers less aggressive rate scaling.

[0072] The size of the macroblock used for video encoding must be considered when recording motion vectors, and there must be one motion vector for each macroblock. In the preferred embodiment, the game-generated motion vectors are generated as per-pixel motion vectors and transformed into per-macroblock motion vectors, finding the arithmetic mean for each macroblock group of per-pixel motion vectors. Petition 870190105936, dated 10 / 18 / 2019, page 68 / 246 32 / 43

[0073] FIG. 8 illustrates the client-side transmission and storage of anticipatory motion vectors for the purposes of player input motion compensation. During the development of video game software, events need to be configured to signal context-sensitive animations triggered by nearby inputs. For example, a game developer might want to send anticipatory motion vectors so that they are available when the player performs an edge-grabbing animation. The developer implements an event that will be triggered whenever the player is both facing and within reach of an edge that can be grabbed. In this example, when the player approaches an edge that can be grabbed while playing the game, the example event is received in “During Runtime, Receive an Event”, step 800.The event type will describe whether the anticipatory motion vectors need to be generated, as in the case of adaptively altered animations, or whether the anticipatory motion vectors were pre-generated offline, as in the case of animations that are never altered but play infrequently or are context-dependent. In the example above, the edge-grabbing animation is stretched based on the player's distance from the edge, meaning that motion vectors will need to be generated at runtime in “Generate Anticipatory Motion Vectors”, step 802. In another case, the motion vectors may have been generated offline and read from storage in “Read Pre-Generated Motion Vectors”, step 804.

[0074] An example of pre-generated motion vectors might be switching between weapons in a large arsenal. The number of possible weapon switching permutations can grow quite large, making it impractical to cache the entire set of resulting motion vectors. In general, if motion vectors take up an excessive amount of space in the limited cache and are not used frequently enough, they are not prime candidates for pre-caching. The anticipated motion vectors are sent to the client in “Send Anticipated and Invalidating Motion Vectors”, step 806. The vectors Petition 870190105936, dated 10 / 18 / 2019, page 69 / 246 33 / 43 of the anticipated movement vectors are added to the movement vector lookup table in “Anticipated Movement Vectors and Invalidators Stored in Cached”, step 808. In one embodiment, the invalidation system functions similarly to the system that triggers the application of movement vectors in the lookup table, but instead, it disables the movement vectors. When a set of movement vectors and invalidators is received in “Anticipated Movement Vectors and Invalidators Stored in Cached”, step 808, the invalidators will need to be registered by the invalidation system.

[0075] The player input motion compensation method, as described above in connection with FIGS. 1 to 6, compares all player inputs with the entries in the motion vector lookup table. In the example above, when the player enters the required input to initiate the edge grab animation, the input will match the input previously cached in the lookup table in “Match Received Player Input”, step 810. If the cached anticipatory motion vectors have not yet been invalidated, the anticipatory motion vectors will be applied in “Apply Player Input Motion Compensation”, step 812.If a matching player input would invalidate the anticipated movement vectors, or if the anticipated movement vectors lose validity after a predetermined time, or if the anticipated movement vectors lose validity after being applied once, the anticipated movement vectors are removed from the lookup table in “Invalidate”, step 814 and are not applied.

[0076] FIG. 9 describes an exemplary signaling method that can invalidate a set of anticipated movement vectors. Invalidation is a type of lookup table update, where a set of anticipated movement vectors is removed from the lookup table, which is preferably cached on the client. In addition to responding to update events received from the server, the update mechanism can monitor player inputs and contain invalidation timers. Petition 870190105936, dated 10 / 18 / 2019, page 70 / 246 34 / 43 When a set of anticipatory movement vectors is cached in the lookup table, its invalidator will likely be registered with the update function / mechanism. An invalidator is any data signal that triggers an invalidation update to the lookup table. The example lookup table, “Lookup Table,” shown in step 900, contains three sets of anticipatory movement vectors: an anticipatory door animation, shown in “Anticipatory Door Animation,” step 902; an anticipatory edge grab animation, shown in “Anticipatory Edge Grab,” step 904; and an anticipatory shot-to-kill animation, shown in “Anticipatory Shot-to-Kill Animation,” step 906. Anticipatory movement vectors can be invalidated after being applied once.In the example, the door-ahead animation motion vectors shown in “Door-Ahead Animation,” step 902, are applied when the player presses the input in “Open Door Input,” step 908, to open a nearby door. Simultaneously, the registration of the door-opening input in “Open Door Input,” step 908, is canceled, and the door-ahead animations shown in “Door-Ahead Animation,” step 902, can be removed from the lookup table in “Invalidate After Use,” step 910. Similarly, other inputs can invalidate door-ahead motion vectors before they are applied. In the example, a set of door-ahead motion vectors for an edge-grabbing animation shown in “Edge-Ahead Grabbing,” step 904, are only valid within a limited distance from an edge.

[0077] If the player moves away from the edge, using the movement input shown in “Movement Input”, step 912, before the player jumps to grab the edge (using the jump input shown in “Jump Input”, step 914), then the anticipated edge-grabbing movement vectors shown in “Anticipated Edge Grabbing”, step 904, will be invalidated in “Invalidate in Inputs”, step 916. Other examples of invalidating inputs might be cases where the player has a harpoon or some other movement-based weapon, which would invalidate movement vectors. Petition 870190105936, dated 10 / 18 / 2019, page 71 / 246 35 / 43 anticipated, and cases where the player pushes a button that activates a special ability. Since invalidation inputs are context-specific, others may be apparent based on the particular implementation. Anticipated movement vectors may also lose validity over time. In the example, a kill shot opportunity is only present to the player for a three-second window. If the player does not press the brawl input, shown in “Brawl Input”, step 918, within the three-second window, the movement vectors for an anticipated kill shot animation, shown in “Anticipated Kill Shot Animation”, step 906, will be invalidated in “Invalidate with Expiration Timer”, step 920. Anticipated movement vectors may have multiple invalidators and vice versa.For example, an anticipated edge grab might be invalidated if a movement input is received or after the edge grab movement vectors have been used, whichever comes first.

[0078] FIG. 10 presents an exemplary method for generating the motion vector library and an exemplary repetitive motion vector library for caching purposes. Since the selected movements are highly repetitive in nature, they will be reproduced in the same way every time they are triggered. This makes it possible to generate motion vectors ahead of time and organize them into libraries. Motion vector library generation can occur at any point during development, but it may make sense to add this process as a stage during the build process or some other feature conditioning phase. In this example, a motion vector library will be generated for each weapon available in a first-person shooter.When the library generation for the first weapon starts in “Library Generation”, step 1000, the first weapon animation is triggered in “Trigger Animation”, step 1002, and the motion vectors are recorded in “Record Motion Vectors”, step 1004. The motion vectors can be generated by the game or generated through more advanced motion estimation techniques. Petition 870190105936, dated 10 / 18 / 2019, page 72 / 246 Traditional 36 / 43 frames, such as those used in the H.264 codec. In a preferred embodiment, game-generated motion vectors, as described in U.S. Provisional Applications 62 / 488,256 and 62 / 634,464, incorporated herein in their entirety, are used to ensure high-quality motion estimates. If the recorded motion vectors are not accurate or correctly quantized, they will introduce artifacting when used during player input motion compensation. Steps 1002 and 1004 are repeated until motion vectors are recorded for each highly repetitive animation in the library. Library generation restarts at step 1000 until all libraries are generated.

[0079] The example shown in “Motion Vector Library”, step 1006 is a very simplified version of a repetitive motion library for a plasma rifle weapon in a first-person shooter game. This example is simplified to include two simple animations: a 2-frame animation for the recoil animation that plays when the player fires the plasma rifle, shown in step 1008, and a 4-frame animation for the rifle's swing motion, which occurs when the player walks forward, shown in step 1010. In a typical real-world environment, a weapon would likely have more animations, and the animations could be much longer than four frames.

[0080] FIG. 11 illustrates the process for caching, applying, and updating motion vector libraries for player input motion compensation. When the game initializes, the server will send the pre-generated motion vector libraries to the client in “ON INITIALIZATION, SEND REPETITIVE MOTION VECTOR LIBRARIES”, step 1100. The client stores the motion vector libraries in memory, typically in the form of a lookup table, in “Cached Repetitive Motion Vectors”, step 1102. In this implementation, the client is generalized to serve the needs of multiple streaming games. In an alternative implementation, a game-specific client could permanently store the motion vector libraries. Petition 870190105936, dated 10 / 18 / 2019, page 73 / 246 37 / 43 movement, without the need to cache repetitive movement vectors from the server.

[0081] At this point, the client can start monitoring the player input and compare the incoming inputs with the entries in the player input motion compensation lookup table. When an incoming input has a corresponding entry in the lookup table in “Corresponding Player Input Received”, step 1104, the motion vectors associated with the player input are used to provide a motion estimate, as exemplarily described in connection with FIGS. 1 to 10, in “Apply Player Input Motion Compensation”, step 1106. There may be certain cases where receiving a specific input may require the lookup table to be changed in “Update Cached Repetitive Motion Vector Mapping”, step 1112.An example of an input that would alter the lookup table is the pause button, which can disable player inputs such as character movement, weapon firing animations, or other gameplay-related movements. After applying the motion vectors and optionally updating the lookup table, the client will continue to monitor player input. After the lookup table is updated in “Update Cached Repetitive Motion Vector Mapping”, step 1112, incoming player input will be compared with the updated entries in the lookup table.

[0082] At any point during the game's runtime, a change in the player's context may require the lookup table to be altered. For example, the player may switch weapons, requiring the cached movement library to be swapped from the previously held weapon to the new weapon. An illustrative implementation could be server-side game monitoring for specific game events. These events will be configured through methods known in the technique during game development and can be sent through the existing message or event system in the game. When a running game instance receives one of these events in Petition 870190105936, dated 10 / 18 / 2019, page 74 / 246 38 / 43 “During Runtime, Receive an Event”, step 1108, a message will be generated and transmitted to the client in “Send Context Update”, step 1110. A context update can be sent for any game event that changes the relationship between player input and the movement vectors contained in the lookup table. Context updates can enable or disable a player input from triggering a set of movement vectors, change the movement vectors associated with a given input, or otherwise add or remove associations between player input and movement vectors for player input movement compensation. When the message reaches the client, the lookup table is modified according to the changing context in “Update Cached Repetitive Movement Vector Mapping”, step 1112.

[0083] FIG. 12 is a diagram illustrating how the movement vector mapping can be updated. The lookup table 1200, which was stored in the client cache 1202, is used by the client to find which movement vectors are associated with a given player input. There may be times during game runtime when contextual changes alter the behavior of player inputs. For example, if a player is moving forward but encounters a movement blocker, such as a wall, the predicted movement vectors for forward movement will need to stop being applied. When this happens, the server will send a context update to the client, as shown in step 1110 in FIG. 11. This movement block event 1204 is received, which signals the client to disable the connection between the forward movement input 1206 and the corresponding movement vectors 1208.The forward movement vectors will no longer be applied when the player uses the forward movement input, until another server event re-enables the connection between the forward movement input and the forward movement vectors. For this example, the game will generate an event when the player's movement becomes unlocked and transmit the context update to the client. Petition 870190105936, dated 10 / 18 / 2019, page 75 / 246 39 / 43 re-establish the connection between the forward input and the move forward movement vectors in the lookup table.

[0084] In another example, the player is holding the plasma rifle, but switches to a shotgun. The motion vector library for the plasma rifle 1210 is stored in the cache along with its firing motion vectors 1209, which corresponds to a specific firing input 1211. The cache also stores motion vector libraries for other weapons, including a shotgun 1212 and a pistol 1214. When the client receives the weapon swap event 1216 from the server, the motion vector library for the plasma rifle 1210 is swapped from lookup table 1200 for the motion vector library for the shotgun 1212.To prevent player input motion compensation from occurring unnecessarily during weapon switching, two events can be used together to first disable the motion vector library of the 1210 plasma rifle while a weapon switching animation is playing, and then switch the two motion vector libraries after the weapon switching animation is complete.

[0085] For longer multi-frame motion vectors, it is possible to extend their application, such that the last frame of motion vectors is applied as the last video frame is actually received from the server. This will allow the server to catch up to the estimated motion at the moment the client runs out of cached motion vectors. The scaling factor for motion vectors is defined as the playbackSpeedScale, calculated for example as shown below: Cached animation time playbackSpeedScale = ------------------------------------------- Cached animation time + delay

[0086] Where delay is defined as the time between the initial player input event and the actual receipt of the video on the client. This delay includes the time it takes to send the input across the network to the server, Petition 870190105936, dated 10 / 18 / 2019, page 76 / 246 40 / 43 server processing including game logic, rendering logic, GPU rendering time, and encoding time, and network time to return the video back to the player. Delay must already be continuously measured in any continuous game streaming environment. The preferred implementation for player input motion compensation utilizes a correlation tag, as described in US Provisional Applications 62 / 488,256 and 62 / 634,464, to correlate player input and actual video. Before an incoming player input is sent to the server, a correlation tag is attached as a unique identifier. When a video frame returns from the server with a correlation tag, the client matches the unique identifier to a previous input. This signals the client to stop estimating motion for the correlated input or undo a previous motion estimate through matching techniques.The length of the cached portion of the animation, or cached animation time, can be calculated by multiplying the number of frames in the cached animation by the length of each frame.

[0087] In FIG. 13, an example animation containing 10 frames is used for player input motion compensation in a game running at 60 frames per second with a 100 ms delay. When player input triggers player input motion compensation, the playbackSpeedScale is calculated for the cached motion vectors, as shown below. animation time stored in cache p L ay b ackS p eedS cal e = ---------------------------------------------- animation time stored in cache + frame delay * 1000 ms frames playbackSpeedScale => , 1000 ms 1 frame* ----3-- + 100 ms \ 60 frames / 166.66 ms playbackSpeedScale = -----------------= 0.625 166.66 ms + 100 ms Petition 870190105936, dated 10 / 18 / 2019, p. 77 / 246 41 / 43

[0088] Player input was received at time 0 ms. The playback rate of the motion vectors is scaled 1300 by the calculated playbackSpeedScale. The first frame of motion vectors is applied to the next available frame in the server's video stream. The scaled motion vector frames will be interleaved to maintain smooth animation. Since motion vector frames are scaled across multiple frames, interpolation is a method that can be used to calculate "how much" of the motion vector to apply to any given frame. An example implementation can use linear interpolation, based on the calculated playbackSpeedScale. For our example, the calculated playbackSpeedScale is 0.625, which will stretch a set of motion vectors across 1.6 display frames. Interpolation is a method for calculating how much of a motion vector to apply over a given frame.That is, the interpolation calculates how far to move a macroblock down a motion vector when the set of motion vectors is stretched over multiple display frames. Therefore, a portion of the first resized motion vectors should be applied over the first display frame at 17 ms, equal to a playbackSpeedScale of 0.625. In the second display frame at 33 ms, the remainder of the first resized motion vectors is applied, calculated as 1 - 0.625 = 0.375, then the first portion of the second resized motion vectors is applied, calculated as the playback speed resizing minus the remaining portion of the first resized motion vectors, or 0.625 - 0.375 = 0.25. In the third display frame at 50 ms, the second set of resized motion vectors continues to be applied; the macroblocks are moved the next 62.5% down the motion vectors.In the fourth display frame at 67 ms, the remainder of the second rescaled motion vectors is applied, calculated as 1 - 0.25 - 0.625 = 0.125, and the first part of the third rescaled motion vectors is applied, calculated as the playbackSpeedScale minus the remainder of the second vectors. Petition 870190105936, dated 10 / 18 / 2019, page 78 / 246 42 / 43 scaled motion 0.625 - 0.125 = 0.5. Linear interpolation continues as the scaled motion vectors are applied.

[0089] Multi-frame motion vectors can send a correlation tag for each frame of the cached animation, to correlate each estimated motion frame to future actual video.

[0090] The delay will depend heavily on the network route and architecture between client and server. This example uses a 100ms delay, but delays can vary between dozens and hundreds of milliseconds. Shorter delays will provide a better player experience, but player input motion compensation techniques can help mask the impact of higher delay times in certain cases. After a 1304 delay, the actual video is received at 1302. For servers located at the edge, or servers that are physically close to consumers, delay times can be as low as 30ms. For more typical server locations, 100ms is more likely. The actual video retains the original animation length of 1306 because it has not been resized. The actual video is applied in accordance with player input motion compensation techniques.

[0091] If the client cannot perfectly match the previous motion compensation, the H.264 encoding standard provides a redundant slice feature, which can temporarily correct any propagated errors. Based on the H.264 profile settings, each slice will be encoded as an intra slice (I-slice) and sent on a rotating schedule at a certain frequency. Since intra slices will not contain motion vectors, matching motion vectors should only be applied when motion vectors actually arrive in the p-slices. This will prevent matching motion vectors from being applied over macroblocks that appeared in an I-slice before the labeled frame returns from the server.

[0092] The above description and drawings should be considered as merely illustrative of the principles of the invention. The invention is not intended to Petition 870190105936, dated 10 / 18 / 2019, page 79 / 246 43 / 43 being limited by the preferred embodiment and can be implemented in a variety of ways, which will be clear to a person of ordinary skill in the art. Numerous applications of the invention will occur readily to those skilled in the art. Therefore, it is not desired to limit the invention to the specific examples disclosed or to the exact construction and operation shown and described. Instead, recourse may be made to all appropriate and equivalent modifications falling within the scope of the invention. Petition 870190105936, dated 10 / 18 / 2019, page 80 / 246

Claims

1 / 4 CLAIMS 1. A computer-implemented method for estimating movement, comprising the steps of: transmitting a lookup table composed of one or more user inputs and one or more associated movement vectors, wherein the lookup table is transmitted with an instruction to cache the lookup table; transmitting an instruction to query the lookup table for corresponding movement vectors upon receipt of a player input; transmitting an instruction to associate a unique tag with the corresponding movement vectors from the lookup table and add the labeled movement vectors to a queue, wherein the labeled movement vectors are summed; transmitting a frame with a unique identifier tag, wherein the tag indicates the chronological point at which the frame contains movement associated with the player input;and transmit an instruction to remove motion vectors from the queue that have a tag associated with the labeled frame received from a server; characterized in that, when the server transmits encoded video frames to a client, the client is instructed to decode the video frames and apply the motion vectors added to the decoded video frames to estimate motion before emitting video.

2. A motion estimation method implemented in a computer, according to claim 1, characterized in that the encoded video frames are decoded without residual processing.

3. A motion estimation computer method, according to claim 1, characterized by the fact that Petition 870250030054, dated 04 / 14 / 2025, page 6 / 18 2 / 4, further comprises the step of instructing the client to apply one or more smoothing functions to the labeled motion vectors summed in the queue.

4. A computer-implemented motion estimation method according to claim 1, characterized in that the unique tags associated with the motion vectors are chronological in nature.

5. A computer-implemented motion estimation method according to claim 1, characterized in that the summed labeled motion vectors are associated with a corresponding macroblock.

6. A computer-implemented method for estimating movement, according to claim 1, characterized in that the cached lookup table is configured to be modified based on player input.

7. A computer-implemented motion estimation method according to claim 1, characterized in that the summed labeled motion vectors are configured to be applied once.

8. A computer-implemented motion estimation method according to claim 1, characterized in that the summed labeled motion vectors describe motion of arbitrary complexity.

9. A computer-implemented motion estimation method according to claim 1, characterized in that the summed labeled motion vectors estimate future feedback for the player input.

10. A computer-implemented motion estimation method according to claim 1, characterized in that the summed labeled motion vectors conform to the H.264 coding standard. Petition 870250030054, dated April 14, 2025, page 7 / 18 3 / 4 11. A system for estimating movement, in which, through a network, a server: transmits an instruction to a client to query a lookup table for corresponding movement vectors upon receiving a player input; transmits an instruction to the client to associate a unique tag with the corresponding movement vectors from the lookup table and adds the labeled movement vectors to a queue, in which the labeled movement vectors are summed; transmits a frame with a unique identifier tag to the client, in which the unique identifier tag indicates the chronological point at which the frame contains movement associated with the player input; and transmits an instruction to the client to remove movement vectors from the queue that have a tag associated with the labeled frame received from the server;characterized by the fact that, when the server transmits encoded video frames to the client, the client is instructed to decode the video frames and apply the motion vectors added to the decoded video frames to estimate motion.

12. Motion estimation system according to claim 11, characterized in that the encoded video frames are decoded without residual processing.

13. A motion estimation system according to claim 11, characterized in that the server further instructs the client to apply one or more smoothing functions to the labeled motion vectors summed in the queue.

14. System for motion estimation, according to claim 11, characterized in that the unique tags associated with the motion vectors are chronological in nature.

15. System for estimating movement, according to claim 11, characterized in that the summed movement vectors labeled Petition 870250030054, dated 04 / 14 / 2025, page 8 / 18 4 / 4 are associated with a corresponding macroblock.

16. Movement estimation system according to claim 11, characterized in that the cached lookup table is configured to be modified based on player input.

17. System for estimating motion, according to claim 11, characterized in that the summed labeled motion vectors are configured to be applied once.

18. System for estimating motion, according to claim 11, characterized in that the summed labeled motion vectors describe motion of arbitrary complexity.

19. System for estimating movement, according to claim 11, characterized in that the summed labeled movement vectors estimate future feedback for the player input.

20. System for estimating motion, according to claim 11, characterized in that the summed labeled motion vectors conform to the H.264 coding standard. Petition 870250030054, dated 04 / 14 / 2025, page 9 / 18

Images