Synchronization and offset of vsync between cloud game server and client

Abstract

A method is disclosed, comprising, at a server, setting a server VSYNC signal to a server VSYNC frequency. The server VSYNC signal corresponds to a video frame generated during a frame period for the server VSYNC frequency. The method also includes, at a client, setting a client VSYNC signal to a client VSYNC frequency. The method further includes transmitting a compressed video frame from the server to the client over a network using the server VSYNC signal, wherein the compressed video frame is based on the generated video frame. The method also includes, at the client, decoding and displaying the compressed video frame. Finally, the method includes, as the client receives the compressed video frame, analyzing the timing of one or more client operations to adjust the relative timing between the server VSYNC signal and the client VSYNC signal.

Description

Technical Field

[0001] This disclosure relates to streaming systems configured for streaming content across a network, and more specifically to synchronizing vertical synchronization (VSYNC) signals between cloud gaming servers and clients to reduce latency between cloud gaming servers and clients. Background of the Invention

[0003] In recent years, there has been a growing push for online services, enabling online or cloud gaming in streaming formats between cloud gaming servers and clients connected via the internet. Streaming formats are gaining popularity because they allow for on-demand game titles, multiplayer gaming between players, asset sharing, instant sharing of experiences between players and / or spectators, allowing friends to watch friends play video games, and letting friends join games they are playing, among other things. Unfortunately, this demand also challenges the limitations of network connectivity and the processing capabilities performed at both the server and client ends, where the processing response must be sufficient to render high-quality images delivered to the client. For example, the results of all game activity performed on the server need to be compressed and transmitted back to the client with low millisecond latency for optimal user experience. Round-trip latency can be defined as the total time between a user's controller input and the display of a video frame on the client; it can include processing control information and transmitting it from the controller to the client, processing control information and transmitting it from the client to the server, generating a video frame on the server in response to that input, processing the video frame and transmitting it to an encoding unit (e.g., a scan output), encoding the video frame, transmitting the encoded video frame back to the client, receiving and decoding the video frame, and any processing or grading of the video frame before display. One-way latency can be defined as a portion of the round-trip latency, including the time from the start of transmitting a video frame to an encoding unit (e.g., a scan output) on the server to the start of displaying the video frame on the client. A portion of both round-trip and one-way latency is associated with the time spent sending data streams from the client to the server and from the server to the client over the communication network. Another portion is associated with processing on the client and server; improvements to these operations (such as advanced strategies related to frame decoding and display) can significantly reduce round-trip and one-way latency between the server and client, providing a higher quality experience for users of cloud gaming services.

[0004] It is against this backdrop that the proposed implementation scheme has emerged. Summary of the Invention

[0005] Embodiments of this disclosure relate to streaming systems configured for streaming content (e.g., games) across a network, and more specifically to synchronizing VSYNC signals between cloud gaming servers and clients to reduce latency between them. In the context of this patent, “synchronization” should be understood as meaning tuning signals such that their frequencies match, but their phases may differ; “offset” should be understood as meaning a time delay between signals, such as the time between when one signal reaches its maximum value and when another signal reaches its maximum value.

[0006] This disclosure discloses a method. The method includes setting a server VSYNC signal at a server to a server VSYNC frequency, the server VSYNC signal corresponding to the generation of multiple video frames at the server during multiple frame periods for the server VSYNC frequency. The method includes setting a client VSYNC signal at a client to a client VSYNC frequency. The method includes transmitting multiple compressed video frames based on the multiple video frames from the server to the client via a network using the server VSYNC signal. The method includes decoding and displaying the multiple compressed video frames at the client. The method includes analyzing the timing of one or more client operations as the client receives the multiple compressed video frames to adjust the relative timing between the server VSYNC signal and the client VSYNC signal.

[0007] Other embodiments of this disclosure disclose a method. The method includes generating a plurality of video frames at a server during a plurality of frame periods, wherein the frame periods are approximately equal in size. The method includes setting a client VSYNC signal to a client VSYNC frequency at a client. The method includes sending a plurality of compressed video frames based on the plurality of video frames from the server to the client. The method includes decoding and displaying the plurality of compressed video frames at the client. The method includes analyzing the timing of one or more client operations as the client receives the plurality of compressed video frames to adjust the relative timing of the client VSYNC signal and the generation of the plurality of compressed video frames at the server.

[0008] Other embodiments of this disclosure disclose a non-transitory computer-readable medium storing a computer program for performing a method. The computer-readable medium includes program instructions for setting a server VSYNC signal at a server to a server VSYNC frequency, the server VSYNC signal corresponding to the generation of a plurality of video frames at the server during a plurality of frame periods for the server VSYNC frequency. The computer-readable medium includes program instructions for setting a client VSYNC signal at a client to a client VSYNC frequency. The computer-readable medium includes program instructions for sending a plurality of compressed video frames based on the plurality of video frames from the server to the client via a network using the server VSYNC signal. The computer-readable medium includes program instructions for decoding and displaying the plurality of compressed video frames at the client. The computer-readable medium includes program instructions for analyzing the timing of one or more client operations as the client receives the plurality of compressed video frames to adjust the relative timing between the server VSYNC signal and the client VSYNC signal.

[0009] Other embodiments of this disclosure disclose a computer system including a processor and a memory coupled to the processor and storing instructions therein that, if executed by the computer system, cause the computer system to perform a method. The method includes setting a server VSYNC signal at a server VSYNC frequency, the server VSYNC signal corresponding to the generation of a plurality of video frames at the server during a plurality of frame periods for the server VSYNC frequency. The method includes setting a client VSYNC signal at a client VSYNC frequency. The method includes sending a plurality of compressed video frames based on the plurality of video frames from the server to the client via a network using the server VSYNC signal. The method includes decoding and displaying the plurality of compressed video frames at the client. The method includes analyzing the timing of one or more client operations as the client receives the plurality of compressed video frames to adjust the relative timing between the server VSYNC signal and the client VSYNC signal.

[0010] Other embodiments of this disclosure disclose another method. The method includes setting a server VSYNC signal at a server to a server VSYNC frequency defining a plurality of frame periods, the server VSYNC signal corresponding to the generation of a plurality of video frames at the server during the plurality of frame periods. The method includes setting a client VSYNC signal at a client VSYNC frequency. The method includes using the server VSYNC signal to send a plurality of compressed video frames based on the plurality of video frames from the server to the client via a network. The method includes decoding and displaying the plurality of compressed video frames at the client. The method includes analyzing the timing of one or more client operations as the client receives the plurality of compressed video frames to set the amount of frame buffer used by the client.

[0011] Other embodiments of this disclosure disclose a non-transitory computer-readable medium storing a computer program for performing a method. The computer-readable medium includes program instructions at a server to set a server VSYNC signal at a server frequency defining a plurality of frame periods, the server VSYNC signal corresponding to the generation of a plurality of video frames at the server during the plurality of frame periods. The computer-readable medium includes program instructions at a client to set a client VSYNC signal at a client frequency. The computer-readable medium includes program instructions for sending a plurality of compressed video frames based on the plurality of video frames from the server to the client via a network using the server VSYNC signal. The computer-readable medium includes program instructions for decoding and displaying the plurality of compressed video frames at the client. The computer-readable medium includes program instructions for analyzing the timing of one or more client operations as the client receives the plurality of compressed video frames to set the amount of frame buffer used by the client.

[0012] Other embodiments of this disclosure disclose a computer system including a processor and a memory coupled to the processor and storing instructions therein that, if executed by the computer system, cause the computer system to perform a method. The method includes setting a server VSYNC signal at a server to a server VSYNC frequency defining a plurality of frame periods, the server VSYNC signal corresponding to the generation of a plurality of video frames at the server during the plurality of frame periods. The method includes setting a client VSYNC signal at a client VSYNC frequency at a client. The method includes sending a plurality of compressed video frames based on the plurality of video frames from the server to the client via a network using the server VSYNC signal. The method includes decoding and displaying the plurality of compressed video frames at the client. The method includes analyzing the timing of one or more client operations as the client receives the plurality of compressed video frames to set the amount of frame buffer used by the client.

[0013] Other embodiments of this disclosure disclose another method. The method includes setting multiple VSYNC signals to multiple VSYNC frequencies at multiple devices, wherein a corresponding device VSYNC signal of a corresponding device is set to a corresponding device VSYNC frequency. The method includes transmitting multiple signals between the multiple devices, the multiple signals being analyzed and used to adjust the relative timing between corresponding device VSYNC signals of at least two devices.

[0014] Other embodiments of this disclosure disclose a non-transitory computer-readable medium storing a computer program for performing a method. The computer-readable medium includes program instructions for setting a plurality of VSYNC signals to a plurality of VSYNC frequencies at a plurality of devices, wherein a corresponding device VSYNC signal of a corresponding device is set to a corresponding device VSYNC frequency. The computer-readable medium includes program instructions for transmitting a plurality of signals between the plurality of devices, the plurality of signals being analyzed and used to adjust the relative timing between corresponding device VSYNC signals of at least two devices.

[0015] Other embodiments of this disclosure disclose a computer system including a processor and a memory coupled to the processor and storing instructions therein that, if executed by the computer system, cause the computer system to perform a method. The method includes setting a plurality of VSYNC signals to a plurality of VSYNC frequencies at a plurality of devices, wherein a corresponding device VSYNC signal of a corresponding device is set to a corresponding device VSYNC frequency. The method includes transmitting a plurality of signals between the plurality of devices, the plurality of signals being analyzed and used to adjust the relative timing between corresponding device VSYNC signals of at least two devices.

[0016] Other embodiments of this disclosure disclose another method. The method includes receiving encoded video frames at a client, wherein a server executes an application to generate rendered video frames, and then encodes the rendered video frames at an encoder on the server to form the encoded video frames, wherein the encoded video frames include compressed one or more encoded slices. The method includes decoding the one or more encoded slices at a decoder on the client to generate one or more decoded slices. The method includes rendering the one or more decoded slices for display at the client. The method includes starting to display the rendered one or more decoded slices before the one or more encoded slices are fully received at the client.

[0017] Other embodiments of this disclosure disclose a non-transitory computer-readable medium storing a computer program for performing a method. The computer-readable medium includes program instructions for receiving encoded video frames at a client, wherein a server executes an application to generate rendered video frames, and then encodes the rendered video frames at an encoder on the server into the encoded video frames, wherein the encoded video frames include compressed one or more encoded slices. The computer-readable medium includes program instructions for decoding the one or more encoded slices at a decoder on the client to generate one or more decoded slices. The computer-readable medium includes program instructions for rendering the one or more decoded slices for display at the client. The computer-readable medium includes program instructions for starting to display the rendered one or more decoded slices before the one or more encoded slices are fully received at the client.

[0018] Other embodiments of this disclosure disclose a computer system including a processor and a memory coupled to the processor and storing instructions therein that, if executed by the computer system, cause the computer system to perform a method. The method includes receiving encoded video frames at a client, wherein a server executes an application to generate rendered video frames, and then encoding the rendered video frames at an encoder on the server to form the encoded video frames, wherein the encoded video frames include compressed one or more encoded slices. The method includes decoding the one or more encoded slices at a decoder on the client to generate one or more decoded slices. The method includes rendering the one or more decoded slices for display at the client. The method includes starting to display the rendered one or more decoded slices before the one or more encoded slices are fully received at the client.

[0019] Other aspects of this disclosure will become apparent from the following detailed description taken in conjunction with the accompanying drawings, which illustrate the principles of this disclosure by way of example. Attached Figure Description

[0020] This disclosure is best understood by referring to the following description taken in conjunction with the accompanying drawings, in which:

[0021] Figure 1A This is a diagram of the VSYNC signal at the beginning of a frame period according to one embodiment of the present disclosure.

[0022] Figure 1B This is a diagram showing the frequency of the VSYNC signal according to one embodiment of this disclosure.

[0023] Figure 2A This is a diagram of a system according to one embodiment of the present disclosure for providing games over a network between one or more cloud gaming servers and one or more client devices in various configurations, wherein the VSYNC signal can be synchronized and offset to reduce one-way latency.

[0024] Figure 2B This is a diagram of an embodiment of the present disclosure for providing a game between two or more peer devices, wherein the VSYNC signal can be synchronized and offset to achieve optimal timing for receiving controller and other information between the devices.

[0025] Figure 2C Various network configurations that benefit from proper synchronization and offset of the VSYNC signal between the source and target devices are shown according to one embodiment of this disclosure.

[0026] Figure 2DThis illustration shows a multi-tenant configuration between a cloud gaming server and multiple clients that benefits from proper synchronization and offset of the VSYNC signal between the source and target devices, according to one embodiment of this disclosure.

[0027] Figure 3 The illustration shows a variation in one-way latency between a cloud gaming server and a client due to clock drift when streaming video frames generated from a video game executed on a server, according to one embodiment of the present disclosure.

[0028] Figure 4 The network configuration of a cloud gaming server and client, according to one embodiment of the present disclosure, is shown when streaming video frames generated from a video game executed on a server, synchronizing and offsetting the VSYNC signal between the server and client to allow overlapping operations at the server and client, and reducing one-way latency between the server and client.

[0029] Figure 5A This is a diagram illustrating, according to one embodiment of the present disclosure, the possible variations in the timing of client completion of decoding relative to the desired display time specified by the server due to drift between the corresponding clocks at the cloud gaming server and the client, as well as variations in the time spent on client and server operations and network latency.

[0030] Figure 5B This includes a histogram showing the timing of client completion of decoding relative to the desired display time specified by the server, according to one embodiment of this disclosure, and showing the increase in decoding time measured in subsequent histograms due to drift between the corresponding clocks at the cloud gaming server and the client.

[0031] Figure 5C This includes a histogram showing the timing of client completion of decoding relative to the desired display time specified by the server, according to one embodiment of this disclosure, and showing consistent measurements of decoding time in subsequent histograms after compensating for measurement drift between the corresponding clocks at the cloud gaming server and the client.

[0032] Figure 6A This is a flowchart illustrating a method for tuning the VSYNC signal between a cloud gaming server and a client to reduce one-way latency, according to one embodiment of the present disclosure.

[0033] Figure 6BThis is a flowchart illustrating a method for aligning VSYNC signals to reduce one-way latency during tuning of VSYNC signals between a cloud gaming server and a client, according to one embodiment of the present disclosure. The method includes constructing a histogram that provides a distribution of the timing of client completion of decoding relative to a desired display time specified by the server. The histogram is configured to determine adjustments for the offset between the VSYNC signals at the server and client, and is further configured to determine the drift between the server's VSYNC signal and the client's VSYNC signal.

[0034] Figure 6C This is a flowchart illustrating an embodiment of the present disclosure of another method for synchronizing the VSYNC signal during the tuning of the VSYNC signal between the cloud gaming server and the client to reduce one-way latency.

[0035] Figure 7 This is a flowchart illustrating a method for tuning a client VSYNC signal relative to compressed video frames generated at a server, according to one embodiment of the present disclosure, wherein the video frames are generated during frame periods of similar size.

[0036] Figure 8A This is a diagram illustrating the construction of a histogram according to one embodiment of the present disclosure, the histogram providing the distribution of the timing of client completion of decoding relative to the desired display time specified by the server, the histogram being configured to determine adjustments for the offset between the VSYNC signals at the server and the client.

[0037] Figure 8B This is a histogram according to one embodiment of the present disclosure, the histogram providing the distribution of the timing of client completion of decoding relative to the desired display time specified by the server, the histogram being configured to determine adjustments for the offset between the VSYNC signals at the server and the client.

[0038] Figure 9 This is a flowchart illustrating a method for constructing a histogram that provides the distribution of timing of client-completed decoding relative to the desired display time specified by the server. The histogram is configured to determine the required buffer size for video frames decoded at the client.

[0039] Figure 10 This is a flowchart illustrating a method for adjusting the relative timing between VSYNC signals between two or more devices, according to one embodiment of this disclosure.

[0040] Figure 11A The diagram illustrates an overlap of receiving, decoding, and rendering decompressed video frames for display at a client, according to one embodiment of the present disclosure.

[0041] Figure 11B This is a flowchart illustrating a cloud gaming method according to one embodiment of the present disclosure, wherein encoded frames are received from a server at the client and decoded and rendered for display, wherein the decoding and display of video frames may be overlaid to reduce one-way latency.

[0042] Figure 12 Components of an exemplary device are shown that can be used to carry out various embodiments of the present disclosure. Detailed Implementation

[0043] While the following detailed description contains numerous specific details for illustrative purposes, those skilled in the art will recognize that many variations and modifications of these details are within the scope of this disclosure. Therefore, in clarifying the various aspects of this disclosure described below, the generality of the appended claims is not diminished, and no limitation is imposed on these claims.

[0044] Generally, various embodiments of this disclosure describe methods and systems configured to reduce latency and / or latency instability between source and target devices when streaming media content (e.g., streaming audio and video from a video game). Specifically, in some embodiments of this disclosure, the VSYNC signal between the cloud gaming server and client is synchronized and offset. Due to clock differences between the cloud gaming server and client, the VSYNC signals of the cloud gaming server and client drift relative to each other. This drift results in latency instability lasting up to one frame period. For example, when the game is executed at 60Hz to generate video frames, there is an additional latency of 0-16.7 ms, which varies over time. An ideal VSYNC relationship between the cloud gaming server and client can be determined by analyzing the worst-case or near-worst-case arrival time of the compressed video frames on the client. This ideal relationship can be established by tuning the VSYNC frequency at the cloud gaming server or client, thereby eliminating latency instability. In other embodiments of this disclosure, the VSYNC signals between gaming devices (e.g., game consoles) are synchronized and offset to provide an ideal VSYNC relationship and minimal one-way latency between peer devices. Specifically, due to clock differences between peer devices (e.g., gaming devices) in head-to-head gaming, their VSYNC signals will drift relative to each other, resulting in latency instability lasting up to one frame period. For example, when a game executes at 60Hz to generate video frames, there will be an additional latency of 0-16.7 ms, which varies over time. By exchanging timestamp information, an ideal VSYNC relationship between peer devices can be determined. This ideal relationship can be established by tuning the VSYNC frequency at either peer device, thereby eliminating latency instability. In other embodiments of this disclosure, dynamic client-side buffering and selective use of video frames received from the cloud gaming server are performed at the client to reduce latency and for tuning. Understanding the server-side timing of video frame generation allows the client to determine the ideal display time for each frame. Based on the variability in the arrival time of compressed video frames at the client, frame buffering (single-buffered, double-buffered, triple-buffered, etc.) can be dynamically adjusted. Latency tuning may also occur, such as choosing to skip the display of late-arriving frames. In other embodiments of this disclosure, the one-way latency between the cloud gaming server and the client can be reduced by overlapping the decoding and display of compressed video frames. The client in the cloud game receives compressed video frames from the cloud gaming server and decodes them. One-way latency is reduced by starting to display the frame before it is fully received or decoded at the client. The timing for submitting the frame for display must estimate the remaining time required to receive and decode the compressed video frame.

[0045] Specifically, latency instability can be introduced between the server and client due to the additional time required to generate complex frames (e.g., scene changes) at the server, the increased time for encoding / compressing complex frames at the server, variable communication paths on the network, and the increased time for decoding complex frames at the client. Latency instability can also be introduced due to clock differences between the server and client, causing drift between the server and client VSYNC signals. In one embodiment, this latency instability can be eliminated by tuning either the server or client VSYNC signal back to synchronization alignment (e.g., operating at the same frequency). In another embodiment, adjusting the timing offset between the server and client VSYNC signals reduces unidirectional latency by considering near-worst-case latency conditions when receiving and displaying video frames at the client. In yet another embodiment, dynamic buffering on the client side provides additional latency tuning by providing more display buffer at the client when latency increases and using less display buffer when latency decreases. In another implementation, one-way latency can be further reduced by overlapping the decoding and display of video frames at the client.

[0046] Based on the general understanding of the various implementation schemes above, exemplary details of the implementation schemes will now be described with reference to the various accompanying drawings.

[0047] Throughout this specification, references to "game," "video game," "game application," or "application" are intended to refer to any type of interactive application that is initiated by executing input commands. For illustrative purposes only, interactive applications include applications for games, word processing, video processing, video game processing, etc. Furthermore, the terms used above are interchangeable.

[0048] Cloud gaming involves executing a video game on a server to generate game-rendered video frames, and then sending those video frames to a client for display. The timing of operations at both the server and client can be related to corresponding vertical synchronization (VSYNC) parameters. When the VSYNC signal is correctly synchronized and / or offset between the server and / or client, the operations performed on the server (e.g., the generation and transmission of video frames over one or more frame periods) are synchronized with the operations performed on the client (e.g., displaying video frames on a display at a display frame or refresh rate corresponding to the frame period). Specifically, the server-generated VSYNC signal and the client-generated VSYNC signal can be used to synchronize operations at the server and client. That is, when the server-generated and / or client-generated VSYNC signals are synchronized and / or offset, the way the server generates and sends video frames is synchronized with how the client displays those video frames.

[0049] VSYNC signaling and Vertical Blanking Interval (VBI) have been incorporated into the generation and display of video frames when streaming media content between a server and a client. For example, the server attempts to generate game-rendered video frames in one or more frame cycles defined by the corresponding server VSYNC signal (e.g., generating one video frame per frame cycle would result in 60Hz operation if the frame cycle is 16.7ms, and generating one video frame every two frame cycles would result in 30Hz operation), then encodes the video frame and transmits it to the client. At the client, the received encoded video frames are decoded and displayed, with the client displaying each video frame rendered for display, starting with the corresponding client VSYNC.

[0050] For the purpose of explanation, Figure 1A This illustrates how the VSYNC signal 111 can indicate the start of a frame period, during which various operations can be performed at the server and / or client during the corresponding frame period. When streaming media content, the server can use the server VSYNC signal to generate and encode video frames, and the client can use the client VSYNC signal to display video frames. For example... Figure 1B As shown, the VSYNC signal 111 is generated at a defined frequency corresponding to the defined frame period 110. Furthermore, VBI 105 defines the time period between when the last raster line of the previous frame period is drawn on the display and when the first raster line (e.g., the top) is drawn on the display. As shown, after VBI 105, the video frames rendered for display are displayed via raster scan lines 106 (e.g., raster line by raster line from left to right).

[0051] Furthermore, various embodiments of this disclosure are disclosed for reducing one-way latency and / or latency instability between source and target devices, such as when streaming media content (e.g., video game content). For illustrative purposes only, various embodiments for reducing one-way latency and / or latency instability are described within a server and client network configuration. However, it should be understood that, as Figures 2A to 2D As shown, the various techniques disclosed for reducing one-way latency and / or latency instability can be implemented in other network configurations and / or on peer-to-peer networks. For example, the various implementations disclosed for reducing one-way latency and / or latency instability can be implemented in various configurations (e.g., server and client, server and server, server and multiple clients, server and multiple servers, client and client, client and multiple clients, etc.) between one or more server and client devices.

[0052] Figure 2AThis diagram illustrates a system 200A, according to one embodiment of the present disclosure, for providing games via network 250 between one or more cloud gaming networks 290 and / or server 260 and one or more client devices 210 in various configurations. This includes synchronization and offset of server and client VSYNC signals, and / or dynamic buffering on the client, and / or overlapping of decoding and display operations on the client to reduce one-way latency between server 260 and client 210. Specifically, according to one embodiment of the present disclosure, system 200A provides games via cloud gaming network 290, wherein the game is remotely executed from a client device 210 (e.g., a thin client) of a corresponding user playing the game. System 200A can provide game control via network 250 to one or more users playing one or more games in single-player or multi-player mode via cloud gaming network 290. In some embodiments, cloud gaming network 290 may include multiple virtual machines (VMs) running on a host hypervisor, wherein one or more VMs are configured to utilize hardware resources available to the host hypervisor to execute game processor modules. Network 250 may include one or more communication technologies. In some implementations, network 250 may include fifth-generation (5G) network technology with advanced wireless communication systems.

[0053] In some implementations, wireless technologies can be used to facilitate communication. Such technologies may include, for example, 5G wireless communication technology. 5G is the fifth generation of cellular network technology. A 5G network is a digital cellular network in which the service area covered by the provider is divided into small geographical areas called cells. Analog signals representing voice and images are digitized in a telephone call, converted by an analog-to-digital converter, and transmitted as a bit stream. All 5G wireless devices in a cell communicate via radio waves with a local antenna array and low-power automatic transceivers (transmitters and receivers) in the cell on frequency channels allocated by the transceivers from a frequency pool reused from other cells. The local antennas are connected to the telephone network and the Internet via high-bandwidth fiber optic or wireless backhaul connections. As in other cellular networks, mobile devices moving from one cell to another are automatically transferred to the new cell. It should be understood that 5G networks are merely example types of communication networks, and embodiments of this disclosure may utilize earlier generations of wireless or wired communication, as well as later generations of wired or wireless technologies that have emerged after 5G.

[0054] As shown in the figure, the cloud gaming network 290 includes a game server 260 that provides access to multiple video games. The game server 260 can be any type of server computing device available in the cloud and can be configured to execute one or more virtual machines on one or more hosts. For example, the game server 260 can manage virtual machines that support game processors that instantiate game instances for users. Therefore, multiple game processors of the game server 260 associated with multiple virtual machines are configured to execute multiple instances of one or more games associated with multiple users. In this way, the backend server supports streaming media (e.g., video, audio, etc.) of multiple game applications to multiple corresponding users. That is, the game server 260 is configured to stream data (e.g., images and / or frames rendered by the corresponding game) back to the corresponding client device 210 via network 250. In this way, computationally complex game applications can be executed at the backend server in response to controller input received and forwarded by the client device 210. Each server is capable of rendering images and / or frames, then encoding (e.g., compressing) and streaming the images and / or frames to the corresponding client device for display.

[0055] For example, multiple users can access the cloud gaming network 290 via the communication network 250 using corresponding client devices 210 configured to receive streaming media. In one embodiment, the client device 210 may be configured as a thin client, providing an interface to a backend server (e.g., game server 260 of the cloud gaming network 290) configured to provide computing capabilities (e.g., including a game title processing engine 211). In another embodiment, the client device 210 may be configured with a game title processing engine and game logic for at least some local processing of video games, and may be further configured to receive streaming content generated by a video game executed at the backend server, or for other content supported by the backend server. For local processing, the game title processing engine includes basic processor-based functionality for executing the video game and services associated with the video game. The game logic is stored on the local client device 210 and is used to execute the video game.

[0056] Specifically, a client device 210 corresponding to a user (not shown) is configured to request access to the game via a communication network 250 (such as the Internet) and to render display images generated by a video game executed by a game server 260, wherein encoded images are delivered to the client device 210 for display in association with the corresponding user. For example, a user can interact with an instance of a video game executed on the game processor of the game server 260 via the client device 210. More specifically, the instance of the video game is executed by a game title processing engine 211. The corresponding game logic (e.g., executable code) 215 implementing the video game is stored and accessible via a data storage area (not shown) and used to execute the video game. The game title processing engine 211 is capable of supporting multiple video games using multiple game logics, each of which can be selected by the user.

[0057] For example, client device 210 is configured to interact with a game title processing engine 211 associated with the game of the corresponding user, such as through input commands used to drive the game. Specifically, client device 210 can receive input from various types of input devices, such as game controllers, tablets, keyboards, gestures captured by cameras, mice, touchpads, etc. Client device 210 can be any type of computing device having at least a memory and processor module, and is capable of connecting to game server 260 via network 250. The backend game title processing engine 211 is configured to generate rendered images, which are delivered via network 250 for display at a corresponding display associated with client device 210. For example, through a cloud-based service, the game-rendered images can be delivered by an instance of the corresponding game running on game execution engine 211 of game server 260. That is, client device 210 is configured to receive encoded images (e.g., encoded from game-rendered images generated by executing a video game) and to display images rendered for display on display 11. In one embodiment, display 11 includes an HMD (e.g., displaying VR content). In some implementations, the rendered image may be obtained directly from a cloud-based service or via a client device 210 (e.g., Remote Play allows you to stream wirelessly or via wired connection to your smartphone or tablet.

[0058] In one implementation, the game server 260 and / or the game title processing engine 211 include basic processor-based functions for executing the game and services associated with the game application. For example, processor-based functions include 2D or 3D rendering, physics, physics simulation, scripting, audio, animation, graphics processing, lighting, shading, rasterization, ray tracing, shadows, culling, transformations, artificial intelligence, etc. Furthermore, services for the game application include memory management, multithreading management, Quality of Service (QoS), bandwidth testing, social networks, management of social friends, communication with friends on social networks, communication channels, text, instant messaging, chat support, etc.

[0059] In one implementation, the cloud gaming network 290 is a distributed game server system and / or architecture. Specifically, a distributed game engine executing game logic is configured as a corresponding instance of a game. Generally, the distributed game engine takes each function of the game engine and distributes these functions to multiple processing entities for execution. These functions can be further distributed across one or more processing entities. Processing entities can be configured with different configurations, including physical hardware, and / or as virtual components or virtual machines, and / or as virtual containers, where a container differs from a virtual machine because it virtualizes an instance of a game application running on a virtualized operating system. Processing entities can utilize and / or rely on servers and their underlying hardware on one or more servers (compute nodes) of the cloud gaming network 290, where the servers can reside on one or more racks. The coordination, allocation, and management of the execution of these functions by the various processing entities are performed by a distributed synchronization layer. In this way, the execution of these functions is controlled by the distributed synchronization layer to support the generation of media (e.g., video frames, audio, etc.) for the game application in response to player controller input. The distributed synchronization layer enables these functions to be performed efficiently across distributed processing entities (e.g., through load balancing), allowing critical game engine components / functions to be distributed and reassembled for more efficient processing.

[0060] The game title processing engine 211 includes a central processing unit (CPU) and a graphics processing unit (GPU) group configured to perform multi-tenant GPU functionality. In another embodiment, multiple GPU devices are combined to perform graphics processing for a single application running on a corresponding CPU.

[0061] Figure 2B This is a diagram according to one embodiment of the present disclosure for providing a game between two or more peer devices, wherein the VSYNC signal can be synchronized and offset to achieve optimal timing for receiving controller and other information between the devices. For example, head-to-head games can be performed using two or more peer devices connected via a network 250 or directly via peer-to-peer communication (e.g., Bluetooth, LAN, etc.).

[0062] As shown in the figure, the game is executed locally on each client device 210 (e.g., a game console) of the corresponding user playing the video game, with the client devices 210 communicating via a peer-to-peer network. For example, an instance of the video game is being executed by the game title processing engine 211 of the corresponding client device 210. The game logic 215 (e.g., executable code) that implements the video game is stored on the corresponding client device 210 and used to execute the game. For illustrative purposes, the game logic 215 may be delivered to the corresponding client device 210 via a portable medium (e.g., optical medium) or via a network (e.g., downloaded from a game provider via the Internet).

[0063] In one implementation, the game title processing engine 211 corresponding to client device 210 includes basic processor-based functions for executing the game and services associated with the game application. For example, processor-based functions include 2D or 3D rendering, physics, physics simulation, scripting, audio, animation, graphics processing, lighting, shading, rasterization, ray tracing, shadows, culling, transformation, artificial intelligence, etc. Furthermore, services for the game application include memory management, multithreading management, Quality of Service (QoS), bandwidth testing, social networks, management of social friends, communication with friends on social networks, communication channels, text, instant messaging, chat support, etc.

[0064] Client device 210 can receive input from various types of input devices, such as game controllers, tablets, keyboards, gestures captured by a camera, mice, touchpads, etc. Client device 210 can be any type of computing device that has at least a memory and processor module and is configured to generate rendered images executed by game title processing engine 211, and to display the rendered images on a display (e.g., display 11, or display 11 including a head-mounted display (HMD), etc.). For example, the rendered images can be associated with an instance of a game running locally on client device 210 to enable gameplay for the corresponding user, such as through input commands used to drive the game. Some examples of client device 210 include personal computers (PCs), game consoles, home theater systems, general-purpose computers, mobile computing devices, tablets, telephones, or any other type of computing device capable of executing game instances.

[0065] Figure 2C Various network configurations that benefit from proper synchronization and offset of the VSYNC signal between the source and target devices according to embodiments of this disclosure are illustrated, including Figures 2A to 2BThe configurations shown are examples of network configurations that benefit from proper frequency alignment of the server and client VSYNC signals, and timing offsets between them, to reduce one-way latency and / or latency variability between the server and client. For example, one network device configuration includes a cloud gaming server (e.g., source) to client (target) configuration. In one embodiment, the client may include a WebRTC client configured to provide audio and video communication within a web browser. Another network configuration includes a client (e.g., source) to server (target) configuration. Yet another network configuration includes a server (e.g., source) to server (e.g., target) configuration. Yet another network device configuration includes a client (e.g., source) to client (target) configuration, where each client may be a game console to, for example, provide head-to-head gaming.

[0066] Specifically, VSYNC signal alignment may include synchronizing the frequencies of the server VSYNC signal and the client VSYNC signal, and may also include adjusting the timing offset between the client VSYNC signal and the server VSYNC signal to eliminate drift and / or maintain an ideal relationship between the server VSYNC signal and the client VSYNC signal to reduce one-way latency and / or latency variability. In one embodiment, to achieve proper alignment, the server VSYNC signal may be tuned to achieve proper alignment between the server 260 and client 210 pair. In another embodiment, the client VSYNC signal may be tuned to achieve proper alignment between the server 260 and client 210 pair. Once the client and server VSYNC signals are aligned, the server VSYNC signal and the client VSYNC signal occur at substantially the same frequency and are offset from each other by a timing offset, which may be adjusted at any time. In another embodiment, VSYNC signal alignment may include synchronizing the frequencies of the VSYNC signals of the two clients, and may also include adjusting the timing offset between their VSYNC signals to eliminate drift and / or achieve optimal timing for receiving controller and other information; either VSYNC signal may be tuned to achieve this alignment. In another implementation, alignment may include synchronizing the frequencies of VSYNC across multiple servers, and may further include synchronizing the frequencies of server and client VSYNC signals, and adjusting the timing offset between the client and server VSYNC signals, for example, in head-to-head cloud gaming. In server-to-client and client-to-client configurations, alignment may include synchronizing the frequencies of server and client VSYNC signals, and providing a correct timing offset between the server and client VSYNC signals. In a server-to-server configuration, alignment may include synchronizing the frequencies of server and client VSYNC signals without setting a timing offset.

[0067] Figure 2D A multi-tenant configuration between a cloud gaming server 260 and one or more clients 210, according to one embodiment of the present disclosure, benefits from proper synchronization and offset of VSYNC signals between source and target devices. In a server-to-client configuration, alignment may include synchronizing the frequency between server and client VSYNC signals, and providing a proper timing offset between server and client VSYNC signals. In one embodiment, in the multi-tenant configuration, the client VSYNC signal is tuned at each client 210 to achieve proper alignment between the server 260 and client 210 pairs.

[0068] For example, in one implementation, a graphics subsystem may be configured to perform multi-tenant GPU functionality, whereby the graphics subsystem can implement graphics and / or rendering pipelines for multiple games. That is, the graphics subsystem is shared among multiple games being executed. Specifically, in one implementation, a game title processing engine may include a group of CPUs and GPUs configured to perform multi-tenant GPU functionality, whereby the CPU and GPU groups can implement graphics and / or rendering pipelines for multiple games. That is, the CPU and GPU groups are shared among multiple games being executed. The CPU and GPU groups may be configured as one or more processing devices. In another implementation, multiple GPU devices are combined to perform graphics processing for a single application running on a corresponding CPU.

[0069] Figure 3This diagram illustrates the general process of executing a video game at a server to generate game-rendered video frames and sending these frames to a client for display. Traditionally, many operations at game server 260 and client 210 are performed within frame periods defined by corresponding VSYNC signals. For example, server 260 attempts to generate game-rendered video frames at 301 within one or more frame periods defined by the corresponding server VSYNC signal 311. Video frames are generated by the game in response to control information delivered from an input device at operation 350 (e.g., user input commands) or game logic not driven by control information. When control information is sent to server 260, transmission jitter 351 may exist, where jitter 351 measures the variation in network latency from the client to the server (e.g., when sending input commands). As shown, the thick arrows indicate the current latency when control information is sent to server 260, but due to jitter, the control information at server 260 may have a range of arrival times (e.g., the range defined by the dashed arrows). At flip time 309, the GPU triggers a flip command, indicating that the corresponding video frame has been fully generated and placed in the frame buffer at server 260. Thereafter, server 260 performs a scan output / scan input (operation 302, where the scan output may be aligned with VSYNC signal 311) on the video frame within the subsequent frame period defined by server VSYNC signal 311 (VBI omitted for clarity). Subsequently, the video frame is encoded (operation 303) (e.g., encoding begins after the occurrence of VSYNC signal 311, and the end of encoding may not be aligned with the VSYNC signal) and transmitted (operation 304, where transmission may not be aligned with VSYNC signal 311) to client 210. At client 210, the encoded video frame is received (operation 305, where reception may not be aligned with client VSYNC signal 312), decoded (operation 306, where decoding may not be aligned with client VSYNC signal 312), buffered, and displayed (operation 307, where the start of display may be aligned with client VSYNC signal 312). Specifically, client 210 begins displaying each video frame rendered for display from the corresponding occurrence of client VSYNC signal 312.

[0070] The one-way latency 315 can be defined as the time from the start of transmitting a video frame to the encoding unit at the server (e.g., scan output 302) to the start of displaying the video frame 307 at the client. That is, considering client buffering, the one-way latency is the time from the server's scan output to the client's display. Each frame has a latency from the start of scan output 302 to the completion of decoding 306. This latency may vary from frame to frame due to server operations such as encoding 303 and transmission 304, network transmission between server 260 and client 210, and accompanying jitter 352 and height variations in client reception 305. As shown, the straight, thick arrows indicate the current latency when the corresponding video frame is sent to client 210, but due to jitter 352, the arrival time of video frames at client 210 may have a range (e.g., the range defined by the dashed arrows). Because one-way latency must be relatively stable (e.g., fairly consistent) for a good playback experience, the conventional result of performing buffer 320 is that the display of individual frames with low latency (e.g., from the start of scan output 302 to the completion of decoding 306) is delayed by several frame cycles. In other words, if network instability or unpredictable encoding / decoding times exist, additional buffering is required to keep the one-way latency consistent.

[0071] According to one embodiment of this disclosure, when streaming video frames generated from a video game executed on a server, the one-way latency between the cloud gaming server and the client may vary due to clock drift. Specifically, the frequency difference between the server's VSYNC signal 311 and the client's VSYNC signal 312 may cause the client's VSYNC signal to drift relative to frames arriving from the server 260. This drift may be due to subtle differences in the crystal oscillators used in each of the server's and client's respective clocks. Furthermore, embodiments of this disclosure reduce the one-way latency by implementing one or more of the synchronization and offset of the VSYNC signals to achieve alignment between the server and client, by providing dynamic buffering on the client, and by overlapping the decoding and display of video frames at the client.

[0072] Figure 4 The present disclosure illustrates a data stream, according to an embodiment of the present disclosure, during streaming of video frames generated from a video game executed on a server, via a network configuration including a highly optimized cloud gaming server 260 and a highly optimized client 210, wherein overlap of server and client operations reduces one-way latency, and synchronization and offset of VSYNC signals between the server and client reduce one-way latency as well as reduce the variability of one-way latency between the server and client. Specifically, Figure 4The desired alignment between the server VSYNC signal and the client VSYNC signal is illustrated. In one embodiment, tuning of the server VSYNC signal 311 is performed to achieve proper alignment between the server and client VSYNC signals, such as in a server-to-client network configuration. In another embodiment, tuning of the client VSYNC signal 312 is performed to achieve proper alignment between the server and client VSYNC signals, such as in a multi-tenant server-to-multi-client network configuration. For illustrative purposes, Figure 4 The tuning of the server VSYNC signal 311 is described in order to synchronize the frequencies of the server VSYNC signal and the client VSYNC signal, and / or adjust the timing offset between the corresponding client VSYNC signal and the server VSYNC signal. Although it should be understood that the client VSYNC signal 312 can also be used for tuning.

[0073] As shown in the figure Figure 4 An improved process is illustrated in an embodiment of this disclosure, in which a video game is executed at a server to generate rendered video frames and the video frames are sent to a client for display. The process is illustrated with respect to the generation and display of individual video frames at both the server and client. Specifically, at 401, the server generates a game-rendered video frame. For example, server 260 includes a CPU (e.g., a game title processing engine 211) configured to execute the game. The CPU generates one or more draw calls for the video frame, wherein the draw calls include commands placed in a command buffer for execution in the graphics pipeline by the corresponding GPU of server 260. The graphics pipeline may include one or more shader programs on the vertices of objects within the scene to generate texture values ​​rendered for the video frame for display, wherein, for efficiency, the operations are performed in parallel by the GPU. At flip time 409, the GPU touches a flip command in the command buffer, the flip command indicating that the corresponding video frame has been fully generated and / or rendered and placed in the frame buffer at server 260.

[0074] At 402, the server performs a scan output of the game-rendered video frames to the encoder. Specifically, the scan output is performed line by line or in groups of consecutive scan lines, where a scan line refers to a single horizontal line, such as a display from one edge of the screen to the other. These scan lines or groups of consecutive scan lines are sometimes referred to as slices, and are referred to as screen slices in this specification. Specifically, scan output 402 may include several processes that modify the game-rendered frame, including overlaying it with another frame buffer or shrinking it to surround it with information from another frame buffer. During scan output 402, the modified video frame is then scanned into the encoder for compression. In one embodiment, scan output 402 is performed at the occurrence 311a of the VSYNC signal 311. In other embodiments, scan output 402 may be performed before the occurrence of the VSYNC signal 311, such as at flip time 409.

[0075] At 403, the game-rendered video frame (which may have been modified) is encoded slice-by-slice at the encoder to generate one or more encoded slices, where the encoded slices are independent of scan lines or screen slices. Thus, the encoder generates one or more encoded (e.g., compressed) slices. In one embodiment, the encoding process begins before the scan output 402 process for the corresponding video frame has been fully completed. Furthermore, the start and / or end of encoding 403 may or may not be aligned with the server VSYNC signal 311. The boundary of the encoded slice is not limited to a single scan line and may consist of a single scan line or multiple scan lines. Furthermore, the end of the encoded slice and / or the start of the next encoder slice may not necessarily occur at the edge of the display (e.g., it may occur somewhere in the middle of the screen or scan line), so that the encoded slice does not need to traverse the entire display screen from edge to edge. As shown, one or more encoded slices may be compressed and / or encoded, including being compressed into an "encoded slice A" with a hash marker.

[0076] At 404, the encoded video frame is transmitted from the server to the client, wherein the transmission may occur on a per-encoded slice basis, where each encoded slice is a compressed encoder slice. In one embodiment, the transmission process 404 begins before the encoding process 403 of the corresponding video frame is fully completed. Furthermore, the start and / or end of the transmission 404 may or may not be aligned with the server's VSYNC signal 311. As shown, the compressed encoded slice A is transmitted to the client independently of other compressed encoder slices used to render the video frame. Encoder slices may be transmitted one at a time or in parallel.

[0077] At 405, the client again receives compressed video frames on a per-coded slice basis. Furthermore, the start and / or end of reception 405 may or may not be aligned with the client's VSYNC signal 312. As shown, compressed coded slice A is received by the client. Transmission jitter 452 may exist between server 260 and client 210, where jitter 452 measures the variation in network latency from server 260 to client 210. The lower the jitter value, the more stable the connection. As shown, the thick straight arrows indicate the current latency when the corresponding video frame is sent to client 210, but due to jitter, the video frames at client 210 may have a range of arrival times (e.g., the range defined by the dashed arrows). Variations in latency may also be due to one or more operations at the server (such as encoding 403 and transmission 404) and networking issues that introduce latency when transmitting video frames to client 210.

[0078] At 406, the client again decodes the compressed video frame on a per-code-slice basis, producing decoded slice A (displayed as a hash-free tag), which is now ready for display. In one embodiment, the decoding process 406 begins before the receiving process 405 of the corresponding video frame is fully completed. Furthermore, the start and / or end of decoding 406 may or may not be aligned with the client's VSYNC signal 312. At 407, the client displays the decoded rendered video frame on its display. That is, for example, the decoded video frame is placed in a display buffer, which is streamed to the display device on a per-scan-line basis. In one embodiment, the display process 407 (i.e., streaming to the display device) begins after the decoding process 406 of the corresponding video frame is fully completed, i.e., the decoded video frame is fully residing in the display buffer. In another embodiment, the display process 407 begins before the decoding process 406 of the corresponding video frame is fully completed. That is, the streaming from the address of the display buffer to the display device occurs during the time when only a portion of the decoded frame buffer resides in the display buffer. Then, the display buffer is updated or populated in a timely manner with the remaining portions of the corresponding video frames for display, ensuring that the display buffer update is performed before these portions are streamed to the display. Furthermore, the start and / or end of display 407 are aligned with the client's VSYNC signal 312.

[0079] In one embodiment, the one-way latency 416 between server 260 and client 210 can be defined as the time elapsed between the start of scan output 402 and the start of display 407. Embodiments of this disclosure are capable of aligning the VSYNC signals between the server and client (e.g., synchronizing frequencies and adjusting offsets) to reduce the one-way latency between the server and client, and to reduce the variability of the one-way latency between the server and client. For example, embodiments of this disclosure are capable of calculating the optimal adjustment of the offset 430 between the server VSYNC signal 311 and the client VSYNC signal 312, such that even under near-worst-case times for server processing (such as encoding 403 and transmission 404), near-worst-case network latency between server 260 and client 210, and near-worst-case client processing (such as receiving 405 and decoding 406), the decoded rendered video frames can be used in time for the display process 407. That is, it is not necessary to determine the absolute offset between the server VSYNC and the client VSYNC; adjusting the offset so that the decoded rendered video frames can be used in time for the display process is sufficient.

[0080] Specifically, the frequencies of the server VSYNC signal 311 and the client VSYNC signal 312 can be aligned through synchronization. Synchronization is achieved by tuning either the server VSYNC signal 311 or the client VSYNC signal 312. For illustrative purposes, tuning is described with reference to the server VSYNC signal 311, but it should be understood that tuning the client VSYNC signal 312 can be performed alternatively. For example, as... Figure 4 As shown, the server frame period 410 (e.g., the time between two occurrences 311c and 311d of the server VSYNC signal 311) is essentially equal to the client frame period 415 (e.g., the time between two occurrences 312a and 312b of the client VSYNC signal 312), which indicates that the frequencies of the server VSYNC signal 311 and the client VSYNC signal 312 are also essentially equal.

[0081] To maintain frequency synchronization between the server and client VSYNC signals, the timing of the server VSYNC signal 311 can be manipulated. For example, the vertical blanking interval (VBI) in the server VSYNC signal 311 can be increased or decreased over a certain period of time to resolve drift between the server VSYNC signal 311 and the client VSYNC signal 312. Manipulating the vertical blanking (VBLANK) lines in the VBI provides adjustment of the number of scan lines used for VBLANK for one or more frame periods of the server VSYNC signal 311. Decreasing the number of scan lines for VBLANK reduces the corresponding frame period (e.g., time interval) between two occurrences of the server VSYNC signal 311. Conversely, increasing the number of scan lines for VBLANK increases the corresponding frame period (e.g., time interval) between two occurrences of the VSYNC signal 311. In this way, the frequency of the server VSYNC signal 311 is adjusted to align the frequencies of the client and server VSYNC signals 311 and 312 to substantially the same frequency. Furthermore, the offset between the server's VSYNC signal and the client's VSYNC signal can be adjusted by increasing or decreasing the VBI for a short period of time before returning the VBI to its original value. In one embodiment, the server's VBI is adjusted. In another embodiment, the client's VBI is adjusted. In yet another embodiment, instead of two devices (server and client), there are multiple connected devices, each of which may have a corresponding VBI that has been adjusted. In one embodiment, each of the multiple connected devices may be an independent peer device (e.g., a serverless device). In another embodiment, the multiple devices may include one or more server devices and / or one or more client devices arranged in one or more server / client architectures, multi-tenant server / client architectures, or some combination thereof.

[0082] Alternatively, in one embodiment, the server's pixel clock (e.g., located at the southbridge of the server's northbridge / southbridge core logic chipset) can be manipulated to perform coarse and / or fine adjustments to the frequency of the server VSYNC signal 311 over a period of time, so that the frequency synchronization between the server VSYNC signal 311 and the client VSYNC signal 312 returns to alignment. Specifically, the pixel clock in the server's southbridge can be overclocked or underclocked to adjust the overall frequency of the server's VSYNC signal 311. In this way, the frequency of the server VSYNC signal 311 is adjusted to align the frequencies of the client and server VSYNC signals 311 and 312 at substantially the same frequency. The offset between the server and client VSYNC can be adjusted by increasing or decreasing the client-server pixel clock over a short period of time before restoring the pixel clock to its original value. In one embodiment, the server pixel clock is adjusted. In another embodiment, the client pixel clock is adjusted. In yet another embodiment, instead of two devices (server and client), there are multiple connected devices, each of which may have a corresponding pixel clock that has been adjusted. In one embodiment, each of the multiple connected devices may be an independent peer device (e.g., a serverless device). In another embodiment, the multiple connected devices may include one or more server devices and one or more client devices arranged in one or more server / client architectures, multi-tenant server / client architectures, or some combination thereof.

[0083] Figure 5A This is a diagram illustrating possible variations in the timing of client 210 completing decoding 406 when streaming video frames generated from a video game executed on server 260, due to drift 390 between the corresponding clocks at cloud gaming server 260 and client 210, and variations in the time spent by server operations such as encoding 403 and transmission 404, network latency, and client operations such as receiving 405 and decoding 406.

[0084] The y-axis 501 displays time in milliseconds. The x-axis 502 displays time in minutes. In embodiments of this disclosure, server 260 sends timestamp information along with compressed video frames to client 210, or the server may send the timestamp information separately from the compressed video frames. In one embodiment, the timestamp information may represent the time, derived from the pixel clock of server 260, that the server VSYNC signal appears immediately preceding the scan output 402 of the corresponding video frame. That is, the timestamp gives an indication of the expected display timing of the corresponding video frame, such as whether the video frame should be displayed immediately. In another embodiment, the timestamp information may represent the time of flip time, derived from the pixel clock of server 260, for example, the time when the rendering of the corresponding video frame is completed. In yet another embodiment, a regular frame period is used instead of the server VSYNC signal, and the timestamp information may represent the start or end time of the corresponding frame period, derived from the pixel clock of server 260.

[0085] Upon completion of decoding 406, client 206 records the time derived from the client's pixel clock and subtracts that time from the timestamp (delivered from the server) to create a "decoding timestamp" (i.e., the time decoding was completed, not the time spent decoding the corresponding video frame). Therefore, the decoding timestamp provides an indication of the availability at the client to display the corresponding video frame relative to the expected display time specified by the server (e.g., indicated by the timestamp). Figure 5A As shown, in one implementation, the decoding timestamp of the first compressed video frame received by the client is assigned a zero value (i.e., it is normalized). Furthermore, all other decoding timestamps are calculated with reference to normalization (i.e., subtracted and normalized). Due to variations in server and client operations and network latency, the decoding timestamps measured for a series of compressed video frames can be plotted as a distribution 510, where the first compressed video frame 511 is assigned a zero decoding timestamp, as previously described. These decoding timestamps can be combined to create a histogram 515, as shown. Figure 5B As shown below, a more comprehensive description will follow. Measurements 520, 530, and 540 illustrate the distribution of decoding timestamps calculated for subsequent video frames received at the client. The decoding timestamps in measurements 520, 530, and 540 are calculated with reference to the normalization defined by the first compressed video frame 511.

[0086] Figure 5BThis includes a histogram illustrating the timing of client-completed decoding relative to the desired display time specified by the server (e.g., a corresponding timestamp), according to one embodiment of this disclosure, and showing the increase in decoding timestamps in subsequent histograms due to drift between the corresponding clocks at the cloud gaming server and the client. Specifically, the decoding timestamps of each of measurements 520, 530, and 540, later determined for subsequent video frames, can be combined to create a corresponding histogram. For example, the decoding timestamps plotted in measurement 520 can be combined to create histogram 525. Similarly, the decoding timestamps plotted in measurement 530 can be combined to create histogram 535, and the decoding timestamps plotted in measurement 540 can be combined to create histogram 545. Each of histograms 515, 525, 535, and 545 may have unique characteristics. As shown in the figure, even though the width of the distribution of the decoded timestamps in each of the measurements 515, 525, 535, and 545 is approximately similar, the histogram determined later shows an increase in the decoded timestamps because they are each shifted to the right in time, such as due to the drift between the server and client clocks used to generate the VSYNC frequency.

[0087] Specifically, measurements of the decoding timestamps determined at later times for subsequent video frames: 520, 530, and 540 (and...) Figure 5B The corresponding histograms 525, 535, and 545 shown in the figure illustrate the difference between the server VSYNC signal 311 and the client VSYNC signal 312 (i.e., Figure 3 The effect of drift 390). The drift 390 between the server and client clocks, and the drift 390 reflected in the corresponding VSYNC signal, can... Figure 5B The line is shown as line 590. For example, due to the inaccuracy of the crystal oscillators used to generate the VSYNC frequency at the server and client, a 10 ppm (parts per million) difference in the clocks at the server and client will cause a drift of approximately one frame period (16.7 ms) every 30 minutes between the server and client; the drift can take the form of decreasing or increasing the decoding timestamp, depending on whether the frequency of the server's VSYNC 311 is slightly higher or slightly lower than that of the client's VSYNC signal 312.

[0088] Figure 5CThis includes a histogram showing the timing of client-side decoding completion relative to the desired display time specified by the server, according to one embodiment of this disclosure, and showing consistent measurements of decoding time in subsequent histograms after compensating for measurement drift between the corresponding clocks at the cloud gaming server and the client. In one embodiment, dynamically regenerating the histogram allows for the calculation of the drift amount; understanding the drift amount allows for corresponding adjustments to the VSYNC frequency and the elimination of drift, resulting in histograms such as 516, 526, 536, and 546 over time. In this way, the plotted measurements 520, 530, and 540 will not show an increase in the decoding timestamp (e.g., along...). Figure 5A (The x-axis 501 is shifted vertically upwards), but it will be drawn as... Figure 5A The measurement 510 in the middle is more horizontally aligned (i.e., it shows no increase in one-way waiting time). This is reflected in Figure 5C In the diagram, line 590' shows zero drift between the server and client clocks, and is reflected in the corresponding VSYNC signal (i.e., the one-way latency is not increased as shown in histograms 526, 536, and 546).

[0089] In another implementation, instead of sending a corresponding server timestamp (e.g., used in decoding timestamp calculation) from the server to the client at each frame period (where the lengths of the frame periods are approximately equal), the client calculates the corresponding server timestamp (for the corresponding video frame being received) by adding the frame period to a previously calculated server timestamp (e.g., for a previously received video frame). An initial server timestamp and / or timing signal of the initial video frame can be delivered from the server to the client to initiate the timing process.

[0090] In another implementation, the drift is calculated using timestamp information sent from the server to the client (either separately from or together with the compressed video frames), for example, by analyzing the timing difference between the timestamp information and the timestamp information received at the client. In still other implementations, instead of using a server VSYNC signal, the server uses a frame period of approximately equal size, and the drift (or a multiple thereof) of the server frame period relative to the client VSYNC can be calculated. Therefore, the server frame period can be adjusted in response to the drift calculation.

[0091] In other embodiments, instead of two devices (server and client), there are multiple connected devices, each of which can measure its drift relative to one or more other devices. In one embodiment, each of the multiple connected devices may be an independent peer device (e.g., a serverless device). In another embodiment, the multiple devices may include one or more server devices and one or more client devices arranged in one or more server / client architectures, multi-tenant server / client architectures, or some combination thereof.

[0092] Therefore, a measured drift between the frequencies of the VSYNC signals between two devices (e.g., a server and a client, or any two of multiple networked devices having a server, a client, and independent peers) can be used to adjust the VSYNC signals at one or more devices. Tuning may include removing or adding raster scan lines for the vertical blanking interval of the corresponding frame period. Tuning may also include overclocking or underclocking the corresponding clock. Tuning can be performed dynamically, where the adjustment of the corresponding VSYNC signal varies over time. In some implementations, the server may use a frame period of approximately equal size instead of a VSYNC signal, and the server frame period may be adjusted in response to drift calculations.

[0093] In addition, align the server's VSYNC signal and the client's VSYNC signal (such as...) Figure 4 (As shown) may include adjusting the server VSYNC signal 311 to provide a correct timing offset 430 between the corresponding client VSYNC signal and the server VSYNC signal. In one embodiment, histogram data (e.g., at least in Figures 5A to 5C As shown in the diagram, a timing offset can be determined so that a predetermined number or threshold (e.g., 99.99%) of received video frames arrive at the client in time for display when the client's VSYNC signal 312 next occurs appropriately. That is, adjusting the timing offset 430 (e.g., ...) allows for the determination of a timing offset such that ... Figure 4 As shown), this is to accommodate near-worst-case scenarios for video frames, ensuring that the video frame is ready for display in time with minimal time (e.g., received, decoded, and placed in the display buffer for streaming output at the client) before the next occurrence of the client's VSYNC signal 312. For illustration, the 99.99% threshold provides for one missing video frame out of ten thousand video frames generated at server 260 and displayed at client 210.

[0094] Specifically, to establish the correct offset 430, the frequency of the server VSYNC signal 311 can be manipulated over a period of time (e.g., one or more frame periods) to shift the timing of one or more occurrences of the server VSYNC signal 311, thereby shifting or adjusting the relative timing offset between the client and server VSYNC signals once the two VSYNC signals have synchronized their respective frequencies. Alternatively, the frequency of the client VSYNC signal 312 can also be manipulated to adjust the relative timing offset between the client and server VSYNC signals. The determination of the correct offset can be performed dynamically (e.g., repeatedly over time), with corresponding dynamic manipulation of the VSYNC signals. In other embodiments, instead of first eliminating the drift between the server and client VSYNC signals and then establishing the correct offset between the VSYNC signals, the offset is maintained by adjusting the relative timing offset more frequently by manipulating the frequency of either the server or client VSYNC signal. In some embodiments, instead of using a server VSYNC signal, the server uses a frame period of approximately equal size and can manipulate the server frame period or the client VSYNC signal to adjust the relative timing offset (or a multiple thereof) of the server frame period relative to the client VSYNC. In other embodiments, instead of two devices (server and client), there are multiple connected devices, each of which can manipulate its VSYNC signal to adjust the relative timing offset of its VSYNC relative to the VSYNC of one or more other devices. In one embodiment, each of the multiple connected devices can be an independent peer device (e.g., a serverless device). In another embodiment, the multiple devices may include one or more server devices and one or more client devices arranged in one or more server / client architectures, multi-tenant server / client architectures, or some combination thereof.

[0095] Through the Figures 2A to 2D Detailed description of various client devices 210 and / or cloud gaming networks 290 (e.g., in game servers 260), Figures 6A to 6C Flowcharts 600A, 600B, and 600C illustrate a method for tuning a VSYNC signal between a cloud gaming server and a client to reduce one-way latency, according to one embodiment of this disclosure.

[0096] Specifically, Figure 6A A method for adjusting the relative timing between the VSYNC signals of a cloud gaming server and a client to reduce one-way latency is illustrated according to one embodiment of the present disclosure. For example, flowchart 600A can be executed to adjust... Figure 4 The timing between the corresponding client and server VSYNC signals is shown.

[0097] At 601, the method includes setting a server VSYNC signal to a server VSYNC frequency at the server. As previously described, the server VSYNC signal corresponds to the generation of multiple video frames at the server during multiple frame periods for the server VSYNC frequency. For example, the server may execute a video game in a streaming mode, causing the server's CPU to execute the video game in response to input commands from the user, so as to use a graphics pipeline to generate game-rendered video frames that can be used for streaming.

[0098] At 603, the method includes setting a client VSYNC signal to a client VSYNC frequency at the client end. The client VSYNC signal is used for rendering to a display associated with the client. That is, the timing for rendering and displaying video frames at the client end can be referenced to the client VSYNC signal. For example, video frames can be displayed starting from the corresponding occurrence of the client VSYNC signal.

[0099] In one implementation, the client VSYNC frequency is approximately set to the server VSYNC frequency. For example, the server may send a control signal to the client, which the client uses to set its client VSYNC signal to the apparent frequency of the server VSYNC signal. That is, the control signal may include the apparent frequency to which the client VSYNC signal is set. In other words, the client VSYNC frequency is set to the same apparent frequency as the server VSYNC frequency, although the actual server and client VSYNC frequencies may not match due to differences in the crystal oscillators used for clocking at the server and client.

[0100] At 605, the method includes sending multiple compressed video frames, based on multiple video frames, from a server to a client over a network using a server VSYNC signal. Specifically, game-rendered video frames generated in response to the server processing the video game in streaming mode are delivered (e.g., during scan output 402) to an encoder configured to perform compression and produce multiple compressed video frames. As previously described, the encoding start of a corresponding video frame may be aligned with the corresponding occurrence of the server VSYNC signal, or may occur before the corresponding occurrence, such as during a flip. The compressed video frames are transmitted and / or delivered to the client for display during a game session. Figure 4 As shown, the transmission of video frames does not need to start in alignment with the server's VSYNC signal, and can begin immediately after a portion or the entire video frame has been encoded.

[0101] At 607, the method includes, at the client, decoding and displaying a plurality of compressed video frames. As previously described, the client receives a plurality of compressed video frames, which are then decoded by a decoder at the client. For example, the client receives one or more encoded slices corresponding to the compressed video frames. The compressed video frames are then decoded and placed into a display buffer. For example, the encoded slices corresponding to the compressed video frames are then decoded such that the decoded video frames are placed in the display buffer. During decoding, the decoded slices may be rendered for display, wherein rendering includes generating screen slices (e.g., scan lines) from the decoded slices of the corresponding video frames, and then streaming the screen slices to the client's display. Specifically, pixel data of the decoded slices of the corresponding video frames may be placed in the correct address of the display buffer for streaming (e.g., per scan line) to the display at the client.

[0102] At 610, the method includes analyzing the timing of one or more client operations as the client receives multiple compressed video frames to adjust the relative timing between the server VSYNC signal and the client VSYNC signal. For example, adjusting the relative timing to achieve proper alignment between the server VSYNC signal and the client VSYNC signal (e.g., synchronizing frequencies and adjusting offsets) reduces one-way latency and reduces the variability of one-way latency between the server and the client. In one embodiment, proper timing between the server VSYNC signal and the client VSYNC signal is achieved by adjusting at least one of the server VSYNC signal and the client VSYNC signal. In the following... Figures 6B to 6C The document provides a more detailed discussion of adjusting the relative timing between the server and client VSYNC signals.

[0103] Figure 6B This is a flowchart 600B illustrating a method for aligning VSYNC signals during tuning between a cloud gaming server and a client, according to one embodiment of this disclosure. For example, alignment includes synchronizing frequencies and / or adjusting offsets between the server's and client's VSYNC signals, such as to reduce one-way latency. Specifically, Figure 6B Provided for in Figure 6A Additional details regarding adjusting the relative timing between the server's VSYNC signal and the client's VSYNC signal, as outlined in Operation 610.

[0104] At 611, the method includes sending timestamp information associated with multiple video frames (e.g., generated by the server as video frames for game rendering) from a server to a client. In one embodiment, the timestamp information is sent to the client along with the multiple compressed video frames. In another embodiment, the timestamp information is sent to the client separately from the multiple compressed video frames.

[0105] Specifically, the timestamp information includes a corresponding timestamp of the corresponding video frame generated at the server, such as one derived from the server's pixel clock. In one implementation, the timestamp of the corresponding video frame may appear at the corresponding occurrence of the server's VSYNC signal used to scan the corresponding video frame to the encoder, such as during scan output (e.g., immediately preceding the scan output of the corresponding video frame). In another implementation, the timestamp of the corresponding video frame may appear at the server at the corresponding flip time. The timestamp provides an indication of the expected display timing of the corresponding video frame as determined by the video game, such as when scanning or streaming to a local display without transmission over a network.

[0106] As the client receives and decodes compressed video frames, timestamp information is processed to create a decoding timestamp, which indicates the availability at the client to display the corresponding video frame relative to the expected display time (timestamp) specified by the server. As previously described, the client records the time derived from its pixel clock when decoding the corresponding video frame is complete. This decoding completion time is subtracted from the corresponding timestamp delivered by the server to create the decoding timestamp. Furthermore, the first compressed video frame being decoded can be normalized such that its decoding timestamp is adjusted to zero using a normalization factor that can be applied (e.g., added to or subtracted from) all subsequently measured and / or calculated decoding times for subsequently received compressed video frames at the client.

[0107] At 613, the method includes constructing one or more histograms based on measured and / or calculated decoding timestamps. Specifically, corresponding histograms are created by merging decoding timestamp information measured for compressed video frames received at the client within a certain time period. As described above, the decoded video frame, measured and normalized for the corresponding video frame, provides an indication of when the decoded video frame is available for display at the client, relative to the expected display time indicated by a server timestamp. In this way, for multiple video frames, the corresponding histograms provide a distribution of the timing at which the client completes decoding relative to the expected display time specified by the server. Therefore, one or more generated histograms can be used to adjust the relative timing between the server's VSYNC signal and the client's VSYNC signal to reduce one-way latency and / or variations in one-way latency between the server and the client.

[0108] At point 615, the relative timing between the server VSYNC signal and the client VSYNC signal is adjusted to synchronize the server VSYNC frequency of the server VSYNC signal and the client VSYNC frequency of the client VSYNC signal. In one implementation, a corresponding histogram is used to determine the drift between the server VSYNC signal and the client VSYNC signal. For example, one or more histograms generated from video frames received from the client are continuously and dynamically updated. The histograms are analyzed to determine the drift between the server VSYNC signal and the client VSYNC signal. For example, Figure 5B This illustrates how drift (e.g., as reflected by line 590) can be determined when plotting multiple histograms, as previously described.

[0109] Alternative methods for determining drift between server and client VSYNC signals can be used for synchronization. For example, decoding timestamps over a period of time can be analyzed to determine a trend of increasing or decreasing decoding timing, which can be used to determine drift. In another example, drift can be calculated by analyzing the difference between timestamp information generated at the server and client-based timing received based on the server's timestamp information. Furthermore, drift can be measured between multiple connected devices, which can be independent peer devices, or server and client devices arranged in a peer-to-peer architecture, a server / client architecture, or some combination thereof.

[0110] Furthermore, based on the timestamp information sent from the server to the client, at least one of the server's VSYNC frequency and the client's VSYNC frequency can be adjusted to compensate for the measured drift. For example, the frequency of either the server's or client's VSYNC signal can be adjusted within a certain time period so that the actual frequencies of the server's and client's VSYNC signals are approximately similar during that time period. In this way, the frequencies of the server's and client's VSYNC signals are synchronized.

[0111] As previously described, the frequency of the corresponding server or client VSYNC signal can be tuned by deleting or adding raster scan lines for the corresponding frame period's vertical blanking interval, wherein the VSYNC signal can be adjusted within a certain time period. Tuning may include overclocking or underclocking the corresponding pixel clock of the server or client within a certain time period. Furthermore, tuning can be performed continuously by dynamically adjusting the corresponding VSYNC signal appropriately over time.

[0112] At point 617, the relative timing between the server and client VSYNC signals is adjusted by adjusting the relative offset between them based on timestamp information. Similarly, the relative phase between the server and client VSYNC signals can be adjusted based on timestamps. Once the frequencies of the server and client VSYNC signals are synchronized, adjustments to the relative phase or offset between them can be applied to either the server or client VSYNC signals.

[0113] Specifically, adjustments to the offset between the server and client VSYNC signals are determined based on the near-worst-case decoding timestamps indicated by the corresponding histogram. For example, a timing offset can be determined such that a predetermined number or threshold (e.g., 99.99%) of received video frames arrive at the client in time for decoding and display when the client VSYNC signal next appears appropriately. In this way, when adjusting the timing offset between the server and client VSYNC signals, even considering near-worst-case scenarios of one-way latency for video frames, video frames in near-worst-case scenarios are received, decoded, and rate-controlled in the display buffer for streaming to the client display. (See reference...) Figures 8A to 8B The determination of the correct timing offset is further described.

[0114] As previously described, the timing offset between the server and client VSYNC signals can be adjusted by tuning the server or client VSYNC signals within one or more frame periods. For example, the timing offset can be adjusted by changing the frequency of the corresponding server or client VSYNC signal within one or more frame periods. Specifically, the frequency of the corresponding server or client VSYNC signal can be adjusted by deleting or adding raster scan lines for the vertical blanking interval of the corresponding frame period, or by overclocking or underclocking the corresponding pixel clock of the server or client.

[0115] In some implementations, instead of first eliminating the drift between the server and client VSYNC signals and then establishing the correct offset between them, the timing offset is maintained by manipulating the frequency of either the server or client VSYNC signal more frequently to adjust the relative timing offset. That is, adjustments to the offset between the server and client VSYNC signals are continuously determined based on near-worst-case decoding timestamps indicated by corresponding histograms, where histograms can be generated over shortened time intervals to allow for frequent determination and manipulation of the timing offset.

[0116] Figure 6C This is a flowchart 600C illustrating another method for aligning VSYNC signals to reduce one-way latency during tuning of VSYNC signals between a cloud gaming server and a client, according to one embodiment of this disclosure. As previously described, alignment includes synchronizing frequencies and / or adjusting offsets between the server's VSYNC signal and the client's VSYNC signal, such as to reduce one-way latency. Specifically, Figure 6C Provided in Figure 6A Additional details regarding adjusting the relative timing between the server's VSYNC signal and the client's VSYNC signal, as outlined in Operation 610.

[0117] Specifically, alternative methods for determining the drift between the server's VSYNC signal and the client's VSYNC signal are used for synchronization, as outlined in operation 620, which includes operations 621 and 623. Specifically, at 621, the method includes periodically sending timing information from the server to the client. For example, timing information determined from the server's pixel clock may include the start of a corresponding frame period, the duration of the corresponding frame period, the timing of the video frame's scan output to the encoder, the flip time of the corresponding video frame, etc.

[0118] Furthermore, at 623, timing information is analyzed to determine the drift between the server VSYNC signal and the client VSYNC signal. For example, the drift can be calculated by analyzing the difference between timing information generated at the server and client-based timing information received based on the server's timing information. In other embodiments, the drift can be measured against the server frame period used to generate video frames, wherein the drift of the server frame period is measured with reference to the client VSYNC signal or a multiple thereof. Additionally, drift can be measured between multiple connected devices, which can be independent peer devices, or server and client devices arranged in a peer-to-peer architecture, a server / client architecture, or some combination thereof.

[0119] Once the drift is determined, the frequency of the server's or client's VSYNC signal can be adjusted to compensate for it, with the adjustment applied over a period of time. As previously described, the measured drift between the frequencies of the server's and client's VSYNC signals can be used to adjust the server's or client's VSYNC signal over a period of time. Tuning the VSYNC signal may include removing or adding raster scan lines for the vertical blanking interval of the corresponding frame period, or it may include overclocking or underclocking the corresponding pixel clock of the server or client.

[0120] After compensating for the frequency drift between the server and client VSYNC signals, the relative phase or offset between the server and client VSYNC signals can be adjusted based on timestamp information, wherein the adjustment can be applied to either the server or client VSYNC signal. Specifically, the relative phase or offset adjustment is performed based on timestamp information associated with video frames generated by the server.

[0121] Specifically, at point 611, the timestamp information is sent from the server to the client. In the implementation, the timestamp information is sent along with multiple video frames, or sent separately from multiple video frames. Operation 611 of flowchart 600C has been previously referred to. Figure 6B The flowchart 600B describes this. For example, the timestamp information determined by the server pixel clock can indicate the timing of the corresponding occurrence of the server VSYNC signal used to scan the corresponding video frame to the encoder (e.g., during scan output). In another implementation, the timestamp information can indicate the timing of the corresponding flip time of the corresponding video frame.

[0122] As previously mentioned, the client uses timestamp information to create decoding timestamps. Each decoding timestamp indicates the availability of the corresponding video frame for display at the client relative to the expected display time (timestamp) specified by the server. Decoding timestamps can be derived by subtracting the time indicated at the client that decoding of the corresponding video frame is complete from the corresponding server-based timestamp information. Normalization can also be applied when generating decoding timestamps.

[0123] Since drift can be performed without using timestamp information, such as generating multiple histograms, a histogram is generated at a specific time, and this histogram is used to adjust the timing offset between the server's VSYNC signal and the client's VSYNC signal. That is, the histogram can be updated by expanding the decoded timestamps included within it, but it is not necessary to generate multiple histograms for different time periods. Specifically, at 613-A, the histogram is constructed based on the decoded timestamps. Operation 613-A of flowchart 600C is similar to the previously referenced... Figure 6B The operation 613 is described in flowchart 600B. For example, for video frames received and decoded by the client, decoding timestamp information is merged over time to provide a distribution of the timing of the client completing decoding relative to the expected display time specified by the server (e.g., server timestamp information).

[0124] At 617, the relative phase or offset between the server VSYNC signal and the client VSYNC signal can be adjusted based on timestamp information, wherein the adjustment can be applied to either the server VSYNC signal or the client VSYNC signal. Operation 617 of flowchart 600C has previously been referenced. Figure 6B Flowchart 600B describes this. Specifically, the adjustment of the offset is determined based on the near-worst-case decoding timestamps indicated by a continuously updated histogram. For example, a timing offset can be determined such that a predetermined number of received video frames (e.g., 99.99%) arrive at the client in time for decoding and display when the client's VSYNC signal next occurs appropriately. (Refer to...) Figures 8A to 8B The determination of the correct timing offset is further described.

[0125] The timing offset between the server VSYNC signal and the client VSYNC signal is adjusted by tuning the server VSYNC signal or the client VSYNC signal within one or more frame periods. For example, the adjustment can be performed by adjusting the frequency of the server VSYNC signal or the client VSYNC signal for one or more frame periods, by deleting or adding raster scan lines for the vertical blanking interval of the corresponding frame period, or by overclocking or underclocking the corresponding pixel clock of the server or client.

[0126] In some implementations, when the correct offset is established between the VSYNC signals, no action is taken. Figure 6C The drift operation 620. Instead, the timing offset is maintained by adjusting the relative timing offset through more frequent manipulation of the frequency of the server VSYNC signal or the client VSYNC signal. That is, the adjustment of the offset between the server VSYNC signal and the client VSYNC signal is continuously determined based on the near-worst-case decoding timestamp indicated by the corresponding histogram, wherein the adjustment of the offset can be performed at shorter intervals.

[0127] Through the Figures 2A to 2D Detailed description of various client devices 210 and / or cloud gaming networks 290 (e.g., in game servers 260), Figure 7 Flowchart 700 illustrates an alternative method for reducing one-way latency between a cloud gaming server and a client, according to one embodiment of the present disclosure. Specifically, flowchart 700 illustrates a method for tuning a client's VSYNC signal by referring to compressed video frames generated at the server, according to one embodiment of the present disclosure, wherein the video frames are generated during frame periods of similar size.

[0128] At 710, the method includes generating multiple video frames at the server during multiple frame periods, wherein the frame periods are approximately equal in size. In one embodiment, the cloud gaming server may disable or not implement a server VSYNC signal because a display is not required at the server when streaming video frames to the client. Instead, when processing video games, the server may utilize regular (e.g., periodic) or near-regular signals for timing during the generation of game-rendered video frames. For example, the server may use frame periods of approximately equal size instead of a server VSYNC signal. The generation of multiple video frames occurs within multiple frame periods, such that game-rendered video frames are generated within corresponding frame periods. The server may execute the video game in a streaming mode, such that the server's CPU executes the video game in response to input commands from the user to generate game-rendered video frames using a graphics pipeline.

[0129] At 720, the method includes setting a client VSYNC signal to a client VSYNC frequency at the client end. The client VSYNC signal is used for rendering to a display associated with the client. Timing for rendering and displaying video frames on the client end can be referenced to the client VSYNC signal, such that a corresponding video frame can be displayed starting from the corresponding occurrence of the client VSYNC signal.

[0130] At 730, the method includes sending multiple compressed video frames, based on multiple video frames, from a server to a client. Specifically, game-rendered video frames are delivered (e.g., during scan output 402) to an encoder at the server, wherein the encoder is configured to perform compression on the game-rendered video frames. The multiple compressed video frames are transmitted (e.g., streamed) to the client for display, such as during a game session. The transmission of the compressed video frames does not need to be aligned with the frame period, such that transmission can occur while a portion of the corresponding video frame is encoded, or while the entire video frame is encoded.

[0131] At 740, the method includes, at the client, decoding and displaying multiple compressed video frames. As previously described, the client receives and decodes the multiple compressed video frames. For example, the client may receive one or more encoded slices of a corresponding compressed video frame and then decode them. The decoded video frames are placed in a display buffer. During decoding, the decoded slices may be rendered for display, wherein rendering includes generating screen slices (e.g., scan lines) from the decoded slices of the corresponding video and then streaming the screen slices to the client's display. For example, pixel data of the decoded slices of the corresponding video frame may be placed in the correct address of the display buffer for streaming (e.g., scan line-by-scan line) to the display.

[0132] At 750, the method includes sending timing information associated with multiple frame periods from a server to a client. The timing information indicates when each frame period begins at the server. Because the frame periods are approximately equal, the timing of one frame period delivered to the client (e.g., a server timestamp) allows the client to track the timing of each frame period (e.g., periodically adding the frame period to the last calculated timestamp). In this way, the client can associate timing information received from the server or calculated at the client with the corresponding video frame received at the client. The timing information determined at the client can provide an indication of the expected display timing of the corresponding video frame determined by a video game executed at the client, such as when the video frame has been generated and theoretically scanned or streamed to a local display without transmission over a network.

[0133] At 760, the method includes analyzing the timing of one or more client operations as the client receives multiple compressed video frames to adjust the relative timing of the generation of the multiple compressed video frames at the server and the client's VSYNC signal. Specifically, the server frame period (e.g., duration) or the client VSYNC signal can be manipulated to adjust the relative timing offset (or a multiple thereof) of the server frame period relative to the client VSYNC signal.

[0134] For example, the drift between the server frame period calculated by the client and the client VSYNC signal can be determined. Compensation for this drift can be applied to either the server frame period or the client VSYNC signal for synchronization. For example, at the client, the frequency of the client VSYNC signal can be tuned by removing or adding raster scan lines for the vertical balance interval of the corresponding frame period, where the VSYNC signal can be adjusted over a certain time period. Furthermore, tuning may include overclocking or underclocking the client's pixel clock. Additionally, to adjust for relative offset, one or more histograms can be constructed at the client to indicate when the decoded video frame is available at the client relative to when the video frame is generated at the server (e.g., the desired display time). The same techniques described above can be used to construct the histogram, with only minor modifications, such as using the frame period determined by the client to indicate when the video frame is generated and intended for display.

[0135] Figure 8AFigure 800A illustrates the construction and use of a histogram 850 according to one embodiment of this disclosure. The histogram provides a distribution of decoded timestamps of video frames, indicating the availability of displaying video frames at the client relative to a desired display time specified by the server, as previously described. The histogram is configured to determine adjustments to the offset between the VSYNC signals at the server and client. As shown, video frames are generated at the server (operation 401), the game-rendered video frames are scanned and output to an encoder (operation 402) for compression (operation 403), and transmitted to the client (operation 404). The client receives the encoded video frames (operation 405), decompresses the encoded video frames (operation 406), and renders the video frames for display (e.g., converting the decoded video frames into scan lines in operation 407).

[0136] As previously described, server-based timestamp information is delivered to the client in association with compressed / encoded video frames to construct one or more histograms for determining offsets; the timestamp information gives the expected display time specified by the server, which may not send compressed video frames for each frame period. Specifically, the histogram may include decoded timestamps, which indicate the availability of displaying video frames at the client relative to the expected display time specified by the server (e.g., based on server-based timestamp information). Since server and client timestamps can be defined by their respective asynchronous clocks, normalizing the decoded timestamps can be beneficial. For example, normalization may include subtracting the value of the first decoded timestamp from all decoded timestamps; this results in an initial decoded timestamp of zero, to which all subsequent timestamps are associated.

[0137] like Figure 8A As shown, the constructed histogram 850 can be used to determine the correct offset between the server's VSYNC signal and the client's VSYNC signal. As previously described, the histogram provides a distribution of decoding timestamps, indicating the availability of the display relative to the desired display time. A VSYNC offset 430 exists between the server and client, ensuring that a predetermined number or threshold (e.g., 99.99%) of received video frames arrive at the client and are decoded in time for display when the client's VSYNC signal next arrives appropriately; in one implementation, the remaining number (e.g., 0.01%) arrives too late to be displayed and may be discarded. In other words, the VSYNC offset 430 adapts to near-worst-case latency when receiving, decoding, and displaying rendered video frames. Figure 8AThe encoding 403, transmission 404, reception 405, and decoding 406 in the diagram show frames close to the worst (e.g., the 99.99th percentile); if the correct VSYNC offset 430 has been established, there will be no margin between the decoding of this frame and its display. One or more client buffers 820 can be implemented to accommodate video frames with lower decoding timestamps (e.g., indicating the lowest one-way latency), such that encoding 403, transmission 404, reception 405, and decoding 406 are merged early in the histogram (e.g., below the 25th percentile). In this particular example, four buffers are required: three for frames during decoding (three buffers 820), and one for the frame currently being displayed (not shown).

[0138] A series of theoretical timing diagrams 850A to 850D are provided for the client VSYNC signal 312, with timing diagram 850C (and accompanying display 407) showing the ideal client VSYNC 312C. Since there is no direct synchronization of the clock or timestamp (e.g., via a third-party timing mechanism such as a universal clock), the offset 430 is not directly set; instead, the current client VSYNC 312 can be adjusted using near-worst-case timing information from the histogram to become the ideal client VSYNC timing 312C, as previously described. Alternatively, as previously described, the server VSYNC can be adjusted to create the correct offset.

[0139] The client collects and / or receives timestamp information from the server. As previously described, timestamp information may include the time when the corresponding video frame was generated (e.g., flip time, when the scan output occurred, the appearance of the server's VSYNC signal when the scan output occurred, etc.). Additional information may be collected by the server and / or the client and used to construct or interpret the histogram, and is referred to as "histogram information," as described more fully below.

[0140] On the server side, additional histogram information may include encoding time statistics, such as: the number of scene changes; the mean and / or standard deviation of I-frame encoding time; and the mean and / or standard deviation of P-frame encoding time. Encoding time statistics may be delivered from the server to the client as periodic messages. Furthermore, histogram information may include the time the encoder takes to prepare encoder slices, which may also be delivered from the server to the client as periodic messages. Additionally, histogram information may include the actual server-side VSYNC timing and the target VSYNC timing, which may be added to the packet header. Furthermore, histogram information may include the average number of slices per I-frame relative to a P-frame.

[0141] At the server, histogram information may include round-trip time (RTT) measurements to derive the one-way network latency for sending encoded slices (e.g., compressed encoder slices). RTT measurements can be used to determine the transmission time required to send packets to the client (e.g., without requiring any further processing by the client, such as decoding and rendering). For example, RTT can be determined by sending heartbeat packets from the server to the client, where the packets include a unique identifier. The client sends a heartbeat response along with the unique identifier back to the server so that the server can calculate the RTT. The one-way network latency is approximately half the RTT. By periodically measuring RTT, network or transmission jitter (e.g., spikes in the RTT) can be analyzed and / or determined when used to construct a histogram. For example, the measured one-way network latency, obtained through RTT measurements, can be used as the transmission time for all received video frames until the next RTT measurement.

[0142] At the client end, additional histogram information may include the decoding time for each received encoded video frame. Additionally, the histogram information may include the rendering preparation time for each decoded video frame, where rendering preparation may include converting the decoded video frame slices into scan lines or screen slices.

[0143] Additionally, at the server level, supplemental histogram information may include a maximum transmission rate, which defines the total network throughput (e.g., bandwidth) that the server considers available to the client. This can be used to determine the maximum rate at which encoder slices of encoded video frames can be transmitted. The maximum rate will fluctuate depending on the stability of the network connection with the client, and the offset can be dynamically adjusted to accommodate this fluctuation. Furthermore, the maximum transmission rate can be adjusted independently of encoder parameters so that slices can be transmitted faster if the encoder is configured not to produce slices at the maximum transmission rate.

[0144] For example, maximum bandwidth or maximum transmission rate can be determined through feedback mechanisms from the client. One way to do this is to have the client return the number of packets it received within an incremental sequence ID (identifier) ​​range or frame range. For example, the client might report 145 out of 150 frames received with sequence IDs between 100 and 250. The server calculates packet loss, knows the amount of bandwidth transmitted during that packet sequence, and can determine what the client's maximum bandwidth is. The client cannot make this decision because the amount of bandwidth transmitted fluctuates due to variable bit rates, scene complexity, etc. That is, the client does not know whether the server is transmitting the maximum bandwidth that the client can handle at any given moment. For example, the maximum bandwidth might be 15 Mbps (megabits per second), but because the user is on a menu, the scene complexity might be low (static video frames have low complexity and no variation between frames). Therefore, only 2 Mbps might be transmitted. So if the client reports 0% packet loss, this does not tell the server whether the client can still handle 15 Mbps. Therefore, the true maximum bandwidth can only be determined when the server transmits the maximum bandwidth.

[0145] Figure 8B Histogram 850 is shown, illustrating the distribution of decoding timestamps. In one implementation, this is normalized such that the numerically smallest decoding timestamp is assigned a zero value (e.g., the smallest decoding timestamp is subtracted from all decoding timestamps). Specifically, the x-axis shows the time in milliseconds corresponding to the decoding timestamps, such as between 0 milliseconds (ms) and over 60 ms. The y-axis shows the number of video frames received by the client for the corresponding decoding timestamp.

[0146] For illustrative purposes only, the decoding timestamp can vary within a range of approximately 60 ms (milliseconds) and indicates a 60 ms variability in the availability of displaying a video frame at the client relative to the expected display time of the video frame specified by the server. That is, some frames may be displayed approximately 60 ms earlier or later than others. This variability in the availability of a particular frame for display can be due to variations in server and client processing, scene complexity, network path variations, packet latency variations, and other factors. By analyzing the worst-case or near-worst-case decoding timestamps, an ideal relationship between the server's VSYNC signal and the client's VSYNC signal can be determined. That is, an ideal relative offset between the timing of the client's VSYNC signal and the server's VSYNC signal can be determined to maximize the number of received and decompressed video frames available for display at the appropriate client VSYNC signal, as previously described. In this way, Figure 800B shows a distribution width of decoding timestamp 755 (e.g., approximately 57 ms), within which 99.99% of the video frames received by the client will arrive and be decoded in time for display when the client VSYNC signal next appears appropriately.

[0147] The distribution width of the decode timestamp 755 (including all decode timestamps up to near the worst-case scenario, but excluding those exceeding the worst-case scenario) can be used to determine the total amount of buffering required to decode a video frame. If the width 755 is less than one frame period, two buffers are needed, as one buffer is required when the frame is decoded, and one buffer is needed for display. If the width is greater than one frame period but less than two frame periods, three buffers are needed, and so on. In the specific example with a width of 57ms, if the frame period is 16.67ms, five frame buffers are required. The decode timestamp indicates the availability of the decoded frame relative to the expected display time; therefore, video frames with lower decode timestamps remain in the buffer for a longer period before display, while video frames with higher decode timestamps remain in the buffer for a shorter period before display.

[0148] In one implementation, the histogram is dynamically regenerated. In another implementation, the frame buffer size is dynamically set by the client over time. In yet another implementation, frames that arrive and are decoded too late to be displayed at the expected display time are skipped (i.e., not displayed).

[0149] Through the Figures 2A to 2D Detailed description of various client devices 210 and / or cloud gaming networks 290 (e.g., in game servers 260), Figure 9Flowchart 900 illustrates a method for constructing a histogram according to one embodiment of the present disclosure, the histogram providing the distribution of elapsed time between the time video frames are generated at a cloud gaming server and the time they arrive at and / or are ready to be displayed at the client, wherein the histogram is configured to determine the buffer size at the client. As previously stated, the histogram is also configured to determine the correct offset between the VSYNC signals at the server and the client.

[0150] Operations 601, 603, 605, 611, and 613 have been previously referred to Figure 6A Flowchart 600A and Figure 6B Flowchart 600B describes the adjustment of the relative timing between the server VSYNC signal and the client VSYNC signal (e.g., synchronizing frequencies and adjusting timing offsets or phases). In summary, at 601, the method includes, at the server, setting a server VSYNC signal to a frequency corresponding to the generation of video frames at the server during the frame period of the server VSYNC signal. At 603, the method includes, at the client, setting a client VSYNC signal corresponding to the frequency, the client VSYNC signal being used for rendering to a display associated with the client. At 605, the method includes, using the server VSYNC signal, sending compressed video frames based on video frames generated from the server to the client over a network.

[0151] At 611, the method includes sending timestamp information associated with a compressed video frame to the client. For example, the timestamp information may be sent together with the compressed video frame or separately, wherein the timestamp information gives an indication of the expected display timing of the corresponding video frame as determined by the video game, such as when theoretically scanning or streaming to a local display without transmission over a network. As the client receives and decodes the compressed video frame, the timestamp information is processed to create a decoded timestamp indicating the availability at the client to display the corresponding video frame relative to the expected display time specified by the server (e.g., a server timestamp). In one embodiment, the decoded timestamp may be normalized because server and client timing may be defined by correspondingly asynchronous separate clocks. A complete discussion of timestamp information is available in [reference]. Figures 6B to 6C and Figures 8A to 8B To provide, and equally applicable Figure 9 .

[0152] At 613, the method includes constructing a histogram based on decoding timestamps measured and / or calculated at the client. For example, a corresponding histogram can be created by incorporating decoding timestamp information associated with compressed video frames received and decoded at the client within a certain time period. Since the decoding timestamps indicate the availability of displaying video frames at the client relative to the expected display time specified by the server (e.g., server timestamp), the histogram also provides the distribution of the timing of decoding completion of video frames received by the client relative to the expected display time specified by the server (e.g., server timestamp information). A complete discussion of timestamp information refers to [reference needed]. Figures 6B to 6C and Figures 8A to 8B To provide, and equally applicable Figure 9 .

[0153] At 910, the method includes measuring the width of a histogram at a specific time point. For example, the distribution width of the decoded timestamps in the histogram can be measured such that a predetermined number or threshold (e.g., 99.99%) of received video frames arrive at the client in time to be displayed when the client's VSYNC signal 312 next occurs appropriately (more specifically, the remaining 0.01% of received video frames are not included when measuring the width). Specifically, the width of the histogram can be used to set the amount of frame buffer required by the client at a specific time. Therefore, at 920, the method dynamically sets the number of display buffers at the client based on the width of the histogram and the frame periods of the synchronized server and client VSYNC signals, where the histogram 750 is generated at a specific time point. As previously stated, if the width is less than one frame period, two frame buffers are required, and so on. In this way, video frames with lower decoded timestamps are kept in the buffer for a longer period, while video frames with higher decoded timestamps are kept in the buffer for a shorter period.

[0154] Through the Figures 2A to 2D Detailed description of various client devices 210 and / or cloud gaming networks 290 (e.g., in game servers 260), Figure 10 Flowchart 1000 illustrates a method for adjusting the relative timing between VSYNC signals of two or more devices according to one embodiment of the present disclosure. Specifically, flowchart 1000 can be used to compensate for drift and / or adjust offset or phase between two or more VSYNC signals of corresponding devices.

[0155] At 1010, the method includes setting multiple VSYNC signals to multiple VSYNC frequencies at multiple devices, wherein a corresponding device VSYNC signal for a corresponding device is set to a corresponding device VSYNC frequency. That is, each of the devices uses a corresponding pixel clock to set its corresponding VSYNC signal. Furthermore, the frequencies can be similar, such as being set to the same apparent frequency, although their actual frequencies may differ due to differences between various pixel clocks. These VSYNC signals can be used for the generation of video frames (e.g., at the server in a server / client architecture) and / or the display of video frames (e.g., at the client in a server / client architecture). Furthermore, these VSYNC signals can be used for the generation and display of video frames (e.g., at devices in a peer-to-peer architecture), where each device executes the video game locally, but their timing for the execution and display of video frames is coordinated.

[0156] At 1020, the method includes transmitting multiple signals among multiple devices, which are analyzed and used to adjust the relative timing between corresponding VSYNC signals of at least two devices. The relative timing can be adjusted between devices configured in a server / client architecture or a peer-to-peer architecture. For example, as previously described, the signals may include server timestamp information or server timing information, which gives an indication of when the server intends to display a corresponding video frame. In this way, the VSYNC signals of multiple devices can be synchronized (e.g., the frequency of the synchronized VSYNC signals) by determining the drift between at least two VSYNC signals. Furthermore, timing offsets and / or timing phases can be adjusted between at least two VSYNC signals.

[0157] Specifically, in one embodiment, at least two of the devices may be configured in a server / client architecture. In another embodiment, the devices are arranged in a multi-tenant configuration (e.g., one server for multiple client devices). For example, the first device may be a server device, wherein a server VSYNC signal is set to a server VSYNC frequency. The server VSYNC signal corresponds to the generation of multiple video frames during application execution at the server device during multiple frame periods for the server VSYNC frequency. Based on the server VSYNC signal, multiple compressed video frames are transmitted over a network from the server device to each of the remaining devices (e.g., client devices). For example, the server VSYNC signal provides timing for the generation and encoding of video frames on the server. The compressed video frames are based on the video frames generated by the server device. Each receiving device (e.g., a remaining device) decodes and displays the received compressed video frames. The display of the decoded video frames can be synchronized between each receiving device.

[0158] Specifically, relative timing can be adjusted between devices to compensate for drift and / or adjust timing offset or phase between the devices' VSYNC signals. A previous reference can be used. Figures 6A to 6C , Figure 7 and Figures 8A to 8B The described techniques determine drift, and timing offset, or phase adjustments. Adjusting the relative timing between the VSYNC signals of two devices can occur at either device and may include adjusting the frequency by removing or adding raster scan lines through the vertical blanking interval of the corresponding frame period of the corresponding device's VSYNC signal for the corresponding device, or by overclocking or underclocking the corresponding clock of the corresponding device.

[0159] Specifically, in one implementation, at least two of the devices can be configured in a peer-to-peer architecture. For example, each device can be an independent peer device. That is, no single device is a server device. In this way, the devices can be configured for peer-to-peer gaming. Each device generates multiple video frames by processing the same video game. Independent peer devices can operate in multiplayer mode for a specific video game using backend server support that controls the multiplayer gaming session. The backend server enables state sharing between devices by managing state data for each user in the multiplayer gaming session. State data may include game state data that defines the state of the corresponding user's (game application's) game progress at a specific point. For example, game state data may include game characters, game objects, game object attributes, game properties, game object states, graphics overlays, etc. In this way, objects and characters can be inserted into each user's game environment participating in the multiplayer gaming session, thereby customizing each user's game progress through state sharing. Furthermore, each user's game progress can be synchronized based on state sharing. That is, video frames displayed on each device can be synchronized, as reflected in the synchronized game progress. In this way, a user may not gain an advantage by receiving and displaying video frames more frequently on their device than other users' games. Alternatively, without involving a backend server, the VSYNC relationship between peers is optimized to minimize the latency between receiving control or status information from another peer and displaying video frames using that information.

[0160] Figure 11A An overlay of receiving, decoding, and rendering decompressed video frames for display at client 210 is illustrated according to one embodiment of this disclosure. Specifically, the one-way latency between a server (not shown) and client 210 in a cloud gaming application can be reduced by overlaying the reception, decoding, and display of specific video frames.

[0161] For example, a client in a cloud gaming application receives and decodes video frames. Specifically, at receive operation 405, the client receives encoded video frame 1105, where the server executes a video game to generate game-rendered video frames, then encodes the video frames at the server's encoder and delivers them to the client as encoded video frame 1105. Encoded video frame 1105 includes one or more encoded slices compressed by the encoder at the server. The client includes a decoder configured to decode one or more encoded slices in the encoded video frame at decode operation 406. In one embodiment, the decoding process begins before the client has fully received the corresponding video frame. Because the decoder performs decoding on an encoded slice-by-slice basis, the decoded video frame 1106 includes one or more encoder slices. The decoded video frame 1106 is then prepared for display, such as rendering the information in the decoded video frame 1106 as scan lines or screen slices. The client then renders video frame 1107 ready for display.

[0162] By having client 210 begin displaying the video frame at operation 407 before fully decoding it at operation 406, the one-way latency between the server and client can be reduced. Specifically, one or more decoded slices of the video frame can be prepared for rendering to the display before the video frame is fully decoded. That is, the display operation at 407 overlaps with the decoding operation at 406. Specifically, the first encoded slice (e.g., slice A) must arrive and be decoded before the client begins scanning the output to the display. Furthermore, all subsequent encoded slices must arrive and be decoded before their corresponding decompressed data is rendered and scanned for display.

[0163] Furthermore, in addition to overlapping the receiving and decoding operations at the client, one or more decoded slices can be displayed and then rendered in preparation for display, even before the encoded video frames sent by the server are fully received at the client. That is, one or more of the receiving, decoding, and display operations at the client can be overlapped for the corresponding video frames. Furthermore, in one embodiment, when multiple operations at both the server and client are overlapped, one or more decoded slices of rendered video frames in preparation for display can be displayed and then rendered at the client, even before the scan output operation at the server is fully completed, wherein the scan output delivers the game rendered video frames to the encoder at the server.

[0164] The overlap between display at operation 407 and decoding at operation 406 can be performed on a per-encoder-slice basis. In this way, an encoded slice can be displayed before one or more subsequent encoded slices are received. For this purpose, forward error correction (FEC) data must be interleaved between encoded slices of the corresponding video frame. Specifically, an encoded slice can be divided into one or more network packets. FEC packets can be used to correct one or more packets associated with a slice. Therefore, FEC packets can be interleaved between packets of multiple slices. In this way, forward error correction can be used earlier to correct lost and / or corrupted slice packets without waiting for the client to receive the entire packet set of the frame (e.g., data and FEC). This provides overlap between decoding and display operations at the client.

[0165] In one implementation, a decoding timestamp can be created for each slice, indicating the availability of displaying the slice at the client relative to the expected display time specified by the server. The decoding timestamp can be calculated by taking the completion time of slice decoding 406 at the client; subtracting the timestamp of the ideal display time for the indication frame received from the server; and adding the time spent using decompressed slice data during the display process 407 (i.e., adding 0 ms if decompressed slice data is needed immediately, adding 8.33 ms if slice data is needed midway through a 16.67 ms frame period, and so on). Normalizing the decoding timestamp in some way, such as subtracting the first decoding timestamp from all other timestamps, may be beneficial.

[0166] Decoding timestamps can be placed in a histogram, similar to... Figures 5A to 5B and Figures 8A to 8B Those shown. Decoding timestamps for the worst-case or near-worst-case scenarios (e.g., 99.999%), as determined by the histogram, can be used to adjust the relative server and client VSYNC timings, thereby reducing one-way latency. If a slice arrives and is decoded late, existing content in the display buffer will be used for display, resulting in visible corruption or "tearing," thus requiring a very high threshold (such as 99.999%) to provide for one lost frame out of a hundred thousand video frames generated at server 260 and displayed at client 210.

[0167] Through the Figures 2A to 2D Detailed description of various client devices 210 and / or cloud gaming networks 290 (e.g., in game servers 260), Figure 11B Flowchart 1100B illustrates a cloud gaming method according to one embodiment of this disclosure, wherein encoded frames are received from a server at the client and decoded and rendered for display, wherein the decoding and display of video frames can be overlaid to reduce one-way latency. As previously stated in Figures 4 to 10As described, this allows for the ability to perform one or more operations at overlapping clients by managing the one-way latency between the server and the client. For example, adjusting the relative timing between the server's VSYNC signal and the client's VSYNC signal can reduce and / or minimize the variability of the one-way latency between the server and the client.

[0168] At 1110, the method includes receiving encoded video frames at the client, wherein the server executes an application to generate rendered video frames, and then encodes the rendered video frames at an encoder at the server as encoded video frames, wherein the encoded video frames comprise one or more compressed encoded slices. For example, the server uses a server VSYNC signal to generate multiple video frames. Each video frame may be sent to the encoder for compression, wherein each video frame may be encoded into one or more encoded slices. As previously described, the encoding start of a corresponding video frame may be aligned with the server VSYNC signal. The compressed video frames are then transmitted to the client, wherein transmission does not require alignment with the server VSYNC signal and may begin immediately after the encoder slices or the complete video frame have been encoded. The client receives the compressed video frames.

[0169] At 1120, the method includes decoding one or more encoded slices at a decoder on the client side to generate one or more decoded slices. In one embodiment, decoding of the one or more encoded slices may begin before the client has fully received the encoded video frame. For example, the client receives one or more encoded slices corresponding to a video frame. Each encoded slice is then decoded and placed into a display buffer, such that the decoded video frame is placed into the display buffer.

[0170] At 1130, the method includes rendering one or more decoded slices for display at a client. Specifically, during the decoding process, decoded slices may be rendered for display, wherein rendering includes generating screen slices (e.g., scan lines) from the decoded slices of the corresponding video frames, and then streaming the screen slices to the client's display.

[0171] At 1140, in one embodiment, the method includes starting to display one or more decoded slices being rendered before the client has fully received one or more encoded slices. Specifically, the decoded slices placed in the display buffer can be immediately streamed to the client's display. Therefore, the client operations of receiving and displaying can overlap.

[0172] In another embodiment, the method includes starting to display one or more decoded slices being rendered at the display before fully decoding one or more encoded slices. Specifically, the decoded slices placed in the display buffer can be immediately streamed to the client's display. Therefore, client-side operations of decoding and display can overlap.

[0173] Figure 12 Components of an exemplary device 1200 are shown that can be used to perform various aspects of embodiments of the present disclosure. For example, Figure 12 An exemplary hardware system is illustrated according to embodiments of the present disclosure, suitable for streaming media content and / or receiving streaming media content (including tuning the VSYNC signal of a server or client to synchronize and / or adjust the offset of the VSYNC signal between the server and the client), suitable for providing dynamic buffering on the client, and suitable for decoding and displaying video frames at overlapping clients. The block diagram illustrates device 1200, which may be incorporated into or may be a personal computer, server computer, game console, mobile device, or other digital device, each of which is suitable for practicing embodiments of the present invention. Device 1200 includes a central processing unit (CPU) 1202 for running software applications and optionally an operating system. CPU 1202 may consist of one or more homogeneous or heterogeneous processing cores.

[0174] According to various implementations, CPU 1202 is one or more general-purpose microprocessors having one or more processing cores. Further implementations may use one or more CPUs with a microprocessor architecture particularly suitable for highly parallel and computationally intensive applications (such as media and interactive entertainment applications) configured for graphics processing during game execution.

[0175] Memory 1204 stores applications and data for use by CPU 1202 and GPU 1216. Storage device 1206 provides non-volatile storage and other computer-readable media for applications and data, and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROMs, DVD-ROMs, Blu-ray discs, HD-DVDs, or other optical storage devices, as well as signal transmission and storage media. User input device 1208 transmits user input from one or more users to device 1200, examples of which may include a keyboard, mouse, joystick, touchpad, touchscreen, still or video recorder / camera, and / or microphone. Network interface 1209 allows device 1200 to communicate with other computer systems via electronic communication networks and may include wired or wireless communication over local area networks and wide area networks such as the Internet. Audio processor 1212 is adapted to generate analog or digital audio output from instructions and / or data provided by CPU 1202, memory 1204, and / or storage device 1206. The components of device 1200, including CPU 1202, graphics subsystem 1214 including GPU 1216 and GPU cache 1218, memory 1204, data storage device 1206, user input device 1208, network interface 1209 and audio processor 1212, are connected via one or more data buses 1222.

[0176] The graphics subsystem 1214 is further connected to the data bus 1222 and components of device 1200. The graphics subsystem 1214 includes a graphics processing unit (GPU) 1216 and a graphics memory 1218. The graphics memory 1218 includes display memory (e.g., a frame buffer) for storing pixel data for each pixel of the output image. The graphics memory 1218 may be integrated into the same device as the GPU 1216, connected to the GPU 1216 as a separate device, and / or implemented within memory 1204. Pixel data may be provided directly from the CPU 1202 to the graphics memory 1218. Alternatively, the CPU 1202 provides the GPU 1216 with data and / or instructions defining the desired output image, and the GPU 1216 generates pixel data for one or more output images based on the data and / or instructions. The data and / or instructions defining the desired output image may be stored in memory 1204 and / or graphics memory 1218. In one implementation, the GPU 1216 includes 3D rendering capabilities for generating pixel data for an output image based on instructions and data that define the geometry, lighting, shading, texturing, motion, and / or camera parameters of a scene. The GPU 1216 may also include one or more programmable execution units capable of executing shader programs.

[0177] The graphics subsystem 1214 periodically outputs pixel data of an image from the graphics memory 1218 for display on the display device 1210 or for projection by a projection system (not shown). The display device 1210 can be any device capable of displaying visual information in response to signals from the device 1200, including CRT, LCD, plasma, and OLED displays. The device 1200 can provide, for example, analog or digital signals to the display device 1210.

[0178] Other implementations for optimizing the graphics subsystem 1214 may include multi-tenant GPU operations that share GPU instances among multiple applications, and distributed GPUs that support a single game. The graphics subsystem 1214 may be configured as one or more processing devices.

[0179] For example, in one implementation, the graphics subsystem 1214 may be configured to perform multi-tenant GPU functionality, whereby one graphics subsystem can implement graphics and / or rendering pipelines for multiple games. That is, the graphics subsystem 1214 is shared among multiple games being executed.

[0180] In other implementations, the graphics subsystem 1214 includes multiple GPU devices that are combined to perform graphics processing for a single application running on a corresponding CPU. For example, the multiple GPUs may perform frame rendering in an alternating manner, where in consecutive frame cycles, GPU 1 renders the first frame, and GPU 2 renders the second frame, and so on, until the last GPU is reached, then the initial GPU renders the next video frame (e.g., if there are only two GPUs, GPU 1 renders the third frame). That is, the GPUs rotate while rendering frames. Rendering operations may overlap, where GPU 2 may begin rendering the second frame before GPU 1 has finished rendering the first frame. In another implementation, different shader operations may be assigned to the multiple GPU devices in the rendering and / or graphics pipeline. The main GPU is performing the main rendering and compositing. For example, in a group comprising three GPUs, the main GPU 1 can perform primary rendering (e.g., first shader operations) and compositing from outputs from GPU 2 and GPU 3, where GPU 2 can perform second shader operations (e.g., fluid effects such as rivers), and GPU 3 can perform third shader operations (e.g., particle smoke), with the main GPU 1 compositing the results from each of GPU 1, GPU 2, and GPU 3. In this way, different GPUs can be assigned to perform different shader operations (e.g., waving flags, wind, smoke generation, fire, etc.) to render video frames. In yet another embodiment, each of the three GPUs can be assigned to different objects and / or portions of the scene corresponding to a video frame. In the above embodiments and implementations, these operations can be performed in the same frame cycle (simultaneously in parallel) or in different frame cycles (sequentially in parallel).

[0181] Therefore, this disclosure describes methods and systems for configuring streaming media content and / or receiving streaming media content (including tuning the VSYNC signal of a server or client to synchronize and / or adjust the offset of the VSYNC signal between the server and the client), providing dynamic buffering on the client, and decoding and displaying video frames at the overlapping client.

[0182] It should be understood that the various implementations defined herein can be combined or assembled into specific implementations using the various features disclosed herein. Therefore, the examples provided are merely some possible examples and are not limited to the various implementations that could be defined by combining various elements. In some examples, some implementations may include fewer elements without departing from the spirit of the disclosed or equivalent implementations.

[0183] The embodiments of this disclosure can be practiced with various computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. The embodiments of this disclosure can also be practiced in distributed computing environments where tasks are performed via remote processing devices based on wired or wireless network links.

[0184] In light of the above embodiments, it should be understood that embodiments of this disclosure can employ various computer-implemented operations involving data stored in a computer system. These operations are those that require the physical manipulation of physical quantities. Any operation described herein that forms part of embodiments of this disclosure is a useful machine operation. Embodiments of this disclosure also relate to devices or apparatuses for performing these operations. The apparatus may be specifically constructed for the desired purpose, or the apparatus may be a general-purpose computer selectively activated or configured by a computer program stored in a computer. Specifically, various general-purpose machines may be used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct more specialized apparatuses to perform the desired operations.

[0185] This disclosure can also be embodied as computer-readable code on a computer-readable medium. A computer-readable medium is any data storage device capable of storing data that can subsequently be read by a computer system. Examples of computer-readable media include hard disk drives, network attached storage devices (NAS), read-only memory, random access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical and non-optical data storage devices. The computer-readable medium may include computer-readable tangible media distributed across network-coupled computer systems, enabling the distributed storage and execution of computer-readable code.

[0186] Although the method operations are described in a specific order, it should be understood that other housekeeping operations may be performed between operations, or operations may be adjusted so that they occur at slightly different times, or they may be distributed in a system that allows processing operations to occur at various intervals associated with the processing, as long as the processing of the covering operations is performed in the desired manner.

[0187] While the foregoing disclosure has been described in considerable detail for the purposes of clarity, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. Therefore, embodiments of the invention are to be considered illustrative rather than restrictive, and embodiments of this disclosure are not limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.

Claims

1. A method for network transmission, comprising: at a server, setting a server VSYNC signal to a server VSYNC frequency, the server VSYNC signal corresponding to generating a plurality of video frames at the server during a plurality of frame periods for the server VSYNC frequency; at a client, setting a client VSYNC signal to a client VSYNC frequency; sending a plurality of compressed video frames based on the plurality of video frames from the server to the client over a network using the server VSYNC signal; decoding and displaying the plurality of compressed video frames at the client; and as the client receives the plurality of compressed video frames, analyzing a timing of one or more client operations to adjust a relative timing between the server VSYNC signal and the client VSYNC signal.

2. The method of claim 1, further comprising: sending timestamp information associated with the plurality of video frames generated from the server to the client.

3. The method of claim 2, wherein the timestamp information is sent with the plurality of compressed video frames, or the timestamp information is sent separately from the plurality of compressed video frames.

4. The method of claim 2, wherein the timestamp information includes a corresponding timestamp of a generated corresponding video frame, wherein the corresponding timestamp occurs at a corresponding flip time at the server, or at a corresponding occurrence of the server VSYNC signal used to scan the corresponding video frame to an encoder.

5. The method of claim 2, further comprising: synchronizing the server VSYNC frequency and the client VSYNC frequency based on the timestamp information sent from the server to the client.

6. The method of claim 2, further comprising: adjusting a relative phase or offset between the server VSYNC signal and the client VSYNC signal based on the timestamp information sent from the server to the client.

7. The method of claim 1, further comprising: adjusting the relative timing by adjusting at least one of the server VSYNC signal and the client VSYNC signal.

8. The method of claim 1, further comprising: continuously analyzing the timing of client operations; and based on the analysis, dynamically adjusting the relative timing by adjusting at least one of the server VSYNC signal and the client VSYNC signal.

9. The method of claim 7, further comprising, at the server or the client, determining a drift between the server VSYNC frequency and the client VSYNC frequency; and at the server or the client, adjusting a corresponding server VSYNC frequency or the client VSYNC frequency over one or more frame periods to compensate for the drift. ​ ​ ​ 10. The method of claim 7, wherein said adjusting at least one of the server VSYNC signal or the client VSYNC signal comprises: deleting or adding raster scan lines to a vertical blanking interval of a frame period of the server VSYNC signal or the client VSYNC signal.

11. The method of claim 7, wherein said adjusting at least one of the server VSYNC signal or the client VSYNC signal comprises: overclocking or underclocking a corresponding clock of the server or the client.

12. The method of claim 1, wherein the server comprises an encoder for generating the plurality of compressed video frames, the plurality of compressed video frames being generated and delivered to the client during a session in response to the server processing an application in a streaming mode.

13. The method of claim 12, wherein the client is configured to receive and decompress the plurality of compressed video frames for rendering to a display associated with the client.

14. The method of claim 1, wherein the relative timing between the server VSYNC signal and the client VSYNC signal is adjusted using timing of the decoding of the plurality of compressed video frames by the client.

15. The method of claim 14, further comprising, analyzing the timing of the decoding of a plurality of encoded slices within the plurality of compressed video frames by the client to adjust the relative timing between the server VSYNC signal and the client VSYNC signal.

16. The method of claim 14, wherein a threshold decode time of the plurality of compressed video frames is analyzed to adjust the relative timing between the server VSYNC signal and the client VSYNC signal.

17. The method of claim 16, wherein the threshold decode time is determined by measuring a distribution of decode times of the plurality of compressed video frames received at the client.

18. The method of claim 1, further comprising: providing a corresponding timestamp generated at the server corresponding to when a corresponding scan of each video frame of the plurality of video frames is output to an encoder; delivering the corresponding timestamps of the plurality of video frames to the client; constructing a histogram giving a timing of a corresponding decode time of each video frame of the plurality of video frames at the client relative to the corresponding timestamps provided at the server; and offsetting the client VSYNC signal from the server VSYNC signal such that a threshold percentage of the plurality of compressed video frames arrive at the client and are decoded in time to be displayed at a corresponding occurrence of the client VSYNC signal indicated by the corresponding timestamps.

19. A method for network transmission, comprising: generating a plurality of video frames at a server during a plurality of frame periods, wherein the frame periods are approximately equal in size; setting a client VSYNC signal to a client VSYNC frequency at a client; sending a plurality of compressed video frames based on the plurality of video frames from the server to the client; decoding and displaying the plurality of compressed video frames at the client; and analyzing timing of one or more client operations to adjust relative timing of the client VSYNC signal and generation of the plurality of compressed video frames at the server as the client receives the plurality of compressed video frames.

20. The method of claim 19, further comprising: periodically sending timestamp information related to the plurality of frame periods from the server to the client.

21. The method of claim 19, further comprising: determining a drift between the server VSYNC frequency and the client VSYNC frequency at the server or the client; and adjusting a corresponding server VSYNC frequency or the client VSYNC frequency to compensate for the drift at the server or the client for one or more frame periods.

22. The method of claim 21, wherein the adjusting the corresponding server VSYNC frequency or the client VSYNC frequency comprises: removing or adding raster scan lines for a vertical blanking interval of a frame period of the server VSYNC signal or the client VSYNC signal.

23. The method of claim 21, the adjusting the corresponding server VSYNC frequency or the client VSYNC frequency comprises: overclocking or underclocking a corresponding clock of the server or the client.

24. A non-transitory computer-readable medium storing a computer program for performing a method, the computer-readable medium comprising: program instructions for setting a server VSYNC signal to a server VSYNC frequency at a server, the server VSYNC signal corresponding to generation of a plurality of video frames at the server during a plurality of frame periods for the server VSYNC frequency; program instructions for setting a client VSYNC signal to a client VSYNC frequency at a client; program instructions for sending a plurality of compressed video frames based on the plurality of video frames from the server to the client using the server VSYNC signal over a network; program instructions for decoding and displaying the plurality of compressed video frames at the client; and program instructions for analyzing timing of one or more client operations to adjust relative timing between the server VSYNC signal and the client VSYNC signal as the client receives the plurality of compressed video frames.

25. The non-transitory computer-readable medium of claim 24, further comprising: program instructions for sending timestamp information associated with the plurality of video frames generated from the server to the client, wherein the timestamp information comprises a corresponding timestamp of a corresponding video frame generated, wherein the corresponding timestamp occurs at a corresponding flip time at the server, or at a corresponding occurrence of the server VSYNC signal used to scan the corresponding video frame to an encoder.

26. The non-transitory computer-readable medium of claim 24, further comprising: program instructions to synchronize the server VSYNC frequency and the client VSYNC frequency based on timestamp information sent from the server to the client associated with the plurality of video frames, wherein the timestamp information comprises a corresponding timestamp of a corresponding video frame generated.

27. The non-transitory computer-readable medium of claim 24, further comprising: program instructions to adjust a relative phase or offset between the server VSYNC signal and the client VSYNC signal based on timestamp information sent from the server to the client associated with the plurality of video frames, wherein the timestamp information comprises a corresponding timestamp of a corresponding video frame generated.

28. The non-transitory computer-readable medium of claim 24, further comprising: program instructions to adjust the relative timing by adjusting at least one of the server VSYNC signal and the client VSYNC signal.

29. The non-transitory computer-readable medium of claim 28, further comprising: program instructions to determine a drift between the server VSYNC frequency and the client VSYNC frequency at the server or the client; and program instructions to adjust a corresponding server VSYNC frequency or the client VSYNC frequency at the server or the client for one or more frame periods to compensate for the drift.

30. The non-transitory computer-readable medium of claim 28, wherein the program instructions to adjust at least one of the server VSYNC signal or the client VSYNC signal comprises: program instructions to delete or add raster scan lines for a vertical blanking interval of a frame period of a corresponding server VSYNC signal or client VSYNC signal.

31. The non-transitory computer-readable medium of claim 28, wherein the program instructions to adjust at least one of the server VSYNC signal or the client VSYNC signal comprises: program instructions to overclock or underclock a corresponding clock of the server or the client.

32. A method for network transmission, comprising: at a server, setting a server VSYNC signal to a server VSYNC frequency defining a plurality of frame periods, the server VSYNC signal corresponding to generation of a plurality of video frames at the server during the plurality of frame periods; at a client, setting a client VSYNC signal to a client VSYNC frequency; sending a plurality of compressed video frames based on the plurality of video frames from the server to the client over a network using the server VSYNC signal; decoding and displaying the plurality of compressed video frames at the client; and timing of one or more client operations as the client receives the plurality of compressed video frames to set an amount of frame buffering used by the client.

33. The method of claim 32, wherein the amount of frame buffering is dynamically set by the client over time.

34. The method of claim 32, wherein the amount of frame buffering defines a number of frames to be buffered at the client.

35. The method of claim 32, further comprising: adjusting a relative timing between the server VSYNC signal and the client VSYNC signal based on the timing of the one or more client operations.

36. The method of claim 32, further comprising: sending timestamp information associated with the plurality of video frames; constructing a histogram that gives a timing of a corresponding decode time at the client of each video frame generated and delivered to the client relative to the corresponding timestamp information; measuring a width of the histogram at a certain time; and dynamically setting a number of display buffers at the client based on the width of the histogram and the frame period.

37. The method of claim 36, wherein the corresponding timestamp information of a corresponding video frame generated occurs at a corresponding flip time at the server or at a corresponding occurrence of the server VSYNC signal used to scan the corresponding video frame to an encoder.

38. The method of claim 32, wherein the amount of frame buffering used by the client is an input to latency tuning, wherein the latency tuning includes determining to skip displaying a frame at the client if the frame arrives later than a predefined time.

39. The method of claim 32, wherein the plurality of compressed video frames that are decoded are displayed using the client VSYNC signal.

40. A non-transitory computer-readable medium storing a computer program for performing a method, the computer-readable medium comprising: program instructions for setting, at a server, a server VSYNC signal to a server VSYNC frequency that defines a plurality of frame periods, the server VSYNC signal corresponding to a plurality of video frames generated at the server during the plurality of frame periods; program instructions for setting, at a client, a client VSYNC signal to a client VSYNC frequency; program instructions for sending, using the server VSYNC signal, a plurality of compressed video frames based on the plurality of video frames from the server to the client over a network; program instructions for decoding and displaying the plurality of compressed video frames at the client; and program instructions for analyzing a timing of one or more client operations as the client receives the plurality of compressed video frames to set an amount of frame buffering used by the client.

41. The non-transitory computer-readable medium of claim 40, wherein in the method the amount of frame buffering is dynamically set by the client over time.

42. The non-transitory computer readable medium of claim 40, wherein in the method, the frame buffering amount defines a number of frames to be buffered at the client.

43. The non-transitory computer readable medium of claim 40, further comprising: program instructions to adjust a relative timing between the server VSYNC signal and the client VSYNC signal based on the timing of the one or more client operations.

44. The non-transitory computer readable medium of claim 40, further comprising: program instructions to send timestamp information associated with the plurality of video frames; program instructions to construct a histogram giving a timing of a corresponding decoding time at the client of each video frame generated and delivered to the client relative to the corresponding timestamp information; program instructions to measure a width of the histogram at a certain time; and program instructions to dynamically set a number of display buffers at the client based on the width of the histogram and the frame period.

45. The non-transitory computer readable medium of claim 44, wherein in the method, the corresponding timestamp information of a corresponding video frame generated occurs at a corresponding flip time at the server or at a corresponding occurrence of the server VSYNC signal used to scan the corresponding video frame to an encoder.

46. The non-transitory computer readable medium of claim 40, wherein in the method, the frame buffering amount used by the client is an input to latency tuning, wherein the latency tuning includes determining to skip displaying a frame at the client if the frame arrives later than a predefined time.

47. The non-transitory computer readable medium of claim 40, wherein in the method, the client VSYNC signal is used to display the plurality of compressed video frames decoded.

48. A computer system comprising: a processor; a memory coupled to the processor and having instructions stored therein that, if executed by the computer system, cause the computer system to perform a method comprising: at a server, setting a server VSYNC signal to a server VSYNC frequency defining a plurality of frame periods, the server VSYNC signal corresponding to a plurality of video frames generated at the server during the plurality of frame periods; at a client, setting a client VSYNC signal to a client VSYNC frequency; sending a plurality of compressed video frames based on the plurality of video frames from the server to the client over a network using the server VSYNC signal; decoding and displaying the plurality of compressed video frames at the client; and as the client receives the plurality of compressed video frames, analyzing a timing of one or more client operations to set a frame buffering amount used by the client.

49. The computer system of claim 48, wherein in the method, the frame buffer amount is dynamically set by the client over time.

50. The computer system of claim 48, wherein in the method, the frame buffer amount defines a number of frames to be buffered at the client.

51. The computer system of claim 48, the method further comprising: sending timestamp information associated with the plurality of video frames; constructing a histogram that gives a timing of a corresponding decode time at the client of each video frame generated and delivered to the client relative to the corresponding timestamp information; measuring a width of the histogram at a certain time; and dynamically setting a number of display buffers at the client based on the width of the histogram and the frame period.

52. A method for network transmission, comprising: setting a plurality of VSYNC signals to a plurality of VSYNC frequencies at a plurality of devices, wherein a corresponding device VSYNC signal of a corresponding device is set to a corresponding device VSYNC frequency; sending a plurality of signals between the plurality of devices; and analyzing the plurality of signals to adjust a relative timing between the corresponding device VSYNC signals of at least two devices.

53. The method of claim 52, wherein a first device of the plurality of devices is a server device, a server VSYNC signal is set to a server VSYNC frequency for the server device, the server VSYNC signal corresponding to a plurality of video frames generated during execution of an application at the server device during a plurality of frame periods for the server VSYNC frequency; sending a plurality of compressed video frames based on the plurality of video frames from the server device to each remaining device of the plurality of devices over a network using the server VSYNC signal; and decoding and displaying the plurality of compressed video frames at each remaining device of the plurality of devices; wherein the displaying of the plurality of compressed video frames at each remaining device of the plurality of devices is synchronized.

54. The method of claim 53, wherein each remaining device of the plurality of devices is a client device configured with the server device in a server and client architecture.

55. The method of claim 54, wherein the server device is configured with the remaining devices of the plurality of devices in a multi-tenant configuration.

56. The method of claim 52, generating a corresponding plurality of video frames when an application is processed at a corresponding device of the plurality of devices; and displaying the corresponding plurality of video frames at the corresponding device of each device of the plurality of devices, wherein the displaying of the corresponding plurality of video frames at the plurality of devices is synchronized.

57. The method of claim 52, wherein each device of the plurality of devices is an independent peer device, wherein the plurality of devices are arranged in a peer-to-peer configuration for peer-to-peer gaming.

58. The method of claim 52, wherein at least one signal of the plurality of signals comprises timestamp information.

59. The method of claim 52, wherein the plurality of signals are analyzed to synchronize device VSYNC signals of the plurality of devices.

60. The method of claim 59, wherein the relative timing between corresponding device VSYNC signals of at least two devices is adjusted by: deleting or adding raster scan lines to a vertical blanking interval of a corresponding frame period of a corresponding device VSYNC signal of a corresponding device.

61. The method of claim 59, wherein the relative timing between corresponding device VSYNC signals of at least two devices is adjusted by: overclocking or underclocking a corresponding clock of a corresponding device.

62. The method of claim 52, further comprising: determining a drift between a first device VSYNC frequency of a first device of the plurality of devices and a second device VSYNC frequency of a second device of the plurality of devices; and adjusting, at the first device or the second device, the corresponding device VSYNC frequency for one or more frame periods to compensate for the drift.

63. A non-transitory computer-readable medium storing a computer program for performing a method, the computer-readable medium comprising: program instructions for setting a plurality of VSYNC signals to a plurality of VSYNC frequencies at a plurality of devices, wherein a corresponding device VSYNC signal of a corresponding device is set to a corresponding device VSYNC frequency; program instructions for sending a plurality of signals between the plurality of devices; and program instructions for analyzing the plurality of signals to adjust a relative timing between corresponding device VSYNC signals of at least two devices.

64. The non-transitory computer-readable medium of claim 63, further comprising: program instructions for setting a server VSYNC signal to a server VSYNC frequency for a server, the server VSYNC signal corresponding to a plurality of video frames generated at the server for application during a plurality of frame periods for the server VSYNC frequency; program instructions for sending a plurality of compressed video frames based on the plurality of video frames from the server to each device of the plurality of devices using the server VSYNC signal over a network; and program instructions for decoding and displaying the plurality of compressed video frames at each device of the plurality of devices; wherein the displaying of the plurality of compressed video frames at the plurality of devices is synchronized.

65. The non-transitory computer-readable medium of claim 64, wherein in the method, each device of the plurality of devices is a client device, wherein in the method, each device of the plurality of devices is configured in a server and client architecture.

66. The non-transitory computer-readable medium of claim 63, further comprising: program instructions for generating a corresponding plurality of video frames for application at a corresponding device of the plurality of devices; and program instructions for sending a corresponding plurality of compressed video frames based on the corresponding plurality of video frames from the corresponding device to each device of the plurality of devices using a corresponding device VSYNC signal of the corresponding device over a network; and program instructions for decoding and displaying the corresponding plurality of compressed video frames at each device of the plurality of devices; wherein the displaying of the corresponding plurality of compressed video frames at the plurality of devices is synchronized. program instructions for displaying the corresponding plurality of video frames at the corresponding device of each of the plurality of devices, wherein the displaying of the corresponding plurality of video frames at the plurality of devices is synchronized.

67. The non-transitory computer-readable medium of claim 66, wherein in the method, each of the plurality of devices is an independent peer device, wherein in the method, the plurality of devices are arranged in a peer-to-peer configuration for peer-to-peer gaming.

68. The non-transitory computer-readable medium of claim 63, wherein in the method, the plurality of signals are analyzed to synchronize device VSYNC signals of the plurality of devices.

69. A computer system comprising: a processor; a memory coupled to the processor and having instructions stored therein that, if executed by the computer system, cause the computer system to perform a method comprising: setting, at a plurality of devices, a plurality of VSYNC signals to a plurality of VSYNC frequencies, wherein a corresponding device VSYNC signal of a corresponding device is set to a corresponding device VSYNC frequency; sending a plurality of signals between the plurality of devices; and analyzing the plurality of signals to adjust a relative timing between the corresponding device VSYNC signals of at least two devices.

70. The computer system of claim 69, the method further comprising: setting, for a server, a server VSYNC signal to a server VSYNC frequency, the server VSYNC signal corresponding to a plurality of video frames generated at the server during a plurality of frame periods for the server VSYNC frequency; sending, using the server VSYNC signal, a plurality of compressed video frames based on the plurality of video frames from the server to each of the plurality of devices over a network; and decoding and displaying the plurality of compressed video frames at each of the plurality of devices; wherein the displaying of the plurality of compressed video frames at the plurality of devices is synchronized.

71. The computer system of claim 70, wherein in the method, each of the plurality of devices is a client device, wherein in the method, each of the plurality of devices is configured in a server and client architecture.

72. The computer system of claim 69, the method further comprising: generating, at a corresponding device of the plurality of devices, a corresponding plurality of video frames of an application; and displaying the corresponding plurality of video frames at the corresponding device of each of the plurality of devices, wherein the displaying of the corresponding plurality of video frames at the plurality of devices is synchronized.

73. The computer system of claim 72, wherein in the method, each of the plurality of devices is an independent peer device, wherein in the method, the plurality of devices are arranged in a peer-to-peer configuration for peer-to-peer gaming.

74. The computer system of claim 69, wherein in the method, the plurality of signals are analyzed to synchronize device VSYNC signals of the plurality of devices. ​ 75. A method of cloud gaming, the method comprising: setting, at a client, a client VSYNC signal to a client VSYNC frequency; receiving, at the client, encoded video frames using a server VSYNC signal having a server VSYNC frequency, wherein a server executes an application to generate rendered video frames, which are then encoded into the encoded video frames at an encoder of the server, wherein the encoded video frames comprise one or more encoded slices that are compressed; decoding, at a decoder of the client, the one or more encoded slices to generate one or more decoded slices; rendering the one or more decoded slices for display at the client; beginning to display the rendered one or more decoded slices before the one or more encoded slices are fully received at the client; and analyzing timing of one or more client operations to adjust a relative timing between the server VSYNC signal and the client VSYNC signal once the plurality of compressed video frames are received.

76. The method of claim 75, further comprising: beginning to decode the one or more encoded slices before the encoded video frames are fully received at the client.

77. The method of claim 75, wherein the beginning to display the rendered one or more decoded slices comprises: beginning to display the rendered one or more decoded slices at a display before the one or more encoded slices are fully decoded.

78. The method of claim 75, wherein the relative timing between the server VSYNC signal and the client VSYNC signal is adjusted by: sending timestamp information associated with the plurality of rendered video frames from the server to the client; wherein a corresponding timestamp of a corresponding rendered video frame occurs at a corresponding flip time at the server, or at a corresponding occurrence of the server VSYNC signal used to scan the corresponding video frame to an encoder, synchronizing the server VSYNC frequency and the client VSYNC frequency based on the timestamp information sent from the server to the client; and adjusting a relative phase or offset between the server VSYNC signal and the client VSYNC signal based on the timestamp information sent from the server to the client.

79. The method of claim 78, wherein the timestamp information is sent with the plurality of compressed video frames or the timestamp information is sent separately from the plurality of compressed video frames.

80. The method of claim 75, wherein the relative timing between the server VSYNC signal and the client VSYNC signal is adjusted by: adding or removing raster scan lines from a vertical blanking interval of a frame period of the server VSYNC signal or the client VSYNC signal. ​ 81. A non-transitory computer readable medium storing a computer program for performing a method of cloud gaming, the computer readable medium comprising: program instructions for setting a client VSYNC signal to a client VSYNC frequency at a client; program instructions for receiving encoded video frames at the client using a server VSYNC signal having a server VSYNC frequency, wherein a server executes an application to generate rendered video frames, which are then encoded into the encoded video frames at an encoder of the server, wherein the encoded video frames comprise one or more encoded slices that are compressed; program instructions for decoding the one or more encoded slices at a decoder of the client to generate one or more decoded slices; program instructions for rendering the one or more decoded slices for display at the client; program instructions for beginning display of the rendered one or more decoded slices at the client prior to full reception of the one or more encoded slices at the client; and program instructions for analyzing timing of one or more client operations once the plurality of compressed video frames are received to adjust a relative timing between the server VSYNC signal and the client VSYNC signal.

82. The non-transitory computer readable medium of claim 81, further comprising: program instructions for beginning decoding of the one or more encoded slices at the client prior to full reception of the encoded video frames at the client.

83. The non-transitory computer readable medium of claim 81, wherein the program instructions for beginning display of the rendered one or more decoded slices comprise: program instructions for beginning display of the rendered one or more decoded slices at a display prior to full decoding of the one or more encoded slices.

84. The non-transitory computer readable medium of claim 81, wherein in the program instructions, the relative timing between the server VSYNC signal and the client VSYNC signal is adjusted by: program instructions for sending timestamp information associated with the plurality of rendered video frames generated from the server to the client; wherein a corresponding timestamp of a corresponding rendered video frame occurs at a corresponding flip time at the server, or at a corresponding occurrence of the server VSYNC signal used to scan the corresponding video frame to an encoder, program instructions for synchronizing the server VSYNC frequency and the client VSYNC frequency based on the timestamp information sent from the server to the client; and program instructions for adjusting a relative phase or offset between the server VSYNC signal and the client VSYNC signal based on the timestamp information sent from the server to the client. ​ 85. The non-transitory computer readable medium of claim 84, wherein in the method, the timestamp information is sent with the plurality of compressed video frames or the timestamp information is sent separately from the plurality of compressed video frames.

86. The non-transitory computer readable medium of claim 81, wherein in the method, the relative timing between the server VSYNC signal and the client VSYNC signal is adjusted by: deleting or adding raster scan lines to a vertical blanking interval of a frame period for the server VSYNC signal or the client VSYNC signal.

87. A computer system comprising: a processor; a memory coupled to the processor and having instructions stored therein that, if executed by the computer system, cause the computer system to perform a method of cloud gaming, the method comprising: at a client, setting a client VSYNC signal to a client VSYNC frequency; at the client, receiving encoded video frames using a server VSYNC signal having a server VSYRC frequency, wherein a server executes an application to generate rendered video frames that are then encoded into the encoded video frames at an encoder of the server, wherein the encoded video frames comprise one or more encoded slices that are compressed; decoding the one or more encoded slices at a decoder of the client to generate one or more decoded slices; rendering the one or more decoded slices for display at the client; beginning to display the rendered one or more decoded slices before the one or more encoded slices are completely received at the client; and once the plurality of compressed video frames are received, analyzing timing of one or more client operations to adjust the relative timing between the server VSYNC signal and the client VSYNC signal.

88. The computer system of claim 87, the method further comprising: beginning to decode the one or more encoded slices before the encoded video frames are completely received at the client.

89. The computer system of claim 87, wherein in the method, the beginning to display the rendered one or more decoded slices comprises: beginning to display the rendered one or more decoded slices at a display before the one or more encoded slices are completely decoded.

90. The computer system of claim 87, wherein in the method, the relative timing between the server VSYNC signal and the client VSYNC signal is adjusted by: sending timestamp information associated with the plurality of rendered video frames generated from the server to the client; wherein a corresponding timestamp of a corresponding rendered video frame occurs at a corresponding flip time at the server or at a corresponding occurrence of the server VSYNC signal used to scan the corresponding video frame to an encoder, ​ synchronizing the server VSYNC frequency and the client VSYNC frequency based on the timestamp information sent from the server to the client; and adjusting the relative phase or offset between the server VSYNC signal and the client VSYNC signal based on the timestamp information sent from the server to the client.

91. The computer system of claim 87, wherein in the method, the relative timing between the server VSYNC signal and the client VSYNC signal is adjusted by: deleting or adding raster scan lines to the vertical blanking interval of a frame period of the server VSYNC signal or the client VSYNC signal.

Images