Synchronization and offset of vsync between cloud gaming server and client

Abstract

To reduce round-trip and one way latency between a server and a client by improving operations associated with decoding and displaying frames.SOLUTION: A method is disclosed that includes setting, at a server, a server VSYNC signal to a server VSYNC frequency. The server VSYNC signal corresponds to generation of video frames during multiple frame periods of the server VSYNC frequency. The method sets, at the client, a client VSYNC signal to a client VSYNC frequency. The method sends compressed video frames from the server to the client via the network using the server VSYNC signal, wherein the compressed video frames are based on the generated video frames. The method decodes and displays, at the client, the compressed video frames and analyzes a timing of one or more client operations as the client receives the plurality of compressed video frames.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to a streaming system configured to stream content over a network, and more particularly to synchronization of vertical synchronization (VSYNC) signals between a cloud gaming server and a client to reduce latency between the cloud gaming server and the client. [Background technology]

[0002] In recent years, there has been a continuous push for online services that enable streaming-style online or cloud gaming between cloud gaming servers and clients connected over a network. Streaming has become increasingly popular due to the availability of on-demand game titles, networking capabilities between players for multiplayer games, asset sharing between players, instant experience sharing between players and / or spectators, allowing friends to watch a friend play a video game, or allowing friends to join a friend's ongoing gameplay. Unfortunately, demand is also pushing the limits of network connection capabilities and the processing running on the server and client that is responsive enough to render high-quality images delivered to the client. For example, the results of all game activity performed on the server should be compressed and sent back to the client with low millisecond latency to ensure the best user experience. Round-trip latency can be defined as the overall time from a user's controller input to the display of a video frame on the client. This may include processing and transmitting control information from the controller to the client, processing and transmitting control information from the client to the server, using that input on the server to generate a video frame responsive to the input, processing and forwarding the video frame to an encoding unit (e.g., scanning out), encoding the video frame, sending the encoded video frame back to the client, receiving and decoding the video frame, and any processing or staging of the video frame before displaying it. One-way latency can be defined as the portion of round-trip latency that consists of the time from the start of the transfer of a video frame to an encoding unit (e.g., scanout) at the server to the start of displaying the video frame at the client. Part of the round-trip and one-way latency is related to the time it takes for the data stream to be transmitted over the communication network from the client to the server and from the server to the client. Another portion is related to processing at the client and server. Improvements in these operations, such as advanced strategies related to decoding and displaying frames, can significantly reduce the round-trip and one-way latency between the server and the client, providing a better experience for users of cloud gaming services.

[0003] It is against this background that the embodiments of the present disclosure have been made. Summary of the Invention

[0004] Embodiments of the present disclosure relate to a streaming system configured to stream content (e.g., games) over a network, and more specifically, to synchronizing VSYNC signals between a cloud gaming server and a client, with the aim of reducing latency between the two. In the context of this patent application, "synchronization" should be interpreted as meaning adjusting the signals so that their frequencies match, but their phases may differ. "Offset" should be interpreted as the time delay between signals, for example, the time between when one signal reaches its maximum and when the other signal reaches its maximum.

[0005] An embodiment of the present disclosure discloses a method. The method includes, at a server, setting a server VSYNC signal to a server VSYNC frequency, the server VSYNC signal corresponding to generating multiple video frames at the server during multiple frame periods of the server VSYNC frequency. The method includes, at a client, setting a client VSYNC signal to the client VSYNC frequency. The method includes transmitting multiple compressed video frames based on the multiple video frames from the server to the client over a network using the server VSYNC signal. The method includes, at the client, decoding and displaying the multiple compressed video frames. The method includes, when the client receives the multiple compressed video frames, analyzing timing of one or more client operations to adjust the relative timing between the server VSYNC signal and the client VSYNC signal.

[0006] An embodiment of the present disclosure discloses a method. The method includes generating a plurality of video frames at a server over a plurality of frame periods, the frame periods being approximately equal in size. The method includes setting a client VSYNC signal at a client VSYNC frequency. The method includes transmitting a plurality of compressed video frames based on the plurality of video frames from the server to the client. The method includes decoding and displaying the plurality of compressed video frames at the client. The method includes, when the client receives the plurality of compressed video frames, analyzing timing of one or more client operations to adjust the relative timing of the client VSYNC signal and generating the plurality of compressed video frames at the server.

[0007] Another embodiment of the present disclosure discloses a non-transitory computer-readable medium storing a computer program for performing a method. The computer-readable medium includes program instructions for: at a server, setting a server VSYNC signal to a server VSYNC frequency, where the server VSYNC signal corresponds to generating multiple video frames at the server during multiple frame periods of the server VSYNC frequency. The computer-readable medium includes program instructions for a client, setting a client VSYNC signal to the client VSYNC frequency. The computer-readable medium includes program instructions for transmitting multiple compressed video frames based on the multiple video frames from the server to the client over a network using the server VSYNC signal. The computer-readable medium includes program instructions for decoding and displaying the multiple compressed video frames at the client. The computer-readable medium includes program instructions for analyzing timing of one or more client operations and adjusting the relative timing between the server VSYNC signal and the client VSYNC signal when the client receives the multiple compressed video frames.

[0008] In another embodiment of the present disclosure, a computer system is disclosed that includes a processor and a memory coupled to the processor, the memory storing instructions that, when executed by the computer system, cause the computer system to perform a method. The method includes, at a server, setting a server VSYNC signal to a server VSYNC frequency, the server VSYNC signal corresponding to generation of multiple video frames at the server during multiple frame periods of the server VSYNC frequency. The method includes, at a client, setting a client VSYNC signal to the client VSYNC frequency. The method includes transmitting multiple compressed video frames based on the multiple video frames from the server to the client over a network using the server VSYNC signal. The method includes, at the client, decoding and displaying the multiple compressed video frames. The method includes, when the client receives the multiple compressed video frames, analyzing timing of one or more client operations to adjust the relative timing between the server VSYNC signal and the client VSYNC signal.

[0009] Another embodiment of the present disclosure discloses another method. The method includes, at a server, setting a server VSYNC signal to a server VSYNC frequency defining a plurality of frame periods, the server VSYNC signal corresponding to generation of a plurality of video frames at the server during the plurality of frame periods. The method includes, at a client, setting a client VSYNC signal to the client VSYNC frequency. The method includes transmitting a plurality of compressed video frames based on the plurality of video frames from the server to the client over a network using the server VSYNC signal. The method includes, at the client, decoding and displaying the plurality of compressed video frames. The method includes, when the client receives the plurality of compressed video frames, analyzing timing of one or more client operations to set an amount of frame buffering to be used by the client.

[0010] Another embodiment of the present disclosure discloses a non-transitory computer-readable medium storing a computer program for performing a method. The computer-readable medium includes program instructions for setting, at a server, a server VSYNC signal to a server VSYNC frequency defining a plurality of frame periods, the server VSYNC signal corresponding to generation of a plurality of video frames at the server during the plurality of frame periods. The computer-readable medium includes program instructions for setting, at a client, a client VSYNC signal to the client VSYNC frequency. The computer-readable medium includes program instructions for transmitting a plurality of compressed video frames based on the plurality of video frames from the server to the client over a network using the server VSYNC signal. The computer-readable medium includes program instructions for decoding and displaying the plurality of compressed video frames at the client. The computer-readable medium includes program instructions for analyzing timing of one or more client operations when the client receives the plurality of compressed video frames to set an amount of frame buffering to be used by the client.

[0011] In another embodiment of the present disclosure, a computer system is disclosed that includes a processor and a memory coupled to the processor, the memory storing instructions that, when executed by the computer system, cause the computer system to perform a method. The method includes setting, at a server, a server VSYNC signal to a server VSYNC frequency defining a plurality of frame periods, the server VSYNC signal corresponding to generation of a plurality of video frames at the server during the plurality of frame periods. The method includes setting, at a client, a client VSYNC signal to the client VSYNC frequency. The method includes transmitting, from the server to the client over a network, a plurality of compressed video frames based on the plurality of video frames using the server VSYNC signal. The method includes decoding and displaying, at the client, the plurality of compressed video frames. The method includes analyzing timing of one or more client operations when the client receives the plurality of compressed video frames to set an amount of frame buffering to be used by the client.

[0012] Another embodiment of the present disclosure discloses another method, the method including setting a plurality of VSYNC signals to a plurality of VSYNC frequencies at a plurality of devices, where corresponding device VSYNC signals of corresponding devices are set to the corresponding device VSYNC frequencies, and transmitting a plurality of signals between the plurality of devices, which are analyzed and used to adjust relative timing between corresponding device VSYNC signals of at least two devices.

[0013] Another embodiment of the present disclosure discloses a non-transitory computer-readable medium storing a computer program for performing a method, the computer-readable medium including program instructions for setting a plurality of VSYNC signals at a plurality of devices to a plurality of VSYNC frequencies, where corresponding device VSYNC signals of corresponding devices are set to corresponding device VSYNC frequencies, and the computer-readable medium including program instructions for transmitting a plurality of signals between a plurality of devices, the program instructions being analyzed and used to adjust the relative timing between corresponding device VSYNC signals of at least two devices.

[0014] In another embodiment of the present disclosure, a computer system is disclosed that includes a processor and a memory coupled to the processor, the memory storing instructions that, when executed by the computer system, cause the computer system to perform a method, the method including setting a plurality of VSYNC signals to a plurality of VSYNC frequencies at a plurality of devices, where corresponding device VSYNC signals of corresponding devices are set to the corresponding device VSYNC frequencies, and transmitting a plurality of signals between the plurality of devices, which are analyzed and used to adjust relative timing between corresponding device VSYNC signals of at least two devices.

[0015] Another embodiment of the present disclosure discloses another method. The method includes receiving encoded video frames at a client, and a server executing an application to generate rendered video frames, which are then encoded at an encoder at the server as encoded video frames, the encoded video frames including one or more compressed encoded slices. The method includes decoding the one or more encoded slices at a decoder at the client to generate one or more decoded slices. The method includes rendering the one or more decoded slices for display at the client. The method includes commencing display of the rendered one or more decoded slices before the one or more encoded slices have been completely received at the client.

[0016] Another embodiment of the present disclosure discloses a non-transitory computer-readable medium storing a computer program for executing a method. The computer-readable medium includes program instructions for receiving encoded video frames at a client, where a server executes an application to generate rendered video frames, which are then encoded by an encoder at the server as encoded video frames, the encoded video frames having one or more compressed encoded slices. The computer-readable medium includes program instructions for decoding the one or more encoded slices at a decoder at the client to generate one or more decoded slices. The computer-readable medium includes program instructions for rendering the one or more decoded slices for display at the client. The computer-readable medium includes program instructions for commencing display of the rendered one or more decoded slices before the one or more encoded slices have been completely received at the client.

[0017] Another embodiment of the present disclosure discloses a computer system including a processor and a memory coupled to the processor, the memory storing instructions that, when executed by the computer system, cause the computer system to perform a method. The method includes receiving encoded video frames at a client, and a server executing an application to generate rendered video frames, which are then encoded by an encoder at the server as encoded video frames, the encoded video frames including one or more compressed encoded slices. The method includes decoding the one or more encoded slices at a decoder at the client to generate one or more decoded slices. The method includes rendering the one or more decoded slices for display at the client. The method includes commencing display of the rendered one or more decoded slices before the one or more encoded slices are completely received at the client.

[0018] Other aspects of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the present disclosure.

[0019] The present disclosure is best understood by reference to the following detailed description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0020] [Figure 1] 1A and 1B are diagrams of a VSYNC signal at the beginning of a frame period and a frequency of a VSYNC signal, respectively, according to one embodiment of the present disclosure. [Figure 2A] 1 is a diagram of a system for providing games over a network between one or more cloud gaming servers and one or more client devices in various configurations, according to one embodiment of the present disclosure, in which VSYNC signals can be synchronized and offset to reduce one-way latency. [Figure 2B]FIG. 1 is a diagram for providing gaming between two or more peer devices, according to one embodiment of the present disclosure, in which VSYNC signals can be synchronized and offset to achieve optimal timing of reception of controller and other information between the devices. [Figure 2C] 1 illustrates various network configurations that benefit from proper synchronization and offset of VSYNC signals between source and target devices, according to one embodiment of the present disclosure. [Figure 2D] 1 illustrates a multi-tenancy configuration between a cloud gaming server and multiple clients that benefits from proper synchronization and offset of VSYNC signals between source and target devices, according to one embodiment of the present disclosure. [Figure 3] 1 illustrates the variation in one-way latency between a cloud gaming server and a client due to clock drift when streaming video frames generated from a video game running on the server, according to one embodiment of the present disclosure. [Figure 4] 1 illustrates a network configuration including a cloud gaming server and clients when streaming video frames generated from a video game running on the server, according to one embodiment of the present disclosure, where the VSYNC signals between the server and clients are synchronized and offset to allow for overlap of operations at the server and clients and to reduce one-way latency between the server and clients. [Figure 5A] FIG. 1 illustrates possible variations in the timing of completion of decoding by a client relative to the desired display time specified by the server due to drift between the clocks at the cloud gaming server and client, as well as variations in client and server operation times and network latency, according to one embodiment of the present disclosure. [Figure 5B]Included is a histogram showing the timing of completion of decoding by a client relative to a desired display time specified by a server, according to one embodiment of the present disclosure, illustrating an increase in measured decoding time in subsequent histograms due to drift between the clocks at the cloud gaming server and the client, respectively. [Figure 5C] According to one embodiment of the present disclosure, a histogram showing the timing of completion of decoding by a client relative to a desired display time specified by the server is included, showing consistent measurements of decoding time in subsequent histograms after compensation for measured drift between the clocks at the cloud gaming server and the client. [Figure 6A] 1 is a flow diagram illustrating a method for coordinating VSYNC signals between a cloud gaming server and a client for the purpose of reducing one-way latency, according to one embodiment of the present disclosure. [Figure 6B] 1 is a flow diagram illustrating a method for aligning VSYNC signals when performing VSYNC signal adjustment between a cloud gaming server and a client for the purpose of reducing one-way latency, including constructing a histogram that provides a distribution of timing of client decoding completion relative to a desired display time specified by the server, according to one embodiment of the present disclosure, wherein the histogram is configured to determine an adjustment of the offset between the VSYNC signals at the server and the client, and the histogram is also configured to determine drift between the server VSYNC signal and the client VSYNC signal. [Figure 6C] 10 is a flow diagram illustrating another method for synchronizing VSYNC signals when performing VSYNC signal coordination between a cloud gaming server and a client for the purpose of reducing one-way latency, according to an embodiment of the present disclosure. [Figure 7] 1 is a flow diagram illustrating a method for adjusting a client VSYNC signal in relation to generation of compressed video frames at a server, where the video frames are generated during similarly sized frame periods, according to one embodiment of the present disclosure. [Figure 8A]FIG. 10 illustrates the construction of a histogram providing a distribution of timing of completion of decoding by a client relative to a desired display time specified by a server, in accordance with one embodiment of the present disclosure, where the histogram is constructed for determining adjustment of the offset between VSYNC signals at the server and the client. [Figure 8B] FIG. 10 is a diagram of a histogram providing a distribution of timing of completion of decoding by a client relative to a desired display time specified by a server, the histogram being constructed to determine an adjustment of the offset between VSYNC signals at the server and the client, according to one embodiment of the present disclosure. [Figure 9] 1 is a flow diagram illustrating a method for constructing a histogram that provides a distribution of timing of completion of decoding by a client relative to a desired display time specified by a server, the histogram being configured to determine the amount of required buffering of decoded video frames at the client. [Figure 10] 1 is a flow diagram illustrating a method for adjusting the relative timing between VSYNC signals between two or more devices, according to one embodiment of the present disclosure. [Figure 11A] 1 illustrates overlapping reception, decoding, and rendering of decompressed video frames for display at a client, according to one embodiment of the present disclosure. [Figure 11B] 1 is a flow diagram illustrating a method for cloud gaming according to one embodiment of the present disclosure, in which encoded frames are received at a client from a server, decoded, and rendered for display, and the decoding and display of video frames may be overlapped to reduce one-way latency. [Figure 12] 1 illustrates components of an exemplary device that can be used to implement aspects of various embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0021] Although the following detailed description includes many specific details for purposes of illustration, those skilled in the art will appreciate that many variations and modifications to the following details are within the scope of the present disclosure. Accordingly, the aspects of the disclosure described below are set forth without loss of generality to, and without imposing limitations on, the claims that follow this description.

[0022] Generally, various embodiments of the present disclosure describe methods and systems configured to reduce latency and / or latency instability between a source device and a target device when streaming media content (e.g., streaming audio and video from a video game). In particular, in some embodiments of the present disclosure, the VSYNC signals between a cloud gaming server and a client are synchronized and offset. Clock differences between the cloud gaming server and the client cause the VSYNC signals of the cloud gaming server and the client to drift relative to each other. This drift leads to latency instability of the maximum frame period. For example, if a game is running at 60 Hz for video frame generation, there is an additional latency of 0 to 16.7 milliseconds that varies over time. By analyzing the worst-case or near-worst-case arrival times of compressed video frames at the client, an ideal VSYNC relationship between the cloud gaming server and the client can be determined. This ideal relationship can be established by adjusting the VSYNC frequency on either the cloud gaming server or the client, thereby eliminating latency instability. In another embodiment of the present disclosure, VSYNC signals between game devices (e.g., game consoles) are synchronized and offset to provide an ideal VSYNC relationship and minimal one-way latency between peer devices. Clock differences between peer devices (e.g., game consoles), particularly in face-to-face gaming, can cause their VSYNC signals to drift relative to one another, resulting in latency instability of up to a frame period. For example, if a game runs at 60 Hz for video frame generation, there is an additional 0-16.7 milliseconds of latency that varies over time. By exchanging timestamp information, the ideal VSYNC relationship between peer devices can be determined. This ideal relationship can be established by adjusting the VSYNC frequency on one of the peer devices, thereby eliminating latency instability. In yet another embodiment of the present disclosure, dynamic client buffering and selective use of video frames received at the client from the cloud gaming server results in reduced and regulated latency. Knowledge of the server-side timing of video frame generation allows the client to determine the ideal display time for each frame. Frame buffering (single buffering, double buffering, triple buffering, etc.) can be dynamically adjusted based on the variability of arrival times of compressed video frames to the client. Latency adjustments, such as selecting to skip display of late-arriving frames, can also occur. In another embodiment of the present disclosure, one-way latency between the cloud gaming server and the client can be reduced by overlapping the decoding of compressed video frames and their display. The cloud gaming client receives compressed video frames from the cloud gaming server and decodes the compressed video frames. One-way latency can be reduced by starting display of the video frames before the frames are fully received or decoded at the client. The timing of transmission for display must predict the remaining time required for receiving and decoding the compressed video frames.

[0023] In particular, latency instability can occur between the server and the client due to the additional time required to generate complex frames (e.g., scene changes) at the server, the increased time to encode / compress the complex frames at the server, variable communication paths over the network, and the increased time to decode the complex frames at the client. Latency instability can also occur due to differences in the server and client clocks, which causes the server and client VSYNC signals to drift. In one embodiment, this latency instability can be eliminated by adjusting either the server VSYNC signal or the client VSYNC signal to bring the server and client VSYNC signals back into synchronized alignment (e.g., operating at the same frequency). In another embodiment, adjusting the timing offset between the server VSYNC signal and the client VSYNC signal reduces one-way latency by accounting for near-worst-case latency conditions in receiving and displaying video frames at the client. In yet another embodiment, client-side dynamic buffering provides additional latency adjustment by providing more display buffer at the client when latency increases and using less display buffer when latency decreases. In another embodiment, one-way latency can be further reduced by overlapping the decoding and display of video frames at the client.

[0024] With the above general understanding of the various embodiments, the details of example embodiments will now be described with reference to the various figures.

[0025] Throughout this specification, references to "game," "video game," or "game application" or "application" are meant to describe any type of interactive application that is directed through the execution of input commands. By way of example only, interactive applications include applications for gaming, word processing, video processing, video game processing, etc. Furthermore, the terms introduced above are interchangeable.

[0026] Cloud gaming involves running a video game on a server to generate game-rendered video frames, which are then transmitted to clients for display. The timing of operations on both the server and the client may be associated with their respective vertical synchronization (VSYNC) parameters. When VSYNC signals are properly synchronized and / or offset between the server and / or client, operations performed on the server (e.g., generating and transmitting video frames over one or more frame periods) are synchronized with operations performed on the client (e.g., displaying video frames on a display at a display frame or refresh rate corresponding to the frame periods). In particular, a server-generated server VSYNC signal and a client-generated client VSYNC signal may be used to synchronize operations on the server and the client. That is, when the server and client VSYNC signals are synchronized and / or offset, the server generates and transmits video frames in sync with how the client displays those video frames.

[0027] VSYNC signaling and vertical blanking intervals (VBIs) are incorporated to generate and display video frames when streaming media content between a server and a client. For example, the server attempts to generate game-rendered video frames at one or more frame periods defined by the corresponding server VSYNC signal (e.g., if the frame period is 16.7 ms, generating a video frame every frame period results in 60 Hz operation, while generating one video frame every two frame periods results in 30 Hz operation), then encodes and transmits the video frames to the client. The client decodes and displays the received encoded video frames, and the client displays each video frame rendered for display beginning with the corresponding client VSYNC.

[0028] For purposes of explanation, FIG. 1A shows how a VSYNC signal 111 may indicate the start of a frame period, where various operations may be performed during the corresponding frame period at the server and / or client. When streaming media content, the server may use a server VSYNC signal to generate and encode video frames, and the client may use a client VSYNC signal to display the video frames. The VSYNC signal 111 is generated at a defined frequency corresponding to a defined frame period 110, as shown in FIG. 1B. Additionally, the VBI 105 defines the period from when the last raster line in the previous frame period was drawn to the display until the first raster line (e.g., top) is drawn to the display. As shown, after the VBI 105, the video frame rendered for display is displayed via raster scan line 106 (e.g., raster line by raster line from left to right).

[0029] Additionally, various embodiments of the present disclosure are disclosed for reducing one-way latency and / or latency variability between a source device and a target device, such as when streaming media content (e.g., video game content). For illustrative purposes only, various embodiments for reducing one-way latency and / or latency variability are described within a server-client network configuration. However, it is understood that various techniques disclosed for reducing one-way latency and / or latency variability may be implemented within other network configurations and / or on peer-to-peer networks, such as those illustrated in FIGS. 2A-2D. For example, various embodiments disclosed for reducing one-way latency and / or latency variability may be implemented between one or more server and client devices in various configurations (e.g., server-client, server-server, server-multiple clients, server-multiple servers, client-client, client-multiple clients, etc.).

[0030] 2A is a diagram of a system 200A for providing games over a network 250 between one or more cloud gaming networks 290 and / or a server 260 and one or more client devices 210 in various configurations, according to embodiments of the present disclosure, where server and client VSYNC signals can be synchronized and offset, and / or dynamic buffering can be performed on the client, and / or decoding and display operations at the client can be duplicated to reduce one-way latency between the server 260 and the client 210. In particular, the system 200A provides games over the cloud gaming network 290, where, according to one embodiment of the present disclosure, the games are executed remotely from the client devices 210 (e.g., thin clients) of corresponding users who are playing the games. The system 200A can provide game control to one or more users playing one or more games over the cloud gaming network 290 over the network 250 in either single-player or multiplayer mode. In some embodiments, cloud gaming network 290 may include multiple virtual machines (VMs) running on a host machine's hypervisor, with one or more virtual machines configured to run a game processor module that utilizes hardware resources available to the host's hypervisor. Network 250 may include one or more communication technologies. In some embodiments, network 250 may include fifth-generation (5G) network technology with advanced wireless communication systems.

[0031] In some embodiments, communication may be facilitated using wireless technology. Such technology may include, for example, 5G wireless communication technology. 5G is the fifth generation of cellular network technology. 5G networks are digital cellular networks, with service areas covered by providers divided into small geographic areas called cells. Analog signals representing sound and images are digitized by the telephone, converted by an analog-to-digital converter, and transmitted as a stream of bits. All 5G wireless devices within a cell communicate over the airwaves with a local antenna array and low-power automatic transceiver (transmitter / receiver) within the cell via frequency channels assigned by the transceiver from a pool of frequencies reused by other cells. The local antennas are connected to the telephone network and the Internet by high-bandwidth optical fiber or wireless backhaul connections. As with other cellular networks, mobile devices moving from one cell to another are automatically transferred to the new cell. It should be understood that 5G networks are merely one example type of communication network, and embodiments of the present disclosure may utilize previous generations of wireless or wired communications, as well as later generations of wired or wireless technologies following 5G.

[0032] As shown, game network 290 includes game server 260, which provides access to multiple video games. Game server 260 may be any type of server computing device available in the cloud and may be configured as one or more virtual machines running on one or more hosts. For example, game server 260 may manage virtual machines supporting game processors that instantiate instances of users' games. Thus, the game server 260's multiple game processors, associated with the multiple virtual machines, are configured to run multiple instances of one or more games associated with the gameplay of multiple users. In this manner, the back-end server support provides streaming gameplay media (e.g., video, audio, etc.) for multiple game applications to a corresponding number of users. That is, the game servers 260 are configured to stream data (e.g., rendered images and / or frames of corresponding game play) back to corresponding client devices 210 over the network 250. In this way, computationally complex game applications can continue to run on the backend servers in response to controller inputs received and forwarded by the client devices 210. Each server can render images and / or frames, then encode (e.g., compress) them and stream them to corresponding client devices for display.

[0033] For example, multiple users may access the cloud gaming network 290 via the communications network 250 using corresponding client devices 210 configured to receive streaming media. In one embodiment, the client devices 210 may be configured as thin clients that interface with a backend server (e.g., a game server 260 of the cloud gaming network 290) configured to provide computing functionality (e.g., including a game title processing engine 211). In another embodiment, the client devices 210 may be configured with a game title processing engine and game logic for at least some local processing of a video game, and may be further utilized to receive streaming content generated by the video game running on the backend server or for other content provided by the backend server support. In the case of local processing, the game title processing engine includes basic processor-based functionality for running the video game and services related to the video game. The game logic is stored on the local client device 210 and is used to run the video game.

[0034] In particular, a client device 210 of a corresponding user (not shown) is configured to request access to a game via a communications network 250, such as the Internet, and to render display images generated by the video game executed by a game server 260, where encoded images are delivered to the client device 210 for display in association with the corresponding user. For example, a user can interact through the client device 210 with an instance of a video game executing on a game processor of the game server 260. More specifically, the instance of the video game is executed by a game title processing engine 211. Corresponding game logic (e.g., executable code) 215 implementing the video game is stored and accessible via a data store (not shown) and is used to execute the video game. The game title processing engine 211 can support multiple video games using multiple game logics, each selectable by a user.

[0035] For example, client device 210 is configured to interact with a game title processing engine 211 associated with a corresponding user's gameplay, such as via input commands used to drive gameplay. In particular, client device 210 may receive input from various types of input devices, such as a game controller, a tablet computer, a keyboard, gestures captured by a video camera, a mouse, a touchpad, etc. Client device 210 may be any type of computing device having at least a memory and a processor module, and may be connected to game server 260 via network 250. The backend game title processing engine 211 is configured to generate rendered images, which are distributed over the network 250 for display on corresponding displays associated with the client devices 210. For example, via a cloud-based service, the game-rendered images may be distributed by an instance of the corresponding game executing on the game execution engine 211 of the game server 260. That is, the client devices 210 are configured to receive encoded images (e.g., encoded from game-rendered images generated through execution of a video game) and display the rendered images for the displays 11. In one embodiment, the displays 11 include HMDs (e.g., displaying VR content). In some embodiments, the rendered images can be streamed wirelessly or wired to a smartphone or tablet directly from the cloud-based service or via the client devices 210 (e.g., PlayStation® Remote Play).

[0036] In one embodiment, game server 260 and / or game title processing engine 211 include basic processor-based functionality for running services related to games and game applications. For example, processor-based functionality may include 2D or 3D rendering, physics, physics simulation, scripting, audio, animation, graphics processing, lighting, shading, rasterization, ray tracing, shadowing, culling, transformations, artificial intelligence, etc. Additionally, game application services may include memory management, multi-threading management, quality of service (QoS), bandwidth testing, social networking, social friend management, communication with social network friends, communication channels, texting, instant messaging, chat support, etc.

[0037] In one embodiment, cloud gaming network 290 is a distributed game server system and / or architecture. Specifically, a distributed game engine that executes game logic is configured as a corresponding instance of a corresponding game. Generally, a distributed game engine takes each function of the game engine and distributes those functions to be executed by multiple processing entities. Individual functions may be further distributed across one or more processing entities. The processing entities may be configured in various configurations, including physical hardware and / or virtual components or virtual machines and / or virtual containers. A container is different from a virtual machine because it virtualizes an instance of a game application running on a virtualized operating system. The processing entities may utilize and / or rely on servers and underlying hardware on one or more servers (computing nodes) of the cloud gaming network 290, which may be located on one or more racks. Coordination, allocation, and management of the execution of their functions across the various processing entities is performed by a distributed synchronization layer. In this manner, the execution of their functions is controlled by the distributed synchronization layer to generate media (e.g., video frames, audio, etc.) for the game application in response to controller inputs by the player. The distributed synchronization layer can efficiently execute those functions across the distributed processing entities (e.g., via load balancing) so that critical game engine components / functions are distributed and restructured for more efficient processing.

[0038] The game title processing engine 211 includes a group of central processing units (CPUs) and graphics processing units (GPUs) configured to perform multi-tenancy GPU functions. In another embodiment, multiple GPU devices are combined to perform graphics processing for a single application running on a corresponding CPU.

[0039] 2B is a diagram for providing a game between two or more peer devices, where, according to one embodiment of the present disclosure, VSYNC signals can be synchronized and offset to achieve optimal timing of receipt of controller and other information between the devices. For example, a face-to-face game can be performed using two or more peer devices connected directly via a network 250 or via peer-to-peer communication (e.g., Bluetooth, local area networking, etc.).

[0040] As shown, a game is running locally on each of the client devices 210 (e.g., game consoles) of corresponding users playing the video game, and the client devices 210 communicate via peer-to-peer networking. For example, an instance of the video game is executed by a game title processing engine 211 of the corresponding client device 210. Game logic 215 (e.g., executable code) that implements the video game is stored on the corresponding client device 210 and used to run the game. Illustratively, the game logic 215 may be distributed to the corresponding client device 210 via portable media (e.g., optical media) or over a network (e.g., downloaded from a game provider via the Internet).

[0041] In one embodiment, the game title processing engine 211 of the corresponding client device 210 includes basic processor-based functionality for executing services related to games and game applications. For example, the processor-based functionality may include 2D or 3D rendering, physics, physics simulation, scripting, audio, animation, graphics processing, lighting, shading, rasterization, ray tracing, shadowing, culling, transformations, artificial intelligence, etc. Additionally, game application services may include memory management, multi-threading management, quality of service (QoS), bandwidth testing, social networking, social friend management, communication with social network friends, communication channels, texting, instant messaging, chat support, etc.

[0042] Client device 210 can receive input from various types of input devices, such as a game controller, a tablet computer, a keyboard, gestures captured by a video camera, a mouse, a touchpad, etc. Client device 210 can be any type of computing device having at least a memory and a processor module, configured to generate rendered images executed by game title processing engine 211 and display the rendered images on a display (e.g., display 11, or display 11 including a head-mounted display—HMD, etc.). For example, the rendered images can be associated with an instance of a game running locally on client device 210 to implement a corresponding user's gameplay, such as via input commands used to drive the gameplay. Some examples of client device 210 include a personal computer (PC), a game console, a home theater device, a general-purpose computer, a mobile computing device, a tablet, a phone, or any other type of computing device capable of running an instance of a game.

[0043] FIG. 2C illustrates various network configurations that benefit from proper synchronization and offset of VSYNC signals between source and target devices, according to embodiments of the present disclosure, including the configurations illustrated in FIGS. 2A-2B. In particular, various network configurations benefit from proper alignment of the frequency of the server and client VSYNC signals and timing offset of the server and client VSYNC signals to reduce one-way latency and / or latency variability between the server and client. For example, one network device configuration includes a cloud gaming server (e.g., source) to client (target) configuration. In one embodiment, the client may include a web RTC client configured to provide audio and video communications within a web browser. Another network configuration includes a client (e.g., source) to server (target) configuration. Yet another network configuration includes a server (e.g., source) to server (e.g., target) configuration. Another network device configuration includes a client (e.g., source) to client (target) configuration, where each client may be, for example, a game console for providing face-to-face gaming.

[0044] In particular, VSYNC signal alignment includes synchronizing the frequencies of the server VSYNC signal and the client VSYNC signal, and may also include adjusting the timing offset between the client VSYNC signal and the server VSYNC signal to eliminate drift and / or reduce unidirectional latency and / or latency variation in order to maintain an ideal relationship between the client VSYNC signal and the server VSYNC signal. To achieve proper alignment, in one embodiment, the server VSYNC signal may be adjusted to enforce proper alignment between the server 260 and client 210 pair. In another embodiment, the client VSYNC signal can be adjusted to achieve proper alignment between a server 260 and client 210 pair. When the client and server VSYNC signals are aligned, the server VSYNC signal and the client VSYNC signal occur at substantially the same frequency and are offset from each other by a timing offset. The timing offset can be adjusted at any time. In another embodiment, alignment of the VSYNC signals may also synchronize the VSYNC frequencies of the two clients, and may include adjusting the timing offset between their VSYNC signals to remove drift and / or achieve optimally timed receipt of controller and other information, and both VSYNC signals may be adjusted to achieve this alignment. In yet another embodiment, alignment may also synchronize the VSYNC frequencies of multiple servers and may include synchronizing the frequencies of the server and client VSYNC signals and adjusting the timing offset between the client and server VSYNC signals, for example, in the case of face-to-face cloud gaming. In server-to-client and client-to-client configurations, alignment may include both synchronizing the frequencies between the server and client VSYNC signals and providing an appropriate timing offset between the server and client VSYNC signals. In server-to-server configurations, alignment may include synchronizing the frequencies between the server and client VSYNC signals without setting a timing offset.

[0045] 2D illustrates a multi-tenancy configuration between a cloud gaming server 260 and one or more clients 210 that benefits from proper synchronization and offset of VSYNC signals between source and target devices, according to one embodiment of the present disclosure. In a server-to-client configuration, coordination may include both synchronizing the frequency between the server VSYNC signal and the client VSYNC signal and providing a proper timing offset between the server VSYNC signal and the client VSYNC signal. In a multi-tenancy configuration, in one embodiment, the client VSYNC signal is adjusted at each client 210 to implement proper alignment between the server 260 and client 210 pair.

[0046] For example, a graphics subsystem may be configured to perform multi-tenancy GPU functionality, and in one embodiment, a single graphics subsystem may implement the graphics and / or rendering pipeline for multiple games. That is, the graphics subsystem is shared among multiple running games. In particular, a game title processing engine may also include a CPU and GPU group configured to perform multi-tenancy GPU functionality, and in one embodiment, a single CPU and GPU group may implement the graphics and / or rendering pipeline for multiple games. That is, the CPU and GPU group is shared among multiple running games. The CPU and GPU group may be configured as one or more processing devices. In another embodiment, multiple GPU devices are combined to perform graphics processing for a single application running on a corresponding CPU.

[0047] FIG. 3 illustrates a typical process for running a video game on a server to generate game-rendered video frames and transmitting those video frames to a client for display. Traditionally, several operations on the game server 260 and client 210 are performed within a frame period defined by their respective VSYNC signals. For example, the server 260 attempts to generate game-rendered video frames at 301 in one or more frame periods, as defined by a corresponding server VSYNC signal 311. The video frames are generated by the game in response to either control information (e.g., a user's input command) delivered from an input device at operation 350 or game logic not driven by control information. Transmission jitter 351 can be present when transmitting control information to the server 260, and jitter 351 measures the variation in network latency from the client to the server (e.g., when transmitting an input command). As shown, the thick arrow indicates the current delay in transmitting control information to server 260, although due to jitter, there may be a range of arrival times for the control information at server 260 (e.g., the range enclosed by the dotted arrow). At flip time 309, the GPU receives a flip command indicating that the corresponding video frame has been fully generated and placed in the frame buffer of server 260. Server 260 then performs a scan-out / scan-in (operation 302, where scan-out may be aligned with VSYNC signal 311) on that video frame over the subsequent frame period defined by server VSYNC signal 311 (VBI omitted for clarity). The video frame is then encoded (operation 303) (e.g., encoding may begin after the occurrence of VSYNC signal 311, and the end of encoding may not be aligned with VSYNC signal 311) and transmitted to client 210 (operation 304, where transmission may not be aligned with VSYNC signal 311). At client 210, the encoded video frames are received (operation 305, reception may not be aligned with client VSYNC signal 312), decoded (operation 306, decoding may not be aligned with client VSYNC signal 312), buffered, and displayed (operation 307, start of display may be aligned with client VSYNC signal 312). In particular, client 210 displays each video frame rendered for display beginning with the corresponding occurrence of client VSYNC signal 312.

[0048] One-way latency 315 can be defined as the latency from the start of the server's transfer of a video frame to the encoding unit (e.g., scanout 302) to the start of the video frame's display at the client 307. That is, one-way latency is the time from server scanout to client display, taking into account client buffering. Individual frames incur latencies from the start of scanout 302 to the completion of decoding 306, which can vary from frame to frame due to a high degree of variability in server operations, such as encoding 303 and transmitting 304, network transmission between the server 260 and the client 210 with attendant jitter 352, and client reception 305. As shown, the straight, thick arrow indicates the current latency in transmitting the corresponding video frame to the client 210, but there can be a range of arrival times for the video frame at the client 210 (e.g., the range bounded by the dashed arrow) due to jitter 352. Because a good playing experience requires one-way latency to be relatively stable (e.g., fairly consistent), buffering 320 is traditionally performed, resulting in the display of individual low-latency frames (e.g., from the start of scanout 302 to the completion of decoding 306) being delayed by a small frame period. This means that in cases of network instability or unpredictable encoding / decoding times, additional buffering is required to maintain consistent one-way latency.

[0049] According to one embodiment of the present disclosure, one-way latency between a cloud gaming server and a client can vary due to clock drift when streaming video frames generated from a video game running on the server. That is, due to differences in the frequencies of the server VSYNC signal 311 and the client VSYNC signal 312, the client VSYNC signal may drift relative to frames arriving from the server 260. The drift may be due to slight differences in the crystal oscillators used in the server and client clocks. Furthermore, embodiments of the present disclosure reduce one-way latency by performing one or more synchronizations and offsets of the VSYNC signal for alignment between the server and client, by providing dynamic buffering at the client, and by overlapping the decoding and display of video frames at the client.

[0050] 4 illustrates the flow of data through a network configuration including a highly optimized cloud gaming server 260 and a highly optimized client 210 when streaming video frames generated from a video game running on the server, according to an embodiment of the present disclosure, overlapping server operations with client operations to reduce one-way latency, and synchronizing and offsetting VSYNC signals between the server and client to not only reduce one-way latency but also reduce variability in one-way latency between the server and client. In particular, FIG. 4 illustrates the desired alignment between the server and client VSYNC signals. In one embodiment, the adjustment of the server VSYNC signal 311 is performed to obtain proper alignment between the server and client VSYNC signals, such as in a server and client network configuration. In another embodiment, the adjustment of the client VSYNC signal 312 is performed to obtain proper alignment between the server and client VSYNC signals, such as in a multi-tenant server to multi-client network configuration. For purposes of illustration, the adjustment of the server VSYNC signal 311 is depicted in FIG. 4 for the purpose of synchronizing the frequency of the server and client VSYNC signals and / or adjusting the timing offset between corresponding client and server VSYNC signals, although it will be understood that the client VSYNC signal 312 can also be used for adjustment.

[0051] As shown, FIG. 4 illustrates an improved process for executing a video game on a server to generate rendered video frames and send those video frames to a client for display, in an embodiment of the present disclosure. This process is illustrated with respect to the generation and display of a single video frame on the server and client. In particular, the server generates a game-rendered video frame at 401. For example, the server 260 includes a CPU (e.g., game title processing engine 211) configured to execute the game. The CPU generates one or more draw calls for the video frame, and the draw calls include commands placed in a command buffer for execution by a corresponding GPU of the server 260 in a graphics pipeline. The graphics pipeline may include one or more shader programs for vertices of objects in a scene to generate texture values ​​rendered into the video frame for display. In this case, operations are performed in parallel via the GPU for efficiency. At flip time 409, the GPU reaches a flip command in the command buffer, indicating that the corresponding video frame has been fully generated and / or rendered and placed in the frame buffer of the server 260.

[0052] At 402, the server performs scanout of the game-rendered video frame to the encoder. In particular, scanout is performed scanline by scanline or in groups of consecutive scanlines, where a scanline refers to, for example, a single horizontal line of the display from edge of the screen to edge of the screen. These scanlines or groups of consecutive scanlines are sometimes referred to as slices, and are referred to herein as screen slices. In particular, scanout 402 may include several processes that modify the game-rendered frame, including overlaying it with another frame buffer or shrinking it to surround it with information from another frame buffer. During scanout 402, the modified video frame is then scanned into the encoder for compression. In one embodiment, scanout 402 is performed at occurrence 311a of VSYNC signal 311. In other embodiments, scanout 402 may be performed before the occurrence of VSYNC signal 311, such as at flip time 409.

[0053] At 403, the game-rendered video frame (possibly modified) is encoded in an encoder slice-by-slices manner to generate one or more encoded slices, where the encoded slices are independent of scanlines or screen slices. Thus, the encoder generates one or more encoded (e.g., compressed) slices. In one embodiment, the encoding process begins before the scanout 402 process of the corresponding video frame is fully finished. Furthermore, the start and / or end of encoding 403 may or may not be aligned with the server VSYNC signal 311. The boundaries of an encoded slice are not limited to a single scan line and may consist of a single scan line or multiple scan lines. Furthermore, the end of an encoded slice and / or the start of the next encoder slice do not necessarily occur at the edge of the display screen (e.g., they could occur somewhere in the center of the screen or the middle of a scan line), and an encoded slice need not completely traverse from one edge of the display screen to the other. As shown, one or more encoded slices may be compressed and / or encoded, including a compressed "encoded slice A" with hash marks.

[0054] At 404, the encoded video frame is transmitted from the server to the client; the transmission may be performed on a per-encoded-slice basis, with each encoded slice being a compressed encoder slice. In one embodiment, the transmission process 404 begins before the encoding process 403 for the corresponding video frame is fully finished. Furthermore, the start and / or end of the transmission 404 may or may not be aligned with the server VSYNC signal 311. As shown, compressed encoded slice A is transmitted to the client independently of the other compressed encoder slices of the rendered video frame. The encoder slices may be transmitted one at a time or in parallel.

[0055] At 405, the client receives the compressed video frames, again for each encoded slice. Furthermore, the beginning and / or end of receive 405 may or may not be aligned with the client VSYNC signal 312. As shown, compressed encoded slice A is received by the client. Transmission jitter 452 may exist between the server 260 and the client 210, where jitter 452 measures the variation in network latency from the server 260 to the client 210. The lower the jitter value, the more stable the connection. As shown, the thick straight arrow indicates the current latency in transmitting the corresponding video frame to the client 210, but due to jitter, there may be a range of arrival times for the video frames at the client 210 (e.g., the range bounded by the dotted arrow). The variation in latency may be due to one or more operations at the server, such as encoding 403 and transmitting 404, as well as network issues that introduce latency when transmitting the video frames to the client 210.

[0056] At 406, the client decodes the compressed video frame, again encoded slice by slice, to produce decoded slice A (shown without hash marks), which is now ready for display. In one embodiment, the decoding process 406 begins before the receiving process 405 is fully completed for the corresponding video frame. Furthermore, the start and / or end of the decoding 406 may or may not be aligned with the client VSYNC signal 312. At 407, the client displays the decoded rendered video frame on the client's display. That is, the decoded video frame is placed in a display buffer from which it is streamed, e.g., scanline by scanline, to a display device. In one embodiment, the display process 407 (i.e., streaming out to the display device) begins after the decode process 406 is fully finished for the corresponding video frame, i.e., after the decoded video frame is fully resident in the display buffer. In another embodiment, the display process 407 begins before the decode process 406 is fully finished for the corresponding video frame. That is, streaming out to the display device begins from an address in the display buffer at a point where only a portion of the decoded frame buffer resides in the display buffer. The display buffer is then updated or filled with the remaining portions of the corresponding video frame in time for display, so that the display buffer update occurs before those portions are streamed out to the display. Furthermore, the start and / or end of the display 407 is aligned with the client VSYNC signal 312.

[0057] In one embodiment, one-way latency 416 between server 260 and client 210 may be defined as the elapsed time from when scanout 402 begins to when display 407 begins. Embodiments of the present disclosure may align (e.g., synchronize frequency and adjust offset) the VSYNC signals between the server and client to reduce one-way latency between the server and client and reduce one-way variation in latency between the server and client. For example, embodiments of the present disclosure may calculate an optimal adjustment to offset 430 between server VSYNC signal 311 and client VSYNC signal 312 so that, even with near-worst-case times required for server processing such as encoding 403 and transmitting 404, near-worst-case network latency between server 260 and client 210, and near-worst-case client processing such as receiving 405 and decoding 406, a decoded, rendered video frame is available in time for display process 407. In other words, it is not necessary to determine an absolute offset between server VSYNC and client VSYNC. It is sufficient to adjust the offset so that the decoded rendered video frame is in time for the display process.

[0058] In particular, the frequencies of the server VSYNC signal 311 and the client VSYNC signal 312 may be aligned by synchronization. The synchronization is achieved by adjusting the server VSYNC signal 311 or the client VSYNC signal 312. For purposes of explanation, the adjustment is described with reference to the server VSYNC signal 311, but it will be understood that the adjustment may instead be performed on the client VSYNC signal 312. For example, as shown in FIG. 4, the server frame period 410 (e.g., the time between two occurrences 311c and 311d of the server VSYNC signal 311) is substantially equal to the client frame period 415 (e.g., the time between two occurrences 312a and 312b of the client VSYNC signal 312). This indicates that the frequencies of the server VSYNC signal 311 and the client VSYNC signal 312 are also substantially equal.

[0059] To maintain synchronization of the frequencies of the server and client VSYNC signals, the timing of the server VSYNC signal 311 can be manipulated. For example, the vertical blanking interval (VBI) of the server VSYNC signal 311 can be increased or decreased over a period of time to account for drift between the server VSYNC signal 311 and the client VSYNC signal 312, for example. Manipulating the vertical blanking interval (VBLANK) lines in the VBI achieves adjusting the number of scan lines used for VBLANK for one or more frame periods of the server VSYNC signal 311. Reducing the number of VBLANK scan lines shortens the corresponding frame period (e.g., time interval) between two occurrences of the server VSYNC signal 311. Conversely, increasing the number of VBLANK scan lines increases the corresponding frame period (e.g., time interval) between two occurrences of the VSYNC signal 311. In this manner, the frequency of the server VSYNC signal 311 is adjusted to align the frequencies between the client and server VSYNC signals 311 and 312 so that they are substantially the same frequency. Additionally, the offset between the server and client VSYNC signals can be adjusted by briefly increasing or decreasing the VBI before returning it to its original value. In one embodiment, the server VBI is adjusted. In another embodiment, the client VBIs are coordinated. In yet another embodiment, instead of two devices (server and client), there may be multiple connected devices, each of which may have a corresponding coordinated VBI. In one embodiment, each of the multiple connected devices may be an independent peer device (e.g., without a server device). In another embodiment, the multiple devices may include one or more server devices and / or one or more client devices arranged in one or more server / client architectures, multi-tenant server / client(s) architectures, or some combination thereof.

[0060] Alternatively, in one embodiment, the server's pixel clock (e.g., located in the southbridge of the server's northbridge / southbridge core logic chipset) can be manipulated to perform coarse and / or fine adjustments to the frequency of the server's VSYNC signal 311 over a period of time to bring the frequencies between the server and client VSYNC signals 311 and 312 back into alignment. Specifically, the server's southbridge pixel clock can be overclocked or underclocked to adjust the overall frequency of the server's VSYNC signal 311. In this manner, the frequency of the server VSYNC signal 311 is adjusted to align the frequencies between the client and server VSYNC signals 311 and 312 so that they are substantially the same frequency. The offset between the server and client VSYNC can be adjusted by briefly increasing or decreasing the client-server pixel clock before restoring it to its original value. In one embodiment, the server pixel clocks are adjusted. In another embodiment, the client pixel clocks are adjusted. In another embodiment, the client pixel clocks are adjusted. In yet another embodiment, instead of two devices (a server and a client), there may be multiple connected devices, each of which may have a corresponding pixel clock that is adjusted. In one embodiment, each of the multiple connected devices may be an independent peer device (e.g., without a server device). In another embodiment, the multiple connected devices may include one or more server devices and one or more client devices arranged in one or more server / client architectures, multi-tenant server / client(s) architectures, or some combination thereof.

[0061] FIG. 5A illustrates possible variations in the timing of completion of decoding 406 by a client 210 when streaming video frames generated from a video game executed on a server 260 due to drift 390 between the clocks of the respective cloud gaming servers 260 and the client 210, as well as variations in the time spent by server operations such as encoding 403 and transmitting 404, network latency, and client operations such as receiving 405 and decoding 406, in accordance with one embodiment of the present disclosure.

[0062] The y-axis 501 shows time in milliseconds. The x-axis 502 shows time in minutes. In an embodiment of the present disclosure, the server 260 can send the timestamp information to the client 210 along with the compressed video frames, or the server can send the timestamp information separately from the compressed video frames. In one embodiment, this timestamp information may represent the time of occurrence of the server VSYNC signal immediately preceding the scanout 402 of the corresponding video frame, derived from the pixel clock of the server 260. That is, the timestamp indicates the desired display timing of the corresponding video frame, such as whether the video frame will be displayed immediately. In another embodiment, this timestamp information may represent a flip time, e.g., the time of completion of rendering of the corresponding video frame, derived from the pixel clock of server 260. In yet another embodiment, a normal frame period is used instead of the server VSYNC signal, and this timestamp information may represent the start or end time of the corresponding frame period, derived from the pixel clock of server 260.

[0063] Upon completion of decoding 406, the client 206 records a time derived from the client's pixel clock and subtracts this time from the timestamp (delivered from the server) to create a "decoding timestamp" (i.e., the time that decoding completed, rather than the time it took to decode the corresponding video frame). The decode timestamp thus indicates the availability of the corresponding video frame for display at the client, compared to the desired display time specified by the server (e.g., indicated by the timestamp). As shown in FIG. 5A, in one embodiment, the decode timestamp of the first compressed video frame received by the client is assigned a value of zero (i.e., is normalized). Additionally, all other decoding timestamps are calculated with reference to the normalization (i.e., subtracted, taking the normalization into account). Due to variations in server and client operation and network latency, the decoding timestamps measured for a series of compressed video frames may be plotted as a distribution 510, as previously described, with the first compressed video frame 511 assigned a decoding timestamp of zero. These decoding timestamps may be binned to create a histogram 515, as shown in FIG. 5B, described more fully below. Measurements 520, 530, and 540 illustrate the distribution of decoding timestamps calculated for subsequent video frames received at the client. The decoding timestamps of measurements 520, 530, and 540 are calculated with reference to the normalization defined by the first compressed video frame 511.

[0064] FIG. 5B includes a histogram illustrating the timing (e.g., corresponding timestamps) of client decoding completion relative to a desired display time specified by the server, according to one embodiment of the present disclosure, showing an increase in the decoding timestamps in subsequent histograms due to drift between the respective clocks at the cloud gaming server and the client. In particular, the decoding timestamps of each of measurements 520, 530, and 540 subsequently determined for subsequent video frames may be binned to create corresponding histograms. For example, the decoding timestamps plotted in measurement 520 may be binned to create histogram 525. Similarly, the decoding timestamps plotted in measurement 530 may be binned to create histogram 535, and the decoding timestamps plotted in measurement 540 may be binned to create histogram 545. Each of histograms 515, 525, 535, and 545 may have unique characteristics. As shown, the width of the distribution of decoded timestamps in each of measurements 515, 525, 535, and 545 is roughly similar, but the histogram determined later shows an increase in the decoded timestamps as they are each shifted to the right, due to, for example, drift between the server clock and client clock used to generate the VSYNC frequency.

[0065] In particular, measurements 520, 530, and 540 of the decoded timestamps determined later for subsequent video frames (and their corresponding histograms 525, 535, and 545 shown in FIG. 5B ) illustrate the effect of the difference between the server VSYNC signal 311 and the client VSYNC signal 312 (i.e., drift 390 in FIG. 3 ). The drift 390 between the server clock and the client clock, as reflected in the corresponding VSYNC signals, can be shown as 590 in FIG. 5B . For example, a 10 ppm (parts per million) difference between the server and client clocks due to inaccuracies in the crystal oscillators used to generate the VSYNC frequency at the server and client, results in a drift of one frame period (16.7 ms) between the server and client approximately every 30 minutes. The drift can take the form of a decrease or increase in the decoded timestamp, depending on whether the server VSYNC 311 is slightly higher or lower frequency than the client VSYNC signal 312.

[0066] FIG. 5C includes a histogram showing the timing of client decoding completion relative to the desired display time specified by the server, according to one embodiment of the present disclosure, showing consistent measurements of decoding time in subsequent histograms after compensation for measured drift between the clocks at the cloud gaming server and the client. In one embodiment, the amount of drift can be calculated by dynamically regenerating the histogram. Knowing the amount of drift allows for corresponding adjustment of the VSYNC frequency and removal of the drift, resulting in histograms 516, 526, 536, 546, etc. over time. In this manner, plotted measurements 520, 530, and 540 do not exhibit an increase in decoding timestamps (e.g., a vertical upward shift along x-axis 501 in FIG. 5A), but are plotted in more horizontal alignment with measurement 510 in FIG. 5A (i.e., no unidirectional increase in latency). This is reflected in FIG. 5C. In this figure, line 590' shows zero drift between the server clock and the client clock, which is reflected in the corresponding VSYNC signal (ie, does not show the unidirectional latency increase of histograms 526, 536, and 546).

[0067] In another embodiment, the server transmits compressed video frames every frame period (the frame periods being approximately equal in length), and instead of a corresponding server timestamp (e.g., used in calculating the decoding timestamp) being sent from the server to the client, the corresponding server timestamp (for the corresponding video frame being received) is calculated by the client by adding the frame period to a previously calculated server timestamp (e.g., for the previous video frame received). An initial server timestamp and / or timing signal for an initial video frame may be delivered from the server to the client to start the timing process.

[0068] In yet another embodiment, the drift is calculated using timestamp information sent from the server to the client separately or together with the compressed video frames, for example, by analyzing the variance between the timestamp information and the timing of receipt of the timestamp information at the client. In yet another embodiment, instead of using the server VSYNC signal, the server can use a frame period of approximately equal size and calculate the drift of the server frame period relative to the client VSYNC (or some multiple thereof). The server frame period can then be adjusted accordingly based on the drift calculation.

[0069] In other embodiments, instead of two devices (server and client), there are multiple connected devices, each of which can measure its drift relative to one or more other devices. In one embodiment, each of the multiple connected devices can be an independent peer device (e.g., without a server device). In another embodiment, the multiple devices can include one or more server devices and one or more client devices arranged in one or more server / client architectures, multi-tenant server / client(s) architectures, or some combination thereof.

[0070] Thus, the measured drift between the frequencies of the VSYNC signals between two devices (e.g., a server and a client, or any two devices in a multiple network device with a server, a client, and independent peers) can be used to adjust the VSYNC signals to one or more devices. The adjustments may include removing or adding raster scan lines in the vertical blanking interval of the corresponding frame period. The adjustments may also include overclocking or underclocking the corresponding clock. The adjustments may be performed dynamically, with the adjustment of the corresponding VSYNC signal changing over time. In some embodiments, instead of using a VSYNC signal, the server may use frame periods of approximately equal size, and the server frame period may be adjusted in response to the calculation of the drift.

[0071] 4 may include adjusting the server VSYNC signal 311 to provide an appropriate timing offset 430 between the corresponding client and server VSYNC signals. In one embodiment, histogram data (e.g., as shown in at least FIGS. 5A-5C) may be used to determine a timing offset such that a predetermined number or threshold (e.g., 99.99 percent) of received video frames arrive at the client in time to be displayed at the next appropriate occurrence of the client VSYNC signal 312. That is, the timing offset 430 (e.g., as shown in FIG. 4) is adjusted to accommodate a near-worst-case scenario of a video frame being ready for display (e.g., received, decoded, and placed in a display buffer for streaming out at the client) in time for the minimum amount of time until the next occurrence of the client VSYNC signal 312. As an example, a 99.99 percent threshold provides for one missing video frame in 10,000 video frames generated at the server 260 and displayed at the client 210.

[0072] In particular, to establish the appropriate offset 430, the frequency of the server VSYNC signal 311 may be manipulated over a period of time (e.g., one or more frame periods) to move the timing of one or more occurrences of the corresponding server VSYNC signal 311, thereby shifting or adjusting the relative timing offset between the client and server VSYNC signals once both VSYNC signals synchronize their respective frequencies. The frequency of the client VSYNC signal 312 may also be similarly manipulated to alternatively adjust the relative timing offset between the client and server VSYNC signals. Determination of the appropriate offset may be performed dynamically, e.g., repeatedly over time, using corresponding dynamic manipulation of the VSYNC signals. In other embodiments, rather than first removing drift between the server and client VSYNC signals and then establishing an appropriate offset between the VSYNC signals, the offset is instead maintained by more frequently manipulating the frequency of the server or client VSYNC signals to adjust the relative timing offset. In yet other embodiments, instead of using a server VSYNC signal, the server can use frame periods of approximately equal size and manipulate the server frame period or the client VSYNC signal to adjust the relative timing offset of the server frame period for the client VSYNC (or multiples thereof). In yet other embodiments, instead of two devices (a server and a client), there are multiple connected devices, each of which can manipulate its own VSYNC signal to adjust the relative timing offset of its VSYNC with respect to the VSYNC of one or more other devices. In one embodiment, each of the multiple connected devices can be an independent peer device (e.g., without a server device). In another embodiment, the multiple devices can include one or more server devices and one or more client devices arranged in one or more server / client architectures, multi-tenant server / client(s) architectures, or some combination thereof.

[0073] Along with detailed descriptions of various client devices 210 and / or cloud gaming network 290 (e.g., within game server 260) in Figures 2A-2D, flowcharts 600A, 600B, and 600C in Figures 6A-6C illustrate a method for coordinating VSYNC signals between a cloud gaming server and a client for the purpose of reducing one-way latency, according to one embodiment of the present disclosure.

[0074] In particular, Figure 6A illustrates a method for adjusting the relative timing between cloud gaming server and client VSYNC signals to reduce one-way latency, according to one embodiment of the present disclosure. For example, flowchart 600A can be implemented to adjust the timing between corresponding client and server VSYNC signals shown in Figure 4.

[0075] At 601, the method includes setting, at a server, a server VSYNC signal to a server VSYNC frequency. As described above, the server VSYNC signal corresponds to generating multiple video frames at the server during multiple frame periods of the server VSYNC frequency. For example, the server may execute a video game in a streaming mode, and a CPU of the server may execute the video game in response to input commands from a user to generate game-rendered video frames using a graphics pipeline enabled for streaming.

[0076] At 603, the method includes setting a client VSYNC signal at a client to a client VSYNC frequency. The client VSYNC signal is used for rendering to a display associated with the client. That is, the timing of rendering and displaying video frames at the client can be based on the client VSYNC signal. For example, video frames can be displayed starting from the occurrence of a corresponding client VSYNC signal.

[0077] In one embodiment, the client VSYNC frequency is set approximately to the server VSYNC frequency. For example, the server can send a control signal to the client, which is used by the client to set the client's VSYNC signal to the apparent frequency of the server's VSYNC signal. That is, the control signal may include the apparent frequency to which the client VSYNC signal is set. That is, the client VSYNC frequency is set to the same apparent frequency as the server VSYNC frequency, but the actual server and client VSYNC frequencies may not match due to differences in the crystals used for the server and client clocks.

[0078] At 605, the method includes transmitting a plurality of compressed video frames based on the plurality of video frames from the server to the client over a network using a server VSYNC signal. In particular, game-rendered video frames generated in response to processing of the video game by the server in streaming mode are delivered (e.g., during scanout 402) to an encoder configured to perform compression and generate a plurality of compressed video frames. As described above, the start of encoding of the corresponding video frames may be aligned with the corresponding occurrence of the server VSYNC signal or may occur before the corresponding occurrence, such as a flip time. The compressed video frames are transmitted and / or delivered to the client for display during the game session. Transmission of the video frames does not need to begin aligned with the server VSYNC signal and can begin as soon as a portion or a complete corresponding video frame has been encoded, as shown in FIG. 4.

[0079] At 607, the method includes decoding and displaying, at the client, the plurality of compressed video frames. As described above, the client receives the plurality of compressed video frames, which are then decoded by a decoder at the client. For example, the client receives one or more encoded slices of the corresponding compressed video frames. The compressed video frames are then decoded and placed in a display buffer. For example, the encoded slices of the corresponding compressed video frames are then decoded and the decoded video frames are placed in the display buffer. During decoding, the decoded slices can be rendered for display, which can include generating screen slices (e.g., scanlines) from the decoded slices of the corresponding video frames, which are then streamed to the client's display. In particular, pixel data of the decoded slices of the corresponding video frames can be placed at appropriate addresses in the display buffer for streaming (e.g., scanline by scanline) to the client's display.

[0080] At 610, the method includes analyzing the timing of one or more client operations to adjust the relative timing between the server VSYNC signal and the client VSYNC signal when the client receives the plurality of compressed video frames. For example, the relative timing is adjusted to achieve proper alignment (e.g., frequency synchronization and offset adjustment) between the server and client VSYNC signals to reduce one-way latency between the server and client and to reduce one-way latency variation. In one embodiment, the proper timing between the server and client VSYNC signals is achieved by adjusting at least one of the server VSYNC signal and the client VSYNC signal. A more detailed discussion of adjusting the relative timing between the server and client VSYNC signals is provided in Figures 6B-6C below.

[0081] 6B is a flow diagram 600B illustrating a method for aligning VSYNC signals when performing VSYNC signal adjustment between a cloud gaming server and a client, according to one embodiment of the disclosure. For example, alignment may include synchronizing frequency and / or adjusting offsets between the server and client VSYNC signals, such as to reduce one-way latency. In particular, FIG. 6B provides additional details regarding the adjustment of relative timing between the server and client VSYNC signals outlined in operation 610 of FIG. 6A.

[0082] At 611, the method includes transmitting timestamp information associated with a plurality of video frames (e.g., generated by the server as video frames rendered by the game) from the server to the client. In one embodiment, the timestamp information is transmitted to the client along with the plurality of compressed video frames. In another embodiment, the timestamp information is transmitted to the client separately from the plurality of compressed video frames.

[0083] In particular, the timestamp information includes a corresponding timestamp of a corresponding video frame generated at the server, derived by the server's pixel clock. In one embodiment, the timestamp of the corresponding video frame may occur at a corresponding occurrence of a server VSYNC signal used to scan the corresponding video frame into the encoder, such as during scanout (e.g., the occurrence of the server VSYNC signal immediately prior to scanout of the corresponding video frame). In another embodiment, the timestamp of the corresponding video frame may occur at a corresponding flip time on the server. The timestamp indicates a desired display timing of the corresponding video frame as determined by the video game, such as scanning or streaming to a local display without transmission over a network.

[0084] When a client receives and decodes a compressed video frame, the timestamp information is processed and compared to a desired display time (timestamp) specified by the server to create a decode timestamp indicating the availability of the corresponding video frame for display at the client. As previously described, the client records the time, derived from the client's pixel clock, upon completion of decoding of the corresponding video frame. This decoding completion time is subtracted from the corresponding timestamp delivered by the server to create the decode timestamp. Additionally, the first compressed video frame decoded may be normalized such that its decode timestamp is adjusted to zero using a normalization factor that may be applied (e.g., added or subtracted) to all subsequently measured and / or calculated decode times to allow subsequent reception of compressed video frames at the client.

[0085] At 613, the method includes constructing one or more histograms based on the measured and / or calculated decoding timestamps. In particular, the corresponding histograms are created by binning measured decoding timestamp information for compressed video frames received at the client over a period of time. As described above, the measured and normalized decoding timestamps for the corresponding video frames indicate when the decoded video frames are displayable at the client relative to the desired display time indicated by the server timestamp. In this manner, the corresponding histograms provide a distribution of timing of completion of decoding by the client relative to the desired display time specified by the server for multiple video frames. Accordingly, the one or more generated histograms may be used to adjust the relative timing between the server and client VSYNC signals in order to reduce one-way latency and / or one-way latency variability between the server and client.

[0086] At 615, the relative timing between the server and client VSYNC signals is adjusted to synchronize the server VSYNC frequency of the server VSYNC signal with the client VSYNC frequency of the client VSYNC signal. In one embodiment, drift is determined between the server VSYNC signal and the client VSYNC signal using corresponding histograms. For example, one or more histograms generated from video frames received by the client are continuously and dynamically updated. The histograms are analyzed to determine drift between the server VSYNC signal and the client VSYNC signal. For example, FIG. 5B illustrates how drift (e.g., as reflected by line 590) can be determined when plotting multiple histograms, as described above.

[0087] Alternative methods for determining drift between server and client VSYNC signals can be used for synchronization. For example, an analysis of decoded timestamps over a period of time can be performed to determine trends in increasing or decreasing decode timing, which can be used to determine drift. In another example, drift can be calculated by analyzing the variance between server-generated timestamp information and client-based timing of receipt of server-based timestamp information. Drift can also be measured between multiple connected devices, which can be independent peer devices, server and client devices, arranged in a peer-to-peer architecture, a server / client architecture, or a combination thereof.

[0088] Furthermore, based on the timestamp information sent from the server to the client, at least one of the server VSYNC frequency and the client VSYNC frequency can be adjusted to compensate for the measured drift. For example, the frequency of the server VSYNC signal or the client VSYNC signal can be adjusted over a period of time so that the actual frequencies of the server and client VSYNC signals are approximately the same over that period. In this way, the frequencies of the server and client VSYNC signals are synchronized.

[0089] As previously mentioned, the frequency of the corresponding server or client VSYNC signal can be adjusted by removing or adding raster scan lines in the vertical blanking interval of the corresponding frame period, and the VSYNC signal can be adjusted for a certain period of time. The adjustment may include overclocking or underclocking the corresponding pixel clock of the server or client for a certain period of time. Furthermore, the adjustment can be performed continuously by dynamically adjusting the corresponding VSYNC signal appropriately over time.

[0090] At 617, the relative timing between the server and client VSYNC signals is adjusted by adjusting the relative offset between the server and client VSYNC signals based on the timestamp information. Similarly, the relative phase between the server and client VSYNC signals can be adjusted based on the timestamp. Once the frequencies of the server and client VSYNC signals are synchronized, adjustments to the relative phase or offset of the server and client VSYNC signals can be applied to either the server or client VSYNC signals.

[0091] In particular, adjusting the offset between the server VSYNC signal and the client VSYNC signal is determined based on the near-worst-case decode timestamps shown in the corresponding histograms. For example, the timing offset can be determined so that a predetermined number or threshold (e.g., 99.99 percent) of received video frames arrive at the client in time to be decoded and displayed upon the next appropriate occurrence of the client VSYNC signal. In this manner, the near-worst-case scenario of one-way latency of video frames is also taken into account when adjusting the timing offset between the server and client VSYNC signals, and the near-worst-case scenario of video frames being received, decoded, and paced in the display buffer for streaming out to the client display. Determining the appropriate timing offset is further described in conjunction with FIGS. 8A-8B.

[0092] As described above, adjusting the timing offset between the server and client VSYNC signals can be achieved by adjusting the server VSYNC signal or the client VSYNC signal by one or more frame periods. For example, adjusting the timing offset can be performed by adjusting the frequency of the corresponding server or client VSYNC by one or more frame periods. In particular, the frequency of the corresponding server or client VSYNC signal can be adjusted by removing or adding raster scan lines in the vertical blanking interval of the corresponding frame period, or by overclocking or underclocking the corresponding pixel clock of the server or client.

[0093] In some embodiments, rather than first removing drift between the server and client VSYNC signals and then establishing an appropriate offset between the VSYNC signals, the timing offset is instead maintained by more frequently manipulating the frequency of the server or client VSYNC signal to adjust the relative timing offset. That is, adjustments to the offset between the server and client VSYNC signals are continually determined based on the near-worst case decode timestamps indicated by corresponding histograms, which can be generated over short periods of time with frequent determination and manipulation of the timing offset.

[0094] 6C is a flow diagram 600C illustrating another method for aligning VSYNC signals when performing VSYNC signal adjustment between a cloud gaming server and a client for the purpose of reducing one-way latency, according to one embodiment of the present disclosure. As previously discussed, alignment may include synchronizing frequencies and / or adjusting offsets between the server and client VSYNC signals, such as for the purpose of reducing one-way latency. In particular, FIG. 6C provides additional details regarding the adjustment of the relative timing between the server and client VSYNC signals outlined in operation 610 of FIG. 6A.

[0095] Specifically, an alternative method for determining drift between the server and client VSYNC signals is used for synchronization as outlined in operation 620, including operations 621 and 623. In particular, at 621, the method includes periodically sending timing information from the server to the client. For example, the timing information determined from the server pixel clock may include the start of the corresponding frame period, the length of time of the corresponding frame period, the scan-out timing of the video frame into the encoder, the flip time of the corresponding video frame, etc.

[0096] Also, at 623, the timing information is analyzed to determine drift between the server VSYNC signal and the client VSYNC signal. For example, drift may be calculated by analyzing the variance between server-generated timing information and client-based timing of receipt of the server-based timing information. In other embodiments, drift may be measured with respect to the server frame period used to generate the video frames, with the drift in the server frame period being measured with reference to the client VSYNC signal, or some multiple thereof. Drift may also be measured between multiple connected devices, which may be independent peer devices, server and client devices, arranged in a peer-to-peer architecture, a server / client architecture, or a combination thereof.

[0097] Once the drift is determined, the frequency of the server VSYNC signal or the client VSYNC signal can be adjusted to compensate for the drift, and the adjustment can be applied over a period of time. As described above, the measured drift between the frequencies of the server and client VSYNC signals can be used to adjust the VSYNC signal at the server or client for a period of time. Adjusting the VSYNC signal can include removing or adding raster scan lines in the vertical blanking interval of the corresponding frame period, or can include overclocking or underclocking the corresponding pixel clock at the server or client.

[0098] After compensating for the drift between the frequencies of the server and client VSYNC signals, the relative phase or offset between the server and client VSYNC signals can be adjusted based on the timestamp information, and the adjustment can be applied to either the server VSYNC signal or the client VSYNC signal. In particular, the relative phase or offset adjustment is performed based on the timestamp information associated with the video frames generated at the server.

[0099] In particular, at 611, timestamp information is transmitted from the server to the client. In an embodiment, the timestamp information is transmitted along with the plurality of video frames or separately from the plurality of video frames. Operation 611 of flowchart 600C was previously described in connection with flowchart 600B of FIG. 6B. For example, the timestamp information, determined by the server pixel clock, may indicate the timing of a corresponding occurrence of a server VSYNC signal used to scan the corresponding video frame into the encoder (e.g., during scan-out). In another embodiment, the timestamp information may indicate the timing of a corresponding flip time of the corresponding video frame.

[0100] As mentioned above, the timestamp information is used by the client to create decoding timestamps, each indicating the availability for display at the client of the corresponding video frame compared to the desired display time (timestamp) specified by the server. The decoding timestamps can be derived by subtracting the time at the client indicating completion of decoding of the corresponding video frame from the corresponding server-based timestamp information. Normalization can also be applied when generating the decoding timestamps.

[0101] Drifting can be performed without using timestamp information; for example, to generate multiple histograms, one histogram is generated at a specific time and used to adjust for the timing offset between the server and client VSYNC signals. That is, a histogram can be updated by extending the decoding timestamps included in the histogram, but generating multiple histograms spanning different time periods is not necessary. In particular, in 613-A, a histogram is created based on the decoding timestamps. Operation 613-A of flowchart 600C is similar to operation 613 described above in connection with flowchart 600B of FIG. 6B. For example, decoding timestamp information is binned for video frames received and decoded by the client over time to provide a distribution of the timing of completion of decoding by the client relative to the desired display time specified by the server (e.g., server timestamp information).

[0102] At 617, the relative phase or offset between the server and client VSYNC signals may be adjusted based on the timestamp information, and the adjustment may be applied to either the server VSYNC signal or the client VSYNC signal. Operation 617 of flowchart 600C was previously described in connection with flowchart 600B of FIG. 6B. In particular, the offset adjustment is determined based on the near-worst-case decoding timestamp indicated by the continuously updated histogram. For example, a timing offset may be determined so that a predetermined threshold number (e.g., 99.99 percent) of received video frames arrive at the client in time to be decoded and displayed upon the next appropriate occurrence of the client VSYNC signal. Determining the appropriate timing offset is further described in connection with FIGS. 8A-8B.

[0103] Adjusting the timing offset between the server and client VSYNC signals is performed by adjusting the server or client VSYNC signal by one or more frame periods. For example, the adjustment can be performed by adjusting the frequency of the server or client VSYNC signal by removing or adding raster scan lines in the vertical blanking interval of the corresponding frame period, or by overclocking or underclocking the corresponding pixel clock of the server or client.

[0104] In some embodiments, the drift operation 620 of Figure 6C is not performed when establishing an appropriate offset between the VSYNC signals. Instead, the timing offset is maintained by more frequently manipulating the frequency of the server or client VSYNC signals to adjust the relative timing offset. That is, adjustments to the offset between the server and client VSYNC signals are continually determined based on the worst-case decoded timestamps indicated by the corresponding histograms, and adjustments to the offset may be performed at reduced intervals.

[0105] 2A-2D , flowchart 700 illustrates an alternative method for reducing one-way latency between a cloud gaming server and a client, according to one embodiment of the present disclosure. In particular, flowchart 700 illustrates a method for adjusting a client VSYNC signal in relation to generation of compressed video frames at a server, where the video frames are generated during similarly sized frame periods, according to one embodiment of the present disclosure.

[0106] At 710, the method includes generating a plurality of video frames at the server over a plurality of frame periods, the frame periods being approximately equal in size. In one embodiment, the cloud gaming server may turn off or not implement a server VSYNC signal because the video frames do not need to be displayed on the server when streaming to the client. Instead, the server may utilize a regular (e.g., periodic) or approximately regular signal used for timing during the generation of game-rendered video frames when processing the video game. For example, the server may use frame periods that are approximately equal in size instead of using a server VSYNC signal. The generation of the plurality of video frames occurs within the plurality of frame periods, such that game-rendered video frames are generated within corresponding frame periods. The server may execute the video game in streaming mode, and the server's CPU may execute the video game in response to input commands from a user to generate game-rendered video frames using a graphics pipeline.

[0107] At 720, the method includes setting a client VSYNC signal at the client to a client VSYNC frequency. The client VSYNC signal is used for rendering to a display associated with the client. Timing of rendering and displaying video frames at the client can be based on the client VSYNC signal, and corresponding video frames can be displayed starting from the occurrence of the corresponding client VSYNC signal.

[0108] At 730, the method includes transmitting a plurality of compressed video frames based on the plurality of video frames from the server to the client. In particular, the game-rendered video frames are delivered to an encoder of the server (e.g., during scanout 402), which is configured to perform compression on the game-rendered video frames. The plurality of compressed video frames are transmitted (e.g., streamed) to the client for display, such as during a game session. Transmission of the compressed video frames need not coincide with a frame period, such that transmission can occur as soon as a portion of a corresponding video frame is encoded or when a complete video frame has been encoded.

[0109] At 740, the method includes decoding and displaying, at the client, the plurality of compressed video frames. As described above, the client receives and decodes the plurality of compressed video frames. For example, the client can receive and decode one or more encoded slices of the corresponding compressed video frames. The decoded video frames are placed in a display buffer. During decoding, the decoded slices can be rendered for display, which includes generating screen slices (e.g., scanlines) from the decoded slices of the corresponding video frames, which are then streamed to the client's display. For example, pixel data for the decoded slices of the corresponding video frames can be placed at appropriate addresses in the display buffer for streaming (e.g., scanline-by-scanline) to the display.

[0110] At 750, the method includes transmitting timing information associated with a plurality of frame periods from the server to the client. The timing information may indicate when each frame period begins at the server. Because the frame periods are approximately equal, the timing of one frame period (e.g., a server timestamp) delivered to the client allows the client to track the timing of each frame period (e.g., by periodically adding the frame period to the last calculated timestamp). In this manner, the client can correlate timing information received from the server or calculated at the client with corresponding video frames received at the client. The client-determined timing information can provide an indication of the desired display timing of corresponding video frames as determined by the video game running at the client, such as when the video frames were generated and ideally scanned or streamed to a local display, without transmission over a network.

[0111] At 760, the method includes analyzing the timing of one or more client operations to adjust the relative timing of the client VSYNC signal and generate multiple compressed video frames at the server when the client receives the multiple compressed video frames. In particular, the server frame period (e.g., duration) or the client VSYNC signal may be manipulated to adjust the relative timing offset of the server frame period with respect to the client VSYNC signal (or some multiple thereof).

[0112] For example, the drift between the client-calculated server frame period and the client VSYNC signal can be determined. Drift compensation can be applied to the server frame period or the client VSYNC signal for synchronization purposes. For example, at the client, the frequency of the client VSYNC signal can be adjusted by removing or adding raster scan lines at the vertical balance interval of the corresponding frame period, or the VSYNC signal can be adjusted for a certain period. Adjustments may also include overclocking or underclocking the client's pixel clock. Furthermore, to adjust for the relative offset, one or more histograms can be created at the client, indicating when decoded video frames are available at the client (e.g., required display time) compared to when the video frames were generated at the server. The histograms can be created using the same techniques described above, with slight modifications, such as using client-determined frame periods to indicate when video frames are generated and displayed.

[0113] 8A is a diagram 800A illustrating the construction and use of a histogram 850 providing a distribution of decode timestamps for video frames, which, as previously described, indicates availability for display at a client of video frames relative to desired display times specified by a server. The histogram is configured to determine an adjustment of the offset between the VSYNC signals at the server and the client, according to one embodiment of the present disclosure. As shown, video frames are generated at the server (operation 401), and scanout is performed on the game-rendered video frames in an encoder (operation 402) for compression (operation 403) and transmission to the client (operation 404). The client receives the encoded video frames (operation 405), decompresses the encoded video frames (operation 406), and renders the video frames for display (e.g., converts the decoded video frames to scanlines in operation 407).

[0114] As previously mentioned, server-based timestamp information is delivered to the client in association with compressed / encoded video frames for the purpose of creating one or more histograms used to determine the offset. The timestamp information provides the desired display time specified by the server. The server may not transmit compressed video frames every frame period. In particular, the histogram may include a decoding timestamp that indicates the availability of the video frame for display at the client compared to the desired display time specified by the server (e.g., the server-based timestamp information). Because the server timestamp and the client timestamp may be defined by separate, unsynchronized clocks, it may be useful to normalize the decoding timestamps. For example, normalization may involve subtracting the value of the first decoding timestamp from all decoding timestamps. This causes the first decoding timestamp to be zero, and all subsequent timestamps to be relative to it.

[0115] As shown in FIG. 8A, the constructed histogram 850 can be used to determine an appropriate offset between the server VSYNC signal and the client VSYNC signal. As previously described, the histogram provides a distribution of decode timestamps, which indicates display availability relative to the desired display time. The VSYNC offset 430 between the server and client exists so that a predetermined number or threshold (e.g., 99.99 percent) of received video frames arrive at the client and are decoded in time to be displayed at the next appropriate occurrence of the client VSYNC signal. In one embodiment, the remaining number (e.g., 0.01 percent) may be too late to be displayed and may leak. In other words, the VSYNC offset 430 corresponds to a near-worst-case latency when receiving, decoding, and displaying rendered video frames. 8A shows a near-worst frame (e.g., the 99.99th percentile). With the proper VSYNC offset 430 established, there is no margin between the decoding of this frame and its display. One or more client buffers 820 can be implemented to accommodate video frames with low decode timestamps (e.g., indicating the shortest one-way latency) so that the encoding 403, transmitting 404, receiving 405, and decoding 406 are binned early in the histogram (e.g., below the 25th percentile). In this particular example, four buffers are required: three for the frame being decoded (the three buffers 820) and one for the currently displayed frame (not shown).

[0116] A series of theoretical timing diagrams 850A-850D are provided for the client VSYNC signal 312, with timing diagram 850C (and accompanying display 407) showing an ideal client VSYNC 312C. Because there is no direct synchronization of clocks or timestamps (e.g., via a third-party timing mechanism such as a universal clock), the offset 430 is not set directly; instead, the current client VSYNC 312 is adjusted using the near-worst-case timing information from the histogram to arrive at the ideal client VSYNC timing 312C, as described above. Alternatively, the server VSYNC can be adjusted to create the appropriate offset, as described above.

[0117] The server's timestamp information is collected and / or received by the client. As mentioned above, the timestamp information may include the time the corresponding video frame was generated (e.g., flip time, when scanout occurred, the occurrence of the server VSYNC signal when scanout occurred, etc.). Additional information may be collected at the server and / or client and used to create or interpret the histogram, and is referred to as "histogram information," as described in more detail below.

[0118] On the server side, additional histogram information may include encoding time statistics such as the number of scene changes, the mean and / or standard deviation of the encoding time for I-frames, and the mean and / or standard deviation of the encoding time for P-frames. The encoding time statistics may be delivered as periodic messages from the server to the client. Furthermore, the histogram information may include the time to prepare encoder slices by the encoder, which may be delivered as periodic messages from the server to the client. The histogram information may also include the actual server-side VSYNC timing and the target VSYNC timing, which may be added to the packet header. Furthermore, the histogram information may include the average number of slices per I-frame versus P-frame.

[0119] At the server, the information for the histogram may include a round-trip time (RTT) measurement to derive the one-way network latency for transmitting encoded slices (e.g., compressed encoder slices). The RTT measurement may be used to determine the transmission time required to send a packet to the client (e.g., without further processing performed by the client, such as decoding and rendering). For example, the RTT can be determined by sending a heartbeat packet from the server to the client. This packet contains a unique identifier. The client sends a heartbeat response back to the server with the unique identifier so that the server can calculate the RTT. The one-way network latency is approximately half the RTT. When used to create a histogram, periodically measuring the RTT can analyze and / or determine network or transmission jitter (e.g., spikes in the RTT). For example, a measurement of one-way network latency measured over the RTT can be used as the transmission time for all video frames received until the next RTT measurement.

[0120] At the client, the additional histogram information may include a decode time for each received encoded video frame. Additionally, the histogram information can include a render preparation time for each decoded video frame, where render preparation can include converting the decoded video frame slices into scanlines or screen slices.

[0121] Additionally, at the server, the additional histogram information may include a maximum transmission rate, which defines the total network throughput (e.g., bandwidth) that the server considers available to the client. This can be used to determine the maximum rate at which encoder slices of encoded video frames can be sent. The maximum rate can vary based on the stability of the network connection to the client, and the offset can be dynamically adjusted to accommodate the variations. Furthermore, the maximum transmission rate can be adjusted independently of the encoder parameters, allowing slices to be sent more quickly if the encoder is configured not to generate slices at the maximum transmission rate.

[0122] For example, the maximum bandwidth or maximum transmission rate can be determined through a feedback mechanism from the client. One way to do this is to have the client return the number of packets received over a range of incremental sequence IDs (identifiers) or a range of frames. For example, a client might report that it received 145 of 150 frames for sequence IDs 100-250. The server can calculate packet loss, know the amount of bandwidth transmitted during that series of packets, and determine the client's maximum bandwidth. The client cannot make this determination because the amount of bandwidth transmitted is constantly fluctuating due to variable bitrates, scene complexity, and so on. This means that the client does not know whether, at any given moment, the server is transmitting the maximum bandwidth the client can handle. For example, the maximum bandwidth may be 15 megabits per second (Mbps), but the scene complexity may be low because the user is viewing a menu (static video frames have low complexity and no inter-frame variation). As a result, only 2 Mbps is transmitted. Therefore, if a client reports 0% packet loss, the server is not informed whether the client can still handle 15 Mbps. Therefore, the true maximum bandwidth can only be determined if the server is transmitting at its maximum bandwidth.

[0123] 8B shows a histogram 850 illustrating the distribution of decode timestamps, which in one embodiment are normalized so that the numerically smallest decode timestamp is assigned a value of zero (e.g., the smallest decode timestamp is subtracted from all decode timestamps). In particular, the x-axis indicates the time of the corresponding decode timestamp in milliseconds, such as from 0 to over 60 milliseconds (ms). The y-axis indicates the number of video frames received by the client for the corresponding decode timestamp.

[0124] Purely for illustrative purposes, the decode timestamps may vary over a range of approximately 60 milliseconds (ms), indicating a 60 ms variance in the availability of video frames for display at the client compared to the desired display time specified by the server. That is, some frames may be available for display approximately 60 ms earlier or later than other frames. Variations in the availability of particular frames for display may be due to variations in server and client processing, scene complexity, network path variations, packet delay variations, and other factors. By analyzing the worst-case or near-worst-case decode timestamps, the ideal relationship between the server VSYNC signal and the client VSYNC signal can be determined. That is, as described above, the ideal relative offset between the timing of the client VSYNC signal and the server VSYNC signal can be determined to maximize the number of received and decompressed video frames that can be displayed at the appropriate client VSYNC signal. Thus, diagram 800B shows the width (e.g., approximately 57 milliseconds) of the distribution of decode timestamps 755, within which 99.99 percent of the video frames received by the client will arrive and be decoded in time for display at the next appropriate occurrence of the client VSYNC signal.

[0125] This width of the distribution of decode timestamps 755 (including all decode timestamps up to, but excluding, the worst-case case) can be used to determine the overall buffering requirements for decoded video frames. If the width 755 is less than a frame period, two buffers are required: one for the frame as it is decoded and one for display. For example, if the width is greater than one frame period but less than two frame periods, three buffers are required. In the specific example of a 57 ms width, five frame buffers are required for a frame period of 16.67 ms. The decode timestamp indicates the availability of decoded frames relative to the desired display time; therefore, video frames with lower decode timestamps are buffered for longer periods before display, and video frames with higher decode timestamps are buffered for shorter periods before display.

[0126] In one embodiment, the histogram is dynamically regenerated. In another embodiment, the amount of frame buffering is dynamically set by the client over time. In yet another embodiment, frames that arrive and are decoded too late to be displayed at the desired display time are skipped (i.e., not displayed).

[0127] 2A-2D , together with the detailed description of the various client devices 210 and / or cloud gaming network 290 (e.g., within game server 260), flowchart 900 of FIG. 9 illustrates a method for constructing a histogram that provides a distribution of the elapsed timing of video frames from when they are generated at a cloud gaming server to when they arrive and / or are ready for display at a client, according to one embodiment of the present disclosure, where the histogram is configured for buffer size determination at the client. As previously mentioned, the histogram is also configured for determining the appropriate offset between VSYNC signals at the server and the client.

[0128] Operations 601, 603, 605, 611, and 613 were previously described in connection with flowchart 600A of FIG. 6A and flowchart 600B of FIG. 6B and disclose adjusting the relative timing between the server and client VSYNC signals (e.g., synchronizing frequency and adjusting timing offset or phase). In summary, at 601, the method includes, at the server, setting a server VSYNC signal to a frequency corresponding to generation of video frames at the server during a frame period of the server VSYNC signal. At 603, the method includes, at the client, setting a client VSYNC signal corresponding to the frequency used for rendering to a display associated with the client. At 605, the method includes transmitting compressed video frames based on the generated video frames from the server to the client over a network using the server VSYNC signal.

[0129] At 611, the method includes transmitting timestamp information associated with the compressed video frame to the client. For example, the timestamp information may be transmitted together with or separately from the compressed video frame, ideally providing an indication of the desired display timing of the corresponding video frame at a time determined by the video game, such as when scanned or streamed to a local display, without transmission over a network. When the client receives and decodes the compressed video frame, the timestamp information is processed and compared to a desired display time specified by the server (e.g., a server timestamp) to create a decode timestamp indicating the availability of the corresponding video frame for display at the client. In one embodiment, the decode timestamp may be normalized because the server and client timing may be defined by corresponding individual clocks that are not synchronized. A complete discussion regarding the timestamp information is provided in connection with FIGS. 6B-6C and 8A-8B, and is equally applicable in connection with FIG. 9.

[0130] At 613, the method includes constructing a histogram based on the decoding timestamps measured and / or calculated at the client. For example, a corresponding histogram may be created by binning decoding timestamp information associated with compressed video frames received and decoded at the client over a period of time. Because the decoding timestamps indicate availability for display at the client of the video frames relative to desired display times specified by the server (e.g., server timestamps), the histogram also provides a distribution of timing of decoding completion of video frames received by the client relative to desired display times specified by the server (e.g., server timestamp information). A full discussion regarding timestamp information is provided in connection with FIGS. 6B-6C and 8A-8B and is equally applicable in connection with FIG. 9.

[0131] At 910, the method includes measuring the width of the histogram at a particular point in time. For example, the width of the distribution of decoded timestamps in the histogram may be measured so that a predetermined number or threshold (e.g., 99.99 percent) of the received video frames arrive at the client within the time indicated at the next appropriate occurrence of the client VSYNC signal 312 (for clarity, the remaining 0.01% of the received video frames are not included when measuring the width). In particular, the histogram width can be used to configure the amount of frame buffering required by the client at a particular point in time. Thus, at 920, the method dynamically configures a number of display buffers at the client based on the histogram width and the frame period of the synchronized server and client VSYNC signals, and histogram 750 is generated at a particular point in time. As noted above, if the width is less than the frame period, then two frame buffers, for example, are required. In this manner, video frames with lower decode timestamps are buffered for longer periods, while video frames with higher decode timestamps are buffered for shorter periods.

[0132] 2A-2D , flowchart 1000 illustrates a method for adjusting the relative timing between VSYNC signals between two or more devices, according to one embodiment of the present disclosure. In particular, flowchart 1000 may be used to compensate for drift and / or adjust the offset or phase between two or more VSYNC signals of corresponding devices.

[0133] At 1010, the method includes setting a plurality of VSYNC signals to a plurality of VSYNC frequencies on a plurality of devices, where corresponding device VSYNC signals of corresponding devices are set to the corresponding device VSYNC frequency. That is, each device sets its corresponding VSYNC signal using a corresponding pixel clock. Furthermore, the frequencies may be similar, e.g., set to the same apparent frequency, but the actual frequencies may differ due to differences between the various pixel clocks. These VSYNC signals may be used to generate video frames (e.g., a server in a server-client architecture) and / or display video frames (e.g., a client in a server-client architecture). These VSYNC signals may also be used to both generate video frames and display video frames. For example, in a peer-to-peer architecture where each device is running a video game locally, the timing may coordinate the execution and display of video frames.

[0134] At 1020, the method includes transmitting a plurality of signals between the plurality of devices, which are analyzed and used to adjust relative timing between VSYNC signals of corresponding devices of at least two devices. The relative timing can be adjusted between devices configured in a server / client architecture or between devices configured in a peer-to-peer architecture. For example, the signals can include server timestamp information or server timing information, as described above, that provides an indication of when corresponding video frames are intended to be displayed by the server. In this manner, the VSYNC signals of the plurality of devices can be synchronized (e.g., synchronizing the frequency of the VSYNC signals) by determining drift between the at least two VSYNC signals. Also, timing offsets and / or timing phases can be adjusted between the at least two VSYNC signals.

[0135] In particular, in one embodiment, at least two of the devices may be configured in a server / client architecture. In another embodiment, the devices are arranged in a multi-tenant configuration (e.g., one server for multiple client devices). For example, a first device may be a server device whose server VSYNC signal is set to a server VSYNC frequency. The server VSYNC signal corresponds to the generation of multiple video frames during execution of an application on the server device for multiple frame periods of the server VSYNC frequency. Multiple compressed video frames are transmitted from the server device to each of the remaining devices (e.g., client devices) in the multiple devices over a network based on the server VSYNC signal. For example, the server VSYNC signal provides timing for the generation and encoding of video frames at the server. The compressed video frames are based on the video frames being generated by the server device. Each receiving device (e.g., the remaining devices) decodes and displays the received compressed video frames. The display of the decoded video frames may be synchronized between each receiving device.

[0136] In particular, relative timing may be adjusted between devices to compensate for drift and / or to adjust the timing offset or phase between the VSYNC signals of the devices. Drift and timing offset or phase adjustments may be determined using the techniques described above in connection with Figures 6A-6C, 7, and 8A-8B. The relative timing adjustments between the VSYNC signals of two devices may occur on either device and may include adjusting the frequency by removing or adding raster scan lines in the vertical blanking interval of the corresponding frame period of the corresponding device's VSYNC signal, or overclocking or underclocking the corresponding clock of the corresponding device.

[0137] In particular, in one embodiment, at least two of the devices may be configured in a peer-to-peer architecture. For example, each device may be an independent peer device; that is, none of the devices is a server device. In that manner, the devices may be configured for peer-to-peer gaming. Each device generates multiple video frames by processing the same video game. The independent peer devices may be operating in a multiplayer mode of a particular video game, with back-end server support controlling the multiplayer game session. The backend server can enable state sharing between devices by managing state data for each user of a multiplayer game session. The state data can include game state data that defines the state of gameplay (of a game application) for that user at a particular point. For example, the game state data can include game characters, game objects, game object attributes, game attributes, game object states, graphic overlays, etc. In this manner, objects and characters can be inserted into the game environment of each user participating in the multiplayer game session, allowing each user's gameplay to be customized to that user through state sharing. Additionally, each user's gameplay can be synchronized based on state sharing. That is, the video frames displayed on each device can be synchronized to reflect synchronized gameplay. In this manner, one user may not benefit from continuously receiving and displaying video frames on a corresponding device earlier than the video frames of other users' gameplay. Alternatively, no back-end server is involved, in which case the VSYNC relationship between peers is optimized to minimize the latency between receiving control or state information from other peers and displaying video frames using the information received from other peers.

[0138] 11A illustrates, in accordance with one embodiment of the present disclosure, overlapping the receiving, decoding, and rendering of decompressed video frames for display at a client 210. In particular, one-way latency between a server (not shown) and a client 210 in a cloud gaming application may be reduced by overlapping the receiving, decoding, and displaying of particular video frames.

[0139] For example, a client of a cloud gaming application receives and decodes video frames. In particular, the client receives encoded video frames 1105 at receiving operation 405, where a server runs a video game to generate game-rendered video frames, which are then encoded by the server's encoder and delivered to the client as encoded video frames 1105. The encoded video frames 1105 include one or more encoded slices that are compressed by the server's encoder. The client includes a decoder configured to decode one or more encoded slices in the encoded video frames at decoding operation 406. In one embodiment, the decoding process begins before the corresponding video frame is fully received at the client. The decoder performs decoding for each encoded slice, so that the decoded video frame 1106 includes one or more encoder slices. The decoded video frame 1106 is then prepared for display, such as by rendering the information in the decoded video frame 1106 into scanlines or screen slices. The client-rendered video frame 1107 is then ready for display.

[0140] One-way latency between the server and the client can be reduced by having the client 210 begin displaying a video frame at operation 407 before the video frame is fully decoded at operation 406. In particular, one or more decoded slices of a video frame may be ready for rendering to the display before the video frame is fully decoded. That is, the display operation at 407 overlaps with the decode operation at 406. In particular, the first encoded slice (e.g., slice A) must arrive and be decoded before client scanout begins displaying it. Furthermore, all subsequent encoded slices must arrive and be decoded before their respective decompressed data can be rendered and scanned out for display.

[0141] Furthermore, in addition to overlapping receive and decode operations at the client, display of one or more decoded slices that are then rendered in preparation for display may occur even before an encoded video frame sent by the server is fully received at the client. That is, one or more receive, decode, and display operations at the client may overlap for the corresponding video frame. Furthermore, in the case of overlapping operations at both the server and the client, one or more decoded slices of a rendered video frame that is then rendered in preparation for display may be displayed at the client even before a scanout operation at the server is fully completed, in which case, in one embodiment, scanout delivers the game-rendered video frame to the server's encoder.

[0142] The overlap of displaying in operation 407 and decoding in operation 406 can be performed on an encoder slice-by-slice basis. In this manner, an encoded slice can be displayed before one or more subsequent encoded slices are received. To do this, forward error correction (FEC) data must be interleaved between the encoded slices of the corresponding video frame. In particular, an encoded slice can be divided into one or more network packets. An FEC packet can be used to correct one or more packets associated with a slice. Thus, an FEC packet may be interleaved between packets of multiple slices. In this manner, forward error correction can be used early to correct missing and / or corrupted packets of a slice without waiting for the entire set of packets (e.g., data and FEC) of a frame to be received by the client. This allows for overlapping of decoding and display operations at the client.

[0143] In one embodiment, a decode timestamp can be created for each slice, which indicates the availability of the slice for display at the client compared to the desired display time specified by the server. The decode timestamp can be calculated by taking the completion time of the slice's decoding 406 at the client, subtracting the timestamp received from the server indicating the frame's ideal display time, and adding the time the decompressed slice will be used in the display process 407 (i.e., adding 0 ms if the decompressed slice data is needed immediately, or adding 8.33 ms if the slice data is needed partway through a 16.67 ms frame period). It may be useful to normalize the decode timestamps in some way, such as by subtracting the first decode timestamp from all other timestamps.

[0144] The decode timestamps can be arranged in a histogram similar to those shown in Figures 5A-5B and 8A-8B. The worst-case or near-worst-case (e.g., 99.999%) decode timestamp determined by the histogram can be used to adjust the relative server and client VSYNC timing, thereby reducing one-way latency. Because slices arriving and decoded late would cause visible corruption or "tearing" by using the existing contents of the display buffer for display, a very high threshold, such as 99.999%, is desirable, specifying that one frame will be missed for every 100,000 video frames generated by the server 260 and displayed by the client 210.

[0145] 2A-2D , together with the detailed description of the various client devices 210 and / or cloud gaming network 290 (e.g., within game server 260), according to an embodiment of the present disclosure, flowchart 1100B of FIG. 11B illustrates a method of cloud gaming in which encoded frames are received at a client from a server, decoded, and rendered for display, and the decoding and display of video frames can be overlapped to reduce one-way latency. The ability to overlap one or more operations at the client is achieved by managing one-way latency between the server and client, as previously described in FIGS. 4-10 . For example, the relative timing between the server and client VSYNC signals can be adjusted to reduce and / or minimize variations in one-way latency between the server and client.

[0146] At 1110, the method includes receiving an encoded video frame at a client, where a server executes an application to generate a rendered video frame, which is then encoded by an encoder at the server as an encoded video frame, where the encoded video frame includes one or more compressed encoded slices. For example, the server generates multiple video frames using a server VSYNC signal. Each video frame can be sent to an encoder for compression, and each video frame can be encoded into one or more encoded slices. As described above, the start of encoding of the corresponding video frame can be aligned with the server VSYNC signal. The compressed video frame is then sent to the client, where the transmission does not need to coincide with the server VSYNC signal and can begin as soon as an encoder slice or complete video frame has been encoded. The compressed video frame is received by the client.

[0147] At 1120, the method includes decoding the one or more coded slices at a decoder at the client to generate one or more decoded slices. In one embodiment, the decoding of the one or more coded slices may begin before the coded video frame is completely received at the client. For example, the client receives one or more coded slices of a corresponding video frame. Each of the coded slices is then decoded and placed in a display buffer, and the decoded video frame is placed in the display buffer.

[0148] At 1130, the method includes rendering one or more decoded slices for display at a client. In particular, during the decoding process, the decoded slices may be rendered for display, where rendering includes generating screen slices (e.g., scanlines) from the decoded slices of corresponding video frames, which are then streamed to a display at the client.

[0149] At 1140, the method includes, in one embodiment, initiating display of one or more decoded slices that are rendered prior to complete reception of the one or more encoded slices at the client. In particular, decoded slices placed in a display buffer may be immediately streamed to the client's display. As such, client operations of receiving and display may overlap.

[0150] In another embodiment, the method includes initiating display of one or more decoded slices rendered on a display before fully decoding one or more encoded slices. In particular, decoded slices placed in a display buffer may be immediately streamed to a client display. As such, client operations of decoding and display may overlap.

[0151] 12 illustrates components of an example device 1200 that can be used to implement aspects of various embodiments of the present disclosure. For example, FIG. 12 illustrates an example hardware system suitable for streaming media content and / or receiving streaming media content, providing dynamic buffering at a client, and redundant decoding and display of video frames at a client, such as adjusting a server or client VSYNC signal to synchronize and / or adjust the offset of the VSYNC signals between the server and client, in accordance with embodiments of the present disclosure. The block diagram illustrates device 1200, which may incorporate or be a personal computer, server computer, gaming console, mobile device, or other digital device, each suitable for implementing embodiments of the present invention. Device 1200 includes a central processing unit (CPU) 1202 that runs software applications and, optionally, an operating system. CPU 1202 may be comprised of one or more homogeneous or heterogeneous processing cores.

[0152] According to various embodiments, CPU 1202 is one or more general-purpose microprocessors having one or more processing cores. Further embodiments may be implemented using one or more CPUs with a microprocessor architecture specifically adapted for highly parallel and computationally intensive applications, such as applications configured for graphics processing during game execution, media and interactive entertainment applications, etc.

[0153] Memory 1204 stores applications and data used by CPU 1202 and GPU 1216. Storage 1206 provides non-volatile storage and other computer-readable media for applications and data and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other optical storage devices, as well as signal transmission and storage media. User input device 1208 communicates user input from one or more users to device 1200 and may include, for example, a keyboard, mouse, joystick, touchpad, touchscreen, still or video recorder / camera, and / or microphone. The network interface 1209 enables the device 1200 to communicate with other computer systems over electronic communication networks, and may include wired or wireless communication over local area networks and wide area networks such as the Internet. The audio processor 1212 is adapted to generate analog or digital audio output from instructions and / or data provided by the CPU 1202, the memory 1204, and / or the storage 1206. The components of the device 1200, including the CPU 1202, the graphics subsystem 1214, e.g., GPU 1216 and GPU cache 1218, the memory 1204, the data storage 1206, the user input devices 1208, the network interface 1209, and the audio processor 1212, are connected via one or more data buses 1222.

[0154] Graphics subsystem 1214 is further connected to data bus 1222 and the components of device 1200. Graphics subsystem 1214 includes a graphics processing unit (GPU) 1216 and graphics memory 1218. Graphics memory 1218 includes display memory (e.g., a frame buffer) used to store pixel data for each pixel of an output image. Graphics memory 1218 may be integrated into the same device as GPU 1216, connected as a separate device from GPU 1216, and / or implemented within memory 1204. Pixel data may be provided directly from CPU 1202 to graphics memory 1218. Alternatively, CPU 1202 may provide data and / or instructions defining desired output images to GPU 1216, which then generates pixel data for one or more output images. The data and / or instructions defining the desired output images may be stored in memory 1204 and / or graphics memory 1218. In an embodiment, GPU 1216 includes 3D rendering functionality that generates pixel data for output images from instructions and data that define scene geometry, lighting, shadowing, texture, motion, and / or camera parameters. GPU 1216 may also include one or more programmable execution units capable of executing shader programs.

[0155] Graphics subsystem 1214 periodically outputs image pixel data from graphics memory 1218 to be displayed on display device 1210 or projected by a projection system (not shown). Display device 1210 may be any device capable of displaying visual information in response to signals from device 1200, including CRT, LCD, plasma, and OLED displays. Device 1200 may provide analog or digital signals to display device 1210, for example.

[0156] Other embodiments for optimizing the graphics subsystem 1214 can include multi-tenancy GPU operations, where a GPU instance is shared among multiple applications, and distributed GPUs supporting a single game. The graphics subsystem 1214 can be configured as one or more processing devices.

[0157] For example, graphics subsystem 1214 may be configured to perform multi-tenancy GPU functions, and in one embodiment, one graphics subsystem may implement the graphics and / or rendering pipeline for multiple games, i.e., graphics subsystem 1214 is shared between multiple running games.

[0158] In other embodiments, graphics subsystem 1214 includes multiple GPU devices that are combined to perform graphics processing for a single application running on a corresponding CPU. For example, multiple GPUs can perform alternative forms of frame rendering, where, for example, GPU1 renders the first frame, GPU2 renders the second frame in successive frame periods, and when the last GPU is reached, the first GPU renders the next video frame (e.g., if there are only two GPUs, GPU1 renders the third frame). That is, GPUs rotate when rendering frames. Rendering operations can overlap, such that GPU2 can begin rendering the second frame before GPU1 finishes rendering the first frame. In another embodiment, multiple GPU devices can be assigned different shader operations in the rendering and / or graphics pipeline. A master GPU performs main rendering and compositing. For example, in a group including three GPUs, master GPU1 can perform main rendering (e.g., the first shader operation) and composite the outputs from slave GPU2 and slave GPU3, slave GPU2 can perform a second shader operation (e.g., a fluid effect such as a river), slave GPU3 can perform a third shader operation (e.g., particle smoke), and master GPU1 can composite the results from each of GPU1, GPU2, and GPU3. In this manner, various GPUs can be assigned to perform various shader operations (e.g., flag waving, wind, smoke generation, fire, etc.) to render a video frame. In yet another embodiment, each of the three GPUs can be assigned to a different object and / or portion of the scene corresponding to the video frame. In the above embodiments and implementations, these operations can be performed in the same frame period (concurrently in parallel) or in different frame periods (sequentially in parallel).

[0159] Accordingly, this disclosure describes methods and systems configured for streaming media content and / or receiving streaming media content, providing dynamic buffering at a client, and overlapping the decoding and display of video frames at a client, including adjusting a VSYNC signal at a server or a client to synchronize and / or adjust the offset of the VSYNC signals between the server and the client.

[0160] It should be understood that the various embodiments defined herein can be combined or incorporated into specific embodiments using various features disclosed herein. Thus, the examples provided are merely some possible examples and are not limited to the various embodiments in which many more embodiments can be defined by combining various elements. In some instances, an embodiment may include fewer elements without departing from the spirit of the disclosed or equivalent embodiments.

[0161] Embodiments of the present disclosure may be practiced with a variety of computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. Embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.

[0162] With the above embodiments in mind, it should be understood that embodiments of the present disclosure may employ various computer-implemented operations involving data stored in computer systems. These operations are operations requiring physical manipulation of physical quantities. Any of the operations described herein that form part of embodiments of the present disclosure are useful machine operations. Embodiments of the disclosure also relate to devices or apparatus for performing these operations. An apparatus may be specially constructed for the required purposes. Alternatively, the apparatus may be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general-purpose machines may be used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct a more specialized apparatus to perform the required operations.

[0163] The present disclosure can also be embodied as computer-readable code on a computer-readable medium. A computer-readable medium is any data storage device that can store data which can thereafter be read by a computer system. Examples of computer-readable media include hard drives, network-attached storage (NAS), read-only memory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tape, and other optical and non-optical data storage devices. The computer-readable medium can also include computer-readable tangible media distributed over network-connected computer systems so that the computer-readable code is stored and executed in a distributed fashion.

[0164] Although the operations of the method are described in a particular order, it should be understood that other housekeeping operations may be performed between operations, or operations may be arranged to occur at slightly different times, or operations may be distributed within the system to allow processing operations to occur at various intervals relative to processing, so long as the processing of the overlay operations is performed in the desired manner.

[0165] Although the foregoing disclosure has been described in some detail for clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. Accordingly, the present embodiments are to be considered as illustrative and not restrictive, and embodiments of the present disclosure are not to be limited to the details provided herein, but may be modified within the scope of the appended claims and their equivalents.

Claims

1. In a plurality of devices, setting a plurality of VSYNC signals to a plurality of VSYNC frequencies, and corresponding device VSYNC signals of corresponding devices are set to the corresponding device VSYNC frequencies; A method comprising transmitting a plurality of signals between the plurality of devices, the signals being analyzed and used to adjust the relative timing between VSYNC signals of corresponding devices of at least two devices.

2. further, the first device is a server device, and setting a server VSYNC signal to the server device at a server VSYNC frequency, the server VSYNC signal corresponding to generating video frames of a plurality of frame periods while executing an application on the server device during the plurality of server VSYNC frequencies; transmitting a plurality of compressed video frames based on the plurality of video frames from the server device to each of the remaining devices of the plurality of devices over a network using the server VSYNC signal; and each of the remaining devices of the plurality of devices decoding and displaying the plurality of compressed video frames; The method of claim 1 , wherein the display of the plurality of compressed video frames on each of the remaining devices of the plurality of devices is synchronized.

3. The method of claim 2 , wherein each of the remaining devices of the plurality of devices is a client device configured in a client architecture that includes a server and the server device.

4. The method of claim 3 , wherein the server device is configured with the remaining devices of the plurality of devices in a multi-tenant configuration.

5. Further, generating a corresponding plurality of video frames when processing the application on a corresponding device of the plurality of devices; and the corresponding device of each of the plurality of devices displays the corresponding plurality of video frames; The method of claim 1 , wherein the display of corresponding video frames on the multiple devices is synchronized.

6. each of the plurality of devices is an independent peer device; The method of claim 1 , wherein the plurality of devices are arranged in a peer-to-peer configuration for peer-to-peer gaming.

7. The method of claim 1 , wherein at least one of the plurality of signals comprises timestamp information.

8. The method of claim 1 , wherein the plurality of signals are analyzed to synchronize device VSYNC signals of the plurality of devices.

9. The relative timing between corresponding device VSYNC signals of at least two devices is 9. The method of claim 8, wherein the adjustment is performed by removing or adding raster scan lines in the vertical blanking interval of the corresponding frame period of the VSYNC signal of the corresponding device.

10. The relative timing between corresponding device VSYNC signals of at least two devices is The method of claim 8, wherein the adjustment is performed by overclocking or underclocking a corresponding clock of a corresponding device.

11. Further, determining a drift between a VSYNC frequency of the first device and a VSYNC frequency of the second device; and The method of claim 1 , wherein the first device or the second device adjusts the VSYNC frequency of the corresponding device by one or more frame periods to compensate for the drift.

12. A computer-readable medium storing a computer program for performing a method, comprising: program instructions for setting a plurality of VSYNC signals to a plurality of VSYNC frequencies in a plurality of devices, wherein corresponding device VSYNC signals of corresponding devices are set to the corresponding device VSYNC frequencies; and A computer-readable medium having program instructions for transmitting a plurality of signals between the plurality of devices, the plurality of signals being analyzed and used to adjust the relative timing between corresponding device VSYNC signals of at least two devices.

13. having program instructions for setting a server VSYNC signal at a server to a server VSYNC frequency, the server VSYNC signal corresponding to the generation of a plurality of video frames of an application at the server during a plurality of frame periods of the server VSYNC frequency; having program instructions for transmitting a plurality of compressed video frames based on a video frame from the server to the plurality of devices over a network using the server VSYNC signal; and each of the plurality of devices further comprising program instructions for decoding and displaying the plurality of compressed video frames; The computer-readable medium of claim 12 , wherein the display of the plurality of compressed video frames on the plurality of devices is synchronized.

14. wherein each of the plurality of devices is a client device; The computer-readable medium of claim 13 , wherein the method comprises each of the plurality of devices configured in a server and client architecture.

15. program instructions for generating corresponding plurality of video frames of an application on corresponding devices of the plurality of devices; and each corresponding device of the plurality of devices further comprising program instructions for displaying the corresponding plurality of video frames; The computer-readable medium of claim 12 , wherein the display of corresponding video frames on the multiple devices is synchronized.

16. wherein each of the plurality of devices is an independent peer device; 16. The computer-readable medium of claim 15, wherein the method comprises the plurality of devices arranged in a peer-to-peer configuration for peer-to-peer gaming.

17. 13. The computer-readable medium of claim 12, wherein the plurality of signals are analyzed to synchronize device VSYNC signals of a plurality of devices.

18. 1. A computer system comprising: a processor; a memory coupled to the processor and storing instructions that, when executed by the computer system, cause the computer system to perform a method, the method comprising: In a plurality of devices, setting a plurality of VSYNC signals to a plurality of VSYNC frequencies, and setting corresponding device VSYNC signals of corresponding devices to the corresponding device VSYNC frequencies; A computer system transmitting a plurality of signals between the plurality of devices, the plurality of signals being analyzed and used to adjust the relative timing between VSYNC signals of corresponding ones of at least two devices.

19. The method further comprises: setting a server VSYNC signal at a server VSYNC frequency, the server VSYNC signal corresponding to generating a plurality of video frames of an application at the server during a plurality of frame periods of the server VSYNC frequency; transmitting a plurality of compressed video frames based on the video frame from the server to the plurality of devices over a network using the server VSYNC signal; and each of the plurality of devices decoding and displaying the plurality of compressed video frames; 20. The computer system of claim 18, wherein the display of the plurality of compressed video frames on the plurality of devices is synchronized.

20. wherein each of the plurality of devices is a client device; 20. The computer system of claim 19, wherein the method comprises each of the plurality of devices configured in a server and client architecture.

21. further generating a plurality of corresponding video frames of the application on corresponding devices of the plurality of devices; and a corresponding device of each of the plurality of devices displays the corresponding plurality of video frames; 20. The computer system of claim 18, wherein the display of corresponding video frames on the multiple devices is synchronized.

22. wherein each of the plurality of devices is an independent peer device; 22. The computer system of claim 21, wherein the method comprises the plurality of devices arranged in a peer-to-peer configuration for peer-to-peer gaming.

23. 20. The computer system of claim 18, wherein the plurality of signals are analyzed to synchronize device VSYNC signals of the plurality of devices.

Images