Method for cloud gaming, computer-readable medium, and computer system

Abstract

To substantially reduce round-trip latency and one-way latency between a server and a client.SOLUTION: A method comprises: generating a video frame when executing a video game at a server and scanning-in the video frame to an encoder scanline-by-scanline; encoding one or more encoder slices as one or more encoded slices at the encoder, where each encoded slice is delivered to a buffer at an encoder fill rate; and draining a first encoded slice from the buffer to stream the first encoded slice at a buffer drain rate over a network to a client, where the first encoded slice is streamed over the network before fully encoding the one or more slices of the video frame.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present disclosure relates to streaming systems configured to stream content across a network, and more specifically, to overlapping operations at cloud gaming servers and / or clients to reduce latency between the cloud gaming servers and clients. [Background technology]

[0002] In recent years, there has been a continuing pursuit for online services that enable online gaming or cloud gaming in a streaming format between cloud gaming servers and clients connected through a network. The streaming format has become increasingly more popular due to the availability of game titles on demand, the ability to network between players for multiplayer gaming, sharing of assets between players, sharing of instantaneous experiences between players and / or spectators, the ability for friends to watch friends play video games and to participate in the ongoing gameplay of friends, etc. Unfortunately, there are also increasing demands on the capacity of network connections and limitations of the processing performed on the server and the client to be responsive enough to render high quality images as delivered to the client. For example, the results of all gaming activities performed on the server need to be compressed and transmitted back to the client at low millisecond latency for the best user experience. Round-trip latency may be defined as the total time between the user's controller input and the display of a video frame at the client, which may include processing and transmitting control information from the controller to the client, processing and transmitting control information from the client to the server, using that input at the server to generate a video frame in response to the input, processing the video frame and transferring the video frame to a coding unit (e.g., scanning out), encoding the video frame, transmitting the encoded video frame back to the client, receiving and decoding the video frame, and any processing or staging of the video frame prior to its display. One-way latency may be defined as a portion of round-trip latency consisting of the time from the start of transfer (e.g., scan-out) of a video frame to an encoding unit at the server to the start of display of the video frame at the client. A portion of the round-trip latency and one-way latency is associated with the time it takes for a data stream to be transmitted from the client to the server and from the server to the client over a communication network. Another portion is associated with processing at the client and server, and improvements in their operation, such as advanced strategies related to frame decoding and display, may result in a substantial reduction in the round-trip latency and one-way latency between the server and the client, resulting in a higher quality experience for users of the cloud gaming service.

[0003] It is against this background that embodiments of the present disclosure have been made. Summary of the Invention

[0004] Embodiments of the present disclosure relate to streaming systems configured to stream content (e.g., gaming) across a network, and more specifically to overlapping encoding and transmitting operations at a server, overlapping receiving and decoding operations at a client, and overlapping receiving, decoding, and rendering operations with display operations at a client.

[0005] An embodiment of the present disclosure discloses a method for cloud gaming. The method includes receiving one or more encoded slices of a video frame at a client, the video frame being generated at a server during execution of a video game and encoded into one or more encoded slices by an encoder at the server. The method includes decoding a first encoded slice at a decoder at the client before completely receiving the one or more encoded slices of the video frame. At the server, a plurality of video frames are generated in response to processing of the video game by the server in a streaming mode during a session with the client. The plurality of video frames are encoded at the encoder into a plurality of compressed video frames, each corresponding compressed video frame including one or more corresponding encoded slices. The plurality of compressed video frames are transmitted to the client over a network.

[0006] In another embodiment, a method for cloud gaming is disclosed. The method includes generating video frames when executing a video game at a server. The method includes scanning the video frames into an encoder per server scanline. The method includes encoding one or more scanlines at the encoder as one or more encoded slices, each encoded slice being delivered to a buffer at an encoder fill rate. The method includes draining a first encoded slice from the buffer to stream the first encoded slice over a network to a client at a buffer drain rate, the first encoded slice being streamed over the network before fully encoding the one or more slices of the video frame.

[0007] In another embodiment, a non-transitory computer-readable medium having stored thereon a computer program for cloud gaming is disclosed, the computer-readable medium including program instructions for receiving at a client one or more encoded slices of video frames, the video frames being generated at a server during execution of a video game and encoded into the one or more encoded slices by an encoder at the server. The computer-readable medium includes program instructions for decoding a first encoded slice at a decoder of the client prior to completely receiving one or more encoded slices of the video frame. At the server, a plurality of video frames are generated in response to processing of the video game by the server in a streaming mode during a session with the client. The plurality of video frames are encoded at an encoder into a plurality of compressed video frames, each corresponding compressed video frame including one or more corresponding encoded slices. The plurality of compressed video frames are transmitted to the client over a network.

[0008] In another embodiment, a non-transitory computer-readable medium having stored thereon a computer program for cloud gaming is disclosed. The computer-readable medium includes program instructions for generating video frames when executing a video game at a server. The computer-readable medium includes program instructions for scanning in the video frames to an encoder per server scan line. The computer-readable medium includes program instructions for encoding one or more encoder slices as one or more encoded slices at the encoder, each encoded slice being delivered to a buffer at an encoder fill rate. The computer-readable medium includes program instructions for draining a first encoded slice from a buffer to stream the first encoded slice over a network to a client at a buffer drain rate, the first encoded slice being streamed over a network before fully encoding the one or more slices of the video frame.

[0009] In yet another embodiment, a computer system includes a processor and a memory coupled to the processor and having instructions stored thereon, the instructions, when executed by the computer system, causing the computer system to perform a method for cloud gaming. The method includes receiving one or more encoded slices of a video frame at a client, the frames being generated at a server during execution of a video game and encoded into the one or more encoded slices by an encoder at the server. The method includes decoding a first encoded slice at a decoder at the client prior to fully receiving the one or more encoded slices of the video frame. At the server, a plurality of video frames are generated in response to processing of the video game by the server in a streaming mode during a session with the client. The plurality of video frames are encoded at the encoder into a plurality of compressed video frames, each corresponding compressed video frame including one or more corresponding encoded slices. The plurality of compressed video frames are transmitted over a network to the client.

[0010] In yet another embodiment, a computer system includes a processor and a memory coupled to the processor and having instructions stored thereon, the instructions, when executed by the computer system, causing the computer system to perform a method for cloud gaming. The method includes generating video frames when executing a video game at a server. The method includes scanning the video frames into an encoder for each server scanline. The method includes encoding one or more encoder slices at the encoder as one or more encoded slices, each encoded slice being delivered to a buffer at an encoder fill rate. The method includes draining a first encoded slice from the buffer to stream the first encoded slice over a network to a client at a buffer drain rate, the first encoded slice being streamed over the network before fully encoding the one or more slices of the video frame.

[0011] Other aspects of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the present disclosure.

[0012] The present disclosure is best understood by reference to the following detailed description taken in conjunction with the accompanying drawing figures. [Brief description of the drawings]

[0013] [Figure 1A] 2 is a diagram of a VSYNC signal at the start of a frame period according to one embodiment of the present disclosure. [Figure 1B] 4 is a diagram of the frequency of a VSYNC signal according to one embodiment of the present disclosure. [Figure 2A] FIG. 1 is a diagram of a system that provides gaming over a network between one or more cloud gaming servers and one or more client devices in various configurations, and that can synchronize and offset VSYNC signals to reduce one-way latency, in accordance with one embodiment of the present disclosure. [Figure 2B] FIG. 1 is a diagram of providing gaming between two or more peer devices according to one embodiment of the present disclosure, where VSYNC signals can be synchronized and offset to achieve optimal timing of receipt of controller and other information between the devices. [Figure 2C] 1 illustrates various network configurations that benefit from proper synchronization and offset of VSYNC signals between source and target devices, according to one embodiment of the present disclosure. [Figure 2D] 1 illustrates a multi-tenancy configuration between a cloud gaming server and multiple clients that benefits from proper synchronization and offset of VSYNC signals between source and target devices, according to one embodiment of the present disclosure. [Diagram 3] 1 illustrates the variation in one-way latency between a cloud gaming server and a client due to clock drift when streaming video frames generated from a video game running on the server, according to one embodiment of the present disclosure. [Figure 4] 1 illustrates a network configuration including a cloud gaming server and clients in which VSYNC signals between the server and clients are synchronized and offset to allow overlapping of operations at the server and clients and reduce one-way latency between the server and clients when streaming video frames generated from a video game running on the server, according to one embodiment of the present disclosure. [Figure 5A] FIG. 1 illustrates overlapping encoding and sending operations at a server and overlapping receiving and decoding operations at a client when streaming video frames generated from a video game running on the server, according to one embodiment of the disclosure. [Figure 5B]1 is a flowchart illustrating a method for cloud gaming according to one embodiment of the disclosure, in which encoding and transmission operations at a server are overlapped to reduce one-way latency between the server and a client. [Figure 6] FIG. 1 illustrates spacing of encoded slices of video frames based on a client's maximum receiving bandwidth when streaming video frames generated from a video game running on a server, according to one embodiment of the disclosure. [Figure 7A] FIG. 1 illustrates overlapping receiving and decoding operations at a client when streaming video frames generated from a video game running on a server, according to one embodiment of the disclosure. [Figure 7B] FIG. 1 is a flowchart illustrating a method for cloud gaming according to one embodiment of the disclosure, in which when streaming video frames generated from a video game running on a server, receiving and decoding operations are overlapped at the client to reduce one-way latency between the server and the client, and rendering of decompressed video frames can also be overlapped with the receiving and decoding operations. [Figure 8A] FIG. 1 illustrates interleaving forward error correction (FEC) packets within encoded slices of video frames when streaming video frames generated from a video game running on a server, according to one embodiment of the disclosure. [Figure 8B] 1 is a diagram of a packet header of an encoded slice according to one embodiment of the disclosure. [Figure 9] 1 illustrates components of an example device that can be used to implement aspects of various embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Although the following detailed description contains many specific details for purposes of illustration, those skilled in the art will appreciate that many variations and modifications to the following details are within the scope of the disclosure. Accordingly, the aspects of the disclosure described below are presented without loss of generality to, and without imposing limitations on, the claims that follow this description.

[0015] Generally speaking, various embodiments of the present disclosure describe methods and systems configured to reduce latency and / or latency volatility between a source device and a target device when streaming media content (e.g., streaming audio and video from a video game). In particular, some embodiments of the present disclosure can reduce one-way latency in cloud gaming applications when streaming video frames generated from a video game running on a server by overlapping video frame encoding and transmission operations at the server. A server in a cloud gaming application may encode video frames and begin transmitting data from the server to clients as soon as the packets are encoded by the server's encoder to reduce one-way latency. This may require information (e.g., the amount of data being encoded at a given time) to be shared within the server between the encoding unit and those units that perform the transmission of data (e.g., streamers operating at the application layer). In some embodiments, the encoder is tuned to encode packets such that the packet size corresponds to the natural amount of data produced by the encoder. In one example, the packet size may correspond to a network-based maximum transmission unit (MTU). In another example, the packet size may correspond to the MTU and may be limited to the encoded slices generated by the encoder. In yet another example, the packet size may correspond to a scanline or screen slice in one embodiment. In that way, the amount of one-way latency and buffering required is reduced. Additionally, in some embodiments of the present disclosure, when streaming video frames generated from a video game running on a server, one-way latency in cloud gaming applications can be reduced by overlapping the receiving and decoding operations of the video frames at the client. In another embodiment of the present disclosure, one-way latency between the cloud gaming server and the client can be reduced by overlapping the receiving and decoding operations of the compressed video frames and their display. The client in the cloud gaming application is configured to receive the compressed video frames from the cloud gaming server and decode the compressed video frames. By starting the decoding operation at the client as early as possible, one-way latency between the server and the client can be reduced. For example, each packet of an encoded slice may be decoded as it is received by the client from the server. Also, one-way latency can be reduced by starting the display of the video frame before the frame is fully received or decoded at the client. The timing of the submission for display should predict the remaining time required for the receiving and decoding of the compressed video frame. As previously introduced, the packet sizing may correspond to the natural amount of data created by the encoder (e.g., MTU, scanline, MTU-sized encoder slice, etc.) that allows each packet to be decoded as it arrives.

[0016] In particular, latency instability may be introduced between the server and the client due to the additional time required to generate a complex frame (e.g., a scene change) at the server, the increased time to encode / compress the complex frame at the server, the variable communication paths through the network, and the increased time to decode the complex frame at the client. Latency instability may also be introduced due to differences in clocks at the server and the client that cause drift between the server VSYNC signal and the client VSYNC signal. In one embodiment, this latency instability can be eliminated by overlapping the encoding and transmitting operations at the server, overlapping the receiving and decoding operations at the client, and / or overlapping the receiving, decoding, and rendering operations with the display operations at the client.

[0017] With the above general understanding of the various embodiments, details of example embodiments will now be described with reference to the various drawing figures.

[0018] Throughout this specification, references to a "game," "video game," "gaming application," or "application" are meant to represent any type of interactive application that is directed through the execution of input commands. By way of example only, interactive applications include applications for gaming, word processing, video processing, video game processing, and the like. Furthermore, the terms introduced above are interchangeable.

[0019] Cloud gaming involves running a video game at a server to generate game-rendered video frames, which are then transmitted to a client for display. The timing of operations at both the server and the client may be tied to respective vertical synchronization (VSYNC) parameters. When VSYNC signals are appropriately synchronized and / or offset between the server and / or client, operations performed at the server (e.g., generating and transmitting video frames over one or more frame periods) are synchronized with operations performed at the client (e.g., displaying video frames on a display at a display frame or refresh rate corresponding to the frame period). In particular, a server VSYNC signal generated at the server and a client VSYNC signal generated at the client may be used to synchronize operations at the server and the client. That is, when the server VSYNC signal and the client VSYNC signal are synchronized and / or offset, the server generates and transmits video frames in sync with how the client displays those video frames.

[0020] When streaming media content between a server and a client, VSYNC signaling and vertical blanking interval (VBI) have been incorporated to generate video frames and display them. For example, the server will generate a game-rendered video frame within one or several frame periods as defined by the corresponding server VSYNC signal (e.g., if the frame period is 16.7 milliseconds, generating a video frame every frame period will result in 60 Hertz operation, and generating one video frame every two frame periods will result in 30 Hertz operation), and then encode and transmit the video frame to the client. At the client, the received encoded video frame is decoded and displayed, and the client displays each rendered video frame for display starting with the corresponding client VSYNC.

[0021] For illustration, FIG. 1A shows how a VSYNC signal 111 can indicate the start of a frame period during which various operations may be performed at the server and / or client. When streaming media content, the server may use the server VSYNC signal to generate and encode video frames, and the client may use the client VSYNC signal to display the video frames. The VSYNC signal 111 is generated at a defined frequency corresponding to a defined frame period 110, as shown in FIG. 1B. In addition, the VBI 105 defines the time period between when the last raster line is drawn on the display for the previous frame period and when the first raster line (e.g., the top) is drawn on the display. As shown after the VBI 105, the video frame rendered for display is displayed via raster scan lines 106 (e.g., raster line by raster line, from left to right).

[0022] Additionally, various embodiments of the present disclosure are disclosed for reducing one-way latency and / or latency volatility between a source device and a target device, such as when streaming media content (e.g., video game content). For purposes of illustration only, various embodiments for reducing one-way latency and / or latency volatility are described in a server and client network configuration. However, it will be understood that the various disclosed techniques for reducing one-way latency and / or latency volatility may be implemented in other network configurations and / or over peer-to-peer networks, such as those illustrated in Figures 2A-2D. For example, various disclosed embodiments for reducing one-way latency and / or latency volatility may be implemented between one or more of a server and client devices in various configurations (e.g., a server and a client, a server and a server, a server and multiple clients, a server and multiple servers, a client and a client, a client and multiple clients, etc.).

[0023] FIG. 2A is a diagram of a system 200A that provides gaming over a network 250 between one or more cloud gaming networks 290 and / or servers 260 and one or more client devices 210 in various configurations according to one embodiment of the disclosure, where server VSYNC signals and client VSYNC signals may be synchronized and offset, and / or dynamic buffering may be performed on the client, and / or encoding and transmission operations may be superimposed on the server, and / or receiving and decoding operations may be superimposed at the client, and / or decoding and display operations may be superimposed on the client to reduce one-way latency between the server 260 and the client 210. In particular, the system 200A provides gaming over a cloud gaming network 290, according to one embodiment of the disclosure, with the games being executed remotely from the client devices 210 (e.g., thin clients) of the corresponding users playing the games. The system 200A can provide gaming control to one or more users playing one or more games through the cloud gaming network 290 over the network 250 in either a single player mode or a multiplayer mode. In some embodiments, the cloud gaming network 290 may include multiple virtual machines (VMs) running on a hypervisor of a host machine, where one or more virtual machines are configured to execute a game processor module that utilizes hardware resources available to the host's hypervisor. The network 250 may include one or more communication technologies. In some embodiments, the network 250 may include a fifth generation (5G) network technology having an advanced wireless communication system.

[0024] In some embodiments, communication may be facilitated using wireless technology. Such technology may include, for example, 5G wireless communication technology. 5G is the fifth generation of cellular network technology. 5G networks are digital cellular networks in which the service area covered by a provider is divided into small geographic areas called cells. Analog signals representing sound and images are digitized at the telephone, converted by an analog-to-digital converter, and transmitted as a stream of bits. All 5G wireless devices in a cell communicate by radio waves through a local antenna array and a low-power automated transceiver (transmitter and receiver) in the cell, and through frequency channels assigned by the transceiver from a pool of frequencies reused in other cells. The local antennas are connected to the telephone network and the Internet by high-bandwidth optical fiber or wireless backhaul connections. As in other cellular networks, mobile devices crossing from one cell to another are automatically transferred to the new cell. It should be understood that a 5G network is merely an example type of communications network, and the disclosed embodiments may utilize earlier generation wireless or wired communications, as well as later generation wired or wireless technologies that come after 5G.

[0025] As shown, the cloud gaming network 290 includes a game server 260 that provides access to multiple video games. The game server 260 may be any type of server computing device available in the cloud and may be configured as one or more virtual machines running on one or more hosts. For example, the game server 260 may manage virtual machines supporting game processors that instantiate instances of games for users. Thus, the game processors of the game server 260 associated with the multiple virtual machines are configured to run multiple instances of one or more games associated with the gameplay of multiple users. In that way, the back-end server support provides streaming of media (e.g., video, audio, etc.) of gameplay of multiple gaming applications to multiple corresponding users. That is, the game servers 260 are configured to stream data (e.g., rendered images and / or frames of corresponding game play) back to the corresponding client devices 210 over the network 250. In that manner, computationally complex gaming applications may be running on the backend servers in response to controller inputs received and forwarded by the client devices 210. Each server may render images and / or frames that are then encoded (e.g., compressed) and streamed to the corresponding client devices for display.

[0026] For example, multiple users may access cloud gaming network 290 over communication network 250 using corresponding client devices 210 configured to receive streaming media. In one embodiment, client devices 210 may be configured as thin clients that provide for interacting with backend servers (e.g., game servers 260 of cloud gaming network 290) configured to provide computing functionality (e.g., including game title processing engine 211). In another embodiment, the client device 210 may be configured with a game title processing engine and game logic for local processing of at least a portion of a video game, and may further be utilized to receive streaming content, such as generated by a video game executing on a back-end server, or other content provided by a back-end server support. For local processing, the game title processing engine includes basic processor-based functionality for executing the video game and services associated with the video game. The game logic is stored locally on the client device 210 and is used to execute the video game.

[0027] In particular, a client device 210 of a corresponding user (not shown) requests access to a game through a communications network 250, such as the Internet, and is configured to render to a display image generated by the video game executed by a game server 260, where the encoded image is delivered to the client device 210 for display in collaboration with the corresponding user. For example, a user may be interacting with an instance of a video game executing on a game processor of the game server 260 through the client device 210. More particularly, the instance of the video game is executed by a game title processing engine 211. Corresponding game logic (e.g., executable code) 215 implementing the video game is stored and accessible through a data store (not shown) and used to execute the video game. The game title processing engine 211 is capable of supporting multiple video games using multiple game logics, each of which is selectable by a user.

[0028] For example, the client device 210 is configured to interact with the game title processing engine 211 in cooperation with a corresponding user's game play, such as through input commands used to drive the game play. In particular, the client device 210 may receive input from various types of input devices, such as a game controller, a tablet computer, a keyboard, gestures captured by a video camera, a mouse, a touchpad, etc. The client device 210 may be any type of computing device having at least a memory and a processor module with the ability to connect to a game server 260 over a network 250. The back-end game title processing engine 211 is configured to generate rendered images, which are distributed over the network 250 for display on a corresponding display in cooperation with the client device 210. For example, game rendered images may be delivered by a corresponding game instance executing on a game executing engine 211 of game server 260 through a cloud-based service. That is, client device 210 is configured to receive encoded images (e.g., encoded from game rendered images generated through execution of a video game) and display the rendered images on display 11. In one embodiment, display 11 includes an HMD (e.g., displaying VR content). In some embodiments, the rendered images may be streamed wirelessly or wired to a smartphone or tablet directly from the cloud-based service or via client device 210 (e.g., PlayStation® Remote Play).

[0029] In one embodiment, the game server 260 and / or the game title processing engine 211 include basic processor-based functionality for executing services associated with games and gaming applications. For example, the processor-based functionality includes 2D or 3D rendering, physics, physics simulation, scripting, audio, animation, graphics processing, lighting, shading, rasterization, ray tracing, shadowing, culling, transformation, artificial intelligence, etc. Additionally, services for gaming applications include memory management, multi-thread management, quality of service (QoS), bandwidth checking, social networking, management of social friends, communication with social networks of friends, communication channels, texting, instant messaging, chat support, etc.

[0030] In one embodiment, the cloud gaming network 290 is a distributed game server system and / or architecture. In particular, a distributed game engine that executes game logic is configured as a corresponding instance of a corresponding game. In general, the distributed game engine takes each of the game engine's functions and distributes them for execution by multiple processing entities. Individual functions may be further distributed across one or more processing entities. The processing entities may include physical hardware and / or be configured in different configurations as virtual components or virtual machines and / or as virtual containers, where a container differs from a virtual machine as it virtualizes an instance of a gaming application running on a virtualized operating system. The processing entities may utilize and / or depend on the servers of the cloud gaming network 290 and their underlying hardware (computing nodes) on one or more servers, which may be located on one or more racks. The coordination, allocation, and management of the execution of their functions to the various processing entities is performed by a distributed synchronization layer. As such, the execution of their functions is controlled by the distributed synchronization layer to enable the generation of media (e.g., video frames, sounds, etc.) for the gaming application in response to controller inputs by the player. The distributed synchronization layer enables the efficient execution of those functions (e.g., through load balancing) across the distributed processing entities, so that critical game engine components / functions are distributed and reassembled for more efficient processing.

[0031] The game title processing engine 211 includes a group of central processing units (CPUs) and graphics processing units (GPUs) that may be configured to perform multi-tenancy GPU functionality. In another embodiment, multiple GPU devices are combined to perform graphics processing for a single application running on a corresponding CPU.

[0032] 2B is a diagram of providing gaming between two or more peer devices, where VSYNC signals can be synchronized and offset to achieve optimal timing of receipt of controller and other information between the devices, according to one embodiment of the disclosure. For example, head-to-head gaming may be performed using two or more peer devices connected directly through a network 250 or through peer-to-peer communications (e.g., Bluetooth, local area networking, etc.).

[0033] As shown, a video game is executed locally on each of the client devices 210 (e.g., game consoles) of corresponding users playing the game, and the client devices 210 communicate through peer-to-peer networking. For example, an instance of the video game is executed by a game title processing engine 211 of the corresponding client device 210. Game logic (e.g., executable code) implementing the video game is stored on the corresponding client device 210 and used to execute the game. By way of example, the game logic 215 may be distributed to the corresponding client device 210 through a portable medium (e.g., optical medium) or over a network (e.g., downloaded from a gaming provider over the Internet).

[0034] In one embodiment, the game title processing engine 211 of the corresponding client device 210 includes basic processor-based functionality for executing games and services associated with gaming applications. For example, the processor-based functionality includes 2D or 3D rendering, physics, physics simulation, scripting, audio, animation, graphics processing, lighting, shading, rasterization, ray tracing, shadowing, culling, transformation, artificial intelligence, etc. Additionally, services for gaming applications include memory management, multi-thread management, quality of service (QoS), bandwidth checking, social networking, management of social friends, communication with social networks of friends, communication channels, texting, instant messaging, chat support, etc.

[0035] Client device 210 may receive input from various types of input devices, such as a game controller, a tablet computer, a keyboard, gestures captured by a video camera, a mouse, a touchpad, etc. Client device 210 may be any type of computing device having at least a memory and a processor module, and is configured to generate rendered images that are executed by game title processing engine 211 and display the rendered images on a display (e.g., display 11, or display 11 including a head mounted display - HMD, etc.). For example, the rendered image may be associated with an instance of a game running locally on client device 210 to implement the gameplay of a corresponding user, such as through input commands used to drive the gameplay. Some examples of client device 210 include a personal computer (PC), a game console, a home theater device, a general purpose computer, a mobile computing device, a tablet, a phone, or any other type of computing device capable of running an instance of a game.

[0036] 2C illustrates various network configurations that would benefit from proper synchronization and offset of VSYNC signals between source and target devices, including those configurations illustrated in FIGs. 2A-2B, in accordance with an embodiment of the present disclosure. In particular, various network configurations would benefit from proper alignment of the frequency of the server VSYNC signal and the client VSYNC signal, as well as proper alignment of the timing offset of the server VSYNC signal and the client VSYNC signal, in order to reduce one-way latency and / or latency variability between the server and the client. For example, one network device configuration includes a cloud gaming server (e.g., source) to client (target) configuration. In one embodiment, the client may include a WebRTC client configured to provide voice and video communications within a web browser. Another network configuration includes a client (e.g., source) to server (target) configuration. Yet another network configuration includes a server (e.g., source) to server (e.g., target) configuration. Another network device configuration includes a client (e.g., source) to client (target) configuration, where each client may be, for example, a gaming console providing head-to-head gaming.

[0037] In particular, alignment of the VSYNC signals may include synchronizing the frequency of the server VSYNC signal and the client VSYNC signal, and may also include adjusting the timing offset between the client VSYNC signal and the server VSYNC signal to maintain an ideal relationship between the server VSYNC signal and the client VSYNC signal in order to eliminate drift and / or reduce one-way latency and / or latency variability. To achieve proper alignment, in one embodiment, the server VSYNC signal may be tuned to implement proper alignment between the server 260 and client 210 pair. In another embodiment, the client VSYNC signals may be synchronized to implement proper alignment between a server 260 and client 210 pair. When the client and server VSYNC signals are aligned, the server and client VSYNC signals occur at substantially the same frequency and are offset from each other by a timing offset that can be adjusted from time to time. In another embodiment, alignment of the VSYNC signals may include synchronizing the VSYNC frequency for two clients and may also include adjusting the timing offset between those VSYNC signals to eliminate drift and / or achieve optimal timing of receipt of controller and other information, and either VSYNC signal may be synchronized to achieve this alignment. In yet another embodiment, alignment may include synchronizing the VSYNC frequency for multiple servers and may also include synchronizing the frequency of the server and client VSYNC signals and adjusting a timing offset between the client and server VSYNC signals, for example for head-to-head cloud gaming. In server-to-client and client-to-client configurations, alignment may include both synchronizing the frequency between the server and client VSYNC signals as well as providing an appropriate timing offset between the server and client VSYNC signals. In a server-to-server configuration, alignment may include synchronizing the frequency between the server and client VSYNC signals without setting a timing offset.

[0038] 2D illustrates a multi-tenancy configuration between a cloud gaming server 260 and one or more clients 210 that benefits from proper synchronization and offset of VSYNC signals between source and target devices, according to one embodiment of the present disclosure. In a server-to-client configuration, alignment may include both synchronizing frequency between the server VSYNC signal and the client VSYNC signal as well as providing a proper timing offset between the server VSYNC signal and the client VSYNC signal. In a multi-tenancy configuration, in one embodiment, the client VSYNC signal is tuned at each client 210 to implement proper alignment between the server 260 and client 210 pair.

[0039] For example, in one embodiment, the graphics subsystem may be configured to perform multi-tenancy GPU functionality, where one graphics subsystem implements a graphics and / or rendering pipeline for multiple games, i.e., the graphics subsystem is shared among multiple games being executed. In particular, in one embodiment, the game title processing engine may include a group of CPUs and GPUs configured to perform multi-tenancy GPU functionality, where one CPU and GPU group implements a graphics and / or rendering pipeline for multiple games, i.e., the CPU and GPU group is shared among multiple games being executed. The CPU and GPU group are configured as one or more processing devices. In another embodiment, multiple GPU devices are combined to perform graphics processing for a single application executing on a corresponding CPU.

[0040] 3 illustrates an overall process of executing a video game at a server to generate game-rendered video frames and sending those video frames to a client for display. Conventionally, some operations at the game server 260 and client 210 are performed within a frame period as defined by their respective VSYNC signals. For example, the server 260 attempts to generate a game-rendered video frame within one or more frame periods as defined by a corresponding server VSYNC signal 311 at 301. The video frames are generated by the game in response to either control information (e.g., a user's input command) delivered from an input device at operation 350 or game logic that is not driven by control information. Transmission jitter 351 may be present when sending control information to server 260, where jitter 351 measures the variation in network latency from the client to the server (e.g., when sending an input command). As shown, the thick arrow indicates the current delay in sending control information to server 260, but due to jitter there may be a range of arrival times for the control information at server 260 (e.g., the range bounded by the dotted arrow). At flip time 309, the GPU arrives at the flip command, indicating that the corresponding video frame has been fully generated and placed in the frame buffer at server 260. The server 260 then performs a scan-out / scan-in on the video frame throughout the subsequent frame period as defined by the server VSYNC signal 311 (VBI omitted for clarity) (operation 302, where the scan-out may be aligned with the VSYNC signal 311). The video frame is then encoded (operation 303) (e.g., the encoding starts after the occurrence of the VSYNC signal 311, and the end of the encoding may not be aligned with the VSYNC signal) and transmitted to the client 210 (operation 304, where the transmission may not be aligned with the VSYNC signal 311). At the client 210, the encoded video frame is received (operation 305, where the reception may not be aligned with the client VSYNC signal 312), decoded (operation 306, where the decoding may not be aligned with the client VSYNC signal 312), buffered, and displayed (operation 307, where the start of the display may be aligned with the client VSYNC signal 312). In particular, the client 210 displays each video frame rendered for display beginning with the corresponding occurrence of a client VSYNC signal 312 .

[0041] One-way latency 315 may be defined as the latency from the start of the transfer of a video frame to an encoding unit at the server (e.g., scanout 302) to the start of display of the video frame at the client 307. That is, one-way latency is the time from server scanout to client display, taking into account client buffering. Individual frames have a latency from the start of scanout 302 to the completion of decoding 306 that may vary from frame to frame due to server operations such as encoding 303 and transmission 304, network transmission between the server 260 and client 210 with jitter 352, and high degree of mismatch in client reception 305. As shown, the thick line indicates the current latency in transmitting a video frame corresponding to the client 210, but due to jitter 352, there may be a range of arrival times for the video frame at the client 210 (e.g., the range bounded by the dotted arrow). Because one-way latency needs to be relatively stable (e.g., remain highly consistent) to achieve a good playing experience, buffering 320 is conventionally performed with the result that the display of individual frames having low latency (e.g., from the start of scanout 302 to the completion of decoding 306) is delayed for several frame periods. That is, when there is network instability or unpredictable encoding / decoding times, extra buffering is required so that one-way latency remains consistent.

[0042] One-way latency between a cloud gaming server and a client may vary due to clock drift when streaming video frames generated from a video game running on a server according to one embodiment of the present disclosure. That is, differences in the frequency of the server VSYNC signal 311 and the client VSYNC signal 312 may cause the client VSYNC signal to drift relative to the frames arriving from the server 260. The drift may be due to very slight differences in the crystal oscillators used in each of the respective clocks at the server and the client. Additionally, embodiments of the present disclosure reduce one-way latency by performing one or more of synchronization and offsetting of the VSYNC signals for alignment between the server and the client, providing dynamic buffering on the client, and overlapping the encoding and transmission of video frames at the server, overlapping the reception and decoding of video frames at the client, and overlapping the decoding and display of video frames at the client.

[0043] 4 illustrates the flow of data through a network configuration including a highly optimized cloud gaming server 260 and a highly optimized client 210 when streaming video frames generated from a video game running on the server in accordance with an embodiment of the present disclosure, where overlapping server and client operations reduces one-way latency, and synchronizing and offsetting VSYNC signals between the server and client reduces one-way latency as well as reducing variability in one-way latency between the server and client. In particular, FIG. 4 illustrates the desired alignment between the server VSYNC signal and the client VSYNC signal. In one embodiment, synchronization of the server VSYNC signal 311 is performed to obtain proper alignment between the server VSYNC signal and the client VSYNC signal, such as in a server and client network configuration. In another embodiment, synchronization of the client VSYNC signal 312 is performed to obtain proper alignment between the server VSYNC signal and the client VSYNC signal, such as in a multi-tenant server-to-multi-client network configuration. For illustrative purposes, the synchronization of the server VSYNC signal 311 is described in FIG. 4 for the purpose of synchronizing the frequency of the server VSYNC signal and the client VSYNC signal and / or adjusting the timing offset between the corresponding client VSYNC signal and the server VSYNC signal, but it will be understood that the client VSYNC signal 312 may also be used for synchronization. In the context of this patent application, "synchronize" should be interpreted as synchronizing the signals so that their frequency matches, but the phases may differ, and "offset" should be interpreted as the time delay between the signals, for example, between when one signal reaches its maximum value and when the other signal reaches its maximum value.

[0044] As shown, Figure 4 illustrates an improved process for executing a video game at a server to generate rendered video frames and sending those video frames to a client for display in a disclosed embodiment. Processes for generating and displaying a single video frame at the server and client are shown. In particular, the server generates a game rendered video frame at 401. For example, the server 260 includes a CPU (e.g., game title processing engine 211) configured to execute the game. The CPU generates one or more draw calls for a video frame, the draw calls including commands that are placed in a command buffer for execution by a corresponding GPU of server 260 in a graphics pipeline. The graphics pipeline may include one or more shader programs for vertices of objects in the scene to generate texture values ​​that are rendered for the video frame for display, with operations performed in parallel across the GPU for efficiency. At flip time 409, the GPU arrives at a flip command in the command buffer indicating that the corresponding video frame has been fully generated and / or rendered at server 260 and placed in the frame buffer.

[0045] At 402, the server performs scanout of the game rendered video frame to the encoder. In particular, scanout is performed scanline by scanline or in groups of consecutive scanlines, where a scanline refers to, for example, a single horizontal line of display from screen edge to screen edge. Those scanlines or groups of consecutive scanlines may be referred to as slices, and are referred to herein as screen slices. In particular, scanout 402 may include some processing to modify the game rendered frame, including overlapping it with another frame buffer or shrinking it to surround it with information from another frame buffer. During scanout 402, the modified video frame is then scanned in to the encoder for compression. In one embodiment, scanout 402 is performed at the occurrence 311a of the VSYNC signal 311. In other embodiments, scanout 402 may be performed before the occurrence of the VSYNC signal 311, such as at flip time 409.

[0046] At 403, the game rendered video frames (possibly modified) are encoded in an encoder on an encoder slice by encoder slice basis to generate one or more encoded slices, the encoded slices being unrelated to scanlines or screen slices. Thus, the encoder generates one or more encoded (compressed) slices. In one embodiment, the encoding process begins before the scanout process 402 is fully completed for the corresponding video frame. Furthermore, the start and / or end of encoding 403 may or may not be aligned with the server VSYNC signal 311. The boundaries of the encoded slices are not constrained to a single scanline, but may consist of a single scanline, or multiple scanlines. In addition, the end of an encoded slice and / or the beginning of the next encoder slice may not necessarily occur at the edge of the display screen (e.g., may occur somewhere in the middle of the screen or in the middle of a scanline), such that the encoded slice need not completely traverse from edge to edge of the display screen. As shown, one or more encoded slices may be compressed and / or encoded, including a compressed "encoded slice A" with hash marks.

[0047] At 404, the encoded video frame is transmitted from the server to the client, where the transmission may occur on a per-encoded slice basis, with each encoded slice being a compressed encoder slice. In one embodiment, the transmission process 404 begins before the encoding process 403 is fully completed for the corresponding video frame. Furthermore, the start and / or end of the transmission 404 may or may not be aligned with the server VSYNC signal 311. As shown, the compressed encoded slice A is transmitted to the client independently of the other compressed encoder slices for the rendered video frame. The encoder slices may be transmitted at once or in parallel.

[0048] At 405, the client receives the compressed video frames, again on a per-encoded slice basis. Additionally, the start and end of receive 405 may or may not be aligned with the client VSYNC signal 312. As shown, the compressed encoded slice A is received by the client. Transmission jitter 452 may exist between the server 260 and the client 210, where jitter 452 measures the variation in network latency from the server 260 to the client 210. Lower jitter values ​​indicate a more stable connection. As shown, the thick line indicates the current latency in sending a video frame corresponding to the client 210, but due to jitter there may be a range of arrival times for the video frames at the client 210 (e.g., the range bounded by the dotted arrows). The variation in latency may be due to networking issues that introduced latency in transmitting the video frames to the client 210, along with one or more operations at the server, such as encoding 403 and transmitting 404.

[0049] At 406, the client decodes the compressed video frame again on a per-encoded slice basis to produce a decoded slice A (shown without hash marks) that is now ready for display. In one embodiment, the decoding process 406 begins before the receive process 405 is fully completed for the corresponding video frame. Furthermore, the start and / or end of the decoding 406 may or may not be aligned with the client VSYNC signal 312. At 407, the client displays the decoded rendered video frame on a display at the client. That is, the decoded video frame is placed in a display buffer, e.g., to be streamed out on a scanline-by-scanline basis to a display device. In one embodiment, the display process 407 begins (i.e., streams out to the display device) after the decode process 406 is fully completed for the corresponding video frame, i.e., the decoded video frame is fully resident in the display buffer. In another embodiment, the display process 407 begins before the composite process 406 is fully completed for the corresponding video frame. That is, streaming out to the display device begins from an address in the display buffer at a time when only a portion of the decoded frame buffer resides in the display buffer. The display buffer is then updated or filled with the remaining portions of the corresponding video frame in time for display, such that updates to the display buffer are performed prior to streaming out those portions to the display. Additionally, the start and / or end of display 407 is aligned with the client VSYNC signal 312.

[0050] In one embodiment, one-way latency 416 between server 260 and client 210 may be defined as the elapsed time between when scanout 402 begins and when display 407 begins. Embodiments of the present disclosure have the ability to align VSYNC signals between the server and client (e.g., synchronize frequency and adjust offset) to reduce the one-way latency between the server and client and to reduce variability in the one-way latency between the server and client. For example, embodiments of the present disclosure are capable of calculating an optimal adjustment to the offset 430 between the server VSYNC signal 311 and the client VSYNC signal 312, so that even in the event of near worst-case times required for server processing such as encoding 403 and transmitting 404, near worst-case network latency between the server 260 and the client 210, and worst-case client processing such as receiving 405 and decoding 406, a decoded rendered video frame is available in time for display processing 407. That is, it is not necessary to determine the absolute offset between the server VSYNC and the client VSYNC, but rather it is sufficient to adjust the offset such that a decoded rendered video frame is available in time for display processing.

[0051] In particular, the frequencies of the server VSYNC signal 311 and the client VSYNC signal 312 may be aligned through synchronization. The synchronization is achieved through synchronizing the server VSYNC signal 311 or the client VSYNC signal 312. For illustrative purposes, the synchronizing is described with respect to the server VSYNC signal 311, but it will be understood that the synchronizing may instead be performed on the client VSYNC signal 312. For example, as shown in Figure 4, the server frame period 410 (e.g., the time between two occurrences 311c and 311d of the server VSYNC signal 311) is substantially equal to the client frame period 415 (e.g., the time between two occurrences 312a and 312b of the client VSYNC signal 312), which indicates that the frequencies of the server VSYNC signal 311 and the client VSYNC signal 312 are also substantially equal.

[0052] To maintain synchronization of the frequencies of the server VSYNC signal and the client VSYNC signal, the timing of the server VSYNC signal 311 may be manipulated. For example, the vertical blanking interval (VBI) in the server VSYNC signal 311 may be increased or decreased over a period of time, such as causing drift between the server VSYNC signal 311 and the client VSYNC signal 312. Manipulation of the vertical blanking interval (VBLANK) lines in the VBI results in adjusting the number of scan lines used for VBLANK during one or more frame periods of the server VSYNC signal 311. Decreasing the number of scan lines of VBLANK reduces the corresponding frame period (e.g., time interval) between two occurrences of the server VSYNC signal 311. Conversely, increasing the number of scan lines of VBLANK increases the corresponding frame period (e.g., time interval) between two occurrences of the VSYNC signal 311. As such, the frequency of the server VSYNC signal 311 is adjusted so that the frequencies between the client VSYNC signal 311 and the server VSYNC signal 312 are matched to be substantially the same frequency. Also, the offset between the server VSYNC signal and the client VSYNC signal may be adjusted by increasing or decreasing the VBI for a short period of time before returning the VBI to its original value. In one embodiment, the server VBI is adjusted. In another embodiment, the client VBI is adjusted. In yet another embodiment, instead of two devices (a server and a client), there may be multiple connected devices, each of which may have a corresponding VBI that is adjusted. In one embodiment, each of the multiple connected devices may be an independent peer device (e.g., with no server device). In another embodiment, the multiple devices may include one or more server devices and / or one or more client devices arranged in one or more server / client architectures, multi-tenant server / client(s) architectures, or some combination thereof.

[0053] Alternatively, in one embodiment, the server's pixel clock (e.g., located in the southbridge of the server's northbridge / southbridge core logic chipset, or in case of a discrete GPU, which generates the pixel clock by itself using its own hardware) may be manipulated to perform coarse and / or fine tuning of the frequency of the server VSYNC signal 311 over a period of time to bring the frequency synchronization between the server VSYNC signal 311 and the client VSYNC signal 312 back into alignment. In particular, the pixel clock in the server's southbridge may be overclocked or underclocked to adjust the overall frequency of the server's VSYNC signal 311. In that way, the frequency of the server VSYNC signal 311 is adjusted to be aligned such that the frequencies between the client VSYNC signal 311 and the server VSYNC signal 312 are substantially the same frequency. The offset between the server VSYNC and the client VSYNC may be adjusted by increasing or decreasing the client-server pixel clock for a short period of time before returning the pixel clock to its original value. In one embodiment, the server pixel clock is adjusted. In another embodiment, the client pixel clock is adjusted. In yet another embodiment, instead of two devices (a server and a client), there may be multiple connected devices, each of which has a corresponding pixel clock that is adjusted. In one embodiment, each of the multiple connected devices may be an independent peer device (e.g., with no server device). In another embodiment, the multiple devices may include one or more server devices and one or more client devices arranged in one or more server / client architectures, a multi-tenant server / client(s) architecture, or some combination thereof.

[0054] 5A is a diagram illustrating overlapping encoding and transmitting operations at the server 260 and overlapping receiving and decoding operations at the client 210 when streaming video frames generated from a video game executing on the server, according to one embodiment of the disclosure. In particular, overlapping the encoding and transmitting operations of video frames at the server 260 and overlapping the receiving and decoding operations of video frames at the client 210 may be performed at the encoder slice level with reference to timeline 501.

[0055] In particular, rendered video frames are generated by a server 260 running a video game. For example, video frames are generated as the video game runs on a CPU in combination with a GPU graphics pipeline of the server 260. Each video frame may be divided into one or more encoder slices, which are then compressed as one or more encoded slices. As previously described, the boundaries of an encoded slice are not constrained to a single scanline, but may consist of a single scanline, or multiple scanlines. Additionally, the end of an encoded slice and / or the beginning of the next encoded slice may not necessarily occur at the edge of a display screen (e.g., may occur somewhere in the middle of the screen or in the middle of a scanline).

[0056] Further, each encoded slice (e.g., compressed) may include one or more packets (e.g., User Datagram Protocol-UDP). If an encoded slice exceeds a maximum transmission unit (MTU), the encoded slice is split into multiple packets, each of which is sized at or smaller than the MTU. The MTU defines the maximum size of a packet when transmitted over UDP. For example, encoded slice A may be split into one or more packets 503-A. Similarly, encoded slice B, encoded slice C, and encoded slice D may each be split into one or more packets.

[0057] At server 260, one or more packets of an encoded slice may be compressed and delivered separately (possibly grouped by slice) without waiting for all encoder slices (e.g., of a video frame) to be compressed (e.g., into encoded slices) and packetized. This may require information (e.g., the amount of data encoded at a given time) to be shared within server 260, such as between the encoding units and those units that perform the transmission of data (e.g., streamers operating at the application layer). In particular, in operation 403, the encoder receives the generated video frame and encodes the video frame on an encoder slice-by-encoder slice basis to generate one or more encoded slices. In some embodiments, the encoder receives the generated video frame (e.g., during a scan-out process) modified by an additional layer or overlay as previously described. As shown, each of the encoded slices of the generated video frame is compressed separately in the encoder. For example, in the encoding process of operation 403, the generated video frame is encoded into encoded slice A (e.g., compressed), encoded slice B, encoded slice C, and encoded slice D. In some embodiments, depending on the number of processors or hardware, multiple slices may be encoded simultaneously (e.g., in parallel). For example, some game consoles may generate four encoded slices in parallel. In more detail, the hardware encoder may be configured to compress multiple encoder slices in parallel (e.g., to generate one or more encoded slices) due to hardware pipelining.

[0058] In operation 404, the encoded slices are transmitted to the client 210 over the network. In particular, the encoded slices are delivered from the encoder to a streamer device, which is configured to transmit the encoded slices to the client 210 over the network. For example, the streamer device is configured to operate at an application layer of the Transmission Control Protocol / Internet Protocol or UDP / IP computer networking model. As shown, according to one embodiment of the present disclosure, the encoding and transmission processes may be overlapped. For example, each encoded slice (e.g., compressed) is transmitted separately such that one or more packets of the corresponding encoded slice are transmitted as a group before the packets of the next encoded slice can be transmitted. In particular, once compressed, the corresponding encoded slice (e.g., one or more packets of the encoded slice) is then ready for transmission by the streamer in operation 404, without waiting for all encoder slices to be compressed for the corresponding video frame. As shown, in timeline 501, encoded slice A is transmitted by the streamer while encoder slice B is being compressed by the encoder (e.g., into encoded slice B) and before encoder slices C and D begin the encoding process. More specifically, before encoder slice B is compressed by the encoder, packets of encoded slice A are transmitted by the streamer over the network to client 210. Similarly, encoded slice B is transmitted by the streamer while encoder slice C is being compressed by the encoder and before encoder slice D starts the encoding process. In cases where multiple slices can be encoded simultaneously (e.g., X number of slices encoded in parallel), X number of encoded slices may be transmitted while the next X number of encoder slices are being encoded (e.g., compressed). Thus, in embodiments of the present disclosure, transmission of data (e.g., encoded slices and / or corresponding encoded slice packets) from the server to the client may begin as soon as the corresponding encoder slices and / or encoder slice packets are encoded and / or compressed, and before a complete video frame is encoded (as is customarily done). In that way, one-way latency between the server and the client may be reduced by beginning transmission of data from the server to the client as soon as a corresponding number of packets of data are encoded by the server.

[0059] At the client 210, the process includes receiving a compressed video frame (e.g., an encoded slice of a video frame) and then decoding the compressed video frame. At the client, the receiving and decoding processes may also be performed at the encoded slice level, such that when one or more packets of a corresponding encoded slice are received, the encoded slice is then decoded using those one or more packets. Thus, the receiving and decoding processes may be overlapped at the client. In particular, at the client 210, one or more encoded slices of a corresponding video frame are received separately in operation 405. For illustrative purposes, in the timeline 501, encoded slice A may be received first, then encoded slice B, and so on, until encoded slice D is finally received.

[0060] In particular, the received encoded slices are then decoded in operation 406. In particular, once received, without waiting for all encoded slices to be received for a corresponding video frame, the received corresponding encoded slices are then ready to be decoded by the decoder in operation 406. For example, in timeline 501, one or more packets of encoded slice A are decoded while encoded slice B is being received and while encoded slice C and encoded slice D are still in transmission over the network. Similarly, one or more packets of Coded Slice B are decoded while Coded Slice C is being received and while Coded Slice D is still in transmission over the network. Thus, in embodiments of the present disclosure, decoding of data (e.g., coded slices and / or coded slice packets) from the server to the client may begin as soon as coded slices and / or coded slice packets are received and before a full complement of coded slices and / or their corresponding packets for a video frame are received. In that way, one-way latency between the server 260 and the client 210 may be reduced by beginning decoding of data as soon as a corresponding number of packets of data are received by the client 210.

[0061] It will be appreciated that the encoded slices of the corresponding video frames may be received at the client in various random orders (e.g., due to network jitter) or in parallel depending on the hardware configuration at the client. That is, in operation 405, the transmitted encoded slices may be received at the client 210 out of order due to jitter. In that case, it may be possible to decode the encoded slices out of order (e.g., receive the encoded slices of the corresponding video frames out of order) to account for the jitter. For example, in a sequence of encoded slices for corresponding video frames transmitted over a network, when one encoded slice is received out of order, the decoding is also performed out of order. That is, encoding is still performed as each encoded slice is received.

[0062] More specifically, each of the encoded slices of the corresponding video frame may be split into one or more network packets at the encoder of the server 260, and the one or more network packets are then transmitted over the network to the client 210. Thus, the client 210 receives one or more packets for each of the one or more slices of the corresponding video frame before a slice of the next video frame and / or corresponding packets of the slice are received. That is, in one embodiment, one or more encoded slices for a corresponding video frame may be received at the client one at a time or in parallel and before one or more encoded slices of the next video frame are received. More specifically, packets for one or more encoded slices of a corresponding video frame may be received at the client one at a time or in parallel and before one or more encoded slices of the next video frame are received.

[0063] With detailed descriptions of the various client devices 210 and / or cloud gaming network 290 (e.g., in game server 260) in Figures 2A-2D, flowchart 500B in Figure 5B illustrates a method for cloud gaming in which server operations can be overlapped to reduce one-way latency between the cloud gaming server and clients, according to one embodiment of the present disclosure.

[0064] At 510, the method includes generating video frames when executing the video game at the server. For example, the server may be executing the video game in a streaming mode, such that a CPU of the server executes the video game in response to input commands from a user to generate game-rendered video frames using a graphics pipeline that can be used for streaming. In particular, the CPU executing the video game in cooperation with the GPU graphics pipeline is configured to generate a plurality of video frames. In cloud gaming, the game-generated video frames are typically rendered for display on a virtual display. The server may perform additional operations on the game-generated video frames. For example, one or more overlays may be added to the corresponding game-generated video frames, such as during a scanout process.

[0065] At 520, the method includes scanning in the game generated video frames, scanline by scanline, into an encoder at the server. As previously described, the game generated video frames may be modified during the scan-out process, such as by adding one or more overlays. Thus, optionally, the modified game generated video frames are scanned in to the encoder for compression in preparation for streaming the video frames to the client.

[0066] At 530, the method includes encoding one or more encoder slices for a corresponding video frame as one or more encoded slices, i.e., the video frame is compressed into a compressed video frame that includes one or more encoded slices. As previously described, each encoded slice may be arranged into one or more scanlines, the minimum size of an encoded slice is one scanline, and the end of an encoded slice may not necessarily occur at an edge of a display screen.

[0067] After encoding, one or more encoded slices are delivered to a buffer. The speed at which the buffer is filled depends on the capabilities of the encoder, such that each encoded slice is delivered to the buffer at the encoder fill rate. For example, the buffer may be filled as fast as the encoder is able to encode video frames, or more specifically, as fast as the encoder is able to generate encoded slices and / or packets of encoded slices.

[0068] In addition, one or more forward error correction (FEC) slices may be generated for a corresponding video frame being compressed by the encoder. The one or more FEC slices may be used by the client to correct any coded slices lost during transmission to the client. In one embodiment, the one or more FEC slices may be interleaved within the one or more coded slices during streaming. That is, for a corresponding video frame having multiple coded slices and multiple FEC slices, the FEC slices may be interleaved within the coded slices during streaming. For example, an FEC slice used to correct a set of coded slices may be delivered immediately after each of the coded slices in the set has been delivered.

[0069] At 540, the method includes draining the first encoded slice from the buffer to stream the first encoded slice over the network to a client at a buffer drain rate. That is, the first encoded slice is delivered over the network to a client configured to receive and decompress the compressed video frame for rendering to a display. More specifically, one or more packets of the first encoded slice are drained from the buffer at the drain rate for streaming. In one embodiment, at least one encoded slice of a video frame is drained before the entire video frame is encoded. In one embodiment, the first encoded slice is streamed over the network before fully encoding one or more slices of the corresponding video frame.

[0070] In one embodiment, the drain rate is dynamically adjusted to be approximately equal to the client's measured maximum receiving bandwidth. In particular, the transmission of the encoded slices of the compressed video frames is performed by spacing the encoded slices according to the client's measured maximum receiving bandwidth. In this way, in one embodiment, the buffer drain rate is decoupled from the encoder fill rate. In particular, FIG. 6 illustrates spacing the encoded slices of the video frames when streaming video frames generated from a video game running on a server according to one embodiment of the disclosure, the spacing being based on the client's 210 maximum receiving bandwidth.

[0071] In one embodiment, to reduce buffering and latency, it is beneficial to tune the Encoder / Decoder (Codec) 610 or codec utilization so that the size of a packet for a corresponding coded slice corresponds to the natural amount of data created by the codec. A coded slice (e.g., compressed) may include one or more packets, which are then delivered to a client over a network. In one embodiment, the packet size is tied to a maximum transmission unit (MTU). When a compressed (coded slice) video encoder slice exceeds the MTU, the coded slice is split into several packets, each of which is sized to be equal to or smaller than the MTU.

[0072] A sequence of encoded slices larger than the MTU is an increased packet loss rate and amount of visual corruption caused when packet loss occurs. For example, if a compressed encoder slice is split into three packets during encoding and one of the packets is lost on the way to the client, the entire encoder slice becomes invalid at the client, so that losing a single packet effectively results in the loss of three packets. Thus, in one embodiment, the packet size may be tuned to the natural amount of data (e.g., MTU) created by the codec. In one implementation, the packet size is tuned to correspond to the UDP packet size. In another implementation, the packet size may be tuned to correspond to the screen slices (e.g., from edge to edge of the screen), and the packet size and the coded slices correspond to the MTU. That is, the coded slices may be sized to the MTU and delivered as one packet. This can increase the number of coded slices of the corresponding video frame. When the packet size is tuned to correspond to something natural in the codec on the server side (e.g., MTU size and / or coded slices), it allows the packets to be decoded on the client side as soon as each of them arrives.

[0073] In one embodiment, compressed video frames are transmitted from the server to the client at a rate based on the maximum available bitrate or bandwidth through the network at a particular point in time. That is, the rate of transmission of encoded slices of compressed video frames is dynamically adjusted according to the currently measured maximum bandwidth. Thus, packets 655 of encoded slices 650 are spaced for delivery to the client according to the maximum available bitrate or maximum bandwidth. That is, instead of spacing packet delivery according to the bitrate of the codec (e.g., concatenating the encoder fill rate 615 with the buffer drain rate 635), packets 655 are delivered according to the maximum bandwidth of the client 210 (with the buffer drain rate 635 tied to the client's maximum bandwidth) (e.g., decoupling the encoder fill rate 615 from the buffer drain rate 635). In that way, packets 655 can be received at the client 210 at a higher rate. In one embodiment, the encoding of packets is performed at the same rate at which they are transmitted, so that both operations are dynamically tuned to the maximum available bandwidth available to the client.

[0074] Additionally, at the server, the additional information may include a maximum transmission rate that defines the total network throughput (e.g., bandwidth) that the server considers available to the client. This maximum transmission rate may be determined by the server by tracking the amount of bandwidth sent to the client for a range of packets combined with messages from the client to the server indicating how much packet loss occurred for the same range of packets. This may be used to determine the maximum rate at which encoded slices of a video frame can be sent out. The maximum rate may vary based on the stability of the network connection to the client, and an offset may be dynamically adjusted to accommodate the variations. Furthermore, the maximum transmission rate may be adjusted independently of the encoder parameters, so that slices can be sent out more quickly if the encoder is configured not to create slices at the maximum transmission rate.

[0075] In one embodiment, the maximum bandwidth at the client is determined by a feedback mechanism from the client. In particular, the streamer 620 operating at the application layer measures the maximum bandwidth of the client using, for example, a bandwidth tester 625. As previously explained, the application layer is used in the UDP suite of protocols used to interconnect network devices over the Internet. For example, the application layer defines the communication protocols and interface methods used to communicate between devices over an IP communication network. During the test, the streamer 620 provides special FEC packets 640 so that the buffer 630 (e.g., a first-in / first-out FIFO buffer) can stream packets at a predefined bit rate, such as a tested maximum bandwidth. In one embodiment, the client returns to the streamer 620 as feedback 690 a number of packets it received over a range of incremental sequence identifiers (IDs), such as for a range of video frames. For example, the client may report something (e.g., 150 video frames), such as 145 of 150 video frames received for sequence IDs 100-250. Thus, the Streamer 620 at the Server 260 can calculate packet loss and dynamically determine what the maximum bandwidth of a client is at a particular time because the Streamer 620 knows the amount of bandwidth that was being transmitted during that sequence of packets (e.g., as tested). The measured maximum bandwidth of a client may be delivered from the Streamer 620 to the Buffer 630 as control information 627, so that the Buffer 630 can dynamically transmit packets at a rate approximately equal to the maximum bandwidth of the client.

[0076] The client 210 is unable to make this determination because the amount of bandwidth transmitted is constantly fluctuating due to variable bitrates, scene complexity, etc. Thus, the client 210 does not know whether the server is transmitting the maximum bandwidth the client can handle at any given moment. For example, the maximum bandwidth may be 15 Mbps (megabits per second), but as a result only 2 Mbps is being transmitted because the scene complexity may be low due to the user being on a menu. Thus, if the client reports 0% packet loss, this does not inform the server 260 whether the client 210 can handle the maximum bandwidth of 15 Mbps. Thus, only when the server 260 is transmitting can the maximum bandwidth be determined as the true maximum bandwidth.

[0077] 7A is a diagram illustrating overlapping receiving and decoding operations at a client when streaming video frames generated from a video game running on a server to reduce one-way latency between the server and the client, according to one embodiment of the disclosure. Additionally, the receiving, decoding, and rendering of decompressed video frames for display may be overlapped at the client to reduce one-way latency between the server and the client, according to one embodiment of the disclosure.

[0078] In particular, the client 210 receives the compressed video frame 705 in operation 405, and a server (not shown) executes a video game to generate game-rendered video frames, which are then encoded in an encoder at the server and delivered to the client as the compressed video frame 705. The compressed video frame 705 includes one or more encoded slices, such as encoded slice A (hashed), as previously described. For example, the compressed video frame 705 includes encoded slices A-G. Each of the encoded slices may be an encoded slice of a different size. For example, encoded slice D is larger than encoded slice E. Furthermore, each of the slices may be divided into one or more packets.

[0079] The client 210 includes a decoder configured to decode one or more encoded slices (e.g., unhashed decoded slice A) of the corresponding compressed video frame in operation 406. In one embodiment, the decoding process begins before the corresponding compressed video frame is completely received at the client. That is, the decoding of the encoded slices for the corresponding video frame begins before all of the encoded slices of the corresponding video frame are received. Because the decoder decodes on an encoded slice-by-encoded slice basis, the decoded video frame 706 includes one or more encoder slices (as decoded). The decoded video frame 706 is then prepared for display, such as rendering the information in the decoded video frame 706 into scanlines or screen slices. Thus, the client rendered video frame 707 is ready for display.

[0080] In a fault-tolerant mode, one-way latency between the server and the client may be reduced by having the client 210 begin displaying a video frame at operation 407 before the video frame is fully decoded at operation 406, according to one embodiment of the present disclosure. In particular, one or more decoded slices of a compressed video frame may be prepared for rendering to the display before the video frame is fully decoded. That is, the display operation at 407 overlaps with the decode operation at 406. In particular, the first encoded slice (e.g., slice A) needs to arrive and be decoded before the client's scanout to the display begins. In addition, all subsequent encoded slices need to arrive and be decoded before their respective decompressed data can be rendered and scanned out for display.

[0081] Furthermore, in addition to overlapping receiving and decoding operations at the client, display of one or more decoded slices that are then rendered in preparation for display may occur before a compressed video frame transmitted by the server is completely received at the client 210. That is, one or more of the receiving, decoding, and display operations at the client may be overlapped with respect to a corresponding video frame.

[0082] Moreover, according to one embodiment of the present disclosure, when multiple operations are overlaid at both the server and the client, one or more decoded slices of a video frame that is then rendered in preparation for display may be displayed at the client even before the scanout operation at the server is fully completed. As previously described, scanout delivers the game-rendered video frames to an encoder at the server, and during the scanout process the game-rendered video frames may optionally be modified (e.g., adding an overlay).

[0083] 2A-2D, flowchart 700B of FIG. 7B illustrates a method for cloud gaming in which client operations can be overlapped to reduce one-way latency between a cloud gaming server and a client, according to one embodiment of the disclosure. In particular, FIG. 7B is a flowchart illustrating overlapping receive and decode operations at a client to reduce one-way latency between a server and a client when streaming video frames generated from a video game running on a server, where rendering of the decompressed video frames may also be optionally overlapped with the receive and decode operations, according to one embodiment of the disclosure.

[0084] At 710, the method includes receiving one or more encoded slices of a compressed video frame at a client. In particular, a plurality of video frames are generated at the server in response to processing of the video game by the server in a streaming mode during a session with the client. The plurality of video frames are encoded at an encoder into a plurality of compressed video frames, each of the plurality of compressed video frames corresponding to a compressed video frame including one or more corresponding encoded slices. The plurality of compressed video frames are transmitted to the client over a network, and more particularly, the encoded slices of the plurality of compressed video frames are transmitted to the client.

[0085] At the client, the method includes receiving a plurality of compressed video frames at a decoder and decompressing the plurality of compressed video frames for rendering to a display associated with the client, where the operations may be overlapped. In particular, at 720, the method includes decoding a first encoded slice at the client prior to completely receiving one or more encoded slices of a corresponding compressed video frame.

[0086] 8A is a diagram illustrating interleaving forward error correction (FEC) packets within encoded slices of video frames when streaming video frames generated from a video game executing on a server, according to one embodiment of the disclosure. Interleaving FEC data between encoded slices of corresponding video frames allows overlapping of receiving, decoding, and rendering operations with display operations at a client, which may be performed on a per-encoded-slice basis. In particular, forward error correction provides for reconstruction of lost encoded slices of corresponding video frames and / or reconstruction of lost packets of encoded slices. Thus, forward error correction allows display of received, decoded, and rendered encoded slices for display at a client before one or more subsequent encoded slices are received and / or decoded.

[0087] For example, an encoded slice may be partitioned into one or more network packets. FEC packets may be used to correct one or more packets associated with a slice. Thus, to accelerate recovery of lost packets using FEC, FEC packets may be interleaved within the data packets of the corresponding compressed video frame (e.g., within the packets of the encoded slice). That is, instead of placing the FEC packets at the end of the grouping of packets for the corresponding compressed video frame, the FEC packets are interleaved within the data packets. In that way, as soon as one or more data packets of the corresponding compressed video frame are lost, the corresponding FEC packets can be used to recover those lost packets almost instantly. In particular, recovery of lost packets of a corresponding compressed video frame can be performed without waiting for the entire set of data packets to be delivered because the corresponding FEC packet is delivered after the data packets that it can recover. Thus, an FEC packet received earlier (e.g., during transmission of an encoded slice of the corresponding compressed video frame), possibly before all data packets have been received for the compressed video frame, may be used to recover one or more lost packets. That is, in embodiments of the present disclosure, forward error correction may be performed earlier (e.g., during reception of a compressed video frame) to correct for lost and / or corrupted packets of an encoded slice, without waiting for the entire set of packets (e.g., data and FEC) of the compressed video frame to be received by the client. Typically, forward error correction of any packet of a compressed video frame is performed after all of the packets for that compressed video frame are received, thereby adding to the one-way latency between the server and the client; the disclosed embodiments perform forward error correction while the client is still receiving the compressed video frame, thereby reducing the one-way latency. Thus, by interleaving the FEC packets within the data packets of the corresponding compressed video frame, this results in overlapping the decoding and display operations at the client.

[0088] In addition, according to one embodiment of the present disclosure, when packets are aligned to the MTU, this can enable easier recovery from packet loss. That is, since packets are aligned and / or sized to the coded slices (i.e., a one-to-one relationship between the coded slices and the corresponding packets), a lost complete coded slice can be recovered using the corresponding FEC. In that way, FEC creation is performed across the data packets, such that the FEC packets can be used to correct one or more corresponding data packets, each data packet corresponding to a coded slice. As shown in FIG. 8A, each of the coded slices is sized to or below the MTU 850, such that one corresponding packet is used to transmit each coded slice over the network.

[0089] 8A illustrates interleaving FEC packets within packets of compressed encoder slices of an encoded video frame, according to one embodiment of the present disclosure. Purely, for example, an encoded video frame may include Encoded Slice 0, Encoded Slice 1, Encoded Slice 2, and Encoded Slice 3. Also, the encoded slices may be sized to MTU 850 or smaller. As shown, FEC-0 and FEC-1 are interleaved with data packets for Encoded Slice 0, Encoded Slice 1, Encoded Slice 2, and Encoded Slice 3.

[0090] In particular, a packet 810a is created for coded slice 0 during compression, and coded slice 0 is sized to compress into one packet. Coded slice 0 may include a padded portion 810b for FEC creation and recovery purposes. It is important to note that in one embodiment, only packet 810a is transmitted from the server to the client, as padded portion 810b is stripped prior to transmission, but can be considered at the client for recovery purposes. Similarly, a packet 811a is created for coded slices during compression, and coded slice 1 may be sized to compress into one packet. Coded slice 1 does not include a corresponding padded portion for FEC creation, as it is the largest packet in the set of packets to recover during FEC, as described below.

[0091] An FEC-0 is generated as packet 820 and transmitted after the packets for Coded Slice 0 and Coded Slice 1. FEC-0 820 may be used to correct for each of those Coded Slices that may be lost and / or FEC-0 820 may be used to correct for lost packets of each of those compressed Coded Slices. The packet 820 packet size for FEC-0 may indicate the size of padding used for compressed Coded Slice 0 and Coded Slice 1 for the purposes of performing FEC creation and recovery. That is, the FEC packets are created based on the sizing of the blocks, and the size of the FEC packets is tuned to the size of the largest data packet in the set of data packets that can be corrected using its corresponding FEC packet. Thus, FEC creation requires knowledge of the sizing of the data packets in the set. However, rather than waiting for each of the data packets in the set corresponding to the FEC packet to be generated before transmitting any of the data packets in the set for the purpose of FEC creation, the data packets in the set may be transmitted instantaneously once they are compressed.

[0092] In particular, packets 810a for encoded slice 0 may be transmitted as soon as they are compressed at the server's encoder without waiting for generation of FEC-0, and may be stored or buffered for FEC creation. Also, packets 811a for encoded slice 1 may be transmitted as soon as they are compressed at the server's encoder without waiting for generation of FEC-0, and may be stored or buffered for FEC creation. Because the FEC packets are created using block sizing, the size of each of the data packets needs to be communicated prior to creating a corresponding FEC packet that can be used to recover any of those data packets that may be lost.

[0093] In one embodiment, the size of each data packet is included in the header for the corresponding data packet that needs to be read for FEC creation purposes. In another embodiment, rather than communicating the size of each data packet in the header for the corresponding data packet that needs to be read for FEC creation purposes, the sizing of each of the packets 810a and 811a for the encoded slice may be known or determined because those packets are stored. Once the size of the largest packet in the set of data packets that can be corrected using the corresponding FEC packet is known, the sizing of the block of packets for FEC-0 may be tuned to the maximum of packets 810a and 811a when generating FEC-0.

[0094] In addition, once the size of FEC-0 is known, the padding size is also known for data packets in the set of data packets that can be corrected using FEC-0 for FEC creation and data recovery purposes. As shown in FIG. 8A, FEC-0 is tuned to the sizing of coded slice 1 packet 811a, which is the largest packet in the set that includes coded slice 0 and coded slice 1. Thus, padding 810b is needed for coded slice 0 for FEC creation and / or data recovery. Coded slice 1 packet 811a does not need any padding for FEC creation and / or data recovery because the size of FEC-0 packet 820 is tuned to the size of coded slice 1 packet 811a (i.e., the largest packet in the set of data packets that can be corrected using FEC-0).

[0095] In another embodiment, the padding may be set to the MTU in one embodiment. In particular, the FEC packets are normalized to the size of the MTU. Also, for purposes of FEC creation and / or data recovery, each of the packets in the set of packets that can be corrected using the corresponding FEC is also padded to the MTU. For example, FEC-0 is padded to the MTU, Coded Slice 0 is padded to the MTU, and Coded Slice 1 is also padded to the MTU for purposes of FEC creation and / or data recovery.

[0096] 8A , FEC-1 may be used to correct a set of data packets including Encoded Slice 2 and Encoded Slice 3. In particular, packet 812a may be created for Encoded Slice 2 during compression, and Encoded Slice 2 may be sized to compress into one packet. Encoded Slice 2 does not include a corresponding padded portion for FEC creation and / or data recovery such that it is the largest packet in size that can be recovered using FEC-1. Also, a packet 813a may be created for coded slice 3 during compression, and coded slice 3 may be sized to be compressed into one packet. Coded slice 3 may include a corresponding padded portion 813b for FEC creation and data recovery that is stripped before transmission. An FEC-1 is generated as a packet 825 and transmitted after coded slice 2 and coded slice 3. The FEC-1 may be used to correct for each of those coded slices that may be lost and / or may be used to correct for lost packets of each of those compressed coded slices. In particular, packet 812a for coded slice 1 may be transmitted as soon as it is compressed in the server's encoder, without waiting for FEC-1 generation, and may be stored and buffered for FEC creation. Also, packet 813a for coded slice 3 may be transmitted as soon as it is compressed in the server's encoder, without waiting for FEC-1 generation, and may be stored and buffered for FEC creation.

[0097] Similarly, the packet size of packet 825 for FEC-1 can indicate the size of padding used for Coded Slice 2 and Coded Slice 3 during encoding. As previously described, the size of each data packet may be included in the header for the corresponding data packet that needs to be read for FEC creation purposes. As previously described, rather than communicating the size of each data packet in the header for the corresponding data packet that needs to be read for FEC creation purposes, the sizing of each of packets 812a and 813a for the coded slices may be known or determined because each of those packets is stored. Once the size of the largest packet in the set of data packets that can be corrected using the corresponding FEC packet is known, the sizing of the block of packets for FEC-1 may be tuned to the largest of packets 812a and 813a when generating FEC-1. In particular, once the size of FEC-1 is known, padding is known for data packets in the set of data packets that can be corrected using FEC-1 for FEC creation and data recovery purposes. As shown in FIG. 8A, FEC-1 is tuned to the sizing of packet 812a of coded slice 2, which is the largest packet in the set that includes coded slice 2 and coded slice 3. Thus, no padding is required for coded slice 2 for FEC creation and / or data recovery. In contrast, padding 813b is required for coded slice 3 for FEC creation and / or data recovery.

[0098] In embodiments of the present disclosure, when streaming media content and / or receiving streamed media content, one or more strategies may be implemented to recover encoded slices and / or packets of encoded slices at the client that are lost during transmission from the server to the client. As previously described, each encoded slice of a corresponding compressed video frame is packetized into one corresponding network packet, such that a packet corresponds to a single encoded slice in a one-to-one relationship. When overlapping all parts of the pipeline, including both server and client operations, a decision needs to be made as to when to wait for additional packets of the corresponding video frame (e.g., when a lost packet has been received) and when to move to the next video frame.

[0099] Information in the header of the data packet may be used to make the decision when to move to the next video frame. In each data packet header, there is a frame index that informs the video game (e.g., gaming application) which video frame the corresponding packet belongs to. In addition, each packet header contains information about the packet index and the total number of packets for that frame. For example, Figure 8B is a diagram of a header 850 of a data packet 800B of an encoded slice for a corresponding video frame, according to one embodiment of the disclosure. For example, the information in the header 850 includes a frame number 851 for the corresponding video frame, a total number of encoded slices for the corresponding video frame 852, a frame generation timestamp (e.g., required in flip time) 853, a cryptographic verification 854, an identifier 855 for the corresponding FEC packet, an encoded slice number 856, and a timestamp 857 specifying when the corresponding encoded slice was generated. Optionally, the size of the data packet 800B may be included in the header 850. The data packet 800B includes a data portion 858.

[0100] At the point where the data packets from the next video frame are received at the client, a decision is made whether to wait for the remaining data packets for the current video frame or to move to the next video frame. In particular, as packets are received for the current video frame (N), tracking of which packets have been received is performed. In one embodiment, as soon as a packet from the next video frame (N+1) is received, any packets (minus FEC packets) that have not been received for the current video frame need to be corrected, such as through a method called "predec". In order to ensure that the entire H.264 video frame remains syntactically correct, Predec is performed to generate syntactically correct H.264 coded slices. That is, when it is determined that a coded slice and / or one or more packets of a coded slice are lost, a filler slice is generated as a replacement for the lost coded slice by inserting a skip macroblock. In this way, the decoder is less likely to crash when there are lost coded slices.

[0101] At this point, corruption may occur on the client side and reference frame invalidation, FEC recovery, or using data from a previous rendered video frame may need to be performed to correct any corrupted areas on the screen (e.g., skipped macroblocks corresponding to lost coded slices).

[0102] For example, in one embodiment, reference frame invalidation may be performed to correct skipped macroblocks for pixels corresponding to the first coded slice. In another embodiment, FEC recovery is performed to correct lost coded slices and / or lost data packets or lost packets of the corresponding coded slices. For example, received FEC packets and / or FEC slices are used to regenerate and / or recover lost coded slices, lost data packets, or lost data packets of the corresponding coded slices.

[0103] In yet another embodiment, a previous video frame is used to compensate for the skipped macroblock. For example, when rendering to a display at the client for the current video frame, color values ​​from the previous video frame for pixels corresponding to the skipped macroblock of the lost coded slice are used. In particular, appropriate data from a frame buffer holding pixel data for the previous video frame is identified to overlay pixels and / or skipped macroblocks for the lost coded slice. For example, the current video frame may have eight slices from which the second coded slice is missing. Because the previous video frame has already been rendered for display at the client, data from the previous video frame may be used to address the missing second coded slice of the current video frame. For purposes of illustration, the previous video frame may be divided into 13 encoded slices. Data for the current video frame rendered for display at the client is placed (e.g., pixel by pixel) in a "new" display buffer. Pixel data for the previous video frame that overlays pixels corresponding to the lost encoded slice may be identified and accessed from the "old" display buffer. That is, pixel data from the previous video frame may be used for the lost second encoded slice of the current video frame.

[0104] FIG. 9 illustrates components of an example device 900 that can be used to implement aspects of various embodiments of the present disclosure. For example, FIG. 9 illustrates an exemplary hardware system suitable for streaming media content and / or receiving streamed media content, including overlapping encoding and sending operations at a server, overlapping receiving and decoding operations at a client, and overlapping receiving, decoding, and rendering operations with display operations at a client, according to embodiments of the disclosure. The block diagram illustrates device 900, which may be incorporated into a personal computer, server computer, game console, mobile device, or other digital device, each of which is suitable for implementing embodiments of the invention. Device 900 includes a central processing unit (CPU) 902 for running software applications and optionally an operating system. CPU 902 may be comprised of one or more homogeneous or heterogeneous processing units.

[0105] According to various embodiments, CPU 902 is one or more general-purpose microprocessors having one or more processing cores. Further embodiments may be implemented using one or more CPUs having a microprocessor architecture specifically adapted for highly parallel and computationally intensive applications, such as applications configured for graphics processing during the execution of games, media applications, and interactive entertainment applications.

[0106] Memory 904 stores applications and data for use by CPU 902 and GPU 916. Storage 906 provides non-volatile storage and other computer-readable media for applications and data, and may include signal transmission and storage media along with fixed disk drives, removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other optical storage devices. User input device 908 communicates user input from one or more users to device 500, examples of which may include a keyboard, mouse, joystick, touchpad, touch screen, still or video recorder / camera, and / or microphone. The network interface 909 enables the device 900 to communicate with other computer systems via electronic communications networks, which may include wired or wireless communications over local area networks and wide area networks such as the Internet. The audio processor 912 is adapted to generate analog or digital audio output from instructions and / or data provided by the CPU 902, memory 904, and / or storage device 906. The components of the device 900, including the CPU 902, the graphics subsystem including the GPU 916 and GPU cache 918, the memory 904, the data storage device, the user input devices 908, the network interface 909, and the audio processor 912, are connected via one or more data buses 922.

[0107] The graphics subsystem 914 is further connected to the data bus 922 and components of the device 900. The graphics subsystem 914 includes a graphics processing unit (GPU) 916 and a graphics memory 918. The graphics memory 918 includes a display memory (e.g., a frame buffer) used to store pixel data for each pixel of an output image. The graphics memory 918 may be integrated into the same device as the GPU 916, may be connected as a separate device to the GPU 916, and / or may be implemented within the memory 904. The pixel data may be provided to the graphics memory 918 directly from the CPU 902. Alternatively, CPU 902 provides data and / or instructions defining a desired output image to GPU 916 from which GPU 916 generates pixel data for one or more output images. The data and / or instructions defining the desired output image may be stored in memory 904 and / or graphics memory 918. In an embodiment, GPU 916 includes 3D rendering capabilities for generating pixel data for output images from instructions and data defining geometry, lighting, shading, texturing, motion, and / or camera parameters for a scene. GPU 916 further includes one or more programmable execution units capable of executing shader programs.

[0108] Graphics subsystem 914 periodically outputs pixel data from graphics memory 918 for an image to be displayed on display device 910 or projected by a projection system (not shown). Display device 910 may be any device capable of displaying visual information in response to signals from device 900, including CRT, LCD, plasma, and OLED displays. Device 900 may provide analog or digital signals to display device 910, for example.

[0109] Other embodiments for optimizing the graphics subsystem 914 include multi-tenancy GPU operation, where a GPU instance is shared between multiple applications and a single game and a distributed GPU supporting the graphics subsystem 914 is configured as one or more processing devices.

[0110] For example, in one embodiment, graphics subsystem 914 may be configured to perform multi-tenancy GPU functionality, where one graphics subsystem runs the graphics and / or rendering pipelines for multiple games, i.e., graphics subsystem 914 is shared between multiple running games.

[0111] In other embodiments, the graphics subsystem 914 includes multiple GPU devices that are combined to perform graphics processing for a single application running on a corresponding CPU. For example, multiple GPUs may perform alternative forms of frame rendering, with GPU1 rendering the first frame, GPU2 rendering the second frame, and so on, in serial frame periods, as soon as the first GPU renders the next video frame, until the last GPU is reached (e.g., if there are only two GPUs, GPU1 renders the third frame). That is, the GPUs rotate when rendering frames. Rendering operations may be overlapped, and GPU2 may start rendering the second frame before GPU1 finishes rendering the first frame. In another embodiment, multiple GPU devices may be assigned different shader operations in the rendering pipeline and / or graphics pipeline. The master GPU performs primary rendering and compositing. For example, in a group including three GPUs, master GPU1 may perform rendering (e.g., first shader operations) and compositing of the output of slave GPU2 and slave GPU3, slave GPU2 may perform second shader (e.g., fluid effects such as rivers) operations, slave GPU3 may perform third shader (e.g., subtle smoke) operations, and master GPU1 composites the results from each of GPU1, GPU2, and GPU3. In that way, different GPUs can be assigned to perform different shader operations (e.g., flag waving, wind, smoke generation, fire, etc.) to render a video frame. In yet another embodiment, each of the three GPUs may be assigned to different objects and / or portions of a scene corresponding to a video frame. In the above embodiments and implementations, the operations may be performed within the same frame period (parallel and simultaneous) or within different frame periods (parallel and sequential).

[0112] Accordingly, this disclosure describes methods and systems configured to stream media content and / or receive streamed media content, including overlapping encoding and transmitting operations at a server, overlapping receiving and decoding operations at a client, and overlapping receiving, decoding, and rendering operations with display operations at the client.

[0113] It should be understood that the various embodiments defined herein can be combined or incorporated into specific embodiments using various features disclosed herein. Thus, the examples provided are only some possible examples and are not limited to the various embodiments in which more embodiments can be defined by combining various elements. In some examples, some embodiments may include fewer elements without departing from the spirit of the disclosed or equivalent embodiments.

[0114] Embodiments of the present disclosure may be practiced in a variety of computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. Embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.

[0115] With the above embodiments in mind, it should be understood that the embodiments of the present disclosure can employ various computer-implemented operations involving data stored in computer systems. These operations are operations requiring physical manipulation of physical quantities. Any of the operations described herein that form part of the embodiments of the present disclosure are useful machine operations. The disclosed embodiments also relate to devices or apparatus for performing these operations. The apparatus may be specially constructed for the required purposes. Alternatively, the apparatus may be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general-purpose machines can be used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct a more specialized apparatus to perform the required operations.

[0116] The present disclosure can also be embodied as computer readable code on a computer readable medium. A computer readable medium is any data storage device that can store data that can be subsequently read by a computer system. Examples of computer readable media include hard drives, network attached storage (NAS), read only memory, random access memory, CD-ROM, CD-R, CD-RW, magnetic tape, and other optical and non-optical data storage devices. The computer readable medium may also include computer readable tangible media distributed across networked computer systems such that the computer readable code is stored and executed in a distributed fashion.

[0117] Although the operations of the method have been described in a particular order, it should be understood that other housekeeping operations may be performed between operations, or operations may be coordinated to occur at slightly different times, or operations may be distributed within the system to allow processing operations to occur at various intervals related to processing, so long as the processing of the overlay operation is performed in the desired manner.

[0118] Although the foregoing disclosure has been described in some detail for clarity of understanding, it will be apparent that certain changes and modifications can be practiced within the scope of the appended claims. Thus, the present embodiments are to be considered as illustrative and not restrictive, and the embodiments of the present disclosure are not to be limited to the details provided herein, but may be modified within the scope and equivalents of the appended claims.

Claims

1. 1. A method for cloud gaming, comprising: generating video frames when executing the video game on the server; scanning said video frames into an encoder for each server scanline; encoding one or more encoder slices at the encoder as one or more encoded slices, each encoded slice being delivered to a buffer at an encoder fill rate; 22. The method of claim 21, further comprising: draining a first encoded slice from the buffer to stream the first encoded slice over a network to a client at a buffer drain rate, the first encoded slice being streamed over the network before fully encoding the one or more slices of the video frame.

2. The method of claim 1 , wherein the server drains at least one encoded slice of the video frame before the entire video frame is encoded.

3. Further, the encoded slices are packetized into one or more network communication packets; sizing each of the one or more network communication packets at the encoder to less than or equal to a maximum transmission unit (MTU); The method of claim 1 , further comprising streaming the one or more network communication packets through the network to the client at the drain rate.

4. 4. The method of claim 3, further comprising sizing each of the one or more encoded slices at or below the MTU such that each encoded slice is packetized into a single corresponding network communication packet.

5. Furthermore, measuring a maximum receive bandwidth of the client; dynamically adjusting the drain rate to approximately equal the maximum reception bandwidth such that transmissions of the one or more encoded slices are spaced based on the measured maximum reception bandwidth of the client; The method of claim 1 , wherein the buffer drain rate is decoupled from the encoder fill rate.

6. generating one or more forward error correction (FEC) slices used to correct the one or more encoded slices; interleaving the one or more FEC slices within the one or more encoded slices when streaming; each of the one or more FEC slices and the one or more encoded slices is packetized into a corresponding packet of a size less than or equal to a maximum transmission unit (MTU); The method of claim 1.

7. further comprising: generating a plurality of video frames at the server in response to processing of the video game by the server in a streaming mode during a session with the client; encoding the plurality of video frames in the encoder into a plurality of compressed video frames, each compressed video frame including one or more encoded slices; The method of claim 1 , further comprising transmitting the plurality of compressed video frames to the client over a network.

8. The method of claim 7 , wherein the client is configured to receive and decompress the plurality of compressed video frames for rendering on a display associated with the client.

9. 1. A non-transitory computer readable medium storing a computer program for performing a method for cloud gaming, the computer readable medium comprising: having program instructions for generating video frames when executing a video game on the server; having program instructions for scanning in the video frame to an encoder for each server scanline; program instructions for encoding one or more encoder slices at the encoder as one or more encoded slices, each encoded slice being delivered to a buffer at an encoder fill rate; A non-transitory computer-readable medium having program instructions for draining a first encoded slice from the buffer to stream the first encoded slice over a network to a client at a buffer drain rate, the first encoded slice being streamed over the network before fully encoding the one or more slices of the video frame.

10. 10. The non-transitory computer-readable medium of claim 9, wherein the server drains at least one encoded slice of the video frame before the entire video frame is encoded.

11. The method includes packetizing the encoded slices into one or more network communication packets; further comprising program instructions for sizing each of the one or more network communication packets at the encoder to less than or equal to a maximum transmission unit (MTU); and program instructions for streaming the one or more network communication packets to the client over the network at the drain rate.

12. 12. The non-transitory computer-readable medium of claim 11, further comprising program instructions for sizing each of the one or more encoded slices at or below the MTU such that each encoded slice is packetized into a single corresponding network communication packet.

13. further comprising program instructions for measuring a maximum receive bandwidth of said client; and program instructions for dynamically adjusting the drain rate to approximately equal a maximum reception bandwidth of the client such that transmissions of the one or more encoded slices are spaced based on the measured maximum reception bandwidth of the client; The non-transitory computer-readable medium of claim 9 , wherein the buffer drain rate is decoupled from the encoder fill rate.

14. program instructions for generating one or more forward error correction (FEC) slices used to correct the one or more encoded slices; and program instructions for interleaving the one or more FEC slices within the one or more encoded slices when streaming, each of the one or more FEC slices and the one or more encoded slices is packetized into a corresponding packet of a size less than or equal to a maximum transmission unit (MTU); 10. The non-transitory computer readable medium of claim 9.

15. further comprising program instructions for generating a plurality of video frames at the server in response to processing of the video game by the server in a streaming mode during a session with the client; program instructions for encoding the plurality of video frames in the encoder into a plurality of compressed video frames, each compressed video frame including one or more encoded slices; 10. The non-transitory computer readable medium of claim 9 having program instructions for transmitting the plurality of compressed video frames to the client over a network.

16. 20. The non-transitory computer-readable medium of claim 15, wherein the method is configured to receive and decompress the plurality of compressed video frames for rendering on a display associated with the client.

17. 1. A computer system comprising: A processor; and a memory coupled to the processor and having instructions stored thereon, the instructions, when executed by the computer system, causing the computer system to perform a method for cloud gaming, the method comprising: generating video frames when executing the video game on the server; scanning said video frames into an encoder for each server scanline; encoding one or more encoder slices at the encoder as one or more encoded slices, each encoded slice being delivered to a buffer at an encoder fill rate; and draining a first encoded slice from the buffer to stream the first encoded slice over a network to a client at a buffer drain rate, the first encoded slice being streamed over the network before fully encoding the one or more slices of the video frame.

18. 20. The computer system of claim 17, wherein the server drains at least one encoded slice of the video frame before the entire video frame is encoded.

19. The method further comprises: Packetizing the encoded slices into one or more network communication packets; sizing each of the one or more network communication packets at the encoder to less than or equal to a maximum transmission unit (MTU); 20. The computer system of claim 17, further comprising: streaming the one or more network communication packets through the network to the client at the drain rate.

20. The method further comprises:

20. The computer system of claim 19, wherein each of the one or more encoded slices is sized at or below the MTU such that each encoded slice is packetized into a single corresponding network communication packet.

21. 1. A method of cloud gaming, comprising: receiving at a client one or more encoded slices of a video frame, the video frame having been generated at a server during execution of a video game and encoded into the one or more encoded slices by an encoder at the server; The method of claim 1, further comprising: decoding a first encoded slice at the client decoder prior to complete reception of the one or more encoded slices of the video frame.

22. 22. The method of claim 21, wherein the client VSYNC signal and the server VSYNC signal are synchronized and offset to allow for reduced latency in display of the decoded slices.

23. each of the one or more encoded slices is packetized into one or more network communication packets; 22. The method of claim 21, wherein each of the one or more network communication packets is sized at the encoder to be equal to or smaller than a maximum transmission unit (MTU).

24. 24. The method of claim 23, wherein each of the one or more encoded slices is sized at or below the MTU such that each encoded slice is packetized into a single corresponding network communication packet.

25. generating a plurality of video frames at the server in response to processing of the video game by the server in a streaming mode during a session with the client; the plurality of video frames are encoded in the encoder into a plurality of compressed video frames, each corresponding compressed video frame including one or more corresponding encoded slices; 22. The method of claim 21, wherein the plurality of compressed video frames are transmitted to the client over a network.

26. further comprising: receiving the plurality of compressed video frames; 26. The method of claim 25, further comprising decompressing the plurality of compressed video frames for rendering on a display associated with the client.

27. further determining that a first encoded slice of the one or more encoded slices for the video frame was lost during transmission from the server to the client; 22. The method of claim 21, further comprising generating a filler slice as a replacement for the first coded slice decoded by the decoder, the decoder inserting skip macroblocks for pixels of a display corresponding to the first coded slice.

28. 30. The method of claim 27, further comprising performing reference frame avoidance to correct the skipped macroblock for the pixels corresponding to the first coded slice.

29. further determining that a first encoded slice of the one or more encoded slices for the video frame was lost during transmission from the server to the client; receiving a forward error correction (FEC) slice corresponding to the lost first encoded slice; The method of claim 21 , further comprising using the FEC slice to regenerate the first encoded slice.

30. further determining that a first encoded slice of the one or more encoded slices for the video frame was lost during transmission from the server to the client; 22. The method of claim 21, further comprising using color values ​​from a previous video frame for pixels corresponding to the first encoded slice.

31. 1. A non-transitory computer readable medium storing a computer program for performing a method for cloud gaming, the computer readable medium comprising: and program instructions for receiving at a client one or more encoded slices of a video frame, the video frame having been generated at a server during execution of a video game and encoded into the one or more encoded slices by an encoder at the server; A non-transitory computer-readable medium having program instructions for decoding a first encoded slice at a decoder of the client prior to complete reception of the one or more encoded slices of the video frame.

32. 32. The non-transitory computer-readable medium of claim 31, in which the method includes synchronizing and offsetting a client VSYNC signal and a server VSYNC signal to enable reduced latency in displaying decoded slices.

33. The method further comprising: packetizing each of the one or more encoded slices into one or more network communication packets; The method further comprising: sizing each of the one or more network communication packets at the encoder to a maximum transmission unit (MTU) or less; 32. The non-transitory computer-readable medium of claim 31 , in which the method, each of the one or more encoded slices is sized at or below the MTU such that each encoded slice is packetized into a single corresponding network communication packet.

34. The method further comprises generating a plurality of video frames at the server in response to processing of the video game by the server in a streaming mode during a session with the client; In the method, the plurality of video frames are encoded in the encoder into a plurality of compressed video frames, each corresponding compressed video frame including one or more corresponding encoded slices; 32. The non-transitory computer-readable medium of claim 31, wherein the method comprises transmitting the plurality of compressed video frames over a network to the client.

35. program instructions for receiving the plurality of compressed video frames; program instructions for decompressing the plurality of compressed video frames for rendering on a display associated with the client; 35. The non-transitory computer readable medium of claim 34, further comprising:

36. further comprising program instructions for determining that a first encoded slice of the one or more encoded slices for the video frame was lost during transmission from the server to the client; and program instructions for generating a filler slice as a replacement for the first coded slice decoded by the decoder, the decoder inserting skip macroblocks for pixels of a display corresponding to the first coded slice; 32. The non-transitory computer readable medium of claim 31 having program instructions for performing reference frame avoidance to correct the skipped macroblock for the pixels corresponding to the first coded slice.

37. further comprising program instructions for determining that a first encoded slice of the one or more encoded slices for the video frame was lost during transmission from the server to the client; receiving a forward error correction (FEC) slice corresponding to the lost first encoded slice; 32. The non-transitory computer-readable medium of claim 31 having program instructions for using the FEC slice to regenerate the first encoded slice.

38. 1. A computer system comprising: A processor; and a memory coupled to the processor and having instructions stored thereon, the instructions, when executed by the computer system, causing the computer system to perform a method for cloud gaming, the method comprising: receiving at a client one or more encoded slices of a video frame, the video frame having been generated at a server during execution of a video game and encoded into the one or more encoded slices by an encoder at the server; The computer system further comprising: decoding a first encoded slice at the client decoder prior to complete reception of the one or more encoded slices of the video frame.

39. 40. The computer system of claim 38, wherein the method includes synchronizing and offsetting the client VSYNC signal and the server VSYNC signal to allow for reduced latency in display of the decoded slices.

40. each of the one or more encoded slices is packetized into one or more network communication packets; 40. The computer system of claim 38, wherein each of the one or more network communication packets is sized at the encoder to be equal to or smaller than a maximum transmission unit (MTU).

Images