Overlaid encoding and transmission at a server

Abstract

A method for cloud gaming. The method includes generating a video frame while executing a video game at a server. The method includes scanning the video frame into an encoder at the server on a scan line by scan line basis. The method includes encoding one or more encoder slices into one or more encoded slices at the encoder, where each encoded slice is transferred to a buffer at an encoder fill rate. The method includes draining a first encoded slice from the buffer to stream the first encoded slice to a client over a network at a buffer drain rate, where the first encoded slice is streamed over the network before one or more slices of a fully encoded video frame.

Description

Technical Field

[0001] This disclosure relates to streaming systems configured for streaming content across networks, and more specifically to overlapping operations at cloud gaming servers and / or clients to reduce latency between cloud gaming servers and clients. Background Technology

[0002] In recent years, there has been a continuous push for online services, allowing online or cloud gaming to be conducted in streaming formats between cloud gaming servers and clients connected via a network. Streaming formats are gaining popularity because they enable on-demand delivery of game titles, multiplayer gaming between players, asset sharing between players, instant sharing of experiences between players and / or spectators, allowing friends to watch a friend play a video game, let friends join a friend's ongoing game, and so on. Unfortunately, this demand also challenges the limitations of network connectivity and the processing capabilities performed at both the server and client ends, whose responses should be sufficient to render high-quality images when transmitted to the client. For example, the results of all game activity performed on the server need to be compressed and transmitted back to the client with low millisecond latency for the best user experience. Round-trip latency can be defined as the total time between user controller input and the display of a video frame at the client; it may include the processing and transmission of control information from the controller to the client, the processing and transmission of control information from the client to the server, the generation of a video frame at the server in response to the input, processing the video frame and transmitting it to an encoding unit (e.g., scan output), encoding the video frame, transmitting the encoded video frame back to the client, receiving and decoding the video frame, and any processing or segmentation of the video frame before display. One-way latency can be defined as a portion of the round-trip latency, including the time from the start of transmitting the video frame to the encoding unit (e.g., scan output) at the server to the start of displaying the video frame at the client. A portion of both round-trip latency and one-way latency is associated with the time it takes for data to flow through the communication network from the client to the server and from the server to the client. Another portion is associated with processing at both the client and server; improvements to these operations, such as advanced strategies related to frame decoding and display, can significantly reduce round-trip latency and one-way latency between the server and client, providing a higher quality experience for users of cloud gaming services.

[0003] It is against this backdrop that the implementation scheme disclosed herein came into being. Summary of the Invention

[0004] The embodiments of this disclosure relate to configuring a streaming system for streaming content (e.g., games) across a network, and more specifically to overlapping encoding and sending operations at the server, overlapping receiving and decoding operations at the client, and overlapping receiving, decoding, and rendering to a display operations at the client.

[0005] This disclosure discloses a method for cloud gaming. The method includes receiving one or more encoded slices of video frames at a client, wherein the video frames are generated when a video game is executed at a server, and the video frames are encoded into one or more encoded slices by an encoder at the server. The method includes decoding a first encoded slice at a decoder at the client before fully receiving the one or more encoded slices of the video frames. At the server, multiple video frames are generated in response to the server processing the video game in a streaming mode during a session with the client. The multiple video frames are encoded into multiple compressed video frames at an encoder, each corresponding compressed video frame including one or more corresponding encoded slices. The multiple compressed video frames are transmitted to the client over a network.

[0006] In another embodiment, a method for cloud gaming is disclosed. The method includes generating video frames while executing a video game at a server. The method includes scanning the video frames scan-line by scan into an encoder at the server. The method includes encoding one or more scan lines at the encoder into one or more encoded slices, wherein each encoded slice is transmitted to a buffer at an encoder fill rate. The method includes ejecting a first encoded slice from the buffer to stream the first encoded slice over a network to a client at a buffer eject rate, wherein the first encoded slice is streamed over the network before one or more slices of the video frame are fully encoded.

[0007] In another embodiment, a non-transitory computer-readable medium storing a computer program for cloud gaming is disclosed. The computer-readable medium includes program instructions for receiving one or more coded slices of video frames at a client, wherein the video frames are generated when a video game is executed at a server, and the video frames are encoded into one or more coded slices by an encoder at the server. The computer-readable medium includes program instructions for decoding a first coded slice at a decoder at the client before fully receiving one or more coded slices of the video frames. At the server, multiple video frames are generated in response to the server processing the video game in a streaming mode during a session with the client. The multiple video frames are encoded into multiple compressed video frames at an encoder, each corresponding compressed video frame including one or more corresponding coded slices. The multiple compressed video frames are transmitted to the client over a network.

[0008] In another embodiment, a non-transitory computer-readable medium stores a computer program for cloud gaming. The non-transitory computer-readable medium includes program instructions for generating video frames when executing a video game at a server. The non-transitory computer-readable medium includes program instructions for scanning video frames scan-line by scan into an encoder at the server. The non-transitory computer-readable medium includes program instructions for encoding one or more encoder slices into one or more encoded slices at the encoder, wherein each encoded slice is transmitted to a buffer at an encoder fill rate. The non-transitory computer-readable medium includes program instructions for ejecting a first encoded slice from the buffer to stream the first encoded slice over a network to a client at a buffer eject rate, wherein the first encoded slice is streamed over the network before one or more slices of a video frame are fully encoded.

[0009] In another embodiment, the computer system includes a processor and a memory coupled to the processor and storing instructions therein, which, if executed by the computer system, cause the computer system to perform a method for cloud gaming. The method includes receiving one or more encoded slices of video frames at a client, wherein the video frames are generated when a video game is executed at a server, and the video frames are encoded into one or more encoded slices by an encoder at the server. The method includes decoding a first encoded slice at a decoder at the client before fully receiving one or more encoded slices of the video frames. At the server, in response to processing the video game in a streaming mode by the server during a session with the client, multiple video frames are generated. The multiple video frames are encoded into multiple compressed video frames at an encoder, each corresponding compressed video frame including one or more corresponding encoded slices. The multiple compressed video frames are transmitted to the client via a network.

[0010] In another embodiment, the computer system includes a processor and a memory coupled to the processor and storing instructions therein, which, if executed by the computer system, cause the computer system to perform a method for cloud gaming. The method includes generating video frames while executing a video game at a server. The method includes scanning the video frames scan-line by scan into an encoder at the server. The method includes encoding one or more encoder slices into one or more encoded slices at the encoder, wherein each encoded slice is transmitted to a buffer at an encoder fill rate. The method includes ejecting a first encoded slice from the buffer to stream the first encoded slice over a network to a client at a buffer eject rate, wherein the first encoded slice is streamed over the network before one or more slices of the video frame are fully encoded.

[0011] Other aspects of this disclosure will become apparent from the following detailed description taken in conjunction with the accompanying drawings, which illustrate the principles of this disclosure by way of example. Attached Figure Description

[0012] This disclosure may be best understood by reference to the following description taken in conjunction with the accompanying drawings, in which:

[0013] Figure 1A This is a diagram of the VSYNC signal at the beginning of a frame period according to one embodiment of the present disclosure.

[0014] Figure 1B This is a frequency diagram of the VSYNC signal according to one embodiment of the present disclosure.

[0015] Figure 2A This is a diagram of a system according to one embodiment of the present disclosure for providing games over a network between one or more cloud gaming servers and one or more client devices in various configurations, wherein the VSYNC signal can be synchronized and offset to reduce one-way latency.

[0016] Figure 2B This is a diagram according to one embodiment of the present disclosure for providing a game between two or more peer devices, wherein the VSYNC signal can be synchronized and offset to achieve optimal timing for receiving controller and other information between the devices.

[0017] Figure 2C Various network configurations that benefit from proper synchronization and offset of the VSYNC signal between the source and target devices are shown according to one embodiment of this disclosure.

[0018] Figure 2D This illustration shows a multi-tenant configuration between a cloud gaming server and multiple clients that benefits from proper synchronization and offset of the VSYNC signal between the source and target devices, according to one embodiment of this disclosure.

[0019] Figure 3 This illustration shows the variation in one-way latency between a cloud gaming server and a client due to clock drift when streaming video frames generated from a video game executed on a server, according to one embodiment of this disclosure.

[0020] Figure 4 The network configuration shown in one embodiment of the present disclosure, including a cloud gaming server and a client, is used when streaming video frames generated from a video game executed on a server. The VSYNC signal between the server and the client is synchronized and offset to allow overlapping of operations at the server and the client, and to reduce one-way latency between the server and the client.

[0021] Figure 5A This is a diagram illustrating the overlap of encoding and sending operations at the server and the overlap of receiving and decoding operations at the client when streaming video frames generated from a video game executed on a server, according to one embodiment of this disclosure.

[0022] Figure 5B This is a flowchart illustrating a method for cloud gaming according to one embodiment of the present disclosure, wherein encoding operations and sending operations are overlapped at the server to reduce one-way waiting time between the server and the client.

[0023] Figure 6 This is a diagram illustrating the interval of encoded slices of video frames based on the client's maximum receive bandwidth when streaming video frames generated from a video game executed on a server, according to one embodiment of this disclosure.

[0024] Figure 7A This is a diagram illustrating the overlap of receiving and decoding operations at the client when streaming video frames generated from a video game executed on a server, according to one embodiment of this disclosure.

[0025] Figure 7B This is a flowchart illustrating a method for cloud gaming according to one embodiment of the present disclosure, wherein when streaming video frames generated from a video game executed on a server, receiving operations and decoding operations are overlapped at the client to reduce one-way latency between the server and the client, wherein rendering of decompressed video frames may also overlap with receiving and decoding operations.

[0026] Figure 8A This is a diagram illustrating the interleaving of forward error correction (FEC) packets within an encoded slice of a video frame during streaming of a video frame generated from a video game executed on a server, according to one embodiment of this disclosure.

[0027] Figure 8B This is a diagram of the header of a data packet of an encoded slice according to one embodiment of this disclosure.

[0028] Figure 9 Components of an exemplary device are shown that can be used to carry out various aspects of the various embodiments of this disclosure. Detailed Implementation

[0029] While the following detailed description contains numerous specific details for illustrative purposes, those skilled in the art will recognize that many variations and modifications of these details are within the scope of this disclosure. Therefore, in clarifying the various aspects of this disclosure described below, the generality of the appended claims is not diminished, and no limitation is imposed on these claims.

[0030] Generally, various embodiments of this disclosure describe methods and systems configured to reduce latency and / or latency instability between a source device and a target device when streaming media content (e.g., streaming audio and video from a video game). Specifically, in some embodiments of this disclosure, when streaming video frames generated from a video game executed on a server, one-way latency in cloud gaming applications can be reduced by overlapping the encoding and transmission operations of the video frames at the server. The server in the cloud gaming application encodes the video frames, and to reduce one-way latency, data transmission from the server to the client can begin once the data packets are encoded by the server's encoder. This may require sharing information (e.g., the amount of data encoded at a given time) between the encoding units within the server and those units performing data transmission (e.g., a streaming device operating at the application layer). In some embodiments, the encoder is tuned to encode data packets such that the packet size corresponds to the natural amount of data created by the encoder. In one example, the packet size may correspond to the network's Maximum Transmission Unit (MTU). In another example, the packet size may correspond to the MTU and is also limited to the encoded slices generated by the encoder. In yet another example, in one implementation, the packet size may correspond to a scanline or a screen slice. In this way, one-way latency and the required buffering amount are reduced. Furthermore, in some embodiments of this disclosure, when streaming video frames generated from a video game executed on a server, one-way latency in a cloud gaming application can be reduced by overlapping the receiving and decoding operations of the video frames at the client. In other embodiments of this disclosure, one-way latency between the cloud gaming server and the client can be reduced by overlapping the receiving and decoding operations of compressed video frames with their display. The client in the cloud gaming application receives compressed video frames from the cloud gaming server and is configured to decode the compressed video frames. One-way latency between the server and the client can be reduced by initiating the decoding operation at the client as soon as possible. For example, each packet encoding a slice can be decoded when it is received by the client from the server. Furthermore, one-way latency can be reduced by starting the display of the video frame before it is fully received or decoded at the client. The timing of submission for display must anticipate the remaining time required for receiving and decoding the compressed video frames. As mentioned earlier, packet size adjustment can correspond to the natural amount of data created by the encoder (e.g., MTU, scan lines, encoder slices resized to the MTU size, etc.), which makes it possible to decode each packet as it arrives.

[0031] Specifically, latency instability can be introduced between the server and client due to the additional time required to generate complex frames at the server (e.g., scene changes), the increased time for encoding / compressing complex frames at the server, variable communication paths on the network, and the increased time for decoding complex frames at the client. Latency instability can also be introduced due to clock differences between the server and client, causing drift between the server's and client's VSYNC signals. In one implementation, this latency instability can be eliminated by overlapping encoding and transmission operations at the server, receiving and decoding operations at the client, and / or receiving and decoding operations at the client with rendering operations for display.

[0032] With the above general understanding of the various implementation schemes, exemplary details of the implementation schemes will now be described with reference to the various accompanying drawings.

[0033] Throughout this specification, references to "game," "video game," "game application," or "application" are intended to refer to any type of interactive application that is initiated by executing input commands. For illustrative purposes only, interactive applications include applications for games, word processing, video processing, video game processing, etc. Furthermore, the terms used above are interchangeable.

[0034] Cloud gaming involves executing a video game on a server to generate game-rendered video frames, which are then sent to a client for display. The timing of operations at the server and client can be correlated with corresponding vertical synchronization (VSYNC) parameters. When the VSYNC signals are correctly synchronized and / or offset between the server and / or client, operations performed on the server (e.g., generating and transmitting video frames within one or more frame periods) are synchronized with operations performed on the client (e.g., displaying video frames on a monitor at display frames or refresh rates corresponding to the frame periods). Specifically, the server-side VSYNC signal generated on the server and the client-side VSYNC signal generated on the client can be used to synchronize operations at the server and client. That is, when the server-side VSYNC signal and the client-side VSYNC signal are synchronized and / or offset, the server's generation and transmission of video frames are synchronized with how the client displays those video frames.

[0035] VSYNC signaling and Vertical Blanking Interval (VBI) have been incorporated for generating and displaying video frames when streaming media content between a server and a client. For example, the server attempts to generate game-rendered video frames within one or more frame cycles defined by the corresponding server VSYNC signal (e.g., generating one video frame per frame cycle results in 60Hz operation if the frame cycle is 16.7 milliseconds, while generating one video frame every two frame cycles results in 30Hz operation), then encodes the video frames and transmits them to the client. At the client, the received encoded video frames are decoded and displayed, with the client displaying each video frame rendered for display starting from the corresponding client VSYNC signal.

[0036] To illustrate, Figure 1A This demonstrates how the VSYNC signal 111 can indicate the start of a frame period, during which various operations can be performed at the server and / or client. When streaming media content, the server can use the server VSYNC signal to generate and encode video frames, while the client can use the client VSYNC signal to display video frames. The VSYNC signal 111 is generated at a defined frequency corresponding to the defined frame period 110, such as... Figure 1B As shown in the figure. Furthermore, VBI 105 defines the time period between when the last raster line of the previous frame period is drawn on the display and when the first raster line (e.g., the top) is drawn on the display. As shown, after VBI 105, video frames rendered for display are displayed via raster scan lines 106 (e.g., raster lines one by one from left to right).

[0037] Furthermore, various embodiments of this disclosure are disclosed for reducing one-way latency and / or latency instability between a source device and a target device, such as when streaming media content (e.g., video game content). For illustrative purposes only, various embodiments for reducing one-way latency and / or latency instability are described in a server and client network configuration. However, it should be understood that the various techniques disclosed for reducing one-way latency and / or latency instability can be implemented in other network configurations and / or on peer-to-peer networks, such as... Figures 2A to 2D As shown in the illustration. For example, the various implementations disclosed for reducing one-way wait times and / or wait time instability can be implemented between one or more server and client devices in various configurations (e.g., server and client, server and server, server and multiple clients, server and multiple servers, client and client, client and multiple clients, etc.).

[0038] Figure 2AThis diagram illustrates a system 200A, according to one embodiment of the present disclosure, for providing games via network 250 between one or more cloud gaming networks 290 and / or server 260 and one or more client devices 210 in various configurations. This includes server VSYNC signals and client VSYNC signals that can be synchronized and offset, and / or dynamic buffering performed on the client, and / or encoding operations on the server that overlap with transmission operations, and / or receiving operations and decoding operations on the client that overlap, and / or decoding operations and display operations on the client that overlap, to reduce one-way latency between server 260 and client 210. Specifically, according to one embodiment of the present disclosure, system 200A provides games via cloud gaming network 290, wherein the game is remotely executed from the client device 210 (e.g., a thin client) of the corresponding user playing the game. System 200A can provide game control via network 250 to one or more users playing one or more games through cloud gaming network 290 in single-player or multi-player mode. In some implementations, cloud gaming network 290 may include multiple virtual machines (VMs) running on a host hypervisor, wherein one or more VMs are configured to utilize the hardware resources available to the host hypervisor to execute a game processor module. Network 250 may include one or more communication technologies. In some implementations, network 250 may include fifth-generation (5G) network technology with advanced wireless communication systems.

[0039] In some implementations, wireless technologies can be used to facilitate communication. Such technologies may include, for example, 5G wireless communication technology. 5G is the fifth generation of cellular network technology. A 5G network is a digital cellular network in which the service area covered by the provider is divided into small geographical areas called cells. Analog signals representing voice and images are digitized in a telephone call, converted by an analog-to-digital converter, and transmitted as a bit stream. All 5G wireless devices in a cell communicate via radio waves with a local antenna array and low-power automatic transceivers (transmitters and receivers) in the cell, whose frequency channels are allocated by the transceivers from a frequency pool reused in other cells. The local antennas are connected to the telephone network and the Internet via high-bandwidth fiber optic or wireless backhaul connections. As with other cell networks, mobile devices moving from one cell to another automatically migrate to the new cell. It should be understood that 5G networks are merely exemplary types of communication networks, and embodiments of this disclosure may utilize previous generations of wireless or wired communication, as well as subsequent generations of wired or wireless technologies after 5G.

[0040] As shown in the figure, the cloud gaming network 290 includes a game server 260 that provides access to multiple video games. The game server 260 can be any type of server computing device available in the cloud and can be configured to execute one or more virtual machines on one or more hosts. For example, the game server 260 can manage virtual machines that support game processors, which instantiate game instances for users. Thus, multiple game processors of the game server 260 associated with multiple virtual machines are configured to execute multiple instances of one or more games associated with multiple users. In this way, the backend server supports streaming media (e.g., video, audio, etc.) of multiple game applications to multiple corresponding users. That is, the game server 260 is configured to stream data (e.g., rendered images and / or frames of the corresponding game) back to the corresponding client device 210 via network 250. In this way, computationally complex game applications can be executed at the backend server in response to controller input received and forwarded by the client device 210. Each server is capable of rendering images and / or frames, then encoding (e.g., compressing) them and streaming them to the corresponding client device for display.

[0041] For example, multiple users can access the cloud gaming network 290 via a communication network 250 using a corresponding client device 210 configured to receive streaming media. In one embodiment, the client device 210 may be configured as a thin client, providing an interface with a backend server (e.g., game server 260 of the cloud gaming network 290) configured to provide computing functions (e.g., including a game title processing engine 211). In another embodiment, the client device 210 may be configured with a game title processing engine and game logic for at least some local processing of video games, and may be further configured to receive streaming content generated by executing video games at the backend server, or for other content supported by the backend server. For local processing, the game title processing engine includes basic processor-based functions for executing video games and services associated with the video games. The game logic is stored on the local client device 210 and is used to execute the video games.

[0042] Specifically, the client device 210 corresponding to the user (not shown) is configured to request access to the game via a communication network 250 (such as the Internet) and to render images generated by the video game executed by the game server 260 for display, wherein encoded images are transmitted to the client device 210 associated with the corresponding user for display. For example, the user can interact with an instance of the video game executed on the game processor of the game server 260 through the client device 210. More specifically, the instance of the video game is executed by the game title processing engine 211. The corresponding game logic (e.g., executable code) 215 implementing the video game is stored and accessible through a data storage area (not shown) and is used to execute the video game. The game title processing engine 211 is capable of supporting multiple video games using multiple game logics, each of which can be selected by the user.

[0043] For example, client device 210 is configured to interact with the associated game title processing engine 211 of the corresponding user's game, such as through input commands used to drive the game. Specifically, client device 210 can receive input from various types of input devices, such as game controllers, tablets, keyboards, gestures captured by cameras, mice, touchpads, etc. Client device 210 can be any type of computing device, having at least a memory and processor module, and capable of connecting to game server 260 via network 250. The backend game title processing engine 211 is configured to generate rendered images, which are transmitted via network 250 for display at a corresponding display associated with client device 210. For example, through a cloud-based service, the game rendered images can be transmitted by an instance of the corresponding game running on the game execution engine 211 of game server 260. That is, client device 210 is configured to receive encoded images (e.g., encoded from game rendered images generated by executing a video game) and to display images rendered for display 11. In one embodiment, display 11 includes an HMD (e.g., displaying VR content). In some implementations, the rendered image can be delivered wirelessly or wired, directly from a cloud-based service or via client device 210 (e.g., Remote Play allows you to stream to your smartphone or tablet.

[0044] In one implementation, the game server 260 and / or the game title processing engine 211 include basic processor-based functions for executing the game and services associated with the game application. For example, processor-based functions include 2D or 3D rendering, physics, physics simulation, scripting, audio, animation, graphics processing, lighting, shading, rasterization, ray tracing, shadows, culling, transformation, artificial intelligence, etc. Furthermore, the services of the game application include memory management, multithreading management, Quality of Service (QoS), bandwidth testing, social networks, social friend management, social network communication with friends, communication channels, SMS, instant messaging, chat support, etc.

[0045] In one implementation, the cloud gaming network 290 is a distributed game server system and / or architecture. Specifically, a distributed game engine executing game logic is configured as a corresponding instance of a game. Typically, the distributed game engine takes each of the functions of the game engine and distributes these functions to numerous processing entities for execution. Individual functions can be further distributed across one or more processing entities. Processing entities can be configured in different configurations, including physical hardware, and / or as virtual parts or virtual machines, and / or as virtual containers, where a container differs from a virtual machine because it virtualizes an instance of a game application running on a virtualized operating system. Processing entities can utilize and / or rely on servers and their underlying hardware on one or more servers (compute nodes) of the cloud gaming network 290, where servers can reside on one or more racks. The coordination, allocation, and management of the execution of these functions by the various processing entities are performed by a distributed synchronization layer. In this way, the execution of these functions is controlled by the distributed synchronization layer to enable the generation of media (e.g., video frames, audio, etc.) for the game application in response to player controller input. The distributed synchronization layer enables these functions to be executed efficiently across distributed processing entities (e.g., through load balancing), allowing critical game engine components / functions to be distributed and reassembled for more efficient processing.

[0046] The game title processing engine 211 includes a central processing unit (CPU) and a graphics processing unit (GPU) group, which can be configured to perform multi-tenant GPU functionality. In another embodiment, multiple GPU devices are combined to perform graphics processing on a single application running on a corresponding CPU.

[0047] Figure 2B This is a diagram according to one embodiment of the present disclosure for providing a game between two or more peer devices, wherein the VSYNC signal can be synchronized and offset to achieve optimal timing for receiving controller and other information between the devices. For example, head-to-head games can be performed using two or more peer devices connected via a network 250 or directly via peer-to-peer communication (e.g., Bluetooth, LAN, etc.).

[0048] As shown in the figure, the game is executed locally on each of the client devices 210 (e.g., game consoles) of the corresponding user playing the video game, where the client devices 210 communicate via peer-to-peer networking. For example, an instance of the video game is executed by the game title processing engine 211 of the corresponding client device 210. The game logic 215 (e.g., executable code) that implements the video game is stored on the corresponding client device 210 and is used to execute the game. For illustrative purposes, the game logic 215 may be transmitted to the corresponding client device 210 via portable media (e.g., optical media) or via a network (e.g., downloaded from a game provider via the Internet).

[0049] In one implementation, the game title processing engine 211 corresponding to client device 210 includes basic processor-based functions for executing the game and services associated with the game application. For example, processor-based functions include 2D or 3D rendering, physics, physics simulation, scripting, audio, animation, graphics processing, lighting, shading, rasterization, ray tracing, shadows, culling, transformation, artificial intelligence, etc. Furthermore, the game application services include memory management, multithreading management, Quality of Service (QoS), bandwidth testing, social networks, social friend management, social network communication with friends, communication channels, SMS, instant messaging, chat support, etc.

[0050] Client device 210 can receive input from various types of input devices, such as game controllers, tablets, keyboards, gestures captured by a camera, mice, touchpads, etc. Client device 210 can be any type of computing device with at least a memory and processor module, and is configured to generate rendered images executed by game title processing engine 211, and to display the rendered images on a display (e.g., display 11, or display 11 including a head-mounted display—HMD, etc.). For example, the rendered image can be associated with an instance of a game running locally on client device 210 to enable gameplay for the corresponding user, such as through input commands used to drive the game. Some examples of client device 210 include personal computers (PCs), game consoles, home theater systems, general-purpose computers, mobile computing devices, tablets, telephones, or any other type of computing device capable of executing game instances.

[0051] Figure 2C Various network configurations that benefit from proper synchronization and offset of the VSYNC signal between the source and target devices according to embodiments of this disclosure are shown, including Figures 2A to 2BThe configurations shown are examples of those network configurations. Specifically, various network configurations benefit from proper frequency alignment of the server's VSYNC signal and the client's VSYNC signal, as well as timing offsets between the server's and client's VSYNC signals, to reduce one-way latency and / or latency variability between the server and client. For example, one network device configuration includes a cloud gaming server (e.g., source) to client (target) configuration. In one embodiment, the client may include a WebRTC client configured to provide audio and video communication within a web browser. Another network configuration includes a client (e.g., source) to server (target) configuration. Yet another network configuration includes a server (e.g., source) to server (e.g., target) configuration. Another network device configuration includes a client (e.g., source) to client (target) configuration, where the clients may each be game consoles to provide, for example, head-to-head gaming.

[0052] Specifically, VSYNC signal alignment may include synchronizing the frequencies of the server VSYNC signal and the client VSYNC signal, and may also include adjusting the timing offset between the client VSYNC signal and the server VSYNC signal to eliminate drift, and / or maintain an ideal relationship between the server VSYNC signal and the client VSYNC signal to reduce one-way latency and / or latency variability. In one embodiment, to achieve proper alignment, the server VSYNC signal may be tuned to achieve proper alignment between the server 260 and client 210 pair. In another embodiment, the client VSYNC signal may be tuned to achieve proper alignment between the server 260 and client 210 pair. Once the client VSYNC signal is aligned with the server VSYNC signal, the server VSYNC signal and the client VSYNC signal occur at substantially the same frequency and are offset from each other by a timing offset that may be adjusted from time to time. In another embodiment, VSYNC signal alignment may include synchronizing the VSYNC frequencies of two clients and may also include adjusting the timing offset between their VSYNC signals to eliminate drift and / or achieve optimal reception timing for controller and other information; either VSYNC signal may be tuned to achieve this alignment. In yet another embodiment, for example, for head-to-head cloud gaming, alignment may include synchronizing the VSYNC frequencies of multiple servers and may also include synchronizing the frequencies of the server's VSYNC signal with the client's VSYNC signal and adjusting the timing offset between the client's VSYNC signal and the server's VSYNC signal. In server-to-client and client-to-client configurations, alignment may include frequency synchronization between the server's VSYNC signal and the client's VSYNC signal, and providing the correct timing offset between the server's VSYNC signal and the client's VSYNC signal. In a server-to-server configuration, alignment may include frequency synchronization between the server's VSYNC signal and the client's VSYNC signal without setting a timing offset.

[0053] Figure 2D This illustration depicts a multi-tenant configuration between a cloud gaming server 260 and one or more clients 210 according to one embodiment of this disclosure. The multi-tenant configuration benefits from proper synchronization and offset of VSYNC signals between the source and target devices. In a server-to-client configuration, alignment may include frequency synchronization between the server's VSYNC signals and the client's VSYNC signals, as well as providing proper timing offset between the server's and client's VSYNC signals. In one embodiment, in the multi-tenant configuration, the client's VSYNC signals are tuned at each client 210 to achieve proper alignment between the server 260 and client 210 pairs.

[0054] For example, in one implementation, the graphics subsystem can be configured to perform multi-tenant GPU functionality, whereby the graphics subsystem can implement graphics and / or rendering pipelines for multiple games. That is, the graphics subsystem is shared among multiple games being executed. Specifically, in one implementation, the game title processing engine may include a CPU and GPU group configured to perform multi-tenant GPU functionality, whereby the CPU and GPU group can implement graphics and / or rendering pipelines for multiple games. That is, the CPU and GPU group is shared among multiple games being executed. The CPU and GPU group may be configured as one or more processing devices. In another implementation, multiple GPU devices are combined to perform graphics processing for a single application executing on a corresponding CPU.

[0055] Figure 3This illustrates the general process of executing a video game at a server to generate game-rendered video frames and sending these frames to a client for display. Traditionally, many operations at game server 260 and client 210 are performed within frame periods defined by corresponding VSYNC signals. For example, server 260 attempts to generate game-rendered video frames at 301 within one or more frame periods defined by corresponding server VSYNC signal 311. The video frames are generated by the game either in response to control information transmitted from an input device at operation 350 (e.g., user input commands) or by game logic not driven by control information. Transmission jitter 351 may exist when sending control information to server 260, where jitter 351 measures the variation in network latency from client to server (e.g., when sending input commands). As shown, the thick arrows indicate the current latency when sending control information to server 260, but due to jitter, the control information at server 260 may have a range of arrival times (e.g., the range defined by the dashed arrows). At flip time 309, the GPU triggers a flip command, indicating that the corresponding video frame has been fully generated and placed in the frame buffer at server 260. Subsequently, server 260 performs a scan output / scan input (operation 302, where the scan output may be aligned with VSYNC signal 311) on the video frame within the subsequent frame period defined by server VSYNC signal 311 (VBI omitted for clarity). The video frame is then encoded (operation 303) (e.g., encoding begins after VSYNC signal 311 appears, and the end of encoding may not be aligned with VSYNC signal 311) and transmitted (operation 304, where transmission may not be aligned with VSYNC signal 311) to client 210. At client 210, the encoded video frame is received (operation 305, where reception may not be aligned with client VSYNC signal 312), decoded (operation 306, where decoding may not be aligned with client VSYNC signal 312), buffered, and displayed (operation 307, where the start of display may be aligned with client VSYNC signal 312). Specifically, client 210 begins displaying each video frame rendered for display from the corresponding occurrence of client VSYNC signal 312.

[0056] The one-way latency 315 can be defined as the time from the start of video frame transmission to the encoding unit at the server (e.g., scan output 302) to the start of video frame display at the client 307. That is, the one-way latency is the time from server scan output to client display, taking into account client buffering. Each frame has a latency from the start of scan output 302 to the completion of decoding 306. This latency may vary from frame to frame due to the high variability of server operations (such as encoding 303 and transmission 304), jitter 352 accompanying network transmission between server 260 and client 210, and client reception 305. As shown in the figure, the bold vertical arrows indicate the current latency when the corresponding video frame is sent to client 210, but due to jitter 352, the video frame at client 210 may have an arrival time range (e.g., the range defined by the dashed arrows). Because one-way latency must be relatively stable (e.g., fairly consistent) for a good playback experience, the traditional result of performing buffer 320 is that individual frame displays with low latency (e.g., from the start of scan output 302 to the completion of decoding 306) are delayed by several frame cycles. That is, if there is network instability or unpredictable encoding / decoding time, additional buffering is needed to keep the one-way latency consistent.

[0057] According to one embodiment of this disclosure, when streaming video frames generated from a video game executed on a server, the one-way latency between the cloud gaming server and the client may vary due to clock drift. Specifically, the frequency difference between the server's VSYNC signal 311 and the client's VSYNC signal 312 may cause the client's VSYNC signal to drift relative to frames arriving from the server 260. This drift may be due to minute differences in the crystal oscillators used in each of the corresponding clocks at the server and client. Furthermore, embodiments of this disclosure reduce the one-way latency by performing synchronization and offset of the VSYNC signals for alignment between the server and client, by providing dynamic buffering on the client, by overlapping the encoding and transmission of video frames at the server, by overlapping the reception and decoding of video frames at the client, and by overlapping the decoding and display of video frames at the client.

[0058] Figure 4 This illustration depicts a data stream, according to an embodiment of the present disclosure, during streaming of video frames generated from a video game executed on a server, via a network configuration including a highly optimized cloud gaming server 260 and a highly optimized client 210. Overlapping server and client operations reduces one-way latency, and synchronizing and offsetting VSYNC signals between the server and client reduces one-way latency and the variability of one-way latency between the server and client. Specifically, Figure 4This indicates the desired alignment between the server's VSYNC signal and the client's VSYNC signal. In one embodiment, tuning of the server's VSYNC signal 311 is performed to achieve proper alignment between the server's and client's VSYNC signals, such as in a server-to-client network configuration. In another embodiment, tuning of the client's VSYNC signal 312 is performed to achieve proper alignment between the server's and client's VSYNC signals, such as in a multi-tenant server-to-multi-client network configuration. For illustrative purposes, Figure 4 The document describes the tuning of the server VSYNC signal 311 for synchronizing the frequencies of the server VSYNC signal and the client VSYNC signal, and / or adjusting the timing offset between the corresponding client VSYNC signal and the server VSYNC signal. However, it is understood that the client VSYNC signal 312 can also be used for tuning. In the context of this patent, "synchronization" should be understood as meaning tuning the signals to make their frequencies match, but their phases may differ; "offset" should be understood as meaning the time delay between signals, for example, the time between when one signal reaches its maximum value and when another signal reaches its maximum value.

[0059] As shown in the figure Figure 4 An improved process is illustrated in an embodiment of this disclosure for executing a video game at a server to generate rendered video frames and sending these video frames to a client for display. The process is illustrated regarding the generation and display of individual video frames at both the server and client. Specifically, the server generates game-rendered video frames at 401. For example, server 260 includes a CPU (e.g., game title processing engine 211) configured to execute the game. The CPU generates one or more draw calls for the video frames, wherein the draw calls include commands placed in a command buffer for execution in the graphics pipeline by the corresponding GPU of server 260. The graphics pipeline may include one or more shader programs on the vertices of objects within the scene to generate texture values ​​for rendering as video frames for display, wherein the operations are performed in parallel by the GPU for improved efficiency. At flip time 409, the GPU touches a flip command in the command buffer, the flip command indicating that the corresponding video frame has been fully generated and / or rendered and placed in the frame buffer at server 260.

[0060] At 402, the server performs a scan output to the encoder for the game-rendered video frame. Specifically, the scan output is performed line by line or in groups of consecutive scan lines, where a scan line refers to a single horizontal line, such as from one edge of the display screen to the other. These scan lines or groups of consecutive scan lines are sometimes referred to as slices, and are referred to as screen slices in this specification. Specifically, scan output 402 may include a number of processes for modifying the game-rendered frame, including overlaying it with another frame buffer or shrinking it to surround it with information from another frame buffer. During scan output 402, the modified video frame is then scanned into the encoder for compression. In one embodiment, scan output 402 is performed at the occurrence 311a of the VSYNC signal 311. In other embodiments, scan output 402 may be performed before the occurrence of the VSYNC signal 311, such as at flip time 409.

[0061] At 403, the game-rendered video frame (which may have been modified) is encoded at the encoder on a slice-by-slice basis to generate one or more encoded slices, where the encoded slices are independent of scan lines or screen slices. Therefore, the encoder generates one or more encoded (e.g., compressed) slices. In one embodiment, the encoding process begins before the scan output 402 process of the corresponding video frame has been fully completed. Furthermore, the start and / or end of encoding 403 may or may not be aligned with the server VSYNC signal 311. The boundary of the encoded slice is not limited to a single scan line and may consist of a single scan line or multiple scan lines. Furthermore, the end of the encoded slice and / or the start of the next encoder slice may not necessarily occur at the edge of the display screen (e.g., it may occur somewhere in the middle of the screen or in the middle of a scan line), so that the encoded slice does not need to traverse the display screen completely from edge to edge. As shown, one or more encoded slices may be compressed and / or encoded, including a compressed "encoded slice A" with a hash marker.

[0062] At 404, the encoded video frame is transmitted from the server to the client, wherein the transmission can occur on a slice-by-slice basis, where each slice is a compressed encoder slice. In one embodiment, transmission 404 begins before the corresponding video frame has fully completed encoding 403. Furthermore, the start and / or end of transmission 404 may or may not be aligned with the server's VSYNC signal 311. As shown, the compressed encoded slice A is transmitted to the client independently of other compressed encoder slices of the rendered video frame. Encoder slices can be transmitted one at a time or in parallel.

[0063] At 405, the client again receives compressed video frames on a slice-by-slice basis. Furthermore, the start and / or end of reception 405 may or may not be aligned with the client's VSYNC signal 312. As shown, compressed slice A is received by the client. Transmission jitter 452 may exist between server 260 and client 210, where jitter 452 measures the variation in network latency from server 260 to client 210. Lower jitter values ​​indicate a more stable connection. As shown, the bold straight arrows indicate the current latency when sending the corresponding video frame to client 210, but due to jitter, there may be a range of arrival times for video frames at client 210 (e.g., the range defined by the dashed arrows). Variations in latency may also be due to one or more operations at the server, such as encoding 403 and transmission 404, and networking issues that introduce latency when transmitting video frames to client 210.

[0064] At 406, the client again decodes the compressed video frame on a slice-by-slice basis, producing a decoded slice A (displayed as a hash-free tag) now ready for display. In one embodiment, the decoding process 406 begins before the corresponding video frame has fully completed the reception process 405. Furthermore, the start and / or end of decoding 406 may or may not be aligned with the client's VSYNC signal 312. At 407, the client displays the decoded rendered video frame on its display. That is, for example, the decoded video frame is placed in a display buffer and streamed to the display device on a scan-line basis. In one embodiment, the display process 407 (i.e., streaming to the display device) begins after the corresponding video frame has fully completed the decoding process 406, i.e., the decoded video frame is fully residing in the display buffer. In another embodiment, the display process 407 begins before the corresponding video frame has fully completed the decoding process 406. That is, streaming to the display device begins from the address of the display buffer while only a portion of the decoded frame buffer is residing in the display buffer. The display buffer is then updated or filled with the remaining portions of the corresponding video frames for display, ensuring that the update of the display buffer is performed before these portions are streamed to the display. Furthermore, the start and / or end of display 407 are aligned with the client's VSYNC signal 312.

[0065] In one embodiment, the one-way latency 416 between server 260 and client 210 can be defined as the elapsed time between the start of scan output 402 and the start of display 407. Embodiments of this disclosure can align the VSYNC signals between the server and client (e.g., synchronization frequency and adjustment offset) to reduce the one-way latency between the server and client, and to reduce the variability of the one-way latency between the server and client. For example, embodiments of this disclosure can calculate the optimal adjustment of the offset 430 between the server VSYNC signal 311 and the client VSYNC signal 312 so that even if the time required for server processing (e.g., encoding 403 and transmission 404) is close to worst-case, the network latency between server 260 and client 210 is close to worst-case, and the client processing (e.g., receiving 405 and decoding 406) is close to worst-case, the decoded and rendered video frames can be used in time for the display process 407. That is, it is not necessary to determine the absolute offset between the server VSYNC and the client VSYNC; it is sufficient to adjust the offset so that the decoded and rendered video frames can be used in time for the display process.

[0066] Specifically, the frequencies of the server VSYNC signal 311 and the client VSYNC signal 312 can be aligned through synchronization. Synchronization is achieved by tuning either the server VSYNC signal 311 or the client VSYNC signal 312. For illustrative purposes, the tuning of the server VSYNC signal 311 is described; however, it is understood that tuning can be performed instead on the client VSYNC signal 312. For example, as... Figure 4 As shown, the server frame period 410 (e.g., the time between two occurrences 311c and 311d of the server VSYNC signal 311) is substantially equal to the client frame period 415 (e.g., the time between two occurrences 312a and 312b of the client VSYNC signal 312), which indicates that the frequencies of the server VSYNC signal 311 and the client VSYNC signal 312 are also substantially equal.

[0067] To maintain frequency synchronization between the server and client VSYNC signals, the timing of the server VSYNC signal 311 can be manipulated. For example, the vertical blanking interval (VBI) in the server VSYNC signal 311 can be increased or decreased over a period of time, such as to handle drift between the server VSYNC signal 311 and the client VSYNC signal 312. Manipulating the vertical blanking (VBLANK) lines in the VBI provides the ability to adjust the number of scan lines used for VBLANK for one or more frame periods of the server VSYNC signal 311. Decreasing the number of scan lines for VBLANK reduces the corresponding frame period (e.g., time interval) between two occurrences of the server VSYNC signal 311. Conversely, increasing the number of scan lines for VBLANK increases the corresponding frame period (e.g., time interval) between two occurrences of the VSYNC signal 311. In this way, the frequency of the server VSYNC signal 311 is adjusted so that the frequency of the client VSYNC signal 312 is aligned to substantially the same frequency as the server VSYNC signal 311. Furthermore, the offset between the server's VSYNC signal and the client's VSYNC signal can be adjusted by briefly increasing or decreasing the VBI and then restoring the VBI to its original value. In one embodiment, the server's VBI is adjusted. In another embodiment, the client's VBI is adjusted. In yet another embodiment, instead of two devices (server and client), there are multiple connected devices, each of which may have a corresponding VBI that is adjusted. In one embodiment, each of the multiple connected devices may be an independent peer device (e.g., no server device). In another embodiment, the multiple devices may include one or more server devices and / or one or more client devices arranged in one or more server / client architectures, multi-tenant server / client(multiple) architectures, or a combination thereof.

[0068] Alternatively, in one implementation, the server's pixel clock (e.g., located at the southbridge of the server's northbridge / southbridge core logic chipset, or, in the case of a discrete GPU, generating its own pixel clock using its own hardware) can be manipulated to perform coarse and / or fine adjustments to the frequency of the server VSYNC signal 311 over a period of time, restoring frequency synchronization alignment between the server VSYNC signal 311 and the client VSYNC signal 312. Specifically, the pixel clock in the server's southbridge can be overclocked or underclocked to adjust the overall frequency of the server's VSYNC signal 311. In this way, the frequency of the server VSYNC signal 311 is adjusted to align the frequency of the client VSYNC signal 312 with that of the server VSYNC signal 311 to substantially the same frequency. The offset between the server VSYNC and the client VSYNC can be adjusted by increasing or decreasing the client-server pixel clock over a short period of time and then restoring the pixel clock to its original value. In one implementation, the server pixel clock is adjusted. In another implementation, the client pixel clock is adjusted. In yet another implementation, instead of two devices (server and client), there are multiple connected devices, each of which may have an adjusted corresponding pixel clock. In one implementation, each of the multiple connected devices may be an independent peer device (e.g., no server device). In another implementation, the multiple connected devices may include one or more server devices and one or more client devices arranged in one or more server / client architectures, multi-tenant server / client(multiple) architectures, or a combination thereof.

[0069] Figure 5A This diagram illustrates the overlap of encoding and transmission operations at server 260 and the overlap of receiving and decoding operations at client 210 when streaming video frames generated from a video game executed on a server, according to one embodiment of this disclosure. Specifically, the overlap of encoding and transmission operations for video frames at server 260 and the overlap of receiving and decoding operations for video frames at client 210 can be seen at the encoder slice level with reference to timeline 501.

[0070] Specifically, the rendered video frames are generated by server 260, which is executing a video game. For example, video frames are generated when the video game is executed on the GPU graphics pipeline of CPU-combined server 260. Each video frame can be divided into one or more encoder slices, which are then compressed into one or more encoded slices. As mentioned earlier, the boundaries of encoded slices are not limited to a single scan line and can consist of a single scan line or multiple scan lines. In addition, the end of an encoded slice and / or the beginning of the next encoded slice may not necessarily occur at the edge of the display screen (e.g., it may occur somewhere in the middle of the screen or in the middle of a scan line).

[0071] Furthermore, each encoded slice (e.g., compressed) can include one or more data packets (e.g., User Datagram Protocol – UDP). If an encoded slice exceeds the Maximum Transmission Unit (MTU), it is fragmented into multiple data packets, each resized to the MTU or smaller. The MTU defines the maximum size of a data packet when sent via UDP. For example, encoded slice A can be fragmented into one or more 503-A data packets. Similarly, encoded slices B, C, and D can each be fragmented into one or more data packets.

[0072] At server 260, one or more packets of encoded slices can be compressed and transmitted individually (and possibly grouped by slice) without waiting for all encoder slices (e.g., video frames) to be compressed (e.g., compressed into encoded slices) and packaged. This may require sharing information within server 260 (e.g., the amount of data encoded at a given time), such as between encoding units and those units performing data transmission (e.g., streaming at the application layer). Specifically, at operation 403, the encoder receives the generated video frames and encodes the video frames on a slice-by-slice basis to generate one or more encoded slices. In some embodiments, the encoder receives video frames generated by a video game that have been modified with additional or overlay layers (e.g., during the scan output process), as previously described. As shown, each of the encoded slices of the generated video frames is compressed individually at the encoder. For example, during the encoding process of operation 403, the generated video frames are encoded into encoded slice A (e.g., compressed), encoded slice B, encoded slice C, and encoded slice D. Depending on the number of processors or hardware, in some embodiments, multiple slices can be encoded simultaneously (e.g., in parallel). For example, some game consoles can generate four encoded slices in parallel. More specifically, due to hardware pipelining, hardware encoders can be configured to compress multiple encoder slices in parallel (e.g., to generate one or more encoded slices).

[0073] At operation 404, the encoded slice is transmitted to client 210 via the network. Specifically, the encoded slice is transferred from the encoder to a streaming device configured to transmit the encoded slice to client 210 via the network. For example, the streaming device is configured to operate on an application layer of Transmission Control Protocol / Internet Protocol or a UDP / IP computer networking model. As shown, according to one embodiment of this disclosure, the encoding process and the transmission process can overlap. For example, each of the encoded slices (e.g., compressed) is transmitted individually, such that one or more packets of the corresponding encoded slice are sent as a group before packets of the next encoded slice can be sent. Specifically, once compressed, the corresponding encoded slice (e.g., one or more packets of the encoded slice) is then ready for transmission by the streaming device at operation 404 without waiting for all encoder slices of the corresponding video frame to be compressed. As shown, in timeline 501, encoded slice A is transmitted by the streaming device while encoder slice B is compressed by the encoder (e.g., compressed into encoded slice B), and before encoder slices C and D have started their encoding process. More specifically, before encoder slice B has been compressed by the encoder, the data packets of encoded slice A are transmitted to client 210 via the network by the streaming device. Similarly, encoded slice B is transmitted by the streaming device while encoder slice C is compressed by the encoder, and before the encoding process of encoder slice D begins. Where multiple slices can be encoded simultaneously (e.g., X slices encoded in parallel), the X encoded slices can be sent while the next X encoder slices are being encoded (e.g., compressed). Therefore, in embodiments of this disclosure, data transmission from server to client (e.g., encoded slices and / or data packets corresponding to encoded slices) can begin once the data packets corresponding to the encoder slices and / or encoder slices have been encoded and / or compressed, and before the entire video frame is encoded (as is conventionally done). In this way, once the value data of the data packets is encoded by the server, the one-way latency between server and client can be reduced by starting data transmission from server to client.

[0074] At client 210, the process includes receiving compressed video frames (e.g., encoded slices of video frames) and then decoding the compressed video frames. At the client, the receiving and decoding processes can also be performed at the encoded slice level, such that as soon as one or more data packets corresponding to a encoded slice are received, those data packets are used to decode the encoded slice. Therefore, the receiving and decoding processes can overlap at the client. Specifically, at client 210, one or more encoded slices corresponding to the video frames are received at operation 405. For illustrative purposes, in timeline 501, encoded slice A can be received first, then encoded slice B, and so on, until finally encoded slice D is received.

[0075] Specifically, at operation 406, the received coded slice is subsequently decoded. Specifically, once received, the corresponding coded slice is ready for decoding by the decoder at operation 406 without waiting for all coded slices of the corresponding video frame to be received. For example, in timeline 501, while receiving coded slice B, and while coded slices C and D are still being transmitted over the network, one or more packets of coded slice A are decoded. Similarly, while receiving coded slice C, and while coded slice D is being transmitted over the network, one or more packets of coded slice B are decoded. Therefore, in embodiments of this disclosure, decoding of data (e.g., coded slices and / or packets of coded slices) from the server to the client can begin once coded slices and / or packets of coded slices are received, and before the complete complement of coded slices of video frames and / or corresponding packets of those coded slices have been received. In this way, the one-way waiting time between server 260 and client 210 can be reduced by starting data decoding once the value data of the packets is received by client 210.

[0076] It is understood that encoded slices of the corresponding video frame can be received at the client in various random orders (e.g., due to network jitter) and can be received in parallel depending on the hardware configuration at the client. That is, in operation 405, due to jitter, the transmitted encoded slices may be received out of order at client 210. In this case, the encoded slices may be decoded out of order to account for the jitter (e.g., the encoded slices of the corresponding video frame being received out of order). For example, in a sequence of encoded slices of the corresponding video frame transmitted over the network, when an encoded slice is received out of order, decoding is also performed out of order. That is, decoding still occurs as each encoded slice is received.

[0077] More specifically, each of the encoded slices corresponding to a video frame can be segmented into one or more network packets at the encoder of server 260 and then transmitted over the network to client 210. Therefore, client 210 receives one or more data packets for each of the one or more slices of the corresponding video frame before receiving the slice of the next video frame and / or the corresponding data packets of the slice. That is, in one embodiment, one or more encoded slices of the corresponding video frame can be received at the client one at a time or in parallel, and this occurs before receiving one or more encoded slices of the next video frame. More specifically, the data packets for one or more encoded slices of the corresponding video frame can be received at the client one at a time or in parallel, and this occurs before receiving one or more encoded slices of the next video frame.

[0078] Through the Figures 2A to 2DDetailed description of various client devices 210 and / or cloud gaming networks 290 (e.g., in game servers 260), Figure 5B Flowchart 500B illustrates a method for cloud gaming according to one embodiment of the present disclosure, wherein server operations can be overlapped to reduce one-way waiting time between the cloud gaming server and the client.

[0079] At 510, the method includes generating video frames while executing a video game on a server. For example, the server can execute the video game in streaming mode, causing the server's CPU to execute the video game in response to input commands from a user, thereby generating game-rendered video frames using a graphics pipeline available for streaming. Specifically, the CPU executing the video game, in collaboration with the GPU graphics pipeline, is configured to generate multiple video frames. In cloud gaming, game-generated video frames are typically rendered for display on a virtual display. The server can perform additional operations on the game-generated video frames. For example, one or more overlays can be added to the corresponding game-generated video frames, such as during a scan output process.

[0080] At 520, the method includes scanning game-generated video frames line-by-line into an encoder at the server. As previously described, the game-generated video frames can be modified during the scanning output process, such as by adding one or more overlays. Therefore, the selectively modified generated video frames are scanned into the encoder for compression in preparation for streaming the video frames to the client.

[0081] At 530, the method includes encoding one or more encoder slices of a corresponding video frame into one or more coded slices. That is, the video frame is compressed into a compressed video frame comprising one or more coded slices. As previously described, each coded slice can be configured as one or more scan lines, wherein the minimum size of the coded slice is one scan line, and wherein the end of the coded slice may not necessarily appear at the edge of the display screen.

[0082] After encoding, one or more encoded slices are transferred to a buffer. The buffer's fill rate depends on the encoder's capacity, so each encoded slice is transferred to the buffer at the encoder's fill rate. For example, the buffer may be filled as quickly as the encoder is able to encode video frames, and more specifically, as quickly as the encoder is able to generate encoded slices and / or packets of encoded slices.

[0083] Furthermore, one or more forward error correction (FEC) slices can be generated for the corresponding video frame being compressed by the encoder. The client can use one or more FEC slices to correct any coded slices lost during transmission to the client. In one implementation, one or more FEC slices can be interleaved within one or more coded slices during streaming. That is, for a corresponding video frame with multiple coded slices and multiple FEC slices, the FEC slices can be interleaved within the coded slices during streaming. For example, FEC slices used to correct a set of coded slices can be transmitted immediately after each of the coded slices in the set has been transmitted.

[0084] At 540, the method includes ejecting a first encoded slice from a buffer to stream the first encoded slice to a client over a network at a buffer ejection rate. That is, the first encoded slice is transmitted over the network to a client configured to receive and decompress compressed video frames for rendering to a display. More specifically, one or more packets of the first encoded slice are ejected from the buffer at an ejection rate for streaming. In one embodiment, at least one encoded slice of a video frame is ejected before the entire video frame is encoded. In one embodiment, the first encoded slice is streamed over the network before one or more slices of the corresponding video frame are fully encoded.

[0085] In one implementation, the overflow rate is dynamically adjusted to approximately equal to the maximum receive bandwidth measured by the client. Specifically, the transmission of multiple coded slices of compressed video frames is performed by coded slices spaced according to the maximum receive bandwidth measured by the client. In this way, in one implementation, the buffer overflow rate is decoupled from the encoder fill rate. Specifically, Figure 6 This is a diagram illustrating the spacing of multiple coded slices of multiple video frames according to one embodiment of the present disclosure, wherein the spacing is based on the maximum receiving bandwidth of the client 210 when streaming video frames generated from a video game executed from a server.

[0086] In one implementation, it is advantageous to tune the encoder / decoder (codec) 610 or use a codec such that the packet size of the corresponding encoded slice corresponds to the natural amount of data created by the codec, thereby reducing buffering and latency. An encoded slice (e.g., compressed) may include one or more packets, which are then transmitted over the network to the client. In one implementation, the packet size is associated with the Maximum Transmission Unit (MTU). When a compressed (as an encoded slice) video encoder slice exceeds the MTU, the encoded slice is split into several packets, each resized to be equal to or less than the MTU.

[0087] One consequence of encoding slices larger than the MTU is an increased packet loss rate and visual corruption when packet loss occurs. For example, if a compression encoder slice is split into three packets during encoding, and one of the packets is lost en route to the client, the entire encoder slice becomes invalid at the client, meaning the loss of a single packet results in the effective loss of three packets. Therefore, in one implementation, the packet size can be tuned to the natural amount of data (e.g., MTU) created by the codec. In one implementation, the packet size is tuned to correspond to the UDP packet size. In another implementation, the packet size can be tuned to correspond to a screen slice (e.g., from edge to edge of the screen), where the packet size and encoding slice correspond to the MTU. That is, the encoding slice is resized to the MTU and can be transmitted as a single packet. This increases the number of encoding slices corresponding to a video frame. When the packet size is tuned to correspond to the natural factors in the codec on the server side (e.g., MTU size and / or encoding slice), it makes it possible to decode packets on the client side as soon as each packet arrives.

[0088] In one implementation, compressed video frames are transmitted from a server to a client at a rate based on the maximum available bit rate or bandwidth on the network at specific points in time. That is, the transmission rate of the encoded slices of the compressed video frames is dynamically adjusted based on the currently measured maximum bandwidth. Therefore, data packets 655 of the encoded slice 650 are spaced out to be transmitted to the client according to the maximum available bit rate or maximum bandwidth. That is, instead of spacing the data packets according to the codec's bit rate (e.g., coupling the encoder fill rate 615 to the buffer overflow rate 635), the data packets 655 are transmitted according to the client 210's maximum bandwidth (e.g., decoupling the encoder fill rate 615 from the buffer overflow rate 635) at a buffer overflow rate 635 associated with the client's maximum bandwidth. In this way, data packets 655 can be received at a higher rate at the client 210. In one implementation, the encoding of the data packets is performed at the same rate at which they are transmitted, such that both operations are dynamically tuned to the maximum available bandwidth at the client.

[0089] In addition, at the server level, supplementary information may include the maximum transmission rate, which defines the total network throughput (e.g., bandwidth) that the server considers available to the client. This maximum transmission rate can be determined by the server by tracking the amount of bandwidth sent to the client for a series of packets, and messages from the client to the server indicating how many packets were lost for the same series. This can be used to determine the maximum rate at which encoded slices of video frames can be sent. The maximum rate fluctuates depending on the stability of the network connection with the client, and the offset can be dynamically adjusted to accommodate this fluctuation. Furthermore, the maximum transmission rate can be adjusted independently of encoder parameters, so that slices can be sent out faster if the encoder is configured not to generate slices at the maximum transmission rate.

[0090] In one implementation, the maximum bandwidth at the client is determined through a feedback mechanism from the client. Specifically, the streaming transmitter 620, operating at the application layer, measures the client's maximum bandwidth, such as using a bandwidth tester 625. As previously described, the application layer is used for the UDP protocol suite for interconnecting network devices over the Internet. For example, the application layer defines communication protocols and interface methods for communication between devices over IP communication networks. During testing, the streaming transmitter 620 provides additional FEC packets 640, enabling the buffer 630 (e.g., a first-in / first-out FIFO buffer) to stream packets at a predefined bit rate (such as the maximum bandwidth being tested). In one implementation, the client returns the number of packets it receives within a range of incremental sequence identifiers (IDs) (such as the range for video frames) as feedback 690 to the streaming transmitter 620. For example, the client might report something like: 145 out of 150 video frames with sequence IDs from 100 to 250 (e.g., 150 video frames). In this way, the streaming device 620 at server 260 can calculate packet loss, and since the streaming device 620 knows the amount of bandwidth being transmitted during the packet sequence (e.g., during testing), it can dynamically determine the client's maximum bandwidth at a given moment. The client's measured maximum bandwidth can be transmitted from the streaming device 620 to the buffer 630 as control information 627, allowing the buffer 630 to dynamically transmit packets at a rate approximately equal to the client's maximum bandwidth.

[0091] Client 210 cannot make this determination because the amount of bandwidth being transmitted fluctuates constantly due to variable bit rates, scene complexity, and other factors. Therefore, client 210 does not know whether the server is transmitting the maximum bandwidth that the client can handle at any given moment. For example, the maximum bandwidth might be 15 Mbps (megabits per second), but because scene complexity might be low, such as the user being on a menu, only 2 Mbps might be transmitted. Therefore, if the client reports 0% packet loss, this does not tell server 260 whether client 210 can handle the maximum bandwidth of 15 Mbps. Thus, the true maximum bandwidth can only be determined when server 260 transmits the maximum bandwidth.

[0092] Figure 7A This diagram illustrates how, according to one embodiment of the present disclosure, the receiving and decoding operations at the client are overlapped when streaming video frames generated from a video game executed on a server to reduce one-way latency between the server and the client. Furthermore, according to one embodiment of the present disclosure, the receiving, decoding, and decompression of video frames for rendering for display (e.g., display operations) can be overlapped at the client to reduce one-way latency between the server and the client.

[0093] Specifically, client 210 receives compressed video frame 705 at operation 405, where a server (not shown) executes a video game to generate game-rendered video frames. These game-rendered video frames are then encoded at an encoder on the server and transmitted to the client as compressed video frame 705. As previously described, compressed video frame 705 includes one or more encoded slices, such as encoded slice A (which is hashed). For example, compressed video frame 705 includes encoded slices A through G. Each of the encoded slices can have a different size. For example, encoded slice D is larger than encoded slice E. Furthermore, each of the slices can be divided into one or more data packets.

[0094] Client 210 includes a decoder configured to decode one or more encoded slices (e.g., decoded slice A – which is not hashed) of the corresponding compressed video frame at operation 406. In one embodiment, the decoding process begins before the client has fully received the corresponding compressed video frame. That is, decoding of the encoded slices of the corresponding video frame begins before all encoded slices of the corresponding video frame are received. Because the decoder decodes on a slice-by-slice basis, the decoded video frame 706 includes one or more encoder slices (when decoded). The decoded video frame 706 is then prepared for display, such as rendering the information in the decoded video frame 706 as scan lines or screen slices. Therefore, the client renders video frame 707 ready for display.

[0095] According to one embodiment of this disclosure, in a fault-tolerant mode, the one-way latency between the server and the client can be reduced by having the client 210 begin displaying the video frame at operation 407 before fully decoding the video frame at operation 406. Specifically, one or more decoded slices of the video frame can be prepared for rendering to the display before the compressed video frame has been fully decoded. That is, the display operation at 407 overlaps with the decoding operation at 406. Specifically, the first encoded slice (e.g., slice A) must arrive and be decoded before the client begins scanning the output to the display. Furthermore, all subsequent encoded slices must arrive and be decoded before their respective decompressed data is rendered and scanned for display.

[0096] Furthermore, besides overlapping the receiving and decoding operations at the client end, one or more decoded slices that are subsequently rendered and ready for display can be displayed even before the client 210 has fully received the compressed video frame sent by the server. In other words, for a given video frame, one or more of the receiving, decoding, and display operations at the client end can be overlapped.

[0097] Furthermore, according to one embodiment of this disclosure, when multiple operations overlap at the server and client, one or more decoded slices of video frames subsequently rendered for display can be displayed at the client even before the scan output operation at the server is fully completed. As previously described, the scan output transmits game-rendered video frames to the encoder at the server, wherein the game-rendered video frames can be optionally modified (e.g., by adding overlays) during the scan output process.

[0098] Through the Figures 2A to 2D Detailed description of various client devices 210 and / or cloud gaming networks 290 (e.g., in game servers 260), Figure 7B Flowchart 700B illustrates a method for cloud gaming according to one embodiment of the present disclosure, wherein client operations can be overlapped to reduce one-way latency between the cloud gaming server and the client. Specifically, Figure 7B This is a flowchart illustrating the overlap of receive and decode operations at the client when streaming video frames generated from a video game executed on a server to reduce one-way latency between the server and the client, wherein rendering of decompressed video frames may also optionally overlap with the receive and decode operations.

[0099] At 710, the method includes receiving one or more encoded slices of compressed video frames at the client. Specifically, in response to the server processing a video game in streaming mode during a session with the client, multiple video frames are generated at the server. The multiple video frames are encoded at an encoder into multiple compressed video frames, each corresponding compressed video frame including one or more corresponding encoded slices. The multiple compressed video frames are transmitted to the client over a network, and more specifically, the encoded slices of the multiple compressed video frames are transmitted to the client.

[0100] At the client end, the method includes receiving multiple compressed video frames at a decoder and decompressing the multiple compressed video frames for rendering to a display associated with the client, wherein these operations can be overlapped. Specifically, at 720, the method includes decoding a first encoded slice at the client before fully receiving one or more encoded slices corresponding to the compressed video frame.

[0101] Figure 8A This diagram illustrates the interleaving of forward error correction (FEC) packets within coded slices of a video frame during streaming of a video frame generated from a video game executed on a server, according to one embodiment of this disclosure. The interleaving of FEC data between coded slices of corresponding video frames allows for overlap of reception, decoding, and rendering operations at the client, which can be performed on a slice-by-slice basis. Specifically, forward error correction provides for the reconstruction of lost coded slices of a corresponding video frame and / or the reconstruction of lost packets from a coded slice. Therefore, forward error correction allows for the reception, decoding, and rendering of coded slices for display at the client before one or more subsequent coded slices have been received and / or decoded.

[0102] For example, a coded slice can be divided into one or more network packets. FEC packets can be used to correct one or more packets associated with a slice. Thus, FEC packets can be interleaved within packets of the corresponding compressed video frame (e.g., within packets of the coded slice) to accelerate the recovery of lost packets using FEC. That is, instead of placing FEC packets at the end of the packet group of the corresponding compressed video frame, FEC packets are interleaved within the packets. In this way, once one or more packets of the corresponding compressed video frame are lost, those lost packets can be recovered almost immediately using the corresponding FEC packets. Specifically, lost packets of the corresponding compressed video frame can be recovered without waiting for the entire packet set to be transmitted. This is because the corresponding FEC packets are transmitted after the packets that the FEC packets can recover. Therefore, FEC packets received earlier (e.g., during the transmission of the coded slice of the corresponding compressed video frame) and possibly before all packets of the compressed video frame are received can be used to recover one or more lost packets. In other words, in embodiments of this disclosure, forward error correction can be performed earlier (e.g., during the reception of a compressed video frame) to correct for lost and / or corrupted packets of coded slices, without waiting for the client to receive the entire set of packets (e.g., data and FEC) of the compressed video frame. Typically, forward error correction for any packets of a compressed video frame is performed after all packets of that compressed video frame have been received—thus increasing the one-way latency between the server and the client. In this embodiment of the disclosure, forward error correction is performed while the client is still receiving the compressed video frame—thereby reducing the one-way latency. This provides overlapping decoding and display operations at the client by interleaving FEC packets within the packets of the corresponding compressed video frame.

[0103] Furthermore, according to one embodiment of this disclosure, when packets are tuned to the MTU, this allows for easier recovery from packet loss. That is, since packets are tuned and / or resized to coded slices (i.e., a one-to-one relationship between coded slices and corresponding packets), the corresponding FEC can be used to recover the lost complete coded slice. In this way, FEC creation is performed across packets, making the FEC packets available for correcting one or more corresponding packets, where each packet corresponds to a coded slice. Figure 8A As shown, each of the encoded slices is resized to be equal to or less than MTU 850, such that each encoded slice is transmitted over the network using a corresponding data packet.

[0104] Figure 8AThis illustrates the interleaving of FEC packets within the data packets of a compressed encoder slice of an encoded video frame according to one embodiment of this disclosure. For example, the encoded video frame may include encoded slice 0, encoded slice 1, encoded slice 2, and encoded slice 3. Additionally, the encoded slices may be resized to an MTU of 850 or smaller. As shown, FEC-0 and FEC-1 are interleaved with the data packets of encoded slice 0, encoded slice 1, encoded slice 2, and encoded slice 3.

[0105] Specifically, during compression, a data packet 810a is created for encoded slice 0, where encoded slice 0 is resized to compress into a single data packet. Encoded slice 0 may include a padding portion 810b for FEC creation and recovery. It is important to note that only data packet 810a is transmitted from the server to the client, because in one implementation, the padding portion 810b is stripped before transmission, but can be considered at the client for recovery purposes. Similarly, during compression, a data packet 811a is created for encoded slice 1, where encoded slice 1 may be resized to compress into a single data packet. Encoded slice 1 does not include the corresponding padding portion for FEC creation, as it is the largest data packet in the set of data packets recovered during FEC, as described below.

[0106] FEC-0 is generated as packet 820 and transmitted after packets from coded slice 0 and coded slice 1. FEC-0 820 can be used to correct each of those coded slices that may be lost, and / or can be used to correct lost packets from each of those compressed coded slices. The packet size of FEC-0 packet 820 can specify the size of the padding portion of coded slice 0 and coded slice 1 that is compressed for the purpose of performing FEC creation and recovery. That is, the FEC packet is created according to block size adjustment, wherein the size of the FEC packet is tuned to the size of the largest packet in the set of packets that can be corrected using the corresponding FEC packet. Therefore, FEC creation requires knowledge of the packet size adjustment in the set. However, it is not necessary to wait for each packet in the set of packets corresponding to the FEC packet to be generated before any packets in the set are transmitted for the purpose of creating FEC; the packets in the set can be transmitted immediately upon compression.

[0107] Specifically, packet 810a of encoded slice 0 can be transmitted immediately after it is compressed at the server's encoder, without waiting for FEC-0 to be generated, and can be stored or buffered for FEC creation. Similarly, packet 811a of encoded slice 1 can also be transmitted immediately after it is compressed at the server's encoder, again without waiting for FEC-0 to be generated, and can be stored or buffered for FEC creation. Because the FEC packets are created using block size adjustment, the size of each packet needs to be communicated before creating a corresponding FEC packet that can be used to recover any of those potentially lost packets.

[0108] In one implementation, the size of each packet is included in the header of the corresponding packet that needs to be read to create the FEC. In another implementation, because each of packets 810a and 811a stores the encoded slices, the size adjustment of each of these packets is known or determinable, rather than conveying the size of each packet in the header of the corresponding packet that needs to be read to create the FEC. Once the size of the largest known packet in the set of packets that can be corrected using the corresponding FEC packet is obtained, the block size of the FEC-0 packets can be tuned to the maximum value in packets 810a and 811a when generating FEC-0.

[0109] Furthermore, once the size of FEC-0 is known, the padding size of the packets in the packet set that can be corrected using FEC-0 for FEC creation and data recovery purposes is also known. For example... Figure 8A As shown, FEC-0 is tuned to the size of packet 811a of coded slice 1, which is the largest packet in the set containing coded slice 0 and coded slice 1. Therefore, coded slice 0 requires padding 810b for FEC creation and / or data recovery. Because the size of FEC-0 packet 820 is tuned to the size of packet 811a of coded slice 1 (i.e., the largest packet in the set of packets that can be corrected using FEC-0), packet 811a of coded slice 1 does not require any padding for FEC creation and / or data recovery.

[0110] In another embodiment, the padding portion can be set to the MTU of one embodiment. Specifically, the FEC packets are normalized to the size of the MTU. Furthermore, for FEC creation and / or data recovery purposes, each packet in the corresponding FEC-corrected packet set can also be padded to the MTU. For example, for FEC creation and / or data recovery purposes, FEC-0 will be resized to the MTU, coded slice 0 will be padded to the MTU, and coded slice 1 will also be padded to the MTU.

[0111] continue Figure 8A For example, FEC-1 can be used to correct a set of packets including encoded slice 2 and encoded slice 3. Specifically, a packet 812a is created for encoded slice 2 during compression, where encoded slice 2 can be resized to compress into a single packet. Encoded slice 2 does not include the corresponding padding portion for FEC creation and / or data recovery because it is the largest packet in the set of packets that can be recovered using FEC-1. Additionally, a packet 813a is created for encoded slice 3 during compression, where encoded slice 3 can be resized to compress into a single packet. Encoded slice 3 may include the corresponding padding portion 813b for FEC creation and data recovery, which is stripped before transmission. FEC-1 is generated as packet 825 and transmitted after encoded slice 2 and encoded slice 3. FEC-1 can be used to correct each of those encoded slices that may be lost, and / or can be used to correct lost packets in each of those compressed encoded slices. Specifically, packet 812a of encoded slice 1 can be transmitted as soon as it is compressed at the encoder on the server, without waiting for the generation of FEC-1, and can be stored or buffered for FEC creation. Furthermore, the data packet 813a of encoded slice 3 can be transmitted as soon as it is compressed at the encoder on the server, without waiting for the generation of FEC-1, and can be stored or buffered for use in FEC creation.

[0112] Similarly, the packet size of FEC-1 packet 825 can specify the size of the padding used for encoding slices 2 and 3 during encoding. As previously mentioned, the size of each packet can be included in the header of the corresponding packet that needs to be read for FEC creation purposes. As previously mentioned, since each of the packets 812a and 813a storing the encoded slices is known or can be determined, rather than conveying the size of each packet in the header of the corresponding packet that needs to be read for FEC creation. Once the size of the largest packet in the set of packets that can be corrected using the corresponding FEC packets is known, the block size of the FEC-1 packets can be tuned to the maximum value in packets 812a and 813a when generating FEC-1. Specifically, once the size of FEC-1 is known, the padding of packets in the set of packets that can be corrected using FEC-1 for FEC creation and data recovery purposes is known. Figure 8AAs shown, FEC-1 is tuned to adjust the size of data packet 812a of coded slice 2, where data packet 812a is the largest data packet in the set containing coded slice 2 and coded slice 3. Therefore, coded slice 2 does not require padding for FEC creation and / or data recovery. On the other hand, coded slice 3 requires padding 813b for FEC creation and / or data recovery.

[0113] In embodiments of this disclosure, one or more strategies can be implemented to recover encoded slices and / or packets of encoded slices lost at the client during transmission from the server to the client when streaming media content and / or receiving streaming media content. As previously described, each encoded slice corresponding to a compressed video frame is packaged into a corresponding network packet, such that one packet corresponds to a single encoded slice in a one-to-one relationship. When all parts of a pipeline, including server and client operations, overlap, it is necessary to determine when to wait for additional packets for the corresponding video frame (e.g., when the lost packet has not yet been received) and when to proceed to the next video frame.

[0114] Information in the packet header can be used to determine when to move to the next video frame. Each packet header contains a frame index that tells the video game (e.g., a game application) which video frame the corresponding packet belongs to. Additionally, each packet header contains information about the packet index and the total number of packets for that frame. For example, Figure 8B This is a diagram of the header 850 of a data packet 800B corresponding to a coded slice of a video frame according to one embodiment of this disclosure. For example, the information in the header 850 includes: the frame number of the corresponding video frame 851, the total number of coded slices of the corresponding video frame 852, a frame generation timestamp (e.g., obtained at flip time) 853, encryption verification 854, an identifier of the corresponding FEC data packet 855, a coded slice number 856, and a timestamp specifying when the corresponding coded slice was generated 857. Optionally, the size of the data packet 800B may be included in the header 850. The data packet 800B includes a data portion 858.

[0115] When a data packet from the next video frame is received at the client, a decision is made whether to wait for the remaining data packets of the current video frame or proceed to the next video frame. Specifically, when a data packet for the current video frame (N) is received, tracking of which data packets have been received is performed. In one implementation, once a data packet from the next video frame (N+1) is received, any data packets that have not yet been received in the current video frame (minus FEC data packets) need to be corrected, such as through a method called "predec". Predec is performed to generate syntactically correct H.264 encoded slices to maintain the syntactically correctness of the entire H.264 video frame. That is, when it is determined that an encoded slice and / or one or more data packets of an encoded slice are lost, a filler slice is generated as a replacement for the lost encoded slice by inserting skipped macroblocks. In this way, the decoder is less likely to crash when there are lost encoded slices.

[0116] At this point, corruption may occur on the client side, and it may be necessary to perform reference frame invalidation, FEC recovery, or use data from previously rendered video frames to repair any damaged areas on the screen (e.g., skipped macroblocks corresponding to missing coded slices).

[0117] For example, in one implementation, a reference frame failure can be performed to repair skipped macroblocks corresponding to pixels in a first coded slice. In another implementation, FEC recovery is performed to correct lost coded slices and / or lost packets or multiple lost packets corresponding to the coded slice. For example, received FEC packets and / or FEC slices are used to regenerate and / or recover lost coded slices or lost packets or multiple lost packets corresponding to the coded slice.

[0118] In another implementation, a previous video frame is used to repair skipped macroblocks. For example, when rendered at the client to a display for the current video frame, color values ​​from the previous video frame are used for pixels corresponding to skipped macroblocks in the missing coded slice. Specifically, appropriate data from the frame buffer storing pixel data from the previous video frame is identified as overlapping pixels and / or skipped macroblocks in the missing coded slice. For example, the current video frame may have eight slices, with a second coded slice missing. Since the previous video frame has already been rendered for display at the client, data from the previous video frame can be used to address the missing second coded slice in the current video frame. For illustrative purposes, the previous video frame may have been divided into thirteen coded slices. Data for the current video frame rendered for display at the client is placed in a “new” display buffer (e.g., pixel-by-pixel). Pixel data from the previous video frame that overlaps with pixels corresponding to the missing coded slice can be identified and accessed from the “old” display buffer. That is, pixel data from the previous video frame can be used for the missing second coded slice in the current video frame.

[0119] Figure 9 Components of an exemplary device 900 that can be used to perform various aspects of embodiments of this disclosure are shown. For example, Figure 9 An exemplary hardware system suitable for streaming media content and / or receiving streaming media content according to embodiments of this disclosure is illustrated, including overlapping encoding and transmission operations at a server, and overlapping reception and decoding operations at a client, as well as overlapping reception, decoding, and rendering to a display operations at the client. This block diagram illustrates device 900, which may be incorporated into or may be a personal computer, server computer, game console, mobile device, or other digital device, each of which is suitable for practicing embodiments of the present invention. Device 900 includes a central processing unit (CPU) 902 for running software applications and optionally an operating system. CPU 902 may consist of one or more homogeneous or heterogeneous processing cores.

[0120] According to various implementations, CPU 902 is one or more general-purpose microprocessors having one or more processing cores. Further implementations may use one or more CPUs with microprocessor architectures specifically designed for highly parallel and compute-intensive applications (such as media and interactive entertainment applications) configured for graphics processing during game execution.

[0121] Memory 904 stores applications and data for use by CPU 902 and GPU 916. Storage device 906 provides non-volatile storage and other computer-readable media for applications and data and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other optical storage devices, as well as signal transmission and storage media. User input device 908 conveys user input from one or more users to device 900, examples of which may include a keyboard, mouse, joystick, touchpad, touchscreen, still or video recorder / camera, and / or microphone. Network interface 909 allows device 900 to communicate with other computer systems via electronic communication networks and may include wired or wireless communication via local area networks and wide area networks (such as the Internet). Audio processor 912 is adapted to generate analog or digital audio output from instructions and / or data provided by CPU 902, memory 904, and / or storage device 906. The components of device 900, including CPU 902, a graphics subsystem including GPU 916 and GPU cache 918, memory 904, data storage device 906, user input device 908, network interface 909 and audio processor 912, are connected via one or more data buses 922.

[0122] The graphics subsystem 914 is further connected to the data bus 922 and components of the device 900. The graphics subsystem 914 includes a graphics processing unit (GPU) 916 and a graphics memory 918. The graphics memory 918 includes display memory (e.g., a frame buffer) for storing pixel data for each pixel of the output image. The graphics memory 918 may be integrated with the GPU 916 in the same device, connected to the GPU 916 as a separate device, and / or implemented in memory 904. Pixel data may be provided directly from the CPU 902 to the graphics memory 918. Alternatively, the CPU 902 provides the GPU 916 with data and / or instructions defining the desired output image, from which the GPU 916 generates pixel data for one or more output images. The data and / or instructions defining the desired output image may be stored in memory 904 and / or graphics memory 918. In an embodiment, the GPU 916 includes 3D rendering capabilities for generating pixel data for the output image based on instructions and data defining the geometry, lighting, shading, texturing, motion, and / or camera parameters of a scene. GPU 916 may further include one or more programmable execution units capable of executing shader programs.

[0123] The graphics subsystem 914 periodically outputs pixel data of an image from the graphics memory 918 for display on the display device 910 or for projection by a projection system (not shown). The display device 910 can be any device capable of displaying visual information in response to signals from the device 900, including CRT, LCD, plasma, and OLED displays. The device 900 can provide, for example, analog or digital signals to the display device 910.

[0124] Other implementations for optimizing the graphics subsystem 914 may include multi-tenant GPU operations shared among multiple applications by GPU instances, and distributed GPUs supporting a single game. The graphics subsystem 914 can be configured as one or more processing devices.

[0125] For example, the graphics subsystem 914 can be configured to perform multi-tenant GPU functionality, where in one implementation, a graphics subsystem can implement graphics and / or rendering pipelines for multiple games. That is, the graphics subsystem 914 is shared among multiple games being executed.

[0126] In other implementations, the graphics subsystem 914 includes multiple GPU devices that are combined to perform graphics processing for a single application running on a corresponding CPU. For example, the multiple GPUs may perform frame rendering in an alternating manner, where GPU 1 renders the first frame, GPU 2 renders the second frame, and so on in consecutive frame cycles until the last GPU is reached, at which point the initial GPU renders the next video frame (e.g., if there are only two GPUs, GPU 1 renders the third frame). That is, the GPUs take turns rendering frames. Rendering operations can overlap, where GPU 2 can begin rendering the second frame before GPU 1 has finished rendering the first frame. In another implementation, the multiple GPU devices may be assigned different shader operations in the rendering and / or graphics pipeline. The main GPU is performing main rendering and compositing. For example, in a group comprising three GPUs, the primary GPU 1 can perform primary rendering (e.g., first shader operations) and compositing of outputs from secondary GPUs 2 and 3, where secondary GPU 2 can perform second shader operations (e.g., fluid effects, such as rivers), and secondary GPU 3 can perform third shader operations (e.g., particle smoke). The primary GPU 1 composes the results from each of GPUs 1, 2, and 3. In this way, different GPUs can be assigned to perform different shader operations (e.g., waving flags, wind, smoke generation, fire, etc.) to render video frames. In yet another embodiment, each of the three GPUs can be assigned to different objects and / or portions of a scene corresponding to a video frame. In the above embodiments and implementations, these operations can be performed in the same frame period (simultaneous parallelism) or in different frame periods (sequential parallelism).

[0127] Therefore, this disclosure describes methods and systems configured for streaming media content and / or receiving streaming media content, including overlapping encoding and transmission operations at the server, overlapping reception and decoding operations at the client, and overlapping reception, decoding, and rendering to a display operations at the client.

[0128] It should be understood that the various implementation schemes defined herein can be combined or assembled into specific implementations using the various features disclosed herein. Therefore, the examples provided are merely some possible examples and are not limited to the various implementations that could be defined by combining various elements. In some examples, some implementations may include fewer elements without departing from the spirit of the disclosed or equivalent implementations.

[0129] The embodiments of this disclosure can be practiced with various computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. The embodiments of this disclosure can also be practiced in distributed computing environments where tasks are performed via remote processing devices based on wired or wireless network links.

[0130] In light of the above embodiments, it should be understood that embodiments of this disclosure can employ various computer-implemented operations involving data stored in a computer system. These operations are those that require the physical manipulation of physical quantities. Any operation described herein that forms part of embodiments of this disclosure is a useful machine operation. Embodiments of this disclosure also relate to devices or apparatuses for performing these operations. The apparatus may be specifically constructed for the desired purpose, or the apparatus may be a general-purpose computer selectively activated or configured by a computer program stored in a computer. Specifically, various general-purpose machines may be used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct more specialized apparatuses to perform the desired operations.

[0131] This disclosure can also be implemented as computer-readable code on a computer-readable medium. A computer-readable medium is any data storage device capable of storing data that can subsequently be read by a computer system. Examples of computer-readable media include hard disk drives, network attached storage (NAS), read-only memory, random access memory, CD-ROM, CD-R, CD-RW, magnetic tape, and other optical and non-optical data storage devices. Computer-readable media may include computer-readable tangible media distributed across network-coupled computer systems, enabling the distributed storage and execution of computer-readable code.

[0132] Although the method operations are described in a specific order, it should be understood that other housekeeping operations may be performed between operations, or operations may be adjusted so that they occur at slightly different times, or they may be distributed in a system that allows processing operations to occur at various intervals associated with the processing, as long as the processing of the covering operations is performed in the desired manner.

[0133] While the foregoing disclosure has been described in considerable detail for the purposes of clarity, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. Therefore, this embodiment is to be considered illustrative rather than restrictive, and various embodiments of this disclosure are not limited to the details given herein, but may be modified within the scope of the appended claims and their equivalents.

Claims

1. A method for cloud gaming, comprising: generating a video frame while executing a video game at a server; inputting the video frame into an encoder at the server scan line by scan line; encoding one or more encoder slices into one or more encoded slices at the encoder, wherein each encoded slice is transferred to a buffer at an encoder fill rate; draining a first encoded slice from the buffer to stream the first encoded slice to a client over a network at a buffer drain rate, wherein the first encoded slice is streamed over the network before the one or more slices of the video frame are fully encoded, the first encoded slice being used for decoding at a decoder of the client before one or more other encoded slices of the video frame are fully received at the client, wherein a client VSYNC signal and a server VSYNC signal are synchronized and offset to enable a reduction in a display latency of a decoded slice.

2. The method of claim 1, wherein the server drains at least one encoded slice of the video frame before the entire video frame is encoded.

3. The method of claim 1, further comprising: wherein an encoded slice is packed into one or more network communication packets, sizing each of the one or more network communication packets to be less than or equal to a maximum transmission unit (MTU) at the encoder; and streaming the one or more network communication packets to the client over the network at the drain rate.

4. The method of claim 3, further comprising: sizing each of the one or more encoded slices to be the MTU or less, such that each encoded slice is packed into a single corresponding network communication packet.

5. The method of claim 1, further comprising: measuring a maximum receive bandwidth of the client; and dynamically adjusting the drain rate based on the maximum receive bandwidth, such that transmission of the one or more encoded slices is spaced apart based on the measured maximum receive bandwidth of the client, wherein the buffer drain rate is decoupled from the encoder fill rate.

6. The method of claim 1, further comprising: generating one or more forward error correction (FEC) slices for correcting the one or more encoded slices; and interleaving the one or more FEC slices within the one or more encoded slices as streamed, wherein each of the one or more FEC slices and the one or more encoded slices are packed into corresponding packets sized to be equal to or less than a maximum transmission unit (MTU).

7. The method of claim 1, further comprising: generating a plurality of video frames at the server during a session with the client in response to processing a video game by the server in a streaming mode; encoding the plurality of video frames into a plurality of compressed video frames at the encoder, each compressed video frame comprising one or more encoded slices; and sending the plurality of compressed video frames to the client over a network. ​ ​ ​ 8. The method of claim 7, wherein the client is configured to receive and decompress the plurality of compressed video frames for rendering to a display associated with the client.

9. A non-transitory computer-readable medium storing a computer program to perform a method for cloud gaming, the computer-readable medium comprising: program instructions to generate a video frame while executing a video game at a server; program instructions to input the video frame to an encoder at the server line-by-line scan; program instructions to encode one or more encoder slices into one or more encoded slices at the encoder, wherein each encoded slice is transferred to a buffer at an encoder fill rate; program instructions to drain a first encoded slice from the buffer to stream the first encoded slice to a client over a network at a buffer drain rate, wherein the first encoded slice is streamed over the network prior to fully encoding the one or more slices of the video frame, the first encoded slice for decoding at a decoder of the client prior to the client fully receiving one or more other encoded slices of the video frame, wherein a client VSYNC signal and a server VSYNC signal are synchronized and offset to enable a reduction in a display latency of a decoded slice.

10. The non-transitory computer-readable medium of claim 9, wherein the server drains at least one encoded slice of the video frame prior to encoding the entire video frame.

11. The non-transitory computer-readable medium of claim 9, further comprising: wherein in the method, an encoded slice is packed into one or more network communication packets, program instructions to resize each of the one or more network communication packets at the encoder to be less than or equal to a maximum transmission unit (MTU); and program instructions to stream the one or more network communication packets to the client over the network at the drain rate.

12. The non-transitory computer-readable medium of claim 11, further comprising: program instructions to resize each of the one or more encoded slices to be the MTU or smaller, such that each encoded slice is packed into a single corresponding network communication packet.

13. The non-transitory computer-readable medium of claim 9, further comprising: program instructions to measure a maximum receive bandwidth of the client; and program instructions to dynamically adjust the drain rate based on the maximum receive bandwidth, such that transmission of the one or more encoded slices are spaced apart based on the measured maximum receive bandwidth of the client, wherein the buffer drain rate is decoupled from the encoder fill rate.

14. The non-transitory computer-readable medium of claim 9, further comprising: program instructions to generate one or more forward error correction (FEC) slices, the FEC slices to correct the one or more encoded slices; and program instructions to stream the one or more FEC slices to the client over the network at the drain rate. program instructions for interleaving the one or more FEC slices within the one or more encoded slices when streaming, wherein each of the one or more FEC slices and the one or more encoded slices are packed into corresponding data packets that are equal to or smaller than a maximum transmission unit (MTU) in size.

15. The non-transitory computer-readable medium of claim 9, further comprising: program instructions for generating a plurality of video frames at the server in response to processing a video game in a streaming mode by the server during a session with the client; program instructions for encoding the plurality of video frames into a plurality of compressed video frames at the encoder, each compressed video frame comprising one or more encoded slices; and program instructions for transmitting the plurality of compressed video frames to the client over a network.

16. The non-transitory computer-readable medium of claim 15, wherein in the method, the client is configured to receive and decompress the plurality of compressed video frames for rendering to a display associated with the client.

17. A computer system comprising: a processor; and a memory coupled to the processor and storing instructions therein that, if executed by the computer system, cause the computer system to perform a method for cloud gaming, the method comprising: generating a video frame when executing a video game at a server; inputting the video frame into an encoder at the server scan line by scan line; encoding one or more encoder slices into one or more encoded slices at the encoder, wherein each encoded slice is transferred to a buffer at an encoder fill rate; draining a first encoded slice from the buffer to stream the first encoded slice to a client over a network at a buffer drain rate, wherein the first encoded slice is streamed over the network before the one or more slices of the video frame are fully encoded, the first encoded slice being used for decoding at a decoder of the client before one or more other encoded slices of the video frame are fully received at the client, wherein a client VSYNC signal and a server VSYNC signal are synchronized and offset to enable a reduction in a display latency of a decoded slice.

18. The computer system of claim 17, wherein in the method, the server drains at least one encoded slice of the video frame before encoding the entire video frame.

19. The computer system of claim 17, the method further comprising: wherein an encoded slice is packed into one or more network communication data packets, sizing each of the one or more network communication data packets to be less than or equal to a maximum transmission unit (MTU) at the encoder; and streaming the one or more network communication data packets to the client over the network at the drain rate.

20. The computer system of claim 19, the method further comprising: each of the one or more encoded slices is sized to be the MTU or less, such that each encoded slice is packaged into a single corresponding network communication data packet.

21. A method of cloud gaming, comprising: receiving one or more encoded slices of a video frame at a client, wherein the video frame is generated when a video game is executed at a server, and the video frame is encoded by an encoder at the server into the one or more encoded slices; and decoding a first encoded slice at a decoder of the client prior to fully receiving the one or more encoded slices of the video frame, wherein a client VSYNC signal and a server VSYNC signal are synchronized and offset to enable a reduction in display latency of a decoded slice.

22. The method of claim 21, wherein each of the one or more encoded slices is packaged into one or more network communication data packets, wherein each of the one or more network communication data packets is sized at the encoder to be less than or equal to a maximum transmission unit (MTU).

23. The method of claim 22, wherein each of the one or more encoded slices is sized to be the MTU or less, such that each encoded slice is packaged into a single corresponding network communication data packet.

24. The method of claim 21, wherein a plurality of video frames are generated at the server in response to processing a video game in a streaming mode by the server during a session with the client, wherein the plurality of video frames are encoded at the encoder into a plurality of compressed video frames, each corresponding compressed video frame comprising one or more corresponding encoded slices, wherein the plurality of compressed video frames are transmitted over a network to the client.

25. The method of claim 24, further comprising: receiving the plurality of compressed video frames; and decompressing the plurality of compressed video frames for rendering to a display associated with the client.

26. The method of claim 21, further comprising: determining that a first encoded slice of the one or more encoded slices of the video frame is lost during transmission from the server to the client; and generating a filler slice as a substitute for the first encoded slice decoded by the decoder, wherein the decoder inserts a skip macroblock of pixels of a display corresponding to the first encoded slice.

27. The method of claim 26, further comprising: performing reference frame invalidation to repair the skip macroblock of the pixels corresponding to the first encoded slice.

28. The method of claim 21, further comprising: determining that a first encoded slice of the one or more encoded slices of the video frame is lost during transmission from the server to the client; receiving a forward error correction (FEC) slice corresponding to the lost first encoded slice; and regenerating the first encoded slice using the FEC slice.

29. The method of claim 21, further comprising: determining that a first encoded slice of the one or more encoded slices of the video frame was lost during transmission from the server to the client; and using color values of a previous video frame for pixels corresponding to the first encoded slice.

30. A non-transitory computer-readable medium storing a computer program to perform a method for cloud gaming, the computer-readable medium comprising: program instructions to receive one or more encoded slices of a video frame at a client, wherein the video frame is generated while a video game is executed at a server, and the video frame is encoded by an encoder at the server into the one or more encoded slices; and program instructions to decode a first encoded slice at a decoder of the client prior to fully receiving the one or more encoded slices of the video frame, wherein a client VSYNC signal and a server VSYNC signal are synchronized and offset to enable a reduction in display latency of a decoded slice.

31. The non-transitory computer-readable medium of claim 30, wherein in the method, each of the one or more encoded slices is packed into one or more network communication packets, wherein in the method, each of the one or more network communication packets is sized at the encoder to be less than or equal to a maximum transmission unit (MTU), wherein in the method, each of the one or more encoded slices is sized to be the MTU or less such that each encoded slice is packed into a single corresponding network communication packet.

32. The non-transitory computer-readable medium of claim 30, wherein in the method, a plurality of video frames is generated at the server during a session with the client in response to processing a video game by the server in a streaming mode, wherein in the method, the plurality of video frames is encoded at the encoder into a plurality of compressed video frames, each corresponding compressed video frame comprising one or more corresponding encoded slices, wherein in the method, the plurality of compressed video frames is transmitted over a network to the client.

33. The non-transitory computer-readable medium of claim 32, further comprising: program instructions to receive the plurality of compressed video frames; and program instructions to decompress the plurality of compressed video frames for rendering to a display associated with the client.

34. The non-transitory computer-readable medium of claim 30, further comprising: program instructions to determine that a first encoded slice of the one or more encoded slices of the video frame was lost during transmission from the server to the client; program instructions to generate a filler slice as a replacement for the first encoded slice decoded by the decoder, wherein the decoder inserts a skip macroblock of pixels of a display corresponding to the first encoded slice; and program instructions to perform reference frame invalidation to repair the skip macroblock of the pixels corresponding to the first encoded slice.

35. The non-transitory computer-readable medium of claim 30, further comprising: program instructions for determining that a first encoded slice of the one or more encoded slices of the video frame was lost during transmission from the server to the client; program instructions for receiving a forward error correction (FEC) slice corresponding to the lost first encoded slice; and program instructions for regenerating the first encoded slice using the FEC slice.

36. A computer system comprising: a processor; and a memory coupled to the processor and storing instructions therein that, if executed by the computer system, cause the computer system to perform a method for cloud gaming, the method comprising: receiving one or more encoded slices of a video frame at a client, wherein the video frame is generated when a video game is executed at a server, and the video frame is encoded by an encoder at the server into the one or more encoded slices; and decoding a first encoded slice at a decoder of the client prior to fully receiving the one or more encoded slices of the video frame, wherein a client VSYNC signal and a server VSYNC signal are synchronized and offset to enable a reduction in display latency of a decoded slice.

37. The computer system of claim 36, wherein each of the one or more encoded slices is packed into one or more network communication packets, wherein each of the one or more network communication packets is sized at the encoder to be less than or equal to a maximum transmission unit (MTU).

Images