Encoder tuning to improve the trade-off between latency and video quality in cloud gaming applications

Abstract

To provide a higher quality experience to users of cloud gaming services.SOLUTION: Provided is a method for cloud gaming. In this method, a plurality of video frames is generated when executing a video game at a cloud gaming server. In this method, the plurality of video frames is encoded at an encoder bit rate, and the plurality of video frames that is compressed is transmitted to a client from a streamer of the cloud gaming server. In this method, a maximum receive bandwidth of a client is measured, the encoding of the plurality of video frames at the streamer is monitored; and a parameter of the encoder is dynamically tuned on the basis of the monitoring of the encoding.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure relates to streaming systems configured to stream content over a network, and more particularly to high performance encoders and decoders for cloud gaming systems, and streaming systems configured for encoder adjustments that are aware of network transmission speed and reliability, as well as overall latency targets. [Background technology]

[0002] In recent years, there has been a continuous push for online services that enable online or cloud gaming in a streaming format between cloud gaming servers and clients connected over a network. The streaming format has become increasingly popular due to the availability of game titles on demand, the ability to network between players for multiplayer gaming, the sharing of assets between players, the sharing of instant experiences between players and / or spectators, the ability for friends to watch friends play video games, the ability for friends to have friends join in on gameplay in progress, etc. Unfortunately, the demands are also pushing against the capabilities of network connections and the limitations of processing running on the server and client that is responsive enough to render high-quality images when delivered to the client. For example, the results of all game activity running on the server need to be compressed and transmitted back to the client with low millisecond latency for the best user experience. Round-trip latency can be defined as the overall time between a user's controller input and the display of a video frame at the client, which may include the processing and transmission of control information from the controller to the client, the processing and transmission of control information from the client to the server, the use of that input at the server to generate a video frame responsive to the input, the processing and transfer of the video frame to an encoding unit (e.g., scan-out), the encoding of the video frame, the transmission of the encoded video frame back to the client, the receipt and decoding of the video frame, and any processing or staging of the video frame before display. One-way latency may be defined as the portion of round-trip latency that consists of the time from the start of transfer (e.g., scan-out) of a video frame to an encoding unit at the server to the start of display of the video frame at the client. Part of the round-trip latency and one-way latency is associated with the time it takes for the data stream to be transmitted over the communication network from the client to the server and from the server to the client. Another part is associated with processing at the client and server; improvements in these operations, such as advanced strategies for decoding and displaying frames, can result in a substantial reduction in the round-trip latency and one-way latency between the server and the client, providing a higher quality experience for users of the cloud gaming service.

[0003] It is against this background that embodiments of the present disclosure have been made. Summary of the Invention

[0004] Embodiments of the present disclosure relate to a streaming system configured to stream content (e.g., games) over a network, and more particularly to a streaming system configured to provide encoder adjustments to improve the trade-off between one-way latency and video quality in a cloud gaming system, where the encoder adjustments may be based on monitoring of client bandwidth, skipped frames, number of encoded I-frames, number of scene changes, and / or number of video frames exceeding a target frame size, and the adjusted parameters may include an encoder bitrate, a target frame size, a maximum frame size, and a quantization parameter (QP) value, and where high performance encoders and decoders serve to reduce the overall one-way latency between the cloud gaming server and the client.

[0005] An embodiment of the present disclosure discloses a method for cloud gaming. The method includes generating a plurality of video frames when executing a video game on a cloud gaming server. The method includes encoding the plurality of video frames at an encoder bitrate, and the compressed plurality of video frames are transmitted from a streamer of the cloud gaming server to a client. The method includes measuring a maximum receive bandwidth of the client. The method includes monitoring encoding of the plurality of video frames at the streamer. The method includes dynamically adjusting parameters of the encoder based on the monitored encoding.

[0006] In another embodiment, a non-transitory computer-readable medium storing a computer program for cloud gaming is disclosed. The computer-readable medium includes program instructions for generating a plurality of video frames when executing a video game on a cloud gaming server. The computer-readable medium includes program instructions for encoding the plurality of video frames at an encoder bitrate, and the compressed plurality of video frames are transmitted from a streamer on the cloud gaming server to a client. The computer-readable medium includes program instructions for measuring a maximum receive bandwidth of the client. The computer-readable medium includes program instructions for monitoring the encoding of the plurality of video frames at the streamer. The computer-readable medium includes program instructions for dynamically adjusting parameters of the encoder based on the monitored encoding.

[0007] In yet another embodiment, a computer system includes a processor and a memory, coupled to the processor, having instructions stored therein, which, when executed by the computer system, cause the computer system to perform a method for cloud gaming. The method includes generating a plurality of video frames when executing a video game on a cloud gaming server. The method includes encoding the plurality of video frames at an encoder bitrate, wherein the compressed plurality of video frames are transmitted from a streamer on the cloud gaming server to a client. The method includes measuring a maximum receive bandwidth of the client. The method includes monitoring encoding of the plurality of video frames at the streamer. The method includes dynamically adjusting parameters of the encoder based on the monitored encoding.

[0008] In yet another embodiment, a method for cloud gaming is disclosed. The method includes generating a plurality of video frames when executing a video game on a cloud gaming server. The method includes predicting a scene change for a first video frame of the video game, the scene change being predicted before the first video frame is generated. The method includes generating a scene change hint, where the first video frame is a scene change. The method includes sending the scene change hint to an encoder. The method includes delivering the first video frame to the encoder, where the first video frame is encoded as an I-frame based on the scene change hint. The method includes measuring a maximum receive bandwidth of a client. The method includes determining whether to encode or not encode a second video frame received at the encoder based on the maximum receive bandwidth of the client and a target resolution of the client display.

[0009] In another embodiment, a non-transitory computer-readable medium storing a computer program for cloud gaming is disclosed. The computer-readable medium includes program instructions for generating a plurality of video frames when executing a video game on a cloud gaming server. The computer-readable medium includes program instructions for predicting a scene change for a first video frame of the video game, the scene change being predicted before the first video frame is generated. The computer-readable medium includes program instructions for generating a scene change hint indicating that the first video frame is a scene change. The computer-readable medium includes program instructions for transmitting the scene change hint to an encoder. The computer-readable medium includes program instructions for delivering the first video frame to the encoder, the first video frame being encoded as an I-frame based on the scene change hint. The computer-readable medium includes program instructions for measuring a maximum receive bandwidth of a client. The computer-readable medium includes program instructions for determining whether to encode or not encode a second video frame received by the encoder based on the maximum receive bandwidth of the client and a target resolution of the client display.

[0010] In yet another embodiment, a computer system includes a processor and a memory coupled to the processor and having instructions stored therein that, when executed by the computer system, cause the computer system to perform a method for cloud gaming. The method includes generating a plurality of video frames when executing a video game on a cloud gaming server. The method includes predicting a scene change for a first video frame of the video game, the scene change being predicted before the first video frame is generated. The method includes generating a scene change hint, where the first video frame is a scene change. The method includes sending a scene change hint to an encoder. The method includes delivering a first video frame to the encoder, the first video frame being encoded as an I-frame based on the scene change hint. The method includes measuring a maximum receive bandwidth of a client. The method includes determining whether to encode or not encode a second video frame received at the encoder based on the maximum receive bandwidth of the client and a target resolution of the client display.

[0011] Other aspects of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the present disclosure.

[0012] The present disclosure can be best understood by referring to the following description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0013] [Figure 1A] FIG. 2 is a diagram of a VSYNC signal at the beginning of a frame period, according to one embodiment of the present disclosure. [Figure 1B] FIG. 1 is a diagram of the frequency of a VSYNC signal, according to one embodiment of the present disclosure. [Figure 2A] 1 is a diagram of a system for providing games over a network between one or more cloud gaming servers and one or more client devices in various configurations, according to one embodiment of the present disclosure, wherein VSYNC signals can be synchronized and offset to reduce one-way latency. [Figure 2B] FIG. 1 is a diagram for providing gaming between two or more peer devices, according to one embodiment of the present disclosure, in which VSYNC signals can be synchronized and offset to achieve optimal timing of reception of controllers and other information between the devices. [Figure 2C] 1 illustrates various network configurations that benefit from proper synchronization and offset of VSYNC signals between source and target devices, according to one embodiment of the present disclosure. [Figure 2D] 1 illustrates a multi-tenancy configuration between a cloud gaming server and multiple clients that benefits from proper synchronization and offset of VSYNC signals between source and target devices, according to one embodiment of the present disclosure. [Figure 3] 1 illustrates the variation in one-way latency between a cloud gaming server and a client due to clock drift when streaming video frames generated from a video game running on the server, according to one embodiment of the present disclosure. [Figure 4] 1 illustrates a network configuration including a cloud gaming server and clients when streaming video frames generated from a video game running on the server, where the VSYNC signals between the server and clients are synchronized and offset, allowing overlap of operations on the server and clients and reducing one-way latency between the server and clients. [Figure 5] FIG. 1 is a flow diagram illustrating a method for cloud gaming, according to one embodiment of the present disclosure, in which encoding of video frames includes adjusting encoder parameters with awareness of network transmission speed and reliability, as well as overall latency targets. [Figure 6] FIG. 10 illustrates measuring a client's bandwidth by a streamer component operating at the application layer, according to one embodiment of the present disclosure, where the streamer is configured to monitor and adjust the encoder so that compressed video frames can be transmitted at a rate within the client's measured bandwidth. [Figure 7] FIG. 1A illustrates setting the quantization parameter (QP) of an encoder to optimize quality and buffer utilization at a client according to one embodiment of the present disclosure; and FIG. 1B illustrates adjusting the target frame size, maximum frame size, and / or QP (e.g., minimum QP and / or maximum QP) encoder settings to reduce the occurrence of I-frames that exceed the true target frame size supported by the client according to one embodiment of the present disclosure. [Figure 8]FIG. 1 is a flow diagram illustrating a method for cloud gaming, according to one embodiment of the present disclosure, in which encoding of video frames includes determining whether to skip video frames or delay the encoding and transmission of video frames when the encoding is long, such as when encoding an I-frame, or when the video frames being generated are large. [Figure 9] A shows a sequence of video frames being compressed by an encoder according to one embodiment of the present disclosure, where the encoder drops the encoding of a video frame after encoding an I-frame when the client bandwidth is low relative to the target resolution of the client's display; B shows a sequence of video frames being compressed by an encoder according to one embodiment of the present disclosure, where in each sequence a video frame is encoded as an I-frame and subsequent video frames are also encoded after a delay in encoding an I-frame when the client bandwidth is medium or high relative to the target resolution of the client's display; and C shows a sequence of video frames being compressed by an encoder according to one embodiment of the present disclosure, where in each sequence a video frame is encoded as an I-frame and subsequent video frames are also encoded after a delay in encoding an I-frame when the client bandwidth is medium or high relative to the target resolution of the client's display. [Figure 10] 1 illustrates components of an exemplary device that can be used to implement aspects of various embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0014] Although the following detailed description includes many specific details for purposes of illustration, those skilled in the art will appreciate that many variations and modifications to the following details are within the scope of the present disclosure. Accordingly, the aspects of the present disclosure described below are set forth without loss of generality to, and without imposing limitations on, the claims that follow this description.

[0015] Generally speaking, various embodiments of the present disclosure describe methods and systems configured to reduce latency and / or latency instability between a source device and a target device when streaming media content (e.g., when streaming audio and video from a video game). Latency instability can occur in the one-way latency between the server and the client due to the additional time required to generate complex frames (e.g., scene changes) at the server, the increased time it takes to encode / compress complex frames at the server, variable communication paths over the network, and the increased time it takes to decode complex frames at the client. Latency instability can also be introduced by clock differences at the server and the client, which cause drift between the server and the client's VSYNC signals. In embodiments of the present disclosure, one-way latency between a server and a client can be reduced in cloud gaming applications by providing high-performance encoding and decoding. When decompressing streaming media (e.g., streaming video, movies, clips, content), it is possible to buffer a significant amount of the decompressed video, and therefore, displaying the streamed content can depend on average decoding capabilities and metrics (e.g., supporting 4K media at 60 Hz depends on the average amount of decoding resources). However, in cloud gaming, the longer it takes to perform encoding and / or decoding operations (even for a single frame), the correspondingly higher one-way latency. Therefore, for cloud gaming, it is beneficial to provide powerful decoding and encoding resources that may seem unnecessary compared to the needs of streaming video applications, and the resources should be optimized for the time required to process the frames that require the longer or longest processing. In other embodiments of the present disclosure, encoder adjustments may be performed to improve the trade-off between latency and video quality in cloud gaming applications. Encoder adjustments are made with awareness of the network transmission speed and reliability, and overall latency targets. In embodiments, if the encoding is taking too long or the data generated is large (e.g., both conditions can occur with compressed I-frames), methods are implemented to determine whether to delay the encoding and transmission of subsequent frames or skip them. In an embodiment, adjustments to the quantization parameter (QP) value, target frame size, and maximum frame size are performed based on the available network speed to the client, e.g., if the network speed is faster, the QP may be reduced. In other embodiments, monitoring of the I-frame occurrence rate is performed and used to set the QP. For example, if the frequency of I-frames is low, the QP can be reduced (e.g., resulting in higher encoding accuracy or higher encoding quality), so that encoding of video frames can be skipped to maintain low one-way latency while sacrificing video playback quality. In this way, high performance encoding and decoding and encoder adjustments performed to improve the trade-off between latency and video quality in cloud gaming applications can lead to reduced one-way latency between the cloud gaming server and the client, smoother frame rates, and more reliable and / or consistent one-way latency.

[0016] With a general understanding of the various embodiments above, illustrative details of the embodiments will now be described with reference to the various figures.

[0017] Throughout the specification, references to a "game" or a "video game" or a "gaming application" are meant to refer to any type of interactive application that is directed through the execution of input commands. By way of example only, interactive applications include applications for games, word processing, video processing, video game processing, etc. Furthermore, the terms introduced above are intended to be interchangeable.

[0018] Cloud gaming involves running a video game on a server to generate game-rendered video frames, which are then transmitted to a client for display. The timing of operations on both the server and the client may be associated with respective vertical synchronization (VSYNC) parameters. When VSYNC signals are properly synchronized and / or offset between the server and / or the client, operations performed on the server (e.g., generating and transmitting video frames over one or more frame periods) are synchronized with operations performed on the client (e.g., displaying video frames on a display at a display frame or refresh rate corresponding to the frame periods). In particular, a server VSYNC signal generated on the server and a client VSYNC signal generated on the client may be used to synchronize operations on the server and the client. That is, when the server and client VSYNC signals are synchronized and / or offset, the server generates and transmits video frames in sync with how the client will display those video frames.

[0019] VSYNC signals and vertical blanking intervals (VBIs) are incorporated to generate and display video frames when streaming media content between a server and a client. For example, the server attempts to generate rendered video frames for a game at one or more frame periods defined by the corresponding server VSYNC signal (e.g., if the frame period is 16.7 ms, generating a video frame every frame period results in 60 Hz operation, while generating one video frame every two frame periods results in 30 Hz operation), then encodes and transmits the video frames to the client. The client decodes and displays the received encoded video frames, and the client displays each rendered video frame for display beginning with the corresponding client VSYNC.

[0020] For illustrative purposes, FIG. 1A shows how a VSYNC signal 111 may indicate the start of a frame period, where various operations may be performed at the server and / or client during the corresponding frame period. When streaming media content, a server may use the server VSYNC signal to generate and encode video frames, and a client may use the client VSYNC signal to display the video frames. The VSYNC signal 111 is generated at a specified frequency corresponding to a specified frame period 110, as shown in FIG. 1B. Additionally, the VBI 105 defines the period between when the last raster line for the previous frame period is drawn on the display and when the first raster line (e.g., top) is drawn on the display. As shown, after the VBI 105, the video frame rendered for display is displayed via raster scan line 106 (e.g., raster line by raster line from left to right).

[0021] Additionally, various embodiments of the present disclosure are disclosed for reducing one-way latency and / or latency stabilization between a source device and a target device, such as when streaming media content (e.g., video game content). For illustrative purposes only, various embodiments for reducing one-way latency and / or latency stabilization are described within a server-client network configuration. However, it is understood that the various techniques disclosed for reducing one-way latency and / or latency stabilization may be implemented within other network configurations and / or on peer-to-peer networks, such as those shown in FIGS. 2A-2D. For example, various embodiments disclosed for reducing one-way latency and / or latency stabilization may be implemented between one or more server and client devices in various configurations (e.g., server-client, server-server, server-multiple clients, server-multiple servers, client-client, client-multiple clients, etc.).

[0022] FIG. 2A is a diagram of a system 200A for providing games between one or more cloud gaming networks 290 and / or servers 260 and one or more client devices 210 over a network 250 in various configurations, where, according to one embodiment of the present disclosure, the server and client VSYNC signals can be synchronized and offset, and / or dynamic buffering can be performed on the client, and / or encoding and transmission operations on the server can be overlapped, and / or receiving and decoding operations on the client can be overlapped, and / or decoding and display operations on the client can be overlapped to reduce one-way latency between the server 260 and the client 210. In particular, according to one embodiment of the present disclosure, system 200A provides games via cloud gaming network 290, with the games being executed remotely from corresponding user client devices 210 (e.g., thin clients) playing the games. System 200A can provide game control to one or more users playing one or more games through cloud gaming network 290 via network 250, in either single-player or multiplayer mode. In some embodiments, cloud gaming network 290 can include multiple virtual machines (VMs) running on a host machine's hypervisor, with one or more virtual machines configured to execute game processor modules utilizing hardware resources available to the host machine's hypervisor. Network 250 can include one or more communication technologies. In some embodiments, network 250 can include fifth-generation (5G) network technology with advanced wireless communication systems.

[0023] In some embodiments, communication may be facilitated using wireless technology. Such technology may include, for example, 5G wireless communication technology. 5G is the fifth generation of cellular network technology. 5G networks are digital cellular networks, where service areas covered by providers are divided into small geographic areas called cells. Analog signals representing sound and images are digitized by the telephone, converted by an analog-to-digital converter, and transmitted as a stream of bits. All 5G wireless devices within a cell communicate over the airwaves using a local antenna array and low-power automatic transceivers (transmitters and receivers) within the cell via frequency channels assigned by the transceiver from a pool of frequencies reused by other cells. The local antennas are connected to the telephone network and the Internet by high-bandwidth optical fiber or wireless backhaul connections. As with other cellular networks, mobile devices moving from one cell to another are automatically transferred to the new cell. It should be understood that a 5G network is merely one exemplary type of communication network, and embodiments of the present disclosure may utilize previous generations of wireless or wired communication, as well as subsequent generations of wired or wireless technologies following 5G.

[0024] As shown, cloud gaming network 290 includes game server 260, which provides access to multiple video games. Game server 260 may be any type of server computing device available in the cloud and may be configured as one or more virtual machines running on one or more hosts. For example, game server 260 may manage virtual machines supporting game processors that instantiate instances of games for users. In this manner, multiple game processors of game server 260, associated with multiple virtual machines, are configured to run multiple instances of one or more games associated with gameplay for multiple users. In this manner, the backend server support provides streaming of gameplay media (e.g., video, audio, etc.) for multiple game applications to multiple corresponding users. That is, the game servers 260 are configured to stream data (e.g., rendered images and / or frames of corresponding gameplay) back to corresponding client devices 210 over the network 250. In this manner, computationally complex game applications may be executed on the backend servers in response to controller inputs received and forwarded by the client devices 210. Each server may render images or frames, which may then be encoded (e.g., compressed) and streamed to corresponding client devices for display.

[0025] For example, multiple users may access cloud gaming network 290 over communications network 250 using corresponding client devices 210 configured to receive streaming media. In one embodiment, client devices 210 may be configured as thin clients that interface with backend servers (e.g., game servers 260 of cloud gaming network 290) configured to provide computing functionality (e.g., including game title processing engine 211). In another embodiment, the client device 210 may be configured with a game title processing engine and game logic for at least some local processing of the video game, and may further be utilized to receive streaming content generated by the video game running on a backend or for other content provided by backend server support. For local processing, the game title processing engine includes basic processor-based functionality for running the video game and services related to the video game. The game logic is stored on the local client device 210 and is used to run the video game.

[0026] In particular, a client device 210 of a corresponding user (not shown) is configured to request access to the game over a communications network 250, such as the Internet, and to render display images generated by the video game executed by a game server 260, the encoded images being delivered to the client device 210 for display in association with the corresponding user. For example, a user may interact with an instance of a video game running on a game processor of a game server 260 via a client device 210. More specifically, the instance of the video game is executed by a game title processing engine 211. Corresponding game logic (e.g., executable code) 215 that implements the video game is stored in and accessible via a data store (not shown) and is used to run the video game. The game title processing engine 211 can support multiple video games using multiple game logics, each of which is selectable by a user.

[0027] For example, client device 210 is configured to interact with game title processing engine 211 in connection with the corresponding user's gameplay, such as via input commands used to drive the gameplay. In particular, client device 210 can receive input from various types of input devices, such as a game controller, a tablet computer, a keyboard, gestures captured by a video camera, a mouse, a touchpad, etc. Client device 210 may be any type of computing device having at least a memory and a processor module that can connect to game server 260 over network 250. Backend game title processing engine 211 is configured to generate rendered images that are distributed over network 250 for display on corresponding displays associated with client device 210. For example, through a cloud-based service, rendered images for a game may be delivered by an instance of the corresponding game executing on a game execution engine 211 of a game server 260. That is, a client device 210 is configured to receive encoded images (e.g., encoded from game-rendered images generated through the execution of a video game) and display the rendered images for display 11. In one embodiment, display 11 includes an HMD (e.g., displaying VR content). In some embodiments, the rendered images may be streamed wirelessly or wired to a smartphone or tablet directly from the cloud-based service or via a client device 210 (e.g., PlayStation® Remote Play).

[0028] In one embodiment, game server 260 and / or game title processing engine 211 include basic processor-based functionality for running services related to games and game applications. For example, processor-based functionality may include 2D or 3D rendering, physics, physics simulation, scripting, audio, animation, graphics processing, lighting, shading, rasterization, ray tracing, shadowing, culling, transformations, artificial intelligence, etc. Additionally, game application services may include memory management, multi-threading management, quality of service (QoS), bandwidth testing, social networking, social friend management, communication with social networks of friends, communication channels, text messaging, instant messaging, chat support, etc.

[0029] In one embodiment, cloud gaming network 290 is a distributed game server system and / or architecture. In particular, a distributed game engine, which executes game logic, is configured as a corresponding instance of a corresponding game. Generally, a distributed game engine takes each function of the game engine and distributes those functions for execution by multiple processing entities. Individual functions may be further distributed across one or more processing entities. The processing entities may be configured in a variety of configurations, including physical hardware, and / or as virtual components or virtual machines, and / or as virtual containers; a container differs from a virtual machine in that it virtualizes an instance of a game application running on a virtualized operating system. The processing entities may utilize and / or rely on servers and their underlying hardware on one or more servers (computing nodes) of the cloud gaming network 290, which may be located in one or more racks. Coordination, allocation, and management of the execution of these functions across the various processing entities is performed by a distributed synchronization layer. In this manner, the execution of these functions is controlled by the distributed synchronization layer to generate media (e.g., video frames, audio, etc.) for the game application in response to controller inputs by the player. The distributed synchronization layer enables these functions to be executed efficiently (e.g., through load balancing) across the distributed processing entities, allowing critical game engine components / functions to be distributed and reconfigured for more efficient processing.

[0030] The game title processing engine 211 includes a central processing unit (CPU) and a group of graphics processing units (GPUs) that can be configured to perform multi-tenancy GPU functions. In another embodiment, multiple GPU devices are combined to perform graphics processing for a single application running on a corresponding CPU.

[0031] 2B is a diagram for providing a game between two or more peer devices, according to one embodiment of the present disclosure, where VSYNC signals can be synchronized and offset to achieve optimal timing of reception of controllers and other information between the devices. For example, a head-to-head game may be performed using two or more peer devices connected directly via a network 250 or via peer-to-peer communication (e.g., Bluetooth, local area networking, etc.).

[0032] As shown, a game is running locally on each of the client devices 210 (e.g., game consoles) of corresponding users playing the video game, and the client devices 210 communicate via peer-to-peer networking. For example, an instance of the video game is executed by a game title processing engine 211 of the corresponding client device 210. Game logic 215 (e.g., executable code) implementing the video game is stored on the corresponding client device 210 and used to run the game. For illustrative purposes, the game logic 215 may be distributed to the corresponding client device 210 via portable media (e.g., optical media) or over a network (e.g., downloaded from a game provider via the Internet).

[0033] In one embodiment, the game title processing engine 211 of the corresponding client device 210 includes basic processor-based functionality for executing services related to games and game applications. For example, the processor-based functionality may include 2D or 3D rendering, physics, physics simulation, scripting, audio, animation, graphics processing, lighting, shading, rasterization, ray tracing, shadowing, culling, transformations, artificial intelligence, etc. Additionally, the game application services may include memory management, multi-threading management, quality of service (QoS), bandwidth testing, social networking, social friend management, communication with social networks of friends, communication channels, text messaging, instant messaging, chat support, etc.

[0034] Client device 210 can receive input from various types of input devices, such as a game controller, a tablet computer, a keyboard, gestures captured by a video camera, a mouse, a touchpad, etc. Client device 210 can be any type of computing device having at least a memory and a processor module, and is configured to generate rendered images executed by game title processing engine 211 and display the rendered images on a display (e.g., display 11, or display 11 including a head-mounted display (HMD)). For example, the rendered images may be associated with an instance of a game running locally on the client device 210, implementing the corresponding user's gameplay, such as via input commands used to drive the gameplay. Some examples of client device 210 include a personal computer (PC), a game console, a home theater device, a general-purpose computer, a mobile computing device, a tablet, a phone, or any other type of computing device capable of running an instance of a game.

[0035] FIG. 2C shows various network configurations that benefit from proper synchronization and offset of the VSYNC signal between the source device and the target device, including the configurations shown in FIGS. 2A-2B. In particular, the various network configurations benefit from proper alignment of the frequencies of the server and client VSYNC signals, and timing offsets of the server and client VSYNC signals for the purpose of reducing one-way latency and / or latency variability between the server and the client. For example, one network device configuration includes a configuration from a cloud gaming server (e.g., source) to a client (target). In one embodiment, the client may include a WebRTC client configured to provide audio and video communication within a web browser. Another network configuration includes a configuration from a client (e.g., source) to a server (target). Yet another network configuration includes a configuration from a server (e.g., source) to a server (e.g., target). Another network device configuration includes a configuration from a client (e.g., source) to a client (target), where the clients may each be, for example, a game console for providing a head-to-head game.

[0036] In particular, the alignment of the VSYNC signal may include synchronization of the frequencies of the server VSYNC signal and the client VSYNC signal, and may also include adjusting the timing offset between the client VSYNC signal and the server VSYNC signal to maintain an ideal relationship between the server and client VSYNC signals for the purpose of removing drift and / or reducing one-way latency and / or latency variability. To achieve proper alignment, in one embodiment, the server VSYNC signal may be adjusted to effect proper alignment between a pair of a server 260 and a client 210. In another embodiment, the client VSYNC signal may be adjusted to achieve proper alignment between a server 260 and client 210 pair. When the client and server VSYNC signals are aligned, the server and client VSYNC signals occur at substantially the same frequency and are offset from one another by a timing offset that can be adjusted at any time. In another embodiment, aligning the VSYNC signals may include synchronizing the VSYNC frequencies of the two clients, adjusting the timing offset between their VSYNC signals to remove drift, and / or achieving proper timing of receipt of controller and other information, and either VSYNC signal may be adjusted to achieve this alignment. In yet another embodiment, alignment may include synchronizing the VSYNC frequencies of multiple servers, and may also include synchronizing the frequencies of server and client VSYNC signals and adjusting the timing offset between the client and server VSYNC signals, for example, for head-to-head cloud gaming. In server-to-client and client-to-client configurations, alignment may include both synchronizing the frequencies between the server and client VSYNC signals and providing appropriate timing offsets between the server and client VSYNC signals. In server-to-server configurations, alignment may include synchronizing the frequencies between the server and client VSYNC signals without setting a timing offset.

[0037] 2D illustrates a multi-tenancy configuration between a cloud gaming server 260 and one or more clients 210 that benefits from proper synchronization and offset of VSYNC signals between source and target devices, according to one embodiment of the present disclosure. In a server-to-client configuration, alignment may include both synchronizing the frequency between the server VSYNC signal and the client VSYNC signal and providing a proper timing offset between the server VSYNC signal and the client VSYNC signal. In a multi-tenancy configuration, in one embodiment, the client VSYNC signal is adjusted at each client 210 to achieve proper alignment between the server 260 and client 210 pair.

[0038] For example, a graphics subsystem may be configured to perform multi-tenancy GPU functions, and in one embodiment, one graphics subsystem may implement the graphics and / or rendering pipeline for multiple games. That is, the graphics subsystem is shared among multiple games being executed. In particular, a game title processing engine may include a CPU and GPU group configured to perform multi-tenancy GPU functions, and in one embodiment, one CPU and GPU group may implement the graphics and / or rendering pipeline for multiple games. That is, the CPU and GPU group is shared among multiple games being executed. The CPU and GPU group can be configured as one or more processing devices. In another embodiment, multiple GPU devices are combined to perform graphics processing for a single application running on a corresponding CPU.

[0039] 3 illustrates a general process for running a video game on a server to generate rendered video frames for the game and send those video frames to a client for display. Traditionally, some operations on the game server 260 and the client 210 are performed within the frame period defined by their respective VSYNC signals. For example, the server 260 attempts to generate rendered video frames for the game at 301 in one or more frame periods as defined by corresponding server VSYNC signals 311. The video frames are generated by the game in response to either control information (e.g., a user's input commands) delivered from an input device in operation 350, or game logic that is not driven by control information. Transmission jitter 351 may be present when sending control information to the server 260, and jitter 351 measures the variation in network latency from the client to the server (e.g., when sending input commands). As shown, the thick arrow indicates the current delay in sending control information to server 260, although due to jitter, there may be a range of arrival times for the control information at server 260 (e.g., the range enclosed by the dotted arrow). At flip time 309, the GPU receives a flip command indicating that the corresponding video frame has been fully generated and placed in the frame buffer of server 260. Server 260 then performs scan-out / scan-in (operation 302, where scan-out may be aligned with VSYNC signal 311) on that video frame over the subsequent frame period defined by server VSYNC signal 311 (VBI omitted for clarity). The video frame is then encoded (operation 303) (e.g., encoding begins after the occurrence of VSYNC signal 311, and the end of encoding may not be aligned with VSYNC signal 311) and transmitted to client 210 (operation 304, where transmission may not be aligned with VSYNC signal 311). At client 210, the encoded video frames are received (operation 305, where reception may not be aligned with client VSYNC signal 312), decoded (operation 306, where decoding may not be aligned with client VSYNC signal 312), buffered, and displayed (operation 307, where start of display may not be aligned with client VSYNC signal 312). In particular, client 210 displays each video frame rendered for display beginning with the corresponding occurrence of client VSYNC signal 312.

[0040] One-way latency 315 may be defined as the latency from the start of the transfer of a video frame to an encoding unit (e.g., scanout 302) at the server to the start of display of the video frame at the client 307. That is, one-way latency is the time from server scanout to client display, taking into account client buffering. Individual frames have a latency from the start of scanout 302 to the completion of decode 306, and this latency may vary from frame to frame due to high variability in server operations such as encoding 303 and transmission 304, network transmission with jitter 352 between the server 260 and client 210, and client receive 305. As shown, the thick straight arrow indicates the current latency in transmitting the corresponding video frame to the client 210, although due to jitter 352, there can be a range of arrival times for the video frame at the client 210 (e.g., the range demarcated by the dotted arrow). Because one-way latency must be relatively stable (e.g., fairly consistent) to achieve a good playback experience, buffering 320 is traditionally performed with the result that the display of individual frames with low latency (e.g., from the start of scanout 302 to the completion of decode 306) is delayed by several frame periods. That is, in the presence of network instability or unpredictable encoding / decoding times, additional buffering is required to ensure that one-way latency remains constant.

[0041] According to one embodiment of the present disclosure, one-way latency between a cloud gaming server and a client may vary due to clock drift when streaming video frames generated from a video game running on the server. That is, differences in the frequency of the server VSYNC signal 311 and the client VSYNC signal 312 may cause the client VSYNC signal to drift relative to frames arriving from the server 260. This drift may be due to slight differences in the crystals used in each of the respective clocks at the server and client. Furthermore, embodiments of the present disclosure reduce one-way latency by performing one or more of synchronization and offset of the VSYNC signal for alignment between the server and client, by providing dynamic buffering at the client, by overlapping the encoding and transmission of video frames at the server, by overlapping the reception and decoding of video frames at the client, and by overlapping the decoding and display of video frames at the client.

[0042] Furthermore, during the encoding of video frames (operation 303), in previous techniques, the encoder determines how much change there is between the current video frame being encoded and one or more previously encoded frames to determine whether there is a scene change (e.g., a complex image of the corresponding generated video frame). That is, a scene change hint can be inferred from the difference between the current frame being encoded and the previous frames that have already been encoded. When streaming content from a server to a client over a network, the encoder at the server can decide to encode video frames that are detected as scene changes with high complexity. Otherwise, the encoder encodes video frames that are not detected as scene changes with lower complexity. However, detecting a scene change in the encoder may take up to one frame period (e.g., adding jitter) such that a video frame is initially encoded with lower complexity (in a first frame period), but then re-encoded with higher complexity (in a second frame period) if a scene change is determined to exist. Similarly, scene change detection may be unnecessarily triggered (such as via a small burst in the image) because the difference between the currently encoded video frame and a previously encoded video frame may exceed a threshold difference value, even if there is no scene change. Thus, when a scene change is detected in the encoder, additional latency due to jitter is introduced in the encoder to accommodate performing scene change detection and re-encoding the video frame with higher complexity.

[0043] FIG. 4 illustrates the flow of data through a network configuration including a highly optimized cloud gaming server 260 and a highly optimized client 210 when streaming video frames generated from a video game running on the server, where, according to an embodiment of the present disclosure, overlapping of server and client operations reduces one-way latency, and synchronization and offsetting of VSYNC signals between the server and client reduces one-way latency while simultaneously reducing the variability of one-way latency between the server and client. In particular, Figure 4 illustrates the desired alignment between the server VSYNC signal and the client VSYNC signal. In one embodiment, adjustments to the server VSYNC signal 311 are performed to obtain proper alignment between the server VSYNC signal and the client VSYNC signal, such as in a server and client network configuration. In another embodiment, adjustment of the client VSYNC signal 312 is performed to obtain proper alignment between the server VSYNC signal and the client VSYNC signal, such as in a multi-tenant server to multiple client network configuration. For illustrative purposes, adjustment of the server VSYNC signal 311 is described in FIG. 4 to synchronize the frequency of the server and client VSYNC signals and / or adjust the timing offset between corresponding client and server VSYNC signals, although it will be understood that the client VSYNC signal 312 may also be used for adjustment. In the context of this patent application, "synchronization" should be interpreted to mean adjusting signals so that they match in frequency but may differ in phase, and "offset" should be interpreted to mean a time delay between signals, for example, the time between when one signal reaches its maximum value and when the other signal reaches its maximum value.

[0044] As illustrated, FIG. 4 illustrates an improved process for executing a video game on a server to generate rendered video frames and sending those video frames to a client for display, in accordance with an embodiment of the present disclosure. The process is illustrated with respect to the generation and display of a single video frame on the server and the client. In particular, the server generates rendered video frames for the game at 401. For example, the server 260 includes a CPU (e.g., game title processing engine 211) configured to execute the game. The CPU generates one or more draw calls for the video frames, which include commands placed in a command buffer for execution by a corresponding GPU of the server 260 in a graphics pipeline. The graphics pipeline may include one or more shader programs for vertices of objects in a scene to generate texture values ​​that are rendered into a video frame for display, with this operation being performed in parallel across the GPU for efficiency. At flip time 409, the GPU reaches a flip command in the command buffer, indicating that the corresponding video frame has been fully generated and / or rendered and placed into a frame buffer at server 260.

[0045] At 402, the server performs scanout of the rendered video frame for the game to the encoder. In particular, scanout is performed scanline by scanline or in groups of consecutive scanlines, where a scanline refers to the display of a single horizontal line, for example, from edge of the screen to edge of the screen. These scanlines or groups of consecutive scanlines are sometimes referred to as slices, and are referred to herein as screen slices. In particular, scanout 402 may include several processes, including modifying the rendered frame for the game, overlaying it with another frame buffer, or shrinking it to surround it with information from another frame buffer. During scanout 402, the modified video frame is then scanned to the encoder for compression. In one embodiment, scanout 402 is performed at occurrence 311a of VSYNC signal 311. In other embodiments, scanout 402 may be performed before occurrence of VSYNC signal 311, such as at flip time 409.

[0046] At 403, the rendered video frames for the game (possibly modified) are encoded in an encoder, one per encoder slice, to generate one or more encoded slices, where the encoded slices are unrelated to scanlines or screen slices. In this manner, the encoder generates one or more encoded (e.g., compressed) slices. In one embodiment, the encoding process begins before the scanout 402 process is fully completed for the corresponding video frame. Additionally, the start and / or end of the encode 403 may or may not be aligned with the server VSYNC signal 311. The boundaries of an encoded slice are not limited to a single scan line, but may consist of a single scan line, or multiple scan lines. Furthermore, the end of an encoded slice and / or the start of the next encoder slice may not necessarily occur at the edge of the display screen (e.g., may occur in the center of the screen or the middle of a scan line), and an encoded slice need not traverse the entire display screen from edge to edge. As shown, one or more encoded slices may be compressed and / or encoded, including a compressed "encoded slice A" with hash marks.

[0047] At 404, the encoded video frame is transmitted from the server to the client; this transmission may occur on a per-encoded-slice basis, with each encoded slice being a compressed encoder slice. In one embodiment, the transmission process 404 begins before the encoding process 403 is fully completed for the corresponding video frame. Furthermore, the start and / or end of transmission 404 may or may not be aligned with the server VSYNC signal 311. As shown, compressed encoded slice A is transmitted to the client independently of the other compressed encoder slices for the rendered video frame. The encoder slices may be transmitted one at a time or in parallel.

[0048] At 405, the client receives the compressed video frames, again for each encoded slice. Furthermore, the start and / or end of reception 405 may or may not be aligned with the client VSYNC signal 312. As shown, compressed encoded slice A is received by the client. Transmission jitter 452 may exist between the server 260 and the client 210, where jitter 452 measures the variation in network latency from the server 260 to the client 210. A lower jitter value indicates a more stable connection. As shown, the thick straight arrow indicates the current latency in transmitting the corresponding video frame to the client 210, but due to jitter there may be a range of arrival times for the video frames at the client 210 (e.g., the range enclosed by the dotted arrow). The variation in latency may also be due to one or more operations at the server, such as encoding 403 or transmitting 404, as well as network issues that introduce latency in transmitting the video frames to the client 210.

[0049] At 406, the client again decodes the compressed video frame for each encoded slice, generating a decoded slice A (shown without hash marks) that is now ready for display. In one embodiment, the decoding process 406 begins before the receiving process 405 is fully completed for the corresponding video frame. Furthermore, the start and / or end of the decoding 406 may or may not be aligned with the client VSYNC signal 312. At 407, the client displays the decoded rendered video frame on a display at the client. That is, the decoded video frame is placed in a display buffer from which it is streamed out to a display device, e.g., scanline by scanline. In one embodiment, the display process 407 (i.e., streaming out to the display device) begins after the decode process 406 is fully completed for the corresponding video frame, i.e., after the decoded video frame is fully resident in the display buffer. In another embodiment, the display process 407 begins before the decode process 406 is fully completed for the corresponding video frame. That is, streaming out to the display device begins at a display buffer address at which only a portion of the decoded frame buffer resides in the display buffer. The display buffer is then updated or filled with the remaining portions of the corresponding video frame in time for display, such that the display buffer updates occur before those portions are streamed out to the display. Furthermore, the start and / or end of display 407 is aligned with the client VSYNC signal 312.

[0050] In one embodiment, one-way latency 416 between server 260 and client 210 may be defined as the elapsed time between when scanout 402 begins and when display 407 begins. Embodiments of the present disclosure may align (e.g., synchronize frequency and adjust offset) VSYNC signals between the server and client to reduce one-way latency between the server and client and reduce variability in one-way latency between the server and client. For example, embodiments of the present disclosure can calculate an optimal adjustment to the offset 430 between the server VSYNC signal 311 and the client VSYNC signal 312 so that, even in the event of near-worst-case times for server processing such as encoding 403 and transmitting 404, near-worst-case network latency between the server 260 and the client 210, and near-worst-case client processing such as receiving 405 and decoding 406, the decoded and rendered video frame will be available in time for the display process 407. In other words, it is not necessary to determine the absolute offset between the server VSYNC and the client VSYNC; it is sufficient to adjust the offset so that the decoded and rendered video frame is available in time for the display process.

[0051] In particular, the frequencies of the server VSYNC signal 311 and the client VSYNC signal 312 may be adjusted by synchronization. The synchronization is achieved by adjusting either the server VSYNC signal 311 or the client VSYNC signal 312. For illustrative purposes, the adjustment is described with reference to the server VSYNC signal 311, but it will be understood that the adjustment may instead be performed on the client VSYNC signal 312. For example, as shown in FIG. 4, the server frame period 410 (e.g., the time between two occurrences 311c and 311d of the server VSYNC signal 311) is substantially equal to the client frame period 415 (e.g., the time between two occurrences 312a and 312b of the client VSYNC signal 312), indicating that the frequencies of the server VSYNC signal 311 and the client VSYNC signal 312 are also substantially equal.

[0052] To maintain synchronization of the frequencies of the server and client VSYNC signals, the timing of the server VSYNC signal 311 can be manipulated. For example, the vertical blanking interval (VBI) of the server VSYNC signal 311 can be increased or decreased over a period of time, e.g., to check for drift between the server VSYNC signal 311 and the client VSYNC signal 312. Manipulation of the vertical blanking interval (VBLANK) in the VBI provides for adjusting the number of scan lines used for VBLANK for one or more frame periods of the server VSYNC signal 311. A decrease in the number of VBLANK scan lines shortens the corresponding frame period (e.g., time interval) between two occurrences of the server VSYNC signal 311. Conversely, an increase in the number of VBLANK scan lines lengthens the corresponding frame period (e.g., time interval) between two occurrences of the VSYNC signal 311. In this way, the frequency of the server VSYNC signal 311 is adjusted so that the frequencies between the client and server VSYNC signals 311 and 312 are substantially the same frequency. Additionally, the offset between the server and client VSYNC signals can be adjusted by briefly increasing or decreasing the VBI before returning it to its original value. In one embodiment, the server VBI is coordinated. In another embodiment, the client VBI is coordinated. In yet another embodiment, instead of two devices (server and client), there may be multiple connected devices, each of which may have a corresponding VBI that is coordinated. In one embodiment, each of the multiple connected devices may be an independent peer device (e.g., without a server device). In another embodiment, the multiple devices may include one or more server devices and / or one or more client devices arranged in one or more server / client architectures, multi-tenant server / client(s) architectures, or some combination thereof.

[0053] Alternatively, the server's pixel clock (e.g., located in the southbridge of the server's northbridge / southbridge core logic chipset, or in the case of a discrete GPU, generating the pixel clock itself using its own hardware) can, in one embodiment, be manipulated to perform coarse and / or fine adjustments to the frequency of the server VSYNC signal 311 over a period of time to bring the frequency synchronization between the server VSYNC signal 311 and the client VSYNC signal 312 back into alignment. Specifically, the pixel clock of the server's southbridge may be overclocked or underclocked to adjust the overall frequency of the server's VSYNC signal 311. In that way, the frequency of the server VSYNC signal 311 is adjusted so that the frequencies between the client and server VSYNC signals 311 and 312 are substantially the same frequency. The offset between the server and client VSYNC can be adjusted by briefly increasing or decreasing the client-server pixel clock before restoring the pixel clock to its original value. In one embodiment, the server pixel clock is adjusted. In another embodiment, the client pixel clocks are aligned. In yet another embodiment, instead of two devices (a server and a client), there may be multiple connected devices, each of which may have a corresponding pixel clock that is aligned. In one embodiment, each of the multiple connected devices may be an independent peer device (e.g., without a server device). In another embodiment, the multiple connected devices may include one or more server devices and one or more client devices arranged in one or more server / client architectures, multi-tenant server / client(s) architectures, or some combination thereof.

[0054] In one embodiment, high-performance codecs (e.g., encoders and / or decoders) may be used to further reduce one-way latency between the cloud gaming server and the client. In traditional streaming systems involving streaming compressed media (such as streaming movies, television programs, and videos), when the streaming media is decompressed at the end target (e.g., the client), a significant amount of the decompressed video may be buffered at the client to accommodate variations in the encoding operation (e.g., longer encoding times), jitter that introduces transmission quality, and variations in the decoding operation (e.g., longer decoding times). Thus, traditional streaming systems may rely on average decoding capabilities and metrics (e.g., average decoding resources) to ensure that video frames are displayed at a desired rate (e.g., supporting 4K media at 60 Hz or displaying a video frame every time a client VSYNC signal occurs) as the decoded content accommodates latency variability.

[0055] However, in cloud gaming environments, buffering is highly limited (e.g., moving to zero buffering) to enable real-time gaming. As a result, variability introduced into the one-way latency between the cloud gaming server and the client can adversely affect downstream operations. For example, if it takes a long time to encode or decode a complex frame, the one-way latency will correspondingly increase (even for a single frame), ultimately increasing response times to the user and adversely affecting the user's real-time experience.

[0056] In one embodiment, for cloud gaming, it is beneficial to provide more powerful decoding and encoding resources that may be unnecessary compared to the needs of streaming video applications. Furthermore, as described in more detail below, encoder resources need to be optimized for the time to process frames that are long or require the longest processing. That is, in embodiments where the encoder can be adjusted to improve the trade-off between one-way latency and video quality in a cloud gaming system, the encoder adjustments may be based on monitoring client bandwidth, skipped frames, the number of encoded I-frames, the number of scene changes, and / or the number of video frames that exceed a target frame size, and the adjusted parameters may include the encoder bitrate, target frame size, maximum frame size, and quantization parameter (QP) value, where high-performance encoders and decoders help reduce the overall one-way latency between the cloud gaming server and the client.

[0057] 2A-2D , together with a detailed description of the various client devices 210 and / or cloud gaming network 290 (e.g., within game server 260), flow diagram 500 of FIG. 5 illustrates a cloud gaming method according to one embodiment of the present disclosure, in which encoding video frames involves adjusting encoder parameters with awareness of network transmission speed and reliability, as well as an overall latency target. A cloud gaming server is configured to stream content to one or more client devices over a network. This process results in smoother frame rates and more reliable latency, and one-way latency between the cloud gaming server and the client is reduced and made more consistent, thereby improving the smoothness of the client's display of video.

[0058] At 510, multiple video frames are generated when a video game is executed on a cloud gaming server. Typically, the cloud gaming server generates rendered video frames for multiple games. For example, game logic for a video game is built based on a game engine or game title processing engine. The game engine includes core functionality that can be used by the game logic to build a game environment for the video game. For example, some functions of the game engine may include a physics engine for simulating physical forces and collisions with objects in the game environment, a rendering engine for 2D or 3D graphics, collision detection, sound, animation, artificial intelligence, networking, streaming, etc. In this way, the game logic does not need to build from scratch the core functionality provided by the game engine.

[0059] The game logic in combination with the game engine is executed by a CPU and a GPU, which may be configured in an Accelerated Processing Unit (APU), that is, the CPU and GPU, along with shared memory, may be configured as a rendering pipeline for generating rendered video frames for the game, such that the rendering pipeline outputs the rendered images for the game, including corresponding color information for each pixel of a targeted and / or virtualized display, as video or image frames suitable for display. In particular, the CPU may be configured to generate one or more draw calls for a video frame, each draw call including commands stored in a corresponding command buffer that are executed by the GPU in the GPU pipeline. Generally, the graphics pipeline may perform shader operations on vertices of objects in a scene to generate texture values ​​for pixels of display. In particular, the graphics pipeline receives input geometry (e.g., vertices of objects in a gaming environment), and vertex shaders construct the primitives or polygons that make up the objects. The vertex shader programs may perform lighting, shading, shadowing, and other operations on the primitives. Depth buffering or Z-buffering is performed to determine which objects are visible when rendered from the corresponding viewpoint. Rasterization is performed to project objects in the 3D game environment onto the 2D plane defined by the viewpoint. Pixel-sized fragments are generated for the objects, and one or more fragments can contribute to the color of a pixel in the image. The fragments can be merged and / or blended to determine the combined color of each pixel in the corresponding video and stored in a frame buffer. Subsequent video frames are generated and / or rendered for display using a similarly configured command buffer, and multiple video frames are output from the GPU pipeline.

[0060] At 520, the method includes encoding a plurality of video frames at an encoder bitrate. In particular, the plurality of video frames are scanned into an encoder where they are compressed before streaming to a client using a streamer operating at the application layer. In one embodiment, each of the video frames rendered for the game can be composited and blended into a corresponding modified video frame with additional user interface features and then scanned into an encoder, which compresses and streams the modified video frame to the client. For brevity and clarity, the method of adjusting encoder parameters disclosed in FIG. 5 is described with respect to encoding multiple video frames, but it is understood that it also supports encoding modified video frames. The encoder is configured to compress the multiple video frames based on the described format. For example, when streaming media content from a cloud gaming server to a client, the Motion Picture Experts Group (MPEG) or H.264 standard may be implemented. In particular, the encoder may perform compression by video frame or by encoder slices of video frames, and as described above, each video frame may be compressed as one or more encoded slices. Generally, when streaming media, video frames are compressed as I-frames (intra-frames) or P-frames (predicted frames), each of which may be divided into encoded slices.

[0061] At 530, the client's maximum receive bandwidth is measured. In one embodiment, the maximum bandwidth experienced by the client is determined by a feedback mechanism from the client. Figure 6 illustrates measuring the client's 210 bandwidth by a streamer of a cloud gaming server, according to one embodiment of the disclosure, where the streamer 620 is configured to monitor and adjust the encoder 610 so that compressed video frames can be transmitted at a rate within the client's measured bandwidth. As shown, compressed video frames, encoded slices, and / or packets are delivered from an encoder 610 to a buffer 630 (e.g., first-in-first-out—FIFO). The encoder delivers the compressed video frames at an encoder fill rate 615. For example, the buffer may fill as fast as the encoder can generate compressed video frames, encoded slices 650, and / or packets of encoded slices 655. Additionally, the compressed video frames are drained from the buffer at a buffer drain rate 635 for delivery to the client 210 over the network 250. In one embodiment, the buffer drain rate 635 is dynamically adjusted to the client's measured maximum receive bandwidth. For example, the buffer drain rate 635 may be adjusted to be approximately equal to the client's measured maximum receive bandwidth. In one embodiment, the encoding of packets is performed at the same rate as they are transmitted, with both operations dynamically adjusted to the maximum available bandwidth available to the client.

[0062] In particular, the streamer 620, operating at the application layer, measures the maximum bandwidth of the client 210, such as by using a bandwidth tester 625. The application layer is used in the User Datagram Protocol / Internet Protocol (UDP / IP) suite of protocols used to interconnect network devices over the Internet. For example, the application layer defines the communication protocols and interface methods used to communicate between devices over an IP communication network. During testing, the streamer 620 provides additional buffered packets 640 (e.g., forward error correction (FEC) packets) to enable the buffer 630 to stream packets from a predefined bit rate, such as the maximum bandwidth being tested. In one embodiment, the client returns as feedback 690 to the streamer 620 the number of packets received over a range of incremental sequence identifiers (IDs), such as a range of video frames. For example, the client may report something like 145 of 150 video frames received with sequence IDs 100-250 (e.g., 150 video frames). In this way, the streamer 620 at the server 260 can calculate packet loss, and because the streamer 620 knows the amount of bandwidth transmitted (e.g., tested) during that sequence of packets, the streamer 620 can dynamically determine what the client's maximum bandwidth is at a particular point in time. The client's measured maximum bandwidth may be delivered as control information 627 from the streamer 620 to the buffer 630, allowing the buffer 630 to dynamically transmit packets at a rate approximately equal to the client's maximum bandwidth. In this way, the transmission rate of compressed video frames, encoded slices, and / or packets may be dynamically adjusted depending on the currently measured client's maximum bandwidth.

[0063] At 540, the encoding process is monitored by the streamer. That is, the encoding of multiple video frames is monitored. In one embodiment, the monitoring is performed by the client 210, and feedback and / or adjustment control signals are provided back to the encoder. In another embodiment, the monitoring is performed by the streamer 620, etc., at the cloud gaming server 260. For example, monitoring of the encoding of the video frames may be performed by the monitoring and adjustment unit 629 of the streamer 620. Various encoding characteristics and / or operations may be tracked and / or monitored. For example, in one embodiment, the rate of occurrence of I-frames within a plurality of video frames may be tracked and / or monitored. Further, in one embodiment, the rate of occurrence of scene changes within a plurality of video frames may be tracked and / or monitored. Also, in one embodiment, the number of video frames that exceed a target frame size may be tracked and / or monitored. Also, in one embodiment, the encoder bit rate used to encode one or more video frames may be tracked and / or monitored.

[0064] At 550, encoder parameters are dynamically adjusted based on monitoring of the encoding of video frames. That is, monitoring of the encoding of video frames affects how the encoder operates when compressing current and future video frames received at the encoder. In particular, the monitoring and adjustment unit 629 is configured to determine which encoder parameters to adjust in response to monitoring of the encoding of video frames and analysis performed on the monitored information. Control signals 621 are transmitted from the monitoring and adjustment unit 629 back to the encoder 610 for use in configuring the encoder. Encoder parameters for adjustment include quantization parameters (QP) (e.g., minimum QP, maximum QP) or quality parameters, target frame size, maximum frame size, etc.

[0065] The adjustments are made with awareness of the network transmission speed and reliability, as well as the overall latency goal. In one embodiment, smooth video playback is prioritized over low latency or image quality. For example, skipping the encoding of one or more video frames is disabled. Specifically, the balance between image resolution or image quality (e.g., 60 Hz) and latency is adjusted using various encoder parameters. In particular, VSYNC signals at the cloud gaming server and client can be synchronized and offset so that one-way latency between the cloud gaming server and client can be reduced, thereby reducing the need to skip video frames to promote low latency. Synchronizing and offsetting VSYNC signals also provides for overlapping operations at the cloud gaming server (scanout, encode, and transmit), overlapping operations at the client (receive, decode, render, display), and / or overlapping operations at the cloud gaming server and client, all of which facilitate reduced one-way latency, reduced variability in one-way latency, real-time generation and display of video content, and consistent video playback at the client.

[0066] In one embodiment, the encoder bitrate is monitored taking into account subsequent frames and their complexity (e.g., predicted scene changes) to predict demand for client bandwidth, where the encoder bitrate can be adjusted according to the predicted demand. For example, when prioritizing smooth video playback, the encoder monitoring and adjustment unit 629 can be configured to determine that the encoder bitrate used exceeds the measured maximum receive bandwidth. In response, the encoder bitrate can be reduced, and frame sizing can also be reduced. When smoothness is a priority, it is desirable to use an encoder bitrate lower than the maximum receiving bandwidth (e.g., an encoder bitrate of 10 Mbps for a maximum receiving bandwidth of 15 Mbps). That way, even if the encoded frames spike above the maximum frame size, they can still be transmitted within 60 Hz (Hertz). In particular, the encoder bitrate can be translated into frame size. A given bitrate and target speed for a video game (e.g., 60 frames per second) translates into the average size of encoded video frames. For example, with an encoder bitrate of 15 Mbps and a given target speed of 60 frames per second, 60 encoded frames share 15 Mbps, with each encoded frame having approximately 250k encoded bits. In this way, controlling the encoder bit rate also controls the frame size of the encoded video frames, so that increasing the encoder bit rate results in more bits for encoding (more precision), and decreasing the encoder bit rate results in fewer bits for encoding (less precision). Similarly, when the encoder bit rate used to encode a group of video frames is within the measured maximum receive bandwidth, the encoder bit rate can be increased and the frame size can also be increased.

[0067] In one embodiment, if smoothness of video playback is a priority, the encoder monitoring and adjustment unit 629 may be configured to determine whether the encoder bitrate used to encode a group of video frames exceeds a measured maximum receive bandwidth. For example, the encoder bitrate may be detected to be 15 Megabits per second (Mbps), while the maximum receive bandwidth may currently be 10 Mbps. In this way, the encoder pushes out more bits than can be sent to the client without increasing one-way latency. As mentioned above, if smoothness is a priority, it may be desirable to use an encoder bitrate lower than the maximum receive bandwidth. In the example above, for the 10 Mbit / s maximum receive bandwidth introduced above, it may be acceptable to have the encoder bitrate set to 10 Mbit / s or less. That way, if the encoded frames spike above the maximum frame size, they can still be transmitted within 60 Hz. The QP value can be adjusted accordingly, with or without reducing the encoder bitrate, and the QP controls the precision used when compressing the video frames. That is, QP controls the amount of quantization performed (e.g., compressing a variable range of values ​​within a video frame into a single quantum value). In H.264, QP ranges from "0" to "51". For example, a QP value of "0" means less quantization, less compression, more precision, and higher quality. For example, a QP value of "51" means less quantization, less compression, more precision, and higher quality. Specifically, the QP value can be increased so that the encoding of a video frame is performed with less precision.

[0068] In one embodiment, when prioritizing smoothness of video playback, encoder monitoring by the monitoring and adjustment unit 629 may be configured to determine that the encoder bit rate used to encode a group of video frames from the plurality of video frames is within the maximum receive bandwidth. As previously introduced, when prioritizing smoothness, it may be desirable to use an encoder bit rate lower than the maximum receive bandwidth. In this manner, there is excess bandwidth available when transmitting the group of video frames. The excess bandwidth can be determined. A QP value can be adjusted accordingly, where the QP controls the precision used when compressing the video frames. In particular, the QP value can be reduced based on the excess bandwidth, so that the encoding is performed more precisely.

[0069] In another embodiment, the characteristics of an individual video game are taken into account when determining I-frame processing and QP settings, particularly when prioritizing smooth video playback. For example, if a video game does not have frequent "scene changes" (e.g., only camera cuts), it may be desirable to have larger I-frames (lower QP or higher encoder bitrate). That is, the number of video frames identified as having a scene change within a group of video frames from multiple video frames being compressed is determined to be less than a threshold number of scene changes. That is, the streaming system can handle the number of scene changes under the current conditions (e.g., measured client bandwidth, required latency, etc.). The QP value can be adjusted accordingly, where the QP controls the precision used when compressing video frames. In particular, the QP value can be reduced so that encoding is performed with greater precision.

[0070] On the other hand, if a video game experiences frequent "scene changes" during gameplay, it may be desirable to keep the I-frame size small (e.g., a higher QP or a lower encoder bitrate). That is, the number of video frames identified as having scene changes within a group of video frames from multiple video frames being compressed is determined to meet or exceed a threshold number of scene changes. That is, the video game is generating too many scene changes for the current conditions (e.g., measured client bandwidth, required latency, etc.). Accordingly, the QP value can be adjusted, where the QP controls the precision used when compressing the video frames. In particular, the QP value can be increased so that encoding is performed with less precision.

[0071] In another embodiment, encoding patterns may be taken into consideration when determining I-frame processing and QP settings, especially when prioritizing smooth video playback. For example, if an encoder generates I-frames infrequently, it may be desirable to have larger I-frames (lower QP or higher encoder bitrate). That is, within a group of video frames from multiple video frames being compressed, the number of video frames compressed as I-frames is within or below a threshold number of I-frames. That is, the streaming system can handle the number of I-frames given the current conditions (e.g., measured client bandwidth, required latency, etc.). The QP value may be adjusted accordingly, where the QP controls the precision used when compressing video frames. In particular, the QP value may be reduced so that encoding is performed with greater precision.

[0072] If the encoder generates I-frames frequently, it may be desirable to keep the I-frame size small (e.g., increase the QP or lower the encoder bitrate). That is, within a group of video frames from multiple video frames being compressed, the number of video frames compressed as I-frames is within or exceeds a threshold number of I-frames. That is, the video game is generating too many I-frames for the current conditions (e.g., measured client bandwidth, required latency, etc.). Accordingly, the QP value can be adjusted, where QP controls the precision used when compressing video frames. In particular, the QP value can be increased so that encoding is performed with less precision.

[0073] In another embodiment, encoding patterns may be taken into consideration when rotating the encoder, especially when prioritizing smooth video playback. For example, if the encoder frequently falls short of the target frame size, it may be desirable to increase the target frame size. That is, it is determined that within a group of video frames from multiple video frames to be compressed and transmitted at the transmission rate, the number of video frames falls below a threshold. Each of the number of video frames is within the target frame size (i.e., equal to or smaller than the target frame size). At least one of the target frame size and the maximum frame size is increased accordingly.

[0074] On the other hand, if the encoder frequently exceeds the target frame size, it may be desirable to reduce the target frame size. That is, within a group of video frames from a plurality of video frames to be compressed and transmitted at a transmission rate, it is determined whether a number of video frames meets or exceeds a threshold. Each of the number of video frames exceeds the target frame size. At least one of the target frame size and the maximum frame size is reduced accordingly.

[0075] 7A illustrates setting an encoder's quantization parameter (QP) to optimize quality and buffer utilization at a client, according to one embodiment of the present disclosure. Graph 720A illustrates the vertical frame size (in bytes) for each generated frame, shown horizontally. The target frame size and maximum frame size are static. In particular, line 711 illustrates the maximum frame size, and line 712 illustrates the target frame size, where the maximum frame size is larger than the target frame size. As shown in graph 720A, there are multiple peaks that include compressed video frames that exceed the target frame size of line 712. Video frames that exceed the target frame size may require multiple frame periods to encode and / or transmit from the cloud gaming server, risking playback jitter (e.g., increased one-way latency).

[0076] Graph 700B shows the encoder response after the QP is set based on the target frame size, maximum frame size, and QP range (e.g., minimum QP and maximum QP) to optimize encoding quality and buffer utilization at the client. For example, the QP may be adjusted and / or tuned based on encoder monitoring of the encoder bitrate, the frequency of scene changes, and the frequency of I-frame generation, as described above. Graph 700B shows the frame size (in bytes) vertically for each generated frame, as shown horizontally. The target frame size at line 712 and the maximum frame size at line 711 remain in the same position as in graph 700A. After the QP adjustment and / or tuning, the number of peaks containing compressed video frames exceeding the target frame size at line 712 is reduced compared to graph 700A. That is, the QP has been adjusted to optimize the encoding of video frames (i.e., to fit within the target frame size) for the current conditions (e.g., measured client bandwidth, required latency, etc.).

[0077] 7B is a diagram illustrating the adjustment of target frame size, maximum frame size, and / or QP (e.g., minimum QP and / or maximum QP) encoder settings to reduce the occurrence of I-frames that exceed the true target frame size supported by the client, according to one embodiment of the present disclosure. For example, the QP may be adjusted and / or tuned based on encoder monitoring of the encoder bitrate, the frequency of scene changes, and the frequency of I-frame generation, as described above.

[0078] Graph 720A shows the vertical frame size (in bytes) for each generated frame, as shown horizontally. For illustrative purposes, graphs 720A of FIG. 7B and 700A of FIG. 7A may reflect similar encoder conditions and are used for encoder adjustment. In graph 720A, the target frame size and maximum frame size are static. In particular, line 711 shows the maximum frame size, and line 712 shows the target frame size, where the maximum frame size is larger than the target frame size. As shown in graph 720A, there are multiple peaks containing compressed video frames that exceed the target frame size at line 712. Video frames that exceed the target frame size may take multiple frame periods to encode and / or transmit from the cloud gaming server, risking playback jitter (e.g., increased one-way latency). For example, the peak reaching the maximum frame size at line 711 may be an I-frame that takes 16 milliseconds or more to transmit to the client, which causes playback jitter by increasing the one-way latency between the cloud gaming server and the client.

[0079] Graph 720B shows the encoder response after at least one of the target frame size and / or maximum frame size has been adjusted to reduce the occurrence of I-frames that exceed the true target frame size supported by the client. The true target frame size may be adjusted based on measured client bandwidth and / or encoder monitoring, including monitoring the encoder bit rate, scene change frequency, and I-frame generation frequency, as described above.

[0080] Graph 720B shows the vertical frame size (in bytes) for each generated frame, as shown horizontally. Compared to graph 720A, the target frame size for line 712' and the maximum frame size for line 711' are lower. For example, the target frame size for line 712' is smaller than that for line 712, and the maximum frame size for line 711' is smaller than that for line 711. After adjusting the target frame size and / or maximum frame size, the maximum size of the peaks of compressed video frames that exceed the target frame size 712' has been reduced for better transmission. Furthermore, when compared to graph 700A, the number of peaks containing compressed video frames that exceed the target frame size 712' has also been reduced. For example, only one peak is shown in graph 720B. In other words, the target frame size and / or maximum frame size has been adjusted to optimize the encoding of video frames (i.e., fit within the target frame size) for the current conditions (e.g., measured client bandwidth, required latency, etc.).

[0081] 2A-2D , together with a detailed description of the various client devices 210 and / or cloud gaming network 290 (e.g., within game server 260), flow diagram 800 of FIG. 8 illustrates a method of cloud gaming according to one embodiment of the present disclosure, in which encoding of video frames includes determining when to skip video frames or when to delay the encoding and transmission of video frames when the encoding is long or when the resulting video frames are large (e.g., when encoding I-frames). In particular, the decision to skip video frames is made taking into account network transmission speed and reliability, and overall latency targets. This process results in smoother frame rates and more reliable latency, making one-way latency between the cloud gaming server and the client lower and more consistent, thereby improving the smoothness of the client's display of video.

[0082] At 810, a plurality of video frames are generated when executing a video game at a cloud gaming server operating in streaming mode. Typically, a cloud gaming server generates rendered video frames for a plurality of games. For example, the generation of rendered video frames for a game is described at 510 in FIG. 5 and is applicable to the generation of video frames in FIG. 8. For example, game logic for a video game may be built on a game engine or game title processing engine. The game logic in combination with the game engine may be executed by a CPU and a GPU, and the CPU and GPU, along with shared memory, may be configured as a rendering pipeline for generating rendered video frames for the game, such that the rendering pipeline outputs the rendered images for the game as video or image frames suitable for display, including color information corresponding to each pixel of a target and / or virtualized display.

[0083] At 820, a scene change is predicted for a first video frame of the video game, the scene change being predicted before the first video frame is generated. In one embodiment, the game logic can cause the CPU to recognize a scene change while the video game is running. For example, the game logic or add-on logic can include code (e.g., scene change logic) that predicts a scene change when generating video frames, such as predicting that a range of video frames will include at least one scene change or predicting that a particular video frame is a scene change. In particular, game logic or add-on logic configured for scene change prediction analyzes game state data collected during the execution of a video game to determine and / or anticipate and / or predict when there will be a scene change, such as within the next X number of frames (e.g., a range), or for an identified video frame. For example, a scene change can be predicted when a character moves from one scene to another in a virtualized game environment, or when a character exits a level and transitions to another level in a video game, or when the transition occurs between two video frames (e.g., a scene cut in a cinematic sequence, or the start of interactive gameplay after a series of menus), and the scene change may be represented by video frames that include a large and complex scene of the virtualized game world or environment.

[0084] Game state data may define the state of the game at that time and may include game characters, game objects, game object attributes, game attributes, game object states, graphic overlays, a character's location within the game world of the player's gameplay, a gameplay scene or game environment, a level of the game application, a character's assets (e.g., weapons, tools, bombs, etc.), loadouts, a character's skill set, a game level, character attributes, character location, number of lives remaining, total number of lives available, armor, trophies, time counter values, and other asset information.

[0085] At 830, a scene change hint is generated and sent to the encoder, the hint indicating that the first video frame is a scene change. In this manner, notification of the upcoming scene change can be provided to the encoder, which can adjust its encoding operations when compressing the identified video frame. The notification provided as a scene change hint can be distributed via an API used to communicate between components or between applications running on components of cloud gaming server 260. In one embodiment, the API may be a GPU API, for example, the API may be running on or called by game logic and / or add-on logic configured to detect scene changes to communicate with the encoder. In one embodiment, the scene change hints may be provided as data control packets formatted such that all components receiving the data control packets can understand what type of information is contained in the data control packet and can understand the appropriate reference to the corresponding rendered video frame. In one embodiment, the communication protocol used for the API, the format for the data control packets, may be defined in a corresponding software development kit (SDK) for the video game.

[0086] At 840, the first video frame is delivered to an encoder. As described above, the game-generated video frame may be composited and blended with additional user interface features to result in a modified video frame that is scanned into the encoder. The encoder is configured to compress the first video frame based on a desired format, such as the MPEG or H.264 standard used for streaming media content from the cloud gaming server to the client. When streaming, video frames are encoded as P-frames, and the next video frame is then encoded as another I-frame, until there is a scene change or the currently encoded frame cannot reference a key frame (e.g., a previous I-frame), in which case the first video frame is encoded as an I-frame based on a scene change hint, and the I-frame may be encoded without referencing any other video frames (e.g., standalone as a key image).

[0087] At 850, the maximum receive bandwidth of the client is measured. As previously mentioned, the maximum bandwidth experienced by the client may be determined by means of a feedback mechanism from the client, as shown in operation 530 of Figures 5 and 6. In particular, a streamer of the cloud gaming server may be configured to measure the client's bandwidth.

[0088] At 860, the encoder receives a second video frame. That is, the second video frame is received after a scene change and is compressed after the first video frame is compressed. The encoder also determines whether to not encode the second video frame (or subsequent video frames) or to delay encoding the second video frame (or subsequent video frames). This determination is made based on the client's maximum receive bandwidth and the target resolution of the client display. That is, the decision to skip or delay encoding takes into account the bandwidth available to the client. Generally, if the current bandwidth experienced by the client is sufficient, a video frame generated and encoded for the client's target display can quickly return to low one-way latency after incurring a latency hit (e.g., generating a large I-frame for a scene change), and the second video frame (and / or subsequent video frames) can still be encoded with a delay. On the other hand, if the current bandwidth experienced by the client is not sufficient, the second video frame (and / or subsequent video frames) may be skipped during the encoding process and not delivered to the client. In this way, it is possible to have fewer skipped frames (and lower latency) if the bandwidth to the client exceeds the bandwidth required to support the target resolution of the display at the client.

[0089] In one embodiment, compressed video frames are transmitted from the server to the client at a rate based on the maximum bit rate or bandwidth available over the network at a particular time. In this manner, the transmission rate of encoded slices of compressed video frames and / or packets of encoded slices is dynamically adjusted according to the currently measured maximum bandwidth. Video frames may be transmitted as they are encoded, such that transmission occurs as soon as encoding is complete, without waiting for the next occurrence of a server VSYNC signal or for the entire video frame to be encoded.

[0090] Furthermore, in one embodiment, encoding of packets is performed at the same rate as they are transmitted, with both operations dynamically adjusting to the maximum available bandwidth available to the client. Also, the encoder bitrate can be monitored, taking into account the upcoming frame and its complexity (e.g., predicted scene changes) to predict client bandwidth demands, and the encoder bitrate can be adjusted according to the predicted demand. Furthermore, the encoder bitrate can be communicated to the client, so that the client can adjust its decoding speed accordingly to match the encoder bitrate.

[0091] In one embodiment, when the transmission rate to the client is low relative to the target resolution of the client display, the second video frame is skipped by the encoder. That is, the second video frame is not encoded. In particular, the transmission rate to the client for a group of compressed video frames exceeds the maximum receive bandwidth. For example, the transmission rate to the client may be 15 megabytes per second (Mbps), but the client's measured receive bandwidth may currently be 5-10 Mbps. In this way, if every video frame were continually pushed to the client, the one-way latency between the cloud gaming server and the client would increase. To promote low latency, the second and subsequent video frames may be skipped by the encoder.

[0092] 9A shows a sequence 900A of video frames being compressed by an encoder that drops the encoding of a second video frame 920 after encoding a first I-frame 905 when client bandwidth is low relative to the target resolution of the client's display, according to one embodiment of the present disclosure. The encoding and transmission blocks of video frames are shown in relation to a VSYNC signal 950. In particular, when extra bandwidth is not available, I-frames take longer to encode, causing one or more skipped frames to keep one-way latency low, which may include the time to display the video frames at the client. As shown, skipping one or more video frames after the I-frame allows for a quick return to low one-way latency (e.g., within one or two frame periods), whereas not skipping the encoding of video frames would otherwise require several frame periods to return to low one-way latency.

[0093] For example, a sequence of video frames 900A includes one encoded I-frame 905, with the remaining frames encoded as P-frames. For purposes of illustration, block 901 and block 902 are encoded as P-frames before block 905, which is encoded as an I-frame. The encoder then compresses the video frames as P frames until the next scene change or until the video frame cannot reference a previous key frame (such as an I frame). Generally, the encoding time for an I frame block may take longer than a P frame block. For example, the encoding time for an I frame block 905 may exceed one frame period. In some cases, the encoding time between a P frame and an I frame may generally be approximately the same, especially when using a high-powered encoder.

[0094] However, the transmission times between I-frames and P-frames differ significantly. As shown, the various transmission times are shown relative to the corresponding encoded video frames. For example, transport block 911 of encoded P-frame block 901 is shown with low latency, such that encoding block 901 and transport block 911 can be performed within one frame period. Also, transport block 912 of encoded P-frame block 902 is shown with low one-way latency, such that encoding block 902 and transport block 912 can also be performed within one frame period.

[0095] On the other hand, transmission block 915A of encoded I-frame block 905 is shown with higher one-way latency, such that encoding block 905 and transmission block 915A occur over several frame periods, thereby introducing jitter into the one-way latency between the cloud gaming server and the client. To provide a real-time experience to the user with less one-way latency, a buffer at the client may not be used to correct for jitter. In that case, the encoder may decide to skip encoding one or more video frames after the I-frame is encoded. For example, video frame 920 is dropped by the encoder, in which case transmission of the encoded video frames reverts to one of low one-way latencies around highlighted region 910 after five subsequent video frames have been encoded as P-frames and transmitted to the client. That is, the fourth or fifth P-frame encoded after I-frame block 905 is encoded will also be transmitted to the client within the same frame period, thereby returning to low one-way latency between the cloud gaming server and the client.

[0096] In one embodiment, if the transmission rate to the client is high relative to the target resolution of the client display, the second video frame is still compressed by the encoder after the delay (i.e., waiting until the I-frame is encoded). In particular, the transmission rate to the client of the group of compressed video frames is within the maximum receive bandwidth. For example, the transmission rate to the client may be 13 megabytes per second (Mbps), but the client's measured receive bandwidth may currently be 15 Mbps. In this way, there is no delay in receiving the encoded video frames at the client, and therefore no increase in one-way latency between the cloud gaming server and the client.

[0097] Furthermore, because the VSYNC signals at the cloud gaming server and client can be synchronized and offset, the one-way latency between the cloud gaming server and client can be reduced, thereby compensating for latency variability caused by jitter at the server or at the client during transmission over the network. Additionally, the synchronization and offset of the VSYNC signal provides for overlapping operations at the cloud gaming server (scan out, encode, and transmit), overlapping operations at the client (receive, decode, render, display), and / or overlapping operations at the cloud gaming server and client, all of which facilitate compensation for latency variability introduced by server or network or client jitter, reduction in one-way latency, reduction in one-way latency variability, real-time generation and display of video content, and consistent video playback at the client.

[0098] FIG. 9B shows a sequence 900B of video frames being compressed by an encoder that takes into account the bandwidth available to the client, resulting in lower latency and fewer or no skipped frames when the bandwidth exceeds the bandwidth required to support the target resolution of the client display, according to one embodiment of the present disclosure. In particular, in sequence 900B, a video frame is encoded as an I-frame, and subsequent video frames are also encoded normally, but after a delay in encoding the I-frame, when the client bandwidth is moderate to moderate for the target resolution of the client display. Because there is moderate bandwidth availability, a moderate amount of excess bandwidth is available to compensate for latency variability (e.g., jitter) between the cloud gaming server and the client, so that frame skipping can be avoided and a return to low one-way latency can be achieved relatively quickly (e.g., within two to four frame periods). The encoding and transmission blocks of video frames are shown relative to VSYNC signal 950.

[0099] The sequence of video frames 900B includes one encoded I-frame 905, with the remaining frames being encoded as P-frames. For illustrative purposes, encode block 901 and encode block 902 as P-frames before encode block 905 is encoded as an I-frame. The encoder then compresses the video frames as P-frames until the next scene change or until the video frames can no longer reference a previous key frame (such as an I-frame). In general, the encoding time for an I-frame block may take longer than a P-frame block, and the transmission of an I-frame may take longer than one frame period. For example, the encoding and transmission time for I-frame block 905 exceeds one frame period. Also shown are various transmission times relative to the corresponding encoded video frames. For example, the encoding and transmission of the video frame preceding I-frame block 905 is shown with low one-way latency, such that the corresponding encoding and transmission blocks can be performed within one frame period. However, transmission block 915B of encoded I-frame block 905 is shown with a higher one-way latency, such that encoding block 905 and transmission block 915B occur over more than one frame period, thereby introducing jitter into the one-way latency between the cloud gaming server and the client. As mentioned above, encoding time can be further reduced by adjusting one or more encoder parameters (e.g., QP, target frame size, maximum frame size, encoder bitrate, etc.) That is, the second or subsequent video frames after an I-frame are encoded with less precision when the transmission rate to the client is medium relative to the target resolution of the client display, and are encoded with less precision when the transmission rate is high relative to the target resolution.

[0100] After I-frame block 905, the encoder continues compressing video frames, although they may be temporarily delayed due to the encoding of the I-frames. Again, the synchronization and offset of the VSYNC signal provides for overlapping operations at the cloud gaming server (scan-out, encode, and transmit), overlapping operations at the client (receive, decode, render, display), and / or overlapping operations at the cloud gaming server and client, all of which facilitate compensating for variability in one-way latency introduced by server or network or client jitter, reducing one-way latency, reducing variability in one-way latency, real-time generation and display of video content, and consistent video playback at the client.

[0101] Because the client bandwidth is moderate with respect to the target resolution of the client display, transmission of the encoded video frames returns to one of low one-way latencies around highlighted region 940, such as after two or three subsequent video frames have been encoded as P-frames and transmitted to the client. Within region 940, P-frames encoded after I-frame block 905 is encoded are also transmitted to the client within the same frame period, thereby returning to low one-way latencies between the cloud gaming server and the client.

[0102] 9C shows a sequence 900C of video frames being compressed by an encoder that takes into account the bandwidth available to the client, such that if the bandwidth exceeds the bandwidth required to support the target resolution of the client display, there may be no or fewer skipped frames while still having lower one-way latency, according to one embodiment of the present disclosure. In particular, in sequence 900C, a video frame is encoded as an I-frame, and subsequent video frames are also encoded normally, but after the delay of encoding the I-frame if the client bandwidth is high relative to the target resolution of the client display. Because of high bandwidth availability, a large amount of excess bandwidth is available to compensate for variability (e.g., jitter) in one-way latency between the cloud gaming server and the client, so that frame skipping can be avoided and a return to low one-way latency can be achieved quickly (e.g., within one to two frame periods). The encoding and transmission blocks of a video frame are shown in relation to the VSYNC signal 950.

[0103] Similar to Figure 9B, the sequence of video frames 900C in Figure 9C includes one encoded I-frame 905, with the remaining frames encoded as P-frames. For illustrative purposes, encode block 901 and encode block 902 as P-frames before encode block 905 is encoded as an I-frame. The encoder then compresses video frames as P-frames until the next scene change or until the video frames can no longer reference previous key frames (e.g., I-frames). In general, the encoding time for I-frame blocks may take longer than P-frame blocks. For example, the encoding time for I-frame block 905 may exceed one frame period. Also shown are various transmission times relative to the corresponding encoded video frames. For example, the encoding and transmission of the video frame preceding I-frame block 905 is shown with low latency, such that the corresponding encoding and transmission blocks can be performed within one frame period. However, transmission block 915C of encoded I-frame block 905 is shown with higher latency, such that encoding block 905 and transmission block 915C occur over two or more frame periods, thereby introducing jitter into the one-way latency between the cloud gaming server and the client. As previously mentioned, encoding time can be further reduced by adjusting one or more encoder parameters (e.g., QP, target frame size, maximum frame size, encoder bitrate, etc.).

[0104] After I-frame block 905, the encoder continues compressing video frames, although they may be temporarily delayed due to the encoding of the I-frames. Again, the synchronization and offset of the VSYNC signal provides for overlapping operations at the cloud gaming server (scan out, encode, and transmit), overlapping operations at the client (receive, decode, render, display), and / or overlapping operations at the cloud gaming server and client, all of which facilitate compensating for latency variability introduced by server or network or client jitter, reducing one-way latency, reducing one-way latency variability, real-time generation and display of video content, and consistent video playback at the client. Because the client bandwidth is high relative to the target resolution of the client display, transmission of the encoded video frames reverts to one of low one-way latencies around highlighted region 970, such as after one or two subsequent video frames are encoded as P-frames and transmitted to the client. Within region 970, P-frames encoded after I-frame block 905 is encoded are also transmitted to the client within one frame period (although they straddle either side of the occurrence of the VSYNC signal), thereby returning to low one-way latencies between the cloud gaming server and the client.

[0105] 10 illustrates components of an exemplary device 1000 that can be used to implement aspects of various embodiments of the present disclosure. For example, FIG. 10 illustrates an exemplary hardware system suitable for streaming media content and / or receiving streamed media content, including providing encoder adjustments that improve the trade-off between one-way latency and video quality in a cloud gaming system for the purpose of reducing latency and providing more consistent latency between clouds, and to improve smoothness of client display of video. The encoder adjustments may be based on monitoring of client bandwidth, skipped frames, number of encoded I-frames, number of scene changes, and / or number of video frames exceeding a target frame size. The adjusted parameters may include an encoder bitrate, a target frame size, a maximum frame size, and a quantization parameter (QP) value. According to embodiments of the present disclosure, high performance encoders and decoders serve to reduce overall one-way latency between the cloud gaming server and the client. The block diagram illustrates a device 1000, which may incorporate or be a personal computer, server computer, game console, mobile device, or other digital device, each of which is suitable for implementing embodiments of the present invention. Device 1000 includes a central processing unit (CPU) 1002 for executing software applications and optionally an operating system. CPU 1002 may be comprised of one or more homogeneous or heterogeneous processing cores.

[0106] According to various embodiments, CPU 1002 is one or more general-purpose microprocessors having one or more processing cores. Further embodiments can be implemented using one or more CPUs with a microprocessor architecture specifically adapted for highly parallel, computationally intensive uses, such as applications configured for graphics processing during game execution, media and interactive entertainment applications, etc.

[0107] Memory 1004 stores applications and data for use by CPU 1002 and GPU 1016. Storage 1006 provides non-volatile storage and other computer-readable media for applications and data and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other optical storage devices and signal transmission and storage media. User input device 1008 communicates user input from one or more users to device 1000; examples of device 1000 may include a keyboard, mouse, joystick, touchpad, touchscreen, still or video recorder / camera, and / or microphone. The network interface 1009 enables the device 1000 to communicate with other computer systems over electronic communications networks, which may include wired or wireless communications over local area networks and wide area networks such as the Internet. The audio processor 1012 is adapted to generate analog or digital audio output from instructions and / or data provided by the CPU 1002, memory 1004, and / or storage 1006. The components of the device 1000, including the CPU 1002, graphics subsystem including the GPU 1016, memory 1004, data storage 1006, user input devices 1008, network interface 1009, and audio processor 1012, are connected via one or more data buses 1022.

[0108] A graphics subsystem 1014 is further connected to a data bus 1022 and to the components of device 1000. The graphics subsystem 1014 includes a graphics processing unit (GPU) 1016 and a graphics memory 1018. The graphics memory 1018 includes display memory (e.g., a frame buffer) used to store pixel data for each pixel of an output image. The graphics memory 1018 can be integrated into the same device as the GPU 1016, connected to the GPU 1016 as a separate device, and / or implemented within memory 1004. Pixel data can be provided directly from CPU 1002 to graphics memory 1018. Alternatively, CPU 1002 provides data and / or instructions defining desired output images to GPU 1016, from which GPU 1016 generates pixel data for one or more output images. The data and / or instructions defining the desired output images can be stored in memory 1004 and / or graphics memory 1018. In one embodiment, GPU 1016 includes 3D rendering functionality for generating pixel data for output images from instructions and data defining geometry, lighting, shading, texturing, motion, and / or camera parameters for a scene. GPU 1016 can further include one or more programmable execution units capable of executing shader programs.

[0109] Graphics subsystem 1014 periodically outputs pixel data for images from graphics memory 1018 to be displayed on display device 1010 or projected by a projection system (not shown). Display device 1010 can be any device capable of displaying visual information in response to signals from device 1000, including CRTs, LCDs, plasma displays, and OLED displays. Device 1000 can provide analog or digital signals to display device 1010, for example.

[0110] Other embodiments for optimizing the graphics subsystem 1014 can include multi-tenancy GPU operation, where a GPU instance is shared among multiple applications and the GPU supporting a single game is distributed. The graphics subsystem 1014 can be configured as one or more processing devices.

[0111] For example, graphics subsystem 1014 may be configured to perform multi-tenancy GPU functions, and in one embodiment, one graphics subsystem may implement the graphics and / or rendering pipeline for multiple games, i.e., graphics subsystem 1014 is shared between multiple games being run.

[0112] In other embodiments, graphics subsystem 1014 includes multiple GPU devices combined to perform graphics processing for a single application running on a corresponding CPU. For example, multiple GPUs can perform an alternative form of frame rendering, where GPU1 renders a first frame and GPU2 renders a second frame in successive frame periods until the last GPU is reached, so that the first GPU renders the next video frame (e.g., if there are only two GPUs, GPU1 renders the third frame). That is, GPUs cycle when rendering frames. Rendering operations can overlap, such that GPU2 can begin rendering the second frame before GPU1 finishes rendering the first frame. In another embodiment, multiple GPU devices can be assigned different shader operations in the rendering pipeline and / or graphics pipeline. A master GPU performs main rendering and compositing. For example, in a group including three GPUs, master GPU1 can perform main rendering (e.g., a first shader operation) and compositing the output from slave GPU2 and slave GPU3, slave GPU2 can perform a second shader operation (e.g., fluid effects such as rivers), slave GPU3 can perform a third shader operation (e.g., particle smoke), and master GPU1 composites the results from each of GPU1, GPU2, and GPU3. In this manner, various GPUs can be assigned to perform various shader operations (e.g., flag waving, wind, smoke generation, fire, etc.) to render a video frame. In yet another embodiment, each of the three GPUs can be assigned to a different object and / or portion of the scene corresponding to the video frame. In the above-described embodiments and implementations, these operations can be performed in the same frame period (concurrently in parallel) or in different frame periods (sequentially in parallel).

[0113] Accordingly, the present disclosure describes methods and systems configured for streaming media content and / or receiving streamed media content, including providing encoder adjustments to improve the trade-off between one-way latency and video quality in a cloud gaming system, where the encoder adjustments may be based on monitoring client bandwidth, skipped frames, number of encoded I-frames, number of scene changes, and / or number of video frames exceeding a target frame size, and the adjusted parameters may include an encoder bitrate, a target frame size, a maximum frame size, and a quantization parameter (QP) value, where the high performance encoder and decoder serve to reduce the overall one-way latency between the cloud gaming server and the client.

[0114] It should be understood that the various embodiments defined herein can be combined or assembled into specific implementations using various features disclosed herein. Thus, the examples provided are only some possible examples and are not limited to the various implementations in which many more implementations can be defined by combining various elements. In some instances, an implementation may include fewer elements without departing from the spirit of the disclosed or equivalent implementations.

[0115] Embodiments of the present disclosure may be practiced with a variety of computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. Embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.

[0116] With the above embodiments in mind, it should be understood that embodiments of the present disclosure can employ various computer-implemented operations involving data stored in computer systems. These operations are operations requiring physical manipulation of physical quantities. Any of the operations described herein that form part of embodiments of the present disclosure are useful machine operations. Embodiments of the disclosure also relate to devices or apparatus for performing these operations. An apparatus may be specially constructed for the required purposes. Alternatively, the apparatus may be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general-purpose machines can be used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct a more specialized apparatus to perform the required operations.

[0117] The present disclosure can also be embodied as computer-readable code on a computer-readable medium. A computer-readable medium is any data storage device that can store data, which can then be read by a computer system. Examples of computer-readable media include hard drives, network-attached storage (NAS), read-only memory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tape, and other optical and non-optical data storage devices. The computer-readable medium can also include computer-readable tangible media distributed over network-coupled computer systems, such that the computer-readable code is stored and executed in a distributed fashion.

[0118] Although the operations of the method are described in a particular order, it should be understood that other housekeeping operations may be performed between operations, or operations may be coordinated to occur at slightly different times or distributed across the system, allowing processing operations to occur at various intervals relative to processing, so long as the processing of overlapping operations is performed in the desired manner.

[0119] Although the foregoing disclosure has been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications can be practiced within the scope of the appended claims. Accordingly, the present embodiments are to be considered as illustrative and not restrictive, and embodiments of the present disclosure are not to be limited to the details provided herein, but may be modified within the scope of the appended claims and their equivalents.

Claims

1. 1. A cloud gaming method, comprising: generating a plurality of video frames when executing a video game on a cloud gaming server; Measure the maximum receive bandwidth of the client device, Encoding the plurality of video frames Filling a buffer with the encoded video frames; dynamically adjusting the buffer drain rate to be equal to or less than the maximum receive bandwidth; the encoded video frames are drained from the buffer at the buffer drain rate for transmission over a network to the client device. Cloud gaming methods.

2. The dynamic adjustment of the buffer discharge rate comprises: dynamically adjusting the buffer drain rate to the maximum receive bandwidth; The cloud gaming method of claim 1 .

3. dynamically adjusting the encoder bit rate to achieve said maximum receive bandwidth; performing the encoding of the plurality of video frames at the encoder bit rate; The cloud gaming method of claim 1 .

4. filling the buffer with the encoded video frames as quickly as possible; The cloud gaming method of claim 1 .

5. the maximum reception bandwidth is measured as a bit rate; The cloud gaming method of claim 1 .

6. modifying at least one of the plurality of video frames by combining the video frame with a user interface feature to generate a modified video frame; draining the modified video frames from the buffer at the buffer drain rate; The cloud gaming method of claim 1 .

7. The measurement of the maximum receive bandwidth comprises: Discharging a plurality of test packets from the buffer at a predetermined bit rate that is a maximum received bandwidth tested; receiving feedback including a total number of the plurality of test packets received for a range of video frames; determining packet loss based on the total number of the plurality of test packets received and the number of the plurality of test packets; determining the maximum reception bandwidth based on the packet loss and the tested maximum reception bandwidth; The cloud gaming method of claim 1 .

8. 1. A computer-readable medium storing a computer program for cloud gaming, the computer-readable medium comprising: having program instructions for generating a plurality of video frames when executing a video game on a cloud gaming server; having program instructions for measuring a maximum receive bandwidth of a client device; having program instructions for encoding the plurality of video frames; having program instructions for filling a buffer with the plurality of encoded video frames; and program instructions for dynamically adjusting a buffer drain rate to be less than or equal to the maximum receive bandwidth, wherein the encoded video frames are drained from the buffer at the buffer drain rate for transmission over a network to the client device. Computer-readable medium.

9. The program instructions for dynamically adjusting the buffer drain rate include: having program instructions for dynamically adjusting the buffer drain rate to the maximum receive bandwidth; The computer-readable medium of claim 8.

10. program instructions for dynamically adjusting an encoder bit rate to achieve said maximum receive bandwidth; having program instructions for performing the encoding of the plurality of video frames at the encoder bit rate; The computer-readable medium of claim 8.

11. having program instructions for filling the buffer with the encoded video frames as quickly as possible; The computer-readable medium of claim 8.

12. the maximum reception bandwidth is measured as a bit rate; The computer-readable medium of claim 8.

13. program instructions for modifying at least one of the plurality of video frames by combining the video frame with a user interface feature to generate a modified video frame; having program instructions for draining the modified video frames from the buffer at the buffer drain rate; The computer-readable medium of claim 8.

14. The program instructions for measuring the maximum receive bandwidth include: program instructions for draining a plurality of test packets from the buffer at a predetermined bit rate that is a maximum received bandwidth tested; program instructions for receiving feedback including a total number of the plurality of test packets received for a range of video frames; program instructions for determining packet loss based on the total number of the plurality of test packets received and the number of the plurality of test packets; and program instructions for determining the maximum receive bandwidth based on the packet loss and the tested maximum receive bandwidth. The computer-readable medium of claim 8.

15. 1. A computer system comprising: a processor; a memory coupled to the processor and having instructions stored therein, the instructions, when executed by the computer system, causing the computer system to perform a cloud gaming method, the cloud gaming method comprising: generating a plurality of video frames when executing a video game on a cloud gaming server; Measure the maximum receive bandwidth of the client device, Encoding the plurality of video frames Filling a buffer with the encoded video frames; dynamically adjusting a buffer drain rate to be less than or equal to the maximum receive bandwidth, and the encoded video frames are drained from the buffer at the buffer drain rate for transmission over a network to the client device. Computer system.

16. In the cloud gaming method, the dynamic adjustment of the buffer drain rate includes: dynamically adjusting the buffer drain rate to the maximum receive bandwidth; 16. The computer system of claim 15.

17. In the cloud gaming method, dynamically adjusting the encoder bit rate to achieve said maximum receive bandwidth; performing the encoding of the plurality of video frames at the encoder bit rate; 16. The computer system of claim 15.

18. In the cloud gaming method, filling the buffer with the encoded video frames as quickly as possible; 16. The computer system of claim 15.

19. In the cloud gaming method, the maximum reception bandwidth is measured as a bit rate; 16. The computer system of claim 15.

20. In the cloud gaming method, modifying at least one of the plurality of video frames by combining the video frame with a user interface feature to generate a modified video frame; draining the modified video frames from the buffer at the buffer drain rate; 16. The computer system of claim 15.

Images