Method and system for maintaining smooth frame rate during streaming of video content

Abstract

Methods and systems include receiving data frames generated for an online game from a server device and encoding the frames using an encoder. Each encoded data frame is transmitted over a network to a client device for rendering. A monitoring engine is used to evaluate the data contained in the encoded frames to determine whether an amount of data contained in the encoded frames exceeds a maximum data limit specified for a communication channel established between the server device and the client device. When the amount of data contained in the encoded frames exceeds the maximum data limit, a signal is sent to the encoder to dynamically reduce an amount of data included in one or more subsequent frames forwarded to the client device for rendering. Adjustments are performed to maintain a frame rate defined for the online game.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to streaming video content, and more particularly to methods and systems for maintaining a smooth frame rate during transmission of streaming video content. BACKGROUND

[0002] With the increasing popularity of mobile devices, there is an exponential growth in the demand for streaming content, such as movies, videos, entertainment content, live game content, and the like, for consumption by users. In order for users to fully engage, high quality video content must be provided with minimal latency. In order to provide high quality video content to client devices with minimal latency, a transmission pipeline with a large amount of bandwidth is required to deliver the streaming content. Further, the streaming content (e.g., video data) needs to be transmitted at a stable frame rate so that the content can be rendered at the client device in a smooth and continuous manner. Typically, the bandwidth for transmitting streaming content over a network, such as the Internet, is defined (i.e., set) by the Internet Service Provider.

[0003] Streaming content, such as videos, capture thousands of individual images in a video sequence. These images are captured in data frames. The frequency at which these frames are provided in a given amount of time represents the frame rate. The frame rate affects the way a user perceives motion in the video content. When the frame rate is inconsistent due to one or more frames being lost, certain details of the video can be lost, resulting in a choppy video. When the loss of frames is minimal and infrequent, the quality of the video perceived by the user can not be greatly affected. However, when frames are lost frequently, a noticeable stutter can be felt on the screen.

[0004] Further, as technology improves over time, the quality and frame rate of the generated videos also improves. Live streaming content providers (e.g., broadcasters) use higher frame rates to provide fast-paced action more completely. Examples of such fast-paced action include live sporting events, such as football, soccer, and the like, high-intensity online gaming, and the like. The increase in video resolution and frame rate places a higher burden on bandwidth requirements.

[0005] Prior to transmission, the content is encoded using an encoder. Encoders for encoding content prior to transmission over the Internet can be broadly classified into two main types - hardware and software. For example, a hardware encoder is a physical computing device / device that is configured to connect directly to a network, such as the Internet, encode video content, and send the encoded video content over the network. A software encoder is a program / programming logic that executes on a computer, establishes a network connection, encodes video content, and pushes the encoded video content over the network.

[0006] Regardless of the type of encoder being used, when encoding data, the encoder is instructed to adhere to a maximum data size per frame, where the maximum size is defined by the available bandwidth of the communication channel used to transmit the video data. However, the encoder does not strictly adhere to the specified maximum size when generating frames. As a result, the encoder can from time to time generate frames that are larger than the specified maximum size. For example, the encoder can generate an I-frame that is many times larger than the specified maximum size. The conventional way of handling such large size frames is to process, encode, and transmit the large frame and discard one or more subsequent frames to allow for the transmission of the large frame, thereby completely ignoring the maximum bitrate specified for the communication channel connection. As previously mentioned, infrequent frame dropping can not affect the overall experience of the user, but if frequent frame dropping occurs, then a noticeable stutter can be felt on the screen. Such noticeable stuttering can affect the overall experience of the user.

[0007] It is in this context that embodiments of the present disclosure arise. SUMMARY

[0008] Embodiments of the present disclosure relate to systems and methods for processing data of an online game or interactive application that generates a stable stream of content. For example, a game engine available on a remote server computing device (or simply "server" herein) executes a user-selected online game and generates a stream of content. User input provided in response to the stream of content is used to affect a game state of the online game. The stream of content is encoded using an encoder and transmitted to a client device for rendering. The stream of content includes a current state of the online game. Data contained in each encoded frame is evaluated to determine whether the data contained in the encoded frame exceeds a content maximum threshold limit defined for a communication channel established between the remote server and the client device. If the data in the encoded frame exceeds the predefined maximum threshold limit, a counter, such as a "sent debt" counter, is incremented. The sent debt counter identifies an amount of data contained in the frame that exceeded the threshold limit set for the communication channel and a number of frames that need to be scaled back in order to pay off the sent debt. When a subsequent content frame is received for encoding by the encoder, the sent debt counter is queried, where the subsequent data frame is generated after a frame that exceeded the threshold limit. When the sent debt counter is non-zero, the encoder is instructed to scale down an amount of data included in the subsequent frame to be below a predefined reduced maximum data limit. The reduced maximum threshold limit of the subsequent frame can be driven by an amount that the first frame exceeded the threshold limit and / or a bandwidth of the communication channel and / or a demand of the online game. The amount of data included in the subsequent frame continues to be scaled down as long as the sent debt is non-zero. Once the sent debt has been paid off (i.e., the sent debt is zero), the encoder is instructed to continue encoding subsequent frames received after the data frame that resulted in the sent debt at a normal rate defined by the predefined maximum threshold limit.

[0009] When a subsequent frame is received, the encoder can evaluate the amount of data included in the subsequent frame and dynamically reduce the amount of data in the subsequent frame to a reduced maximum data limit specified for the online game based on a signal received at the encoder. The reduced maximum data limit can be predefined based on the type of online game selected to play. When the subsequent frame is evaluated based on the signal received at the encoder, the encoder can determine that the amount of data included in the subsequent frame is less than the reduced maximum data limit. In this case, the encoder will simply proceed to encode the subsequent frame without changing the amount of data included in the subsequent frame because the amount of data in the subsequent frame is well within the reduced maximum data limit.

[0010] By providing instructions to the encoder to include an appropriate amount of data in each frame sent to the client device, a smooth and consistent frame rate can be maintained. Maintaining the frame rate results in the relevant content of the online game being rendered at the client device without causing any stutters in the rendering process. The encoder can reduce the amount of data in the subsequent frame by identifying and preserving certain data included in the subsequent frame that is critical to rendering the image and discarding certain other data included in the subsequent frame that is not critical. Alternatively, the encoder can use compression techniques to reduce the data.

[0011] In some cases, a frame data processor can be used to process the data of each frame generated by the online game (i.e., the frame data source) before forwarding the processed data frames to the encoder for encoding. The processing of the data can be to determine the complexity of the data included in the frame so that the encoder can make informed decisions as to which portions of the processed data to preserve and which portions to discard.

[0012] In one implementation, a method is disclosed. The method includes receiving data frames generated for an online game from a server device. The data frames represent a current state of the online game. The data frames are encoded using an encoder. The encoded data frames are transmitted over a network to a client device for rendering. Data included in the encoded frames is evaluated to determine an amount of data included in the encoded frames transmitted to the client device. When the amount of data in the encoded frames exceeds a maximum data limit specified for a communication channel established between the server device and the client device, a signal is sent to the encoder to dynamically reduce an amount of data included in one or more subsequent frames to be forwarded to the client device for rendering. Adjustments are performed to maintain a frame rate defined for the online game.

[0013] In another implementation, a system for processing data of an online game is disclosed. The system includes a server configured to execute an instance of an online game and generate data frames for transmission to a client device. The system also includes an encoder configured to receive data frames of an online game and encode the data frames according to a maximum data limit specified for a communication channel established between the server and the client device. A monitoring system is included in the server and is configured to monitor an amount of data included in each frame and generate a signal to the encoder to dynamically reduce an amount of data included in one or more subsequent frames when an amount of data included in a particular frame preceding the one or more subsequent frames exceeds the maximum data limit.

[0014] In another alternative implementation, a method is disclosed. The method includes receiving data frames generated for an online game from a server device. The data frames represent a current state of the online game. The method also includes verifying whether a previously encoded data frame exceeds a maximum data limit specified for a communication channel established between the server device and a client device. When it is determined that the previously encoded frame exceeds the maximum data limit, an amount of data included in the frame is dynamically reduced to a reduced maximum data limit prior to encoding the data frame to generate an encoded frame. The encoded data frame is forwarded to the client device over the communication channel.

[0015] Other aspects of the disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0016] The disclosure will be best understood by reference to the following description in conjunction with the accompanying drawings.

[0017] Figure 1 A simplified physical view of a cloud gaming system for maintaining a smooth frame rate for a streaming interactive application, such as an online streaming game, in accordance with one implementation of the present disclosure is shown.

[0018] Figure 2A A simplified block diagram of different components of a server device for providing a stable frame rate of online game content generated during gameplay of an online game to a client device in accordance with one implementation of the present disclosure is shown.

[0019] Figure 2B A simplified block diagram of different components of a server device for providing a stable frame rate of online game content generated during gameplay of an online game to a client device in accordance with an alternative implementation of the present disclosure is shown.

[0020] Figure 3AA simplified block diagram illustrating various components of a monitoring engine for identifying the amount of data an encoder needs to include in each data frame transmitted to a client device, according to one implementation of the present disclosure.

[0021] Figure 3B A simplified block diagram illustrating different components of a frame data processor for conditioning data provided to an encoder for encoding according to one implementation of the present disclosure is shown.

[0022] Figure 4 A simple process flow diagram is shown for instructing an encoder on the amount of data included in each frame generated by a frame source, according to one implementation of the present disclosure.

[0023] Figure 5 A simple example of using a maximum bit rate in one implementation to adjust content in different frames generated by a frame source (eg, an online gaming application or an online interactive application) to provide continuous frames to a client is shown.

[0024] Figure 6 An operational flow of a method of processing a data frame for onward transmission to a client device according to one implementation is shown.

[0025] Figure 7 A simplified block diagram of an example interactive application system, such as a gaming system, is shown according to various implementations of the present invention.

[0026] Figure 8 An exemplary information service provider architecture according to one implementation of the present invention is shown, the architecture being used to provide information content and services to users who are geographically dispersed and connected via a network. DETAILED DESCRIPTION

[0027] Although the following detailed description contains many specific details for illustrative purposes, those skilled in the art will appreciate that many variations and modifications of the following details are within the scope of the present disclosure. Therefore, the various aspects of the present disclosure described below are set forth without any loss of generality of the appended claims and without imposing any limitations on the claims.

[0028] In general, various implementations of the present disclosure describe systems and methods for evaluating the amount of data contained in each data frame generated by an online streaming application (e.g., an online game application) and dynamically reducing the amount of data in one or more subsequent frames when the amount of data contained in the frame exceeds a maximum data limit specified for the online game. The maximum data limit can be based on a maximum bitrate that a communication connection established between a server device generating the data frames and a client device requesting the data by selecting the streaming application is able to handle. The amount of data contained in each subsequent frame can be dynamically reduced by compressing the data using a compression algorithm or by selectively discarding portions of the data included in the frame that are not critical to rendering the image or by other means generally available or by specialized algorithms.

[0029] Conventional ways of handling data frames include detecting large data frames that exceed a specified maximum data limit and, in response, discarding one or more subsequent data frames so that the maximum bitrate of the communication connection can be adhered to. This results in inconsistent frame rates and can result in choppy video when frame dropping becomes too frequent.

[0030] On the other hand, various implementations discussed herein include a system configured to detect large frames and, instead of discarding one or more subsequent frames, the system is configured to instruct an encoder to adjust the bitrate of one or more subsequent frames downward in order to be able to pay off a sent debt caused by sending too much data. The system can detect the excess bitrate transmitted in a frame and treat the excess data as a “overdrawn bitrate” account. The system can pay off the excess data transmitted in a frame by reducing the data from subsequent frames that follow the frame that was “overdrawn.” A monitoring engine within the system is configured to track the amount of excess data transmitted in a frame that caused the bitrate to be overdrawn, determine the number of subsequent frames to use to pay off the excess data, and provide instructions to the encoder to dynamically reduce the amount of data in the determined number of subsequent frames. The number of subsequent frames that need to have data dynamically reduced can be determined based on the amount of excess data included in the frame and / or the needs of the online game or the type of online game, etc.

[0031] A “sent debt” counter is used to track the amount of excess data included in a frame and the number of subsequent frames to use to pay off. The sent debt counter is updated after each frame is forwarded to a client device. For each frame received at the encoder, the monitoring engine is used to query the sent debt counter to determine whether the sent debt counter is zero or non-zero. If the sent debt counter is non-zero, the monitoring engine instructs the encoder to adjust the amount of data in the frame currently received at the encoder for encoding and then transmit the adjusted encoded frame to the client device. On the other hand, if the sent debt counter is zero, the monitoring engine instructs the encoder to continue encoding the current frame using the normal bitrate defined for the online game (i.e., the maximum data limit).

[0032] The content generated by online games is video content. The amount of data contained in each frame defines the bit rate, which is measured in megabits per second (Mbps). The bit rate depends in part on the resolution of the video content. In order to provide a realistic gaming experience, the game logic can be configured to provide high resolution video frames. In contrast to low resolution video frames that include a lower amount of data, high resolution video frames include a higher amount of data (i.e., a higher bit rate). Generally, highly compressed video data has less data (i.e., a lower bit rate) than lightly compressed or uncompressed video data. As a result, compressed video data has a lower video quality (i.e., a lower video resolution). In general, for high quality video content, a bit rate of 40 Mbps is recommended, and this bit rate is for uncompressed or very lightly compressed video. The bit rate is defined in terms of the bandwidth requirements of the communication channel used to transmit the video content.

[0033] The video data is typically transmitted at a frame rate defined for the online game. The frame rate describes the speed at which the video data is played at the receiving device, and this can depend on the type of online game being played. When more frames are played per second, the rendered video appears smoother. Increasing the frame rate will result in an increase in the bit rate, as more data needs to be provided in the frames. Therefore, the frame rate can be defined for the online game based on the smoothness expected / defined for the online game. Since the frame rate also affects the bit rate, the frame rate can be defined in terms of the available bandwidth of the communication channel used to transmit the video content to the client device. Typically, the frame rate is standardized in the industry that generates the content, and can be defined as approximately 30 frames per second, or 60 fps or 120 fps.

[0034] The monitoring engine ensures a smooth and consistent frame rate of the content of the interactive streaming application provided to the client device and avoids frame drops by dynamically adjusting the bit rate of the data included in each frame. For an online game that is selected to be played, the video content included in the initial frame generated for the online game can have a higher bit rate because the content can include the entire scene of the game and this bit rate can be higher than the maximum allowable bit rate for the online game. The monitoring engine detects the bit rate overflow caused by the large initial frame and timely instructs the encoder to reduce the bit rate in one or more subsequent frames following the large initial frame to allow the system to pay off the data debt caused by the bit rate overflow of the initial frame. The monitoring engine continues to monitor the bit rate of each frame generated for the online game, queries the sent debt counter to determine whether the subsequent frames received at the encoder need to be adjusted, and sends a signal to the encoder instructing the encoder to dynamically adjust the bit rate of the subsequent frames during encoding or just encode the subsequent frames without any bit rate adjustment. The encoded subsequent frames are forwarded to the client device for rendering. The bit rate down adjustment in the subsequent frames can sometimes have an adverse effect on the resolution quality of the subsequent frames, which is a small price to pay to maintain the frame rate as opposed to the intermittent frame drops observed with the use of conventional encoders. The intermittent frame drops can affect the overall quality of the video content provided to the user, especially when the number of frames dropped continues for a long period of time or more frequently, as the video becomes very choppy.

[0035] A user can set up their user account at a cloud system, such as a cloud gaming system, to enable the user to access various online games available therein. One or more cloud game servers of the cloud gaming system can execute an instance of an online game locally on the cloud and generate streaming content for transmission to a client device of the user for rendering. When the user plays the online game and provides game inputs, the game inputs are transmitted to the cloud game server to affect the outcome of the online game. In response to the game inputs, the cloud game server generates data frames for transmission to the client device for rendering. The generated data frames are encoded using an encoder available in the cloud system to generate encoded frame data. The encoded data frames are forwarded to a network interface, where the frame data is packetized according to a transmission protocol defined for a communication channel established between the cloud game server and the client device over a network, such as the Internet, and transmitted to the client device.

[0036] The encoder provides the encoded frame data to a monitoring engine available in the cloud system. The monitoring engine can be integrated within the encoder, or available within the cloud game server and communicatively connected to the encoder to allow interaction with the encoder. The monitoring engine checks the amount of data included in the encoded frame. The monitoring engine verifies whether the amount of data contained in the encoded frame exceeds a predefined maximum data limit defined for each frame of the online game. If the encoded frame exceeds the predefined maximum data limit, the monitoring engine determines the extent to which the amount of data included in the encoded frame exceeds the maximum data limit and sends a signal to the encoder to dynamically reduce the bit rate of one or more subsequent data frames received after the large frame that exceeded the maximum data limit. In response to the communication from the monitoring engine, when the encoder receives a subsequent frame from the cloud game engine, the encoder injects a new and reduced bit rate during encoding of the subsequent frame. The reduced bit rate can be predefined for the online game. The encoded subsequent frame with the reduced bit rate is forwarded to the client device for rendering. The monitoring engine continues to verify each encoded data frame received from the encoder to determine the amount of data included therein and instruct the encoder to dynamically adjust the subsequent frames in order to maintain a frame rate while adhering to the bit rate limits of the communication channel.

[0037] The monitoring engine can query a "sent debt" counter maintained for the online game. The sent debt counter maintains a running count of data overflow as well as the number of subsequent frames that need to have data dynamically reduced. Typically, the initial frames from the online game can contain a large amount of data because the initial frames include images of the initial game scene of the online game and the subsequent frames can include updates to the images contained in the initial frames. However, at times, depending on user input and depending on the game type and the current game state of the game, the subsequent frames can include a large amount of data. This can be the case when the user moves to a different scene and the online game is providing data for the new scene. By querying the sent debt counter and closely monitoring the bit rate of each frame received at the encoder, the monitoring engine is able to determine which of the frames include an excessive amount of data and how many subsequent frames need to have data dynamically reduced and instruct the encoder accordingly. The encoder adjusts the data frames in accordance with the instructions provided by the monitoring engine. The sent debt counter is updated by the monitoring engine after the data frames are adjusted. The updated sent debt counter is used to instruct the encoder.

[0038] When the sent debt counter is non-zero, the monitoring engine sends a signal to the encoder to adjust the data in the subsequent frame. The encoder, in response to the signal, assesses the amount of data contained in the subsequent frame received at the encoder. If the amount of data contained in the subsequent frame is less than the reduced maximum data limit defined for the online game, the encoder encodes the data only without any adjustment to the data contained in the subsequent frame. For example, if the reduced maximum data limit for the subsequent frame is defined to be about 10,000 bits per frame, and the subsequent frame contains only about 8,000 bits, no further adjustment to the contents of the subsequent frame is necessary because it is already below the reduced maximum data limit. In this example, the encoder will simply encode the data contained in the subsequent frame and transmit it to the client device. The encoded subsequent frame will be used to pay off some of the data that overflowed from the previous large frame. After the subsequent frame is transmitted, the monitoring engine updates the sent debt counter.

[0039] When the amount of data contained in the subsequent frame is above the reduced maximum data limit but does not exceed the predefined maximum data limit and the "sent debt" counter is non-zero, the encoder dynamically injects a new reduced bit rate (i.e., the reduced maximum data limit) into the subsequent frame and forwards the encoded subsequent frame to the client device. Injecting the new reduced bit rate can include compressing the data in the subsequent frame using compression techniques. The monitoring engine manages the frame rate of the online game with the help of the sent debt counter, making it consistent and smooth so that the content of the online game can be viewed without latency or noticeable stuttering.

[0040] With the general understanding of embodiments of the present invention, example details of various implementations will now be described with reference to the various drawings.

[0041] Figure 1An overview of a cloud gaming system 300 according to one implementation is provided for providing a stable stream of data frames related to game play of an online game to client devices. A plurality of client devices 100 (100-1, 100-2, 100-3,... 100-n) are used by different players (i.e., users playing an online game) to access a cloud gaming system (CGS) 300, social media applications, and other interactive applications over a network 200, such as the Internet. The client devices 100 can access the CGS 300 from different geographical locations. The client devices 100 can be any type of client computing device having a processor, memory, capable of accessing a network (e.g., a local area network (LAN), a wide area network (WAN), a wireless LAN, a MAN (metropolitan area network), a personal / private area network (PAN), a virtual private network (VPN), etc.) using wired, wireless, 4G, 5G, or other communication capabilities, and are portable or non-portable. For example, the client devices can be a smartphone, a mobile device, a tablet computer, a desktop computer, a personal computer, a wearable device, or a hybrid device or other digital device including a display or touchscreen having a portable form factor.

[0042] A client device having 5G communication capabilities can include a mobile device or any other computing device capable of connecting to a 5G network. In one implementation, a 5G network is a digital cellular network in which service areas are divided into multiple “cells” (i.e., small geographic areas). Analog data generated at a mobile device is digitized and transmitted as radio waves using frequency channels to local antennas within a cell, which can be reused in geographically separate cells. The local antennas are connected to the Internet and telephone networks through high-bandwidth fiber-optic cables or other similar wireless communication. 5G networks are capable of transmitting data at higher data rates because they use higher frequency radio waves for communication, providing lower network latency.

[0043] The client devices can run an operating system and include a network interface that provides access through the network 200 to various game applications or interactive applications (applications) available on the cloud game servers in the CGS 300, or can be thin clients with a network interface to communicate with cloud game servers 302 that provide the computing functionality. A player can access the CGS 300 using a user account and select an online game available at the CGS 300 for game play, where the game play is controlled by the player using control / input options provided in the client device or using a controller communicatively connected to the client device. The player’s user account can be verified against user data stored in the user account data store 308 to ensure that the player is entitled to access and play a game title of the online game stored in the game title data store 310 before providing access to the online game.

[0044] In some implementations, various applications (e.g., online games) can be executed remotely on one or more cloud game servers 302 of the cloud game system 300, and game play related data from the executing applications is generated as frames that are streamed over the network 200 to the client devices 100. In other implementations, online games can be executed locally at the client devices, and metadata from the executing online games can be transmitted over the network 200 to the cloud game servers 302 to influence the game state. Frames of game play data are forwarded to an encoder for encoding, and the encoded frames are forwarded to the client devices for rendering.

[0045] It should be noted that a player and a user both refer to a person that has a user account and is entitled to access and interact with applications available on the cloud game system 300, including online games. A player or user is a person that has previously played or is currently selected to play an online game. A “user” can also be a person (e.g., a spectator) that has selected an online game to watch a player play the game before selecting the online game to play the game.

[0046] The cloud gaming system (CGS) 300 can include a network of backend servers distributed in different geographical locations and configured to execute instances of one or more online gaming applications and / or other interactive applications available at the backend servers. The backend servers 302 can be cloud gaming servers or cloud application servers configured to execute instances of one or more online gaming / interactive applications. For simplicity, reference will be made to cloud gaming servers (or simply “game servers”) executing instances of online gaming applications, but the implementations disclosed herein can be extended to any other interactive application that can be used to provide frames of interactive data to users. The game servers can be any type of server computing device available in a cloud gaming system, including but not limited to standalone servers, servers as part of a server farm or data center, etc. Further, the game servers can manage one or more virtual machines that support game processors that execute instances of online games on a host.

[0047] The online game executed by the cloud game servers 302 can be a single-player game or a multiplayer game. In some implementations, the online game can be a massively multiplayer online (MMO) game that allows multiple players and spectators from different geographical locations to access and play / watch the online game. The cloud game servers 302 can include a multiplayer distributed game engine communicatively connected to the game logic of the online game. Generally, a game engine is a software layer that serves as the foundation for a game, such as an MMO game, and provides a framework for developing the online game. The game engine abstracts the details of performing common related tasks (i.e., game engine tasks) required for each game, while the online game developer provides game logic that provides the details of how the online game will be played. The game engine framework includes a number of reusable components for handling several functional parts of the online game (i.e., core features) that make the online game come alive. Basic core features handled by the game engine can include physics (e.g., collision detection, collision response, trajectories, object motion based on gravity, friction, etc.), graphics, audio, artificial intelligence, scripting, animation, networking, streaming, optimization, memory management, threading, localization support, etc. The reusable components include processing engines for handling the core features identified for the online game.

[0048] During game play of the online game, the game engine manages the game logic of the online game, collects and transmits player inputs received from different client devices to the game logic, manages the allocation and synchronization of the functional parts of the game engine to process the game data generated by the game logic in an optimal manner, and generates frames of game data that are transmitted back to the client devices for rendering. A number of game engines can currently be used to provide different core functionalities, and a suitable game engine can be selected based on the functionalities specified for executing the online game.

[0049] The game logic analyzes the inputs provided by the players at the respective client devices during a game play session, updates the game state of the online game based on the inputs, manages the saved data of each player playing the online game, and generates game data that is processed by the distributed game engine before being streamed to the user's client device during the game play session. The game state of the online game is used to identify game state metrics of the online game and user metrics of the players, which can be stored separately in game play metrics 314. For example, user metrics related to the complexity of each player's game play can be stored in user metrics data store 314a, and game state metrics of the online game can be stored in game state metrics data store 314b. The game state identifies the overall state of the online game at a particular point and is influenced by the game play complexity of one or more players. The user metrics of each player include any game customizations provided by the player for the online game. The game state provided by the game logic using the distributed game engine enables the superimposition / insertion of objects and characters into the game environment of the players participating in the game play session. In some implementations, the game logic for each instance of the game can be executed on one server or multiple servers based on how the game logic is configured.

[0050] The game inputs provided by each player of the online game during game play, the game state of the online game, the location of each player in the online game, and the details of the activities performed by each player can be stored by the game engine as telemetry data within the game state metrics 314b or separately by the game engine 304 within a telemetry data store (not shown). The details included in the telemetry data identify the characteristics of each activity attempted, completed by the player, and the player attributes of the player attempting, completing each activity in the online game. When the online game needs to be recovered or restarted from a particular point, the telemetry data associated with each player can be used to recreate a portion of the online game. Further, the telemetry data can be used to generate data frames for forwarding to the client devices.

[0051] The data frames generated by the game engine are forwarded to the encoder 305 for encoding before being transmitted to the client devices for rendering. The encoder 305 can be a hardware encoder or a software encoder and is configured to encode the data contained within each frame according to a defined encoding algorithm. The encoded data frames are also forwarded to the monitoring engine 306. The monitoring engine evaluates the amount of data contained within each encoded frame and verifies the amount of data against a predefined maximum data limit specified for the online game based on the communication connection established between the cloud gaming server 302 and the client device 100. The maximum data limit is driven by the bandwidth capability of the communication connection, which can be set by the network service provider.

[0052] When the monitoring engine 306 detects that the amount of data transmitted in a particular frame exceeds the predefined maximum data limit, the monitoring engine 306 sends a signal to the encoder 304 to reduce the amount of data included in subsequent frames. The monitoring engine 306 can maintain a “transmitted debt” counter that acts as an overdraft account and records a count of the excess amount of data transmitted in a particular frame that needs to be repaid, and the number of subsequent frames from which the excess “debt” is to be repaid. In some implementations, the excess data can be repaid from one or more subsequent frames. The monitoring engine can determine the number of subsequent frames from which to repay the excess data transmitted in a particular frame based on the extent to which the amount of data in the particular frame exceeds the maximum data limit (i.e., bit rate) set for the communication connection, the type of data being transmitted, the type of online game for which the data is being generated, the demands of the online game, the bandwidth capabilities of the communication connection, or a combination of two or more thereof. For example, if the online game is a high-intensity game, the repayment must be completed more quickly, as opposed to an online game in which multiple users are watching. To maintain the bit rate of the communication channel, the repayment must be completed more quickly in high-intensity games.

[0053] The monitoring engine queries the transmitted debt counter to determine whether the data included in the subsequent frame needs to be dynamically adjusted and sends an appropriate signal to the encoder to that effect. The signal generated by the monitoring engine 306 can include an indicator to indicate whether the subsequent frame needs to be dynamically adjusted and, if so, to indicate the reduced maximum data limit specified for the adjusted subsequent frame. Like the repayment speed, the reduced maximum data limit can be set based on the game type or game demands. In response to the signal provided by the monitoring engine, the encoder first evaluates the amount of data contained in the subsequent frame. If the amount of data contained in the subsequent frame exceeds the reduced maximum data limit, the encoder dynamically adjusts the data contained within the subsequent frame to be equal to or lower than the reduced maximum data limit. On the other hand, if the amount of data contained in the subsequent frame does not exceed the reduced maximum data limit, the encoder only encodes the data in the subsequent frame and transmits the encoded data over the network to the client device. In this case, no adjustment is made to the data included in the subsequent frame because the amount of data is already less than the reduced maximum limit set for the subsequent frame. For example, if the reduced maximum data limit for the subsequent frame is set to 10,000 bytes and the data included in the subsequent frame is 9,000 bytes, the encoder does not adjust the data in the subsequent frame. Instead, it simply encodes the data and forwards the encoded data to the network interface for packaging for continued transmission over the network to the client device. The encoding of the subsequent frame can include the portion of the data in the large frame that exceeds the maximum data limit to repay the transmitted debt caused by the system.

[0054] After encoding and transmitting the subsequent frame, the encoder interacts with the monitoring engine 306 to update the sent debt counter to account for the subsequent frame sent to the client device. As shown in the example above, the sent debt counter is updated even when no adjustment to the amount of data in the subsequent frame is needed. The querying of the sent debt counter by the monitoring engine and the signaling to the encoder continues as long as data frames are being generated for the online game.

[0055] The current implementation allows the system to pay back the excess data sent in a particular frame by reducing the data in subsequent frames. Each frame generated by the online game is required to adhere to a maximum data limit for the data included in the frame. However, the data included in each frame generated by the online game does not always adhere to the maximum data limit. The monitoring engine discussed in various implementations handles this issue by dynamically paying back the excess data from the frames that follow the large frame, such that the frames as a whole follow the specified maximum bitrate. The number of frames for which the bitrate is adjusted can be 1 or 10 or more, depending on the extent to which the data in the large frame detected exceeds the maximum bitrate. Reducing the data in one or more subsequent frames (i.e., lowering the bitrate included in the subsequent frames) allows the system to recover from the burden of the bitrate overflow caused by the large frame. An excessive burden of data can result in latency or dropped frames. The dynamic adjustment of data in subsequent frames ensures that all frames are delivered to the client device and maintains a consistent frame rate defined for the online game that satisfies the maximum data rate. The reduction of data in subsequent frames can result in a decrease in the quality of the subsequent frames delivered to the client device, but this decrease in quality is temporary as the system is able to quickly pay back the sent debt and continue to send frames at the maximum data rate defined for the online game after the sent debt is paid back.

[0056] Figure 2A An example game cloud system in one implementation is shown that uses a monitoring system to maintain a frame rate for an online streaming game. A game engine 304 responds to a user selecting an online game to play by executing an instance of the online game on a server, such as a cloud game server 302. User input provided at the user's client device 100 is communicated to the game engine to affect the game state of the online game. The online game generates data frames 135 representing the current game state of the game. The data frames 135 are forwarded to an encoder 305 available to the cloud game server 302. The encoder 305 can be available on the cloud game server 302 executing the instance of the online game, or available on a different server within the cloud game system 300, or can be a separate computing device that includes the encoder logic. In the implementation shown, the encoder 305 is shown as a hardware encoder that can be directly connected to the network 200. The implementation is not limited to using a hardware encoder, but can readily be extended to also use a software encoder. Figure 2A The implementation shown, the encoder 305 is shown as a hardware encoder that can be directly connected to the network 200. The implementation is not limited to using a hardware encoder, but can readily be extended to also use a software encoder.

[0057] The encoder 305 is designated to not exceed a suggested frame size, where the suggested frame size can follow the bandwidth of the communication channel established between the server 302 and the client device 100. The encoder can not always follow the suggested frame size and can include data that exceeds the suggested frame size. To not overload the communication channel with data, the cloud gaming system uses the monitoring engine 306 to track the amount of data included in each encoded frame and control the bit rate of subsequent frames after a frame that exceeds the frame size.

[0058] The encoder 305 encodes the data according to an encoding algorithm and forwards the encoded frame to a network interface defined within one of the servers of the cloud gaming system, such as the cloud gaming server 302 or another server. The network interface packages the encoded data frame according to a transmission protocol defined for the communication channel and forwards the data package through the network to the client device 100 for rendering. The encoder 305 also provides the encoded data frame forwarded to the client device 100 to the monitoring engine 306. The monitoring engine 306 evaluates the amount of data in the encoded frame. When the encoded frame includes data that exceeds the maximum data limit for the frame, the monitoring engine 306 detects the excess data and instructs the encoder to reduce the amount of data included in one or more subsequent frames. The monitoring engine maintains and dynamically updates a sent debt counter to indicate excess data in the encoded frames sent to the client device 100.

[0059] When the encoder receives a subsequent frame from the game engine (i.e., frame source), the encoder checks whether the monitoring engine 306 has signaled to adjust the data in the subsequent frame. When the encoder detects a signal from the monitoring engine 306, the encoder identifies a predefined reduced maximum data limit and then verifies the amount of data contained in the subsequent frame to determine whether it is within the predefined reduced maximum data limit. The signal from the monitoring engine 306 can include an instruction to reduce the amount of data contained in the subsequent frame and a reduced maximum data limit that the encoder can use to reduce the data within the subsequent frame. Based on the signal, the encoder checks whether the subsequent frame includes data that is within the reduced maximum data limit. If the amount of data within the subsequent frame is within the reduced maximum data limit, the encoder 305 will encode the subsequent frame without making any adjustments to the data. On the other hand, if the data in the subsequent frame exceeds the reduced maximum data limit, the encoder will dynamically adjust the data in the subsequent frame and then encode the frame. The encoded subsequent frame is forwarded to the network interface for packetization, where the data from the subsequent frame can be combined with some data from a previous large frame that was sent by the encoder to the network interface for packetization and transmission to the client device, thereby paying off the excess data of the previous large frame. The encoder then forwards the data packet that includes the data from the encoded subsequent frame and the excess data from the previous large frame to the client device 100 for rendering. As the name implies, the reduced maximum data limit is less than the regular maximum data limit specified for the frame. For example, the maximum data limit for a frame can be set to 20,000 bytes, while the reduced maximum data limit can be set to 10,000 bytes. The reduced maximum data limit can be set based on the type of game or the needs of the game, for example.

[0060] Figure 2B An alternative implementation of a game cloud system is shown that uses a monitoring system to maintain the frame rate of an online streaming game. In addition to the frame data processor 309 between the frame source and the encoder, the components of the game cloud system identified in this implementation are similar to those described with reference to Figure 2A The components shown and described. As a result, Figure 2A and Figure 2BCommon components are denoted using the same reference numerals and are configured to perform similar functions. In this implementation, rather than forwarding the generated frames directly to the encoder, the game engine now forwards the frames to a frame data processor 309 for processing. As part of the processing, the frame data processor 309 is configured to analyze the data contained in each frame to determine the complexity of the data contained in the frame. The information provided by the frame data processor 309 is used to identify data that is critical to rendering the images contained in the frame for the user at the client device. The critical areas can be areas in which the game scene or game objects / avatars are changed as a result of incoming user input to the online game, or areas in which actions / events occur in response to incoming user input to the online game. In some implementations, critical areas that include changes can be identified as being specific to the user for which the data is being generated. For example, the game engine can generate an I-frame to include data for constructing the entire scene of the game when rendered at the client device, and the game engine can generate subsequent frames as P-frames. The P-frames include changes to the game scene contained in the I-frame, where the changes can include user-specific object movement or action performance or event occurrence, and are identified from processing the user’s input to the online game. However, not all changes contained in the P-frames can be critical to the user. For example, the occurrence of an event can have been recorded as a change and included in the P-frames generated by the game engine. However, the event can have occurred behind the avatar controlled by the user. As a result, the event can not be important to the user because the user has moved past the point or location where the event occurred. The frame data processor 309 can contextually analyze the data contained in each frame generated by the online game to determine which data is critical or important to render to the user. Note that data that is critical to a first user can or can not be critical to a second user. As a result, the content rendered for the first user can be different from the content rendered for the second user, even when the first user and the second user are interacting within the same scene.

[0061] The frame data processor 309 forwards the results of the analysis to the encoder 305 along with the frame data. When instructed to dynamically reduce the amount of data contained within the frames, the encoder 305 uses the information provided by the frame data processor 309 to determine which data to retain and which data to discard in each frame. The encoder 305 dynamically reduces the amount of data in the frames by retaining the data identified as critical to the user and encoding the retained data during encoding. The amount of data that needs to be reduced is according to the maximum data reduction limit specified by the monitoring engine 306. The resulting encoded frames are forwarded to the client device. Note that the frames in which the amount of data is dynamically reduced are the frames after the large frame (i.e., the subsequent frames).

[0062] In an alternative implementation, when encoding a frame to be dynamically reduced, the encoder can reduce the amount of data included in the non-critical section and leave the data of the critical section as is. In yet another alternative implementation, the encoder can compress the data contained in the frame using any of the lossy or lossless techniques such that the subsequent frames adhere to the reduced maximum data limit specified for the online game.

[0063] Figure 3A Various components of a monitoring engine for maintaining a steady stream of frames for an online game in one implementation are shown. The monitoring engine 306 can be part of a game engine that executes the game logic or can be a separate engine that receives the data frames generated by the game engine. The monitoring engine 306 includes a bit rate monitor 311, a bit rate adjuster 312, and a sent data debt tracker (or simply “debt tracker” herein) 313 to track and adjust the bit rate in each frame. The above components are provided as examples and should not be considered exhaustive or limiting. Fewer or more components can be used.

[0064] A frame source, such as a game engine, generates data frames during the game play of an online game. These data frames are transmitted to a client device for rendering. User inputs provided at the client device via controls or an interactive user interface of the client device or from an external controller communicatively coupled to the client device are transmitted to the game engine executing an instance of the online game at a server device. The user inputs are used to affect the game state of the online game. The updated game state is provided in subsequent frames to the client device for rendering. In the case of a streaming online game, the frames are generated in a continuous manner. The amount of data included in each frame can vary depending on the game state of the online game. An initial frame generated for the online game can be large as it can include data related to the scene of the online game. Subsequent frames generated for the online game can be smaller as they can include only the changes detected in the initial scene. During the game play, the game engine can generate large data frames from time to time. This can be the case when the game engine detects that a user has moved from one scene to another scene or from one level to the next level. The generated data frames are forwarded to an encoder 305 for encoding before transmitting the encoded frames to the client device.

[0065] The encoder 305 is configured to receive the data frames and encode the content according to the encoding details defined for the online game. For example, the encoder can need to adhere to a particular frame rate / frequency (e.g., 30 frames per second (fps), 60 fps, 120 fps, etc.) and a maximum bit rate of the communication channel used to transmit the data. The encoded frames are forwarded by the encoder 305 to the bit rate monitor 311.

[0066] The bitrate monitor 311 is configured to assess the amount of data contained within an encoded frame using a predefined maximum data limit 311a specified for the online game. Based on the communication channel used to transmit the streaming data to the client device, each online game can have its own predefined maximum data limit 311a or can follow a common predefined maximum data limit 311a. When the amount of data in an encoded frame exceeds the maximum data limit 311a specified for the online game, the bitrate monitor 311 calculates the extent to which the data in the encoded frame exceeds the maximum data limit 311a. The bitrate monitor 311 then determines a reduced maximum data limit 311b that will be used to adjust the data in subsequent frames and uses a frame number calculation engine 311c to calculate the number of subsequent frames for which content needs to be dynamically adjusted in order to pay off the data overflow caused by the encoded frame. The frame number calculation engine 311c can calculate the number of subsequent frames on which to spread the debt queue to pay off the data overflow as appropriate. For example, in the case of a live streaming game, the number of frames over which the debt queue (i.e., to pay off the overflow) can be spread can be large, such as between about 15 to about 50 frames, as the latency caused by spreading the debt queue can not adversely affect the overall viewing experience of the user. Whereas, in the case of an interactive streaming game, the spreading of the debt queue can have to be more aggressive, such that the debt queue can be paid off more quickly, to maintain a light / minimal latency. Thus, for interactive streaming games, the debt queue can be spread over a smaller number of frames, such as between about 1 to 10 frames or about 1 to 5 frames or about 1 to 3 frames. The frame number calculation engine 311c thus calculates the extent of spreading of the debt queue based on the type of data being generated, the requirements of the game, the extent of the bitrate overflow caused by the encoded frame, etc.

[0067] The bitrate overflow and the number of subsequent frames over which to spread the debt queue are communicated by the bitrate monitor to the transmitted data debt tracker 313 along with instructions to update one or more counters maintained by the debt tracker 313. The debt tracker 313, in response to the instructions from the bitrate monitor 311, can update a bitrate debt counter 313a with the bitrate overflow and a frame debt counter 313b with the number of subsequent frames.

[0068] The bit rate monitor additionally instructs the bit rate adjuster 312 to send a signal to the encoder to adjust the bit rate of the subsequent frames and specifies a reduced maximum data limit 311b to be followed by the subsequent frames. The reduced maximum data limit can be predefined and can be based on the bandwidth of the communication channel and the type, requirements of the online game. In response to the instruction from the bit rate monitor 311, the bit rate adjuster 312 generates a signal with a signal generator 312a to the encoder 305 to adjust the amount of data in the subsequent frames downward to pay off the data overflow caused by the encoded frames sent to the client device 100. The encoder adjusts the amount of data in the subsequent frames according to the instructions included in the signal. As the subsequent frames are adjusted, encoded, and sent over the internet, the bit rate debt counter 313a and the frame debt counter 313b are updated to account for the amount of debt paid off from the subsequent frames. The evaluation of the encoded frames, the querying of the sent data debt tracker, the instruction of the bit rate monitor to the bit rate adjuster, and the bit rate adjuster 132 sending a signal to adjust the subsequent frames are performed as the encoded frames are generated by the encoder and before the subsequent frames are received from the frame source (e.g., the online game) and continue until the sent debt is fully paid off. Reference will be made to Figure 4 The detailed description uses an example of a sequence of operations performed by the bit rate monitor 311 and the bit rate adjuster 312 to instruct the encoder.

[0069] Figure 3B Different components of the frame data processor 309 in one implementation are shown that are used to identify the complexity of the content contained in the frames. The frame data processor 309 is configured to receive data frames generated at the frame source (e.g., a game application) and analyze the data contained in the frames before forwarding the frames to the encoder for further analysis and encoding. The frame data processor 309 uses a frame data analyzer 309a to analyze the data contained in the frames to understand the complexity of the data. The frame data analyzer 309a can perform a contextual analysis of the data to determine how the data relates to each user. The contextual analysis can be used to identify different data as being critical to present to different users that can be playing or watching the same game and within the same scene of the online game (e.g., a multi-player online game). For example, movement of objects behind a user / user avatar in an online game can be non-critical, while movement or activity of objects in the visual vicinity of the user can be critical. Alternatively, objects that the user is interacting with or activities that the user is performing can be more critical to the user than objects that another user is interacting with in the same scene or activities performed by another user.

[0070] The frame data adjuster 309b receives the results of the context analysis from the frame data analyzer 309a and uses the analysis results to identify data related to critical portions of the frame and data related to portions of the frame that are non-critical to the user. The frame data processor 309 forwards the data frame along with the information resulting from the context analysis to the encoder 305. When the encoder 305 needs to dynamically adjust the data in the frame in response to a signal from the bit rate adjuster 312, the encoder 305 uses the information provided by the frame data processor 309. The encoder 305 encodes the adjusted data frame and forwards the encoded frame to the client device for rendering. Note that the frame that is dynamically adjusted follows a large frame that caused the data overflow.

[0071] The data overflow can be repaid over multiple subsequent frames following the large frame. In some implementations, it can be the case that a particular subsequent frame of the multiple subsequent frames following the large frame can itself include data that exceeds the maximum data limit. In this case, the bit rate debt counter 313a and the frame debt counter 313b are updated to account for the excess data included in the particular subsequent frame. In addition, the frame number calculation engine 311c can identify an additional number of frames that need to be dynamically adjusted to repay the additional sent data debt resulting from the particular subsequent frame exceeding the limit, which is added to the appropriate debt counter. However, such a case can be rare, as most of the time, a large data frame is followed by a series of smaller data frames from which the excess data is easily repaid before the next large frame is received.

[0072] The dynamic adjustment of the bit rate in the subsequent frames allows the excess data to be repaid while maintaining the frame rate and minimizing the latency during rendering of the frames. This allows the system to provide the user with video content that does not include dropped frames or affect the quality of the video.

[0073] Figure 4 A series of processing flow operations are shown in one implementation that are performed using components of the bit rate monitor 311 and the bit rate adjuster 312 of the monitoring engine 306 to instruct the encoder 305 to adjust the bit rate in order to maintain the frame rate. An initial frame is generated by a frame source, such as a game engine executing an online game. Although specific implementations are discussed with reference to online games, the implementations are extendable to any other application that generates streaming content. In the case of playing an online game for the first time, the initial frame can be related to the initial scene of the online game that the user selected to play. In the case of continuing an online game from a point at which the online game was previously paused, the initial frame can be related to the scene of the pause point of the online game. The initial frame can also include user-related content, such as a user-customized avatar, user attributes, etc. The initial frame is forwarded to the encoder for encoding before being transmitted to the client device, as shown by bubble 1.

[0074] When the initial frame arrives at the encoder, the encoder checks whether there are any instructions from the bit rate monitor 311 to dynamically adjust the amount of data contained in the initial frame, and this check is done by the encoder before encoding the initial frame. Since this is the first frame received for an online game (either newly started or resumed from a pause point), both the bit rate debt counter 313a and the frame debt counter 313b are zero. Thus, the bit rate monitor does not instruct to adjust the amount of data contained in the initial frame. As a result, the initial frame is encoded according to the encoding algorithm defined in the encoder 305 to generate an encoded frame. The encoded frame is forwarded to the bit rate monitor 306 to enable the bit rate monitor 306 to determine the size of the encoded frame, as shown by bubble 2.

[0075] Even for the initial frame, the bit rate monitor 311 queries the sent debt data tracker 313 in response to receiving the encoded frame to determine whether there is any data overflow that needs to be repaid, as shown by bubble 4. The data tracker 313 queries the bit rate debt counter 313a and the frame debt counter 313b to determine whether there is any remaining data overflow. If one or both of these counters 313a, 313b are non-zero, it means that there is some data overflow that still needs to be repaid. If both counters are zero, it indicates that the data overflow has been fully repaid by the system. For the initial frame arriving at the encoder, both the bit rate debt counter 313a and the frame debt counter 313b are zero because there is no bit rate overflow recorded for the online game yet. In this case, there is no prior bit rate overflow signal sent by the bit rate monitor 311 to the bit rate adjuster 312.

[0076] Next, the bit rate monitor 311 determines the size of the encoded frame by evaluating the amount of data contained in the encoded initial frame against the maximum data limit predefined for the online game. When the bit rate monitor 311 determines that the size of the encoded initial frame is greater than the maximum data limit defined for the communication channel, the bit rate monitor 311 calculates the bit rate overflow and updates the bit rate debt counter 313a with the bit rate overflow (i.e., the amount of data in bytes that exceeds the maximum data limit). In addition, based on the bit rate overflow, the type of data being generated, the needs of the online game, and the maximum bit rate limit of the communication channel, the bit rate monitor 311 determines the number of frames in which to repay the data overflow of the initial frame. For example, for a streaming interactive video game, the bit rate monitor 311 can use an aggressive number of frames to repay the data overflow, while for a streaming game being watched by a multi-bit audience, the bit rate monitor 311 can use a more lenient number of frames to repay the data overflow. The number of frames to use to repay the data overflow is updated to the frame debt counter 313b, as shown by bubble 4.

[0077] Based on the data overflow determination, a signal is sent from the bit rate monitor 311 to the network interface to adjust the packets of encoded data being sent to the client, as shown by bubble 3. In packaging the encoded data according to the transmission protocol being followed for transmitting data to the client device over the internet, the package adjustment signal or lack thereof is used by the network interface. The packaged data is forwarded to the client device for decoding and rendering. When the signal from the bit rate monitor indicates that there is excess data in the encoded initial frame, the network interface adjusts the encoded data included in the package during packaging so that the data transmitted in the package is within the data bandwidth of the communication channel. The remaining excess encoded data is saved in a cache memory and included in the data package of one or more subsequent frames.

[0078] In addition to updating the bit rate debt counter 313a and the frame debt counter 313b and signaling the network interface to adjust the data package, the bit rate monitor 311 also instructs the bit rate adjuster 312 to send a signal to the encoder 305 to indicate that there has been a sent debt overflow that needs to be repaid from one or more subsequent frames, as shown by bubble 5. The bit rate adjuster 312 sends a signal to the encoder to dynamically inject a reduced maximum data limit for the data included in the subsequent frames received at the encoder, as shown by bubble 6. The reduced maximum data limit can be predefined and can be based on the type of data being generated and the needs of the game.

[0079] When the second frame (i.e., a subsequent frame of the initial frame) is received at the encoder 305, the encoder 305 checks whether any signal has been received from the bit rate adjuster 312. In response to the signal from the bit rate adjuster 312, the encoder 305 dynamically injects the predefined reduced maximum data limit by dynamically adjusting the data contained in the second frame. The encoder can reduce the amount of data contained in the second frame by compressing the data using compression logic. In some implementations, different types of compression logic can be implemented based on the amount of data that needs to be adjusted. Alternatively, selective data identified as being critical to the construction of the image can be retained in the second frame and the rest of the non-critical data can be discarded. Different ways of reducing the data in the second frame can be used and are not limited to compression or selective inclusion / discard. The encoder then encodes the adjusted data and forwards the encoded second data frame to the network interface.

[0080] The network interface receives the adjusted and encoded data of the second frame and the signal from the bit rate monitor to include some excess data from the initial frame. Depending on the signal from the bit rate monitor, the network interface includes some data left over from the large frame and data of the second frame, thereby paying off some sent debt, and forwards the packet of the encoded and adjusted second frame to the client device for rendering. Since the second frame includes data equal to or less than the reduced maximum data limit, there is enough room within the frame to include the data left over from the large frame.

[0081] After forwarding the second data frame to the network interface, the bit rate monitor 311 updates the bit rate debt counter and the frame debt counter. Since the amount of data contained in the second frame is within the reduced maximum data limit, there is no data overflow caused by the data contained in the second frame. Rather, some sent debt has been paid off. The bit rate monitor 311 captures the amount of sent debt paid off with the second frame by adjusting the bit rate debt counter 313a and the frame debt counter 313b. After adjusting the debt counters (313a, 313b), the bit rate monitor 311 forwards the updated signal to the bit rate adjuster 312, providing the current value in the respective debt counter. When the debt counter is non-zero, the bit rate adjuster forwards another signal to the encoder to adjust the data of the third frame received at the encoder to pay off the sent debt. As long as content frames are generated, the process of receiving data frames from the frame source, evaluating the amount of data contained in each frame, dynamically adjusting the amount of data contained in each frame by sending appropriate signals to the encoder, and forwarding the adjusted content to the client device continues for the online game.

[0082] Figure 5A simple example table is shown that represents frame data processing performed using an encoder and monitoring engine, according to one implementation. In this example, at time Tl, a frame (e.g., frame 10) is received at the encoder. Frame 10 can be an initial frame (I-frame) generated when an online game is selected to play, or can be a frame generated during game play of a current session or continued from a previously paused game session. Time Tl can be a time representing the start / continuation of a game or can be some time during the playing of an online game. Frame 10 is shown to include 40,000 bytes of data. The content frame properties of the online game can be defined to have a data frame rate of 60 frames per second (fps). The maximum bit rate of each frame can be defined according to the bandwidth availability of a communication channel established between a server executing the online game and a client device used to interact with the online game. For example, the maximum bit rate can be defined to be 10 megabits per second (Mbps). The specified bit rate defines the amount of data that can be included in each frame. In the example above, each frame is defined to have a maximum data limit of 10,000,000 / 60 / 8 = 20,833 bytes per frame.

[0083] Frame 10 is encoded using the encoder and the encoded frame of data is forwarded to the bit rate monitor for evaluation. The bit rate monitor evaluates the amount of data contained in frame 10 and determines that frame 10 includes excess data of about 20,000 bytes over the amount of data defined in the frame properties for the online game. The bit rate monitor also queries the sent debt counters (the bit rate debt counter and the frame debt counter) to see if either of these sent debt counters is non-zero. The bit rate monitor uses the information from the sent debt counters to determine if any excess sent debt needs to be repaid using one or more subsequent frames. If frame 10 is the initial frame generated for the current game play session, then the sent debt counters will be zero. On the other hand, if frame 10 is from the current game play session of the online game and the previous frame 9 did not have any excess data and did not have to be adjusted, then the sent debt counters will be zero. Based on the determination from the sent debt counter query (i.e., 'adjust this frame' is "no"), the bit rate monitor sends a signal to the network interface to package the encoded frame 10 data according to the bandwidth limitations of the communication channel. In addition, when the bit rate monitor detects that the encoded frame 10 includes excess data, the bit rate monitor also causes the bit rate debt counter to be increased by 20,000 (the data over the maximum data allowed) and the frame debt counter to be increased by 2. The number of frames to adjust is determined by the amount of excess data included in the encoded frame and the reduced maximum byte limit defined for the content. The portion of the data of frame 10 (i.e., about 20,000 bytes of data) is packaged and sent to the client, while the remaining portion of the data (i.e., about 20,000 bytes of excess data) is saved in the cache to be included in a subsequent frame. No adjustment is made to the frame 10 data (i.e., no dynamic reduction of the data included in frame 10) other than saving some of the excess encoded data in the local cache. The bit rate monitor then signals the bit rate adjuster to send a signal to the encoder to at least adjust the next subsequent frame to repay the sent debt and the signal is sent as soon as frame 10 is forwarded to the network interface.

[0084] At time T2, the encoder receives frame 11. Upon receiving frame 11, the encoder determines that a signal has been received from the bit rate adjuster to adjust the data in frame 11 (i.e., 'adjust this frame' is "yes"). The encoder determines the reduced maximum bytes that can be included in subsequent frames (i.e., 10,000 bytes) and evaluates frame 11 to determine if frame 11 needs to be adjusted. There can be instances where no adjustment of subsequent frames is needed, even though a signal was sent to the encoder to adjust the subsequent frames. This can occur when the amount of data included in the subsequent frame (i.e., frame 11 in this example) is below the reduced maximum data limit. Based on this evaluation, the encoder determines that frame 11 includes only 9,000 bytes, which is below the reduced maximum data limit of 10,000 bytes (i.e., there is no bit rate overflow in frame 11). As a result, the encoder encodes only the data for frame 11 and forwards the encoded frame 11 data to the network interface with instructions to include some portion of the frame 10 data stored in the cache. In this example, frame 11 leaves approximately 11,000 bytes available to process frame 10 over budget, so the overall frame data does not exceed the maximum data limit of 20,000 bytes.

[0085] After forwarding the encoded frame 11 data to the network interface with instructions, the bit rate monitor decrements the bit rate debt counter by 11,000 bytes and decrements the frame debt counter by 1, as the 11,000 bytes of frame 10 are sent in the subsequent frame 11. The adjusted and encoded data included in frame 11 is packaged and forwarded to the client device for rendering. The bit rate monitor instructs the bit rate adjuster to send another signal to the encoder with updated information for the debt counters.

[0086] At T3, frame 12 is received at the encoder and is approximately 10,000 bytes, well below the reduced maximum data limit and well below the maximum data limit. Based on the signal from the bitrate adjuster, the encoder evaluates frame 12 to determine if the data in frame 12 needs to be adjusted. However, based on the evaluation of frame 12, the encoder determines that there is no bitrate overflow in frame 12. As a result, the encoder determines that the data in frame 12 does not have to be adjusted prior to encoding. On the other hand, if frame 12 included data that exceeded the reduced maximum data limit (e.g., if frame 12 included 12,000 or 15,000 bytes), the encoder would adjust the data in frame 12 by compressing or eliminating / discarding non-critical data so that the data in frame 12 is below the reduced maximum data limit. The encoder then encodes the frame 12 data and forwards it to the network interface along with instructions to include any data carried over from frame 10. As a result, frame 12 leaves approximately 10,000 bytes available to account for the over-budgeted frame 10, so the total frame data does not exceed the 20,000 byte maximum data limit. Thus, when frame 12 is packaged, the network interface processes the remaining 9,000 bytes of frame 10 data along with the data of frame 12 and forwards the packet of encoded frame 12 data and frame 10 data to the client for rendering. In forwarding the encoded frame 12 data to the network interface, the bitrate monitor decrements the bitrate debt counter by 9,000 and decrements the frame debt counter by 1, resulting in both debt counters equaling zero. Thus, the bitrate monitor instructs the bitrate adjuster to send a signal to the encoder to continue encoding subsequent frames at the maximum data limit.

[0087] When frame 13 is received at the encoder, the encoder evaluates the amount of data in frame 13 based on the signal from the bitrate adjuster to continue the maximum data limit for subsequent frames and determines that the data is below the maximum data limit. The signal from the bitrate adjuster to the encoder indicates that no adjustment of the data in frame 13 is needed because both debt counters indicate a value of zero. Based on this determination, the encoder encodes the frame 13 data and forwards the encoded frame 13 data to the network interface along with instructions to package and send only the frame 13 data to the client device. The client device receives the frame 13 data and renders it. The process of receiving data frames at the encoder, evaluating and updating the debt counters, instructing the bitrate adjuster to send appropriate signals to the encoder, and sending appropriate signals to the network interface continues as long as frames are being generated for the online game.

[0088] Figure 6Operations of a method for dynamically adjusting bitrates of one or more subsequent frames of an online game in order to maintain a frame rate defined for the online game are shown in accordance with one implementation. The method begins when a data frame is received at an encoder, operation 610. In response to user input provided from a client device to an online game executing at a server, a data frame can be generated by an online game engine, and the data frame can include a game state of the online game. The online game can specify a maximum data limit per frame, but the game engine can not always adhere to the maximum data limit specified for the online game when generating data frames.

[0089] As shown in operation 620, the data frame is encoded by the encoder before being transmitted to the client device. The encoder encodes the data within the frame and forwards the encoded frame to a network interface for continued transmission to the client device, even if the data contained in the frame exceeds the maximum data limit. The encoded data frame is also forwarded to a bitrate monitor within the monitoring engine. The bitrate monitor assesses the amount of data contained in the encoded frame, as shown in operation 630. When the bitrate monitor detects data that exceeds the maximum data limit, the bitrate monitor instructs a bitrate adjuster within the monitoring engine to signal the encoder to dynamically adjust the amount of data contained in one or more subsequent frames, as shown in operation 640. The bitrate monitor can calculate the amount of data overflow caused by the frame and update one or more counters (e.g., bitrate debt counter, frame debt counter). In updating the one or more counters, the bitrate monitor uses the updated information included in the debt counters to instruct the bitrate adjuster. The instructions given to the bitrate adjuster can also include a predefined reduced maximum data limit to use when dynamically adjusting the amount of data in one or more subsequent frames. The reduced maximum data limit can be defined based on the bandwidth of the communication channel established between the server device and the client device. Alternatively, the encoder can identify a predefined reduced maximum data limit for the online game. The encoder uses the instructions from the bitrate adjuster to dynamically adjust the amount of data contained in subsequent frames so that the data overflowed from the large frame can be repaid from the subsequent frames.

[0090] The adjustment of the subsequent frames can be done using compression logic or by identifying and eliminating non-critical data in the frames or by other ways that can be used to reduce data without affecting the image quality included in the frames. The various implementations described herein ensure smooth rendering of the content frames of the online game. Some frames that can have been dynamically adjusted to repay data overflow can have a lower resolution due to the data reduction, but the low quality frames will last for a short period of time before continuing with regular resolution. Moreover, the frame rate defined for the online game is maintained, resulting in the rendering of the video of the online game proceeding without any stuttering. Other advantages can be envisioned by those skilled in the art upon viewing the various implementations.

[0091] The various implementations discussed in this application can be implemented on a cloud system. According to one implementation of the present disclosure, an overview of an example cloud system will be described that supports collecting user input to update interactive application data and using the updated application data to generate content frames for transmission to client devices. The example cloud system described herein is a game cloud system (GCS) that enables access to and play of video games stored therein. The GCS can be a cloud computing system running over a network to support a plurality of players playing one or more online game applications through corresponding game plays. Data related to these game plays is provided in the form of frames that are encoded by an encoder and transmitted as a data stream over the network to client devices. In particular, the system includes a GCS that interacts with one or more social media providers and client devices, all connected through a network (e.g., the Internet). A plurality of client devices associated with a plurality of users can connect to the network to access services provided by the GCS and social media providers. The users can be players or spectators following a particular player’s game play or online game.

[0092] In one embodiment, the GCS includes one or more game servers, a video recorder, a tag processor, and an account manager (including a user profile manager), a game selection engine (i.e., game selector), a game session manager, user access logic, a network interface, and a social connection manager. The GCS can also include a plurality of storage systems, such as game state storage, user accounts, user data storage, etc., which can generally be stored in a data store. Other game storage systems can include game code storage, recorded game storage, tag data storage, video game title storage, and game network user storage. In one embodiment, the GCS is a system that can provide online game applications, services, game-related digital content, and interconnectivity between the system, applications, users, and social networks. The GCS can communicate with user devices and social media providers through the social connection manager via the network interface. The social connection manager can be configured to associate and / or connect to one or more friends identified using a social media provider. In one embodiment, each social media provider includes at least one social graph that illustrates social network relationships of users.

[0093] Players (i.e., users) are able to access services provided by the GCS via the game session manager. For example, the account manager enables authentication of players and access to game titles on the GCS. The account manager stores information about users (players, spectators) and manages a user profile for each user. The account manager can use the user information for authentication. In addition, game titles owned by the users can be managed by the account manager. For example, video games stored in the game title storage can be available to users who own or have access to those video games.

[0094] In one embodiment, a user can access services provided by the GCS and social media providers via a client device through a communication connection established over a network. The client device can include any type of device having a processor and memory, whether wired or wireless, portable or non-portable. In one embodiment, the client device can take the form of a smart phone, tablet computer, or hybrid device that provides touch screen capability in a portable form factor. One exemplary client device can include a portable phone device that runs an operating system and is capable of accessing various applications (apps) that can be obtained over a network and executed locally on the portable device (e.g., smart phone, tablet computer, laptop computer, desktop computer, etc.).

[0095] The client device can include a display that acts as an interface for the user to send input commands and display frames and / or information received from the GCS and social media providers. The display can be configured as a touch screen, or a display generally provided by a flat panel display, cathode ray tube (CRT), or other device capable of rendering a display picture. Alternatively, the client device 100 can have its display separate from the device, similar to a desktop or laptop computer.

[0096] In one embodiment, the client device is configured to communicate with the GCS to enable a user (player) to play a video game. For example, the player can select (e.g., by game title, etc.) a video game available in the game title data store via a game selection engine. The selected video game is enabled, and an instance of the video game is loaded for execution by a game server on the GCS. In one embodiment, the game play is executed primarily in the GCS, and data frames generated from the game play are encoded using an encoder and streamed to the client device as a stream of encoded game video frames from the GCS. User input commands used to drive the game play are transmitted back to the GCS. The encoded video frames received from the streaming game play are decoded at the client device and rendered on the display of the client device.

[0097] In one embodiment, after a player selects an available game title to play, the game session for the selected game title can be started by the player through the game session manager. The game session manager first accesses the game state storage in the data storage to retrieve the saved game state (if any) of the last session played by the player (for the selected game) so that the player can resume playing the game from the previous game play stop point. Once the resume or start point is identified, the game session manager can instruct the game execution engine in the game server to execute the game code of the selected game title from the game code storage. After the game session starts, the game session manager can pass the game video frames (i.e., streaming video data) to the encoder for encoding, and the encoded video frames are transmitted to the client device via the network interface.

[0098] During game play, the game session manager can communicate with the game server, the recording engine in the video recorder, and the tag processor to produce or save a recording (e.g., video) of the game play or game play session. In one embodiment, the video recording of the game play can include any tag content and other game related metadata entered or provided during the game play. The tag content can also be saved through snapshots. The video recording of the game play and any game metrics corresponding to the game play can be saved in the recorded game storage. Any tag content can be saved in the tag data storage.

[0099] User input commands provided by a user at a client device are transmitted to the game server via the game session manager. The user input commands are used to affect the outcome of the corresponding game play of the video game. The input commands (e.g., controller inputs or user inputs) include input commands for driving the game play and user interaction inputs such as tag content (e.g., text, images, video recording clips, etc.). The game input commands and any user game play metrics (how long the user played the game, etc.) can be stored in the game network user storage. The selection information related to the game play of the video game can be used to enable a plurality of features that can be available to the user.

[0100] Figure 7 is a block diagram of a game system 700 in accordance with various embodiments of the present invention. The game system 700 is configured to provide video streaming to one or more clients 710 via a network 715. The network is similar to the network 705 described above with respect to FIG. 7A. Figure 1 , Figure 2A and Figure 2BThe game system 700 generally includes a video server system 720 and an optional game server 725. The video server system 720 is configured to provide video streams to one or more clients 710 with minimal quality of service. For example, the video server system 720 can receive game commands that change the state of a video game or the perspective within a video game and provide updated video streams to the client devices 710 that instantaneously reflect such changes with minimal lag time. The video server system 820 can be configured to provide video streams in a variety of alternative video formats, including formats that have not yet been defined. Further, the video streams can include video frames that are configured to be presented to users at a variety of frame rates. Typical frame rates are 30 frames per second, 60 frames per second, and 120 frames per second. But higher or lower frame rates are included in alternative embodiments of the application.

[0101] The clients 710 (individually referred to herein as 710A, 710B, etc.) can include head-mounted displays, terminals, personal computers, game consoles, tablet computers, telephones, set-top boxes, kiosks, wireless devices, digital tablets, standalone devices, handheld game-playing devices, etc. Generally, the clients 710 are configured to receive encoded video streams, decode the video streams, and present the resulting video to a user, e.g., a game player. The process of receiving encoded video streams and / or decoding the video streams generally includes storing individual video frames in a receive buffer of the client. The video streams can be presented to the user on a display integral to the client 710 or on a separate device such as a monitor or television. The clients 710 are optionally configured to support more than one game player. For example, a game console can be configured to support two, three, four, or more simultaneous players. Each of these players can receive a separate video stream, or a single video stream can include frame regions specifically generated for each player, e.g., frame regions generated based on the perspective of each player. The clients 710 are optionally geographically dispersed. The number of clients included in the game system 700 can vary widely, from one or two to thousands, hundreds of thousands, or more. As used herein, the term "game player" is used to refer to a person playing a game, and the term "game-playing device" is used to refer to a device used to play a game. In some embodiments, a game-playing device can refer to multiple computing devices that cooperate to deliver a game experience to a user. For example, a game console and an HMD can cooperate with a video server system 720 to deliver a game viewed through the HMD. In one embodiment, a game console receives a video stream from a video server system 720, and the game console forwards the video stream or updates to the video stream to the HMD for rendering.

[0102] Client 710 is configured to receive a video stream via network 715. Network 715 can be any type of communication network, including a telephone network, the Internet, a wireless network, a power line network, a local area network, a wide area network, a private network, etc. In typical embodiments, the video stream is transmitted via a standard protocol, such as TCP / IP or UDP / IP. Alternatively, the video stream is transmitted via a proprietary standard.

[0103] A typical example of client 710 is a personal computer that includes a processor, non-volatile memory, a display, decoding logic, network communication capabilities, and an input device. The decoding logic can include hardware, firmware, and / or software stored on a computer readable medium. Systems for decoding (and encoding) video streams are well known in the art and vary depending on the particular encoding scheme used.

[0104] Client 710 can, but need not, further include a system configured for modifying the received video. For example, the client can be configured to perform further rendering, to overlay one video image on another video image, to crop a video image, etc. For example, client 710 can be configured to receive various types of video frames, such as I-frames, P-frames, and B-frames, and to process these frames into images for display to a user. In some embodiments, members of client 710 are configured to perform further rendering, shading, conversion to 3D, or similar operations on the video stream. Members of client 710 are optionally configured to receive more than one audio or video stream. For example, the input device of client 710 can include a single-hand game controller, a two-hand game controller, a gesture recognition system, a gaze recognition system, a voice recognition system, a keyboard, a joystick, a pointing device, a force feedback device, a motion and / or position sensing device, a mouse, a touch screen, a neural interface, a camera, an input device to be developed, etc.

[0105] The video stream (and optionally the audio stream) received by client 710 is generated and provided by video server system 720. As described further elsewhere herein, the video stream includes video frames (and the audio stream includes audio frames). The video frames are configured (e.g., they include pixel information in an appropriate data structure) to meaningfully contribute to an image displayed to a user. As used herein, the term "video frame" is used to refer to a frame that primarily includes information configured to contribute to (e.g., to implement) an image shown to a user. Most of the teachings herein regarding "video frames" can also apply to "audio frames."

[0106] Client 710 is generally configured to receive input from a user. These inputs can include game commands configured to change the state of a video game or otherwise affect the course of gameplay. Game commands can be received using input devices and / or can be automatically generated by computing instructions executing on client 710. Received game commands are communicated from client 710 to video server system 720 and / or game server 725 via network 715. For example, in some embodiments, game commands are communicated to game server 725 via video server system 720. In some embodiments, separate copies of game commands are communicated from client 710 to game server 725 and video server system 720. The communication of game commands is optionally dependent on the identity of the command. Optionally, game commands are communicated from client 710A over a different route or communication channel used to provide audio or video streams to client 710A.

[0107] Game server 725 is optionally operated by a different entity than video server system 720. For example, game server 725 can be operated by the publisher of a multiplayer game. In this example, video server system 720 is optionally treated by game server 725 as a client and is optionally configured to appear to game server 725 as a prior art client executing a prior art game engine. Communication between video server system 720 and game server 725 is optionally via network 715. Thus, game server 725 can be a prior art multiplayer game server that sends game state information to multiple clients, one of which is video server system 720. Video server system 720 can be configured to communicate with multiple instances of game server 725 simultaneously. For example, video server system 720 can be configured to provide multiple different video games to different users. Each of these different video games can be supported by different game servers 725 and / or published by different entities. In some embodiments, several geographically dispersed instances of video server system 720 are configured to provide game video to multiple different users. Each of these instances of video server system 720 can communicate with the same instance of game server 725. Communication between video server system 720 and one or more game servers 725 optionally occurs via a dedicated communication channel. For example, video server system 720 can be connected to game server 725 via a high bandwidth channel dedicated to communication between the two systems.

[0108] The video server system 720 includes at least a video source 730, an I / O device 745, a processor 750, and a non-transitory storage device 755. The video server system 720 can include one computing device or be distributed among multiple computing devices. The computing devices are optionally connected via a communication system such as a local area network.

[0109] The video source 730 is configured to provide a video stream, e.g., a streaming video or a series of video frames forming a dynamic picture. In some embodiments, the video source 730 includes a video game engine and rendering logic. The video game engine is configured to receive game commands from a player and to maintain a copy of a video game state based on the received commands. The game state includes the positions of objects in a game environment and typically includes a point of view. The game state can also include attributes, images, colors, and / or textures of the objects.

[0110] The game state is typically maintained based on game rules and game commands such as moving, turning, attacking, setting a focus, interacting, using, etc. A portion of the game engine is optionally disposed in a game server 725. The game server 725 can maintain a copy of the state of a game based on game commands received from multiple players using geographically dispersed clients. In these cases, the game state is provided by the game server 725 to the video source 730, where a copy of the game state is stored and rendering is performed. The game server 725 can receive game commands directly from the clients 710 via the network 715 and / or can receive game commands via the video server system 720.

[0111] The video source 730 typically includes rendering logic, e.g., hardware, firmware, and / or software stored on a computer readable medium such as the storage device 755. The rendering logic is configured to create video frames of a video stream based on the game state. All or a portion of the rendering logic is optionally disposed in a graphics processing unit (GPU). The rendering logic typically includes processing stages configured to determine three-dimensional spatial relationships between objects and / or apply appropriate textures, etc., based on the game state and the point of view. The rendering logic produces raw video, which is then typically encoded before being transmitted to the clients 710. For example, the raw video can be encoded according to Adobe Flash® or Microsoft Silverlight®. The video source 730 is optionally disposed in a game server 725. Standard,.wav, H.264, H.263, On2, VP6, VC-1, WMA, Huffyuv, Lagarith, MPG-x. Xvid. FFmpeg, x264, VP6-8, realvideo, mp3, etc. The encoding process produces a video stream, which is optionally packetized for transmission to a decoder on a remote device. The video stream is characterized by frame size and frame rate. Typical frame sizes include 800x600, 1280x720 (e.g., 720p), 1024x768, but any other frame size can be used. Frame rate is the number of video frames per second. The video stream can include different types of video frames. For example, the H.264 standard includes "P" frames and "I" frames. I frames include information to refresh all macroblocks / pixels on the display device, while P frames include information to refresh a subset thereof. The data size of P frames is typically smaller than I frames. As used herein, the term "frame size" is intended to refer to the number of pixels within a frame. The term "frame data size" is used to refer to the number of bytes required to store a frame.

[0112] In an alternative embodiment, the video source 730 includes a video recording device, such as a camera. The camera can be used to generate delayed video or live video, which can be included in the video stream of a computer game. The resulting video stream optionally includes both rendered images and images recorded using a still camera or video camera. The video source 730 can also include a storage device configured to store previously recorded video to be included in the video stream. The video source 730 can also include a motion or positioning sensing device configured to detect motion or position of an object (e.g., a person), and logic configured to determine a game state or produce video based on the detected motion and / or position.

[0113] The video source 730 is optionally configured to provide overlays configured to be placed over other video. For example, these overlays can include a command interface, a login explanation, a message to a game player, an image of another game player, a video feed (e.g., a webcam video) of another game player. In embodiments of the client 710A that include a touchscreen interface or a gaze detection interface, the overlays can include a virtual keyboard, a joystick, a touchpad, etc. In one example of an overlay, a player's voice is overlaid on the audio stream. The video source 730 optionally also includes one or more audio sources.

[0114] In embodiments in which the video server system 720 is configured to maintain a game state based on input from more than one player, each player can have a different perspective including a position and orientation of the view. The video source 830 is optionally configured to provide a separate video stream for each player based on the perspective of each player. In addition, the video source 730 can be configured to provide a different frame size, frame data size, and / or encoding to each of the clients 710. The video source 730 is optionally configured to provide 3D video.

[0115] The I / O device 745 is configured to cause the video server system 720 to send and / or receive information, such as video, commands, information requests, game state, gaze information, device motion, device position, user motion, client identity, player identity, game commands, security information, audio, etc. The I / O device 745 generally includes communication hardware such as a network card or modem. The I / O device 745 is configured to communicate with the game server 725, the network 715, and / or the clients 710.

[0116] The processor 750 is configured to execute logic, e.g., software, included within the various components of the video server system 720 discussed herein. For example, the processor 750 can be programmed with software instructions to perform the functions of the video source 730, the game server 725, and / or the client limiter 760. The video server system 720 optionally includes more than one instance of the processor 750. The processor 750 can also be programmed with software instructions to execute commands received by the video server system 720, or to coordinate the operation of the various elements of the game system 700 discussed herein. The processor 750 can include one or more hardware devices. The processor 750 is an electronic processor.

[0117] The storage device 755 includes a non-transitory analog and / or digital storage device. For example, the storage device 755 can include an analog storage device configured to store video frames. The storage device 755 can include a computer-readable digital storage device, e.g., a hard drive, an optical drive, or a solid state storage device. The storage device 715 is configured to store (e.g., through an appropriate data structure or file system) video frames, artificial frames, video streams including both video frames and artificial frames, audio frames, audio streams, etc. The storage device 755 is optionally distributed among multiple devices. In some embodiments, the storage device 755 is configured to store software components of the video source 730 discussed elsewhere herein. These components can be stored in a just-in-time ready-to-serve format.

[0118] The video server system 720 optionally also includes a client limiter 760. The client limiter 760 is configured to remotely determine the capabilities of a client, such as the client 710A or 710B. These capabilities can include the capabilities of the client 710A itself as well as the capabilities of one or more communication channels between the client 710A and the video server system 720. For example, the client limiter 760 can be configured to test the communication channel through the network 715.

[0119] The client limiter 760 can manually or automatically determine (e.g., discover) the capabilities of the client 710A. Manual determination includes communicating with a user of the client 710A and asking the user to provide the capabilities. For example, in some embodiments, the client limiter 760 is configured to display an image, text, etc. within a browser of the client 710A. In one embodiment, the client 710A is an HMD that includes a browser. In another embodiment, the client 710A is a game console with a browser that can be displayed on the HMD. The displayed object requests user input information, such as the operating system of the client 710A, the processor, the video decoder type, the network connection type, the display resolution, etc. The user input information is communicated back to the client limiter 760.

[0120] Automatic determination can occur, for example, by executing an agent on the client 710A and / or by sending a test video to the client 710A. The agent can include computing instructions embedded in a web page or installed as an add-on, such as java script. The client limiter 760 optionally provides the agent. In various embodiments, the agent can find the processing capabilities of the client 710A, the decoding and display capabilities of the client 710A, the latency reliability and bandwidth of the communication channel between the client 710A and the video server system 720, the display type of the client 710A, the presence of a firewall on the client 710A, the hardware of the client 710A, the software executing on the client 710A, the registry entries within the client 710A, etc.

[0121] The client limiter 760 includes hardware, firmware, and / or software stored on computer- readable media. The client limiter 760 is optionally disposed on a computing device separate from one or more other elements of the video server system 720. For example, in some embodiments, the client limiter 760 is configured to determine the characteristics of a communication channel between the client 710 and more than one instance of the video server system 720. In these embodiments, the information discovered by the client limiter can be used to determine which instance of the video server system 720 is best suited to deliver streaming video to one of the clients 710.

[0122] Figure 8Implementations of an information service provider architecture that can be used to provide access to different games are shown. An information service provider (ISP) 802 provides a number of information services to users 800 that are geographically dispersed and connected through a network 850. The network 850 can be similar to the network 200 of Figure 1 , Figure 2A and Figure 2B Although various implementations have been discussed with reference to providing fast access to games, the implementations can be extended to provide faster access to other interactive applications, such as virtual tours, etc., that provide content that can include various levels of content that can be accessed using one or more visual cues or other tools or application points or rewards or rewards that a user obtains during interaction with the interactive application. For example, the ISP 802 can provide only one type of service, such as games, or can provide multiple services, such as games, stock price updates, broadcast media, news, sports, games, etc. In addition, the services provided by each ISP can be dynamic, i.e., services can be added or removed at any point in time. Thus, the ISP that provides a particular type of service to a particular individual can change over time. For example, when a user is at her home town, the user can be provided services by an ISP that is close to the user, and when the user travels to a different city, the user can be provided services by a different ISP. The home ISP will transfer the required information and data from the user's game or access profile to the new ISP through the connection module, so that the user information will "follow" the user to the new city, making the data closer to the user and easier to access. In another implementation, a master-server relationship can be established between a master ISP that manages the information for the user and a server ISP that interfaces directly with the user under the control of the master ISP. In another implementation, as the client moves around the world, the data is transferred from one ISP to another (i.e., during a data center switch assigned to the user), and this transfer can be based on the compatibility of the services provided by the respective ISPs, so that the ISP 902 that is in a better position to serve the user is the ISP that provides these services.

[0123] The ISP 802 includes an application service provider (ASP) 806 that provides computer-based services to customers over a network. Software provided using the ASP model is sometimes also called on-demand software or software as a service (SaaS). A simple form of providing access to a particular application program, such as customer relationship management, is using a standard protocol such as HTTP. For example, the application software resides on the vendor's system and is accessed by the user through a web browser using HTML or a special-purpose client software provided by the vendor or via other remote interface such as a thin client.

[0124] Services provided over a wide geographic area often use cloud computing. Cloud computing is a way of delivering computing capability as a service. Instead of having an IT department in-house, users need not be experts in the technology infrastructure in the "cloud" that supports them. Cloud computing can be divided into different services, such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). Cloud computing services are often provided online from a web browser, with the software and data stored on servers. The term cloud is used as a metaphor for the Internet (e.g., using servers, storage and logic), based on how computer networks are depicted in diagrams and drawings, and is an abstract concept that hides the complexity of the infrastructure it is hiding.

[0125] Further, the ISP 802 includes a game processing server or provider (GaPS) 808 that is used by game clients to play single and multiplayer video games. Most video games played over the Internet are run through a connection to a game server. Typically, the game uses a dedicated server application that collects data from the players and distributes it to the other players. This is more efficient and effective than a peer-to-peer arrangement, but it requires a separate server to host the server application. In another implementation, the GaPS establishes communications between the players, and their respective game-playing devices exchange information without relying on a centralized GaPS.

[0126] A dedicated GaPS is a server that runs independently of the clients. Such a server typically runs on dedicated hardware located within a data center, providing more bandwidth and dedicated processing power. For most PC-based multiplayer games, a dedicated server is the preferred method of hosting a game server. Large multiplayer online games run on dedicated servers, which are typically hosted by the software company that owns the game title, allowing them to control and update the content.

[0127] A broadcast processing server or provider (BPS) 810 distributes audio or video signals to an audience. Broadcasting to a very small range of audience is sometimes called narrowcasting. The last leg of broadcast distribution is how the signal gets to the radio or television receiver. It can come from an over-the-air broadcast, either terrestrial television networks or a local translator or repeater station, or satellite television from a dish antenna, either directly or through a cable operator, or a webcast platform through a home computer. The Internet can also bring radio or television to the receiver, especially through a multicast capability or "Internet radio". Historically, broadcasting has been defined in terms of geographical area, such as national broadcasts or regional broadcasts. However, with the proliferation of the Internet, broadcasts are not defined by geography, as content can reach almost any country in the world.

[0128] A storage service provider (SSP) 812 provides computer storage space and related management services. The SSP also provides periodic backups and archiving. By providing storage as a service, a user can order more storage as needed. Another major advantage is that the SSP includes backup services, and if a computer's hard drive fails, the user will not lose all of their data. In addition, multiple SSPs can have full or partial copies of a user's data, allowing the user to access the data in an efficient manner regardless of the user's location or the device used to access the data. For example, a user can access personal files at a home computer as well as on a mobile phone as the user moves about.

[0129] A communication provider 814 provides connectivity to users. One type of communication provider is an Internet service provider (ISP) that provides access to the Internet. ISPs connect their customers using data transmission technologies suitable for delivering Internet Protocol datagrams, such as dial-up, DSL, cable modems, fiber, wireless, or dedicated high-speed interconnections. A communication provider can also provide messaging services, such as email, instant messaging, and SMS texting. Another type of communication provider is a network service provider (NSP) that sells bandwidth or network access by providing direct backbone access to the Internet. Network service providers can include telecommunications companies, data carriers, wireless communication providers, Internet service providers, cable television operators that provide high-speed Internet access, and the like.

[0130] A data exchange 804 interconnects several modules within the ISP 802 and connects those modules to the client devices 820 of the users 800 through the network 850. The data exchange 804 can cover a small area where all of the modules of the ISP 802 are in close proximity, or it can cover a large geographic area when the different modules are geographically dispersed. For example, the data exchange 804 can include fast gigabit Ethernet (or faster) within the racks of a data center, or an intercontinental virtual local area network (VLAN).

[0131] The users 800 access remote services using respective client devices 820, which include at least a CPU, memory (not shown), a display, and I / O. The client devices can be PCs, mobile phones, netbooks, tablet computers, game systems, PDAs, and the like. In one implementation, the ISP 802 identifies the type of device used by the client and adjusts the communication methods employed. In other cases, the client devices use standard communication methods, such as HTML, to access the ISP 802.

[0132] Embodiments of the application can be practiced with various computer system configurations including hand-held devices, microcomputer systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers and the like. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.

[0133] With the above embodiments in mind, it should be understood that the application can employ various computer-implemented operations involving data stored in computer systems. These operations are those requiring physical manipulation of physical quantities. Any of the operations described herein that form part of the application are useful machine operations. The application also relates to a device or an apparatus for performing these operations. The apparatus can be specially constructed for the required purposes, or it can be a general purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general purpose machines can be used with computer programs written in accordance with the teachings herein, or it can be more convenient to construct a more specialized apparatus to perform the required operations.

[0134] The application can also be embodied as computer readable code on a computer readable medium. Alternatively, the data exchange interworking described above can be used to download the computer readable code from a server. The computer readable media is any data storage mechanism that can store the data which is later to be read by a computer system. Examples of the computer readable media include a hard drive, a network attached storage, a read-only memory, a random access memory, a CD-ROM, a CD-R, a CD-RW, a magnetic tape and other optical and non-optical data storage devices. The computer readable media can include computer readable tangible medium distributed over a network-coupled computer system so that the computer readable code is stored and executed in a distributed fashion.

[0135] Although the method operations were described in a specific order, it should be understood that other housekeeping operations can be performed in between operations, or operations can be adjusted so that they occur at slightly different times, or can be distributed in different order, or distributed across different systems, without departing from the scope of the present application. Additionally, the description above expressly sets forth neither the essential nor the

[0136] While the foregoing application has been described in some detail for purposes of clarity and the specific embodiments, it will be apparent to those skilled in the art that certain changes and modifications can be practiced within the scope of the appended claims. Therefore, the present implementations should not be limited to the specific implementation described, but should be given broad coverage, within the metes of the following claims as interpreted in the light of this specification.

[0137] An example overall system architecture of a game console is described herein. The example game console can include Playstation (PS3) or Playstation (PS4) entertainment device, which may be compatible with a controller for implementing embodiments of the present invention. Although the PS3 game console system architecture is explained in detail, it should be noted that the various implementations described herein may be extended to the system architecture of different game consoles or computing devices. A system unit is provided to which various peripheral devices may be connected. The system unit is similar to Figure 1 The cloud gaming system 300 includes a processor, such as an 8-core processor in a PS3, or a multi-core processor in a PS4; Dynamic random access memory (XDRAM) units, or graphics dynamic random access memory such as GDDR5 as in the PS4; a real compositor graphics unit (e.g., a 550MHz GPU) with dedicated video random access memory (VRAM) units as in the PS3, or an 800 or 900MHz GPU with shared graphics memory in the PS4 and PS4 Pro; and an I / O bridge. The system unit also includes Blu-ray Disc for reading from disks and a removable slot-mounted hard disk drive (HDD). Disk Read Only Memory (BD- ) (Optical) disk reader, which can be accessed through the I / O bridge. Optionally, the system unit also includes a device for reading Compact Flash cards, Memory A memory card reader for memory cards and the like, which can similarly be accessed through the I / O bridge, and a built-in DVR is included in the PS4 for recording games.

[0138] The I / O bridge also connects to six Universal Serial Bus (USB) 2.0 ports; a Gigabit Ethernet port; an IEEE 802.11b / g wireless networking (Wi-Fi) port; and a port capable of supporting up to seven Bluetooth connections. Wireless link port.

[0139] In operation, the I / O bridge handles all wireless, USB and Ethernet data, including data from one or more game controllers, such as the DualShock 3 controller in the PS3 or the PS4 DualShock 4 controller in the PS4. For example, when a user is playing a game, the I / O bridge receives data from the game controller via a Bluetooth link and directs it to the processor (in the PS3) or multi-core processor (in the PS4), which updates the current state of the game accordingly. In addition, other image and motion sensors provide data captured by the user during gameplay to the I / O bridge, which directs it to the corresponding processor. The game controller (e.g., the game controller of the PS4) includes a share button option for sharing the game, a clickable touchpad, a rechargeable battery (lithium-ion or other type), etc.

[0140] In addition to the game controller, the wireless, USB and Ethernet ports provide connectivity for other peripheral devices, such as: a remote control; a keyboard; a mouse; a portable entertainment device, such as Sony Playstation Portable; a video camera, such as Sony PlayStation Eye; a microphone headset; and a removable hard drive. Thus, such peripheral devices can in principle be connected wirelessly to the system unit; for example, the portable entertainment device can communicate via an ad hoc Wi-Fi connection, while the microphone headset can communicate via a Bluetooth link.

[0141] Providing these interfaces means that the Playstation 3 device can also be compatible with other peripheral devices, such as a digital video recorder (DVR), a set-top box, a digital camera, a portable media player, a voice-over-IP telephone, a mobile phone, a printer and a scanner.

[0142] In addition, a legacy memory card reader can be connected to the system unit via the USB port, enabling reading of memory cards of the type used by or the device.

[0143] In the present implementation, the game controller is operable to communicate wirelessly with the system unit via a Bluetooth link. However, the game controller can instead be connected to a USB port, thereby also providing power for charging the battery of the game controller. In addition to one or more analog joysticks and conventional control buttons, the game controller is also sensitive to six degrees of freedom of motion, corresponding to translation and rotation in each axis. Thus, in addition to or instead of conventional button or joystick commands, gestures and movements of the user's hands of the game controller can be translated into inputs for the game. Optionally, other wireless-enabled peripheral devices, such as the Playstation TM ​The portable device) can be used as a controller. In the case of the Playstation TM In the case of the portable device, additional game or control information (e.g. control instructions or the number of lives) can be provided on the screen of the device. Other alternative or supplementary control devices can also be used, such as a dance mat (not shown), a light gun (not shown), a steering wheel and pedals (not shown) or a custom controller, such as a single or several large buttons for a fast response quiz game (also not shown).

[0144] The remote control is also operable to communicate wirelessly with the system unit via a Bluetooth link. The remote control comprises controls suitable for Blu Ray TM the operation of the disc BD-ROM reader and navigation of the disc content.

[0145] In addition to conventional pre-recorded and recordable compact discs (CDs) and so-called Super Audio CDs, Blu Ray TM The disc BD-ROM reader is also operable to read compact disc read-only memories (CD-ROMs) compatible with the Playstation and the Playstation 2 device. In addition to conventional pre-recorded and recordable DVDs, Blu Ray TM The disc BD-ROM reader is also operable to read digital versatile disc read-only memories (DVD-ROMs) compatible with the Playstation 2 TM and the Playstation 3 TM device. In addition to conventional pre-recorded and recordable DVDs, Blu Ray TM The disc BD-ROM reader is further operable to read BD-ROMs compatible with the Playstation 3 device, as well as conventional pre-recorded and recordable Blu Ray discs.

[0146] The system unit is operable to provide audio and video generated or decoded by the Playstation 3 or Playstation 4 device via the real synthesizer graphics unit to a display and sound output device, such as a monitor or television set with a display and one or more loudspeakers, through audio and video connectors. The audio connectors can comprise conventional analog and digital outputs, while the video connectors can variously comprise component video, S-video, composite video and one or more high-definition multimedia interface (HDMI) outputs. Thus, the video output can be in formats such as PAL or NTSC, or 720p, 1080i or 1080p high definition, as well as 4K, HDR, etc.

[0147] Audio processing (generation, decoding, etc.) is performed by a processor. For example, the operating system of the Playstation 3 device supports surround sound, Dolby Digital Plus, DTS-HD Master Audio, and Dolby True-HD are Dolby Digital Plus, DTS-HD Master Audio, and Dolby True-HD are

[0148] In the present implementation, the video camera includes a CMOS (complementary metal-oxide semiconductor) image sensor (but a charge-coupled device (CCD) image sensor can also be used), an LED indicator, and a hardware-based real-time data compression and encoding device so that compressed video data can be transmitted in a suitable format, such as based on the MPEG (Motion Picture Experts Group) standard within the image, for decoding by the system unit. The camera LED indicator is arranged to illuminate in response to suitable control data from the system unit, for example to indicate adverse lighting conditions. The implementation of the video camera can be connected to the system unit in various ways through a USB, Bluetooth, or Wi-Fi communication port. The implementation of the video camera can include one or more associated microphones and also be able to transmit audio data. In the implementation of the video camera, the CCD can have a resolution suitable for high-definition video capture. In use, the images captured by the video camera can be incorporated into a game, for example, or interpreted as game control inputs.

[0149] In general, in order to have successful data communication with a peripheral device such as a video camera or remote control through one of the communication ports of the system unit, suitable software such as a device driver should be provided. Device driver technology is well known and will not be described in detail here except to say that those skilled in the art will appreciate that a device driver or similar software interface can be required in the described present implementation.

[0150] The processor has an architecture that includes four basic components: an external input and output fabric that includes memory controllers and a dual bus interface controller; a main processor, referred to as a power processing element; eight co-processors, referred to as synergistic processing elements (SPEs); and a ring data bus that connects the above components, referred to as an element interconnect bus. The total floating point performance of the processor is 218 GFLOPS compared to 6.2 GFLOPs for the Emotion Engine of the Playstation 2 device.

[0151] The Power Processing Element (PPE) is based on a bi-directional simultaneous multithreaded Power- compatible PowerPC core (PPU) running at an internal clock of 3.2 GHz. It includes a 512 kB level 2 (L2) cache and a 32 kB level 1 (L1) cache. The PPE is capable of eight unit operations per clock cycle, which translates to 25.6 GFLOPs at 3.2 GHz. The primary role of the PPE is to act as a controller for the synergistic processing elements, which handle most of the computational workload. In operation, the PPE maintains a job queue, schedules jobs for the synergistic processing elements and monitors their progress. As a result, each synergistic processing element runs a kernel whose role is to fetch a job, execute the job and synchronize with the PPE.

[0152] Each synergistic processing element (SPE) includes a corresponding synergistic processing unit (SPU), and a corresponding memory flow controller (MFC), which in turn includes a corresponding dynamic memory access controller (DMAC), a corresponding memory management unit (MMU) and a bus interface (not shown). Each SPU is a RISC processor running at 3.2 GHz clock and includes 256 kB of local RAM (which in principle can be extended to 4 GB). Each SPE gives a theoretical 25.6 GFLOPS of single precision performance. The SPU can operate on 4 single precision floating point members, 4 32-bit numbers, 8 16-bit integers or 16 8-bit integers in a single clock cycle. In the same clock cycle, it can also perform a memory operation. The SPU does not access system memory XDRAM 1426 directly; the 64-bit address formed by the SPU is passed to the MFC, which instructs its DMA controller to access memory via the element interconnect bus and the memory controller.

[0153] The element interconnect bus (EIB) is a logical looped communication bus within the processor that connects the above-mentioned processor elements, i.e. the PPE, the memory controller, the dual bus interface and the 8 SPEs, a total of 12 participants. The participants can simultaneously read and write to the bus at a rate of 8 bytes per clock cycle. As mentioned previously, each SPE includes a DMAC for scheduling longer read or write sequences. The EIB includes four lanes, two in a clockwise direction and two in a counter-clockwise direction. Thus, for twelve participants, the longest step-wise data flow between any two participants is six steps in the appropriate direction. Thus, the theoretical peak instantaneous EIB bandwidth is 96B per clock for 12 slots, which equals a theoretical peak bandwidth of 307.2 GB / s (gigabytes per second) at a clock rate of 3.2 GHz.

[0154] The memory controller includes an XDRAM interface developed by Rambus Incorporated. The memory controller interfaces with Rambus XDRAM 1426 with a theoretical peak bandwidth of 25.6 GB / s.

[0155] The dual bus interface includes a Rambus System Interface. This interface is organized into 12 lanes, each 8 bits wide, with five of the paths being inbound and seven outbound. This provides a theoretical peak bandwidth of 62.4 GB / s between the processor and the I / O bridge via the controller and between the processor and the real synthesizer graphics unit via the controller (36.4 GB / s outbound, 26 GB / s inbound).

[0156] Data sent by the processor to the real synthesizer graphics unit will typically include a display list, which is a series of commands to draw vertices, apply textures to polygons, specify lighting conditions, etc.

[0157] Embodiments can include capturing depth data to better identify real-world users and guide the activities of avatars or scenes. The object can be something held by a person, or it can also be the person's hand. In this specification, the terms "depth camera" and "three-dimensional camera" refer to any camera that is capable of obtaining distance or depth information in addition to two-dimensional pixel information. For example, a depth camera can utilize controlled infrared illumination to obtain distance information. Another example depth camera can be a stereo camera pair that uses two standard cameras to triangulate distance information. Similarly, the term "depth sensing device" refers to any type of device that is capable of obtaining distance information in addition to two-dimensional pixel information.

[0158] Recent advances in three-dimensional imaging open the door to increased possibilities for real-time interactive computer animation. In particular, in addition to conventional two-dimensional video imaging, new "depth cameras" provide the ability to capture and render in three dimensions. With the new depth data, embodiments of the present invention allow computer-generated objects to be placed in real time at various locations within a video scene, including behind other objects.

[0159] Moreover, embodiments of the present application provide users with a real-time interactive gaming experience. For example, users can interact in real-time with various computer-generated objects. In addition, video scenes can be altered in real-time to enhance the user's experience. For example, computer-generated clothing can be inserted over a user's clothing and computer-generated light sources can be used to cast virtual shadows within a video scene. Thus, using embodiments of the present application and a depth camera, users can experience an interactive environment in their own living room. Similar to a regular camera, a depth camera captures two-dimensional data for a plurality of pixels that make up a video image. These values are color values for the pixels, typically red, green, and blue (RGB) values for each pixel. In this way, objects captured by the camera are displayed as two-dimensional objects on a monitor.

[0160] Embodiments of the present application also contemplate a distributed image processing configuration. For example, the present application is not limited to image capture and display image processing occurring in one or even two locations, such as in a CPU or in a CPU and one other element. For example, input image processing can easily be performed in the relevant CPU, processor, or device that can perform the processing; essentially all image processing can be distributed throughout an interconnected system. Thus, the present application is not limited to any particular image processing hardware circuitry and / or software. Embodiments described herein are also not limited to any particular combination of general purpose hardware circuitry and / or software, or any particular source of instructions executed by processing components.

[0161] With the above embodiments in mind, it should be understood that the application can employ various computer-implemented operations involving data stored in computer systems. These operations are those requiring physical manipulation of physical quantities. Usually, though not necessarily, these quantities take the form of electrical, magnetic, or optical signals capable of being stored, transferred, combined, compared, and otherwise manipulated. Further, the manipulations performed are often referred to in terms, such as producing, identifying, determining, or comparing.

[0162] Any of the operations described herein that form part of the application are useful machine operations. The application also relates to a device or an apparatus for performing these operations. The apparatus can be specially constructed for the required purposes, or it can be a general purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general purpose machines can be used with computer programs written in accordance with the teachings of the present application, or it can be more convenient to construct a more specialized apparatus to perform the required operations.

[0163] The foregoing invention can also be implemented in other computer system configurations, including hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers and the like. The invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.

[0164] The present invention can also be embodied as computer readable code on a computer readable medium. The computer readable medium is any data storage device that can store data which can thereafter be read by a computer system. Examples of the computer readable medium include hard drives, network attached storage devices (NAS), read-only memory, random-access memory, CDs, DVDs, magnetic tape and other optical and non-optical data storage devices. The computer readable medium can also be distributed over a network of computers or other devices on a distributed computing network, so that the computer readable code is stored and executed in a distributed fashion.

[0165] While the foregoing invention has been described in some detail for purposes of clarity and the specific embodiments described are shown by way of example and not limitation, it will be apparent to those ordinarily skilled in the art that various changes and modifications can be practiced that are within the scope of the appended claims. Therefore, the embodiments of this invention are to be considered as illustrative and not restrictive, and the invention is not to be limited to the details given herein, but can be modified within the scope and equivalents of the appended claims.

[0166] It should be understood that various features of the herein disclosed can be combined or assembled into particular implementations of the various embodiments defined herein. Therefore, the examples provided are merely examples and are not limiting as to what can become the various implementations of the embodiments defined herein by the combination of the various elements. In some examples, some implementations can include fewer elements than the combination of various elements defined herein, without departing from the spirit of the disclosed or equivalent implementations.

[0167] Embodiments of the present disclosure can be practiced with various computer system configurations, including hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers and the like. The embodiments of the present disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wired or wireless network.

[0168] With the above embodiments in mind, it should be understood that the embodiments of the disclosure can employ various computer-implemented operations that are described in terms of the operations of a computer system. The operations are those requiring physical manipulations of physical quantities. Any of the operations described herein that form part of the embodiments of the disclosure are useful machine operations. The embodiments of the disclosure are also related to the apparatus or devices used to perform the operations. These devices can be specially constructed for the required purposes, or they can be selectively activated or configured from a general purpose computer selectively activated or configured by a computer program stored in the computer. Specifically, various general purpose machines can be used with computer programs written in accordance with the teachings herein, or it can be more convenient to construct a more specialized apparatus to perform the required operations.

[0169] The present disclosure can also be embodied as computer readable code on a computer readable medium. The computer readable medium is any data storage device that can store data which can thereafter be read by a computer system. Examples of the computer readable medium include a hard drive, network attached storage (NAS), read-only memory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical and non-optical data storage devices. The computer readable medium can include computer-readable tangible medium distributed over a network-coupled computer system so that the computer readable code is stored and executed in a distributed fashion.

[0170] Although the method operations were described in a specific order, it should be understood that other housekeeping operations can be performed in between operations, or operations can be adjusted so that they occur at slightly different times, or can be distributed in different order, or distributed across different systems, without departing from the spirit of the process. Likewise, the various tasks can be distributed amongst numerous computer systems.

[0171] While the foregoing disclosure has been described in some detail for purposes of clarity and the specific embodiments of the invention have been described, it is not intended to limit the application to the described embodiments or as having an exclusive use of all the features that are described or that have been set forth in the accompanying claims. Various alternatives, modifications, additions and equivalents can be used. Therefore, the foregoing disclosure in its entirety is to be interpreted no more narrowly than as is apparently indicated by the specification and claims.

Claims

1. A method for processing data of an online game, the method comprising: receiving, from a server device, a frame of data generated for an online game, the frame of data representing a current state of the online game; encoding the frame of data using an encoder, the encoded frame of data being transmitted over a network to a client device for rendering; evaluating the data contained in the encoded frame to determine an amount of data contained in the encoded frame that is transmitted to the client device; when the amount of data in the encoded frame exceeds a maximum data limit specified for a communication channel established between the server device and the client device, determining a number of subsequent frames from which to reduce data, wherein the subsequent frames are frames that follow the encoded frame that exceeds the maximum data limit, sending a signal to the encoder to dynamically reduce an amount of data from subsequent frames that are forwarded to the client device for rendering, wherein a sum of the reduced amount of data for the number of subsequent frames is the amount of data in the encoded frame that exceeds the maximum data limit; and performing the reduction to maintain a frame rate defined for the online game.

2. The method of claim 1, wherein dynamically reducing the amount of data in the subsequent frames comprises, determining the number of subsequent frames for which to reduce an amount of data; and specifying a reduced maximum data limit for data to be included in each of the number of subsequent frames, the reduced maximum data limit used to encode the subsequent frames.

3. The method of claim 2, wherein the reduced maximum data limit is determined based on an extent to which the data in the encoded frame exceeds the maximum data limit.

4. The method of claim 2, wherein the number of subsequent frames for which the reduced maximum data is to be implemented depends on a type of online game.

5. The method of claim 2, wherein the number of subsequent frames for which the reduced maximum data is to be implemented depends on data requirements of the online game.

6. The method of claim 2, wherein the reduced maximum data limit is set based on an available bandwidth of the communication channel.

7. The method of claim 2, wherein dynamically reducing the amount of data comprises performing compression on the data included in the frames using a compression technique, wherein a level of compression is dictated by the compression technique, wherein the compression technique follows lossy compression or lossless compression.

8. The method of claim 2, wherein dynamically reducing the amount of data in the subsequent frames comprises, identifying data pertaining to particular portions of the subsequent frames that are to be retained and other particular portions of the subsequent frames that are to be discarded; and encoding the subsequent frames using the data pertaining to the particular portions that are to be retained.

9. The method of claim 1, wherein dynamically reducing further comprises, after completing the dynamic data reduction of the number of subsequent frames, continuing to encode frames of data for the online game at the maximum data limit. ​ ​ 10. The method of claim 1, wherein the maximum data limit is defined based on an available bandwidth of the communication channel established between the client device and the server device.

11. The method of claim 1, wherein the dynamic reduction in the subsequent frames is done after the frame that exceeded the maximum data limit has been forwarded through the network to the client device.

12. The method of claim 1, wherein the dynamic reduction includes processing the data of each of the subsequent frames of the online game before forwarding the data of each of the subsequent frames to the encoder for encoding.

13. The method of claim 12, wherein the processing of the data of subsequent frames includes, evaluating the data included in the subsequent frame to identify portions of data that are critical to constructing an image included in the subsequent frame and other portions of data that are not critical, the evaluating to identify critical portions being based on a contextual relationship of content contained in the subsequent frame to a user of the client device; and forwarding the data of the subsequent frame and results of the evaluating to the encoder, wherein the encoder uses the results of the evaluating to focus on encoding portions of data in the subsequent frame that are identified as critical to the user.

14. A system for processing data of an online game, the system comprising: a server configured to execute an instance of the online game and generate data frames for transmission to a client device; an encoder configured to receive the data frames of the online game and encode the data frames according to a maximum data limit specified for a communication channel; the communication channel established between the server executing the online game and the client device, the communication channel for transmitting encoded data frames for rendering; and a monitoring system configured to monitor an amount of data included in each of the encoded data frames and, when the amount of data included in a particular frame exceeds the maximum data limit, determine a number of subsequent frames from which to reduce data, wherein the subsequent frames are frames after the particular frame that exceeded the maximum data limit, and generate a signal to dynamically reduce the amount of data from the subsequent frames, wherein a sum of the reduced amount of data for the number of subsequent frames is the amount of data that the particular frame exceeded the maximum data limit.

15. The system of claim 14, wherein the monitoring system comprises, a bit rate monitor configured to inspect each frame to determine an amount of data included in each frame encoded by the encoder to continue transmission to the client device and, when the particular frame includes data that exceeds the maximum data limit specified for the communication channel, interact with a debt tracker; the debt tracker configured to track the excessive amount of data included in the particular frame and a number of subsequent frames generated after the particular frame for which data needs to be dynamically reduced; and a bit rate adjuster configured to receive from the bit rate monitor an indication of the excess data included in the particular frame transmitted to the client device and responsively instruct the encoder to dynamically reduce the amount of data included in the number of subsequent frames to a reduced maximum data limit.

16. The system of claim 15, wherein the reduced maximum data limit is determined based on the needs of the online game.

17. The system of claim 15, wherein the encoder is configured to, receive from the bit rate adjuster a first signal indicating an excess amount of data included in the particular frame, the first signal specifying the reduced maximum data limit for data of each subsequent frame and the number of subsequent frames for which the amount of data must be reduced; and reduce the data identified as included in each subsequent frame according to the reduced maximum data limit specified in the first signal from the bit rate adjuster.

18. The system of claim 15, wherein the number of subsequent frames is determined based on the amount by which the particular frame exceeds the maximum data limit.

19. The system of claim 15, wherein the bit rate monitor interacts with the debt tracker to obtain the number of subsequent frames following the particular frame for which the amount of data needs to be dynamically reduced, the number of subsequent frames being dynamically updated by the bit rate monitor after each subsequent frame is transmitted to the client device, and the bit rate adjuster interacts with the bit rate monitor to obtain a current number of subsequent frames for which data needs to be reduced and issues a first signal to the encoder to dynamically reduce data in the subsequent frames to be encoded for transmission to the client device when the current number of subsequent frames is non-zero.

20. The system of claim 19, wherein the debt tracker includes a frame counter for tracking the current number of subsequent frames for which the data needs to be dynamically reduced.

21. The system of claim 20, wherein the bit rate monitor is further configured to provide an indicator to the bit rate adjuster when the frame counter reaches zero; and the bit rate adjuster is further configured to send a second signal to the encoder to continue encoding the subsequent frames with data at the maximum data limit.

22. The system of claim 15, wherein the encoder is further configured to, determine whether an amount of data included in a subsequent frame of the subsequent frames received after the particular frame exceeds the reduced maximum data limit; when the subsequent frame exceeds the reduced maximum data limit, evaluate the data of the subsequent frame to identify a portion of the data that is critical to constructing an image included within the subsequent frame and discard other data within the frame that is not critical to constructing the image.

23. The system of claim 14, the system further comprising a frame data processor configured to, receive a frame of data from the online game; analyze the data contained in the frame to understand a complexity of the data contained in the frame; and providing information to the encoder related to the complexity of the data contained in the frame, wherein the information is used by the encoder to determine specific data portions to retain in the frame and other specific data portions to discard in the frame.

24. The system of claim 14, further comprising a frame data processor configured to, receive a data frame from the online game; analyze the data contained in the frame to understand the complexity of the data contained in the frame; and adjust the data contained in the frame to reduce the complexity of the frame, the adjusted data frame being forwarded to the encoder for encoding.

25. The system of claim 24, wherein the adjusted data frame is used by the encoder to determine specific data portions to retain and other specific data portions to discard.

26. The system of claim 14, wherein the maximum data limit is defined based on available bandwidth of the communication channel established between the client device and the server.

27. A method for processing data of an online game, the method comprising: receiving a data frame generated for an online game from a server device, the data frame representing a current state of the online game; verifying whether a previously encoded data frame exceeds a maximum data limit specified for a communication channel established between the server device and a client device; when determining that the previously encoded data frame exceeds the maximum data limit, determining a number of subsequent frames from which to reduce data, wherein the subsequent frames are frames subsequent to the previously encoded data frame that exceeds the maximum data limit, dynamically reducing an amount of data from the subsequent frames prior to encoding the subsequent frames to generate an encoded data frame, wherein a sum of the reduced amount of data for the number of subsequent frames is an amount of data that the previously encoded data frame exceeds the maximum data limit; and forwarding the encoded data frame to the client device over the communication channel.

28. The method of claim 27, wherein the verifying comprises, querying a debt counter maintained for a current session of the online game for a number of frames generated subsequent to the previously encoded frame that need to be dynamically reduced and an amount that the previously encoded data frame exceeds the maximum data limit; and when the number of frames that need to be dynamically reduced is non-zero, providing a signal to the encoder to dynamically reduce the amount of data included in the frames.

29. The method of claim 28, wherein when the number of frames that need to be dynamically reduced is zero, providing a second indicator to the encoder to continue encoding the data frames at the maximum data limit.

30. The method of claim 27, wherein the data included in the frames is dynamically reduced when the amount of data contained within the frames is verified to exceed the reduced maximum data limit. ​

Images