Method and system for maintaining a smooth frame rate during transmission of streaming video content - Patents.com

Abstract

The method and system receive frames of data generated for an online game from a server device and encode the frames using an encoder. Each encoded frame of data is transmitted over a network to a client device for rendering. A monitoring engine is used to evaluate the data contained in the encoded frame and determine whether the amount of data contained in the encoded frame exceeds a maximum data limit specified for a communication channel established between the server device and the client device. When the amount of data contained in an encoded frame exceeds the maximum data limit, a signal is sent to the encoder to dynamically reduce the amount of data contained in one or more subsequent frames transferred to the client device for rendering. The adjustment is performed to maintain a specified frame rate for the online game.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE This disclosure relates to streaming video content, and more particularly, to methods and systems for maintaining a smooth frame rate during the transmission of streaming video content. [Background technology]

[0002] As mobile devices become more popular, the demand for streaming content, such as movies, videos, entertainment content, and live gaming content, for user consumption has increased exponentially. Providing high quality video content with minimal latency is essential to maintain full user attention. To provide high quality video content to client devices with minimal latency, a transmission pipeline with a significant amount of bandwidth must be available to transfer the streaming content. Furthermore, streaming content (e.g., video data) must be transmitted at a steady frame rate so that the content can be rendered smoothly and continuously at the client device. Typically, the bandwidth for transmitting streaming content over a network, such as the Internet, is defined (i.e., set) by an Internet Service Provider.

[0003] Streaming content, such as video, captures thousands of individual images in a video sequence. These images are captured into frames of data. The frequency with which these frames are presented within a given time represents the frame rate. The frame rate affects how a user perceives motion in the video content. If there is a frame rate inconsistency caused by dropping or missing frames for one or more frames, some of the video detail may be lost and the video may appear choppy. If the dropped frames are minimal and infrequent, the user's perceived quality of the video will not be significantly affected. However, if the dropped frames become frequent, a noticeable hitch may be perceived on the screen.

[0004] Furthermore, as technology improves over time, so do the quality and frame rates of the videos produced. Higher frame rates are used by live streaming content providers (such as broadcasters) to provide a more in-depth look at the fast-paced action. Examples of such fast-paced action include live sports such as soccer, football, and especially intense online gaming. The increased video resolution and frame rates put a significant strain on bandwidth requirements.

[0005] Content is encoded using an encoder before transmission. Encoders used to encode content before transmission over the Internet can be broadly categorized into two main types: hardware and software. For example, a hardware encoder is a physical computing appliance / device that is configured to connect directly to a network such as the Internet, encode video content, and transmit the encoded video content over the network. A software encoder is a program / programming logic that runs on a computer, establishes a network connection, encodes the video content, and pushes the encoded video content out to the network.

[0006] Regardless of the type of encoder used, the encoder is instructed to adhere to a maximum size for each frame of data when encoding the data, which is determined by the available bandwidth of the communication channel used to transmit the video data. However, encoders do not strictly adhere to the specified maximum size when generating frames. As a result, the encoder may sometimes generate frames that are larger than specified. For example, an encoder may generate an I-frame that is many times larger than the specified maximum size. The traditional way to handle such large sized frames is to process, encode, and transmit the large frame, and then drop one or more subsequent frames to allow for the transmission of the large frame, thereby completely ignoring the maximum bit rate specified for the communication channel connection. As mentioned above, dropping a single frame occasionally may not affect the overall user experience, but dropping frames frequently will cause a noticeable problem on the screen. This noticeable problem may affect the user's overall experience.

[0007] It is within this context that the embodiments of the present disclosure have been made. Summary of the Invention

[0008]

[0003] Embodiments of the present disclosure relate to a system and method for processing data for an online game or interactive application that generates a steady stream of content. For example, a game engine available on a remote server computing device (or simply referred to herein as a "server") executes an online game selected by a user and generates streaming content. User inputs made in response to the streaming content are used to affect a game state of the online game. The streaming content is encoded using an encoder and transmitted to a client device for rendering. The streaming content includes a current state of the online game. Data contained in each encoded frame is evaluated to determine whether the data contained in the encoded frame exceeds a maximum content threshold limit established for a communication channel established between the remote server and the client device. When the data of an encoded frame exceeds the predefined maximum threshold limit, a counter, such as a "sent debt" (SD) counter, is incremented. The sent debt counter (SDC) identifies the amount of data contained in a frame that exceeds the threshold limit established for the communication channel and the number of frames that need to be reduced to make up for the sent debt. The transmission debt counter is consulted when the encoder receives a subsequent frame of content for encoding, and subsequent frames of data are generated after the frame that exceeded the threshold limit. If the transmission debt counter is non-zero, the encoder is instructed to reduce the amount of data included in the subsequent frame to below a predefined reduced maximum data limit. The reduced maximum threshold limit of the subsequent frame(s) may be caused by the amount by which the first frame exceeded the threshold limit, and / or the bandwidth of the communication channel, and / or the demands of an online game. The reduction in the amount of data included in the subsequent frames continues as long as the transmitted debt is not zero.Once the transmitted debt has been replenished (i.e., the transmitted debt becomes zero), the encoder is instructed to resume encoding subsequent frames received after the frame of data in which the transmitted debt was incurred, at a normal rate determined by a predefined maximum threshold limit.

[0009] When a subsequent frame is received, the encoder can evaluate the amount of data included in the subsequent frame and dynamically reduce the amount of data of the subsequent frame to a reduced maximum data limit specified for the online game based on the signal received at the encoder. The reduced maximum data limit may be predetermined based on the type of online game selected for game play. When evaluating the subsequent frame based on the signal received at the encoder, the encoder may determine that the amount of data included in the subsequent frame is less than the reduced maximum data limit. In such a case, the encoder just proceeds to encode the subsequent frame without changing the amount of data included in the subsequent frame because the amount of data of the subsequent frame is well within the reduced maximum data limit.

[0010] By instructing the encoder to include an appropriate amount of data in each frame sent to the client device, a smooth and consistent frame rate can be maintained. Maintaining the frame rate results in the client device rendering the associated content of the online game without any disruption to the rendering. The encoder can reduce the amount of data in the subsequent frame(s) by identifying and retaining certain data that is important to the rendering of the image(s) in the subsequent frame, and discarding other certain data in the subsequent frame that is not important. Alternatively, the encoder may use compression techniques to reduce the data.

[0011] In some cases, a frame data processor may be used to process each frame of data generated by the online game (i.e., the frame data source) before forwarding the processed frame of data to an encoder for encoding. Processing the data may be a matter of determining the complexity of the data contained in the frame, so that the encoder can make an informed decision as to which portion(s) of the processed data to keep and which to discard.

[0012] In one embodiment, a method is disclosed. The method includes receiving a frame of data generated for an online game from a server device. The frame of data represents a current state of the online game. The frame of data is encoded using an encoder. The encoded frame of data is transmitted over a network to a client device for rendering. Data included in the encoded frame is evaluated to determine an amount of data included in the encoded frame to be transmitted to the client device. When an amount of data of the encoded frame exceeds a maximum data limit specified for a communication channel established between the server device and the client device, a signal is sent to the encoder to dynamically reduce an amount of data included in one or more subsequent frames transferred to the client device for rendering. The adjustment is performed to maintain a defined frame rate for the online game.

[0013] In another embodiment, a system for use in processing online game data is disclosed. The system includes a server configured to execute an instance of the online game and generate frames of data for transmission to a client device. The system also includes an encoder configured to receive the frames of online game data and encode the frames of data according to a maximum data limit specified for a communication channel established between the server and the client device. The server includes a monitoring system configured to monitor the amount of data included in each of the frames and generate a signal to the encoder to dynamically reduce the amount of data included in the one or more subsequent frames when the amount of data included in a particular frame that precedes one or more subsequent frames exceeds the maximum data limit.

[0014] In another alternative embodiment, a method is disclosed. The method includes receiving a frame of data generated for an online game from a server device. The frame of data represents a current state of the online game. The method further includes checking whether a previously encoded frame of data exceeds a maximum data limit specified for a communication channel established between the server device and the client device. If it is determined that the previously encoded frame exceeds the maximum data limit, dynamically reducing an amount of data included in the frame to the reduced maximum data limit before encoding the frame to generate an encoded frame. The encoded frame of data is transferred to the client device via the communication channel.

[0015] Other aspects of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the present disclosure.

[0016] The disclosure may be best understood by referring to the following description taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0017] [Figure 1] 1 illustrates a simplified material view of a cloud gaming system used to maintain smooth frame rates for streaming interactive applications, such as online streaming games, according to one embodiment of the present disclosure. [Diagram 2] 2A shows a simplified block diagram of different components of a server device used to provide a client device with a stable frame rate of online game content generated during gameplay of an online game according to one embodiment of the present disclosure, and FIG 2B shows a simplified block diagram of different components of a server device used to provide a client device with a stable frame rate of online game content generated during gameplay of an online game according to another embodiment of the present disclosure. [Diagram 3] 3A shows a simplified block diagram of various components of a monitor engine used to identify the amount of data an encoder needs to include in each frame of data sent to a client device, according to one embodiment of the present disclosure, and FIG. 3B shows a simplified block diagram of different components of a frame data processor used to condition the data provided to the encoder for encoding, according to one embodiment of the present disclosure. [Figure 4] 1 shows a flow diagram of a simple process used to instruct an encoder as to the amount of data to include in each frame generated by a frame source, according to one embodiment of the present disclosure. [Diagram 5] This shows a simple example of using a maximum bit rate to adjust the content of various frames generated by a frame source (such as an online gaming application or an online interactive application) and provide consecutive frames to a client in one embodiment. [Figure 6] 4 illustrates an operational flow for a method of processing a frame of data for onward transmission to a client device, according to one embodiment. [Figure 7]1 illustrates a simplified block diagram of an exemplary interactive application system, such as a gaming system, in accordance with various embodiments of the present invention. [Figure 8] 1 illustrates an exemplary information service provider architecture for delivering information content and services to geographically distributed and connected users over a network in accordance with one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Although the following detailed description contains many specific details for purposes of illustration, those skilled in the art will appreciate that many variations and modifications to the following details are within the scope of the disclosure. Accordingly, the aspects of the disclosure described below are presented without loss of generality to, and without imposing limitations on, the claims that follow this description.

[0019] Generally speaking, various embodiments of the present disclosure describe systems and methods for evaluating the amount of data included in each frame of data generated by an online streaming application (e.g., an online gaming application) and when the amount of data included in the frame exceeds a maximum data limit specified for the online game and dynamically reduces the amount of data for one or more subsequent frames. The maximum data limit may be based on a maximum bit rate that can be handled by a communication connection established between a server device generating the frame of data and a client device requesting the data by selecting the streaming application. The amount of data included in each subsequent frame may be dynamically reduced by compressing the data using a compression algorithm, selectively discarding portions of the data that are not important to the rendering of the image(s) included in the frame, or by other means via commonly available or specialized algorithms.

[0020] Traditional methods of handling data frames include detecting large data frames that exceed a specified maximum data limit and, accordingly, dropping one or more subsequent data frames to track the maximum bit rate of the communication connection. This can lead to inconsistent frame rates and choppy video when frames are dropped too frequently.

[0021] On the other hand, various embodiments discussed herein include a system configured to detect a large frame, and instead of dropping one or more subsequent frames, the system is configured to instruct an encoder to adjust the bitrate so that one or more subsequent frames can make up for the transmitted debt caused by the transmission of the excess data. The system may detect the excess bitrate transmitted in a frame and treat the excess data as an "overdraw bitrate" account. The system may make up for the extra data transmitted in a frame by reducing data from subsequent frames following the "overdrawn" frame. A monitoring engine in the system is configured to track the amount of excess data transmitted in the frame that caused the bitrate overdraw, determine a number of subsequent frames to make up for the excess data, and provide instructions to the encoder to dynamically reduce the amount of data for the determined number of subsequent frames. The number of subsequent frames from which data needs to be dynamically reduced may be determined based on the amount of excess data contained in the frame and / or the demand for online games, or the type of online games, etc.

[0022] A "transmitted debt" counter is used to track the amount of excess data contained in a frame and the number of subsequent frames used to make up for it. After each frame is transferred to the client device, the transmit debt counter is updated. For each frame received by the encoder, the monitor engine is used to query the transmit debt counter to determine if it is zero or non-zero (i.e., not zero). If the transmit debt counter is not zero, the monitor engine instructs the encoder to adjust the amount of data of the frame currently being received by the encoder for encoding and to transmit the adjusted encoded frame to the client device. On the other hand, if the transmit debt counter is zero, the monitor engine instructs the encoder to resume encoding of the current frame using the normal bitrate (i.e., maximum data limit) defined for the online game.

[0023] The content generated by online games is video content. The amount of data contained in each frame defines the bitrate, which is measured in megabits per second (Mbps). The bitrate depends in part on the resolution of the video content. To provide a realistic gaming experience, the game logic may be configured to provide high-resolution video frames. High-resolution video frames contain a greater amount of data (i.e., a higher bitrate) as opposed to low-resolution video frames that contain a smaller amount of data. Typically, highly compressed video data contains less data (i.e., a lower bitrate) than low-compressed or uncompressed video data. As a result, the compressed video data has a lower video quality (i.e., a lower video resolution). Generally speaking, for good quality video content, a bitrate of 40 Mbps is recommended, which is for uncompressed or very lightly compressed video. The bitrate is determined according to the bandwidth requirements of the communication channel used to transmit the video content.

[0024] Video data is typically transmitted at a frame rate defined for online games. The frame rate represents the speed at which the video data is played on the receiving device, which varies depending on the type of online game being played. More frames per second results in a smoother rendered video. Increasing the frame rate increases the bit rate, as more data must be provided in the frames. Thus, a frame rate may be defined for an online game based on the level of smoothness expected / defined for the online game. As the frame rate also impacts the bit rate, the frame rate may be defined according to the available bandwidth of the communication channel used to transmit the video content to the client device. Typically, frame rates are standardized across content generating industries and are defined as approximately 30 frames per second, 60 fps, or 120 fps.

[0025] The monitoring engine provides the client device with a smooth and consistent frame rate of the content of the interactive streaming application and avoids dropped frames by dynamically adjusting the bit rate of the data contained in each frame. For the online game selected to be played, the video content contained in the initial frame generated for the online game may have a high bit rate because the content may include an entire scene of the game and this bit rate may be higher than the maximum bit rate allowed for the online game. The monitoring engine detects the bit rate overrun caused by the large initial frame and immediately instructs the encoder to lower the bit rate in one or more subsequent frames following the large initial frame so that the system can make up for the data debt caused by the bit rate overrun of the initial frame. The monitoring engine continues to monitor the bit rate of each frame generated for the online game, queries the transmission debt counter to determine whether subsequent frames received at the encoder need to be adjusted, and sends a signal to the encoder to instruct the encoder to dynamically adjust the bit rate of the subsequent frames during encoding or just encode the subsequent frames without adjusting any bit rate. The encoded subsequent frames are forwarded to the client device for rendering. Adjusting the bit rate of subsequent frames downwards may adversely affect the resolution quality of subsequent frames, as opposed to the intermittent stuttering or frame dropping observed using conventional encoders, where the price of maintaining the frame rate is small. Intermittent frame dropping, especially if the number of dropped frames is choppy for long periods or frequently, can affect the overall quality of the video content provided to the user, as the video becomes very choppy.

[0026] A user may set up a user account with a cloud system, such as a cloud gaming system, to allow the user access to various online games available therein. One or more cloud gaming servers of the cloud gaming system may run instances of the online games locally on the cloud and generate streaming content for transmission to the user's client devices for rendering. As a user plays the online games and provides game inputs, the game inputs are transmitted to the cloud gaming server(s) and affect the outcome of the online games. In response to the game inputs, the cloud gaming server(s) generate frames of data for transmission to the client devices for rendering. The generated frames of data are encoded using an encoder available within the cloud system to generate encoded frame data. The encoded frames of data are forwarded to a network interface, and the frame data is packetized and transmitted to the client devices according to a transmission protocol defined for a communication channel established between the cloud gaming server(s) and the client devices over a network, such as the Internet.

[0027] The encoder provides the encoded frame data to a monitoring engine available in the cloud system. The monitoring engine may be integrated within the encoder or available within a cloud gaming server and communicatively connected to the encoder to enable interaction with the encoder. The monitoring engine examines the amount of data included in the encoded frame. The monitoring engine checks whether the amount of data included in the encoded frame exceeds a predefined maximum data limit defined for each frame of the online game. If the encoded frame exceeds the predefined maximum data limit, the monitoring engine determines the extent to which the amount of data included in the encoded frame exceeds the maximum data limit and sends a signal to the encoder to dynamically reduce the bitrate of one or more subsequent frames of data received after the large frame that exceeded the maximum data limit. In response to the communication from the monitoring engine, when the encoder receives a subsequent frame from the cloud gaming engine, the encoder introduces a new reduced bitrate during encoding of the subsequent frame. The reduced bitrate may be predefined for the online game. The encoded subsequent frames with the reduced bitrate are forwarded to the client device for rendering. The monitoring engine continues to examine all encoded data frames received from the encoder to determine the amount of data they contain and instructs the encoder to dynamically adjust subsequent frame(s) to maintain the frame rate while adhering to the bit rate limitations of the communication channel.

[0028] The monitoring engine may query a "transmitted debt" counter maintained for the online game. The transmit debt counter maintains a running tally of excess data and the number of subsequent frames for which data needs to be dynamically reduced. Typically, an initial frame of an online game may contain a large amount of data. This is because the initial frame contains images of an initial game scene of the online game, and subsequent frames may contain updates to the images contained in the initial frame. However, sometimes subsequent frames may contain a large amount of data due to user input or the type of game or current state of the game. This is the case when the user moves to a different scene and data for the new scene is being provided by the online game. By querying the transmit debt counter and closely monitoring the bitrate of each frame received at the encoder, the monitoring engine can determine which of the frames contain an excessive amount of data and how many subsequent frames need to be dynamically reduced in data, and instruct the encoder accordingly. The encoder adjusts the frame(s) of data according to the instructions given by the monitoring engine. After adjusting the frame(s) of data, the transmit debt counter is updated by the monitoring engine. The updated transmission debt counter is used to indicate to the encoder.

[0029] If the transmission debt counter is not zero, the monitoring engine sends a signal to the encoder to adjust the data of the subsequent frame. In response to the signal, the encoder evaluates the amount of data contained in the subsequent frame received by the encoder. If the amount of data contained in the subsequent frame is less than the reduced maximum data limit defined for the online game, the encoder does not adjust any of the data contained in the subsequent frame, but simply encodes the data. For example, if the reduced maximum data limit for the subsequent frame is defined to be about 10,000 bits per frame, and the subsequent frame contains only about 8,000 bits, there is no need to further adjust the content of the subsequent frame since it is already below the reduced maximum data limit. In this example, the encoder simply encodes the data contained in the subsequent frame and transmits it to the client device. The encoded subsequent frame is used to make up for some of the excess data from the previous large frame. After transmitting the subsequent frame, the monitoring engine updates the transmission debt counter.

[0030] If the amount of data contained in the subsequent frame exceeds the reduced maximum data limit, but does not exceed the predefined maximum data limit, and the "transmitted debt" counter is not zero, the encoder dynamically introduces a new reduced bitrate to the subsequent frame (i.e., reducing the maximum data limit) and transmits the encoded subsequent frame to the client device. Introducing the new reduced bitrate may include compressing the data of the subsequent frame using a compression technique. The monitoring engine, utilizing the transmitted debt counter, manages the framerate of the online game consistently and smoothly, allowing the online game content to be displayed without latency or noticeable glitches.

[0031] With a general understanding of the embodiments of the present invention, illustrative details of various implementations will now be described with reference to the various drawings.

[0032] 1 presents an overview of a cloud gaming system 300 used to provide a steady stream of frames of data related to gameplay of an online game to client devices according to one embodiment. Multiple client devices 100 (100-1, 100-2, 100-3, ... 100-n) are used by various players (i.e., users playing online games) to access a cloud gaming system (CGS) 300, social media applications, and other interactive applications over a network 200, such as the Internet. The client devices 100 may be accessing the CGS 300 from different geographic locations. The client device 100 can be any type of client computing device with a processor, memory, using wired, wireless, 4G, 5G or other communication capabilities, and involving access to a portable or non-portable network (e.g., a local area network (LAN), a wide area network (WAN), a wireless LAN, a MAN (metropolitan area network), a personal / private area network (PAN), a virtual private network (VPN), etc.). For example, the client device can be a smartphone, a mobile device, a tablet computer, a desktop computer, a personal computer, a wearable device, or other digital device including a monitor and touch screen with a hybrid or portable form factor.

[0033] A client device with 5G communication capabilities may include a mobile device or any other computing device that can connect to a 5G network. In one embodiment, the 5G network is a digital cellular network with a coverage area divided into multiple "cells" (i.e., small geographic areas). Analog data generated by the mobile device is digitized and transmitted as radio waves to a local antenna in the cell using frequency channels that can be reused by geographically separate cells. The local antenna is connected to the telephone network and the Internet by high-bandwidth optical fiber or other similar wireless connections. Because the 5G network uses high-frequency radio waves for communication, data can be transmitted at a higher data rate, resulting in lower network latency.

[0034] The client device may include a network interface that runs an operating system and provides access to various gaming or interactive applications (apps) available on the cloud gaming server of CGS 300 over the network 200, or may be a thin client with a network interface that communicates with a cloud gaming server 302 that provides computing capabilities. A player may access CGS 300 using a user account and select an online game available on CGS 300 for gameplay, which is controlled by the player using control / input options provided on the client device or using a controller communicatively connected to the client device. The player's user account may be verified against user data stored in user account data store 308 to verify that the player is entitled to access and play game titles of online games stored in game title data store 310 before providing access to the online game.

[0035] In some implementations, various applications (e.g., online games) may be executed remotely on one or more cloud gaming servers 302 of the cloud gaming system 300, and gameplay-related data from the executing applications is generated as frames and streamed to the client device 100 over the network 200. In other implementations, the online game may be executed locally on the client device, and metadata from the executing online game may be sent over the network 200 to a backend server of the cloud gaming server(s) 302 to affect the state of the game. The frames of gameplay data are forwarded to an encoder for encoding, and the encoded frames are forwarded to the client device for rendering.

[0036] It should be noted that both players and users refer to people / individuals who have a user account and are eligible to access and interact with applications (including online games) available in the cloud gaming system 300. A player or user is someone who has previously played an online game or has currently selected an online game to play. A "user" may also be someone who selected an online game to watch a player's (e.g., a spectator) gameplay before selecting the online game for gameplay.

[0037] The cloud gaming system (CGS) 300 may include a network of backend servers distributed in different geographic locations and configured to run instances of one or more online game applications and / or other interactive applications available on the backend servers. The backend servers 302 may be cloud game servers or cloud application servers configured to run instances of one or more online game / interactive applications. For simplicity, we refer to cloud game servers (or simply "game servers") running instances of online game applications, but the embodiments disclosed herein may be extended to any other interactive application that may be used to provide frames of interactive data to users. Game servers may be any type of server computing device available in a cloud gaming system, including, but not limited to, standalone servers, servers that are part of a server farm or data center, etc. Additionally, the game servers may manage one or more virtual machines that support game processors that run instances of online games on the hosts.

[0038] The online games executed by the cloud gaming server 302 may be single-player games or multiplayer games. In some implementations, the online games may be massively multiplayer online (MMO) games that allow multiple players and spectators across different geographic locations to access and play / watch the online games. The cloud gaming server 302 may include a multiplayer distributed game engine communicatively connected to the game logic of the online game. Generally speaking, a game engine is a software layer that underpins games, such as MMO games, and provides a framework used to develop online games. The game engine abstracts the details of performing common related tasks (i.e., game engine tasks) required for any online game, while the online game developer provides the game logic that provides the details of how the online game is played. The game engine framework includes multiple reusable components to handle several working parts (e.g., core features) of the online game that bring the online game to life. The basic core features handled by the game engine may include physics (e.g., collision detection, collision reaction, trajectory, object movement based on gravity, etc., friction), graphics, audio, artificial intelligence, scripting, animation, networking, streaming, optimization, memory management, threading, localization support, etc. The reusable components include a process engine used to handle the core features identified for an online game.

[0039] During gameplay of an online game, the game engine collects and transmits player inputs received from different client devices to the game logic, manages the game logic of the online game, and optimally manages the allocation and synchronization of the functional parts of the game engine to process the game data generated by the game logic, and generates frames of game data that are sent back to the client devices for rendering. Various game engines are currently available providing different core functionalities, and an appropriate game engine may be selected based on the functionality indicated for executing the online game.

[0040] The game logic analyzes inputs made by players at their respective client devices during a gameplay session, updates the game state of the online game based on the inputs, manages saved data for each player playing the online game, and generates game data that is processed by the distributed game engine before being streamed to the users' client devices during the gameplay session. The game state of the online game is used to identify game state metrics for the online game and user metrics for the players, which may be stored separately in gameplay metrics 314. For example, user metrics related to the gameplay complexity of each player may be stored in user metrics data store 314a, and game state metrics for the online game may be stored in game state metrics data store 314b. The game state identifies the overall state of the online game at a particular point in time and is affected by the gameplay complexity of one or more players. The user metrics for each player include any game customizations made by the players for the online game. The distributed game engine may use the game state provided by the game logic to overlay / insert objects and characters into the game environment of the players participating in the gameplay session. In some implementations, the game logic for each instance of a game may run on a server or on multiple servers depending on how the game logic is configured.

[0041] Details of game inputs provided by each player of the online game during game play, the game state of the online game, the location of each player of the online game, and activities performed by each player may be stored by the game engine as telemetry data within game state metrics 314b or separately by the game engine 304 in a telemetry data store (not shown). Details included in the telemetry data identify characteristics of each activity attempted and completed by a player, and player attributes of the player who attempted and completed each activity of the online game. If the online game needs to be resumed from a particular point, the telemetry data associated with each player may be used to recreate portions of the online game. Additionally, the telemetry data may be used to generate a data frame for transfer to a client device.

[0042] The frames of data generated by the game engine are forwarded to an encoder 305 for encoding before being sent to the client device for rendering. The encoder 305 may be a hardware encoder or a software encoder and is configured to encode the data contained within the frames according to a defined encoding algorithm. The encoded frames of data are also forwarded to a monitoring engine 306. The monitoring engine evaluates the amount of data contained within each encoded frame and verifies the amount of data against a predefined maximum data limit specified for the online game based on the communication connection established between the cloud gaming server 302 and the client device 100. The maximum data limit is determined by the bandwidth capabilities of the communication connection, which may be set by a network service provider.

[0043] When the monitoring engine 306 detects an amount of data transmitted in a particular frame that exceeds a predetermined maximum data limit, the monitoring engine 306 sends a signal to the encoder 304 to reduce the amount of data included in subsequent frames. The monitoring engine 306 may maintain a "transmitted debt" counter that acts as an overdraft account and keeps a tally of the amount of excess data transmitted in a particular frame that needs to be made up and the number of subsequent frames that need to make up the excess "debt". In some implementations, the excess data may be made up from one or more subsequent frames. The number of subsequent frames to make up for the excess data transmitted in a particular frame may be determined by the monitoring engine based on the extent to which the amount of data in a particular frame exceeds the maximum data limit (i.e., bit rate) set for the communication connection, the type of data being transmitted, the type of online game for which the data is generated, the requirements of the online game, the bandwidth capabilities of the communication connection, or a combination of two or more thereof. For example, if the online game is a particularly intense game, the make-up may need to occur more quickly as opposed to an online game that is being watched by multiple users. In order to maintain the bit rate of the communication channel, the make-up may need to occur more quickly for particularly intense games.

[0044] The monitoring engine queries the transmission debt counter to determine whether the data included in the subsequent frame needs to be dynamically adjusted, and sends an appropriate signal to that effect to the encoder. The signal generated by the monitoring engine 306 may include an indicator of whether the subsequent frame needs to be dynamically adjusted, and if so, the reduced maximum data limit specified for adjusting the subsequent frame. The reduced maximum data limit, as well as the rate of compensation, can be set based on the type of game or the requirements of the game. In response to the signal provided by the monitoring engine, the encoder first evaluates the amount of data included in the subsequent frame. If the amount of data included in the subsequent frame exceeds the reduced maximum data limit, the encoder dynamically adjusts the data included in the subsequent frame to be equal to or less than the reduced maximum data limit. On the other hand, if the amount of data included in the subsequent frame does not exceed the reduced maximum data limit, the encoder simply encodes the data of the subsequent frame and transmits the encoded data over the network to the client device. In this case, the data included in the subsequent frame is not adjusted because the amount of data is already below the reduced maximum limit set for the subsequent frame. For example, if the reduced maximum data limit for a subsequent frame is set to 10,000 bytes, and a subsequent frame contains 9,000 bytes of data, the encoder does not adjust the data for the subsequent frame. Instead, it encodes the data and forwards the encoded data to the network interface for transmission over the network to the client device. Onward Transmission The encoding of subsequent frames may include some data from larger frames that exceed the maximum data limit to make up for transmission debt incurred by the system.

[0045] After encoding and transmitting a subsequent frame, the encoder interacts with the monitoring engine 306 to update the transmission debt counter to account for subsequent frames transmitted to the client device. As shown in the example above, the transmission debt counter is updated even if the amount of data does not need to be adjusted in subsequent frames. The monitoring engine continues to query the transmission debt counter and send a signal to the encoder as long as frames of data are generated for the online game.

[0046] In the current embodiment, the system can compensate for the excess data sent in a particular frame by reducing the data of the subsequent frames. Each frame generated by the online game must comply with the maximum data limit for the data contained in the frame. However, the data contained in each frame generated by the online game does not always comply with the maximum data limit. The monitoring engine described in various embodiments handles this by dynamically compensating for the excess data from frames following the big frame, so that the entire frame complies with the specified maximum bit rate. Depending on the extent to which the data of the detected big frame exceeds the maximum bit rate, the number of frames for which the bit rate is adjusted can be one or ten or more. By reducing the data of one or more subsequent frames (i.e., reducing the bit rate contained in the subsequent frame(s)), the system can recover from the burden of the bit rate overage caused by the large frame. Data overload can lead to latency or dropped frames. Dynamically adjusting the data of the subsequent frames ensures that all frames are delivered to the client device and maintains a consistent frame rate defined for the online game that meets the maximum data rate. Although the reduction in data for subsequent frames may result in a degradation in the quality of subsequent frames delivered to the client device, such degradation is temporary because the system can quickly make up for the transmission debt and resume transmitting frames at the maximum data rate defined for the online game after the transmission debt has been made up.

[0047] FIG. 2A illustrates an exemplary gaming cloud system that uses a monitoring system to maintain the frame rate of an online streaming game in one implementation. A game engine 304 runs an instance of the online game on a server, such as a cloud gaming server 302, in response to a user selecting an online game for gameplay. User input made on the user's client device 100 is communicated to the game engine to affect the game state of the online game. The online game generates frames of data 135 that represent the current game state of the game. The frames of data 135 are forwarded to an encoder 305 available on the cloud gaming server 302. The encoder 305 may be available on the cloud gaming server 302 running the instance of the online game, or on another server in the cloud gaming system 300, or may be a separate computing device that includes encoder logic. In the implementation illustrated in FIG. 2A, the encoder 305 is shown to be a hardware encoder that can be directly connected to the network 200. The implementation is not limited to the use of a hardware encoder, but can be easily extended to the use of a software encoder.

[0048] The encoder 305 is specified not to exceed a suggested frame size, which may be in accordance with the bandwidth of the communication channel established between the server 302 and the client device 100. The encoder may not always adhere to the suggested frame size, and data beyond the suggested frame size may be included. To avoid overwhelming the communication channel with data overload, the cloud gaming system operates the monitoring engine 306 to track the amount of data included in each encoded frame and to control the bitrate of subsequent frames following a frame that exceeds the size of the frame.

[0049] The encoder 305 encodes the data according to an encoding algorithm and forwards the encoded frames to a network interface defined in one of the servers of the cloud gaming system (e.g., the cloud gaming server 302 or another server). The network interface packetizes the encoded data frames according to a transmission protocol defined for the communication channel and forwards the packets of data over a network to the client device 100 for rendering. The encoder 305 also provides the encoded frames of data forwarded to the client device 100 to the monitoring engine 306. The monitoring engine 306 evaluates the amount of data of the encoded frames. If the encoded frames include data that exceeds the maximum data limit of the frame, the monitoring engine 306 detects the excess data and instructs the encoder to reduce the amount of data included in one or more subsequent frames. The monitoring engine maintains a transmission debt counter and dynamically updates the transmission debt counter to indicate the excess data of the encoded frames transmitted to the client device 100.

[0050] When the encoder receives a subsequent frame from the game engine (i.e., frame source), the encoder checks whether the monitor engine 306 has sent a signal to adjust the data of the subsequent frame. When the encoder detects the signal from the monitor engine 306, the encoder identifies the predefined reduced maximum data limit and then examines the amount of data included in the subsequent frame to determine whether it is within the predefined reduced maximum data limit. The signal from the monitor engine 306 may include instructions to reduce the amount of data included in the subsequent frame and the reduced maximum data limit that may be used by the encoder to reduce the data in the subsequent frame. Based on the signal, the encoder checks whether the subsequent frame contains data within the reduced maximum data limit. If the amount of data in the subsequent frame is within the reduced maximum data limit, the encoder 305 encodes the subsequent frame without adjusting any data. On the other hand, if the data of the subsequent frame exceeds the reduced maximum data limit, the encoder dynamically adjusts the data of the subsequent frame before encoding the frame. The encoded subsequent frame is forwarded to the network interface for packetization. In this case, data from the subsequent frame may be combined with a portion of the data from the previous large frame sent by the encoder to the network interface for packetization and transmission to the client device to make up for the excess data of the previous large frame. The encoder then forwards the packets of data, including the encoded data from the subsequent frame and the excess data from the previous large frame, to the client device 100 for rendering. As the name suggests, the reduced maximum data limit is less than the normal maximum data limit specified for a frame. For example, the maximum data limit for a frame may be set to 20,000 bytes, while the reduced maximum data limit may be set to 10,000 bytes. For example, the reduced maximum data limit may be set based on the type of game or the requirements of the game.

[0051] FIG. 2B illustrates an alternative embodiment of a game cloud system that engages a monitoring system to maintain the frame rate of an online streaming game. The game cloud system components identified in this embodiment are similar to those shown and described with reference to FIG. 2A, except for the frame data processor 309, which is between the frame source and the encoder. As a result, the components common to FIGS. 2A and 2B are referenced using the same reference numbers and are configured to perform similar functions. In this embodiment, instead of forwarding generated frames directly to the encoder, the game engine now forwards the frames to the frame data processor 309 for processing. As part of the processing, the frame data processor 309 is configured to analyze the data contained in each frame to determine the complexity of the data contained within the frame. The information provided by the frame data processor 309 is used to identify data that is important for rendering the image contained in the frame for the user at the client device. The important areas may be areas where changes have occurred to the game scene or game objects / avatars as a result of the user's input to the online game, or where an action / event has occurred in response to the user's input to the online game. In some implementations, the important areas that contain changes may be identified as being specific to the user for whom the data is being generated. For example, an I-frame may be generated by a game engine to include data for constructing an entire scene of a game when rendered on a client device, and the game engine may generate subsequent frames as P-frames. The P-frame includes changes to the scene of the game included in the I-frame, and the changes may include the movement of an object, or the performance of an action, or the occurrence of an event that is specific to a user and identified from the processing of a user's input to an online game. However, not all changes included in a P-frame may be significant to a user. For example, the occurrence of an event may be recorded as a change and included in the P-frame generated by the game engine.However, the event may be occurring behind the user-controlled avatar. As a result, the event may not be important to the user because the user has already passed the point or location where the event is occurring. The frame data processor 309 may contextually analyze the data contained in each frame generated by the online game to determine which data is important to render to the user. Note that data that is important to a first user may or may not be important to a second user. As a result, the content rendered for the first user may not be the same as the content rendered for the second user, even if the first and second users are interacting within the same scene.

[0052] The frame data processor 309 forwards the results of the analysis along with the frame data to the encoder 305. The encoder 305 uses the information provided by the frame data processor 309 to determine what data to retain and what data to discard in each frame when the encoder is instructed to dynamically reduce the amount of data contained in the frame. During encoding, the encoder 305 dynamically reduces the amount of data in the frame by retaining data identified as important to the user and encoding the retained data. The amount of data that needs to be reduced is subject to the reduced maximum data limit specified by the monitoring engine 306. The resulting encoded frame is forwarded to the client device. It should be noted that the frames for which the amount of data is dynamically reduced are the frames following the large frame (i.e., subsequent frames).

[0053] In another embodiment, the encoder may reduce the amount of data contained in non-critical sections while leaving the data in important sections intact when encoding a dynamically reduced frame. In yet another alternative embodiment, the encoder may compress the data contained in the frame using either lossy or lossless techniques so that subsequent frames comply with the reduced maximum data limits specified for the online game.

[0054] FIG. 3A illustrates various components of a monitoring engine used in one embodiment to maintain a steady stream of frames of an online game. The monitoring engine 306 may be part of the game engine that executes the game logic or may be a separate engine that receives frames of data generated by the game engine. The monitoring engine 306 includes a bitrate monitor 311, a bitrate adjuster 312, and a transmit data debt tracker (or simply referred to herein as a "debt tracker") 313 for tracking and adjusting the bitrate of each frame. The aforementioned components are presented as examples and should not be considered exhaustive or limiting. Fewer or additional components may be used.

[0055] A frame source, such as a game engine, generates frames of data during gameplay of an online game. These frames of data are sent to a client device for rendering. User input made at the client device, either through the controls or interactive user interface of the client device, or from an external controller communicatively coupled to the client device, is sent to the game engine, which executes an instance of the online game at a server device. The user input is used to affect the game state of the online game. The updated game state is provided to the client device in subsequent frames for rendering. In the case of streaming of an online game, frames are generated continuously. Depending on the game state of the online game, the amount of data included in each frame may vary. Initial frames generated for an online game may be large, as they may include data related to a scene of the online game. Subsequent frames generated for an online game may be smaller, as they may only include changes detected in the initial scene. During gameplay, the game engine may generate large frames of data from time to time. This may be the case when the game engine detects a user moving from one scene to another, or from one level to the next. The generated frames of data are forwarded to an encoder 305 for encoding before transmitting the encoded frames to the client device.

[0056] The encoder 305 is configured to receive the frames of data and encode the content according to the encoding details defined for the online game. For example, the encoder may need to adhere to a particular frame rate / frequency (e.g., 30 frames per second (fps), 60 fps, 120 fps, etc.) and maximum bit rate of the communication channel used to transmit the data. The encoded frames are forwarded by the encoder 305 to a bit rate monitor 311.

[0057] The bitrate monitor 311 is configured to evaluate the amount of data contained within the encoded frame using a predefined maximum data limit 311a specified for the online game. Each online game may have its own predefined maximum data limit 311a or may follow a common predefined maximum data limit 311a based on the communication channel used to transfer the streaming data to the client device. When the amount of data of the encoded frame exceeds the maximum data limit 311a specified for the online game, the bitrate monitor 311 calculates the extent to which the data of the encoded frame exceeds the maximum data limit 311a. The bitrate monitor 311 then determines a reduced maximum data limit 311b to be used to adjust the data of the subsequent frame(s) and uses a frame number calculation engine 311c to calculate the number of subsequent frames for which the content needs to be dynamically adjusted to make up for the data overage caused by the encoded frame. The frame number calculation engine 311c can calculate, on a case-by-case basis, the number of subsequent frames to distribute the debt queue to make up for the data overage. For example, for a live streaming game, the number of frames over which the debt queue (i.e., excess make-up) may be spread may be large, such as about 15 to about 50 frames, since the latency caused by the debt queue spreading may not adversely affect the user's overall viewing experience. On the other hand, for an interactive streaming game, the debt queue spreading may need to be more aggressive so that the debt queue can be cleared sooner to maintain light / minimal latency. As a result, for an interactive streaming game, the debt queue may be spread over a small number of frames, such as about 1 to 10 frames, about 1 to 5 frames, or about 1 to 3 frames. Thus, the frame number calculation engine 311c calculates the extent of the debt queue spreading based on the type of data being generated, the demands of the game, the extent of the bitrate excess caused by the encoded frames, etc.

[0058] The bitrate overage and the number of subsequent frames to spread the debt queue are communicated by the bitrate monitor to the transmitted data debt tracker 313 along with instructions to update one or more counters maintained by the debt tracker 313. In response to instructions from the bitrate monitor 311, the debt tracker 313 can update a bitrate debt counter 313a with the bitrate overage and a frame debt counter 313b with the number of subsequent frames.

[0059] The bitrate monitor further sends a signal to the bitrate adjuster 312 to instruct the encoder to adjust the bitrate of the subsequent frame, specifying a reduced maximum data limit 311b following the subsequent frame. The reduced maximum data limit may be predefined and may be based on the bandwidth of the communication channel and the type and requirements of the online game. In response to the instruction from the bitrate monitor 311, the bitrate adjuster 312 uses the signal generator 312a to generate a signal to the encoder 305 to adjust the amount of data downward in the subsequent frame to make up for the excess of data caused by the encoded frame being sent to the client device 100. The encoder adjusts the amount of data in the subsequent frame according to the instruction included in the signal. As the subsequent frame is adjusted, encoded, and sent over the Internet, the bitrate debt counter 313a and the frame debt counter 313b are updated to account for the amount of debt made up from the subsequent frame. The evaluation of the encoded frame, the querying of the transmitted data debt tracker, the instructing of the bit rate adjuster by the bit rate monitor, and the sending of signals by the bit rate adjuster 132 to adjust subsequent frames continues until an encoded frame is generated by the encoder and the generated and transmitted debt is fully satisfied before receiving a subsequent frame from the frame source (such as an online game). An example sequence of operations performed to instruct the encoder using the bit rate monitor 311 and the bit rate adjuster 312 is described in detail with reference to FIG.

[0060] FIG. 3B illustrates different components of the frame data processor 309 that are used in one embodiment to identify the complexity of the content contained in a frame. The frame data processor 309 is configured to receive frames of data generated at a frame source (e.g., a gaming application) and analyze the data contained in the frame before forwarding the frame to an encoder for further analysis and encoding. The frame data processor 309 works in conjunction with the frame data analyzer 309a to analyze the data contained in the frame to understand the complexity of the data. The frame data analyzer 309a can perform a contextual analysis of the data to determine how the data is relevant to each user. The contextual analysis can be used to identify different data that is important to present to different users who may be playing or viewing the same game and who are in the same scene of an online game (such as a multiplayer online game). For example, the movement of objects behind a user / user's avatar in an online game may not be important, but the movement of objects or activities that are visually in the vicinity of the user may be important. Alternatively, an object that a user is interacting with or an activity that a user is performing may be more important to a user than an object that another user is interacting with or an activity that another user is performing in the same scene.

[0061] The frame data adjuster 309b receives the results of the context analysis from the frame data analyzer 309a and uses the results of the analysis to identify data associated with portions of the frame that are important and data associated with portions of the frame that are not important to the user. The frame data processor 309 forwards the frames of data along with the information obtained from the context analysis to the encoder 305. The encoder 305 uses the information provided by the frame data processor 309 when the encoder 305 needs to dynamically adjust the data in a frame in response to a signal from the bit rate adjuster 312. The encoder 305 encodes the adjusted frames of data and forwards the encoded frames to the client device for rendering. It should be noted that the frames that are dynamically adjusted follow the large frame that caused the data overage.

[0062] The excess of data may be made up with a number of subsequent frames following the large frame. In some implementations, there may be a scenario in which a particular subsequent frame in the number of subsequent frames following the large frame may itself contain data that exceeds the maximum data limit. In this scenario, the bitrate debt counter 313a and the frame debt counter 313b are updated to account for the excess data contained in the particular subsequent frame. Additionally, the frame number calculation engine 311c may identify an additional number of frames that needs to be dynamically adjusted to make up for the additional transmitted data debt incurred as a result of the particular subsequent frame exceeding the limit, which is added to the appropriate debt counter. However, such a scenario rarely occurs because in most cases, the data of the large frame is followed by a series of smaller data frames, making it easy to make up for the excess data before receiving the next larger frame.

[0063] By dynamically adjusting the bit rate of subsequent frames, the system can compensate for the extra data while maintaining the frame rate and minimizing the wait time while the frames are rendered, allowing the system to deliver video content to users without dropped frames or compromised video quality.

[0064] FIG. 4 illustrates a series of process flow operations performed by the combined bitrate monitor 311 and bitrate adjuster 312 components of the monitoring engine 306 to instruct the encoder 305 to adjust the bitrate to maintain the frame rate, in one embodiment. The initial frame is generated by a frame source, such as a game engine running an online game. Although a particular embodiment is described with reference to an online game, the embodiment can be extended to any other application that generates streaming content. If the online game is being played for the first time, the initial frame may relate to the first scene of the online game that the user selected for gameplay. If the online game is being resumed from a point where the online game was previously paused, the initial frame may relate to a scene from the paused point of the online game. The initial frame may also include user-related content, such as a user-customized avatar, user attributes, etc. As shown in bubble 1, the initial frame is forwarded to the encoder for encoding before transmission to the client device.

[0065] When an initial frame arrives at the encoder, the encoder checks whether there is any indication from the bitrate monitor 311 to dynamically adjust the amount of data included in the initial frame, and such a check is performed by the encoder before encoding the initial frame. Because this is the initial frame received in an online game (either newly played or resumed from a paused point), the bitrate debt counter 313a and the frame debt counter 313b are both zero. Therefore, there is no indication from the bitrate monitor to adjust the amount of data included in the initial frame. As a result, the initial frame is encoded according to the encoding algorithm defined in the encoder 305 to generate an encoded frame. The encoded frame is forwarded to the bitrate monitor 306 to enable the bitrate monitor 306 to determine the size of the encoded frame, as indicated by bubble 2.

[0066] In response to receiving the encoded frame, even if it is an initial frame, the bitrate monitor 311 queries the transmitted debt data tracker 313 to determine whether there is any data overage that needs to be made up, as shown in bubble 4. The data tracker 313 queries the bitrate debt counter 313a and the frame debt counter 313b to determine whether there is any remaining data overage. If one or both of these counters 313a, 313b are not zero, it means that there is some data overage that still needs to be recovered. If these counters are both zero, it indicates that the data overage has been completely made up by the system. For an initial frame arriving at the encoder, the bitrate debt counter 313a and the frame debt counter 313b are both zero because the online game has not yet recorded a bitrate overage. In this case, there is no previous bitrate overage signal sent by the bitrate monitor 311 to the bitrate adjuster 312.

[0067] Next, the bitrate monitor 311 determines the size of the encoded frame by evaluating the amount of data contained in the encoded initial frame against a predefined maximum data limit for the online game. If the bitrate monitor 311 determines that the size of the encoded initial frame is greater than the defined maximum data limit for the communication channel, the bitrate monitor 311 calculates the bitrate overage and updates the bitrate debt counter 313a with the bitrate overage (i.e., the amount of data in bytes that exceeds the maximum data limit). Further, based on the bitrate overage, the type of data being generated, the requirements of the online game, and the maximum bitrate limit of the communication channel, the bitrate monitor 311 determines the number of frames to make up for the data overage of the initial frame. For example, for a streaming interactive video game, the bitrate monitor 311 can use an aggressive number of frames to make up for the data overage, while for a streaming game being viewed by multiple spectators, the bitrate monitor 311 can use a more liberal number of frames to make up for the data overage. As shown in bubble 4, the number of frames to make up for the data overage is updated to the frame debt counter 313b.

[0068] Based on the determination of the data excess, a signal is sent from the bit rate monitor 311 to the network interface to adjust the packets of encoded data being transmitted to the client, as shown in bubble 3. The packet adjustment signal, or the absence of a packet adjustment signal, is used by the network interface when packetizing the encoded data in accordance with a transmission protocol for transmitting data over the Internet to the client device. The packetized data is forwarded to the client device for decoding and rendering. If the signal from the bit rate monitor indicates excess data in the encoded initial frame, the network interface adjusts the encoded data included in the packet during packetization so that the data communicated in the packet fits within the data bandwidth of the communication channel. The remaining excess encoded data is kept in a cache memory and included in the data packets of one or more subsequent frames.

[0069] In addition to updating bitrate debt counter 313a and frame debt counter 313b and signaling the network interface to adjust the data packets, bitrate monitor 311 instructs bitrate adjuster 312 to send a signal to encoder 305 indicating that there is a transmitted excess debt that needs to be made up from one or more subsequent frames, as shown in bubble 5. Bitrate adjuster 312 sends a signal to the encoder to dynamically introduce a reduced maximum data limit for data contained in subsequent frames received at the encoder, as shown in bubble 6. The reduced maximum data limit can be predetermined and can be based on the type of data being generated and the requirements of the game.

[0070] When a second frame (i.e., a frame following the initial frame) is received at the encoder 305, the encoder 305 checks whether any signal from the bit rate adjuster 312 has been received. In response to the signal from the bit rate adjuster 312, the encoder 305 dynamically introduces a predefined reduced maximum data limit by dynamically adjusting the data included in the second frame. The encoder may reduce the amount of data included in the second frame by compressing the data using compression logic. In some implementations, various types of compression logic may be implemented based on the amount of data that needs to be adjusted. Alternatively, selective data identified as important to constructing the image may be retained in the second frame, and the remaining non-essential data may be discarded. Various methods of reducing the data of the second frame may be used, including but not limited to compression or selective inclusion / discarding. The encoder then encodes the adjusted data and forwards the encoded frame of second data to the network interface.

[0071] The network interface receives the adjusted and encoded data for the second frame along with a signal from the bit rate monitor to include a portion of the excess data from the earlier frame. The network interface includes the portion of the remaining data from the larger frame and the data of the second frame in accordance with the signal from the bit rate monitor, thereby compensating for a portion of the transmitted debt, and forwards the encoded and adjusted second frame packet to the client device for rendering. Because the second frame includes data below the reduced maximum data limit, there is enough space in the frame to include the remaining data from the larger frame.

[0072] After the second frame of data is forwarded to the network interface, the bitrate monitor 311 updates the bitrate debt counter and the frame debt counter. Since the amount of data included in the second frame is within the reduced maximum data limit, there is no data overage caused by the data included in the second frame. Instead, a portion of the transmitted debt has been compensated. The bitrate monitor 311 captures the amount of the transmitted debt compensated in the second frame by adjusting the bitrate debt counter 313a and the frame debt counter 313b. After adjusting the debt counters (313a, 313b), the bitrate monitor 311 forwards an updated signal to the bitrate adjuster 312 to present the current value for each debt counter. If the debt counters are not zero, the bitrate adjuster forwards another signal to the encoder to adjust the data of the third frame received by the encoder to compensate for the transmitted debt. The process of receiving frames of data from a frame source, evaluating the amount of data contained in each frame, sending appropriate signals to the encoder to dynamically adjust the amount of data contained in each frame, and forwarding the adjusted content to the client device continues throughout the online game for as long as frames of content are being generated.

[0073] FIG. 5 shows a simple sample table illustrating frame data processing performed using an encoder and monitoring engine according to one embodiment. In this example, at time T1, a frame (e.g., frame 10) is received at the encoder. Frame 10 may be an initial frame (I-frame) generated when an online game is selected for play, or may be a frame generated during play of a current session or from resuming play from a previously paused play session. Time T1 may be a time representing the start / resume of play, or may be a time during play of an online game. Frame 10 is shown to contain 40,000 bytes of data. Content frame attributes of the online game may be defined such that the frame rate of the data is 60 frames per second (fps). The maximum bit rate of each frame may be defined according to the bandwidth availability of a communication channel established between a server running the online game and a client device used to interact with the online game. For example, the maximum bit rate may be defined as 10 megabits per second (Mbps). The specified bit rate defines the amount of data that may be included in each frame. In the above example, the maximum data limit for each frame is defined to be 10,000,000 / 60 / 8 = 20,833 bytes per frame.

[0074] Frame 10 is encoded using an encoder, and the encoded frame of data is forwarded to a bitrate monitor for evaluation. The bitrate monitor evaluates the amount of data contained in frame 10 and determines that frame 10 contains approximately 20,000 bytes of extra data than is dictated by the frame attributes of the online game. The bitrate monitor also queries the transmission debt counters (bitrate debt counter and frame debt counter) to see if any of these transmission debt counters are non-zero. The bitrate monitor uses information from the transmission debt counters to determine if there is an excess transmitted debt that needs to be made up using one or more subsequent frames. If frame 10 is an initial frame generated for the current play session, then the transmission debt counter will be zero. On the other hand, if frame 10 is from the current play session of the online game, and there was no extra data in the previous frame 9 and no adjustment was required, then the transmission debt counter will be zero. Based on the determination from the transmission debt counter query (i.e., "adjust this frame" is "no"), the bitrate monitor sends a signal to the network interface to packetize the data of encoded frame 10 according to the bandwidth limitations of the communication channel. Additionally, when the bitrate monitor detects excess data contained in encoded frame 10, the bitrate monitor also increments the bitrate debt counter by 20,000 (data beyond the maximum data allowed) and increments the frame debt counter by 2. The number of frames to adjust is determined by the amount of excess data contained in the encoded frames and the reduction of the maximum byte limit imposed by the content. A portion of the data in frame 10 (i.e., approximately 20,000 bytes worth of data) is packetized and sent to the client, while the remaining portion of the data (i.e., approximately 20,000 bytes of excess data) is retained in a cache for inclusion in a subsequent frame. Other than retaining a portion of the excess encoded data in a local cache, no adjustments are made to the data in frame 10 (i.e., there is no dynamic reduction of the data contained in frame 10).The bit rate monitor then sends a signal to the bit rate adjuster to signal the encoder to adjust at least the next subsequent frame to make up for the transmitted debt, and this signal is sent as soon as frame 10 is forwarded to the network interface.

[0075] At time T2, the encoder receives frame 11. Upon receiving frame 11, the encoder determines that a signal has been received from the bit rate adjuster to adjust the data for frame 11 (i.e., "Adjust this frame" is "Yes"). The encoder determines the reduced maximum bytes that a subsequent frame may contain (i.e., 10,000 bytes) and evaluates frame 11 to determine whether frame 11 needs to be adjusted. Even though the encoder has a signal to adjust a subsequent frame, there may be cases where the subsequent frame does not need to be adjusted. This may occur if the amount of data contained in the subsequent frame (i.e., frame 11 in this example) is below the reduced maximum data limit. Based on the evaluation, the encoder determines that frame 11 contains only 9,000 bytes, which is below the reduced maximum data limit of 10,000 bytes (i.e., there is no bit rate overrun for frame 11). As a result, the encoder will just encode the data for frame 11 and forward the encoded data for frame 11 to the network interface, with instructions to include the portion of the data for frame 10 stored in the cache. In this example, frame 11 allows approximately 11,000 bytes to be used to process frame 10, which is over budget, so that the entire frame data does not exceed the maximum data limit of 20,000 bytes.

[0076] When 11,000 bytes of frame 10 are transmitted in the subsequent frame 11, the bitrate monitor decrements the bitrate debt counter by 11,000 bytes and the frame debt counter by 1 after forwarding the encoded data of frame 11 with instructions to the network interface. The adjusted and encoded data contained in frame 11 is packetized and forwarded to the client device for rendering. The bitrate monitor instructs the bitrate adjuster to send another signal to the encoder with the updated information in the debt counter.

[0077] At T3, frame 12 is received at the encoder, which is approximately 10,000 bytes, the reduced maximum data limit, and is significantly below the maximum data limit. Based on the signal from the bit rate adjuster, the encoder evaluates frame 12 to determine whether the data of frame 12 needs to be adjusted. However, based on the evaluation of frame 12, the encoder determines that frame 12 does not have a bit rate overage. As a result, the encoder determines that the data of frame 12 does not need to be adjusted before encoding it. On the other hand, if frame 12 contains data that exceeds the reduced maximum data limit (e.g., frame 12 contains 12,000 or 15,000 bytes), the encoder adjusts the data of frame 12 by compressing or removing / discarding non-essential data so that the data of frame 12 is below the reduced maximum data limit. The encoder then encodes the data of frame 12 and forwards it to the network interface with instructions to include any remaining data of frame 10. As a result, frame 12 allows approximately 10,000 bytes to be used to process frame 10, which is over budget, so that the entire frame data does not exceed the maximum data limit of 20,000 bytes. Thus, once frame 12 is packetized, the network interface processes the remaining 9,000 bytes of frame 10 with the data from frame 12 and forwards the packets of encoded frame 12 data along with the data for frame 10 to the client for rendering. Upon forwarding the encoded frame 12 data to the network interface, the bitrate monitor decrements the bitrate debt counter by 9,000 and decrements the frame debt counter by 1, so that both debt counters become zero. As a result, the bitrate monitor instructs the bitrate adjuster to send a signal to the encoder to resume encoding of subsequent frames at the maximum data limit.

[0078] When frame 13 is received at the encoder, the encoder evaluates the amount of data for frame 13 based on a signal from the bit rate adjuster and resumes the maximum data limit for subsequent frames, determining that the data is below the maximum data limit. A signal from the bit rate adjuster to the encoder indicates that the data for frame 13 does not need to be adjusted since both debt counters indicate a value of zero. Based on this determination, the encoder encodes the frame 13 data and forwards the encoded frame 13 data to the network interface, which instructs the frame 13 data to be just packetized and sent to the client device. The client device receives and renders the frame 13 data. This process of receiving frames of data at the encoder, evaluating and updating the debt counters, and sending appropriate signals to the bit rate adjuster to the encoder and instructing the network interface to send appropriate signals continues as frames are generated for the online game.

[0079] 6 illustrates operations of a method used to dynamically adjust a bit rate of one or more subsequent frames for an online game to maintain a frame rate defined for the online game, according to one embodiment. The method begins at operation 610 when a frame of data is received at an encoder. The frame of data may be generated by an online game engine in response to user input made from a client device to an online game running on a server and may include a game state of the online game. Although the online game may specify a maximum data limit for each frame, the game engine may not always adhere to the maximum data limit specified for the online game when generating the frame of data.

[0080] The frames of data are encoded by an encoder before being sent to the client device, as shown in operation 620. The encoder encodes the data in the frames and forwards the encoded frames to the network interface for transmission to the client device, even if the data contained in the frames exceeds the maximum data limit. Onward Transmission The encoded data frame is also forwarded to a bitrate monitor in the monitoring engine. The bitrate monitor evaluates the amount of data included in the encoded frame, as shown in operation 630. When the bitrate monitor detects data that exceeds the maximum data limit, the bitrate monitor sends a signal to a bitrate adjuster in the monitoring engine to instruct the encoder to dynamically adjust the amount of data included in one or more subsequent frames, as shown in operation 640. The bitrate monitor may calculate the amount of data overage caused by the frame and update one or more counters (e.g., a bitrate debt counter, a frame debt counter). Upon updating the one or more counters, the bitrate monitor uses the updated information included in the debt counters to instruct the bitrate adjuster. The instruction to the bitrate adjuster may also include a predefined reduced maximum data limit to use when it is necessary to dynamically adjust the amount of data for one or more subsequent frames. The reduced maximum data limit may be determined based on the bandwidth of a communication channel established between the server device and the client device. Alternatively, the encoder may identify a predefined reduced maximum data limit for an online game. The encoder uses the indication from the bit rate adjuster to dynamically adjust the amount of data included in subsequent frames, compensating for the excess of data from the large frame from subsequent frames.

[0081] Adjustments to subsequent frames can be made using compression logic, or through identification and removal of non-essential data from the frames, or through other means that can be used to reduce data without affecting the quality of the image contained in the frame. Various implementations described herein ensure smooth rendering of frames of online game content. Some of the frames that may have been dynamically adjusted to compensate for the data overage may have a lower resolution due to the reduced data, but the period of time that the low quality frames last before resuming normal resolution is very short. Furthermore, the frame rate defined for the online game is maintained, so that the rendering of the online game video occurs without any issues. Other advantages may be envisioned by those skilled in the art when considering various implementations.

[0082] Various embodiments described herein may be implemented in a cloud system. According to one embodiment of the present disclosure, an exemplary cloud system is outlined that supports the collection of user inputs to update interactive application data and uses the updated application data to generate frames of content for transmission to client devices. The exemplary cloud system described herein is a gaming cloud system (GCS) that allows access to and play video games stored therein. The GCS may be a cloud computing system that operates over a network to support multiple players playing one or more online gaming applications through corresponding gameplays. Data related to these gameplays are provided in frames that are encoded by an encoder and transmitted as a data stream over the network to client devices. Specifically, the system includes one or more social media providers and a GCS that interacts with client devices, all of which are connected over a network (e.g., the Internet). Multiple client devices associated with multiple users may be connected to the network to access services provided by the GCS and the social media providers. A user may be a player or a spectator who follows the gameplay of a particular player or of an online game.

[0083] In one embodiment, the GSC includes a game server, a video recorder, a tag processor, and an account manager, which includes a user profile manager, a game selection engine (i.e., a game selector), a game session manager, a user access logic, a network interface, and a social connection manager. The GCS may further include multiple storage systems, such as a game state store, a user account, and a user data store, which may generally be stored in a data store. Other game storage systems may include a game code store, a game record store, a tag data store, a video game title store, and a game network user store. In one embodiment, the GCS is a system that can provide online gaming applications, services, gaming-related digital content, and interactivity among systems, applications, users, and social networks. The GCS may communicate with user devices and social media providers through a social connection manager via a network interface. The social connection manager may be configured to associate and / or connect to one or more friends identified using the social media provider. In one embodiment, each social media provider includes at least one social graph that indicates the user social network connections.

[0084] Players (i.e., users) can access services provided by the GCS through a game session manager. For example, an account manager enables authentication of players and access to game titles on the GCS. The account manager stores information about users (players, spectators) and manages a user profile for each user. The user's information may be used by the account manager for authentication purposes. Additionally, game titles owned by users may be managed by the account manager. For example, video games stored in a game title store are available to users who own or are given access to those video games.

[0085] In one embodiment, a user can access services provided by the GCS and social media providers via a client device with a communication connection established over a network. The client device can include any type of device having a processor and memory, wired or wireless, portable or non-portable. In one embodiment, the client device can take the form of a smartphone, tablet computer, or a hybrid that provides touch screen capabilities in a portable form factor. One exemplary client device may include a mobile phone device running an operating system and including various applications (apps) that can be acquired over a network and executed on a local portable device (e.g., smartphone, tablet, laptop, desktop, etc.).

[0086] The client device may include a display that serves as an interface for the user to input commands and data and / or display frames and / or transmit information received from the GCS and social media providers. The display may be configured as a touch screen, or a display typically provided by a flat panel display, cathode ray tube (CRT), or other device capable of rendering a display. Alternatively, the client device 100 may have its display separate from the device, similar to a desktop or laptop computer.

[0087] In one embodiment, a client device is configured to communicate with the GCS to enable a user (player) to play a video game. For example, a player may select a video game (e.g., by game title) available in a game title data store via a game selection engine. The selected video game is activated and an instance of the video game is loaded in the GCS for execution by a game server. In one embodiment, game play is primarily performed in the GCS, and frames of data generated from the game play are encoded using an encoder and sent to the client device as a stream of encoded game video frames from the GCS. User input commands to drive the game play are sent back to the GCS. The encoded video frames received from the streamed game play are decoded at the client device and rendered on a display of the client device.

[0088] In one embodiment, after a player selects an available game title to play, a game session of the selected game title may be initiated by the player via a game session manager. The game session manager first accesses a game state store in the data store to retrieve the saved game state of the player's last played session (for the selected game), if any, and allows the player to resume game play from the previous game play stop point. Once a resume or start point is identified, the game session manager may notify the game execution engine of the game server to execute game code of the selected game title from the game code store. After the game session is initiated, the game session manager may pass game video frames (i.e., streaming video data) to an encoder for encoding, and the encoded video frames are transmitted to the client device via a network interface.

[0089] During gameplay, the game session manager may communicate with the game server, the video recorder's recording engine, and the tag processor to generate or store a recording (e.g., video) of the gameplay or gameplay session. In one embodiment, the video recording of the gameplay may include any tag content entered or provided during gameplay, as well as other game related metadata. Tag content may also be stored via snapshots. The video recording of the gameplay may be stored in a record game store along with any game metrics corresponding to the gameplay. Any tag content may be stored in a tag data store.

[0090] User input commands provided by a user at a client device are communicated to a game server via a game session manager. The user input commands are used to affect the outcome of a corresponding gameplay of the video game. Input commands (e.g., controller input or user input) include input commands used to drive gameplay and user interactive input such as tag content (e.g., text, images, recorded clips, etc.). Game input commands as well as any user play metrics (e.g., the amount of time a user plays a game) may be stored in a user store on the game network. Selection information related to gameplay of the video game may be used to enable a number of features available to the user.

[0091] FIG. 7 is a block diagram of a gaming system 700 in accordance with various embodiments of the present invention. The gaming system 700 is configured to provide video streams to one or more clients 710 over a network 715. The network is similar to the network 200 shown in FIGS. 1, 2A, and 2B. The gaming system 700 typically includes a video server system 720 and an optional game server 725. The video server system 720 is configured to provide a minimum quality of service video stream to the one or more clients 710. For example, the video server system 720 may receive game commands that change a state or perspective within a video game and provide the client 710 with an updated video stream that immediately reflects the change with minimal latency. The video server system 820 may be configured to provide video streams in a wide variety of alternative video formats, including formats yet to be defined. Additionally, the video streams may include video frames configured to be presented to a user at a wide variety of frame rates. Typical frame rates are 30 frames per second, 60 frames per second, and 120 frames per second. However, alternative embodiments of the present invention include higher or lower frame rates.

[0092] The clients 710, individually referred to herein as 710A, 710B, etc., may include head mounted displays, terminals, personal computers, game consoles, tablet computers, telephones, set-top boxes, kiosks, wireless devices, digital pads, standalone devices, handheld game play devices, and / or the like. Typically, the clients 710 are configured to receive encoded video streams, decode the video streams, and present the resulting video to a user, e.g., a player of a game. The process of receiving the encoded video streams and / or decoding the video streams typically involves storing individual video frames in a receive buffer of the client. The video streams may be presented to the user on a display integrated with the client 710 or on a separate device such as a monitor or television. The clients 710 are optionally configured to accommodate multiple game players. For example, a game console may be configured to accommodate two, three, four or more players simultaneously. Each of these players may receive a separate video stream, or a single video stream may include regions of frames generated specifically for each player, e.g., generated based on each player's viewpoint. The clients 710 are optionally geographically distributed. The number of clients included in the gaming system 700 may vary widely, from one or two to thousands, tens of thousands, or more. As used herein, the term "game player" is used to refer to a person who plays a game, and the term "game playing device" is used to refer to a device used to play a game. In some embodiments, a game playing device may refer to multiple computing devices that work together to deliver a gaming experience to a user. For example, a game console and an HMD may work with the video server system 720 to deliver a game viewed through an HDM. In one embodiment, the game console receives a video stream from the video server system 720, and the game console forwards the video stream or updates of the video stream to the HMD for rendering.

[0093] The client 710 is configured to receive the video stream via a network 715. The network 715 can be any type of communications network, including a telephone network, the Internet, a wireless network, a power line network, a local area network, a wide area network, and / or a private network, etc. In a typical embodiment, the video stream is communicated via a standard protocol, such as TCP / IP or UDP / IP. Alternatively, the video stream is communicated via a proprietary standard.

[0094] A typical example of the client 710 is a personal computer with a processor, non-volatile memory, a display, decoding logic, network communication capabilities, and input devices. The decoding logic may include hardware, firmware, and / or software stored on a computer-readable medium. Systems for decoding (and encoding) video streams are known in the art and vary depending on the particular encoding scheme used.

[0095] The clients 710 may, but need not, further include a system configured to modify the received video. For example, the clients may be configured to perform further rendering, such as overlaying one video image on another and / or cropping a video image. For example, the clients 710 may be configured to receive various types of video frames, such as I-frames, P-frames, and B-frames, and process the frames into images for display to a user. In some embodiments, one of the clients 710 is configured to perform operations on the video stream, such as further rendering, shading, or 3D transformation. One of the clients 710 is optionally configured to receive multiple audio or video streams. The input devices of the clients 710 may include, for example, a one-handed game controller, a two-handed game controller, a gesture recognition system, an eye gaze recognition system, a voice recognition system, a keyboard, a joystick, a pointing device, a force feedback device, a motion and / or position sensing device, a mouse, a touch screen, a neural interface, a camera, and / or input devices under development.

[0096] The video stream (and optionally the audio stream) received by the client 710 is generated and provided by the video server system 720. As further described elsewhere herein, the video stream includes video frames (and the audio stream includes audio frames). The video frames are configured to contribute meaningfully to an image displayed to a user (e.g., a video frame includes pixel information in a suitable data structure). As used herein, the term "video frame" is used to refer to a frame that primarily includes information configured to contribute to, e.g., act on, an image shown to a user. Most of the teachings herein regarding "video frames" can also be applied to "audio frames."

[0097] The client 710 is typically configured to receive inputs from a user. These inputs may include game commands configured to change the state of or affect game play in a video game. The game commands may be received using an input device and / or may be generated automatically by computing instructions executed on the client 710. The received game commands are communicated from the client 710 to the video server system 720 and / or the game server 725 over the network 715. For example, in some embodiments, the game commands are communicated to the game server 725 via the video server system 720. In some embodiments, separate copies of the game commands are communicated from the client 710 to the game server 725 and the video server system 720. The communication of the game commands optionally depends on the identity of the command. The game commands are optionally communicated from the client 710A over a path or communication channel that is different from the path or communication channel used to provide the audio or video stream to the client 710A.

[0098] The game server 725 is optionally operated by a different entity than the video server system 720. For example, the game server 725 may be operated by a publisher of a multiplayer game. In this example, the video server system 720 is optionally viewed as a client by the game server 725 and is optionally configured to appear from the perspective of the game server 725 to a conventional client running a conventional game engine. Communication between the video server system 720 and the game server 725 is optionally performed over the network 715. Thus, the game server 725 may be a conventional multiplayer game server that transmits game state information to multiple clients, including the video server system 720. The video server system 720 may be configured to communicate with multiple instances of the game server 725 simultaneously. For example, the video server system 720 may be configured to provide multiple different video games to different users. Each of these different video games may be supported by a different game server 725 and / or published by a different entity. In some embodiments, several geographically distributed instances of the video server system 720 are configured to provide game videos to multiple different users. Each of these instances of the video server system 720 may communicate with the same instance of the game server 725. Communication between the video server system 720 and one or more game servers 725 optionally occurs over a dedicated communication channel. For example, the video server system 720 may be connected to the game server 725 over a high-bandwidth channel dedicated to communication between these two systems.

[0099] The video server system 720 comprises at least a video source 730, an I / O device 745, a processor 750, and non-transitory storage 755. The video server system 720 may include one computing device or may be distributed across multiple computing devices, optionally connected via a communication system such as a local area network.

[0100] Video source 730 is configured to provide a video stream, e.g., streaming video, or a series of video frames forming a movie. In some embodiments, video source 730 includes a video game engine and rendering logic. The video game engine is configured to receive game commands from a player and maintain a copy of the video game state based on the received commands. This game state includes the positions of objects within the game environment, and typically the viewpoint. The game state may also include object properties, images, colors, and / or textures.

[0101] The game state is typically maintained based on game rules and game commands such as movement, turning, attacking, focusing, interaction, and / or use. Portions of the game engine are optionally located in a game server 725. The game server 725 may maintain a copy of the game state based on game commands received from multiple players using geographically distributed clients. In these cases, the game state is provided by the game server 725 to a video source 730, where the copy of the game state is stored and rendered. The game server 725 may receive game commands directly from the clients 710 over the network 715 and / or via the video server system 720.

[0102] The video source 730 typically includes rendering logic, e.g., hardware, firmware, and / or software stored on a computer-readable medium such as storage 755. The rendering logic is configured to create video frames of a video stream based on the state of the game. All or part of the rendering logic is optionally located in a graphics processing unit (GPU). The rendering logic typically includes processing stages configured to determine three-dimensional spatial relationships between objects and / or apply suitable textures, etc., based on the state and viewpoint of the game. The rendering logic generates raw video, which is then typically encoded prior to communication to the client 710. For example, the raw video may be encoded according to Adobe Flash® standards .wav, H.264, H.263, On2, VP6, VC-1, WMA, Huffyuv, Lagarith, MPG-x, Xvid, FFmpeg, x264, VP6-8, realvideo, mp3, or the like. The encoding process generates a video stream that is optionally packaged for delivery to a decoder on a remote device. A video stream is characterized by a frame size and a frame rate. Typical frame sizes include 800x600, 1280x720 (e.g., 720p), 1024x768, but any other frame size may be used. The frame rate is the number of video frames per second. A video stream may include various types of video frames. For example, the H.264 standard includes "P" frames and "I" frames. An I frame contains information to refresh all macroblocks / pixels on a display device, while a P frame contains information to refresh a subset of them. A P frame is usually smaller in data size than an I frame. As used herein, the term "frame size" is meant to refer to the number of pixels in a frame. The term "frame data size" is used to refer to the number of bytes required to store a frame.

[0103] In an alternative embodiment, video source 730 includes a video recording device, such as a camera. The camera may be used to generate delayed or live video that can be included in a video stream of a computer game. The resulting video stream optionally includes both rendered images and images recorded using a still or video camera. Video source 730 may also include a storage device configured to store pre-recorded video for inclusion in the video stream. Video source 730 may also include a motion or position sensing device configured to detect the motion or position of an object, such as a person, and logic configured to determine game state or generate video based on the detected motion and / or position.

[0104] Video source 730 is optionally configured to provide overlays configured to be placed over other video. For example, these overlays may include command interfaces, login prompts, messages for game players, images of other game players, video feeds of other game players (e.g., webcam video). In embodiments in which client 710A includes a touchscreen interface or an eye gaze detection interface, the overlays may include a virtual keyboard, joystick, and / or touchpad, and the like. In one example of an overlay, a player's voice is overlaid on an audio stream. Video source 730 optionally further includes one or more audio sources.

[0105] In embodiments where the video server system 720 is configured to maintain game state based on input from multiple players, the viewpoint, including the position and direction of the view, may be different for each player. The video source 830 is optionally configured to provide a separate video stream to each player based on each player's viewpoint. Additionally, the video source 730 may be configured to provide different frame sizes, frame data sizes, and / or encoding to each of the clients 710. The video source 730 is optionally configured to provide 3D video.

[0106] The I / O devices 745 are configured for the video server system 720 to transmit and / or receive information such as video, commands, information requests, game state, gaze information, device actions, device location, user actions, client identification, player identification, game commands, security information, and / or audio. The I / O devices 745 typically include communications hardware such as a network card or modem. The I / O devices 745 are configured to communicate with the game server 725, the network 715, and / or the clients 710.

[0107] The processor 750 is configured to execute logic, e.g., software, contained within the various components of the video server system 720 discussed herein. For example, the processor 750 may be programmed with software instructions to perform the functions of the video source 730, the game server 725, and / or the client qualifier 760. The video server system 720 optionally includes multiple instances of the processor 750. The processor 750 may also be programmed with software instructions to execute commands received by the video server system 720 or to coordinate the operation of the various elements of the game system 700 discussed herein. The processor 750 may include one or more hardware devices. The processor 750 is an electronic processor.

[0108] Storage 755 includes non-transitory analog and / or digital storage devices. For example, storage 755 may include an analog storage device configured to store video frames. Storage 755 may include computer-readable digital storage, such as a hard drive, optical drive, or solid-state storage. Storage 715 is configured to store (e.g., in a suitable data structure or file system) video frames, artificial frames, video streams including both video frames and artificial frames, audio frames, and / or audio streams, etc. Storage 755 is optionally distributed across multiple devices. In some embodiments, storage 755 is configured to store software components of video source 730 discussed elsewhere herein. These components may be stored in a format that allows them to be readily provided as needed.

[0109] The video server system 720 optionally further comprises a Client Qualifier 760. The Client Qualifier 760 is configured to remotely identify the capabilities of a client, such as client 710A or 710B. These capabilities may include both the capabilities of the client 710A itself as well as the capabilities of one or more communication channels between the client 710A and the video server system 720. For example, the Client Qualifier 760 may be configured to test the communication channel via the network 715.

[0110] The Client Qualifier 760 may manually or automatically identify (e.g., discover) the capabilities of the Client 710A. Manual identification includes communicating with a user of the Client 710A to ask the user to inform the capabilities. For example, in some embodiments, the Client Qualifier 760 is configured to display images and / or text, etc., in a browser of the Client 710A. In one embodiment, the Client 710A is an HMD that includes a browser. In another embodiment, the Client 710A is a game console with a browser that may be displayed on an HMD. The displayed objects request the user to input information such as the operating system, processor, video decoder type, network connection type, display resolution, etc. of the Client 710A. The information entered by the user is transmitted back to the Client Qualifier 760.

[0111] The automatic identification may be performed, for example, by running an agent on the client 710A and / or by sending a test video to the client 710A. The agent may include computing instructions, such as JavaScript, embedded in a web page or installed as an add-on. The agent is optionally provided by the client qualifier 760. In various embodiments, the agent may find out the processing capabilities of the client 710A, the decoding and display capabilities of the client 710A, the latency reliability and bandwidth of the communication channel between the client 710A and the video server system 720, the display type of the client 710A, firewalls present on the client 710A, the hardware of the client 710A, software running on the client 710A, and / or registry entries within the client 710A, etc.

[0112] The Client Qualifier 760 includes hardware, firmware, and / or software stored on a computer-readable medium. The Client Qualifier 760 is optionally located on a computing device separate from one or more other elements of the video server system 720. For example, in some embodiments, the Client Qualifier 760 is configured to identify characteristics of a communication channel between the client 710 and multiple instances of the video server system 720. In these embodiments, the information discovered by the Client Qualifier may be used to identify an instance of the video server system 720 that is best suited to deliver streaming video to one of the clients 710.

[0113] FIG. 8 illustrates an embodiment of an information service provider architecture that may be used in providing access to various games. Information service provider (ISP) 802 provides multiple information services to geographically distributed and connected users 800 via network 850. Network 850 may be similar to network 200 of FIGS. 1, 2A, and 2B. Although various embodiments have been discussed with respect to providing fast access to games, the embodiments may be extended to provide fast access to other interactive applications, such as virtual tours, that provide content that may include different levels of content that are accessible using one or more visual cues or other tools or application points or awards or rewards acquired by the user during interaction with the interactive application. For example, ISP 802 may provide only one type of service, such as games, or a variety of services, such as games, stock price updates, broadcast media, news, sports, gaming, etc. Additionally, the services provided by each ISP may be dynamic, i.e., services may be added or removed at any time. Thus, the ISPs that provide a particular type of service to a particular individual may change over time. For example, a user may be served by an ISP near the user while the user is in his / her hometown, and when the user moves to another town, the user may be served by a different ISP. The local ISP, through the connection module, will transfer the necessary information and data from the user's gaming or access profile to the new ISP, so that the user information "follows" the user to the new town, making the data closer to the user and more accessible. In another embodiment, a master-server relationship may be established between a master ISP that manages information about the user, and a server ISP that interfaces directly with the user under control from the master ISP.In another embodiment, data may be transferred from one ISP to another as the client moves around the world (i.e., during a switch in the data center assigned to the user), and such transfer may be based on compatibility of the services offered by each ISP, such that the ISP 902 that is better positioned to serve the user is the ISP that provides these services.

[0114] The ISP 802 includes an Application Service Provider (ASP) 806 that provides computer-based services to customers over a network. Software provided using the ASP model is sometimes called on-demand software or software as a service (SaaS). A simple form of providing access to a particular application program (e.g., customer relationship management) uses a standard protocol such as HTTP. The application software resides, for example, on the vendor's system and is accessed by the user through a web browser using HTML, or by dedicated client software provided by the vendor, or through other remote interfaces such as thin clients.

[0115] Services delivered across large geographical regions often use cloud computing. Cloud computing is a computing style in which dynamically scalable, often virtualized resources are provided as a service over the Internet. Users do not need to be experts in the technical infrastructure in the "cloud" that supports them. Cloud computing can be divided into various services such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). Cloud computing services often provide common business applications online that are accessed from a web browser, but the software and data are stored on servers. The term cloud is used as a metaphor for the Internet (e.g., with servers, storage, and logic) based on how the Internet is depicted in computer network diagrams, and is an abstraction for the complex infrastructure it hides.

[0116] Additionally, the ISP 802 includes a Game Processing Server or Provider (GaPS) 808 that is used by game clients to play single and multiplayer video games. Most video games played over the Internet operate through a connection to a game server. Typically, games use dedicated server applications that collect data from players and distribute it to other players. This is more efficient and effective than a peer-to-peer configuration, but requires a separate server to host the server application. In another embodiment, the GaPS establishes communication between players and their respective game playing devices to exchange information without relying on a centralized GaPS.

[0117] A dedicated GaPS is a server that runs independently of the clients. Such servers usually run on dedicated hardware located in a data center, providing greater bandwidth and dedicated processing power. Dedicated servers are the preferred method of hosting game servers for most PC-based multiplayer games. Massively multiplayer online games run on dedicated servers that are usually hosted by the software company that owns the game title, allowing the dedicated server to control and update the content.

[0118] A broadcast processing server or provider (BPS) 810 distributes audio or video signals to viewers. Broadcasting to a very small audience is sometimes called narrowcasting. The final step in broadcast distribution is how the signal reaches the listener or viewer, which may come over the air to an antenna and receiver, similar to a radio or television station, or through cable television or cable radio (or "wireless cable"), either through a station or directly from a network. The Internet can also deliver either radio or television to the recipient, especially using multicast, which can share signals and bandwidth. Historically, broadcasts have been scoped by geographical area, such as national or local broadcast. However, with the proliferation of high-speed Internet, content can reach most countries in the world, so broadcasts are no longer geographically defined.

[0119] Storage Service Providers (SSPs) 812 provide computer storage space and related management services. SSPs also provide periodic backups and archiving. By providing storage as a service, users can order more storage as needed. Another major advantage is that SSPs include backup services so that users do not lose all of their data if their computer's hard drive fails. Furthermore, multiple SSPs can have full or partial copies of user data, allowing users to access data in an efficient manner, regardless of the user's location or the device being used to access the data. For example, while the user is on the move, the user can access personal files on their home computer and mobile phone.

[0120] Communications providers 814 provide connectivity to users. One type of communications provider is an Internet Service Provider (ISP), which provides access to the Internet. ISPs connect customers using data transmission technologies suitable for delivery of Internet protocol datagrams, such as dial-up, DSL, cable modem, fiber, wireless, or dedicated high-speed interconnects. Communications providers may also provide messaging services, such as email, instant messaging, and SMS text. Another type of communications provider is a Network Service Provider (NSP), which sells bandwidth or network access by providing direct backbone access to the Internet. Network service providers may consist of telecommunications companies, data carriers, wireless communication providers, Internet service providers, cable television operators providing high-speed Internet access, and the like.

[0121] A data exchange 804 interconnects several modules in the ISP 902 and connects these modules to client devices 820 of users 800 via a network 850. The data exchange 804 can cover a small area where all modules of the ISP 802 are in close proximity, or it can cover a large geographic area when the various modules are geographically distributed. For example, the data exchange 804 could include a high-speed Gigabit Ethernet (or even faster) in a cabinet of a data center, or an intercontinental virtual area network (VLAN).

[0122] Users 800 access remote services using respective client devices 820, which include at least a CPU, memory (not shown), display, and I / O. Client devices can be PCs, cell phones, netbooks, tablets, gaming systems, PDAs, etc. In one embodiment, the ISP 802 recognizes the type of device used by the client and adjusts the communication method employed. In other cases, client devices access the ISP 802 using standard communication methods such as HTML.

[0123] Embodiments of the invention may be practiced with a variety of computer system configurations including hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.

[0124] With the above embodiments in mind, it should be understood that the present invention may employ various computer-implemented operations involving data stored in computer systems. These operations are operations requiring physical manipulation of physical quantities. Any of the operations described herein that form part of the present invention are useful machine operations. The present invention also relates to devices or apparatus for performing these operations. The apparatus may be specially constructed for the required purposes, or the apparatus may be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general-purpose machines may be used with computer programs written in accordance with the teachings herein, or it may prove more convenient to construct a more specialized apparatus to perform the required operations.

[0125] The invention can also be embodied as computer readable code on a computer readable medium. Alternatively, the computer readable code may be downloaded from a server using the aforementioned data exchange interconnect. The computer readable medium is any data storage device that can store data, which can then be read by a computer system. Examples of computer readable media include hard drives, network attached storage (NAS), read-only memory, random access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical and non-optical data storage devices. The computer readable medium may include computer readable tangible media distributed across network-connected computer systems such that the computer readable code is stored and executed in a distributed fashion.

[0126] Although the method operations have been described in a particular order, it should be understood that other housekeeping operations may be performed between the operations, or the operations may be adjusted to occur at slightly different times, or may be distributed within a system that allows processing operations to occur at various intervals related to processing, so long as the processing of the overlay operation is performed in the desired manner.

[0127] Although the foregoing invention has been described in some detail for clarity of understanding, it will be apparent that changes and modifications can be practiced that are within the scope of the appended claims. Thus, the present embodiments are to be considered as illustrative and not restrictive, and the invention is not to be limited to the details given herein, but may be modified within the scope and equivalents of the described embodiments.

[0128] An exemplary overall system architecture of a gaming console is described herein. The exemplary gaming console may include a Sony® Playstation3® (PS3) or Playstation4® (PS4) entertainment device, which may be compatible with a controller for implementing embodiments of the present invention. Although the PS3 gaming console system architecture is described in detail, it should be noted that the various implementations described herein are extendable to the system architecture of different gaming consoles or computing devices. A system unit is provided to which various peripheral devices can be connected. The system unit is similar to the cloud gaming system 300 of FIG. 1. The system unit includes a processor, which may be an 8-core processor as in the PS3, or a multi-core processor as in the PS4, a Rambus® Dynamic Random Access Memory (XDRAM) unit as in the PS3, or a graphics dynamic random access memory such as GDDR5 as in the PS4, a Reality Synthesizer graphics unit (e.g., a 550 MHz GPU) with a dedicated video random access memory (VRAM) unit as in the PS3, or an 800 or 900 MHz GPU with shared graphics memory in the PS4 and PS4 Pro, and an I / O bridge. The system unit also includes a BluRay® Disc Read Only Memory (BD-ROM®) (optical) disc reader for reading from discs and removable slot-in hard disk drives (HDDs), accessible through the I / O bridge. Optionally, the system unit also includes a memory card reader for reading Compact Flash® memory cards, Memory Stick® memory cards, and the like, also accessible through the I / O bridge, and, in the PS4, a built-in DVR for recording games.

[0129] The I / O bridge also connects to six Universal Serial Bus (USB) 2.0 ports, a Gigabit Ethernet port, an IEEE 802.11b / g wireless network (Wi-Fi) port, and a Bluetooth® wireless link port capable of supporting up to seven Bluetooth connections.

[0130] In operation, the I / O bridge handles all wireless, USB, and Ethernet data, including data from one or more game controllers (such as a DualShock3 controller as in the PS3, or a PS4 DualShock4 controller as in the PS4). For example, when a user is playing a game, the I / O bridge receives data from the game controller over a Bluetooth link and sends it out to the processor (in the PS3) or multi-core processor (in the PS4), which updates the current state of the game accordingly. In addition, other image and movement sensors provide data captured during the user's gameplay to the I / O bridge, which sends the data out to the respective processor. Game controllers (e.g., PS4 game controllers) include a share button option for sharing games, a clickable touchpad, a rechargeable battery (Li-Ion or other type), etc.

[0131] The wireless, USB and Ethernet ports also provide connectivity for other peripheral devices, in addition to game controllers, such as remote controls, keyboards, mice, portable entertainment devices such as the Sony Playstation Portable® entertainment device, video cameras such as the EyeToy® camcorder, microphone headsets, and removable hard drives. Thus, such peripheral devices may in principle be connected to the system unit wirelessly; for example, a portable entertainment device may communicate via a Wi-Fi ad-hoc connection, while a microphone headset may communicate via a Bluetooth link.

[0132] The provision of these interfaces means that Playstation3 devices are also potentially compatible with other peripheral devices such as digital video recorders (DVRs), set-top boxes, digital cameras, portable media players, voice over IP telephones, mobile phones, printers and scanners.

[0133] In addition, a legacy memory card reader may be connected to the system unit via a USB port, allowing reading of memory cards of the type used by Playstation® or Playstation2® devices.

[0134] In this embodiment, the game controller is operable to wirelessly communicate with the system unit via a Bluetooth® link. However, the game controller can instead be connected to a USB port, which also provides power to charge the game controller's battery. In addition to one or more analog joysticks and traditional control buttons, the game controller responds to movement in six degrees of freedom, corresponding to translation and rotation in each axis. As a result, gestures and actions by the user of the game controller can be translated as inputs to the game in addition to or instead of traditional button or joystick commands. Optionally, other wireless-enabled peripheral devices, such as a Playstation® portable device, can be used as a controller. In the case of a Playstation® portable device, additional game or control information (e.g., control instructions or remaining lives) can be presented on the device's screen. Other alternative or supplemental control devices, such as custom-made controllers, such as a dance mat (not shown), a light gun (not shown), a handle and pedals (not shown), or one or more large buttons (also not shown) for a quick-answer quiz game, can also be used.

[0135] The remote control is also operable to communicate wirelessly with the system unit via a Bluetooth link. The remote control comprises suitable controls for operation of the BluRay disc BD-ROM reader and for navigation of the disc contents.

[0136] The BluRay® disc BD-ROM reader is operable to read conventional recorded compact discs (CDs) and recordable CDs, as well as so-called Super Audio CDs, as well as compact disc read-only memories (CD-ROMs) compatible with Playstation and PlayStation2 devices. The BluRay® disc BD-ROM reader is also operable to read conventional recorded DVDs and recordable DVDs, as well as digital versatile disc read-only memories (DVD-ROMs) compatible with PlayStation2 and PlayStation3 devices. The BluRay® disc BD-ROM reader is further operable to read conventional recorded Blu-Ray discs and recordable Blu-Ray discs, as well as BD-ROMs compatible with PlayStation3 devices.

[0137] The system unit is operable to provide audio and video, either generated or decoded by the Playstation3 or Playstation4 device, via the Reality Synthesizer graphics unit through audio and video connectors to a display and sound output device, such as a monitor or television set having a display and one or more loudspeakers. The audio connectors may include conventional analog and digital outputs, while the video connectors may variously include component video, S-video, composite video, and one or more High Definition Multimedia Interface (HDMI®) outputs. As a result, the video output may be in a format such as PAL or NTSC, or in high definition, 720p, 1080i or 1080p, and 4K, HDR.

[0138] Audio processing (such as generation and decoding) is performed by the processor. For example, the operating system of the Playstation3 device supports Dolby® 5.1 surround sound, Dolby® Theatre Surround (DTS), and decoding of 7.1 surround sound from Blu-Ray® discs.

[0139] In this embodiment, the video camera includes a CMOS (complementary metal oxide semiconductor) image sensor (although a charge-coupled device (CCD) image sensor may also be used), an LED indicator, and a hardware-based real-time data compression and encoding device so that compressed video data may be transmitted in a suitable format, such as an intra-image based MPEG (motion picture expert group) standard, for decoding by the system unit. The camera LED indicator is arranged to illuminate in response to suitable control data from the system unit, for example to indicate adverse lighting conditions. Video camera embodiments may variously connect to the system unit via USB, Bluetooth, Wi-Fi communication ports. Video camera embodiments may include one or more associated microphones and may also be capable of transmitting audio data. In video camera embodiments, the CCD may have a resolution suitable for capturing high-definition video. In use, images captured by the video camera may be incorporated into a game, for example, or may be interpreted as game control inputs.

[0140] Generally, a suitable piece of software, such as a device driver, should be provided so that data communication with a peripheral device, such as a video camera or a remote control, can be normally performed through one of the communication ports of the system unit. Device driver technology is well known and will not be described in detail here, suffice to say that a person skilled in the art would understand that a device driver or a similar software interface may be required in the described embodiment.

[0141] The processor has an architecture that includes four basic components: an external input and output structure including a memory controller and a dual bus interface controller, a main processor called the Power Processing Element, eight co-processors called Synergistic Processing Elements (SPEs), and a circular data bus connecting the above components called the Element Interconnect Bus. The total floating-point performance of the processor is 218 GFLOPS, compared to 6.2 GFLOPS for the Emotion Engine of the Playstation 2 device.

[0142] The Power Processing Element (PPE) is based on a two-way simultaneous multithreading compliant PowerPC core (PPU) running on a 3.2 GHz internal clock. This core contains a 512 kB level 2 (L2) cache and a 32 kB level 1 (L1) cache. The PPE is capable of eight single-place operations per clock cycle, translating to 25.6 GFLOPS at 3.2 GHz. The primary role of the PPPE is to act as a controller for the Synergistic Processing Elements, which handle the majority of the computational load. During operation, the PPE maintains a job queue, scheduling jobs for the Synergistic Processing Elements and monitoring their progress. As a result, each Synergistic Processing Element executes a kernel. The kernel's role is to pick jobs, execute them, and synchronize with the PPE.

[0143] Each Synergistic Processing Element (SPE) includes a Synergistic Processing Unit (SPU) and a Memory Flow Controller (MFC), which in turn includes a Dynamic Memory Access Controller (DMAC), a Memory Management Unit (MMU), and a bus interface (not shown). Each SPU is a RISC processor clocked at 3.2 GHz and including 256 kB of local RAM, expandable in principle to 4 GB. Each SPE exhibits a theoretical 25.6 GFLOPS of single precision performance. An SPU can process four single precision floating point numbers, four 32-bit numbers, eight 16-bit integers, or sixteen 8-bit integers in a single clock cycle. In the same clock cycle, an SPU can also perform memory operations. An SPU does not directly access the system memory XDRAM 1426. 64-bit addresses formed by an SPU are passed to the MFC, which instructs its DMA controller to access memory via the element interconnect bus and memory controller.

[0144] The Element Interconnect Bus (EIB) is a logically circular communication bus internal to the processor that connects the above mentioned processor elements, i.e. PPE, memory controller, dual bus interface, and eight SPEs, totaling 12 participants. Participants can simultaneously read and write to the bus at a rate of 8 bytes per clock cycle. As previously mentioned, each SPE contains a DMAC for scheduling longer read and write sequences. The EIB has four channels, two of which are clockwise and two of which are counterclockwise. As a result, for 12 participants, the longest stepwise data flow between any two participants is 6 steps in the appropriate direction. Thus, the theoretical peak instantaneous EIB bandwidth of the 12 slots, when fully utilized through arbitration between participants, is 96B per clock. This equates to a theoretical peak bandwidth of 307.2GB / s (gigabytes per second) at a clock rate of 3.2GHz (gigahertz).

[0145] The memory controller includes an XDRAM interface developed by Rambus Incorporated. The memory controller interfaces with a Rambus XDRAM 1426, which has a theoretical peak bandwidth of 25.6 GB / s.

[0146] The dual bus interface includes a Rambus FlexIO® system interface. The interface is organized into 12 channels, each 8 bits wide, with 5 paths for receiving and 7 paths for transmitting. This provides a theoretical peak bandwidth of 62.4GB / s (36.4GB / s transmit, 26GB / s receive) between the processor and the I / O bridge through the controller, and between the Reality Synthesiser graphics unit through the controller.

[0147] The data sent by the processor to the Reality Synthesiser graphics unit typically includes a display list, which is a series of commands for drawing vertices, textures polygons, specifying lighting conditions, and so on.

[0148] An embodiment may include capturing depth data to better identify real-world users and direct avatars or scene activity. The object may be held by a person or may be a person's hand. In this description, the terms "depth camera" and "three-dimensional camera" refer to any camera capable of acquiring two-dimensional pixel information as well as distance or depth information. For example, a depth camera may utilize controlled infrared illumination to acquire distance information. Another exemplary depth camera may be a stereo camera pair that uses two reference cameras to triangulate distance information. Similarly, the term "depth sensing device" refers to any type of device capable of acquiring two-dimensional pixel information along with distance information.

[0149] Recent advances in three-dimensional imaging offer the opportunity to expand the possibilities of real-time interactive computer animation. In particular, new "depth cameras" provide the ability to capture and map three dimensions in addition to the usual two-dimensional video images. The new depth data allows embodiments of the invention to position computer-generated objects in various locations within a video scene in real time, including behind other objects.

[0150] Moreover, the embodiments of the present invention provide a real-time interactive gaming experience for the user. For example, the user can interact with various computer-generated objects in real-time. Furthermore, the video scene can be modified in real-time to enhance the user's experience. For example, computer-generated costumes can be worn over the user's clothes, and computer-generated light sources can be utilized to cast virtual shadows in the video scene. Thus, using the embodiments of the present invention and the depth camera, the user can experience an interactive environment in his or her living room. Similar to a normal camera, the depth camera captures two-dimensional data for a number of pixels that comprise the video image. These values ​​are color values ​​for the pixels, typically red, green, and blue (RGB) values ​​for each pixel. In this manner, the objects captured by the camera appear as two-dimensional objects on the monitor.

[0151] The embodiments of the present invention also contemplate a distributed image processing configuration. For example, the present invention is not limited to the processing of captured images and displayed images being performed in one location or even two locations, such as in the CPU or in the CPU and one other element. For example, input image processing can easily be performed directly in the associated CPU, processor or device capable of performing the processing, and essentially all of the image processing can be distributed throughout the interconnected system. Thus, the present invention is not limited to any particular image processing hardware circuitry and / or software. The embodiments described herein are not limited to any particular combination of general purpose hardware circuitry and / or software, nor to any particular source for the instructions executed by the processing components.

[0152] With the above embodiments in mind, it should be understood that the invention may employ various computer-implemented operations involving data stored in computer systems. These operations include operations requiring physical manipulation of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transformed, combined, compared, and otherwise manipulated. Further, the manipulations performed are often referred to in terms, such as creating, identifying, determining, or comparing.

[0153] Any of the operations described herein that form part of the invention are useful machine operations. The invention also relates to a device or apparatus for performing these operations. The apparatus may be specially constructed for the required purposes, or the apparatus may be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general-purpose machines may be used with computer programs written in accordance with the teachings herein. Alternatively, it may prove more convenient to construct a more specialized apparatus to perform the required operations.

[0154] The above described invention may be practiced with other computer system configurations including hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.

[0155] The present invention may also be embodied as computer readable code on a computer readable medium. A computer readable medium is any data storage device that can store data that can be subsequently read by a computer system, including an electromagnetic carrier wave. Examples of computer readable media include hard drives, network attached storage (NAS), read only memory, random access memory, CD-ROM, CD-R, CD-RW, magnetic tape, and other optical and non-optical data storage devices. The computer readable medium may also be distributed over a network-coupled computer system such that the computer readable code is stored and executed in a distributed fashion.

[0156] Although the foregoing invention has been described in some detail for clarity of understanding, it will be apparent that changes and modifications can be practiced that are within the scope of the appended claims. Thus, the present embodiments are to be considered as illustrative and not restrictive, and the invention is not to be limited to the details given herein, but may be modified within the scope of the appended claims and their equivalents.

[0157] It should be understood that the various embodiments defined herein can be combined or incorporated into specific implementations using various features disclosed herein. Thus, the examples provided are only some possible examples and are not limited to the various implementations that can be defined by combining various elements. In some instances, an implementation may include fewer elements without departing from the spirit of the disclosed or equivalent implementations.

[0158] Embodiments of the present disclosure may be practiced in a variety of computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. Embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.

[0159] With the above embodiments in mind, it should be understood that the disclosed embodiments may employ various computer-implemented operations involving data stored in computer systems. These operations are operations requiring physical manipulation of physical quantities. Any of the operations described herein that form part of the disclosed embodiments are useful machine operations. The disclosed embodiments also relate to devices or apparatus for performing these operations. The apparatus may be specially constructed for the required purposes. Alternatively, the apparatus may be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general-purpose machines may be used with computer programs written in accordance with the teachings herein. Alternatively, it may be more convenient to construct a more specialized apparatus to perform the required operations.

[0160] The present disclosure can also be embodied as computer readable code on a computer readable medium. A computer readable medium is any data storage device that can store data that can be subsequently read by a computer system. Examples of computer readable media include hard drives, network attached storage (NAS), read only memory, random access memory, CD-ROM, CD-R, CD-RW, magnetic tape, and other optical and non-optical data storage devices. A computer readable medium can include computer readable tangible media distributed across networked computer systems such that the computer readable code is stored and executed in a distributed fashion.

[0161] Although the operations of the method have been described in a particular order, it should be understood that other housekeeping operations may be performed between operations, or operations may be coordinated to occur at slightly different times, or operations may be distributed within the system to allow processing operations to occur at various intervals related to processing, so long as the processing of the overlay operations is performed in the desired manner.

[0162] Although the foregoing disclosure has been described in some detail for clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. Thus, the present embodiments are to be considered as illustrative and not restrictive, and the embodiments of the present disclosure are not to be limited to the details provided herein, but may be modified within the scope and equivalents of the appended claims.

Claims

1. receiving a frame of generated data for an online game from a server device, the frame of data representing a current state of the online game; encoding the frames of data using an encoder, the encoded frames of data being transmitted over a network to a client device for rendering; evaluating the data included in the encoded frame to determine an amount of data included in the encoded frame that will be transmitted to the client device; 22. A method for providing a game system comprising: a server device and a client device that, when an amount of data in the encoded frame exceeds a maximum data limit specified for a communication channel established between the server device and the client device, sends a signal to the encoder to dynamically reduce the amount of data included in one or more subsequent frames transferred to the client device for rendering; the dynamic reduction is performed to maintain a defined frame rate for the online game.

2. The dynamic reduction of the amount of data included in the one or more subsequent frames comprises: determining the number of subsequent frames for which the amount of data needs to be reduced; and 2. The method of claim 1, further comprising: specifying a reduced maximum data limit for data included in each of the number of subsequent frames, the reduced maximum data limit being used to encode the one or more subsequent frames.

3. The method of claim 2 , wherein the reduced maximum data limit is determined based on the extent to which the data of the encoded frame exceeds the maximum data limit.

4. The method of claim 2 , wherein the number of subsequent frames to which the reduced maximum data is applied depends on a type of online game.

5. The method of claim 2 , wherein the number of subsequent frames to which the reduced maximum data is applied depends on the data requirements of the online game.

6. The method of claim 2 , wherein the reduced maximum data limit is set based on an available bandwidth of the communication channel.

7. 3. The method of claim 2, wherein the dynamic reduction of the amount of data comprises performing compression on the data contained in the frame using a compression technique, the level of compression being determined by the compression technique, the compression technique being in accordance with lossy or lossless compression.

8. The dynamic reduction of the amount of data in the subsequent frames, comprising: Identifying data associated with certain portions of the subsequent frame to retain and certain other portions of the subsequent frame to discard; and The method of claim 2 , further comprising encoding the subsequent frame using the data associated with the particular portion to be retained.

9. The method of claim 1, wherein the dynamic reduction further comprises resuming encoding of frames of data of the online game at the maximum data limit when the dynamic reduction of data for the number of subsequent frames is completed.

10. The method of claim 1 , wherein the maximum data limit is based on an available bandwidth of the communication channel established between the client device and the server device.

11. 2. The method of claim 1, wherein the dynamic reduction of the one or more subsequent frames occurs after the frame having data that exceeds the maximum data limit is transferred over the network to the client device.

12. 2. The method of claim 1, wherein the dynamic reduction processes the data for each of the one or more subsequent frames of the online game before forwarding the data for each of the one or more subsequent frames to the encoder for encoding.

13. The processing of the data for the subsequent frame, comprising: evaluating the data included in the subsequent frames to identify portions of data that are important for constructing an image included in the subsequent frames and other portions of data that are not important, the evaluation for identifying the portions of data that are important being based on a contextual relationship of content included in the subsequent frames to a user of the client device; and 13. The method of claim 12, further comprising forwarding the data of the subsequent frame including the results of the evaluation to the encoder, the encoder using the results of the evaluation to focus encoding of portions of data in the subsequent frame identified as important to the user.

14. A system for processing data for an online game, comprising: a server configured to run instances of the online game and generate frames of data for transmission to client devices; an encoder configured to receive the frames of data for the online game and encode the frames of data in accordance with a maximum data limit specified for a communication channel; the communication channel is established between the server executing the online game and the client device for transmitting the frames of data for rendering; and a monitoring system configured to monitor an amount of data included in each of the frames, and to generate a signal to dynamically reduce an amount of the data included in one or more subsequent frames if the amount of the data included in a particular frame exceeds the maximum data limit.

15. The monitoring system includes: a bitrate monitor configured to examine each frame to determine an amount of data contained in each frame encoded by the encoder for onward transmission to the client device, and to interact with a debt tracker when the particular frame contains data that exceeds the maximum data limit specified for the communication channel; the debt tracker is configured to track an amount of excess data contained in the particular frame and a number of subsequent frames generated after the particular frame of data that need to be dynamically reduced; and 15. The system of claim 14, further comprising a bit rate adjuster configured to receive from the bit rate monitor an indication of the excess data included in the particular frame sent to the client device and responsively instruct the encoder to dynamically reduce the amount of data included in the number of subsequent frames to a reduced maximum data limit.

16. 16. The system of claim 15, wherein the reduced maximum data limit is determined based on requirements of the online game.

17. The encoder comprises: receiving a first signal from the bit rate adjuster indicating an excess amount of data contained in the particular frame, the first signal specifying the reduced maximum data limit for each subsequent frame of data and the number of the subsequent frames by which the amount of data needs to be reduced; and 16. The system of claim 15, configured to reduce the data identified for inclusion in each subsequent frame in accordance with the reduced maximum data limit specified in the first signal from the bit rate adjuster.

18. The system of claim 15 , wherein the number of subsequent frames is determined based on the amount by which the particular frame exceeds the maximum data limit.

19. The bitrate monitor interacts with the debt tracker to obtain a number of the subsequent frames following the particular frame for which the amount of data needs to be dynamically reduced, and the number of the subsequent frames is dynamically updated by the bitrate monitor after each subsequent frame is transmitted to the client device; 20. The system of claim 17, wherein the bit rate adjuster interacts with the bit rate monitor to obtain a number of current subsequent frames for which the data needs to be reduced, and if the current number of subsequent frames is not zero, initiates the first signal to the encoder to dynamically reduce the data in the subsequent frames to be encoded for transmission to the client device.

20. 20. The system of claim 19, wherein the debt tracker includes a frame counter for tracking the number of frames subsequent to the current one for which the data needs to be dynamically reduced.

21. the bit rate monitor is further configured to provide an indicator to the bit rate adjuster when the frame counter reaches zero; 21. The system of claim 20, wherein the bit rate adjuster is further configured to send a second signal to the encoder to resume encoding the subsequent frame with data at the maximum data limit.

22. The encoder, determining whether the amount of data included in a subsequent frame of the one or more subsequent frames received after the particular frame exceeds the reduced maximum data limit; 16. The system of claim 15, further configured to: if the subsequent frame exceeds the reduced maximum data limit, evaluate the data of the subsequent frame to identify portions of data that are important to constructing an image contained in the subsequent frame and discard other data in the frame that is not important to constructing the image.

23. The method further includes a frame data processor, the frame data processor comprising: receiving a frame of data from the online game; Analyzing the data contained in the frame to understand the complexity of the data contained in the frame; and 15. The system of claim 14, configured to provide information to the encoder related to the complexity of the data contained in the frame, the information being used by the encoder to determine certain portions of the data of the frame to retain and other certain portions of the data of the frame to discard.

24. The method further includes a frame data processor, the frame data processor comprising: receiving a frame of data from the online game; Analyzing the data contained in the frame to understand the complexity of the data contained in the frame; and 15. The system of claim 14, configured to adjust the data included in the frame to reduce complexity of the frame, and to forward the adjusted frame of data to the encoder for encoding.

25. 25. The system of claim 24, wherein the adjusted frames of data are used by the encoder to determine certain portions of data to retain and certain other portions of data to discard.

26. The system of claim 14 , wherein the maximum data limit is based on an available bandwidth of the communication channel established between the client device and the server.

27. A method for receiving a frame of generated data for an online game from a server device, the frame of data representing a current state of the online game; determining whether a previously encoded frame of data exceeds a maximum data limit specified for a communication channel established between said server device and a client device; if it is determined that the previously encoded frame has exceeded the maximum data limit, dynamically reducing an amount of data included in the frame to a reduced maximum data limit before encoding the frame of data to generate a frame of encoded data; and and transmitting the encoded frames of data over the communication channel to the client device.

28. In the above confirmation, Querying a debt counter maintained for a current session of the online game to obtain the number of frames generated after the previously encoded frame that need to be dynamically reduced and the amount by which the previously encoded frame of data exceeded the maximum data limit; and 28. The method of claim 27, further comprising providing a signal to an encoder to dynamically reduce the amount of data included in the frames if the number of frames that need to be dynamically reduced is not zero.

29. 30. The method of claim 28, further comprising providing a second indicator to the encoder to resume encoding the frames of data at the maximum data limit if the number of frames that need to be dynamically reduced is zero.

30. 28. The method of claim 27, wherein the data included in the frame is dynamically reduced when an amount of the data included in the frame is determined to exceed the reduced maximum data limit.

Images