Detection and compensation of display lag in game system
The cloud gaming system addresses latency and media output inconsistencies by dynamically allocating resources and optimizing network usage, ensuring a synchronized and high-definition gameplay experience across diverse devices and connections.
Patent Information
- Application Number
- JP2024108770
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2019-03-28
- Filing Date
- 2024-07-05
- Publication Date
- 2024-11-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Cloud-based gaming systems face challenges in providing a consistent gameplay experience with low latency and high-definition media output across various input/output devices and network connections, especially when multiple players are located in different geographical areas with varying internet connections.
A cloud gaming system with a game application programming interface (API) and platform that dynamically allocates hardware resources and monitors network bandwidth to optimize gameplay experience, supporting multiple game sessions with high-definition media streams and low latency, using a server system to process player inputs and generate output frames in real-time.
The system ensures a seamless and synchronized gameplay experience for multiple players with minimal delay, adapting to different device and network conditions, and supports high-performance real-time gaming sessions with simultaneous review media streams.
Smart Images

Figure 2024156673000001_ABST
Abstract
Description
[Technical field]
[0001] This application relates generally to computer technology, including, but not limited to, methods and systems for managing a server system to support online interactive sessions corresponding to one or more real-time user interactive applications. [Background technology]
[0002] Internet-connected electronic devices can support a variety of cloud-based media and entertainment applications, including media streaming applications in which a server streams content to a user device, gaming applications in which a user interacts with a game running on a server from a user device, and various social media and communication applications that allow multiple users to simultaneously interact with each other and with cloud-hosted content and applications through their Internet-connected devices. Among cloud-based applications, cloud gaming presents some unique challenges due to the widely varying hardware requirements of game titles, the diverse topologies in which cloud-based games may be played (e.g., by a single player, by multiple players at a single location, or by multiple players at multiple locations), the need for reliable, latency-free transmission of player inputs to the game server running the game session, and game session outputs from the game server to the player's device / display, the widely varying expectations of players regarding speed and responsiveness of game play, and, in some circumstances, the desire to provide near real-time game content to the audience. Another challenge of cloud-based gaming relates to providing a consistent gameplay experience to a player regardless of where the player is located (e.g., close or far from the server), how the player connects to the game service (e.g., via a high or low speed Internet connection), what type of device (e.g., a personal device or a media device connected to a media streaming device) they use to play the game, or what type of device (e.g., a general-purpose personal device or a dedicated game controller) they use to view the gameplay output (e.g., a personal device or a media device connected to a media streaming device). Summary of the Invention
[0003] In particular, there is a need for a cloud gaming system that supports multiple game sessions for multiple game titles that can run simultaneously with acceptable latency and responsiveness, including for multiple players playing the same game title from the same or different locations using a wide variety of input / output devices and network connections. Additionally, there is a need for a cloud gaming system that upon receiving player inputs (e.g., game inputs entered on end user game devices / controllers) in a game session, quickly processes the user inputs and outputs high definition images reflecting the results of the player input actions for all game players simultaneously with acceptable latency. There is also a need for a gaming system that, in some circumstances, provides a high definition video stream of game play activity that allows spectators to follow the game play in real time on their respective display devices. Thus, it would be beneficial to provide a cloud gaming system with efficient game processing and output mechanisms to enhance the gaming experience in a wide range of gaming settings, from spontaneous game play with multiple users gathered in the same location to online interactive game play with multiple users from different locations. [Means for solving the problem]
[0004] overview DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The embodiments described herein are directed to providing a gaming application programming interface (API) and cloud platform to enable efficient, portable, and low latency hosting of third party gaming content. An embodiment dynamically allocates cloud gaming hardware resources and monitors and utilizes available network bandwidth to individual end users to provide an optimal cloud gaming experience. An embodiment provides multiple performance tiers, including tiers that support high performance real-time gaming sessions with high definition media output and end user streams. An embodiment is configured to support different subscription models and / or provide one or more simultaneous real-time gameplay and / or review media streams with little or no latency to one or more actual game streams (e.g., video stream output to client devices of users participating in an online / cloud gaming session via either a video stream mobile app or a browser-based program). In an embodiment, the simultaneous gameplay and / or review videos are provided to one or more users with little or no latency via a media streaming site such as YouTube®.
[0005] In one aspect of the present application, a method for controlling gameplay processing is implemented in a server system including one or more processors and a memory storing one or more programs executed by the one or more processors, the method including receiving an input event from a game controller located at a remote location during a game session operating in a current game state, the input event including a first command generated by a user interaction with the game controller during the game session, the method including determining a first frame to be displayed at the remote location during the user interaction, the first frame being one of a plurality of output frames transmitted by the server during the game session prior to the server receiving the input event, the method including determining a first game state associated with the first frame, the first game state being a game state prior to the current game state, the method including processing gameplay output according to (i) the first command and (ii) the first game state, rendering a response frame based on the gameplay output, and transmitting the response frame for display at the remote location.
[0006] In another aspect of the application, a method of rendering an online interactive game session is implemented in a server system including one or more processing cores and a memory storing a program for execution by the one or more processing cores. The method includes receiving a first command from a first client device associated with the online game session, determining a type of the first command and a first expected response delay time associated with the first command type, determining a network delay time, determining a first introduced delay time based on a comparison of the network delay time to the first expected delay time, generating a first number of intermediate frames that, when transmitted at a predetermined frame rate, occupy a transmission time corresponding to the first introduced delay time, generating a first response frame reflecting an initial result of the first command, and transmitting the first response frame following the first number of intermediate frames at the predetermined frame rate such that the first response frame is received at a first media device associated with the first client device at a time corresponding to the first expected response delay time.
[0007] In another aspect of the present application, a method is implemented in a server system including a plurality of virtual machines, each of the virtual machines having a respective resource profile. The method includes receiving a request from a client device to establish a real-time interactive game session, the request being received through a network connection with the client device, and including determining device capabilities of an output device associated with the client device, determining a connection capability of the network connection, determining one or more target quality parameters for the real-time interactive game session based on the device capabilities and the connection capabilities, selecting a first virtual machine from the plurality of virtual machines based on the one or more target quality parameters, establishing the real-time interactive game session with the client device, and providing the real-time interactive game session with resources for processing input from the client device and generating gameplay output in accordance with the input processed within the real-time interactive game session in accordance with a resource profile of the first virtual machine.
[0008] In another aspect of the application, a method is implemented in a server system including one or more processors and a memory storing one or more programs executed by the one or more processors, the method including establishing a real-time interactive game session with a first client device, the game session being associated with a particular game type, monitoring in-game performance data associated with a user of the first client device during the game session, determining an acceptable level of a gameplay experience for the user of the first client device according to the in-game performance data, and adjusting game session resources based on the acceptable level of the gameplay experience, the game session resources including a frame rate, a resolution, a latency level, or a streaming source.
[0009] According to one aspect of the present application, a server system includes a memory storing instructions for causing the server system to perform any of the methods described above.
[0010] Moreover, according to an aspect of the present application, the instructions stored in the memory of the server system include instructions for causing the server system to perform any of the methods described above.
[0011] Other embodiments and advantages will be apparent to those skilled in the art in light of the description and drawings herein.
[0012] For a better understanding of the various described embodiments, reference should be made to the following description of the embodiments in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout. [Brief description of the drawings]
[0013] [Figure 1] 1 is an exemplary online interactive gaming environment according to one embodiment. [Diagram 2] FIG. 1 is a block diagram illustrating an exemplary client device of a gaming environment according to an embodiment. [Diagram 3] FIG. 2 is a block diagram illustrating an example of a media device in a gaming environment according to some embodiments. [Figure 4] FIG. 2 is a block diagram illustrating an exemplary server of a gaming environment according to an embodiment. [Figure 5A] 1 illustrates an exemplary gaming environment according to an embodiment. [Figure 5B] 1 illustrates an example of a game scenario according to an embodiment. [Figure 5C] 1 illustrates an example of a game scenario according to an embodiment. [Figure 6] FIG. 2 is a flow diagram of a gameplay process according to one embodiment. [Figure 7] 5A-5C are flow diagrams of various trigger frame determination processes according to certain embodiments. [Figure 8] 5A-5C are flow diagrams of various trigger frame determination processes according to certain embodiments. [Figure 9] 5A-5C are flow diagrams of various trigger frame determination processes according to certain embodiments. [Figure 10]5A-5C are flow diagrams of various trigger frame determination processes according to certain embodiments. [Figure 11] 5A-5C are flow diagrams of various trigger frame determination processes according to certain embodiments. [Figure 12] 5A-5C are flow diagrams of various trigger frame determination processes according to certain embodiments. [Figure 13] FIG. 2 is a flow diagram of a delay time detection and compensation process according to an embodiment. [Figure 14A] 1 is an exemplary table of response time settings according to an embodiment. [Figure 14B] 1 is an exemplary table of response time settings according to an embodiment. [Figure 14C] 1 is an exemplary device / network assessment module according to an embodiment. [Figure 15] 1 is an exemplary online interactive gaming environment according to one embodiment. [Figure 16] 4 is an exemplary sequence of frames rendered on a display according to an embodiment. [Figure 17] FIG. 13 illustrates a delay time introduced according to an embodiment. [Figure 18] FIG. 13 illustrates a delay time introduced according to an embodiment. [Figure 19A] 1 is an exemplary online interactive gaming environment according to one embodiment. [Figure 19B] 1 illustrates an exemplary screenshot of an online interactive gaming environment according to an embodiment. [Figure 20A] 1 is an exemplary online interactive gaming environment according to one embodiment. [Figure 20B] 1 illustrates an exemplary screenshot of an online interactive gaming environment according to an embodiment. [Figure 21] FIG. 4 is a flow diagram of a delay time adjustment process according to an embodiment. [Figure 22] 1 is an example implementation of a resource repository according to an embodiment. [Figure 23]FIG. 4 is a flow diagram of a resource allocation process according to an embodiment. [Figure 24A] 1 is an example implementation of a repository for user playability profiles according to an embodiment. [Figure 24B] 4 is an exemplary table of resource configurations according to an embodiment. [Diagram 25] FIG. 4 is a flow diagram of a resource adjustment process according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0014] Like reference numbers refer to corresponding parts throughout the drawings.
[0015] Detailed Description The embodiments described herein are directed to providing a cloud platform and APIs that enable efficient, portable, and low latency hosting of cloud gaming content, including third party gaming content. An embodiment dynamically allocates cloud gaming hardware resources (e.g., CPU, GPU, memory, I / O, and video stream encoders) and monitors and utilizes available network bandwidth to individual end users to simultaneously provide an optimal online gaming experience to a community of game players. An embodiment provides multiple performance tiers, including a tier that supports high performance real-time gaming sessions with high definition media streams for end users. An embodiment is configured to support different subscription models and / or provide one or more simultaneous real-time gameplay and / or review media streams with little or no latency to one or more actual game streams (e.g., video streams output to client devices of users participating in an online / cloud gaming session via either a video stream mobile application or a browser-based program). In an embodiment, the real-time gameplay and / or review media streams are provided to one or more users with little or no latency via a media streaming site such as YouTube.
[0016] In one embodiment of a cloud gaming environment, a server system processes player input. and providing hardware resources for a real-time interactive gaming session to generate an output stream for display to one or more players and optionally a game spectator. In response to a request to establish a real-time interactive gaming session, the server device determines the device capabilities (e.g., hardware and / or software capabilities) of the requesting client device (i.e., the player's controller device), the connection capabilities (e.g., bandwidth, latency and / or error rate) of the player network connections, and one or more target quality parameters (e.g., output video stream resolution, game response latency, etc.) of the gaming session, and accordingly associates one of its virtual machines with the real-time interactive session to establish the session.
[0017] In one embodiment, the processing and encoding capabilities of the game data video stream (e.g., to generate output video streams for players and / or spectators) are managed for one or more processing cores (e.g., a GPU core and an encoder core) within a server system that hosts a real-time, online, and interactive gaming environment. For example, in one embodiment, the one or more processing cores (e.g., 16.67 GPU cores and 16.67 encoder cores, respectively) are managed for one or more processing cores (e.g., 16.67 GPU cores and 16.67 encoder cores, respectively) within a server system that hosts a real-time, online, and interactive gaming environment. The system operates with multiple processing slices (executing on a core for 1000 ms) and the server system assigns each of the multiple processing slices to a subset of the multiple online game sessions to be executed thereon. For one of the processing slices, the server system determines a time-sharing processing schedule such that the corresponding subset of game sessions share the duty cycle of the processing slice and execute in parallel according to their respective real-time data processing needs. Furthermore, to expedite image encoding within a time interval, the encoder of the server system does not need to wait until the GPU makes available all data for an image frame. Rather, in one embodiment, a portion of an image frame is encoded as soon as the information required to encode that portion is provided by the GPU, regardless of whether other portions of the image frame unrelated to the encoded portion are made available by the GPU.
[0018] Furthermore, the server system can dynamically generate some frames in response to user commands received from users playing the online game session. The server system determines an expected response delay time, an actual communication and processing delay time, and an actual transmission delay time depending on the type of user command. The user command is then executed in the online game session by generating a set of frames reflecting the effect of the command. The set of frames when transmitted at a given frame rate occupies a transmission time corresponding to the actual transmission delay time and can be received at the user's client device within a time corresponding to the expected response delay time.
[0019] 1 is an exemplary online interactive gaming environment 100 according to an embodiment. The online interactive gaming environment 100 includes one or more client devices (e.g., client devices 102 and 104). Each client device 102 executes one or more game applications. Game sessions may be executed on a particular game application to allow users of the client devices 102 to play online interactive games hosted by a server system 114. In an embodiment, a client device 102 (e.g., a host client) is configured to invite one or more other client devices 102 to participate in a game scene of a particular game application. The game sessions of these client devices 102 are synchronized to display the same game scene, optionally with separate perspectives corresponding to each user.
[0020] Conversely, the server system 114 may host one or more The server system 114 hosts an online interactive gaming platform to support the client devices 102 to play one or more game applications. Specifically, the server system 114 includes a plurality of user accounts associated with the client devices 102 and authenticates users of the client devices associated with each of the one or more game applications. The server system 114 renders and refreshes scenes of the online interactive games on the client devices 102 that participate in corresponding game sessions associated with the scenes. In one embodiment, the server system 114 evaluates the capabilities of the client devices 102 and / or the quality of the communication connection between the server system 114 and each of the client devices 102, and adaptively generates a synchronous data stream for the game session associated with the client devices 102. By these means, the server system 114 is configured to facilitate synchronous game sessions of the online interactive games on two or more client devices 102 simultaneously and with substantially low latency.
[0021] In one embodiment, the server system 114 includes a game server 122 and a media streaming server 124. The game server 122 is configured to simultaneously provide two or more media streams for an online interactive game session executed on the first client device 102A. The two or more media streams include a low-latency stream and a normal-latency stream that are provided to the first client device 102A and the reviewer client device 104, respectively, via one or more communication networks 112. Optionally, the normal-latency stream is provided for instructional purposes. While a user of the first client device 102 is playing a game session on the first client device 102A, the game session is recorded and broadcast to one or more spectators via the normal-latency stream, i.e., the spectators can review the game session on the reviewer client device 104. The low-latency stream corresponds to the gameplay of the online interactive game session and has a faster response rate and lower transmission latency than the normal-latency stream corresponding to the associated review session. For example, a low latency stream may have a predefined frame rate of 60 frames per second (fps) and may include a time interval of 16.67 milliseconds. The normal delay stream has a predefined frame rate of 30 fps and a frame rate of 33.33 ms. During each time interval, at least one frame is provided to the reviewer client device 104. In one embodiment, the normal latency stream has a lower resolution than the low latency stream.
[0022] In an embodiment, the client device 102 or 104 has a display screen integrated therein for displaying media content. In an embodiment, the client device 102 or 104 is coupled to a media device 106 and an output device 108. Specifically, the client device 102 or 104 may be communicatively coupled to the media device 106 directly (e.g., via a Bluetooth® or other wireless communication link), via a local network 110 (e.g., a Wi-Fi® network), or via one or more communication networks 112. In an embodiment, the client device (102 or 104) and the media device 106 are local to each other (e.g., in the same room, in the same house, etc.). The media device 106 is further coupled to one or more output devices 108 (e.g., a television, a display monitor, a sound system, speakers, etc.) capable of outputting visual and / or audio content. The media device 106 is configured to output content to the output device 108. In one embodiment, the media device 106 may be a casting device (e.g., a Chromecaset® by Google Inc.) or may otherwise be capable of performing a casting function. It is an apparatus that includes a monitoring function.
[0023] In an embodiment, one or more of the client devices 102 or 104 can communicate data and share information with each other, with a central server or cloud computing system (e.g., server system 114), and / or with other networked devices (e.g., another client device 102 or 104, a media device 106, and an output device 108). Data communication can be accomplished using a variety of custom or standard wireless protocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave, Bluetooth Smart, ISA, IEEE 802.15.4, IEEE 802.15.5, IEEE 802.15.6, IEEE 802.15.7, IEEE 802.15.8, IEEE 802.15.9, IEEE 802.15.10, IEEE 802.15.11, IEEE 802.15.12, IEEE 802.15.13, IEEE 802.15.14, IEEE 802.15.15, IEEE 802.15.16, IEEE 802.15.17, IEEE 802.15.18, IEEE 802.15.19 ... 100.11a, WirelessHART, MiWi, etc.), and / or various The communication may be performed using either a custom or standard wired protocol (e.g., Ethernet, HomePlug, etc.), or any other suitable communication protocol, including communication protocols not yet developed as of the filing date of this document. In one embodiment, the online interactive gaming environment 100 includes a conventional networking device (e.g., a router) through which a set of client devices 102 and 104 and their corresponding media and output devices (if any) are communicatively coupled to one another on a local network 110 (e.g., a local area network), which in turn is communicatively coupled to a communication network 112 (e.g., a wide area network and the Internet). In one embodiment, each of the client devices 102 and 104 optionally communicates with one or more other client devices, their respective media devices 106, or their respective output devices 108 using one or more wireless communication networks (e.g., ZigBee, Z-Wave, Insteon, Bluetooth, Wi-Fi, and / or other wireless communication networks).
[0024] In an embodiment, the client devices 102 are separate from one another, i.e., they are not located in the same room or structure. A game may be initiated by launching a game application (e.g., game application 228 of FIG. 2) to run on each client device 102. In an embodiment, for each client device 102, the game application independently establishes an online game session 116 with the server system 114. The online game sessions 116 of two or more client devices 102 (e.g., 102A and 102BB) are associated with one another (e.g., because they are played in the same game area of the game application) and thus share a game scene in the game application. The associated online game sessions 116 are synchronized with one another, with each online game session 116 showing the same game scene, optionally with a unique player perspective or perspective corresponding to the respective client device 102. Thus, a user of each client device 102 can play a game on the respective client device and affect the output from the online game session 116 on the other client devices 102.
[0025] Alternatively, in some other embodiments, after the game application of the first client device 102A establishes the online game session 116, one or more second client devices 102B are invited to join the online game session 116 by an invitation message, e.g., a message having a link (e.g., a URL address) for joining the online game session 116 is sent to each of the second client devices 102B. An appropriate controller configuration is provided to each second client device 102B that is invited to join the online game session 116. In this application, when the second client device 102B joins the online game session 116, the server system 114 creates a separate game session 116 for each individual second client device 102B. Each second client device The separate game session 116 of the first client device 102B may be synchronized with the game session 116 of the first client device 102A and share the same scene, but with a unique player perspective corresponding to each second client device 102B. After each second client device 102B receives the appropriate controller configuration and joins (or, more precisely, starts) its associated online game session 116, the user may play games on the respective second client device 102B and affect the output of the online game session 116 running on the other client devices 102.
[0026] The client device 102 is a device capable of including and executing one or more separate user applications, including gaming applications. In an embodiment, the client device 102 is a smartphone, a tablet device, a laptop computer, a desktop computer, or a multimedia device. In an embodiment, the client device 102 is a dedicated game controller that includes game controls (e.g., one or more buttons, joysticks, touch screen affordances, motion controls, pressure controls, vision controls, voice controls, and / or other haptic interfaces) configured to control some aspect of game play when activated or otherwise manipulated. In an embodiment, the client device 102 includes one or more user applications configured to operate in conjunction with the media device 106. In an embodiment, the applications include a media device application for pairing the client device 102 with the media device 106 and configuring the media device 106. The applications also include one or more applications capable of casting related content to the media device 106. In an embodiment, an application casts data and / or content to the media device 106 by sending the data / content directly to the media device 106 (e.g., over a local network) and / or by directing the media device 106 to a remote location (e.g., a URL or other link to a location at a server system) from which the media device 106 can stream or otherwise receive the data / content. The media device 106 receives the data / content from the application and / or remote location and outputs visual and / or audio content corresponding to the received data / content to the output device 108. Thus, an online game session 116 is established between the game application running on the client device 102, the remote server system 114, and the media device 106.
[0027] In one embodiment, as part of the process of linking associated online game sessions 116, the server system 114 evaluates the capabilities of each corresponding client device 102 and / or the quality of the communication connection between the server system 114 and the client device 102. In one embodiment, the server system 114 measures the network latency between the client device 102 and the server system 114. If the measured latency is above a threshold and a low latency connection is available, the server system 114 can suggest that the client device 102 change to the low latency connection or can prompt the user of the client device 102 to change to the low latency connection. For example, if the client device 102 is on a cellular wireless connection 118 and a local network is available, the server system 114 can suggest that the client device 102 should connect through the available local network. In one embodiment, the latency threshold requirement varies between games. For example, some games (e.g., action games) are best experienced with a low latency connection, while some other games (e.g., online board or card games) are less demanding with respect to latency. The server system 114 may provide a number of different services associated with different types of games. Connection recommendations can be made taking into account these different requirements.
[0028] In an embodiment, as part of the client device 102 initiating or joining a game session 116, the server system 114 communicates with the client device 102 to configure a controller (e.g., game controller configuration and / or interface) on the client device 102. In an embodiment, this involves the server system 114 evaluating whether the client device 102 has the resources and communication capabilities required for the controller. Depending on the resources available at the client device 102, the connection quality, and the requirements for the game, the controller may be implemented differently at the client device 102. In an embodiment, the game may be played using a web page-based controller interface. For example, the controller interface for the game may be embedded in a web page, which is rendered in a web browser on the client device 102. Alternatively, in an embodiment, a standardized controller is implemented in a predefined application that is not specific to the game or directly associated with the game (e.g., a casting device application such as Chromecast or Google Cast by Google, Inc., or other media device application) or in the operating system of the client device 102. For example, a device operating system or a given application on the client device 102 may have a controller sub-module. The controller sub-module includes one or more standardized controller configurations, templates, etc. Each of the standardized controller configurations configures the controller sub-module to utilize input devices and / or sensors on the client device 102 in some manner to implement a virtual controller. The standardized controller configurations may vary by game and / or type of client device.
[0029] Further, in some embodiments, a game has a specific controller configuration that may be implemented on a controller sub-module. Such configuration may be stored on the server system 114 and transmitted to the client device 102 as part of the process of the client device 102 joining or initiating an online game session 116. In some embodiments, the specific controller configuration may be a completely custom controller or a mix of standard and custom controllers. Further, in some embodiments, a game requires a specific application associated with the game. For example, a game may require a controller application specifically associated with the game. In some embodiments, the client device 102 may be instructed to download a specific application or a predefined application as part of initiating or joining a session 116. For example, if the client device 102 does not already have a predefined application (having a controller sub-module) or a specific application associated with the game and such an application is required for play, the server system 114 prompts the user of the client device 102 that a download is required and instructs the client device 102 to request the user's permission to proceed.
[0030] In one embodiment, the server system 114 stores user information associated with each user account of one or more game applications (e.g., game application 228 of FIG. 2) hosted on the server system 114. Examples of user information include, but are not limited to, user account information (e.g., identification and password), membership type, preferences, and activity history. In one embodiment, the server system 114 stores session data associated with online game sessions played on the client devices 102. Each online game session 116 Examples of session data for include, but are not limited to, frame rate, rendering specifications, typical latency requirements, GPU allocation information, encoder allocation information, an identification of the associated session, and latest status information.
[0031] In an embodiment, the server system 114 provides a gaming API and cloud platform to enable efficient, portable, low latency hosting of third party gaming content used in the online gaming session 116. In an embodiment, the gaming API and cloud platform is enabled by the server system 114, which further includes one or more of the front end server 134, the media streaming server 124, the gaming server 122, and one or more third party content servers 136. In an embodiment, the gaming API platform is created and / or hosted by the gaming server 122 and enables the gaming session 116 together with the front end server 134 and the content server 136. The front end server 134 is configured to provide services to users of the gaming session 116 and manage users' accounts. Optionally, users subscribe to gaming services via the front end server 134. The content server 136 provides gaming content associated with the gaming session 116.
[0032] In one embodiment, the front-end server 134 manages user accounts associated with the client devices 102 and 104, e.g., subscriptions by the user accounts to membership in one or more online interactive games. After the client devices 102 log into their respective user accounts and join their online game sessions 116, the game server 122 sets up the game sessions 116 and manages each particular game session 116 for each client device 102 by retrieving game content from the content server 136, transmitting the game content to a game application running on the client device 102, identifying user requests or actions, rendering gameplay output for the client device 102 in response to the user requests or actions, and storing game state data during each game session 116. The game server 122 includes one or more processing units (e.g., a CPU 138, a GPU 140, and an encoder 142), a memory 146, and a data buffer 144 for temporarily storing multimedia content generated by the GPU 140, and further The data buffer 144 optionally provides the multimedia content to an encoder 142 for encoding (eg, normalization or compression). The data buffer 144 is optionally integrated with or separate from the memory 146.
[0033] In an embodiment, the game server 122 dynamically allocates cloud gaming hardware resources (e.g., GPU 140 and encoder 142) and monitors and utilizes available network bandwidth to individual end users to provide an optimal cloud gaming experience. In an embodiment, the game server 122 provides multiple performance tiers, including a tier supporting high performance real-time gaming sessions with high definition video / media streams. In an embodiment, the game server 122 is configured to support different subscription models and / or provide one or more simultaneous real-time gameplay and / or review media streams corresponding to one or more actual game streams with little or no latency (e.g., video streams output to client devices of users participating in an online / cloud gaming session via either a video stream mobile app or a browser-based program). Specifically, the game server 122 is configured to generate simultaneous media streams for gameplay and review video, and the media streaming server 104 is provided with the review video for the simultaneous gameplay. Such review video is then sent to the game server 122 via the game server 122. The video is provided with little or no delay to one or more users via a media streaming site, such as YouTube, optionally managed by a media streaming server 124.
[0034] Some embodiments enable the hosting of public events in conjunction with game competitions. For example, in connection with a multiplayer game event or competition based on a hosted game, a cloud gaming site hosted by game server 122 may transmit to selected reviewer client devices 104, optionally via media streaming server 123, (a) relevant (b) one or more simultaneous ancillary or supplemental media streams, including an accompanying commentary track / stream; (c) game streams from different player perspectives; (c) a highlight stream showing particularly exciting game action based on cloud server analysis; scoring of multiple game sessions associated with a game event; (d) one or more game perspective streams reflecting gameplay sessions 116 of a plurality of active gamers; and / or (e) a cluster of game perspective streams represented by the active gamers. One or more action streams, possibly including real-time picture-in-picture (PIP) video and their corresponding game play responses, are sent to the wood game server system 114. Instruction tracks from active gamers and / or commentators may be broadcast or streamed.
[0035] According to an embodiment, examples of third-party content that may be effectively hosted by content server 136 include, but are not limited to, sports games, racing games, role-playing games (RPGs), and first-personal shooter (FPS) games. Different instances of these games may have widely varying cloud hardware requirements and network and network bandwidth requirements (e.g., to ensure an optimal user gaming experience consistent with different subscription performance tiers) based on different associated latency requirements and expectations, output video resolution, game server computational load and video encoding / streaming resources.
[0036] In an embodiment, the front-end server 134 provides an account management API and / or software module that monitors subscriber gameplay activity and associated requests (e.g., an end user requesting to invite other players to join a game session, upgrade in-game tools, and / or improve game performance) and transmits or makes available API-related information to the third-party content server 136 to enable the content provider to track the settings (including, but not limited to, billing information, in-game credits, subscription levels, etc.) of their subscribers and / or followers. In an embodiment, a content provider of hosted content may offer one or more different subscription models for hosted content via the same hosting platform. In an embodiment, a user (e.g., a subscriber to a gaming service) is allowed unlimited access and gameplay to all games offered by the content provider on the hosting platform. In an embodiment, a user is allowed unlimited access and gameplay to one or more specific game franchises (e.g., a specific football or first person shooter franchise) offered by the content provider on the hosting platform. In one embodiment, subscriptions are for limited participation by a user, which may be limited based on gameplay time, level of hardware resources committed to the end user, or type / location of the end user's device. In one embodiment, the Account API and module configure and monitor gameplay sessions, allowing content providers to track each subscriber's gaming activity according to their most current subscription information, even during active gameplay. to.
[0037] The server system 114 enables cloud functionality that allows a user to move around and continue the first game session, such as pausing a first game stream of a first game session running on a first client device 102 and resuming the first game stream on a second game session on a second client device 102. The server system 114 also supports multiple players on a large scale, providing a richer and more persistent cloud-based world. The server system 114 uses a cloud-based system to store session data associated with different game sessions 116 of the same user or different game sessions 116 of different users.
[0038] The server system 114 displays the game content on multiple client devices 102 and 104, including, but not limited to, mobile phones, tablet computers, desktop computers, and televisions. Optionally, the game content is dynamically adjusted to conform to the specifications of these client devices 102 and 104. In one embodiment, the client devices 102 and 104 have limited or no storage capabilities because the game API platform provides instant access and requires little or no user device storage (e.g., a user can start playing in 5 seconds and save 250 GB of console hard drive space).
[0039] In addition to game content, the server system 114 also streams to the client devices 102 and 104 add-on content, such as new league rosters, statistics, and preview access to initial titles, which are optionally updated periodically (e.g., daily or hourly, and easily updated and upgraded). In one embodiment, the add-on content includes search results of an Internet search or a database search.
[0040] In an embodiment, the server system 114 supports a live online community associated with the gaming application. Users (e.g., subscribers of the service) participate in live events, tournaments, or activities on a corresponding gaming API platform throughout the day. Examples of live events, tournaments, or activities include watching live gaming sessions played by other users, posting achievements to the public domain (e.g., YouTube), and getting live tips and coaching videos. For example, the game server 122 provides two or more live streams 130 and 132 in response to user actions. While maintaining a first game stream 130 on the first gaming session 116 of the first client device 102A for the game player, the server system 114 also broadcasts a second live review stream 132 (e.g., a YouTube stream) to one or more other client devices 104 (e.g., of the subscribers). The second live review stream 132 allows the user to share his / her gaming experience with the audience. Optionally, the second live stream is a playback of the screen of the player's first client device 102A. The server system 114 may obtain an audio stream of the players describing the first game session 116, or a video stream of the players playing and describing the first game session 116. The audio stream is optionally played for the audience while the second live review stream 132 is played for the audience. The video stream is optionally played in an embedded window within the second live review stream 132.
[0041] Some embodiments provide on-the-go gaming, allowing users to take their desired games to any location or client device. For example, a user may A user can start an online game session 116 at his / her origin on the mobile device 102A and then seamlessly resume the game session 116 at his / her destination on the laptop computer 102B. Also, in an embodiment, based on the resources of different client devices available to the user when the game session 116 is handed over between different devices 102, the server system 114 (particularly the game server 122) can dynamically deploy different sets of hardware resources (e.g., GPU 140 and encoder 142) to optimize the user's gaming experience based on the resources (e.g., client hardware capabilities and network bandwidth) of different end users' current devices.
[0042] In the server system 114, the front-end server 134 and the game server 122 can have their own user account systems. In one example, the user account system for the front-end server 134 is used to manage subscriptions to specific game content and services, and the user account system for the game server 122 (e.g., YouTube or Google accounts) is used to manage the game experience (e.g., rendering game content to meet specific game criteria) and many other purposes. In an embodiment, these two user account systems share customer and usage data (e.g., social, friends, presence, authentication, account information, billing information). The content front-end server 134 also provides a service layer that sits on top of the technology layer enabled by the game server 122. In an embodiment, the game content server manages an additional user account system for accessing their content. Optionally, the additional user account system for the game content is integrated with the user account system for the front-end server 134 that manages user subscriptions.
[0043] FIG. 2 is a block diagram illustrating an exemplary client device 102 of a gaming environment 100 according to an embodiment. Throughout this application, unless otherwise specified, references to a client device 102 correspond to one or more of the client devices 102A, 102B, and 104 described with reference to FIG. 1. Examples of client devices 102 include, but are not limited to, mobile phones, tablet computers, laptop computers, desktop computers, and wearable personal devices. In an embodiment, the client device 102 is a dedicated game controller that includes game control inputs 210 (e.g., one or more buttons, joysticks, touch screen elements, motion controls, pressure controls, vision controls, voice controls, and / or other tactile interface elements configured to control some aspect of game play when activated). The client device 102 includes one or more processing units (CPUs) 202, one or more network interfaces 204, memory 206, and one or more communication buses 208 (sometimes referred to as chipsets) for interconnecting these components. The client device 102 includes one or more input devices 210 that facilitate user input, such as a keyboard, mouse, voice command input unit or microphone, touch screen display, touch sensitive input pad, gesture capture camera, or other input buttons or controls. Additionally, some client devices 102 may use a microphone and voice recognition, or a camera and gesture recognition, to supplement or replace an interface that requires contact (e.g., keyboard and buttons). In an embodiment, the client device 102 includes one or more cameras, scanners, or photo sensor units, for example, to capture images of a graphic series code printed on an electronic device. In an embodiment, the client device 102 includes one or more output devices that enable the presentation of a user interface and the display of content, including one or more speakers and / or one or more visual displays. Optionally, the client device 102 includes a force device 212. Optionally, the client device 102 includes a location detection device 214, such as a GPS (Global Positioning Satellite) or other geographic location receiver, for determining the location of the client device 102. The client device 102 may also include a proximity detection device 215, e.g., an IR sensor, for determining the proximity of the media device 106 and / or other client devices 102. The client device 102 may also include one or more sensors 213 (e.g., an accelerometer, gyroscope, etc.) for sensing movement, orientation, and other parameters of the client device 102, which may be used as inputs (e.g., for input 210 described above).
[0044] The memory 206 may be DRAM, SRAM, DDR RAM, or other random access 202。 Memory 206 optionally includes one or more storage devices located remotely from the one or more processing units 202. Memory 206, or alternatively, the non-volatile memory in memory 206, includes a non-transitory computer-readable storage medium. In an embodiment, memory 206, or the non-transitory computer-readable storage medium of memory 206, stores the following programs, modules, and data structures, or a subset or superset thereof: An operating system 216 that contains instructions for handling various basic system services and for performing hardware-dependent tasks; one or more network interfaces 204 (wired or wireless) and a network communications module 218 for connecting the client device 102 to other devices (e.g., server systems 114, media devices 106, and other client devices 102) via one or more networks 110 and / or 112, such as the Internet, other wide area networks, local area networks, metropolitan area networks, etc.; a user interface module 220 for enabling presentation of information (e.g., applications, widgets, websites and their web pages, and / or graphical user interfaces for presenting games, audio and / or video content, text, etc.) on the client device 102 via one or more output devices 212 (e.g., a display, speakers, etc.); an input processing module 222 for detecting one or more user inputs or interactions from one of the one or more input devices 210 and interpreting the detected inputs or interactions; an input event reporting module 223 for reporting input identification and / or timestamp information to the server system 114 for use in the delay time calculation; a web browser module 225 for navigating, requesting (e.g., via HTTP), and displaying websites and web pages, including a web interface for participating in the session 116; a media device application 226 for interacting with the media device 106, including logging into a user account associated with the interactive media device 106, controlling the media device 106 when associated with the user account, and editing and reviewing settings and data associated with the media device 106; a game application 228 for providing a game on the client device 102, including facilitating corresponding game play and facilitating the invitation of additional players; a game controller module 230 for providing a game play input interface to the game application 228; a data download module 231 for downloading data (e.g., the game controller configuration 456 (FIG. 4), the game application 228 and other applications, module and application updates, and data in the memory 206) from the server system 114 and other content hosts and providers; Client device data 232 that stores data associated with the gaming application 228 and other applications / modules, including at least the following: - client device settings 234 for storing information associated with the client device 102 itself, including common device settings (e.g., service tier, device model, storage capacity, processing capabilities, communication capabilities, etc.); - a media device settings 236 for storing information associated with a user account of the media device application 226, including one or more of account access information and information for device settings (e.g., service tier, device model, storage capacity, processing capabilities, communication capabilities, etc.); - game application settings 238 for storing information associated with a user account of the game application 228, including one or more of account access information, in-game user preferences, game play history data, and information about other players; - a game controller configuration 240 for storing information associated with the configuration of the game controller module 230 for the game application 228 (e.g., a configuration received from the game controller configuration 456 of FIG. 4); Location / proximity data 242, which includes information associated with the presence, proximity or location of either the client device 102 and the media device 106.
[0045] In one embodiment, the game controller module 230 is part (e.g., a sub-module) of the media device application 226 or another application in memory 206. In one embodiment, the game controller module 230 is part of the operating system 216. In one embodiment, the game controller module 230 is a separate module or application.
[0046] In some embodiments of the client device 102, the media device application 226 (and corresponding media device settings 236) and the game application 228 (and corresponding game application settings 238) are optional. Depending on the particular game that the client device 102 is invited to participate in, the media device application 226 and the game application 228 may not be required to play. If any of these applications are necessary to play the game (e.g., the game uses the game controller module 230 in the media device application 226) and the application is not in memory 206, the client device 102 may be prompted to download the application.
[0047] Each of the above-identified elements may be stored in one or more of the aforementioned memory devices and corresponds to a set of instructions for performing the functions described above. The above-identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures, modules, or data structures, and thus various subsets of these modules may be combined or otherwise rearranged in various embodiments. In an embodiment, memory 206 optionally stores a subset of the above-identified modules and data structures. Additionally, memory 206 optionally stores additional modules and data structures not described above.
[0048] FIG. 3 is a block diagram illustrating an example media device 106 of the gaming environment 100 according to one embodiment. The media device 106 typically includes one or more processing units ( The client device 102 may include a processor (CPU) 302, one or more network interfaces 304, a memory 306, and one or more communication buses 308 for interconnecting these components (sometimes referred to as a chipset). Optionally, the media device 106 includes a proximity / location detection unit 310, such as an IR sensor, for determining the proximity of the client device 102.
[0049] Memory 306 includes high-speed random access memory, such as DRAM, SRAM, DDRRAM, or other random access solid-state memory devices, and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile semiconductor storage devices. Memory 306 optionally includes one or more storage devices located remotely from the one or more processing units 302. Memory 306, or alternatively, the non-volatile memory within memory 306, includes a non-transitory computer-readable storage medium. In an embodiment, memory 306, or the non-transitory computer-readable storage medium of memory 306, stores the following programs, modules, and data structures, or a subset or superset thereof: An operating system 316 that contains instructions for handling various basic system services and for performing hardware-dependent tasks; one or more network interfaces 304 (wired or wireless) and a network communications module 318 for connecting the media device 106 to other computers or systems (e.g., server system 114 and client device 102) via one or more networks 110 and / or 112, such as the Internet, other wide area networks, local area networks, metropolitan area networks, cable television systems, satellite television systems, IPTV systems, etc.; a content decoding module 320 for decode content signals received from one or more content sources (e.g., the server system 114 for output from the game session 116) and outputting the content in the decoded signals to an output device 108 coupled to the media device 106; a proximity / location determination module 322 for determining the proximity of the client device 102 based on proximity related information detected by the proximity detection unit or provided by the server system 114; a media presentation module 324 for controlling the presentation of media; a display event reporting module 325 for reporting display event identification and / or timestamp information to the server system 114 for use in the delay time calculation; a latency calculation module 326 for calculating latency values based on latency data 334 reported by other components within the gaming environment; Media device data 328 for storing data including at least the following: - a media device configuration 330 for storing information associated with a user account of a media device application, including one or more of account access information and information for device settings (e.g., service tier, device model, storage capacity, processing capabilities, communication capabilities, etc.); - location / proximity data 332, which includes information associated with the presence, proximity or location of either the client device 102 and the media device 106; - Delay data 334, which includes information (eg, timestamps) necessary for the delay calculation module 326 to calculate delay values.
[0050] Each of the above-identified elements may be stored in one or more of the aforementioned memory devices and corresponds to a set of instructions for performing the functions described above. The above-identified modules or programs (i.e., sets of instructions) may be implemented as separate software programs. The modules and data structures need not necessarily be implemented as individual programs, procedures, modules, or data structures, and thus various subsets of these modules may be combined or otherwise rearranged in various embodiments. In an embodiment, memory 306 optionally stores a subset of the modules and data structures identified above. Additionally, memory 306 optionally stores additional modules and data structures not described above.
[0051] FIG. 4 is a block diagram illustrating an exemplary server in a server system 114 of a gaming environment 100 according to an embodiment. The server system 114 typically includes one or more processing units (e.g., CPU 138, GPU 140, and encoder 142), one or more network interfaces 404, memory 146, and one or more communication buses 408 interconnecting these components (sometimes referred to as a chipset). The server system 114 may optionally include one or more input devices 410 to facilitate user input, such as a keyboard, a mouse, a voice command input unit or microphone, a touch screen display, a touch sensitive input pad, a gesture capture camera, or other input buttons or controls. Additionally, the server system 114 may use a microphone and voice recognition, or a camera and gesture recognition to supplement or replace a keyboard. In an embodiment, the server system 114 optionally includes one or more cameras, scanners, or photo sensor units for capturing images of, for example, graphic series codes printed on an electronic device. The server system 114 may also include one or more output devices 412 that enable presentation of a user interface and display of content, including one or more speakers and / or one or more visual displays.
[0052] The memory 146 may be DRAM, SRAM, DDR RAM, or other random access Memory 146 may include high speed random access memory such as solid state memory devices, and optionally non-volatile memory such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile semiconductor storage devices. Memory 146 optionally includes one or more storage devices located remotely from the one or more processing units. Memory 146, or alternatively, the non-volatile memory in memory 146, includes a non-transitory computer readable storage medium. In an embodiment, memory 146, or the non-transitory computer readable storage medium of memory 146, stores the following programs, modules, and data structures, or a subset or superset thereof: An operating system 416 that contains instructions for handling various basic system services and for performing hardware-dependent tasks; a network communications module 418 for connecting the server system 114 to other devices (e.g., the server system 114, the client devices 102, and various servers in the media devices 106) via one or more network interfaces 404 (wired or wireless) and one or more networks 110 and / or 112, such as the Internet, other wide area networks, local area networks, metropolitan area networks, etc.; A user interface module 420 for enabling presentation of information on the client device 102 (e.g., applications, widgets, websites and their web pages, and / or graphical user interfaces for presenting games, audio and / or video content, text, etc.); A media device module 422 (optional) that executes to provide server-side functionality for device provisioning, device control, and user account management associated with the media device 106; Proximity to determine the proximity of the client device 102 to the media device 106 based on location information of either the client device 102 or the media device 106 degree / position determination module 424; a game server module 426 for providing server-side functionality associated with a game (e.g., game application 228), including, but not limited to, setting up game sessions, storing session state data and other game-associated data, processing gameplay input from client devices 102, and displaying gameplay output in response to such input; a media streaming server module 438 for hosting a media streaming site, receiving concurrent or auxiliary media streams associated with the online game session, and providing the concurrent media streams to a client device 104 for simultaneous display with the online game session running on the game application 228 of the same client device 104 or a separate client device 102; a front-end server module 440 for managing user accounts associated with the client device 102, e.g., subscriptions by user accounts to memberships in one or more online interactive games, for forwarding subscriber requests to the game server module 426, for enabling services to subscribers, and for monitoring subscriber game play activity and associated requests; a media content server module 442 for providing access to gaming content hosted by one or more third party content providers; A device / network assessment module 444 for assessing the device and network capabilities of the client device 102, including, but not limited to, assessing the network bandwidth of the connection to the client device 102 and whether the client device 102 has the necessary modules or applications to play the game; A data transmission module 446 for providing data (e.g., game controller configuration 456, software updates, etc.) to the client device 102; Server system data 448, including: - client device settings 450 for storing information associated with the client device 102, including common device settings (e.g., service tier, device model, storage capacity, processing capabilities, communication capabilities, etc.); - media device settings 452 (optional) for storing information associated with a user account of the media device application 422, including one or more of account access information and information for device settings (e.g., service tier, device model, storage capacity, processing capabilities, communication capabilities, etc.); - location / proximity data 454, which includes information associated with the presence, proximity or location of either the client device 102 and the media device 106; - a game controller configuration 456 for storing controller configurations for various games; user information 458 for storing information associated with a user account for each of one or more game applications (e.g., game application 228 of FIG. 2) hosted on the server system 114, including, for example, user account information (e.g., identification and password), membership type, preferences, and activity history; - a game session event log 460 for storing event data associated with the game session (e.g., game state data, input events, display events, other game related data), including, for example, data 460-1 for a first game session and data 460-2 for a second game session, the session data 460 for each game session including, but not limited to, frame rate, rendering specifications, typical latency requirements, GPU allocation information, encoder allocation information, an identification of the associated session, latest status information associated with the respective game session, a log of input events, and a log of display events; - Response time to store expected delay values for various user command types Setting 462, a resource repository 464 for storing virtual machine resource profiles and container images; - Resource Settings 466 for storing configurations of available resources based on user tolerance levels; A data buffer 144 for temporarily storing gameplay multimedia content generated by the GPU 140 associated with one or more output media streams.
[0053] In one embodiment, the game server module 426 includes the following programs, modules, or a subset or superset thereof: an intent determination module 428 for determining the user's intent behind a particular input by comparing user input transit times (e.g., between the client device 102 and the server system 114) with display transit times (e.g., between the media device 106 and the server system 114) and matching input events with their respective trigger frames; A delay time adjustment module 430 for determining the number of intermediate frames for the GPU 140 to insert between (i) a current frame being processed at the time a user input is received and (ii) a response frame indicating the result of the received input; a resource allocation module 432 (optionally referred to herein as a “session orchestrator”) for receiving session requests from endpoints (e.g., the controller 102) and determining which resources to allocate to the session; A resource adjustment module 434 that determines the delay tolerance of a particular user.
[0054] In an embodiment, memory 146 further includes a data buffer 144 configured to couple encoder 142 to GPU 140. In particular, data buffer 144 temporarily stores game play multimedia content generated by GPU 140 associated with one or more output media streams such that encoder 142 can retrieve the game play multimedia content from data buffer 144 and encode the retrieved content into one or more media streams, e.g., for standardization, speed, or compression.
[0055] Each of the above-identified elements may be stored in one or more of the aforementioned memory devices and corresponds to a set of instructions for performing the functions described above. The above-identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures, modules, or data structures, and thus various subsets of these modules may be combined or otherwise rearranged in various embodiments. In an embodiment, memory 146 optionally stores a subset of the above-identified modules and data structures. Additionally, memory 146 optionally stores additional modules and data structures not described above.
[0056] Display Lag Detection and Compensation The various implementations of cloud-based gaming platforms described above provide many benefits (e.g., portability, scalability, efficiency, ease of access and control, etc.). However, the cloud-based nature of these gaming platforms comes with various challenges, such as variability in network and processing resources, which may adversely affect the gameplay experience if not properly accounted for. Such challenges may potentially create a non-uniform gaming experience due to variable latency times introduced in the network 110 / 112 between the player devices 102 and the server system 114. The following disclosure provides an understanding of the different challenges that may exist in a real-time interactive cloud-based gaming environment. Various embodiments are described that detect and compensate for various types of latency. By compensating for these latency times, the embodiments described herein provide a smooth and uniform gaming experience for each player, regardless of available network and processing resources.
[0057] FIG. 5A illustrates an exemplary gaming environment 500 from which some sources of delay time are described. The gaming environment 500 is an exemplary embodiment of the gaming environment 100 (FIG. 1), with corresponding components similarly labeled. The gaming environment 500 includes a client device 102 (also referred to herein as a "game controller" or "controller") that a player (or "user") uses to control various aspects of a game (or "game play"), for example, by activating or manipulating inputs 210 (FIG. 2). The gaming environment 500 also includes a media device 106 (e.g., a set-top box) and an output device 108 (e.g., a television or other output display). The controller 102 and the media device 106 are communicatively coupled to a local network 110 (shown as a wireless router in this example) via local communication links 502 and 504, respectively (e.g., via Wi-Fi). The local network 110 is communicatively connected to a server system 114 via communication link 506, via a communication network 112 (e.g., the Internet). The server system 114 includes a game server 122 (FIG. 1).
[0058] Although the gaming environment 500 shown in the drawings includes only a single local network 110 with a single controller 102, some embodiments of the gaming environment 500 may include multiple local networks 110, some of which include more than one controller 102 (e.g., in the case of multiplayer games sharing the same game session, as described with reference to Figures 1-4 above).
[0059] Several elements present within the gaming environment 500 can introduce latency that is perceptible (e.g., affects at least one frame) and time-varying. For example, the local network 110 (e.g., Wi-Fi) can introduce varying amounts of latency into the communication links 502 and 504. When there is no contention on the channel, the average latency is very low (e.g., less than 1 millisecond). However, in busy environments such as an apartment building with overlapping Wi-Fi networks, or a gaming environment with multiple wireless client devices, average latency amounts in the range of 10-50 milliseconds are more common, with outliers in the 200 millisecond range.
[0060] Additionally, the communications network 112 (e.g., the Internet) can introduce latency into the communications link 506. This latency may be less variable than Wi-Fi for most users, however, during peak gaming hours (evenings), media sharing (e.g., over cable modems) as well as network saturation can result in delayed or dropped packets. The average latency depends on the distance from the local network 110 to the edge server of the server system 114, with the amount of latency ranging from, for example, 20-30 milliseconds.
[0061] The latency introduced in the above-described networks may vary based on the direction of traffic flow (e.g., from the controller 102 to the server 122, or conversely, from the server 122 to the media device 106) due to asymmetries in network demand and link capacity. Thus, the latency on the link 506 from the router to the server may not match the latency from the server back to the router, etc.
[0062] The game server 122 may also introduce latency. There is a latency between the arrival of an input event at the GPU 140 and the output of a frame from the encoder 142. However, in one embodiment, this delay time is fully traceable and, as a result, known by the game server 122 .
[0063] Finally, there is a delay between the arrival of a frame at the output device 108 (e.g., a television) and the display of that frame. This may depend on the nature of the processing at the output device, including the display mode (e.g., gaming mode vs. non-gaming mode). For example, a television may have a display lag on the order of 15-30 ms, or on the order of 50-60 ms. A bad television may have a display lag of 120 ms.
[0064] The different types of latency mentioned above can have a significant impact on the gameplay experience. Figures 5B and 5C show two example gameplay experiences that involve the same user input but result in completely different outputs due to different latency levels. However, before describing these examples in detail, an example of the gameplay process needs to be described first.
[0065] Delay Time Compensation 6 is a flow diagram of a game play process 600 according to one embodiment. The process may be performed in an electronic server (e.g., server system 114, or more specifically, game server 122) having one or more processors (e.g., CPU 138 and / or GPU 140) and a memory (e.g., memory 146) storing one or more programs for execution by the one or more processors, a media device (e.g., media device 106) having one or more processors (e.g., CPU 302) and a memory (e.g., memory 306) storing one or more programs for execution by the one or more processors, and / or a user device (e.g., controller 102) having one or more processors (e.g., CPU 202) and a memory (e.g., memory 206) storing one or more programs for execution by the one or more processors. In one embodiment, the server, media device, and user device include one or more programs and a memory storing one or more respective programs for execution by the one or more respective processors, the one or more programs including instructions for performing process 600. In one embodiment, each non-transitory computer-readable storage medium stores one or more respective programs that, when executed by the electronic server, media device, and user device using one or more respective processors, cause the electronic server, media device, and user device to perform process 600.
[0066] A user of the controller 102 (also referred to herein as a "player") uses the controller 102 to affect events in a game represented by video frames (e.g., 510) displayed on the output device 108 (see FIG. 5A). When the player decides to affect game play (e.g., by moving a virtual player, shooting a hockey puck, etc.), the player activates (602) or otherwise manipulates (e.g., presses a button) an input 210 on the controller 102. The activation or manipulation of an input 210 on the controller 102 is sometimes referred to herein as an "input event" or a "command." The input event is transmitted (604) over communication links 502 and 506 (via networks 110 and 112) to the server system 114 (e.g., to an event log 460 associated with the game session).
[0067] Upon receiving 606 an input event, the server system 114 (e.g., the intent determination module 428 of the game server 122) determines which frame was displayed on the output device 108 at the time the user activated the input associated with the received input event. A frame is then determined (608) for each action that the user has performed. The frame displayed to the user at the time the user activated the input is referred to herein as a "trigger frame" because it triggered the user to respond by activating an input. For example, in a hockey game, if a frame displays an open shot, this triggers the user to respond by activating an input control mapped to a "shoot puck" function. The trigger frame is frame 510 showing an open shot (e.g., frame 510-1, FIG. 5B), and the input event is the user's activation of the "shoot puck" control on controller 102 in response to seeing trigger frame 510.
[0068] Upon determining the trigger frame, the game server 122 (e.g., the intent determination module 428) determines (610) the state of the game at the time the trigger frame was displayed to the user (referred to herein as the “trigger state”). In one embodiment, the intent determination module 428 determines the trigger state by referencing a log of game states maintained in an event log 460 (FIG. 4). In one embodiment, the event log 460 includes a log of game states indexed by frame fingerprints, frame IDs, and / or game time data (e.g., timestamps or clock data). In one embodiment, the intent determination module 428 determines the trigger state by determining a game time index associated with the trigger frame and referencing the event log 460 to determine the state of the game that existed at the time of the game time index associated with the trigger frame. Depending on how much time has passed between displaying the trigger frame on the output device 108 and receiving the input event at the game server 122, the trigger state may be in the past, relative to the current state being processed by the game server 122.
[0069] Returning to the previous example, if a trigger frame (indicating an open shot on the goal) is associated with game time index T1, the state of the game at time index T1 includes the virtual shooter, the virtual defender, the virtual puck, the virtual goal, and the positions of each of these objects. Depending on the state of the game at time index T1, more specifically, the positions of each of the aforementioned virtual objects at time index T1, there is a clear path between the puck and the goal. In other words, one or more algorithms controlling the rules of the gameplay allow the virtual puck to travel from the virtual player shooting the puck to the virtual goal at the moment during the display of the trigger frame (time index T1) without being stopped by any other virtual player between the shooter and the goal. However, in some scenarios, when an input event (e.g., "shoot puck") arrives at the server, the server is currently processing the gameplay at a subsequent state T2, which may include an advanced gameplay state where the virtual puck no longer has a clear path to the goal. In these scenarios, if the server correctly determines the trigger state at T1, the trigger state is a past state with respect to the state T2 that the server is currently processing.
[0070] Upon determining the trigger state, the game server 122 (e.g., GPU 140) processes (612) a subsequent game state (sometimes referred to herein as a "game play output") according to (i) the input event (e.g., "shoot puck") and (ii) the trigger state (e.g., including a clear path from the puck to the goal). In one embodiment, processing the game play output includes inputting the input event into an algorithm or game engine that determines the game play output based on the input event and the corresponding game state. For example, the game engine may determine the next game state based on the state / position of each player and the puck with respect to the goal during the current game state, as well as any input commands (e.g., "move," "shoot," or "block") received for the virtual player during the current game state. In one embodiment, the input event And processing a subsequent game state (gameplay output) in accordance with the trigger state includes processing the input event as if it were available to the server at the time the server was processing a game state proximate to the trigger state (e.g., the next state after the trigger state, or a state that follows close to the trigger state).
[0071] Upon processing the gameplay output, the game server 122 renders (614) a frame or a series of frames that show the processed gameplay output. The frame (or the first of a series of frames) that shows the gameplay output is referred to herein as a "response frame." For example, if the input events and trigger conditions result in a gameplay output that includes movement of a particular virtual player, the response frame is a frame that shows the particular virtual player in a modified spatial position with respect to other objects in the frame that matches the direction specified by the user input. Alternatively, if the input events and trigger conditions result in a gameplay output of a particular virtual player shooting a puck, the response frame is the first of a series of frames that shows the particular virtual player shooting a hockey puck (e.g., frame 510-3, FIG. 5B). In an embodiment, rendering the response frame includes introducing a new virtual object, modifying an existing virtual object, or modifying any other aspect of the gameplay according to the processed gameplay output, including including the new virtual object, the modified existing virtual object, or any other aspect of the modified gameplay in the response frame.
[0072] The server system 114 begins encoding the response frame (e.g., using the encoder 142) and transmitting (616) the encoded response frame to the media device 106. Upon receiving the encoded response frame from the server system 114, the media device 106 decodes (e.g., using the content decoding module 320) the response frame and displays (620) the decoded response frame to the user (e.g., using the output device 108).
[0073] 5B and 5C, two sequences of video frames (510 and 520) are shown illustrating the same input event (shooting the puck) but with different response frames (successful shot 510-2 vs. blocked shot 520-3) due to different amounts of latency present in game environment 500. These sequences are examples of game play process 600 applied to game environment 500.
[0074] FIG. 5B illustrates a first scenario 550 including a sequence of video frames 510 showing three virtual players (A, B, and C) playing a hockey game, and a table 512 of game states T1-T3 (e.g., stored in log 460 of FIG. 4). Player A is controlled by a user of controller 102, and players B and C are controlled by other users of other controllers, by computer-controlled algorithms, or by a combination thereof. In state T1, player A has a clear shot on goal (shown as "clear" in table 512), and therefore the game server sends frame 510-1 indicating this state to the user's display 108. When the user controlling player A sees frame 510-1 on display 108, the user sees that player A has a clear shot on goal, and therefore decides to command player A to shoot the puck. In other words, frame 510-1 triggers the user to input a "shoot" command. The "shoot" command is sent to game server 122 as an input event. When the game server 122 receives the "Shoot" input (shown as "In" in table 512), the game server is currently processing state T2 where Player A no longer has a clear shot (shown as "No Shot" in table 512). The game server 122 correctly determines that the trigger frame (denoted as "T" in table 512) was frame 510-1. According to the state of the game when frame 510-1 is displayed (trigger state T1), player A still had a clear shot on the goal. Therefore, the game server 122 processes the subsequent state T3 according to the "shoot" command and the T1 state (clear shot). According to the game engine, if the player shoots while having a clear shot, the subsequent state includes a successful shooting sequence, which is processed in state T3 (denoted as "score" in table 512). Therefore, the game server renders a response frame 510-2 indicating that player A shooting the puck has passed player C, and sends the response frame to the user. From the user's perspective, the response frame indicates the action intended by the user at the time of the input event. In this way, the game server processes the game play based on the user's intention by correctly determining the trigger state according to the user's input.
[0075] FIG. 5C shows a second scenario 552 including a sequence of video frames 520 showing the same game and players as scenario 550, as well as a table 522 of game states T1-T3 (e.g., stored in log 460 of FIG. 4). As in the previous scenario, in state T1, player A has a clear shot on goal (shown as "Clear" in table 522). Thus, the game server sends frame 520-1 showing this state to the user's display 108. When the user sees frame 520-1 on the screen 108, the user sees that player A has a clear shot on goal, and therefore decides to command player A to shoot the puck. The "Shoot" command is sent as an input event to the game server 122. As in the previous scenario, when the game server 122 receives the "Shoot" input (shown as "In" in table 522), the game server is now processing state T2, where player A no longer has a clear shot (shown as "shot" in table 522). However, unlike the previous scenario, the game server 122 does not precisely determine the trigger frame (denoted as "T" in table 522). Instead, the game server assumes that the trigger frame is the last frame to be rendered according to the current state T2, which in this example is frame 520-2. Alternatively, the game server may The game server may not attempt to determine the frame to be used to determine the current state T2 (no shot), but instead processes the game play output based on the current state T2 (no shot). In either case, the game server processes the subsequent state T3 according to the "shoot" command and the T2 state (no shot). According to the game engine, if the player shoots without a clear shot, the subsequent state will include a blocked shot sequence, which is processed in state T3 (denoted as "Blocked" in table 522). Thus, the game server renders a response frame 520-3 indicating that player A is attempting to shoot the puck but is blocked by player C, and transmits the response frame to the user. From the user's perspective, the response frame indicates an action that the user did not intend at the time of the input event. Specifically, the user intends to have player A shoot while player C was out of the way. Instead, player A did not shoot as quickly as the user intended, resulting in the shot being blocked. Thus, by failing to correctly determine the trigger state that corresponds to the user's input, the game server may process game play events contrary to the user's intentions, which could potentially cause the user (and many other users) to lose interest in playing the game and / or using the game environment 500.
[0076] In each of the two scenarios above, the input events occur simultaneously. However, depending on how long it takes for the input events to reach the game server, the response frames will show two very different outcomes. This is because the server may not know that the user has entered input. This is because if a user input is received while processing a game state (e.g., T2) that is later in time than the game state (e.g., T1) that triggered the server to perform the action, the server may erroneously process the game output based on erroneous information regarding the timing of the user input. Because it is important for the gaming platform to avoid this type of inconsistency, it is important for the gaming platform to detect and compensate for the various delay times introduced into the game environment that cause these delays. By detecting the various delay times, the game playing platform can more accurately correlate input events with the actual trigger state (as in scenario 550). By making these correlations, the gaming platform reduces the effects of uncontrollable and / or undetectable delay times by processing each input event in a manner consistent with the user's intent. Thus, the various embodiments described herein are improvements to gaming platforms that do not attempt to determine or erroneously determine the exact trigger state that corresponds to a user input.
[0077] In certain scenarios, depending on how much time has elapsed between the triggering state and the current state processed by the game server, certain gameplay output may contradict what has already been displayed to one or more users. For example, in FIG. 5C, frame 520-3 shows a blocked shot. However, if the game server were to If, during frame 510-3, the game server determines that the trigger condition was T1, then in one embodiment, the game server attempts to reconcile the user's intent with the current state of the game. In other words, the user's intent is to shoot the puck while player A has a clear shot, while the current state of the game (T3) shows player C between player A and the goal. To align the user's intent (the puck moving towards the goal) with the current state (player C in the path of the puck), the game server may attempt to reconcile the user's intent with the current state (player C in the path of the puck) even though player C is in that path (e.g., frame 510-3, FIG. 5B). A series of response frames may be rendered that move toward the current game state. The response frames may appear to be inconsistent with the current game state. However, they are consistent with the user's intent in the past game state (trigger). Game developers may pre-plan for these inconsistencies, for example, by designing animations that adapt inconsistent game states. Exemplary adjustment animations include instantly shifting a virtual character or object to an intended position (even if this appears to violate the physics in the game), or advancing the game state in an intended manner that does not show the correct animation (e.g., updating the score without showing the puck's arrival at the goal, or classifying a monster as having sustained damage even though it appears to have deviated from its path before being shot). In one embodiment, adjusting the current game state to the game state intended by the user at the time of user interaction (intended game state) includes modifying a frame that shows the current game state to create a subsequent frame that shows the intended game state.
[0078] Delay Time Detection The following description describes various approaches to detecting various latency times in a game environment, according to an embodiment. Latency time detection is a necessary step that allows the game server to accurately determine the trigger frame for a particular user input (step 608, FIG. 6), thereby allowing the game server to determine the trigger state, and therefore the user's intent behind the input, as described above. Knowledge of the correct trigger frame (and, more specifically, the trigger state) allows the game server 122 to process an output that more accurately reflects the user's intent by considering a gameplay state closer to the time the user makes the input (e.g., a button press), instead of the gameplay state when the input arrives at the server (which may correspond to a later gameplay state).
[0079] The following are specific delay values that the server can access in one embodiment, and 1 is a brief discussion of latency from the perspective of a game server (e.g., server 122), including certain latency values that the server does not have access to in the real world. With respect to latency values that the server does not have access to, certain embodiments for detecting or approximating those values are described with reference to FIGS.
[0080] In one embodiment, the game server 122 has access to information necessary to calculate a processing delay, which is the amount of time it takes to process an input event and send a resulting response frame. In one embodiment, the processing delay varies from event to event. In one embodiment, the processing delay varies depending on the complexity of the game state (e.g., the number of game play events being processed simultaneously). In one embodiment, to calculate the processing delay for a particular input event, the server system 114 records a first timestamp corresponding to the time the input event arrives at the server system 114 (e.g., at an edge server), as well as a second timestamp corresponding to the time the encoded response frame leaves the server system 114. From these two timestamps, the server system 114 calculates the processing delay associated with the input event (e.g., by subtracting the first timestamp from the second timestamp).
[0081] In one embodiment, the game server 122 also has access to information necessary to calculate an average round trip time (RTT) between the server system 114 and the controller 102. In one embodiment, the server system 114 calculates this RTT value by sending one or more test packets (e.g., pings) to the controller 102, receiving corresponding responses, and calculating one or more average response times. In one embodiment, the game server 122 also has access to information necessary to calculate an RTT between the server 114 and the media device 106. In one embodiment, the server system 114 calculates this RTT value by sending one or more test packets (e.g., pings) to the media device 106, receiving corresponding responses, and calculating one or more average response times. However, due to asymmetric network delay times, as discussed above, average RTT information alone may not be adequate to determine the exact one-way transit time from the controller 102 to the server system 114 or from the media device 106 to the server system 114. If the server system 114 cannot directly calculate the one-way transit time described above, then the server system 114 may not be able to accurately determine the trigger frame.
[0082] With access to the above RTT values, the server system 114 (e.g., intent determination module 428) can approximate the trigger frame by assuming an average RTT value for each input event, dividing the RTT value in half, and adding in the amount of assumed output device display lag (e.g., television display lag). Although the delay between two network nodes is often asymmetric (forward and reverse delays are not equal), half the RTT value is the average of the forward and reverse delays, and therefore half the RTT value can be used as an approximation for the one-way delay (also referred to herein as "one-way transit time" and "one-way delay time").
[0083] In one embodiment, the intent determination module 428 uses (i) a one-way delay time value between the controller and the server, (ii) a one-way delay time value between the media device and the server, or (iii) a combination thereof to more accurately determine the trigger frame. The following discussion, with reference to Figures 7-12, describes one embodiment for measuring or more accurately approximating the one-way transit time to more accurately determine the trigger frame for a particular input event.
[0084] 7 to 12 are flowcharts of trigger frame determination processes 700 to 1200 according to an embodiment. The processing may be performed in an electronic server (e.g., server system 114, or more specifically, game server 122) having one or more processors (e.g., CPU 138 and / or GPU 140) and memory (e.g., memory 146) storing one or more programs for execution by the one or more processors, in a media device (e.g., media device 106, also referred to as a "display" when combined or otherwise coupled with display device 108) having one or more processors (e.g., CPU 302) and memory (e.g., memory 306) storing one or more programs for execution by the one or more processors, and / or in a user device (e.g., controller 102) having one or more processors (e.g., CPU 202) and memory (e.g., memory 206) storing one or more programs for execution by the one or more processors. In one embodiment, the server, media device, and user device include one or more programs and a memory storing the one or more programs for execution by one or more respective processors, the one or more programs including instructions for performing the respective processes. In one embodiment, a respective non-transitory computer-readable storage medium stores one or more programs including instructions that, when executed by the electronic server, media device, or user device using one or more respective processors, cause the electronic server, media device, or user device to perform one or more of methods 700-1200.
[0085] 7 illustrates a trigger frame determination process 700 according to one embodiment. In the process 700, the server 114, the media device 106, and the controller 102 are independent components that use asynchronous clocks. Thus, analysis of timestamps sent from the controller to the server and / or from the media device to the server will not provide an accurate assessment of the delay between the respective components. In one embodiment, one or more of the clocks are synchronized. However, the server may use the process 700 to determine the trigger frame regardless of whether any of the aforementioned clocks are synchronized.
[0086] The process 700 begins with the server 114 sending the first frame of a series of frames to the media device 106 for display to the user on the output device 108 (702). Once the media device 106 displays the first frame to the user (704), the media device 106 communicates to the server 114 (e.g., to the event log 460 associated with the game session) that the first frame has just been displayed (706). This communication is referred to herein as a "display event" and, in one embodiment, is reported by the display event reporting module 325 of the media device 106. Each time the media device 106 displays a frame, the media device 106 (e.g., the reporting module 325) sends a corresponding display event back to the server. In one embodiment, the display event includes a frame ID corresponding to the frame that was just displayed. Meanwhile, the user observes the first frame and activates an input when they decide to manipulate the controller to affect gameplay. The controller 102 detects (708) the user's input and transmits (710) the user's input to the server 114. This communication is referred to herein as an "input event" and, in one embodiment, is reported by an input event reporting module 223 of the controller 102. Each time the controller 102 detects a user input, the controller 102 transmits a corresponding input event to the server. In one embodiment, the input event includes an input ID that corresponds to the particular input activated by the user.
[0087] Meanwhile, the server that serially renders the frames for display and sends them to the user Display events corresponding to rendered frames sent to the user (e.g., during step 702) are continuously received (712). Upon receiving an input event (e.g., from step 710) (712), the intent determination module 428 matches the input event with the most recently received display event (714). Assuming similar one-way delay times between the controller and the server and between the media device and the server, the input events and their corresponding display events should arrive at approximately the same time. Thus, by matching the input event with the display event received closest in time to the time the input event was received, the intent determination module 428 approximates the user's input event intent by classifying the frame associated with the matched display event as a trigger frame.
[0088] The inaccuracies may result from differences in the upstream connections associated with the controller 102 and the media device 106 due to variability in the upstream links (e.g., links 502 and 504, FIG. 5A). For example, the media device 106 may use a wired connection, while the controller 102 may use a wireless connection that involves additional latency. In one embodiment, the intent determination module 428 accounts for these variables by averaging the difference in latency between the two links (e.g., by comparing the average server-media device RTT to the average server-controller RTT).
[0089] 8 illustrates a trigger frame determination process 800 according to an embodiment. In process 800, the server 114 and the controller 102 have synchronized clocks. In this process and other processes involving clock synchronization (e.g., processes 900-1200 described below), an embodiment achieves synchronization via a Network Time Protocol (NTP) server. In an embodiment, one or more of the aforementioned clocks are periodically resynchronized due to clock drift.
[0090] Process 800 begins with the controller 102 detecting a user input (802) and the reporting module 223 sending a corresponding input event to the server 114 (804), as described in process 700 above. However, in process 800, the input event further includes a timestamp (e.g., TS1). The server 114 receives the input event at time TS2 (806) and determines the one-way input-to-display delay time by comparing TS1 and TS2 (e.g., by taking the absolute value of the difference between the timestamps). The server 114 (e.g., the intent determination module 428) then approximates (810) the unidirectional display-to-server latency by using the unidirectional display-to-server latency as a proxy (e.g., by setting the unidirectional display-to-server latency equal to the unidirectional input-to-display latency).
[0091] The intent determination module 428 then approximates (812) the one-way server-to-display latency. In one embodiment, the server-to-display latency is approximated by using a time-averaged or moving window of the display-to-server latency as a proxy (e.g., by setting the server-to-display latency equal to the display-to-server latency). Alternatively, the server-to-display latency is approximated using the most recently measured server-to-display RTT (e.g., by dividing the RTT by 1 / 2) and, optionally, adding an expected display lag (as discussed above with reference to the latency of the output device 108).
[0092] Upon determining the value of the one-way server-to-display delay (e.g., D milliseconds), the intent determination module 428 transmits a trigger frame to the media device before TS1. It is determined that the frame is a frame that has been processed (814).
[0093] 9 shows a trigger frame determination process 900 according to one embodiment. In the process 900, the media device 106 and the controller 102 have synchronized clocks. In one embodiment, the server 114 also has a clock that is synchronized to the media device's clock, the controller's clock, or both.
[0094] Process 900 begins with the controller 102 detecting a user input (902) and the reporting module 223 sending a corresponding input event with timestamp TS1 (904) as described in process 800 above. The server 114 receives the input event (906) and renders a response frame (908), assuming that the current game state triggered the input event or that a past game state triggered the input event (e.g., offset by a predetermined amount). The server 114 includes timestamp TS1 in the response frame (e.g., in the frame's metadata) and sends it to the media device 106. The media device 106 receives the response frame (910) and causes the response frame to be displayed to the user with a second timestamp (e.g., TS2). The media device 106 (e.g., the delay calculation module 326) measures (912) the difference between the two timestamps and determines the amount of delay from when the user enters an input on the controller 102 to when the user sees a corresponding response on the display 108 (referred to herein as the "thumb-to-display delay"). In one embodiment, the media device 106 stores the timestamp data as delay data 334, and the delay calculation module 326 accesses the stored timestamp data to calculate the various delays described above. In one embodiment, the delay calculation module 326 combines the thumb-to-display delay with other known delays (e.g., of one or more of processes 700-1200) to better approximate the trigger frame.
[0095] 10 shows a trigger frame determination process 1000 according to one embodiment. In the process 1000, the media device 106 and the controller 102 have synchronized clocks. In one embodiment, the server 114 also has a clock that is synchronized to the media device's clock, the controller's clock, or both.
[0096] The process 1000 begins with the server 114 rendering (1002) a first frame at timestamp TS1 and sending the frame to the media device 106 at timestamp TS1 included in the frame (e.g., included in the frame's metadata). The media device 106 receives (1004) the frame and displays the frame at timestamp TS2. The media device (e.g., the delay calculation module 326) compares (1006) the two timestamps TS1 and TS2 to directly determine a one-way server-to-media device delay time, and the reporting module 325 reports (1008) the determined one-way server-to-media device delay time to the server 114. In one embodiment, the report is included in a display event corresponding to the first frame. The server 114 receives the server-to-media device delay time, and the intent determination module 428 compares (1012) it to other known delay times (e.g., from one or more of the processes 700-1200, such as data regarding recent media device-to-server delay times) to better approximate the trigger frame. In one embodiment, since a display event may be assumed to arrive in close proximity to a corresponding input event, process 1000 further includes the controller 102 detecting (1014) a user input and the reporting module 223 of the controller sending (1016) an input event including an input ID of the detected user input to the server 114. The server's intent determination module 428 optionally matches the input event with the closest display event, as described in process 700 above, to better approximate the trigger frame.
[0097] 11 shows a trigger frame determination process 1100 according to one embodiment. In the process 1100, the media device 106 and the controller 102 have synchronized clocks. In one embodiment, the server 114 also has a clock that is synchronized to the media device's clock, the controller's clock, or both.
[0098] Process 1100 begins with server 114 (1102) rendering a first frame with timestamp TS1, similar to process 1000. However, in process 1100, server 114 may not be able to include a timestamp in the frame transmission. In that case, server 114 transmits the first frame without timestamp TS1. Media device 106 receives the first frame (1104) and displays the first frame with timestamp TS2. Media device 106 (e.g., reporting module 325) then reports timestamp TS2 to server 114 (1106), either alone or in conjunction with the display event transmission described above. Server 114 receives timestamp TS2 (1108), and server intent determination module 428 compares timestamp TS2 to timestamp TS1 to directly measure the delay from the server to the media device (1110). In one embodiment, the intent determination module 428 compares (1112) the server-to-media device latency to other known latency times (e.g., from one or more of processes 700-1200, such as recent media device-to-server latency data) to better approximate the trigger frame. In one embodiment, since a display event may be assumed to arrive in close time to a corresponding input event, the process 1100 further includes the controller 102 detecting (1114) a user input and the controller's reporting module 223 sending (1116) an input event to the server 114 that includes an input ID of the detected user input. The server's intent determination module 428 optionally matches the input event to the closest display event, as described in process 700 above, to better approximate the trigger frame.
[0099] 12 illustrates a trigger frame determination process 1200 according to one embodiment. In the process 1200, the media device 106 and the controller 102 have synchronized clocks. In one embodiment, the server 114 also has a clock that is synchronized to the media device's clock, the controller's clock, or both.
[0100] The process 1200 exploits the assumption that if most televisions are good at lip sync (delaying the audio signal to match the television's picture delay time, e.g., aligning with the receiver via HDMI 1.3b), the audio signal can be used to measure the delay time from the media device to the eyeball. In one embodiment, the server 114 sends a separate audio tone to the media device 106 at initialization (e.g., during a login or setup process) at a first timestamp (e.g., TS1) (1202), and the media device 106 causes the audio tone to be played on the output device 108 at a second timestamp (e.g., TS2) (1204). Alternatively, the media device 106 independently plays the separate audio tone at initialization (at TS2) without being prompted by the server 114 (1204). Upon playing the audio tone, the media device (e.g., the reporting module 325) sends a report to the server 114 including the second timestamp TS2 (1206). When the audio tone reaches the controller 102, the controller 102 detects (1208) the audio tone (e.g., using an embedded microphone) at a third timestamp (e.g., TS3) and sends (1204) a report to the server 114 that includes the third timestamp TS3.
[0101] The server 114 receives a "voice transmission" report including TS2 from the media device 105, and Upon receiving (1212) a "voice detected" report including TS3 from the controller 102, the intent determination module 428 compares timestamps TS2 and TS3 to determine the delay from the output device to the controller. Since the user can be assumed to be in close proximity to the controller 102, the delay from the output device to the controller can be assumed to be equivalent to the delay from the output device to the ear. In one embodiment, the intent determination module 428 assumes a predetermined typical seating distance (e.g., 10 feet) from the speaker of the display device 108 to account for the propagation time of sound waves, or a seating distance programmed by the user. In one embodiment, the intent determination module 428 does not re-measure the delay from the display device to the ear, assuming that the display lag is fixed throughout the session. In one embodiment, the intent determination module 428 also compares TS1 to TS2 and / or TS3 to determine the respective end-to-end delay times to better approximate the trigger frame.
[0102] In one embodiment, the audio signal includes an encoded frame ID and / or encoded timestamp TS1, and the server 114 periodically transmits the audio signal, for example, via a high-frequency audio modem (e.g., a whistle), and the controller 102 listens for the audio signal as described above. Alternatively, the media device 106 receives the frame ID and / or timestamp TS1 (e.g., by either of processes 1000 or 1100) and locally synthesizes an audio tone for propagation from the output device 108. In this embodiment, the controller directly receives the identification of the trigger frame itself (e.g., the frame ID and / or timestamp TS1) and proceeds to report the identified trigger frame directly to the server 114. In one embodiment, a correction is performed to account for audio modem delay.
[0103] In one embodiment, instead of an audio tone, the server 114 renders the frame with an image pattern that identifies the frame and / or timestamp TS1 (1202), and a camera or photodetector on the controller 102 detects (1208) the image pattern on the display device 108. The other steps are similar to those described above. In one embodiment, the image pattern includes a variation in intensity (e.g., compared to a baseline value of the measured intensity).
[0104] 13 is a flow diagram of a latency detection and compensation process 1300 according to an embodiment. The process may be performed in an electronic server (e.g., server system 114, or more specifically, game server 122) having one or more processors (e.g., CPU 138 and / or GPU 140) and a memory (e.g., memory 146) storing one or more programs for execution by the one or more processors. In an embodiment, the server includes one or more programs and a memory storing one or more programs for execution by the one or more processors, the one or more programs including instructions for performing process 1300. In an embodiment, a non-transitory computer-readable storage medium stores one or more respective programs including instructions that, when executed by a server with one or more processors, cause the server to perform process 1300.
[0105] Process 1300 begins when server 114 receives an input event from a game controller 102 located at a remote location (e.g., local network 110, FIGS. 1 and 5A) in response to displaying a trigger frame at the remote location (e.g., local network 110, FIGS. 1 and 5A), as described with reference to step 606 of FIG. 6. In one embodiment, the server receives the input event during a game session operating in a current game state (e.g., state T2 of FIG. 5B). The input event includes a user command generated by a user interaction with the game controller (e.g., actuation or control key manipulation) during the game session.
[0106] It is important to note that although the input event is generated by the controller in response to a previously transmitted frame (the "trigger frame") being displayed to the remote user, the input event may not necessarily identify which of the previously transmitted frames was the triggered frame. Thus, the process continues with the server determining the trigger frame (1304), as described with reference to step 608 of FIG. 6 above. More specifically, the server (e.g., intent determination module 428) determines which of a number of frames previously transmitted by the server to the media device 106 during the game session prior to the server receiving the input event is the frame displayed remotely during user interaction (e.g., 510-1 of FIG. 5B). In various embodiments, The server determines the trigger frame by performing one or more of the processes described above with reference to FIGS.
[0107] The process continues with the server using the determined trigger frame to determine a game state (the "trigger state") associated with the trigger frame, as described with reference to step 610 of Figure 6. In one embodiment, depending on how long it takes for the input event to reach the server, the current game state may have already progressed beyond the trigger state, and thus the trigger state (e.g., state T1 in Figure 5B) will be the game state prior to the current game state (e.g., state T2 in Figure 5B).
[0108] Upon determining the trigger state, the game server processes the gameplay output according to (i) the trigger state and (ii) the user command contained in or described by the input event (1308), as described with reference to step 612 of FIG. 6. Specifically, for each input event received by the server, the server matches each user command described in the input event with a respective trigger state in order to more accurately process each command in the context of the game state that triggered the user to initiate the command, thereby conforming to, satisfying, and / or fulfilling the user intent behind each command. In one embodiment, to process the gameplay output, the game server must reconcile the current game state with the trigger state, particularly if the two states are in conflict, as described above.
[0109] The server renders (1310) a response frame showing the gameplay output, as described with reference to step 614 of FIG. 6 above, and transmits (1312) the result frame to the remote user for display, as described with reference to step 616 of FIG. 6 above.
[0110] Delay time adjustment for gameplay tuning Response time is an important aspect of online games that directly impacts the user experience. One way to describe response time is the amount of time between the moment a user performs an action (e.g., pressing a "jump" button on a game controller) and the moment the result of that action is displayed to the user (e.g., a digitally rendered player jumping on a screen). Response time can be affected by any of the various sources of latency mentioned above. Most of the sources of latency fall into two categories: processing latency and network latency.
[0111] Processing latency is the result of the amount and quality of processing resources dedicated to a particular game or online game session and may be affected by the number, speed, and efficiency of processors or processing cores allocated to process user input and to render the corresponding responses. Processing latency may be further affected by the complexity of the user input or the number and complexity of concurrent inputs being processed by other users in the game session.
[0112] Network latency is a result of the quality of the communications network (e.g., 110 and / or 112) used to support an online game session and may be affected by any number of external factors, such as interference, or by internal factors, such as various levels of traffic and available bandwidth. As the distance between a user (e.g., 102) and a particular game server system (e.g., 114) increases, the amount of physical network elements required to support the game session increases, which increases the likelihood that more network latency will be introduced. For example, a first user (e.g., 102A) located several miles away from a data center housing a particular game server may experience a longer latency during an online session hosted by the game server than a second user (e.g., 102B) participating in the same session but located across the street from that data center.
[0113] While processing and network latency are often outside the game developer's sphere of influence, still other types of latency may be intentionally added and adjusted by the developer to optimize the user experience. By adjusting the latency added by the developer, a properly equipped game server can counteract many of the negative effects introduced by uncontrollable types of latency (e.g., processing and network latency).
[0114] Furthermore, delay time adjustments serve to optimize the user experience by modeling game actions on real life actions. Certain user inputs in a game context are more susceptible to different amounts of delay time than others, such as a computer mouse that is overly responsive to slight movements of the hand. For example, in an ice hockey game, the response time for inputs associated with movement should be carefully adjusted to be sensitive enough to guarantee a competitive response time (e.g., avoid being checked by an opponent), but not so sensitive to prevent accurate movement (e.g., preparing a shot without overcompensating). Furthermore, different types of inputs may warrant different amounts of response time. For example, delay time values associated with inputs for shooting and blocking the puck may be adjusted for faster response times than those associated with inputs for controlling player movement, because more precision may be required to move a player into position and prepare a shot than, for example, to accurately determine when to shoot.
[0115] Game developers can tune response times for different inputs or types of inputs. In some embodiments, response time values for a particular game vary by platform, but the response times are stable for each platform. In embodiments involving online streaming, response times vary based on network conditions and speed of light constraints. Additionally, the various types of processing and network delay times mentioned above make tuning response times more complicated, as the amount of time that passes between an input and a response is not consistent for each user, or even for a single user over time.
[0116] Various embodiments of methods and systems for adjusting latency in conjunction with adjusting gameplay are described below. Various embodiments described herein serve to make performance variations based on network conditions less noticeable to users, resulting in a better user experience. In one embodiment, on a game streaming system where a game is hosted and displayed on a local client (e.g., 102A) from a computer in a geographically distant server (e.g., 114), the game dynamically shifts the number of frames rendered between registering an input event and displaying the corresponding response to the player. For a particular game, a developer may dynamically shift the number of frames rendered between registering an input event (e.g., movement, action, etc.) and displaying the corresponding response to the player. For each type of input (e.g., action, button press, controller orientation change, etc.), the system defines an ideal or intended number of frames or amount of time (e.g., milliseconds) between the input and the response. In an embodiment, the game system provides an API that reports for each frame either the existing input latency conditions or a recent time period that represents what user is looking at. In an embodiment, one or more users play a game on their respective game controllers (also referred to herein as "client devices" or "controllers") under variable input latency conditions. In an embodiment, the users generate input events by manipulating a joystick, pressing a button, selecting an affordance, moving, or otherwise manipulating an input device such as a game controller. Upon generating an input event, the controller associated with the user transmits the input event to a server (e.g., 114) that runs the game.
[0117] In one embodiment, the server queries (e.g., by querying a streamer API) the amount of existing latency, either the most recent value or a recent time-stamped estimate of the input latency (e.g., network latency). The server decreases or increases the number of frames (referred to herein as "intermediate frames") that pass before generating a frame (referred to herein as a "response frame") that reflects a response to the input event. The server transmits the intermediate frames, followed by the response frame, over the network to the media device. Upon transmitting the response frame to the media device, the media device displays the response frame to the user (e.g., on output device 108). In one embodiment, the delta in frames (e.g., number of intermediate frames) that are added or subtracted from the time-rendering is calculated based on the frame rate (e.g., in frames / s) being displayed in the current session. The change in delta is calculated based on the frame rate (e.g., in frames / s) being displayed in the input session. The amount of time that passes between a force and a response is reduced or increased so that the response time is as close as possible to the ideal or intended response time for a particular type of input. It is important to note that the embodiments disclosed herein do not require a total delay or buffering of frames, since different input events may require different amounts of adjustment to meet the ideal response time. Rather, aspects of the embodiments disclosed herein focus on increasing or decreasing per event the number of frames between an input and a corresponding response.
[0118] As mentioned above, game developers may define an ideal or intended number of frames or amount of time between each input and each response for different types of input events. In one embodiment, these relationships are defined in a response time configuration 462 stored in the server system 114 (see FIG. 4). FIGS. 14A and 14B are example response time configurations 462 for user inputs (also referred to herein as "commands") that are associated with ideal or intended response times (also referred to herein as "expected delay time values" or "target delay time values"). In one embodiment, an index of commands for a particular game is stored in a table (e.g., 1412) along with a corresponding command type (also referred to herein as "expected delay time type" or "delay time category"). For example, "duck" and "shoot" commands belong to a first category or command type (e.g., "Type 1"), "walk" and "jump" belong to a second category or command type, and so on. In one embodiment, an index of command types is stored in a table (e.g., 1414) along with a corresponding expected delay value. For example, a "Type 1" command is associated with an expected (e.g., ideal or intended) delay of 16 ms, a "Type 2" command is associated with an expected delay of 40 ms, etc. Alternatively, a table (e.g., 1416) is used to directly associate commands with expected delays. For example, "dodge" and "shoot" commands are associated with an expected delay of 20 ms, "move" and "jump" commands are associated with an expected delay of 40 ms, etc. In one embodiment, the tables 1412, 1414, and / or 1416 are stored in the memory 146 of the server system 114 (eg, in the response time settings 462).
[0119] FIG. 14C is an example embodiment of a device / network assessment module 444 of the server system 114. The device / network assessment module 444 obtains network latency information 1420 for each user / controller participating in a particular game session. In one embodiment, the network latency information is round trip timing (RTT) information associated with the amount of time it takes for a user input to be transmitted from a controller to a server combined with the amount of time it takes for a corresponding response frame to be transmitted from the server to a media device. In one embodiment, the RTT information further includes the amount of time it takes for the server to process the user input and generate a corresponding response frame. Additionally or alternatively, the network latency information is any of the latency values described with respect to FIGS. 5-12 above. For example, in one embodiment, if RTT information is not readily available to the network assessment module 444, various one-way transit times are determined as described above, and the network assessment module 444 combines the determined transit times (e.g., one-way controller to server delay time and one-way server to media device delay time) to determine unknown delay time information (e.g., delay time from controller to server and from server to media device).
[0120] 15 is an exemplary online interactive gaming environment 1500 according to one embodiment. Gaming environment 1500 is similar to gaming environments 100 (FIG. 1) and 500 (FIG. 5A), and corresponding components are similarly numbered.
[0121] Time t 1In the example, the controller 102A transmits a user command associated with the input (e.g., a command from the table 1412) to the server 114 over a network (e.g., one or more local and / or non-local communication networks 110 / 112). In one embodiment, the user command is the result of the user manipulating a control on the game controller (e.g., pressing a button, moving a joystick, rotating the controller itself, etc.) or otherwise interacting with the game controller. The user's motivation to manipulate the game controller to issue a user command (i.e., a "trigger") is based on the particular game being played. For example, if a user is playing an online ice hockey game, the user may decide to utilize the opening of the goal to press a "shoot" button to shoot the puck into the goal. In one embodiment, the trigger (e.g., the opening of the goal) is caused by an image communicated from the server 114 and rendered on an output device (e.g., 108) used to view the game play by the user of the controller 102A. Thus, upon seeing an image (eg, an image frame showing an opening for a goal), the user is motivated to respond by issuing an appropriate command (eg, shoot the puck).
[0122] Time t 2 At t, the server 114 receives the command. 1 and 2The amount of time between the command and the response frame is a function of various factors associated with the network latency discussed above. Upon receiving a command, the server processes the command by updating the current game state according to the command and generating a response frame that reflects the updated game state. In one embodiment, processing the command includes determining which command was received, determining the type of command, determining or updating a network latency value, and determining an amount of latency to introduce based on the network latency value and the type of command received (e.g., by looking up a table, determining an expected latency associated with the command, and comparing the expected latency to the network latency). and generating one or more intermediate frames based on the determined amount of introduced delay (e.g., by multiplying the current frame rate by the difference between the expected delay and the network delay).
[0123] Time t 3 At t, the server 114 transmits the intermediate frames (if any) and the response frame to the media device 106 via the network 110 / 112 for display on the output device 108. 2 and 3The amount of time between is a function of various factors associated with the processing latency described above. In one embodiment, the processing latency is affected by the process for generating the response frame. In one embodiment, generating the response frame includes (i) processing the response (e.g., determining what the first image frame of a series of image frames showing a moving puck will be based on each player's current position in the game scene and other factors such as the shot trajectory and player strength), (ii) rendering a frame reflecting the response (e.g., rendering the first frame of a series of frames showing a player shooting the puck), (iii) encoding the frame, and (iv) packetizing the frame for streaming across the network. In one embodiment, the response frame is one of a sequence of frames transmitted at a predetermined frame rate. In one embodiment, the predetermined frame rate is determined according to the network characteristics (e.g., available bandwidth) of the online game session, for example, by using a rate control process, and the predetermined frame rate is maintained even if it is determined that the network latency value has changed. For example, the response frame may be one of a sequence of frames transmitted at a predetermined frame rate, for example, a time before receiving a user command (e.g., t 2 A given frame rate at the time when the corresponding response frame is transmitted (e.g., t 3 ) in the event-by-event delay adjustments, the predetermined frame rate remains constant regardless of how much delay is added (e.g., by inserting more intermediate frames) or subtracted (e.g., by inserting fewer or no intermediate frames at all). By not changing the overall frame rate for per-event delay adjustments, other aspects of the gameplay (e.g., other gameplay events and interactions, viewing quality, etc.) may proceed unimpeded by being unaffected by the delay adjustments.
[0124] Time t 4At t, the media device 106 receives the response frame 1610 and causes the output device 108 to display the response frame. 3 and 4 The amount of time between is a function of various factors associated with network latency, as discussed above. In some embodiments, the network path traversed by the response frame is different from the network path traversed by the user command. In various embodiments, the network latency value is calculated based on the transit time from the controller to the server in one direction (e.g., t 1 and 2 ), one-way transit time from the server to the media device (e.g., t 3 and 4 t ), and / or processing delays (e.g., t 2 and 3 Based on the above.
[0125] FIG. 16 is an exemplary sequence of frames rendered on the display 108 according to an embodiment. In this example, frames 1610-1 through 1610-6 are transmitted and displayed at a predetermined frame rate of 50 frames / second (1 frame every 20 milliseconds). Also, for simplicity, assume that network delay time is negligible. The timestamps shown in FIG. 16 are the time at which each respective frame is generated by the server 114. At t=0 milliseconds, the server generates frame 1610-1, which shows two ice hockey players A and B controlled by two users and their respective controllers (e.g., 102A and 102B). Player A has the puck and player B is in goal. At this time, the server receives a "move" command from the controller controlling player A. In this particular example, the "move" command also includes a trajectory (to the left of the player). After receiving the "move" command, Upon receiving the response frame, the server (e.g., delay adjustment module 430) determines (e.g., by referencing tables 1412 and 1414, or table 1416) that the "move" command is a command type associated with an expected delay of 40 ms. Because the predetermined frame rate is one frame per 20 ms and the server must meet the expected delay of 40 ms before generating a response frame, delay adjustment module 430 determines that one intermediate frame should be generated as a placeholder because 40 ms (expected delay) - 20 ms (actual delay) equals 20 ms, and 20 ms x 1 / 20 frames / ms (frame rate) equals one frame. Thus, GPU 140 generates (or otherwise be processed, rendered, or encoded) one intermediate frame 1610-2 at t=20 ms, and then generates response frame 1610-3 at t=40 ms. In one embodiment, the encoding The decoder 142 encodes one or more intermediate frames as a skip frame or a series of skip frames.
[0126] Continuing with the example of FIG. 16, looking at the movement of player A, the user controlling player A decides to shoot the puck immediately before the user controlling player B has an opportunity to block the shot. The server receives a "SHOOT" command at t=160 ms, simultaneously with the generation of frame 1610-4. The delay time adjustment module 430 determines (e.g., by referencing tables 1412 and 1414, or table 1416) that the "SHOOT" command is associated with only an expected delay time of 20 ms. Because the predetermined frame rate is 1 frame per 20 ms, and the server far meets the expected delay time of 20 ms before generating a response frame, the delay time adjustment module 430 determines that no intermediate frame should be generated because 20 ms (expected delay time) - 20 ms (actual delay time) equals 0 ms, and 0 ms x 1 / 20 frames / ms (frame rate) equals 0 frames. Thus, GPU 140 generates response frame 1610-5 20 ms later, at t=180 ms, without rendering any intermediate frames (or being otherwise processed, rendered, etc.). Response frame 1610-5 is the first in a series of response frames indicating the outcome of the "shoot" command, and frame 1610-6 is the second in the series of response frames at t=200 ms.
[0127] 17 and 18 show an example technique for determining how much delay to introduce, where intermediate frames are shown as solid colors and response frames are shown as stripes.
[0128] In FIG. 17, the server receives the same command over two different networks or over the same network with two different delay values. The network delay 1720 is lower than the network delay 1710. As a result, the corresponding response frames would be sent back to the media device at different times without the introduced delay. By rendering additional intermediate frames at the lower network delay 1720 to the scenario, the server ensures that the expected overall delay (e.g., response time) is consistent regardless of the amount of network delay. From different perspectives, the response time is kept consistent by (i) generating more intermediate frames (scenario 1720) for another scenario, (ii) generating fewer intermediate frames (scenario 1710) for another scenario, or (iii) a combination of (i) and (ii).
[0129] In Figure 18, the server receives two different commands that require different amounts of processing delay. For example, a "run" command requires a first amount of processing delay 1810, while a more resource intensive "pan" command requires a second amount of processing delay 1820, which is greater than the first. 820. If each command is associated with the same expected delay and received at the same time (assuming similar network delays), the server accounts for differences in processing delays by (i) generating more intermediate frames for the other scenario (scenario 1810), (ii) generating fewer intermediate frames for the other scenario (scenario 1820), or (iii) a combination of (i) and (ii).
[0130] In one embodiment, the delay time adjustment module 430 determines the amount of delay time to be introduced by (i) determining or estimating the amount of actual delay time between a user command and a corresponding response frame (e.g., measuring network delay time as described above and optionally taking into account processing delay time), (ii) determining the expected delay time (e.g., by referencing the response time setting 462 described above), and (iii) determining the difference between the actual delay time and the expected delay time. If the actual delay time is shorter than the expected (intended) delay time, the server introduces the additional delay time. Conversely, if the actual delay time is longer than the expected (intended) delay time, the server removes the additional delay time (or does not introduce any at all). In one embodiment, to address each of the aforementioned scenarios, the server always adds a default amount of delay time to be introduced, and both adding and subtracting delay time to be introduced are possible. After determining the amount of delay time to be introduced, the delay time adjustment module determines the number of intermediate frames to be rendered by multiplying the amount of delay time to be introduced by the frame rate. In one embodiment, the delay time adjustment module 430 determines the number of intermediate frames to be rendered using the following equation: Number of intermediate frames = Frame rate x (Expected delay time - Actual delay time). For scenarios where the actual delay time is higher than the expected delay time, the number of intermediate frames is negative. Thus, in embodiments where there is always a reference value for intermediate frames, the delay time adjustment module 430 subtracts the appropriate number of intermediate frames from the reference value. However, in embodiments where there is no reference value for intermediate frames, the delay time adjustment module 430 simply does not cause intermediate frames to be rendered in such scenarios.
[0131] In one embodiment, the amount of delay time introduced is determined based on the estimated response frame arrival time. For example, if the response frame is predicted to arrive at the media device earlier than the expected or intended arrival time (e.g., based on network and / or processing delay values determined as described above), the server renders additional intermediate frames as discussed above. Conversely, if the response frame is predicted to arrive at the media device later than the expected or intended arrival time, the server renders fewer intermediate frames as discussed above. In one embodiment, the delay time adjustment module 430 determines the number of intermediate frames to be rendered using the following equation: Number of intermediate frames = Frame rate x (Target arrival time - Predicted arrival time). For scenarios where the predicted arrival time is later than the target arrival time, the number of intermediate frames is negative. Thus, in an embodiment where a reference value for intermediate frames is always present, the delay time adjustment module 430 subtracts the appropriate number of intermediate frames from the reference value. However, in an embodiment where a reference value for intermediate frames is not present, the delay time adjustment module 430 simply does not cause intermediate frames to be rendered in such scenarios.
[0132] FIG. 19A is an exemplary online interactive gaming environment 1900 according to an embodiment. The gaming environment 1900 is an exemplary embodiment corresponding to the gaming environment 1500, with the addition of a controller 102B and media devices / output devices 106B / 108B of a second user located at a site distant from the first user and the server. The first and second users are both playing the same game. In this example, the distance between the second user and the server is farther than the distance between the first user and the server. Thus, the second user experiences more network latency. Also, in this example, for simplicity of explanation, For this reason, the display delay between each media device 106 and the display 108 is assumed to be zero.
[0133] Time t 0At time t, the first controller 102A transmits a user command (e.g., “shoot”) to the server 114 via the network 110 / 112 (see FIG. 15 ), and the command is 2 It takes 40 ms to reach the server at time t 3 The response frame is sent to the first media device 106A and the second media device 106B at time t 5 takes 40 milliseconds to reach the first media device 106A at time t 6 It takes 60 milliseconds for the event to reach the second media device 106B at 1020. Thus, the second user (controlling player B with controller 102B) will not see the response frame on display 108B until 20 milliseconds after the first user (controlling player A with controller 102A) sees the response frame on display 108A. As a result of the 20 millisecond delay, the response time of the second user may be unduly delayed. Furthermore, player A may be able to score before the second user sees those events unfolding on display 108B, and the game state thereafter may be updated accordingly.
[0134] FIG. 19B shows screenshots 1920-1928 rendered by displays 108A and 108B with timestamps corresponding to the timestamps of FIG. 19A. Player A's response time is 20 ms faster than Player B's response time, so Player A will respond at t 5 = 100 ms, while player B sees response frame 1926 20 ms later. As a result, 6At =120 ms, player A scores, but from player B's perspective, there is still time to block player A's shot, even though the game state has already been updated at the server (e.g., by adding a point to player A's team's score). In some scenarios, player B appears to successfully block player A's shot in a future frame, generating an expected game state that conflicts with the actual game state. These kinds of conflicting game states can be avoided by introducing a delay time according to various embodiments described above.
[0135] 20A and 20B show an example game environment 1900 and corresponding user view, but with an additional intermediate frame added in accordance with various embodiments described above. Figures 20A / 20B correspond to Figures 19A / 19B, with the differences circled. Specifically, by adding the additional intermediate frame 2000, the server introduces an additional 20 milliseconds of delay time into Player A's response time, thereby causing response frame 1926 to arrive at media devices 106A and 106B simultaneously.
[0136] FIG. 21 is a flow diagram of a delay time adjustment process 2100 according to an embodiment. The process may be performed in an electronic server (e.g., server system 114, or more specifically, game server 122) having one or more processors (e.g., CPU 138 and / or GPU 140) and a memory (e.g., memory 146) storing one or more programs for execution by the one or more processors. In an embodiment, the server includes one or more programs and a memory storing one or more programs for execution by the one or more processors, the one or more programs including instructions for performing the process 2100. In an embodiment, a non-transitory computer-readable storage medium stores one or more respective programs, the one or more respective programs including instructions that, when executed by a server with one or more processors, cause the server to perform the process 2100.
[0137] The process 2100 includes receiving (2102) a first command (e.g., “shoot”) from a first client device (e.g., game controller 102A) associated with the online game session, determining (2104) a type of the first command (e.g., “Type 1” in table 1412) and a first expected response delay time associated with the type of first command (e.g., “20 milliseconds” in table 1414 or table 1416), and calculating (2105) a round trip time 1420-1 between the first game controller and a server. 16, or 2010-1 in FIG. 20; determining 2106 a network delay time (by measuring and / or estimating, or by any of the other methods described above with reference to FIGS. 5-12); determining 2108 a first introduced delay time (e.g., a delay time that is intentionally added and / or adjusted to compensate for other types of delay times, such as the determined network delay time) based on a comparison of the network delay time with a first expected delay time; and determining 2109 a first number of intermediate frames (e.g., frame 1610-2 in FIG. 16, or 2010-1 in FIG. 20) that, when transmitted at a predetermined frame rate, occupy a transmission time corresponding to the first introduced delay time. 16, or frame 1926 of FIG. 20B) reflecting an initial result of the first command; and transmitting (2114) the first response frame following a first number of intermediate frames at a predetermined frame rate such that the first response frame is received at a media device (e.g., 106) associated with the first game controller at a time corresponding to a first expected response delay time.
[0138] By using intermediate frames to adjust the response time between a command and a corresponding response frame, the server can maintain the overall aspect of the game session (e.g., a given frame rate) while adjusting a particular aspect (e.g., the response time to a particular type of user input). From an online gaming perspective, a stable frame rate is optimal for a smooth and high-quality user experience. Furthermore, by implementing game adjustments on a local level (e.g., only for a particular type of input), other aspects of the game continue unhindered, such as non-critical game events that do not affect subsequent game states, but are continuously rendered, adding to the fluidity and quality of the gameplay experience.
[0139] Resolution based interactive graphics scaling Real-time rendering of visual simulations has performance requirements (e.g., number of pixels) that scale with the rendered resolution. For example, if a visual simulation of video frames showing a gameplay scene is carried over a network to a user, the capabilities of the user's network and / or display device may constrain the maximum resolution that can be supported for a given user. With multiple users, each with different network and display capabilities, an efficient infrastructure allows performance appropriate to the specific resolution of each user's connection. The following description includes various embodiments of an efficient, resolution-appropriate virtualization method and system of underlying hardware.
[0140] Interactive graphics applications, such as online video games, utilize underlying hardware (e.g., server system 114) to achieve a target image resolution and frame rate. For example, when resulting image frames are streamed over a network (e.g., network 112) to an endpoint (e.g., media device 106), the capabilities of the endpoint and the network connection will determine the maximum resolution that can be sustained.
[0141] Various embodiments of methods and systems for establishing an interactive game session and allocating resources to the session are described below. Referring to FIG. In one embodiment, the server system 114 receives a request to establish a session (e.g., a game session) from the client device 102A. The server system (e.g., device / network assessment module 444) determines characteristics of the controller 102A, the media device 106, and / or the display device 108, and characteristics of the network 110 / 112. Based on the device and network characteristics, the server system (e.g., resource allocation module 432) determines one or more target quality parameters for the session, such as resolution, frame rate, and latency. According to the target quality parameters, the server (e.g., resource allocation module 432) assigns a specific virtual machine or container to the session and establishes the session according to the assignment. Resources are provided to the session according to a resource profile (e.g., stored in resource repository 464) associated with the specific virtual machine or container assigned to the session. By allocating a dedicated virtualization of basic hardware resources in the server system, the hardware resource consumption of the gaming application (e.g., in a data center) is adapted to the capabilities of both the network and the endpoint, thereby optimizing efficiency while still achieving the desired performance metrics for each session.
[0142] FIG. 22 is an example implementation of resource repository 464. The repository includes virtual machine settings 2202 (containing settings for M virtual machines), container image repository 2204 (containing images for N containers), and resource profile 2206 (containing settings for P profiles). A virtual machine is an operating system or application environment installed on software that mimics dedicated hardware, and a container is a virtualized operating system. Although there are differences between the two, both implement the concept of virtualization. Thus, unless otherwise stated, virtual machine and container are used interchangeably throughout this disclosure.
[0143] The virtual machines or containers are provisioned based on target quality parameters. For example, the virtual machines or containers are provided based on target resolutions (e.g., "small", "medium", "large" or 720p, 1080p, 1440p). In this example, virtual machine 2202-1 is referred to as "small" and can be used to provide an output stream having a resolution of 720p, virtual machine 2202-2 is referred to as "medium" and can be used to provide an output stream having a resolution of 1080p, virtual machine 2202-3 is referred to as "large" and can be used to provide an output stream having a resolution of 1440p, etc. In an embodiment, each virtual machine and / or container is associated with a resource profile 2206. Each resource profile includes settings regarding specific resources and resource levels that can be assigned to the virtual machine and / or container. These resources are for processing inputs (e.g., input events as described above) and generating outputs (e.g., response frames as described above) within a session. Examples of resources include one or more of: graphical processor bandwidth at the server system, general purpose processor bandwidth at the server system, graphical memory at the server system, general purpose memory at the server system, storage capacity at the server system, and input / output channels at the server system. Once the resource allocation module 432 assigns or otherwise associates a particular virtual machine or container to a session, the session is provided with available resources according to the resource profile 2206 associated with that particular virtual machine or container.
[0144] FIG. 23 is a flow diagram of a resource allocation process 2300 according to one embodiment. The process includes one or more processors (e.g., CPU 138 and / or GPU 140) and one or more programs for execution by the one or more processors. 2300。 In one embodiment, the server includes a memory that stores one or more programs and one or more programs for execution by one or more processors, the one or more programs including instructions for performing process 2300. In one embodiment, a non-transitory computer-readable storage medium stores one or more respective programs that, when executed by a server with one or more processors, cause the server to perform process 2300.
[0145] The process begins when the server 114 receives 2302 a request to establish a session from a client device (e.g., the controller 102). In one embodiment, the client device 102 requests to establish a real-time interactive session, and the request is received over a network connection with the client device 102 (e.g., the network 110 / 112).
[0146] Upon receiving the request, the server (e.g., device / network assessment module 444) determines (2304) the device capabilities of the devices associated with the client device 102 (e.g., media device 106, output device 108, and / or the client device 102 itself). In one embodiment, the device capabilities are a maximum resolution of the display output on the output device (e.g., 108) associated with the client device, a maximum frame rate of the display output on the output device (e.g., 108) associated with the client device, or both. In one embodiment, the output device itself communicates the device capabilities information to the server directly or through the client device. In one embodiment, the device capabilities information is stored locally on the client device, and the client device transmits the device capabilities information with an initial request to establish a session. In one embodiment, the client device transmits the device capabilities information in response to a request by the device / network assessment module 444.
[0147] Additionally, the server (e.g., device / network assessment module 444) determines (2306) the connection capabilities of the network connection (e.g., network 110 / 112). In one embodiment, the connection capabilities are the bandwidth of the network connection and / or one or more latency values associated with the network connection. Examples of latency values and methods for obtaining them are discussed above with reference to FIGS. 5-12.
[0148] Upon evaluating the device and network capabilities, the server (e.g., resource allocation module 432) determines (2308) one or more target quality parameters for the real-time interactive session based on the device capabilities and the network connection capabilities. In an embodiment, the one or more target quality parameters include one or more of a target resolution, a target frame rate, and / or a target latency time for content (e.g., response frames) sent to the output device.
[0149] Upon determining the target quality parameters, the server (e.g., resource allocation module 432) selects 2310 a virtual machine or container based on the one or more target quality parameters. In one embodiment, resource allocation module 432 (i) compares the one or more target quality parameters to corresponding parameters in a resource profile of each of the multiple virtual machines, (ii) determines a resource profile including parameters that most closely match the one or more target quality parameters, and (iii) selects as the first virtual machine the virtual machine having the resource profile with parameters that most closely match the one or more target quality parameters. For example, if the target frame rate is 120 fps, the resource allocation module 432 compares the target frame rate to parameters in the resource profiles 2206 that would support a frame rate of 120 fps and determines whether a particular resource profile (e.g., 2206-2) is the target. It determines that the virtual machine that can best support the target frame rate and selects the virtual machine associated with profile 2206-2 (eg, Virtual Machine 2).
[0150] In one embodiment, the resource allocation module 432 selects a virtual machine or container by (i) comparing one or more target quality parameters with corresponding parameters in a resource profile of each of the multiple virtual machines, (ii) selecting one or more virtual machines having a resource profile with parameters equal to or greater than the one or more target quality parameters as virtual machine candidates, and (iii) selecting a virtual machine candidate having the least resource intensive resource profile as the first virtual machine. For example, if the target frame rate is 120 fps, the resource allocation module 432 compares the target frame rate with parameters in the resource profile 2206 that selects as candidates resources having resources equal to or greater than the resources required to support the target frame rate (e.g., resources that support target frame rates of 100 fps, 120 fps, 240 fps, etc.), and selects the candidate having the least resource intensive resource profile (e.g., a profile that can support 100 fps, but not necessarily 120 fps). By selecting the least resource intensive resource profile, the server allocates only the resources needed and reserves additional resources for other sessions, thereby achieving the target performance level while optimizing efficiency at the server.
[0151] In one embodiment, selecting a virtual machine includes associating the selected virtual machine with the real-time interactive session and maintaining the association regardless of any changes in device or network connection capabilities. In one embodiment, the association is maintained for a predetermined period of time. In one embodiment, the resource allocation module 432 re-evaluates the association upon detecting a change in device or network connection capabilities. In one embodiment, the resource allocation module 432 re-evaluates the association based on the detected change in device or network connection capabilities only if the change is greater than a predetermined threshold. By limiting the reallocation of virtual machines or containers, the server achieves a targeted performance level while optimizing stability and therefore efficiency in the server.
[0152] Once the virtual machine or container is selected, the server 114 establishes (2312) a real-time interactive session according to the selected virtual machine or container and provides (2314) resources to the real-time interactive session for processing input (e.g., input events as described above) and generating output (e.g., response frames as described above) within the real-time interactive session according to a resource profile of the first virtual machine. Examples of resources for processing input and generating output include one or more of: graphical processor bandwidth in the server system, general purpose processor bandwidth in the server system, graphical memory in the server system, general purpose memory in the server system, storage capacity in the server system, and / or input / output channels in the server system. In an embodiment, the resources are provided by allocating a respective portion of one or more of the resources to the real-time interactive session. In an embodiment, the resources are provided by mapping a resource profile of the selected virtual machine or container to a respective portion of one or more of the resources (e.g., to a particular memory partition or a particular input / output channel). This is provided by:
[0153] In some embodiments, one or more subsets of virtual machines or containers are made available as different service tiers based on the level of resources provided. For example, users who pay for a higher service tier may receive priority access to virtual machines or containers with resource profiles that provide relatively advanced resources (e.g., “large” resolution, high processor bandwidth allocation, etc.). In some embodiments, users of the online gaming environment 100 gain access to higher service tiers by paying a premium, by achieving in-game rewards or high in-game performance statistics, or by taking advantage of promotions offered by particular gaming companies. In some embodiments, users build custom packages based on particular preferences (e.g., high resolution and average frame rate, average resolution and high frame rate, etc.). In these embodiments, different versions of the custom package are associated with one or more virtual machines or containers.
[0154] It is important to note that due to the nature of the real-time interactive session games described above, controller-to-display latency is an issue that must be monitored in order to maintain a high quality and consistent gameplay experience. Thus, the provision of virtual machines or containers as described above ensures that the resources required to ensure minimum latency criteria can be met by reserving dedicated resources for individual users or groups of users. Furthermore, as the use of game sessions scales up to games involving multiple players (e.g., massively multiplayer online role-playing games), the virtual provision of resources through virtual machines and containers further ensures a high quality gameplay experience by facilitating a level playing field for each player (e.g., by ensuring that minimum latency criteria are met for each player). For example, if two players are playing the same online game, the first session between the game server and a first client device may be assigned a different virtual machine or container than the virtual machine or container assigned to the second session between the game server and a second client device, even though both the first and second client devices are playing the same game in parallel. By allocating virtual machines or containers on a per-session basis, hardware resources are optimized while ensuring the desired gameplay experience for each user.
[0155] In one embodiment, applications are designed using a software development kit that takes into account the size of the underlying container and can "right-size" the application's use of hardware and / or virtualized hardware to achieve the desired performance for a given size. In one embodiment, instances of containers of various sizes are made available in close proximity to the endpoints. When an endpoint connects to the resource allocation module 432 and establishes a new session, the device / network assessment module 444 assesses the endpoint and network capabilities, and the resource allocation module 432 determines the size. Once determined, the session is assigned to a virtual machine or container of the corresponding size, and content delivery begins at the corresponding resolution, frame rate, or latency level. By "right-sizing" the application's consumption of hardware resources to both the network and endpoint capabilities, the server achieves the desired performance level while optimizing efficiency.
[0156] Adjustments for user-specific network conditions In online, interactive, real-time gaming environments, latency is a major factor in gameplay quality. Latency is one of the key factors affecting gameplay performance. As discussed above, there are numerous network and processing elements that can introduce various levels of latency into a game environment. Minimizing the negative effects of latency often comes at the expense of processing power and complexity, especially when processing many gameplay inputs from many game sessions and streaming gameplay output content to many users in parallel. It is therefore important that a game environment (e.g., server system 114 and / or game server 122) allocates processing resources wisely.
[0157] One way to allocate processing resources is to consider the user's needs and experience of the game environment on a per-user basis and allocate resources accordingly. For example, different users have different levels of tolerance to adverse playability events. Some users may be more sensitive to certain levels of latency than others. For example, some players may object to controller-to-display latency of up to 20 milliseconds, while other players may only observe issues associated with latency at 120 milliseconds. Given constrained resources, the ability to determine tolerance to adverse playability and conditions allows better allocation decisions to be made. Benefits of allocating resources on a user-specific basis include a better experience for all users due to more highly optimized processing efficiency, which leads to the ability to support more users and lowers the cost of providing content to each user.
[0158] Various embodiments of methods and systems for adjusting network conditions (allocation of network resources) on a user-specific basis are described below. In an embodiment, the game server (e.g., resource adjustment module 458, FIG. 4) considers each user's available set of gameplay statistics (e.g., profile 2402, FIG. 24B) and determines a window for playability for each user based on each user's gameplay statistics. Exemplary gameplay statistics include in-game performance (e.g., how well the user plays the game), controller interaction (e.g., overcompensation for movements, response time, and / or actuation speed), the type of game the user is playing (e.g., fast-paced vs. slow-paced), and user preferences (inferred by the game server or specified by the user). In an embodiment, the game server determines per-user playability based on self-reported data. In an embodiment, the game server observes the user's behavior in the game to determine how to classify the play experience and uses that information to determine where the stream is provided from and how the server conveys the experience. In one embodiment, the above approach is augmented by applying machine learning to determine the specific weighting for each gameplay statistic towards classifying a user's playability.
[0159] 24A is an example implementation of a repository for user playability profiles 2402. In one embodiment, the playability profiles (e.g., representing gameplay statistics for N users) are stored as user information 458 in memory 146 (FIG. 4).
[0160] In one embodiment, the game play statistics include in-game performance data. Examples of a user's in-game performance data include a user's skill level for a particular game (e.g., a measure of how well the user plays the game). In one embodiment, the game server (e.g., resource adjustment module 458) determines a user's skill level by comparing the user's activities or achievements to particular game-specific benchmarks, such as the amount of time it takes the user to reach a checkpoint, how many wins the user achieves against a particular opponent or challenge, how many points the user earns, etc. In one embodiment, the resource adjustment module 458 compares the user's in-game performance data (e.g., skill level) with corresponding performance data of other users, or with an average performance metric representing multiple users currently playing or who have played the game.
[0161] In one embodiment, the gameplay statistics include controller interactivity data, such as data describing a user's interactions with the game controller 102. In one embodiment, the controller interactivity data describes a user response time, such as the amount of time between (i) a stimulus rendered in an output frame being displayed to a user (e.g., on the display 108) and (ii) the user responding to that stimulus by sending gameplay input to a server (e.g., by interacting with a control key on the game controller 102). For example, referring to FIG. 5B, a trigger frame 510-1 is triggered at a first gameplay time t 1 and the user starts the second gameplay time t 2 When responding to a message, the interaction delay from the display to the controller is t 2 and 1 A user having a faster reaction time is associated with controller interaction data that represents a shorter response time.
[0162] In some embodiments, game play statistics include controller actuation speed data, such as data describing how quickly a user registers successive inputs (e.g., by acting on two control keys in succession or by acting on the same control key twice in succession). For example, with reference to FIG. 16, if a user acts a first control key to move player A into position (frame 1610-3) and then acts a second control key to cause player A to shoot the puck (frame 1610-5) in an attempt to score, the user's controller actuation speed is the amount of time taken between the actuation of the first control key and the actuation of the second control key. A user who is able to more quickly actuate multiple control keys on the game controller 102 is associated with controller actuation data that represents a shorter actuation time between successive acts or actions.
[0163] In some embodiments, gameplay statistics include controller performance data, such as data describing how precisely a user interacts with the controller 102. For example, referring to FIG. 16, when a user moves Player A to a position (frame 1610-3), a user with relatively high skill will interact with the appropriate controls (e.g., press and hold the "move" button) long enough to move the virtual player to the correct position, while a user with relatively low skill may over- or under-compensate (e.g., hold the "move" button too long or not long enough), thereby moving the virtual player to an incorrect position and missing the shot. A user who is able to more precisely manipulate the controls on the game controller 102 may be associated with more precise controller performance data.
[0164] In one embodiment, the playability profile 2402 includes the type of game being played. For example, a fast-paced hockey game, such as the games described in the various examples above, may require faster gameplay and less lag time than a more laid-back strategy game that does not require quick decisions or movements. Thus, statistics such as controller response time and accuracy may have different meanings, or be completely meaningless, in the context of other statistics, such as performance or skill level, depending on the type of game.
[0165] In one embodiment, the playability profile 2402 includes playability preferences regarding tolerance levels for some aspects of gameplay. For example, a user may wish to play a game regardless of how well the user performs in the game or interacts with the controller. A user may prefer a certain level of resolution, frame rate, and / or latency. For example, if a user is accustomed to or prefers a particular gameplay experience associated with a particular level of latency, providing a faster response time may be most unnecessary (since the user prefers a slower response time) and may be worst detrimental (since the user may not be able to play the game as well if he or she cannot adapt to the faster response time). As yet another example, a user may prefer a slower frame rate than the server system 114 can provide due to personal viewing preferences. As a result, the user's preferences are a factor in how the server system 114 allocates resources. In an embodiment, the user manually inputs these preferences, and the server (e.g., resource adjustment module 458) stores the user's preferences according to the manually entered settings. Alternatively, the server infers these preferences. In an embodiment, the server infers the user's preferences based on the user's gameplay statistics (e.g., in-game behavior and performance data).
[0166] Upon determining the various gameplay statistics as described above, the resource adjustment module 458 assigns a gameplay experience tolerance level to the user according to one or more of the above-mentioned gameplay statistics (e.g., in-game performance data). The gameplay experience tolerance level is a metric that describes a level of service that the game server can present to the user without adversely affecting the user's perceived gameplay experience. The gameplay experience tolerance level may also be described as a detrimental latency level, a playability level, a quality expectation level, and / or a minimum service tolerance level. In an embodiment, the experience tolerance level represents a particular frame rate, a particular resolution, and / or a particular latency level that is determined to have a below-threshold amount of impact on the perceived gameplay experience associated with the user of the first client device.
[0167] FIG. 24B is an exemplary embodiment of a resource setting table 466. In one embodiment, the resource setting 466 is stored in memory 146 (FIG. 4). According to the exemplary resource setting 466, a first tolerance level ("1") is associated with a particular resource profile ("A"). An example of a resource profile is described above with reference to FIG. 22 (resource profile 2206). By assigning or allocating a profile of resources (e.g., server processor bandwidth) to a user having a particular gameplay experience tolerance level, the server system 114 provides the user with a desired gameplay experience (e.g., within the user's tolerance) with only the minimum resources necessary to provide the level of gameplay experience.
[0168] 25 is a flow diagram of a resource adjustment process 2500 according to an embodiment. The process may be performed on an electronic server (e.g., server system 114, or more specifically, game server 122) having one or more processors (e.g., CPU 138 and / or GPU 140) and a memory (e.g., memory 146) that stores one or more programs for execution by the one or more processors. In an embodiment, the server includes one or more programs and a memory that stores the one or more programs for execution by the one or more processors, the one or more programs including instructions for performing process 2500. In an embodiment, a non-transitory computer-readable storage medium stores one or more respective programs that, when executed by a server with one or more processors, cause the server to perform process 2500.
[0169] The process begins when the server 114 establishes (2502) a real-time interactive game session with a first client device (e.g., the game controller 102). , the game session is associated with a particular game type (e.g., a fast-paced role-playing game or a slow-paced strategy game). While a user of a game controller 102 plays a game via a game session, the server (e.g., resource adjustment module 458) monitors (2504) in-game performance data associated with the user during the game session. Examples of in-game performance data include game play statistics, performance metrics, skill level, controller response time, controller operation time, and / or controller accuracy, as described above. In one embodiment, the in-game performance data is stored in a profile 2402 for the user.
[0170] Based on the in-game performance data (e.g., gameplay skill, controller interaction), game type, and / or user preferences, the resource adjustment module 458 determines (2506) an acceptable gameplay experience level for the user. For example, if the user has a relatively high skill level, faster response (shorter controller response time), faster game control ability (shorter controller operation time), and / or higher controller precision, the adjustment module 458 assigns an acceptable level corresponding to a higher resource profile (e.g., allocating more resources to the user's game session) and / or an acceptable level corresponding to a higher quality gameplay experience (e.g., higher frame rate, higher resolution, and / or lower latency). On the other hand, if the user has a relatively low skill level, slow response (longer controller response time), slower game control ability (longer controller manipulation time), and / or lower controller precision, the adjustment module 458 assigns a tolerance level corresponding to a lower resource profile (e.g., allocating more resources to the user's game session) and / or a tolerance level corresponding to a lower quality gameplay experience (e.g., lower frame rate, lower resolution, and / or higher latency). In one embodiment, the tolerance level is stored in the user's playability profile 2402 along with other gameplay statistics.
[0171] The resource adjustment module 458 adjusts (2508) server resources for the user's game session based on the gameplay experience tolerance level. Exemplary session resources include graphical processor bandwidth at the server system, general purpose processor bandwidth at the server system, graphical memory at the server system, general purpose memory at the server system, storage capacity at the server system, input / output channels at the server system, and / or streaming sources. In one embodiment, the resource adjustment module 458 allocates resources to the game session by allocating virtual machines or containers, as described above with respect to FIGS. 22-23. By adjusting the allocation of session resources, the game server affects one or more of a frame rate, resolution, or latency level associated with a gameplay output stream (e.g., a video stream indicative of gameplay output). Alternatively or additionally, in one embodiment, the resource adjustment module 458 directly adjusts a frame rate, resolution, and / or latency level associated with a gameplay output stream.
[0172] In one embodiment, the resource adjustment module 458 determines a particular user's gameplay experience tolerance level by initially setting the tolerance level at a predetermined starting point (e.g., determined to be higher than necessary to ensure a positive gameplay experience for most users) and then gradually adjusting the tolerance level until one or more of the user's gameplay statistics are adversely affected. The level just before the user's gameplay statistics are adversely affected is determined as the user's gameplay experience tolerance level. For example, when a user establishes a game session, the game server may monitor the user's ability to excel in the game. The initial bandwidth allocation is set at a level known or assumed to be high enough to avoid adverse effects. The resource adjustment module 458 then gradually reduces the bandwidth allocation for the session until the user's gameplay experience begins to show signs of being adversely affected (e.g., by the user's in-game performance beginning to decrease below a threshold). The lowest bandwidth level at which the user's gameplay is not affected, or is not affected beyond a threshold, is then recorded as part of the user's acceptable gameplay experience level.
[0173] In an embodiment, a user may have access to the user's game experience tolerance level. For example, the game server may provide the user with information in the user's playability profile 2402. By providing an option for the user to view the playability profile information, the user may use associated gameplay statistics as a metric to track the user's gameplay progress (e.g., whether the user is improving their in-game performance) and / or compare certain gameplay statistics to the gameplay statistics of other users.
[0174] Various embodiments described herein efficiently utilize bandwidth and computing resources to provide an optimal gameplay experience for users based on their own unique capabilities as determined by monitoring each user's gameplay performance. By tailoring server resources per user according to each user's particular play style, skills, needs, and / or preferences, various embodiments described herein provide a better experience for all users, the ability to support more users, and lower the cost of providing content to each user due to more highly optimized processing efficiency.
[0175] Disclosure Note Reference will now be made in detail to various embodiments, examples of which are illustrated in the accompanying drawings. In the above detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention and the described embodiments. However, the invention may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure the embodiments.
[0176] Terms such as "first", "second", etc. may be used herein to describe various elements, but it should be understood that these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first device can be called a second device, and similarly, a second device can be called a first device, as long as all occurrences of the first device are named consistently and all occurrences of the second device are named consistently, without changing the meaning of the description. A first device and a second device are both devices, but are not the same device.
[0177] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of the claims. As used in describing the embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. As used herein, the terms "comprises" and / or "comprising" refer to and encompass any and all possible combinations of stated features, integer values, steps, operations, elements, and / or specifies the presence of a component but does not preclude the presence or addition of one or more other features, integer values, steps, operations, elements, components, and / or groups thereof. It will be further understood.
[0178] As used herein, the term "if" may be interpreted as "when" or "upon" or "pursuant to a determination" or "in response to a determination" or "in response to a detection," meaning that the aforementioned condition is true, depending on the context. Similarly, the phrase "upon determining that [the aforementioned condition is true]" or "upon determining that [the aforementioned condition is true]" or "when [the aforementioned condition is true]" may be interpreted as "upon determining" or "in response to determining" or "pursuant to determining" or "upon detecting" or "in response to detecting" that the aforementioned condition is true, depending on the context.
[0179] The above description has been set forth with reference to specific embodiments for purposes of explanation. However, the above illustrative description is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in light of the above teachings. The embodiments have been chosen and described to best explain the principles of the invention and its practical application, and thereby enable others skilled in the art to best utilize the invention and its various embodiments, with various modifications as appropriate for the particular uses contemplated.
Claims
1. 1. A method comprising: In a server system having a plurality of virtual machines each having a respective resource profile, receiving a request from a client device to establish a real-time interactive game session, the request being received over a network connection with the client device; determining device capabilities of an output device associated with the client device; determining a connection capability of the network connection; determining one or more target quality parameters for the real-time interactive gaming session based on the device capabilities and the connection capabilities; selecting a first virtual machine of the plurality of virtual machines based on the one or more target quality parameters; establishing the real-time interactive gaming session with the client device; providing multiple resources to the real-time interactive gaming session to process input from the client device in accordance with a resource profile of the first virtual machine and to generate gameplay output in accordance with the input processed within the real-time interactive gaming session.
2. The method of claim 1 , wherein the device capabilities of the output device include one or more of a maximum resolution of a display output of the output device or a maximum frame rate of a display output of the output device.
3. The method of claim 1 or 2, wherein the connection capabilities of the network connection include one or more of: a bandwidth of the network connection; or a latency of the network connection.
4. 4. The method of claim 1, wherein the one or more target quality parameters include one or more of a target resolution of the content to be transmitted to the output device, a target frame rate of the content to be transmitted to the output device, and a target latency of the content to be transmitted to the output device.
5. 5. The method of claim 1, wherein the multiple resources for processing input and generating output comprise one or more of: graphical processor bandwidth in the server system, general purpose processor bandwidth in the server system, graphical memory in the server system, general purpose memory in the server system, storage capacity in the server system, and input / output channels in the server system.
6. 6. The method of claim 5, wherein providing multiple resources for processing input and generating output within the real-time interactive session comprises allocating respective portions of one or more of the multiple resources to the real-time interactive session.
7. 6. The method of claim 5, wherein providing multiple resources for processing input and generating output within the real-time interactive session comprises mapping the resource profile of the first virtual machine to respective portions of one or more of the multiple resources.
8. Selecting the first virtual machine includes: comparing the one or more target quality parameters to corresponding parameters in a resource profile of each of the plurality of virtual machines; determining a resource profile including parameters that most closely match the one or more target quality parameters; and selecting as the first virtual machine a virtual machine having a resource profile having parameters that most closely match the one or more target quality parameters.
9. Selecting the first virtual machine includes: comparing the one or more target quality parameters to corresponding parameters in a resource profile of each of the plurality of virtual machines; selecting one or more virtual machines having a resource profile with parameters equal to or greater than the one or more target quality parameters as candidate virtual machines; and selecting as the first virtual machine a candidate virtual machine having a least resource intensive resource profile.
10. 10. The method of claim 1, wherein selecting the first virtual machine includes associating the first virtual machine with the real-time interactive session, and establishing the real-time interactive session includes allocating resources to the real-time interactive session according to a resource profile of the first virtual machine.
11. 11. The method of claim 10, further comprising: maintaining an association between the first virtual machine and the real-time interactive session regardless of changes in network connection characteristics.
12. The method of claim 10 , further comprising maintaining an association between the first virtual machine and the real-time interactive session for a predetermined period of time.
13. 11. The method of claim 10, further comprising re-evaluating the selection of the first virtual machine after a predetermined period of time and / or upon detecting a change in network connection characteristics that exceeds a predetermined threshold.
14. Multiple virtual machines, one or more processors; and a memory storing one or more programs for execution by said one or more processors, said one or more programs including instructions for performing a method according to any of claims 1 to 13.
15. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by an electronic server having one or more processors, cause the electronic server to perform a method according to any one of claims 1 to 13.
16. 1. A method for controlling a game play process, comprising:
1. An electronic game server having one or more processors and a memory storing one or more programs for execution by said one or more processors, receiving an input event from a remotely located game controller during a game session operating in a current game state, the input event being a first command generated by user interaction with the game controller during a game session, the method further comprising: determining a first frame to be displayed at a remote location during the user interaction, the first frame comprising one of a plurality of output frames transmitted by the server during the game session before the server receives the input event, the method further comprising: determining a first game state associated with the first frame, the first game state being a game state prior to the current game state; (i) processing game play output according to the first command and (ii) the first game state; rendering a response frame based on the gameplay output; and and transmitting a response frame for display at the remote location.
17. Determining the first game state includes: determining a game time index associated with the first frame; and determining a game state that existed at a time corresponding to a game time indicator associated with the first frame.
18. Processing gameplay output is 18. A method according to any of claims 16 to 17, comprising inputting the input events into a game engine which determines a game play output based on the input events and a game state, the algorithm being based on the particular game being played.
19. Rendering the response frame includes: (i) introducing new virtual objects or (ii) modifying existing virtual objects in accordance with said gameplay output; including the new virtual object or the modified existing virtual object in the response frame; and and updating a subsequent game state according to the new or modified virtual object.
20. the current game state is associated with a second frame showing a game scene that is inconsistent with a game state intended by the user at the time of the user interaction; The method of any of claims 16 to 19, wherein processing the gameplay output includes adjusting the current game state to a game state intended by the user at the time of the user interaction.
21. 21. The method of claim 20, wherein adjusting the current game state to a game state intended by the user at the time of the user interaction comprises modifying a frame that depicts the current game state to create a subsequent frame that depicts the intended game state.
22. and receiving, at the server during the game session, a plurality of display events from the remotely located media device, each display event including frame identification data for a respective frame displayed at the remote location; Determining the first frame displayed at a remote location during the user interaction includes: determining which of a plurality of display events was received closest in time to the input event; and identifying a frame associated with the most recently received display event as the first frame based on frame identification data included in the most recently received display event.
23. the input event further comprises a first timestamp indicating a timing of the user interaction; Determining the first frame displayed at the remote location during the user interaction includes: assigning, at the server, a second timestamp to the received input event, the second timestamp indicating a time of receipt at the server; determining a one-way transit time of the input event from the controller to the server based on a comparison of the first timestamp and a second timestamp; approximating a one-way transit time from the server to a remotely located media device based on the determined one-way transit time of the input event from the controller to the server; and identifying as the first frame a frame previously transmitted by the server to the media device according to the approximated one-way transit time from the server to the media device.
24. the input event further comprises a first timestamp indicating a timing of the user interaction; Determining the first frame to be displayed at the remote location during the user interaction includes: including the first timestamp in metadata of the response frame; and transmitting the response frame having the first timestamp to a remotely located media device for use in determining a delay time from a controller to a media device.
25. Before receiving the input event, Rendering an output frame; and Including a first timestamp in metadata of the output frame, the first timestamp indicating a drawing timing or a transmission timing of the output frame; transmitting the output frame having the first timestamp to a remotely located media device for use in determining a one-way server-to-media device delay time; Determining the first frame to be displayed at a remote location during the user interaction includes: receiving, at the server, a display event from the remotely located media device, the display event including a one-way server-to-media device latency value based on the one-way server-to-media device latency determination; and A method according to any of claims 16 to 24, comprising identifying the output frame as a first frame based on the determined one-way server-to-media device delay value.
26. Before receiving an input event, Rendering an output frame; and Including a first timestamp in metadata of an output frame, the first timestamp indicating a drawing timing or a transmission timing of the output frame; transmitting an output frame having a first timestamp to the remotely located media device; Determining a first frame to be displayed at a remote location during the user interaction includes: receiving, at the server, a display event from a remotely located media device, the display event including a first timestamp and a second timestamp, the second timestamp indicating a display timing of the output frame; based on a comparison of the first timestamp and the second timestamp, determining a delay time value from the server to the media device; and identifying the output frame as a first frame based on the determined one-way server-to-media device delay value.
27. Before receiving an input event, Rendering the output frame; tagging the output frames with a first audio signal; transmitting the output frames along with the first audio signal to a remotely located media device; and associating a transmission timing of the output frame with a first timestamp; Determining a first frame to be displayed at a remote location during the user interaction includes: and (i) receiving an audio signal acknowledgment, the audio signal acknowledgment remotely confirming detection of the first audio signal by the game controller, and (ii) receiving a second timestamp, the second timestamp indicating a timing of transmission of the input event from the controller to the server; determining a one-way server-to-controller delay time value based on a comparison of the first timestamp and the second timestamp; determining a one-way controller-to-server latency value based on a comparison of the one-way server-to-controller latency value and a round trip time (RTT) latency value from the server to the controller; and identifying the output frame as a first frame based on the determined one-way controller-to-server delay time value.
28. Before receiving an input event, Rendering the output frame; tagging the output frames with a first image pattern; transmitting the output frame having the first image pattern to a remotely located media device; and associating a transmission timing of the output frame with a first timestamp; Determining a first frame to be displayed at the remote location during the user interaction includes: receiving, upon an input event from the game controller, (i) an image pattern acknowledgment response, the image pattern acknowledgment response remotely acknowledging detection of a first image pattern by the game controller; and (ii) a second timestamp, the second timestamp indicating a timing of transmission of the input event from the controller to the server; determining a one-way server-to-controller delay time value based on a comparison of the first timestamp and the second timestamp; determining a one-way controller-to-server latency value based on a comparison of the one-way server-to-controller latency value and a round trip time (RTT) latency value between the server and the controller; and identifying the output frame as a first frame based on the determined one-way controller-to-server delay time value.
29. one or more processors; and a memory storing one or more programs for execution by said one or more processors, said one or more programs including instructions for carrying out a method according to any of claims 16 to 28.
30. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by the one or more processors, cause the electronic game server to perform a method according to any one of claims 16 to 28.
31. 1. A method of rendering an online interactive gaming session, comprising:
1. A server system including one or more processing cores and a memory storing a program for execution by the one or more processing cores, receiving a first command from a first client device associated with the online game session; determining a first command type and a first expected response delay time associated with the first command type; Determining a network delay time; determining a first introduced delay based on a comparison of the network delay and the first expected delay; generating a first number of intermediate frames that, when transmitted at a predetermined frame rate, occupy a transmission time corresponding to the first introduced delay time; generating a first response frame reflecting an initial result of the first command; transmitting the first response frame, followed by the first number of intermediate frames, at a predetermined frame rate such that the first response frame is received at a first media device associated with the first client device at a time corresponding to a first expected response delay time.
32. determining the predetermined frame rate according to network characteristics of the online game session; 32. The method of claim 31, further comprising: upon determining a subsequent network delay time different from the first, maintaining the predetermined frame rate.
33. 33. The method of claim 31 or 32, further comprising updating a subsequent game state according to the first command, the subsequent game state being initially reflected in the first response frame.
34. Determining the first number of intermediate frames comprises: determining an expected arrival time of a response frame based on the network delay time; determining a target arrival time of a response frame based on the expected response delay time; comparing the predicted arrival time to the target arrival time to obtain a delta time; and multiplying the delta time by a predetermined frame rate.
35. A method according to any of claims 31 to 34, wherein determining the first number of intermediate frames comprises multiplying the introduced delay time by the predetermined frame rate.
36. determining a processing delay associated with executing the first command on the server, the first introduced delay time further comprising: The method according to any one of claims 31 to 35, which is based on
37. A method according to any one of claims 31 to 36, wherein determining the network delay time comprises determining (i) a round-trip communication delay time between a first client device and the server, and / or (ii) a round-trip communication delay time between the server and the first media device.
38. A method according to any one of claims 31 to 37, wherein determining the network delay time comprises determining (i) a one-way communication delay time between the first client device and the server, and / or (ii) a one-way communication delay time between the server and the first media device.
39. The method further comprises receiving a second command from the first client device, the second command type being different from the first command type, the method further comprising: determining a second expected response delay associated with the second command type, the second expected response delay being different from the first expected response delay, the method further comprising: determining an updated network delay time; determining a second introduced delay based on a comparison of the updated network delay and the second expected delay; generating a second number of intermediate frames that, when transmitted at the predetermined frame rate, occupy a transmission time corresponding to the second introduced delay time; generating a second response frame reflecting an initial result of the second command; 39. The method of claim 31, further comprising: transmitting the second response frame at the predetermined frame rate followed by the second number of intermediate frames such that the second response frame is received at the first media device at a time corresponding to the second expected response delay time.
40. receiving a third command from a second client device, the third command having a type equal to a type of the first command; associating the third command with a first expected response delay time; determining that a network latency associated with the second client device is different from a network latency associated with the first client device; determining a third introduced delay time different from the first introduced delay time based on a comparison of a network delay time associated with the second client device to the first expected delay time; generating a third number of intermediate frames, different from the first number of intermediate frames, that when transmitted at the predetermined frame rate occupies a transmission time corresponding to the third introduced delay time; generating a third response frame reflecting an initial result of the third command; 40. The method of claim 31, further comprising: transmitting the third response frame at the predetermined frame rate, following the third number of intermediate frames, such that the third response frame is received at a second media device at a time corresponding to the first expected response delay time.
41. receiving a fourth command from the first client device, the fourth command having a type equal to a type of the first command; Associating the fourth command with a first expected response delay time; determining that a current network latency differs from a network latency associated with the first command; determining a fourth introduced delay time different from the first introduced delay time based on a comparison of the updated network delay time and the first expected delay time; generating a fourth number of intermediate frames different from the first number of intermediate frames, the fourth number of intermediate frames occupying a transmission time corresponding to the fourth introduced delay time when transmitted at the predetermined frame rate; generating a fourth response frame reflecting an initial result of the fourth command; 41. The method of claim 31, further comprising: transmitting the fourth response frame at the predetermined frame rate, following the fourth number of intermediate frames, such that the fourth response frame is received at the first media device at a time corresponding to the first expected response delay time.
42. determining a second network latency associated with a second client device associated with the online game session; determining a fifth introduced delay time based on a comparison of the second network delay time and the first expected delay time; and determining a fifth number of intermediate frames that, when transmitted at a predetermined frame rate associated with the second client device, occupy a transmission time corresponding to the fifth introduced delay time; in response to determining that the fifth number of intermediate frames is higher than the first number of intermediate frames, generating one or more additional intermediate frames until the fifth number of intermediate frames are generated; generating a fifth response frame reflecting an initial result of the first command; and transmitting the fifth response frame, followed by the first number of intermediate frames, followed by the one or more additional intermediate frames, at the predetermined frame rate associated with the second client device, such that the fifth response frame is received at a media device associated with the second client device at a time corresponding to a first expected response delay time.
43. in response to determining that the fifth number of intermediate frames is less than the first number of intermediate frames, generating a sixth response frame reflecting an initial result of the first command; 43. The method of claim 42, further comprising: transmitting, following the fifth response frame, a subset of the first number of intermediate frames corresponding to the fifth number of intermediate frames at a predetermined frame rate associated with the second client device such that the fifth response frame is received at a media device associated with the second client device at a time corresponding to the first expected response delay time.
44. one or more processors; and a memory storing one or more programs for execution by said one or more processors, said one or more programs including instructions for carrying out a method according to any of claims 31 to 43.
45. A non-transitory computer readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by an electronic game server having the one or more processors, cause the electronic game server to perform a method according to any one of claims 31 to 43.
46. 1. A method comprising:
1. A server system having one or more processors and a memory storing one or more programs for execution by said one or more processors, establishing a real-time interactive game session with a first client device, the game session being associated with a particular game type, the method comprising: monitoring in-game performance data associated with a user of the first client device during the game session; determining an acceptance level of a game playing experience for a user of the first client device according to the in-game performance data; and adjusting game session resources based on the gameplay experience acceptance level, the game session resources including frame rate, resolution, latency level, or streaming source.
47. 47. The method of claim 46, wherein monitoring in-game performance data includes determining a skill level of a user of the first client device and comparing the skill level to an average skill level representing a plurality of users of other client devices.
48. 48. The method of any of claims 46 to 47, wherein monitoring in-game performance data comprises evaluating game play performance with respect to one or more predefined game-specific game play performance benchmarks.
49. 49. The method of any one of claims 46 to 48, wherein monitoring in-game performance data includes monitoring interaction response data, the interaction response data representing a time between a rendered stimulus and a corresponding gameplay input received from the first client device.
50. The method of any of claims 46 to 49, wherein monitoring in-game performance data includes monitoring controller interaction data, the controller interaction data representing the time between subsequent user manipulations of a game controller.
51. The method according to any of claims 46 to 50, wherein assigning a gameplay experience acceptance level is further according to said particular game type.
52. A method according to any one of claims 46 to 51, wherein assigning a game playing experience tolerance level is further in accordance with user preference data relating to game session resources.
53. 53. The method of claim 52, wherein the user preference data is reported by the user or inferred from the in-game performance data.
54. 54. A method according to any of claims 46 to 53, wherein the acceptable gameplay experience level represents a minimum frame rate determined to have a below-threshold amount of impact on a perceived gameplay experience associated with a user of the first client device, and adjusting the game session resources comprises setting the frame rate to a minimum level.
55. 55. The method of any one of claims 46 to 54, wherein the gameplay experience tolerance level represents a minimum resolution determined to have a below-threshold amount of impact on a perceived gameplay experience associated with a user of the first client device, and adjusting the game session resources comprises setting the resolution to a minimum level.
56. 56. A method according to any of claims 46 to 55, wherein the acceptable gameplay experience level represents a maximum level of latency determined to have a below-threshold amount of impact on a perceived gameplay experience associated with a user of the first client device, and adjusting the game session resources comprises setting the latency level to a maximum level.
57. 57. The method of claim 46, wherein the gameplay experience tolerance level represents a particular frame rate, a particular resolution, and / or a particular latency level determined to have a less than threshold amount of impact on a perceived gameplay experience associated with a user of the first client device, and adjusting the game session resources includes switching from a current streaming source to a different streaming source optimized for the particular frame rate, the particular resolution, and / or the particular latency level.
58. Following adjusting the game session resources, monitoring in-game performance data associated with the first client device; determining whether a change in the in-game performance data exceeds a predetermined threshold; in response to determining that a change in in-game performance data exceeds the predetermined threshold, reassigning an acceptable game playing experience level to the first client device according to the in-game performance data; The method of any of claims 46-57, further comprising: readjusting said game session resources based on said reallocated gameplay experience tolerance level.
59. 1. An electronic server comprising: one or more processors; and a memory storing one or more programs for execution by said one or more processors, said one or more programs including instructions for performing a method according to any of claims 46 to 58.
60. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by an electronic server having one or more processors, cause the electronic server to perform a method according to any of claims 46 to 58.
Citation Information
Patent Citations
Program, information storage medium, and game machine
JP2010029393A
Information processing system, information processing device, information processing program, and information processing method
JP2015082180A
