Shadow tracing for real-time interactive simulation of complex systems analysis

Abstract

An electronic computing system saves a pre-error state of a processing unit by: receiving a first stream of input; buffering the first stream of input to generate a buffered stream of input identical to the first stream of input; transmitting the first stream to a primary instance of a first program; transmitting the buffered stream to a secondary instance of the first program; executing the primary instance using the first stream in real time; executing the secondary instance using the buffered stream with a predefined time delay relative to executing the primary instance using the first stream; detecting an error state resulting from executing the primary instance; and responsive to detecting the error state, pausing the secondary instance and saving a current state of the secondary instance, wherein the current state of the secondary instance corresponds to the pre-error state of the primary instance.

Description

Technical Field

[0001] The present application relates generally to computer technology, including but not limited to methods and systems for preserving one or more pre-error states during processing of real-time user-interactive applications. Background Art

[0002] Troubleshooting a real-time interactive application can be difficult if it enters an abnormal or failed state. Failures can result in crash dumps and logs, which in some cases indicate the code paths that were executed. However, developers trying to understand the set of inputs or stimuli that caused the application to enter an abnormal or failed state do not have many tools at their disposal. Therefore, a system is needed that provides more detailed and useful information about the application's state before it failed. Summary of the Invention

[0003] The embodiments described herein are directed to providing a processing system that implements a delayed version of a primary instance of a real-time interactive application. The delayed version operates using the same inputs as the primary instance, but runs at a configurable delay behind the primary instance. If the primary instance enters an undesirable state, this allows the delayed instance to be paused before the undesirable state occurs, thereby enabling inspection, stepping, and other diagnostic functions.

[0004] In one aspect, some embodiments include a method for saving a pre-error state of a processing unit implemented on a computer system having one or more processors and a memory storing one or more programs for execution by the one or more processors. The method includes receiving a first stream of input; buffering the first stream of input to generate a buffered stream of input identical to the first stream of input; transmitting the first stream to a primary instance of a first program; transmitting the buffered stream to a secondary instance of the first program; executing the primary instance using the first stream in real time; executing the secondary instance using the buffered stream with a predefined time delay relative to executing the primary instance using the first stream; detecting an error state resulting from executing the primary instance; and responsive to detecting the error state, pausing the secondary instance and saving a current state of the secondary instance, wherein the current state of the secondary instance corresponds to the pre-error state of the primary instance.

[0005] In some embodiments, the error state results from executing the primary instance using a first input in a first stream (e.g., the first input causes the error state), and the method includes pausing the secondary instance before processing an input in a buffered stream that corresponds to the first input in the first stream (e.g., pausing the secondary instance before the secondary instance has an opportunity to process an equivalent of the first input, thereby preventing the error state from occurring in the secondary instance).

[0006] In some embodiments, the auxiliary instance runs concurrently with the primary instance after a predefined time delay. In some embodiments, the auxiliary instance runs after the primary instance terminates (e.g., due to a failure).

[0007] In some embodiments, the error state masks aspects of a pre-error state of the primary instance; and saving the current state of the secondary instance includes recording aspects of the current state of the secondary instance that correspond to aspects of the pre-error state of the primary instance, thereby preserving aspects of the pre-error state that would have been hidden if the instance had not been promptly suspended. In some embodiments, the aspects of the pre-error state of the primary instance are first stored data associated with the primary instance, and the aspects of the current state of the secondary instance are second stored data associated with the secondary instance, and the second stored data corresponds to the first stored data, thereby allowing data that would otherwise be lost or unreadable in the event of a failure to be preserved.

[0008] In some embodiments, executing the primary instance includes generating a first output stream using a first processing unit; executing the secondary instance includes generating a second output stream using a second processing unit. In some embodiments, detecting an error state includes detecting an indicator of an error state (e.g., faulty data, a flag, missing expected data, etc.) in the first output stream. In some embodiments, detecting the error state includes detecting the error state from an indicator (e.g., a fault signal) generated by the first processing unit.

[0009] In some embodiments, the method further includes, after saving the current state of the auxiliary instance, (i) resuming the auxiliary instance to obtain a subsequent state of the auxiliary instance, and (ii) saving the subsequent state of the auxiliary instance, the subsequent state of the auxiliary instance corresponding to a subsequent pre-error state of the primary instance, thereby allowing a state closer to the failure state to be saved and, as a result, providing more relevant data about potential causes of the failure before the failure occurred. In some embodiments, pausing the auxiliary instance includes stopping the delivery of the buffered stream to the auxiliary instance, and resuming the auxiliary instance includes delivering a single input in the buffered stream to the auxiliary instance, thereby allowing a more controlled approach to an impending failure, resulting in greater data accessibility.

[0010] In some implementations, saving the current state of the auxiliary instance includes providing the current state of the auxiliary instance for inspection (eg, by a troubleshooter or a programmer).

[0011] In some implementations, some of the operations or subsets of operations described above may be combined and / or the order of some of the operations or subsets of operations may be changed.

[0012] According to some aspects of the present application, a computer system includes a memory storing instructions for causing the computer system to perform any of the above operations.

[0013] Furthermore, according to some aspects of the present application, instructions stored in a memory of a computer system include instructions for causing the computer system to perform any of the operations described above.

[0014] Other embodiments and advantages will be apparent to those skilled in the art from the description in this specification and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals refer to corresponding parts throughout the various figures.

[0016] Figure 1 is an example online interactive gaming environment according to some implementations.

[0017] Figure 2 is a block diagram illustrating an example client device of a gaming environment according to some implementations.

[0018] Figure 3 is a block diagram illustrating an example media device of a gaming environment according to some implementations.

[0019] Figure 4 is a block diagram illustrating an example server of a gaming environment according to some implementations.

[0020] Figure 5A Depicted is an example gaming environment according to some implementations.

[0021] Figure 5B and Figure 5C Depicted is an example gaming scenario according to some implementations.

[0022] Figure 6 is a flow chart of a game play process according to some embodiments.

[0023] Figure 7 is an example processing system according to some implementations.

[0024] Figure 8 is a block diagram illustrating an example processing system in accordance with some implementations.

[0025] Figure 9 An example scenario illustrating a game play process according to some embodiments is shown.

[0026] Figure 10 is a flow chart illustrating an example method for preserving a pre-error state in a processing system according to some implementations.

[0027] Like reference numerals designate corresponding parts throughout the drawings. DETAILED DESCRIPTION

[0028] The embodiments described herein are intended to provide a processing environment that enables a delayed version of a primary instance of a real-time interactive application. The delayed version (also referred to herein as a "shadow process") operates with the same inputs as the primary instance, but with a configurable delay relative to the primary instance. If the primary instance enters an undesirable state, the delayed instance is paused before the undesirable state occurs, enabling inspection, stepping, and other diagnostic functions.

[0029] Sample game environment

[0030] In order to provide more context for some embodiments described herein, an example gaming environment is now described. The embodiment of the example gaming environment described in this section is intended to provide a cloud platform and API to achieve efficient, portable, low-latency hosting of cloud gaming content including third-party gaming content. Some embodiments dynamically allocate cloud gaming hardware resources (e.g., CPU, GPU, memory, input / output, and video stream encoders) and monitor and utilize the network bandwidth available to individual end users to simultaneously provide the best online gaming experience to the community of gamers. Some embodiments provide multiple performance layers, including layers that support high-performance, real-time gaming sessions, and high-definition media streams for end users. Some embodiments support different subscription models and / or are configured to provide one or more concurrent real-time gaming entertainment (gameplay) and / or review media streams that are consistent with one or more actual game streams (e.g., video streams output to the client devices of users participating in online / cloud gaming sessions via mobile applications or browser-based programs) with almost no delay or no delay. In some implementations, real-time game play and / or review media streams are provided to one or more users with little or no delay via a media streaming website such as YouTube.

[0031] In some embodiments of a cloud gaming environment, a server system provides hardware resources for a real-time, interactive gaming session for processing player input and generating output streams for display to one or more players and, optionally, to a spectator of the game. In response to a request to establish a real-time, interactive gaming session, the server system determines the device capabilities (e.g., hardware and / or software capabilities) of the requesting client device (i.e., the player's controller device), the connection capabilities (e.g., bandwidth, latency, and / or error rate) of the network connection, and one or more target quality parameters (e.g., resolution of the output video stream, game response latency, etc.) of the gaming session, and, accordingly, associates one of its virtual machines with the real-time, interactive session to establish the session.

[0032] In some embodiments, the processing and encoding capabilities of game data (e.g., generating output video streams for players and / or spectators) are managed for one or more processing cores (e.g., GPU cores and encoder cores) in a server system that hosts a real-time, online, and interactive gaming environment. For example, in some embodiments, one or more processing cores operate with multiple processing slices (e.g., each processing slice executes for 16.67 ms on the core), and the server system assigns each of the multiple processing slices to a subset of multiple online game sessions to be executed thereon. For one of the processing slices, the server system determines a time-sharing processing schedule such that the corresponding subset of game sessions share the duty cycle of the processing slice and execute in parallel according to their corresponding real-time data processing needs. In addition, to accelerate image encoding within a time interval, the server system's encoder does not need to wait until the GPU has made all the data for the image frame available. Instead, in some embodiments, a portion of the image frame is encoded as soon as the GPU provides the information required to encode the portion of the frame, independent of whether other portions of the image frame unrelated to the encoded portion are available to the GPU.

[0033] Furthermore, the server system can dynamically generate multiple frames in response to user commands received from a user playing an online game session. Based on the type of user command, the server system determines the expected response latency, the actual communication and processing latency, and the actual transmission latency. The user command is then executed in the online game session by generating a frameset that reflects the effect of the command. The frameset, when transmitted at a predetermined frame rate, takes a transmission time corresponding to the actual transmission latency and can be received at the user's client device within a time corresponding to the expected response latency.

[0034] Figure 1 1 is an example online interactive gaming environment 100 according to some embodiments. The online interactive gaming environment 100 includes one or more client devices (e.g., client devices 102 and 104). Each client device 102 executes one or more game applications. A game session can be run on a particular game application to allow a user of a client device 102 to play an online interactive game hosted by a server system 114. In some embodiments, a client device 102 (e.g., a host client) is configured to invite one or more other client devices 102 to join a game scene of a particular game application. The game sessions of these client devices 102 are synchronized to display the same game scene, optionally with different perspectives corresponding to their respective users.

[0035] Instead, the server system 114 hosts an online interactive gaming platform to support client devices 102 in playing one or more gaming applications, including a specific gaming application. Specifically, the server system 114 includes multiple user accounts associated with the client devices 102 and authenticates the user of the client device in association with each of the one or more gaming applications. The server system 114 renders and refreshes the scene of the online interactive game on the client device 102 that joins the corresponding gaming session associated with the scene. In some embodiments, the server system 114 assesses the capabilities of the client device 102 and / or the quality of the communication connection between the server system 114 and each client device 102, and adaptively generates a synchronized data stream for the gaming session associated with the client device 102. Through these means, the server system 114 is configured to facilitate simultaneous and substantially low-latency gaming sessions of an online interactive game on two or more client devices 102.

[0036] In some embodiments, the server system 114 includes a game server 122 and a media streaming server 124. The game server 122 is configured to simultaneously provide two or more media streams for an online interactive game session running on a first client device 102A. The two or more media streams include a low-latency stream and a normal-latency stream, which are provided to the first client device 102A and the reviewer client device 104, respectively, via one or more communication networks 112. Optionally, the normal-latency stream is provided for educational purposes. When a user of the first client device 102A plays a game session on the first client device 102A, the game session is recorded and broadcast to one or more viewers via the normal-latency stream, i.e., the viewers can review the game session on the reviewer client device 104. The low-latency stream corresponds to the game play of the online interactive game session and has a faster response rate and lower transmission delay than the normal-latency stream corresponding to the associated review session. For example, the low-latency stream has a predefined frame rate of 60 frames per second (fps) and provides at least one frame during each 16.67 ms time interval to the first client device 102A, and the normal-latency stream has a predefined frame rate of 30 fps and provides at least one frame during each 33.33 ms time interval to the reviewer client device 104. In some implementations, the normal-latency stream has a lower resolution than the low-latency stream.

[0037] In some embodiments, client device 102 or 104 has a display screen integrated therein for displaying media content. In some embodiments, client device 102 or 104 is coupled to media device 106 and output device 108. Specifically, client device 102 or 104 can be communicatively coupled to media device 106 directly (e.g., via a Bluetooth or other wireless communication link), via a local network 110 (e.g., a Wi-Fi network), or via one or more communication networks 112. In some embodiments, client device (102 or 104) and media device 106 are located locally to each other (e.g., in the same room, in the same house, etc.). Media device 106 is further coupled to one or more output devices 108 (e.g., a television, display monitor, sound system, speakers, etc.) that can output visual and / or audio content. Media device 106 is configured to output content to output device 108. In some embodiments, media device 106 is a casting device (e.g., Chromecast by Google Inc.) or otherwise includes casting functionality.

[0038] In some embodiments, one or more client devices 102 or 104 can communicate data and share information with each other, a central server or cloud computing system (e.g., server system 114), and / or other network-connected devices (e.g., another client device 102 or 104, a media device 106, and an output device 108). Data communications can be performed using any of a variety of custom or standard wireless protocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.11a, WirelessHART, MiWi, etc.) and / or any of a variety of custom or standard wired protocols (e.g., Ethernet, HomePlug, etc.), or any other suitable communication protocol, including communication protocols not yet developed as of the date of this document. In some embodiments, the online interactive gaming environment 100 includes conventional network equipment (e.g., a router) via which a collection of client devices 102 and 104 and their corresponding media and output devices (if any) are communicatively coupled to each other on a local network 110 (e.g., a local area network), and the local network 110 is communicatively coupled to a communication network 112 (e.g., a wide area network and the Internet). In some embodiments, each of the client devices 102 and 104 optionally communicates with one or more other client devices, a corresponding media device 106, or a corresponding output device 108 using one or more radio communication networks (e.g., ZigBee, Z-Wave, Insteon, Bluetooth, Wi-Fi, and / or other radio communication networks).

[0039] In some embodiments, the client devices 102 are located far from each other, i.e., they are not located in the same room or even building. Figure 2 102B) to start a game for execution on each client device 102. In some embodiments, the game application independently establishes an online game session 116 with the server system 114 for each client device 102. The online game sessions 116 of two or more client devices 102 (e.g., 102A and 102B) are related to each other (e.g., because they are all playing in the same game domain of the game application) and, therefore, share a game scene in the game application. The related online game sessions 116 are synchronized with each other, and each online game session 116 optionally shows the same game scene from a unique player perspective corresponding to the corresponding client device 102. Thus, the user of each client device 102 can play the game on the corresponding client device and affect the output of the online game sessions 116 on other client devices 102.

[0040] Alternatively, in some other embodiments, after the gaming application of the first client device 102A establishes an online gaming session 116, it invites one or more second client devices 102B to join the online gaming session 116 via an invitation message. For example, a message containing a link (e.g., a URL address) to join the online gaming session 116 is sent to each of the second client devices 102B. Appropriate controller configurations are provided to each second client device 102B invited to join the online gaming session 116. In this application, when the second client devices 102B join the online gaming session 116, the server system 114 creates a separate gaming session for each individual second client device 102B. Each individual online gaming session 116 of the corresponding second client device 102B is synchronized with the online gaming session 116 of the first client device 102A and shares the same scene with the online gaming session 116, but may have a unique player perspective corresponding to the corresponding second client device 102B. After each second client device 102B has received the appropriate controller configuration and joined the online game session 116 (more precisely, started its associated online game session 116), the user can play the game on the corresponding second client device 102B and affect the output of the online game session 116 running on the other client devices 102.

[0041] The client device 102 is a device that includes and can run one or more different user applications including a gaming application. In some embodiments, the client device 102 is a smartphone, a tablet device, a laptop computer, a desktop computer, or a multimedia device. In some embodiments, the client device 102 is a dedicated game controller that includes game controls (e.g., one or more buttons, joysticks, touch screen affordances, motion controls, pressure controls, visual controls, audio controls, and / or other tactile interfaces) configured to control certain aspects of gaming when activated or otherwise manipulated. In some embodiments, the client device 102 includes one or more user applications configured to operate in conjunction with the media device 106. In some embodiments, the application includes a media device application for pairing the client device 102 with the media device 106 and configuring the media device 106. The application also includes one or more applications that can deliver associated content to the media device 106. In some implementations, the application delivers data and / or content to the media device 106 by sending the data / content directly to the media device 106 (e.g., via a local network) and / or by directing the media device 106 to a remote location (e.g., a URL or other link to a location at a server system) from which the media device 106 can stream or otherwise receive the data / content. The media device 106 receives the data / content from the application and / or remote location and outputs visual and / or audio content corresponding to the received data / content to the output device 108. Thus, an online gaming session 116 is established between the gaming application running on the client device 102, the remote server system 114, and the media device 106.

[0042] In some embodiments, as part of the process of linking related online game sessions 116, the server system 114 assesses the capabilities of each corresponding client device 102 and / or the quality of the communication connection between the server system 114 and the client device 102. In some embodiments, the server system 114 measures the network latency between the client device 102 and the server system 114. If the measured latency is above a threshold and a lower-latency connection is available, the server system 114 can recommend that the client device 102 change to a lower-latency connection, or invite the user of the client device 102 to change the client device 102 to a lower-latency connection. For example, if the client device 102 is on a cellular wireless connection 118 and a local network is available, the server system 114 can recommend that the client device 102 connect via the available local network. In some embodiments, the latency threshold requirement varies between games. For example, some games (e.g., action games) are best experienced over a lower-latency connection, while other games (e.g., online board or card games) may not require high latency. The server system 114 may make connection recommendations in light of these different requirements associated with different types of games.

[0043] In some embodiments, as part of a client device 102 starting or joining an online game session 116, the server system 114 communicates with the client device 102 to set up a controller (e.g., a game controller configuration and / or interface) on the client device 102. In some embodiments, this includes the server system 114 assessing whether the client device 102 has the resources and communication capabilities required for a controller. Depending on the available resources at the client device 102, the quality of the connection, and the game requirements, the controller may be implemented differently at the client device 102. In some embodiments, a game may be played using a web-based controller interface. For example, the game's controller interface may be embedded in a web page, and the web page rendered in a web browser on the client device 102. Alternatively, in some embodiments, a standardized controller is implemented in a predefined application that is not specific to or directly associated with a game (e.g., a delivery device application, such as Chrome or Google Cast from Google Inc., or other media device applications), or in the operating system of the client device 102. For example, the device operating system or predefined application on the client device 102 may have a controller submodule. The controller submodule includes one or more standardized controller configurations, templates, etc. Each standardized controller configuration configures the controller submodule to implement a virtual controller in some manner utilizing input devices and / or sensors on the client device 102. The standardized controller configuration used may vary depending on the type of game and / or client device.

[0044] Furthermore, in some embodiments, a game may have a specific controller configuration that can be implemented on a controller submodule. Such a configuration may be stored at the server system 114 and transmitted to the client device 102 as part of the process by which the client device 102 joins or starts an online game session 116. In some embodiments, the specific controller configuration may be a completely custom controller or a mix of standard and custom controllers. Additionally, in some embodiments, a game may require a specific application associated with the game. For example, a game may require a controller application specifically associated with the game. In some embodiments, the client device 102 may be instructed to download a specific application or a predefined application as part of starting or joining an online game session 116. For example, if the client device 102 does not already have a predefined application (and controller submodule) or a specific application associated with the game, and such an application is required for playing the game, the server system 114 may instruct the client device 102 to prompt its user to download the application and request permission from the user to proceed.

[0045] In some implementations, the server system 114 stores information associated with each of the one or more game applications hosted on the server system 114 (e.g., Figure 2 In some embodiments, the server system 114 stores session data associated with online game sessions played on the client device 102. Examples of session data for each online game session include, but are not limited to, frame rate, rendering specifications, normal latency requirements, GPU allocation information, encoder allocation information, identification of the associated session, and latest status information.

[0046] In some embodiments, server system 114 provides a gaming API and cloud platform to enable efficient, portable, low-latency hosting of third-party gaming content used in online gaming sessions 116. In some embodiments, the gaming API and cloud platform are implemented by server system 114, which further includes one or more of the following: a front-end server 134, a media streaming server 124, a gaming server 122, and one or more third-party content servers 136. In some embodiments, the gaming API platform is created and / or hosted by gaming server 122 and, in conjunction with front-end server 134 and content server 136, enables online gaming sessions 116. Front-end server 134 is configured to provide services to users of online gaming sessions 116 and manage their accounts. Optionally, users subscribe to gaming services via front-end server 134. Content server 136 provides gaming content related to online gaming sessions 116.

[0047] In some embodiments, the front-end server 134 manages user accounts associated with the client devices 102 and 104, such as subscriptions to memberships in one or more online interactive games through the user accounts. After a client device 102 logs into its corresponding user account and joins its online game session 116, the game server 122 sets up the online game session 116 and manages each specific online game session 116 for the corresponding client device 102 by retrieving game content from the content server 136, sending the game content to a game application executing on the client device 102, identifying user requests or actions, rendering game play output for the client device 102 in response to the user requests or actions, and storing game state data during the corresponding online game session 116. The game server 122 includes one or more processing units (e.g., a CPU 138, a GPU 140, and an encoder 142), memory 146, and a data buffer 144 that temporarily stores multimedia content generated by the GPU 140 and provides the multimedia content to the encoder 142 for further encoding (e.g., normalization or compression). The data buffer 144 is optionally integrated into the memory 146 or separate from the memory 146 .

[0048] In some embodiments, the game server 122 dynamically allocates cloud gaming hardware resources (e.g., GPU 140 and encoder 142) and monitors and utilizes the network bandwidth available to individual end users to provide the best cloud gaming experience. In some embodiments, the game server 122 provides multiple performance tiers, including a tier that supports high-performance, real-time gaming sessions using high-definition video / media streaming. In some embodiments, the game server 122 supports different subscription models and / or is configured to provide one or more concurrent real-time gaming entertainment and / or review media streams that are consistent with one or more actual game streams (e.g., video streams output to the client devices of users participating in online / cloud gaming sessions via mobile applications or browser-based programs) with little or no delay. Specifically, the game server 122 is configured to generate concurrent media streams and review videos for gaming entertainment, and provides the review videos for concurrent gaming entertainment to the media streaming server. Such review videos are provided to one or more users via media streaming websites, such as YouTube, with little or no delay. The media streaming site is optionally managed by the media streaming server 124.

[0049] Some embodiments enable hosting of public events in conjunction with gaming tournaments. For example, in conjunction with a multiplayer gaming event or tournament based on a hosted game, a cloud gaming site hosted by a gaming server 122 may optionally broadcast or stream the following to a specific reviewer client device 104 via a media streaming server 123: (a) one or more concurrent auxiliary or supplemental media streams, including an associated commentary track / stream, (b) a gaming stream from the perspective of different competitors, a highlight stream showing particularly compelling gaming action based on cloud server analysis and / or scoring of multiple gaming sessions associated with the gaming event, (c) one or more gaming viewpoint streams reflecting online gaming sessions 116 of one or more active players, and / or (d) a tutorial track from one or more active players and / or reviewers, which may include real-time picture-in-picture (PIP) video of the active players sent to the cloud gaming server system 114 and their corresponding gaming responses.

[0050] According to some embodiments, examples of third-party content that can be effectively hosted by content server 136 include, but are not limited to, sports games, racing games, role-playing games (RPGs), and first-person shooter (FPS) games. Different instances of these games may have widely varying cloud hardware requirements and networks based on different associated latency requirements and expectations, output video resolutions and game server computational workloads and video encoding / streaming resources, and network bandwidth (e.g., to ensure an optimal user gaming experience—in some cases consistent with different subscription performance tiers).

[0051] In some embodiments, the front-end server 134 provides an account management API and / or software module that monitors game play activity and subscriber-related requests (e.g., requests by end users to invite other players to participate in game sessions, upgrade their game tools, and / or game performance) and transmits or makes available the associated information to the third-party content server 136 via the API to enable content providers to track their subscribers and / or followers' settings (including but not limited to billing information, in-game credits, subscription levels, etc.). In some embodiments, the content provider of hosted content can offer one or more different subscription models for hosted content via the same hosting platform. In some embodiments, users (e.g., subscribers of a gaming service) are granted unlimited access and gameplay to all games offered by the content provider on the hosting platform. In some embodiments, users are granted unlimited access and gameplay to one or more specific game franchises (e.g., a specific football or first-person shooter franchise) offered by the content provider on the hosting platform. In some embodiments, a subscription is for limited participation by a user - participation can be limited based on game play time, hardware resource levels committed by the end user, or end user device type / location. In some embodiments, the account API and modules configure and monitor game play sessions and enable content providers to track the gaming activity of corresponding subscribers based on their latest subscription information - even during active game play.

[0052] The server system 114 implements cloud functionality that allows users to move back and forth, for example, suspending a first game stream of a first game session executed on a first client device 102A and resuming the first game stream on a second game session of a second client device 102B to continue the first game session. The server system 114 also supports multiple players at scale and provides richer, more persistent cloud-based worlds. The server system 114 uses a cloud-based system to store session data related to different online game sessions 116 of the same user or different online game sessions 116 of different users.

[0053] The server system 114 renders game content on a plurality of client devices 102 and 104, including but not limited to mobile phones, tablet computers, desktop computers, and televisions. Optionally, the game content is dynamically adjusted to conform to the specifications of these client devices 102 and 104. In some embodiments, the client devices 102 and 104 have limited or no storage capabilities because the game API platform provides instant access and requires no or minimal user device storage (e.g., a user can start playing in 5 seconds and save 250GB of console hard drive space).

[0054] In addition to game content, the server system 114 also streams additional content, such as new league rosters, statistics, and preview access to early titles, which is optionally updated regularly (e.g., updated from time to time, daily, or hourly), to the client devices 102 and 104. In some embodiments, the additional content includes search results from an internet search or a database search.

[0055] In some embodiments, the server system 114 supports a real-time online community associated with the game application. Users (e.g., subscribers of the service) participate in live events, tournaments, or activities on the corresponding game API platform throughout the day. Examples of live events, tournaments, or activities include watching live game sessions played by other users, publishing achievements to a public domain (e.g., YouTube), and obtaining live tips and instructional videos. For example, in response to user actions, the game server 122 provides two or more real-time streams 130 and 132. While maintaining the first game stream (first real-time stream 130) on the online game session 116 of the first client device 102A of the game player, the server system 114 also broadcasts a second real-time review stream (second real-time stream 132) (e.g., YouTube stream) to one or more other client devices 104 (e.g., subscribers). The second real-time review stream allows the user to share his or her gaming experience with an audience. Optionally, the second real-time stream is a reproduction of the screen image of the player's first client device 102A. The server system 114 can obtain an audio stream of the player explaining the online game session 116, or a video stream of the player playing and explaining the online game session 116. The audio stream can optionally be played for the audience at the same time as the second real-time commentary stream is played for the audience. The video stream can optionally be played in an embedded window in the second real-time commentary stream.

[0056] Some embodiments provide on-the-go gaming, allowing a user to take his or her desired gaming experience to any location or client device. For example, a user can start an online gaming session 116 on a mobile device (first client device 102A) during his or her commute, and then seamlessly continue the online gaming session 116 on a laptop (second client device 102B) at his or her destination. Furthermore, in some embodiments, based on the different client device resources available to the user when the online gaming session 116 switches between different client devices 102, the server system 114 (specifically, the game server 122) can dynamically deploy different sets of hardware resources (e.g., GPU 140 and encoder 142) to optimize the user's gaming experience based on the different end-user's current device resources (e.g., client hardware capabilities and network bandwidth).

[0057] In server system 114, front-end server 134 and game server 122 can have corresponding user account system.In an example, the user account system for front-end server 134 is used to manage the subscription to specific game content and service, and the user account system (for example, YouTube or Google account) for game server 122 is used to manage game experience (for example, rendering game content to meet specific game standards) and many other purposes.In some embodiments, these two user account systems share clients and use data (for example, social, friends, presence (presence), authentication, account information, billing information).And, content front-end server 134 is provided in the service layer above the technical layer realized by game server 122.In some embodiments, game content server management is used to access the additional user account system of their content.Alternatively, the additional user account system for game content is integrated with the user account system of the front-end server 134 that is used to manage user subscription.

[0058] In some embodiments, the server system includes a state saving system 170 for handling real-time instances and delayed instances of the gaming application. Figure 7-10 Various implementations of the state preservation system 170 are described.

[0059] Figure 2 is a block diagram illustrating an example client device 102 of a gaming environment 100 according to some embodiments. Throughout this application, unless otherwise noted, references to client device 102 correspond to references to Figure 1One or more of the client devices 102A, 102B, and 104 described herein. Examples of client devices 102 include, but are not limited to, mobile phones, tablet computers, laptop computers, desktop computers, and wearable personal devices. In some embodiments, client device 102 is a dedicated game controller that includes game control inputs (e.g., one or more buttons, joysticks, touch screen elements, motion controls, pressure controls, visual controls, audio controls, and / or other tactile interface elements configured to control certain aspects of game play when activated). Client device 102 includes one or more processing units (CPUs) 202, one or more network interfaces 204, memory 206, and one or more communication buses 208 for interconnecting these components (sometimes referred to as chipsets). Client device 102 includes one or more input devices 210 to facilitate user input, such as a keyboard, mouse, voice command input unit or microphone, touch screen display, touch-sensitive input pad, gesture capture camera, or other input buttons or controls. In addition, some client devices 200 may use a microphone and voice recognition or a camera and gesture recognition to supplement or replace interfaces that require contact (e.g., keyboard and buttons). In some embodiments, the client device 102 includes one or more cameras, scanners, or photo sensor units for capturing images of, for example, a graphic series code printed on an electronic device. In some embodiments, the client device 102 includes one or more output devices 212 capable of presenting a user interface and displaying content, including one or more speakers and / or one or more visual displays. Optionally, the client device 102 includes a location detection device 214, such as a GPS (global positioning satellite) or other geolocation receiver, for determining the location of the client device 102. The client device 102 may also include a proximity detection device 215, such as an IR sensor, for determining the proximity of the media device 106 and / or other client devices 102. The client device 102 may also include one or more sensors 213 (e.g., accelerometers, gyroscopes, etc.) for sensing the motion, orientation, and other parameters of the client device 102, which may be used as input (e.g., for the input described above).

[0060] Memory 206 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally, includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. Memory 206 optionally includes one or more storage devices remotely located from one or more processing units 202. Memory 206, or alternatively, non-volatile memory within memory 206, includes non-transitory computer-readable storage media. In some embodiments, memory 206 or the non-transitory computer-readable storage media of memory 206 stores the following programs, modules, and data structures, or a subset or superset thereof:

[0061] □ Operating system 216, including processes for handling various basic system services and for performing hardware-related tasks;

[0062] a network communication module 218 for connecting the client device 102 to other devices (e.g., server system 114, media device 106, and other client devices 102) via one or more network interfaces 204 (wired or wireless) and one or more networks 110 and / or 112, such as the Internet, other wide area networks, local area networks, metropolitan area networks, and the like;

[0063] A user interface module 220 for implementing the presentation of information (e.g., a graphical user interface for presenting applications, widgets, websites and their pages, and / or games, audio and / or video content, text, etc.) at the client device 102 via one or more output devices 212 (e.g., a display, speakers, etc.);

[0064] □ Input processing module 222 for detecting one or more user inputs or interactions from one of the one or more input devices 210 and interpreting the detected inputs or interactions;

[0065] □ Input event reporting module 223, used to report input identification and / or timestamp information to the server system 114 for use in delay calculation;

[0066] □ Web browser module 225, for navigating, requesting (e.g., via HTTP), and displaying websites and their pages, including a web interface for joining online game sessions 116;

[0067] □ A media device application 226 for interacting with the media device 106 , including logging into a user account associated with the media device 106 , controlling the media device 106 while associated with the user account, and editing and reviewing settings and data associated with the media device 106 ;

[0068] □ Game application 228, which is used to provide games on the client device 102, including facilitating the corresponding game play and facilitating invitations of other players;

[0069] □ Game controller module 230, used to provide a game play input interface to the game application 228;

[0070] □ Data download module 231 for downloading data (e.g., game controller configuration 456 ( Figure 4 ), gaming applications 228 and other applications, updates to modules and applications and data in memory 206); and

[0071] The client device data 232 stores at least data associated with the game application 228 and other applications / modules, including:

[0072] o Client device settings 234, which is used to store information associated with the client device 102 itself, including common device settings (e.g., service tier, device model, storage capacity, processing power, communication capabilities, etc.);

[0073] o Media device settings 236 for storing information associated with a user account of the media device application 226, including one or more of account access information and device settings (e.g., service tier, device model, storage capacity, processing power, communication capabilities, etc.);

[0074] Game application settings 238 for storing information associated with a user account for the game application 228, including one or more of account access information, in-game user preferences, game play history data, and information about other players;

[0075] o Game controller configuration 240, for storing configurations of the game controller module 230 for the game application 228 (e.g., from Figure 4 information associated with the game controller configuration 456 (the configuration received); and

[0076] o Location / proximity data 242 , including information associated with the presence, proximity, or location of any of the client device 102 and the media device 106 .

[0077] In some implementations, the game controller module 230 is part (e.g., a submodule) of the media device application 226 or another application in the memory 206. In some implementations, the game controller module 230 is part of the operating system 216. In some implementations, the game controller module 230 is a distinct module or application.

[0078] In some embodiments of the client device 102, the media device application 226 (and corresponding media device settings 236) and the game application 228 (and corresponding game application settings 238) are optional. Depending on the specific game that the client device 102 is invited to join, the media device application 226 and the game application 228 may not be required for play. If any of these applications is required to play the game (e.g., the game uses the game controller module 230 in the media device application 226), and the application is not in the memory 206, the client device 102 can be prompted to download the application.

[0079] Each in the element identified above can be stored in one or more aforementioned storage devices, and corresponds to the instruction set for performing the above functions.The module or program (that is, instruction set) identified above do not need to be implemented as independent software program, process, module or data structure, and therefore the various subsets of these modules can be combined or otherwise rearranged in various embodiments.In some embodiments, memory 206 is stored in the module identified above and the subset of data structure alternatively.In addition, memory 206 stores additional modules and data structure not described above alternatively.

[0080] Figure 3 is a block diagram illustrating an example media device 106 of a gaming environment 100 according to some embodiments. The media device 106 typically includes one or more processing units (CPUs) 302, one or more network interfaces 304, memory 306, and one or more communication buses 308 for interconnecting these components (sometimes referred to as a chipset). Optionally, the media device 106 includes a proximity / position detection unit 310, such as an IR sensor, for determining the proximity of the client device 102.

[0081] Memory 306 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. Memory 306 optionally includes one or more storage devices located remotely from one or more processing units 302. Memory 306, or alternatively, non-volatile memory within memory 306, includes non-transitory computer-readable storage media. In some embodiments, memory 306 or the non-transitory computer-readable storage media of memory 306 stores the following programs, modules, and data structures, or a subset or superset thereof:

[0082] □ Operating system 316, including processes for handling various basic system services and for performing hardware-related tasks;

[0083] a network communication module 318 for connecting the media device 106 to other computers or systems (e.g., server system 114 and client device 102) via one or more network interfaces 304 (wired or wireless) and one or more networks 110 and / or 112 (such as the Internet, other wide area networks, local area networks, metropolitan area networks, cable television systems, satellite television systems, IPTV systems, etc.);

[0084] a content decoding module 320 for decoding content signals received from one or more content sources (e.g., server system 114 output from online gaming session 116) and outputting the content in the decoded signals to an output device 108 coupled to media device 106;

[0085] □ A proximity / position determination module 322 for determining the proximity of the client device 102 based on proximity-related information detected by the proximity detection unit 310 or provided by the server system 114;

[0086] □ Media display module 324, used to control media display; and

[0087] □ Display event reporting module 325, for reporting display event identification and / or timestamp information to the server system 114 for use in delay calculation;

[0088] □ Delay calculation module 326, for calculating delay values ​​based on delay data 334 reported by other components in the gaming environment;

[0089] □ Media device data 328, used to store at least the following data:

[0090] o Media device settings 330, for storing information associated with a user account of a media device application, including one or more of account access information and device setting information (e.g., service tier, device model, storage capabilities, processing capabilities, communication capabilities, etc.);

[0091] o Location / proximity data 332 , including information associated with the presence, proximity, or location of any of the client device 102 and the media device 106 ; and

[0092] o Delay data 334, which includes information necessary for the delay calculation module 326 to calculate delay values ​​(eg, timestamps).

[0093] Each element identified above can be stored in one or more aforementioned memory devices, and corresponds to the instruction set for performing the above functions. The modules or programs (that is, instruction sets) identified above do not need to be implemented as independent software programs, processes, modules or data structures, and therefore the various subsets of these modules can be combined or otherwise rearranged in various embodiments. In some embodiments, memory 306 optionally stores the modules identified above and the subset of data structure. In addition, memory 306 optionally stores additional modules and data structures not described above.

[0094] Figure 4 is a block diagram illustrating an example server in a server system 114 of a gaming environment 100 according to some embodiments. The server system 114 typically includes one or more processing units (e.g., a CPU 138, a GPU 140, and an encoder 142), one or more network interfaces 404, memory 146, and one or more communication buses 408 for interconnecting these components (sometimes referred to as a chipset). The server system 114 may optionally include one or more input devices 410 to facilitate user input, such as a keyboard, mouse, voice command input unit or microphone, touchscreen display, touch-sensitive input pad, gesture capture camera, or other input buttons or controls. Furthermore, the server system 114 may use a microphone and voice recognition or a camera and gesture recognition to supplement or replace a keyboard. In some embodiments, the server system 114 may optionally include one or more cameras, scanners, or photo sensor units for capturing images of, for example, a graphical serial code printed on an electronic device. The server system 114 may also include one or more output devices 412 capable of presenting a user interface and displaying content, including one or more speakers and / or one or more visual displays.

[0095] Memory 146 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. Memory 146 optionally includes one or more storage devices located remotely from one or more processing units. Memory 146, or alternatively, non-volatile memory within memory 146, includes non-transitory computer-readable storage media. In some embodiments, memory 146 or the non-transitory computer-readable storage media of memory 146 stores the following programs, modules, and data structures, or a subset or superset thereof:

[0096] □ Operating system 416, including processes for handling various basic system services and for performing hardware-related tasks;

[0097] □ A network communications module 418 for connecting the server system 114 to other devices (e.g., the various servers in the server system 114 , the client devices 102 , and the media devices 106 ) via one or more network interfaces 404 (wired or wired) and one or more networks 110 and / or 112 (such as the Internet, other wide area networks, local area networks, metropolitan area networks, etc.);

[0098] □ User interface module 420 for enabling presentation of information at the client device 102 (e.g., a graphical user interface for presenting applications, widgets, websites and their web pages and / or games, audio and / or video content, text, etc.);

[0099] □ Media device module 422 (optional), executed to provide server-side functionality for device provisioning, device control, and user account management associated with the media devices 106;

[0100] □ A proximity / position determination module 424 for determining proximity of the client device 102 to the media device 106 based on location information of either the client device 102 or the media device 106;

[0101] A game server module 426 for providing server-side functionality associated with a game (e.g., a game application 228), including, but not limited to, setting up a game session, storing session state data and other game-related data, processing game play input from a client device 102, and rendering game play output in response to the game play input;

[0102] a media streaming server module 438 for hosting a media streaming site, receiving concurrent auxiliary or supplemental media streams associated with an online gaming session, and providing the concurrent media streams to a client device 104 for concurrent display with an online gaming session being executed on a gaming application 228 on the same client device 104 or a different client device 102;

[0103] A front-end server module 440 for managing user accounts associated with client devices 102, such as subscriptions to one or more online interactive games through the user accounts, providing services to the subscribers by forwarding subscriber requests to the game server module 426, and monitoring game play activity and subscriber-related requests;

[0104] □ Media content server module 442, for providing access to gaming content hosted by one or more third-party content providers;

[0105] □ Device / network assessment module 444, for assessing the device and network capabilities of the client device 102, including but not limited to assessing the network bandwidth of the connection to the client device 102 and assessing whether the client device 102 has the modules or applications required to play the game;

[0106] □ Data transfer module 446 for providing data (e.g., game controller configuration 456, software updates, etc.) to client device 102; and

[0107] □Server system data 448, including:

[0108] o Client device settings 450, for storing information associated with the client device 102, including common device settings (e.g., service tier, device model, storage capacity, processing power, communication capabilities, etc.);

[0109] o Media device settings 452 (optional), for storing information associated with a user account of the media device application 422, including one or more of account access information and device settings information (e.g., service tier, device model, storage capacity, processing power, communication capabilities, etc.);

[0110] o Location / proximity data 454 , including information associated with the presence, proximity, or location of any of the client device 102 and the media device 106 ;

[0111] o Game Controller Configuration 456, which stores controller configurations for various games; and

[0112] o User information 458, which is used to store and manage one or more gaming applications hosted on the server system 114 (e.g., Figure 2 Information associated with a user account for each of the gaming applications 228 in the game, including, for example, user account information (e.g., identification and password), membership type, preferences, and activity history; and

[0113] A game session event log 460 is used to store event data associated with a game session (e.g., game state data, input events, display events, and other game-related data), including, for example, data 460-1 of a first game session and data 460-2 of a second game session, wherein the session data for each game session includes, but is not limited to, frame rate, rendering specifications, normal latency requirements, GPU allocation information, encoder allocation information, an identifier of the related session, the latest state information associated with the corresponding game session, a log of input events, and a log of display events;

[0114] ○Response time setting 462, used to store expected delay values ​​for various user command types;

[0115] o Resource repository 464 for storing virtual machine resource profiles and container images; and

[0116] o Resource Settings 466, which stores the configuration of available resources based on user tolerance levels; and

[0117] □ Data buffer 144, used to temporarily store gaming multimedia content generated by GPU 140 in association with one or more output media streams.

[0118] In some embodiments, the game server module 426 includes the following programs, modules, or subsets or supersets thereof:

[0119] an intent determination module 428 for comparing user input transmission times (e.g., between the client device 102 and the server system 114) with display transmission times (e.g., between the media device 106 and the server system 114) and determining the user's intent behind a particular input by matching the input event with the corresponding trigger frame;

[0120] □ Latency adjustment module 430 for determining the number of intermediate frames that GPU 140 inserts between (i) the current frame being processed when user input is received and (ii) a response frame showing the result of the received input:

[0121] □ A resource allocation module 432 (optionally referred to herein as a "session coordinator") for receiving a session request from an endpoint (e.g., controller 102) and determining which resources to allocate to the session; and

[0122] □ Resource adjustment module 434, used to determine the delay tolerance of a specific user.

[0123] In some embodiments, the memory 146 also includes a data buffer 144 configured to couple the encoder 142 to the GPU 140. Specifically, the data buffer 144 temporarily stores the gaming multimedia content generated by the GPU 140 in association with one or more output media streams, so that the encoder 142 can retrieve the gaming multimedia content from the data buffer 144 and encode the retrieved content into one or more media streams, for example, for normalization, speed, or compression.

[0124] Each in the element identified above can be stored in one or more aforementioned memory devices, and corresponds to the instruction set for performing the above functions.The module or program (that is, instruction set) identified above do not need to be implemented as independent software program, process, module or data structure, and therefore the various subsets of these modules can be combined or otherwise rearranged in various embodiments.In some embodiments, memory 146 optionally stores the module identified above and the subset of data structure.In addition, memory 146 optionally stores additional modules and data structure not described above.

[0125] Various embodiments of the above-mentioned cloud-based gaming platforms provide many benefits (e.g., portability, scalability, efficiency, ease of access and control, etc.). However, the cloud-based nature of these gaming platforms brings various challenges, such as the variability of network and processing resources, which, if not properly considered, may have a negative impact on the gaming experience. Due to the variable delays introduced in the network 110 / 112 between the player device (client device 102) and the server system 114, such challenges may potentially create an uneven gaming experience. The following disclosure describes various embodiments of detecting and compensating for different types of delays that may exist in a real-time interactive cloud-based gaming environment. By compensating for these delays, the embodiments described herein provide a smooth, unified gaming experience for each player, regardless of the available network and processing resources.

[0126] Figure 5A Depicting an example gaming environment 500, from which several sources of latency will be described. Gaming environment 500 is a graphical representation of gaming environment 100 ( Figure 1 ), with corresponding components labeled similarly. The gaming environment 500 includes a client device 102 (also referred to herein as a "game controller" or "controller") that a player (or "user") uses to control the game, for example, by activating or manipulating an input device 210 ( Figure 2 ) to control various aspects of the game (or "game play"). The gaming environment 500 also includes a media device 106 (e.g., a set-top box) and an output device 108 (e.g., a television or other output display). The controller 102 and the media device 106 are communicatively coupled to a local network 110 (depicted as a wireless router in this example) via local communication links 502 and 504, respectively (e.g., via WiFi). The local network 110 is communicatively coupled to a server system 114 via a communication network 112 (e.g., the Internet) via a communication link 506. The server system 114 includes a game server 122 ( Figure 1 ).

[0127] Although the gaming environment 500 depicted in the figure includes only a single local network 110 with a single controller 102, some embodiments of the gaming environment 500 may include multiple local networks 110, some of which include more than one controller 102 (e.g., for multi-player games sharing the same gaming session, as described above with reference to FIG). Figures 1 to 4 described).

[0128] Several elements present in gaming environment 500 can introduce latency that is perceptible (e.g., affecting at least one frame) and varies over time. For example, local network 110 (e.g., WiFi) can introduce varying amounts of latency in communication links 502 and 504. If there is no contention on the channel, the average latency can be very low (e.g., <1 ms). However, in busy environments such as apartment buildings with overlapping WiFi networks or gaming environments with multiple wireless client devices, average amounts of latency in the 10-50 ms range are more common, with outliers exceeding 200+ ms.

[0129] In addition, the communication network 112 (e.g., the Internet) can introduce latency into the communication link 506. For most users, this latency variability may not be as high as WiFi; however, during peak gaming hours (late evening), media sharing (e.g., on a cable modem), and network saturation, packets may be delayed or lost. The average latency will depend on the distance from the local network 110 to the edge server of the server system 114, with an example latency in the range of 20-30 ms.

[0130] Due to asymmetries in network demand and link capacity, the delay introduced by the aforementioned network may vary based on the direction of traffic flow (e.g., from the controller 102 to the server 122 versus from the server 122 to the media device 106). Thus, the delay on the link 506 from the router to the server may not match the delay from the server back to the router, etc.

[0131] Additionally, game server 122 may introduce latency. There is a delay from the time an input event arrives at GPU 140 to the time a frame is output from encoder 142. However, in some implementations, this latency is fully traceable and, therefore, known to game server 122.

[0132] Finally, there is a delay between when a frame arrives at the output device 108 (e.g., a television) and when the frame is displayed. This may depend on the nature of the processing in the output device, including the display mode (e.g., gaming mode vs. non-gaming mode). For example, a television may have a display lag as low as 15-30ms, or the display lag may be as high as 50-60ms. A poor quality television may have a display lag of over 120ms.

[0133] The different types of delays mentioned above may have a significant impact on the gaming experience. Figure 5B and 5C Two example game play experiences are shown that include identical user inputs but result in completely different outputs due to different levels of latency. However, before describing these examples in detail, it is first necessary to describe the example game play process.

[0134] Figure 6 6 is a flow chart of a game play process 600 according to some embodiments. The process can be performed at an electronic server (e.g., server system 114, or more specifically, game server 122) having one or more processors (e.g., CPU 138 and / or GPU 140) and memory storing one or more programs for execution by the one or more processors (e.g., memory 146); a media device (e.g., media device 106) having one or more processors (e.g., CPU 302) and memory storing one or more programs for execution by the one or more processors (e.g., memory 306); and / or a user device (e.g., controller 102) having one or more processors (e.g., CPU 202) and memory storing one or more programs for execution by the one or more processors (e.g., memory 206). In some embodiments, the server, media device, and user device include one or more programs and memory storing the one or more programs for execution by the one or more respective processors, and the one or more programs include instructions for performing process 600. In some embodiments, corresponding non-transitory computer-readable storage media stores one or more corresponding programs that include instructions that, when executed by the electronic server, media device, and user device, cause the electronic server, media device, and user device having one or more corresponding processors to perform process 600.

[0135] The user of the controller 102 (also referred to herein as a "player") uses the controller 102 to affect events in the game, which are depicted by video frames (e.g., 510) displayed on the output device 108 (see Figure 5A ). When a player decides to affect game play (e.g., by moving a virtual player, shooting a hockey puck, etc.), the player activates (602) or otherwise manipulates an input on controller 102 (e.g., pressing a button). Activation or manipulation of an input on controller 102 is sometimes referred to as an "input event" or "command." The input event is communicated (604) to server system 114 (e.g., to game session event log 460 associated with the game session) via communication links 502 and 506 (over networks 110 and 112).

[0136] Upon receiving an input event (606), the server system 114 (e.g., the intent determination module 428 of the game server 122) determines (608) which frame to display on the output device 108 when the user activates the input associated with the received input event. The frame displayed to the user when the user activates the input is referred to herein as a "trigger frame" because it triggers the user to respond by activating the input. For example, in a hockey game, if the frame shows an open shot, this would trigger the player to respond by activating an input control mapped to a "shoot puck" function. A trigger frame is a frame that shows an open shot (e.g., Figure 5B ), and the input event is the user activating the “Puck Shot” control on controller 102 in response to having seen the trigger frame.

[0137] After determining the trigger frame, the game server 122 (e.g., intent determination module 428) determines (610) the state of the game at the time the trigger frame is displayed to the user (referred to herein as the "trigger state"). In some implementations, the intent determination module 428 determines the state of the game by consulting the game session event log 460 ( Figure 4 ) to determine the trigger state. In some embodiments, the game session event log 460 includes a log of game states indexed by frame fingerprints, frame IDs, and / or game time data (e.g., timestamps or clock data). In some embodiments, the intent determination module 428 determines the trigger state by determining a game time index associated with a trigger frame and consulting the game session event log 460 to determine the game state that existed at the game time index associated with the trigger frame. Depending on how much time elapses between displaying the trigger frame on the output device 108 and receiving the input event on the game server 122, the trigger state may be in the past relative to the current state being processed on the game server 122.

[0138] Returning to the previous example, if a trigger frame (showing an open shot on goal) is associated with game time index T1, the game state at time index T1 includes the positions of the virtual shooter, virtual defenders, virtual puck, virtual goal, and each of these objects. Based on the game state at time index T1, or more specifically, based on the positions of each of the aforementioned virtual objects at time index T1, there is an unobstructed path between the puck and the goal. In other words, the one or more algorithms controlling the game play rules will allow the virtual puck to travel from the virtual player who shot the puck to the virtual goal at a certain moment (time index T1) when the trigger frame is displayed without being stopped by any other virtual players between the shooter and the goal. However, in some cases, when an input event (e.g., "puck shot") arrives at the server, the server is currently processing a subsequent state of the game play, T2, which may include a further state of the game play in which the virtual puck no longer has an unobstructed path to the goal. In these cases, if the server correctly determines the trigger state to be T1, then the trigger state is a past state relative to the state T2 currently being processed by the server.

[0139] Having determined the trigger state, the game server 122 (e.g., GPU 140) processes (612) a subsequent game state (sometimes referred to herein as a "game play output") based on (i) the input event (e.g., "puck shot") and (ii) the trigger state (e.g., including an unobstructed path from the puck to the goal). In some embodiments, processing the game play output includes inputting the input event into an algorithm or game engine that determines the game play output based on the input event and the corresponding game state. For example, the game engine can determine the next game state based on the state / position of each player and the puck relative to the goal during the current game state and any input commands received for the virtual player during the current game state (e.g., "move," "shoot," or "block"). In some embodiments, processing the subsequent game state (game play output) based on the input event and the trigger state includes processing the input event as if the input event were available to the server when processing a game state proximate to the trigger state (e.g., the next state after the trigger state, or the state immediately after the trigger state).

[0140] While processing the game play output, the game server 122 renders (614) a frame or series of frames depicting the processed game play output. A frame (or the first frame in a series of frames) depicting the game play output is referred to herein as a "response frame." For example, if the input event and trigger state result in game play output that includes movement of a particular virtual player, the response frame is a frame that depicts the particular virtual player in a modified spatial position relative to other objects in the frame that is consistent with the direction specified by the user input. Alternatively, if the input event and trigger state result in game play output of a particular virtual player shooting a hockey puck, the response frame is the first frame in a series of frames that depicts the particular virtual player taking a hockey shot (e.g., Figure 5B In some implementations, presenting the response frame includes introducing a new virtual object, modifying an existing virtual object, or modifying any other aspect of the gameplay in the response frame based on the processed gameplay output and including the new virtual object, the modified existing virtual object, or any other aspect of the modified gameplay.

[0141] Server system 114 (e.g., using encoder 142) encodes the response frame and transmits 616 the encoded response frame to media device 106. Upon receiving the encoded response frame from server system 114, media device 106 (e.g., using content decoding module 320) decodes the response frame and causes display 620 of the decoded response frame to a user (e.g., using output device 108).

[0142] Return to Figure 5B and 5C , depicts two sequences of video frames (510 and 520) showing the same input event (a shot on a hockey puck) but with different response frames (a successful shot vs. a blocked shot) due to different amounts of latency present in the gaming environment 500. These sequences are examples of a game play process 600 applied to the gaming environment 500.

[0143] Figure 5B A first scene 550 is depicted, comprising a series of video frames 510 showing three virtual players (A, B, and C) playing a hockey game and a table 512 of game states T1-T3 (e.g., stored in Figure 4460). Player A is controlled by the user of controller 102, and players B and C are controlled by other users of other controllers, by computer-controlled algorithms, or a combination thereof. At state T1, player A has an unobstructed shot at the goal (denoted as "Unobstructed" in table 512); therefore, the game server transmits frame 510-1 representing this state to the user's display (output device 108). When the user controlling player A views frame 510-1 on the display (output device 108), the user sees that player A has an unobstructed shot at the goal and therefore decides to instruct player A to shoot the puck. In other words, frame 510-1 triggers the user to input a "shoot" command. The "shoot" command is sent to the game server 122 as an input event. When the game server 122 receives the "shoot" input (denoted as "In" in table 512), the game server is currently processing state T2, at which point player A no longer has an unobstructed shot (denoted as "No Shot" in table 512). However, the game server 122 correctly determines that the trigger frame (denoted as "T" in table 512) is frame 510-1. According to the game state when frame 510-1 was displayed (trigger state T1), player A still has an unobstructed shot at the goal; therefore, the game server 122 processes the subsequent state T3 based on the "shoot" command and the T1 state (unobstructed shot). According to the game engine, if a shot is taken while the player has an unobstructed shot, the subsequent state includes a successful shot sequence, and the sequence is processed in state T3 (denoted as "score" in table 512). Thus, the game server renders a response frame (frame 510-2) depicting player A shooting the puck past player C and transmits the response frame to the user. From the user's perspective, the response frame describes the action that the user expected when entering the event. Thus, by correctly determining the trigger state corresponding to the user's input, the game server processes the game play according to the user's intent.

[0144] Figure 5C A second scene 552 is depicted, which includes a sequence of video frames 520 showing the same game and players as scene 550, and a table 522 of game states T1-T3 (e.g., stored in Figure 4522). Similar to the previous scenario, at state T1, player A has an unobstructed shot at the goal (denoted as "Unobstructed" in table 522); therefore, the game server transmits frame 520-1 representing this state to the user's display (output device 108). When the user views frame 520-1 on the screen (output device 108), the user sees that player A has an unobstructed shot at the goal and therefore decides to command player A to shoot the puck. The "Shoot" command is sent to the game server 122 as an input event. As in the previous scenario, when the game server 122 receives the "Shoot" input (denoted as "In" in table 522), the game server is currently processing state T2, where player A no longer has an unobstructed shot (denoted as "No Shot" in table 522). However, unlike the previous scenario, the game server 122 does not correctly determine the trigger frame (denoted as "T" in table 522). Instead, the game server considers the trigger frame to be the last frame rendered based on the current state T2, which in this example is frame 520-2. Alternatively, the game server may not even determine a trigger frame, but instead process the game play output based on the current state T2 (no shot). In either case, the game server processes the subsequent state T3 based on the "shoot" command and the T2 state (no shot). According to the game engine, if the player shoots without an unobstructed shot, the subsequent state includes a blocked shot sequence, and this sequence is processed in state T3 (referred to as "blocked" in table 522). Thus, the game server renders a response frame (frame 520-3) depicting Player A attempting a shot on the puck but being blocked by Player C, and transmits this response frame to the user. From the user's perspective, the response frame describes an action that the user did not expect when inputting the event. Specifically, the user expected Player A to shoot at the same time that Player C was unblocked; instead, Player A did not shoot as quickly as the user expected, and the shot was blocked. Thus, by failing to correctly determine the trigger state corresponding to the user input, the game server may process game play events contrary to the user's intent, which may cause the user (and many other users) to lose interest in playing the game and / or using the game environment 500.

[0145] In each of the two scenarios described above, the input events occur simultaneously; however, depending on how long it takes for the input events to reach the game server, the response frames describe two very different outcomes. This is because if the server receives the user's input while processing a game state (e.g., T2) that is later than the game state that triggered the user's input (e.g., T1), the server may incorrectly process the game output based on incorrect information about the timing of the user's input. Because it is critical for the game platform to avoid such inconsistencies, it is important for the game platform to detect and compensate for the various delays introduced into the game environment that cause these delays. By detecting the various delays, the game platform can more accurately correlate input events with the actual triggering state (as in scenario 550). By making these correlations, the game platform can accurately correlate input events with the actual triggering state (e.g., T2). Figure 1 Each input event is processed in a consistent manner to reduce the impact of uncontrollable and / or undetectable delays. Thus, the various embodiments described herein are improvements to gaming platforms that do not attempt to determine or incorrectly determine the exact trigger state corresponding to user input.

[0146] In some scenarios, depending on how much time has passed between the triggering state and the current state being processed by the game server, a particular game play output may contradict what has already been displayed to one or more users. For example, Figure 5C , frame 520-3 depicts the blocked shot. However, if the game server determines during state T3 that the triggering state is T1, in some embodiments, the game server attempts to retroactively reconcile the user's intent with the current state of the game. In other words, the user's intent is for the puck to be shot at the same time that Player A takes an unobstructed shot, and the current state of the game (T3) shows Player C between Player A and the goal. To reconcile the user's intent (the puck moving toward the goal) with the current state (Player C in the puck's path), the game server may render a series of response frames in which the puck moves toward the goal despite Player C being in the way (e.g., Figure 5B Response frames may appear inconsistent with the current game state; however, they are consistent with the user's intent during the past (triggering) game state. Figure 1Game developers can plan for these unexpected situations in advance, for example, by designing animations that reconcile inconsistent game states. Example reconciling animations include immediately displacing a virtual character or object to a desired location (even if this may appear to violate the physics of the game), or advancing the game state in a desired manner while not showing the correct animation (e.g., updating the score without showing the puck reaching the goal, or classifying a monster as injured even though the monster appears to have moved out of the way before being shot). In some embodiments, reconciling the current game state with the game state that the user expects when the user interacts (the desired game state) includes modifying a frame depicting the current game state to create a subsequent frame depicting the desired game state.

[0147] Shadow tracking environment

[0148] The following embodiments are intended to provide a processing environment for implementing a delayed version of a primary instance of a real-time interactive application. The delayed version (also referred to herein as a "shadow process") operates with the same inputs as the primary instance, but with a configurable delay relative to the primary instance. If the primary instance enters an undesirable state, the delayed instance is paused before the undesirable state occurs, enabling inspection, stepping, and other diagnostic functions.

[0149] In some embodiments, each instance of an application is an executing version of the application, each version being executed by a separate processing capability / processor. For example, in some embodiments, each instance is executed by: a microprocessor (CPU), one or more cores of a multi-core CPU, a graphics processing unit (GPU), and / or one or more cores of a multi-core GPU. In some embodiments, each instance of an application is a simulation running on a corresponding processing capability (e.g., a GPU). By running parallel simulations in lockstep (where the second simulation receives delayed inputs or performs delayed operations), one instance is an early detector that warns the system that an error or fault (e.g., an anomaly, failure, or unexpected state) is about to occur. This allows the system to stop and examine the state of the program before a fault occurs and reconstruct one or more pre-fault states; by the time a fault has occurred, it may have destroyed valuable information that was available before the fault occurred.

[0150] With only a single instance of an application, the processing system can capture that instance when it fails; however, by then, the system may not understand why the failure occurred. In other words, with only a single instance, the system cannot look back at the state before the failure; it may not be able to reconstruct the pre-failure state and / or recover all the information necessary to determine the cause of the failure.

[0151] On the other hand, when the secondary instance of the delay is running in parallel, the system can stop the simulation or stop the operation of the actual processing unit at the point in time when the simulation or processing unit is about to fail. In other words, because the main instance is running first and is using the same input stream to execute or simulate the same application, the system already knows the future state of the delayed instance. With this knowledge, the system can pause execution before the failure occurs and carefully step forward to track the information of interest (such as register data or variables). The system can dump one or more of these pre-error states and access information that would not be accessible if the failure occurred, thereby enabling the system to reconstruct the exact state of execution or simulation before the failure. The reconstructed pre-failure state enables the system or developer to determine why a failure will occur in the second instance in the future.

[0152] Some of the embodiments described herein describe online gaming applications (e.g., Figure 1-6 The gaming application described in this paper is used as an example processing environment. However, knowing in advance that a processing system is about to fail is useful in many other types of processing environments, of which an online gaming server is just one example. For example, the various embodiments described herein can be implemented in any processing environment involving a GPU. While CPUs are relatively transparent from a debugging perspective, GPUs are relatively opaque because reverse engineering the events in the GPU immediately before a failure is more difficult. This makes determining the cause of a failure difficult in some cases and impossible in others. For example, many aspects of a GPU are not easily inspected. Hidden states may exist, and the lack of built-in debugging capabilities makes it difficult to reverse engineer what happened at the time of the failure. More specifically, because GPUs have many different types of registers and pipelines involved in parallel operations, there are many more aspects to consider than with some CPUs, which are relatively easy to inspect. Even if it were possible to inspect every register, every variable, every aspect of every pipeline, etc., continuously dumping all of this information in a runtime environment (e.g., while an application or program is running) may be impractical due to the large amount of information associated with each processing state. However, if a specific event is known in advance, it is more reasonable to consider all relevant information at that point. In other words, developers can recover relevant information in a more feasible amount. Depending on how many cycles of data existed before the failure, developers will need more data to reconstruct the conditions that led to the failure and thus obtain the information needed to repair the application or program and avoid future failures.

[0153] In addition to being relevant to processing systems that include GPUs, the embodiments described in this specification are relevant to any processing system involving a processor having characteristics that make it difficult to debug or otherwise determine the cause of a failure condition.

[0154] Some of the various embodiments described herein are implemented on a server, such as the game server 122 (referenced above). Figure 1 Description) or dedicated state preservation system 170 (hereinafter referred to Figure 7-10 Description). In these embodiments, when there is idle processing power on the server, auxiliary processes are executed for specific applications. For example, if a game server uses only a subset of the available GPUs to host an online game session, one or more idle GPUs can be used to execute a delayed auxiliary game session. In this way, if a particular session fails, the auxiliary session saves one or more pre-failure states. This provides developers with the opportunity to capture failure information in a live production environment with consumer interaction, which provides a more realistic use case than sessions executed individually in a lab or test environment. In this way, developers can access real-time failure information in a production environment without having to replicate test situations in a lab environment. In addition, for embodiments that only use idle processing power, developers can access this information without negatively impacting the processing power available for new sessions. In some embodiments, the pre-failure state preservation processing system described herein runs continuously for each session (e.g., for each session described above).

[0155] Regardless of the processing environment, the various embodiments described herein all have the technical effect of preserving the processing state before the error, thereby allowing the cause of real-time fault events in the evaluation program, application and / or virtualization simulation to be evaluated. The creation of a slightly delayed instance of the same program with the same input sequence and timing provides an opportunity to stop the program before a known fault occurs (for example, because it has already occurred in the main instance). Having the ability to analyze a program before it fails is an improvement to general debuggers, which can only look back after the failure and cannot always access the information necessary to fully reconstruct the pre-fault conditions that caused the fault to begin. In addition, the embodiments described herein are equivalent to improvements to debuggers that use slightly different versions of the program (for example, a production version versus a laboratory version, or a real-time version versus a diagnostic test version), because the immediate implementation allows real-time analysis of consumer-oriented programs in a production environment, thereby facilitating the actual execution of the evaluation program in the field. Furthermore, the embodiments described herein represent an improvement to debugging programs that use different input data sets (e.g., consumer input versus lab input) because the presently described implementations allow for real-time analysis of consumer-oriented programs manipulated by actual input from real consumer users in the field, thereby facilitating more realistic operation of the program for evaluation in the context of consumer use. In the example of online gaming, the presently described implementations facilitate saving pre-bug game states for both the production version of the game and for actual game input provided by game users during real-time game play and under actual gaming conditions.

[0156] Additionally, having two instances of the same program run in lockstep (one of which is delayed) is an improvement over techniques that require running the same program multiple times, because even if it were possible to determine the internal processing states of the GPU before and after the failure, those internal processing states may change each time the program is run, thereby invalidating the usefulness of information describing those internal states for subsequent executions of the program.

[0157] As discussed above, the various embodiments described herein allow for forward debugging. Compared to post-mortem debugging, where a program or developer can only access a subset of the data of interest (e.g., data necessary to reconstruct the conditions that led to the fault), forward debugging provides access to more data that has not yet been corrupted by the fault before the fault occurs. In other words, the debugger or developer knows that the data that was in the registers at a particular moment will cause the fault. With this knowledge and access to the data, the developer can apply forward debugging techniques (e.g., evaluating each successive processing state up to the fault) in order to more accurately determine the cause of the fault. More specifically, the developer can more easily determine the exact combination of input sequence and processing state that caused the fault, and with this knowledge, the program can be updated so that in the future, the same input sequence and processing state will not result in the same error.

[0158] Pre-error state saving system

[0159] Figure 7 1 is an example pre-error processing state saving system 170 (also referred to as "processing system" 170) according to some embodiments. The processing system 170 includes a processing unit (e.g., CPU 705) for controlling the overall processing of the system 170, and a memory 708 (see Figure 8 7. The processing system 170 further includes two or more processing units (e.g., GPUs) 721 and 722 for executing a primary instance and a secondary instance of a program, respectively. The processing system 170 includes a delay module 710 for controlling the timing of input 172 relative to the primary and secondary instances of the program. The delay module 710 includes at least one buffer 718 for buffering one stream relative to another. The processing system further includes an error detection module 730 for detecting when the primary GPU 721 enters an undesirable state (e.g., an abnormal or failed state caused by an error or fault). The processing system further includes a check module 740 for facilitating evaluation of a saved pre-error state of the secondary GPU 722.

[0160] In some embodiments, the master GPU 721 and the input 172 correspond to the above reference Figure 1-6140 (in the game server 122) and game input. As mentioned above, it is important to note that these game embodiments are merely examples of interactive programs executed in real time on a processing unit, and the embodiments described herein are equally applicable to other non-game examples. However, because some of the concepts described herein may be more easily understood by reference to example applications, the above-described example game environment will be used as the basis for the example applications, although these concepts are equally applicable to non-game applications.

[0161] In the context of a gaming application, input 172 is supplied by a user of a game controller, generating an input stream (also referred to as input events or commands) as the user plays the game. The game engine (e.g., processing unit 721) receives one or more inputs 172 and processes and outputs a game state based on (i) the one or more inputs 172 and (ii) the current game state, as described above with respect to Figures 5A-5C and Figure 6 For non-gaming applications, the processing unit 721 receives one or more inputs 172 and generates a subsequent processing state based on (i) the one or more inputs 172 and (ii) the current processing state.

[0162] Applying the above example to Figure 7 In the example of processing system 170, processing system 170 receives a stream of input 172. Delay module 710 separates the stream into a primary stream 711 and a secondary stream 712. The only difference between each stream is the timing at which the corresponding GPUs 721 and 722 receive each input in the stream. For example, if delay module 710 receives a first input and a second input with a difference of 500 ms, primary GPU 721 receives the first and second inputs 500 ms apart, and secondary GPU 722 also receives those same inputs 500 ms apart. However, due to buffer 718, secondary stream 712 is delayed overall relative to primary stream 711. Thus, in the aforementioned example, if buffer 718 is configured to delay secondary stream 712 by 1 second, GPU 721 receives the first and second inputs at 0 and 0.5 seconds (corresponding to the 500 ms difference between the inputs), and GPU 722 receives the first and second inputs at 1.0 and 1.5 seconds (corresponding to the 500 ms difference between the inputs and the 1 second delay added by the buffer).

[0163] In some embodiments, the delay module 710 only buffers the auxiliary stream 712 (e.g. Figure 7). Alternatively, the delay module 710 buffers both the main stream 711 and the auxiliary stream 712, but with separate delays. For example, a buffer (not shown) operating on the main stream 711 may delay the stream by 1.0 seconds, and the buffer 718 may delay the auxiliary stream 712 by 2.0 seconds. As long as the auxiliary stream 712 is delayed relative to the main stream 711, the examples described herein can be implemented.

[0164] During normal operation, the primary GPU 721 and the secondary GPU 722 process the respective primary stream 711 and secondary stream 712 until the processing system 170 stops receiving input 172 (e.g., because the user has stopped playing the game). However, if the primary GPU 721 enters an undesirable state (e.g., due to a malfunction), the processing system 170 causes the secondary GPU 722 to suspend its processing.

[0165] In some embodiments, upon determining that the primary GPU 721 has entered an undesirable state, it sends a pause signal 724 to the secondary GPU 722. Additionally or alternatively, the error detection module 730 monitors output data 732 (e.g., a subsequent processing state) from the GPU 721 and detects that the primary GPU 721 has entered an undesirable state based on information in the output data or the absence of expected information in the output data. Consequently, the error detection module 730 sends a pause signal 734 to the secondary GPU 722. Upon receiving the pause signal 724 or 734, the secondary GPU 722 pauses processing in any processing state it is currently processing (the current processing state) and sends information 742 about the current state to the inspection module 740. This information 742 is referred to herein as saved information about the pre-error state, or information about the saved pre-error state. The saved information 742 can be accessed by a debugger or a developer, and either or both can apply forward debugging techniques, including but not limited to causing the auxiliary GPU 722 to step forward by sending a signal 744 instructing the GPU 722 to process a state subsequent to the current pre-error state. As successive states are processed, the auxiliary GPU 722 sends information 742 about each successive state for inspection and / or evaluation by the forward debugger or developer.

[0166] Figure 8is a block diagram illustrating an example processing system 170 according to some embodiments. The processing system 170 typically includes one or more processing units (e.g., CPU 705, GPU 721 and 722), one or more network interfaces 804, memory 708, and one or more communication buses 808 (sometimes referred to as chipsets) for interconnecting these components. The processing system 170 may optionally include one or more input devices 810 to facilitate user input, such as a keyboard, mouse, voice command input unit or microphone, touch screen display, touch-sensitive input pad, gesture capture camera, or other input buttons or controls. In addition, the processing system 170 may use a microphone and voice recognition or a camera and gesture recognition to supplement or replace the keyboard. In some embodiments, the processing system 170 may optionally include one or more cameras, scanners, or light sensor units for capturing images of, for example, a graphical serial code printed on an electronic device. The processing system 170 may also include one or more output devices 812 that enable the presentation of user interfaces and display content, including one or more speakers and / or one or more visual displays. Examples of displayed content include information related to the processing status captured by the inspection module 740.

[0167] Memory 708 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state storage devices; and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. Memory 708 optionally includes one or more storage devices located remotely from one or more processing units. Memory 708, or alternatively, non-volatile memory within memory 708, includes non-transitory computer-readable storage media. In some embodiments, memory 708 or the non-transitory computer-readable storage media of memory 708 stores the following programs, modules, and data structures, or a subset or superset thereof:

[0168] • Operating system 816, including processes for handling various basic system services and for performing hardware-related tasks;

[0169] A network communications module 818 for connecting the processing system 170 to other devices (e.g., various servers, client devices, and / or media devices) via one or more network interfaces (wired or wireless) and one or more networks, such as the Internet, other wide area networks, local area networks, metropolitan area networks, and the like;

[0170] A user interface module 820 for enabling presentation of information at a client device or at a developer device (e.g., a graphical user interface for presenting applications, widgets, websites and web pages and / or games, audio and / or video content, text, etc.) to view information related to the processing status captured by the inspection module 740;

[0171] Input delay module 710 for controlling the timing offset of auxiliary stream 712 relative to primary stream 711 (e.g., by configuring buffer 718);

[0172] An error detection module 730 for detecting an error condition of a primary processing unit (e.g., GPU 721) and, in some embodiments, sending a pause signal to a secondary processing unit (e.g., GPU 722) as a result of the error detection;

[0173] An inspection module 740 for saving information associated with one or more pre-error states of an auxiliary processing unit (e.g., GPU 722) for inspection and / or evaluation in a forward debugging context;

[0174] A program engine 822 for determining a process output or output state based on (i) one or more user inputs 172, and (ii) a process state corresponding to the user input;

[0175] Processing system data 850, including:

[0176] o Program library 852 for storing programs to be executed or simulated by processing units (e.g., GPUs 721 and 722);

[0177] o Error status data 854 for storing information associated with the error status of the primary GPU 721 (e.g., a dump resulting from a failure); and

[0178] o Pre-error state data 856 for storing information associated with the pre-error state of the secondary GPU 722 (e.g., register data, variables, internal state information, and various other data that may be inaccessible if a failure occurs); and

[0179] • A data buffer 718 for temporarily storing user input 172 received in an input stream so that a delayed stream of input can be provided to the auxiliary GPU 722 .

[0180] Each element of above identification can be stored in one or more aforementioned storage devices, and corresponds to the instruction set for performing the above functions.The module or program (that is, instruction set) of above identification need not be implemented as independent software program, process, module or data structure, and therefore, each subset of these modules can be combined or otherwise rearranged in various embodiments.In some embodiments, memory 708 optionally stores the subset of the module and data structure of above identification.In addition, memory 708 optionally stores additional modules and data structure not described above.

[0181] In some embodiments, the processing system 170 and the memory 708 may further include the Figure 4 Any or all components, modules, and data described for the server system 114 and memory 146.

[0182] Example Shadow Process

[0183] Figure 9 Depicts an example primary instance 900 of a particular program (852-1, 852-2, ...) executed by a primary GPU 721 and a secondary (delayed) instance 950 of a particular program (852-1, 852-2, ...) executed by a secondary GPU 722 in a processing system 170, according to some embodiments. In some embodiments, the program (852-1, 852-2, ...) is as described above with reference to Figure 1-6 The described gaming application.

[0184] A master instance 900 of a program (852-1, 852-2, ...) executes on GPU 721. While in current processing state 902 (game state GS0), processing system 170 receives first user input 904 instructing virtual player A to move downward. Delay module 710 passes first user input 904 to GPU 721 without delay (or, alternatively, passes the input to GPU 721 without being processed by the delay module). Master GPU 721 processes first user input 904 based on current processing state 902 (game state GS0) and generates a subsequent processing state 906 (game state GS1) as a result of current state 902 and first user input 904.

[0185] At the same time, a second instance 950 of the program (852-1, 852-2, ...) executes on the GPU 722. The delay module 710 buffers the first user input 904 and passes it to the second instance 950 at a time subsequent to the time when the first instance 900 receives the first user input 904. The secondary GPU 722 processes the first user input 904 according to the current processing state 952 (corresponding to the state 902 of the first instance) and generates a subsequent processing state 956 (corresponding to the state 906 of the first instance) as a result of the current state 952 and the first user input 904.

[0186] Processing system 170 receives second user input 908 instructing player A to shoot the puck. Delay module 710 passes second user input 908 to GPU 721 without delay (or alternatively, passes the input to GPU 721 without being processed by the delay module). Primary GPU 721 processes second user input 908 based on current processing state 906 (game state GS1) and generates a subsequent processing state 910 (game state GS2) as a result of current state 906 and second user input 908.

[0187] Meanwhile, delay module 710 buffers second user input 908 and passes it to secondary instance 950 at a time subsequent to first instance 900 receiving second user input 908. Secondary GPU 722 processes second user input 908 according to current processing state 956 (corresponding to state 906 in the first instance) and generates subsequent processing state 960 (corresponding to state 910 in the first instance) as a result of current state 956 and second user input 908.

[0188] The processing system 170 receives a third user input 912 instructing player B to block the puck. The delay module 710 passes the third user input 912 to the GPU 721 without delay (or alternatively, the input is passed to the GPU 721 without being processed by the delay module). The primary GPU 721 processes the third user input 912 according to the current processing state 910 (game state GS2) and generates a subsequent processing state 914 (error state) as a result of the current state 910 and the third user input 912. In some embodiments, the primary GPU 721 sends a pause signal 724 to the secondary GPU 722 upon entering an undesirable state (see Figure 7 ) (The pause signal is Figure 9 In some embodiments, the error detection module 730 detects an error in the output 732 of the primary GPU 721 and sends a halt signal 734 to the secondary GPU 722 (see Figure 7 ).

[0189] At the same time, the delay module 710 buffers the third user input 912 so that the third user input 912 is passed to the auxiliary instance 950 at a time subsequent to the first instance 900 receiving the third user input 912. However, before the auxiliary GPU 722 has an opportunity to process the third user input 912, the auxiliary GPU 722 receives the pause signal 920 (and / or signal 734) from the primary GPU 721 (or the error detection module 730), and as a result, processing is paused. In this way, execution of the auxiliary instance 950 of the program (852-1, 852-2, ...) is paused in the current processing state (which is consistent with the state 910 in the first instance). The current processing state is a pre-error state because it is consistent with the processing state in the primary instance that occurred before the error state. Information 742 associated with the current pre-error state (see Figure 1 ) is sent to the review module 770 for evaluation. In some embodiments, the secondary GPU 722 generates one or more subsequent pre-error states 961 (e.g., game state GS2a), and information 742 associated with these states is sent to the review module 740 for further evaluation. In some embodiments, the secondary GPU 722 generates successive pre-error states until the secondary GPU 722 reaches an error state.

[0190] In some embodiments, one or more of first user input 904, second user input 908, and third user input 912 are different multiple user inputs. For example, in the example above, multiple third user inputs 912 may have caused error state 914 in primary instance 900. Thus, individually stepping through each of multiple third user inputs 912 and generating intermediate pre-error states in auxiliary instance 950 can provide more accurate information detailing the cause of the error. Specifically, a debugger or developer will be able to more accurately determine the exact input and processing state that caused the failure, which will provide a better basis for modifying the application's programming to avoid future failures due to violating input / state combinations.

[0191] Method for saving state before error

[0192] Figure 10is a flow chart illustrating an example method 1000 for saving a pre-error processing state according to some embodiments. Method 1000 is optionally governed by instructions stored in a computer memory or non-transitory computer-readable storage medium and executed by one or more processors of processing system 170. The computer-readable storage medium may include a magnetic or optical disk storage device, a solid-state storage device (such as flash memory), or other one or more non-volatile storage devices or devices. The instructions stored on the computer-readable storage medium may include one or more of the following: source code, assembly language code, object code, or other instruction formats interpreted by one or more processors. Some operations in method 1000 may be combined and / or the order of some operations may be changed.

[0193] The method 1000 is performed by a processing system 170 that includes one or more processing units (e.g., CPU 705) and a memory that stores programs executed by the processing cores. The processing system 170 receives a first stream (or main stream 711) of an input 172 ( Figure 7 ), and buffers the first input stream (1002) to generate a buffered input stream that is identical to the input first stream. In some embodiments, the input is network packets sent from one or more client devices over a communications network (e.g., the Internet). In some embodiments, each stream is buffered, where the buffer for the first stream has a first depth (e.g., a depth of zero) and the buffer for the second stream has a second depth (e.g., a depth of N, where N is associated with a predetermined offset between the two streams). Although the input in the buffered stream is delayed relative to the input in the first stream, the time between the corresponding inputs is preserved (e.g., the difference between the first user input 904 and the second user input 908 to the primary instance 900 is equal to the difference between the first user input 904 and the second user input 908 to the auxiliary instance 950). Thus, in some embodiments, from the perspective of the auxiliary instance of the program executed by the second processing unit, the input event is not delayed at all. In other words, the input sequence (values ​​and timing) arriving at the first processing unit (e.g., GPU 721) is the same as the input sequence (values ​​and timing) arriving at the second processing unit (e.g., GPU 722), except for the above-mentioned global offset relative to the buffer.

[0194] The processing system 170 transmits (1004) the first stream to a primary instance 900 of a first program (852-1, 852-2, ...) (e.g., executing on a primary GPU 721) and transmits the buffered stream to a secondary instance 950 of the first program (852-1, 852-2, ...) (e.g., executing on a secondary GPU 722). The first (primary) processing unit (e.g., GPU 721) executes (1006) the primary instance 900 in real time using the first stream, and the second (secondary) processing unit (e.g., GPU 722) executes the secondary instance 950 using the buffered stream with a predetermined time delay relative to the execution of the primary instance 900 using the first stream.

[0195] The processing units (e.g., GPUs 721 and 722) execute each instance of the program (852-1, 852-2, ...) until the primary processing unit (e.g., GPU 721) or the error detection module 730 detects (1008) an error state resulting from executing the primary instance. In some embodiments, detecting the error state includes identifying a fault in the first instance. In some embodiments, detecting the error state includes identifying invalid output data or an undesired output state from the primary processing unit (e.g., GPU 721). For embodiments in which the primary processing unit detects an internal fault, the primary processing unit sends a pause signal or flag to the secondary processing unit. Additionally or alternatively, for embodiments in which the primary processing unit does not or cannot detect the internal fault (e.g., because the processing unit is completely shut down and / or cannot continue to process data), the error detection module 730, which continues to monitor the status of the primary processing unit, sends a pause signal or flag to the secondary processing unit.

[0196] In response to detecting the error state, the master GPU 721 or the error detection module 730 sends a signal to the master GPU 721 via the pause signal 724 and / or the pause signal 734 (see Figure 7 ) pauses (1010) the auxiliary instance 950, thereby saving the current state of the auxiliary instance 950, where the current state of the auxiliary instance 950 corresponds to the pre-error state of the primary instance 900.

[0197] Notes on this disclosure

[0198] Reference has been made in detail to various embodiments, examples of which are illustrated in the accompanying drawings. In the foregoing detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention and the described embodiments. However, the invention may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail in order not to unnecessarily obscure aspects of the embodiments.

[0199] It will be understood that although the terms "first," "second," etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first device may be referred to as a second device, and similarly, a second device may be referred to as a first device without changing the meaning of the description, as long as all occurrences of the first device are consistently renamed and all occurrences of the second device are consistently renamed. The first device and the second device are both devices, but they are not the same device.

[0200] The terms used herein are for the purpose of describing specific embodiments only and are not intended to limit the claims. As used in the description of the embodiments and the appended claims, the singular forms "a", "an" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that the terms "and / or" used herein refer to and cover any and all possible combinations of one or more associated listed items. It will also be understood that when used in this specification, the terms "include" and / or "comprising" specify the presence of the features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0201] As used herein, the term “if” may be interpreted to mean “when,” “upon,” or “in response to determining,” or “upon determining,” or “in response to detecting,” a stated conditional precedent is true, depending on the context. Similarly, the phrase “if it is determined that [the stated conditional precedent is true],” or “if [the stated conditional precedent is true],” or “when [the stated conditional precedent is true],” may be interpreted to mean, “upon determining that the stated conditional precedent is true,” or “in response to determining that the stated conditional precedent is true,” or “upon determining that the stated conditional precedent is true,” or “upon detecting that the stated conditional precedent is true,” or “in response to detecting that the stated conditional precedent is true,” depending on the context.

[0202] For purposes of explanation, the foregoing description has been described with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the invention to the precise forms disclosed. In light of the above teachings, many modifications and variations are possible. The embodiments were chosen and described in order to best explain the principles of the invention and its practical application, thereby enabling others skilled in the art to best utilize the invention and various embodiments with various modifications as are suitable for the specific application contemplated.

Claims

1. A method for saving a pre-error state of a processing unit, comprising: At a computer system having one or more processors and a memory storing one or more programs for execution by the one or more processors: Receive a first stream of input; buffering the first stream of input to generate a buffered stream of input identical to the first stream of input; transmitting the first stream of input to a primary instance of a first program; passing the buffered stream of input to an auxiliary instance of the first program; executing the master instance in real time on the input first stream; executing the auxiliary instance on the buffered stream of the input with a predefined time delay relative to executing the primary instance on the first stream of the input; as well as detecting an error condition caused by said master instance executing on said input first stream; In response to detecting the error condition: sending a signal, via the primary graphics processing unit, to the secondary graphics processing unit, wherein the signal suspends the secondary instance; as well as A current state of the auxiliary instance is saved, wherein the current state of the auxiliary instance corresponds to a pre-error state of the primary instance.

2. The method according to claim 1, wherein the error condition being caused by executing the main instance on a first input in a first stream of inputs; and The pausing the auxiliary instance includes: pausing the auxiliary instance before processing an input in the buffered stream of inputs corresponding to the first input in the first stream of inputs.

3. The method according to claim 1, wherein The secondary instance runs concurrently with the primary instance after the predefined time delay.

4. The method according to claim 1, wherein The error state masks aspects of the pre-error state of the primary instance; and wherein saving the current state of the secondary instance comprises: recording aspects of the current state of the auxiliary instance, wherein the aspects of the current state of the auxiliary instance correspond to aspects of the pre-error state of the primary instance, The aspect of the pre-error state of the primary instance is first stored data associated with the primary instance, and the aspect of the current state of the auxiliary instance is second stored data associated with the auxiliary instance.

5. The method according to claim 4, wherein The second stored data corresponds to the first stored data.

6. The method according to claim 1, wherein: Executing the primary instance includes generating a first output stream using a first processing unit; and Executing the auxiliary instance includes generating a second output stream using a second processing unit.

7. The method according to claim 6, wherein: Detecting an error condition includes detecting an indicator of an error condition in the first output stream.

8. The method according to claim 6, wherein: Detecting the error status includes detecting the error status from an indicator generated by the first processing unit.

9. The method of claim 1 , further comprising, after saving the current state of the auxiliary instance: continuing the auxiliary instance to obtain a subsequent state of the auxiliary instance; and Saving the subsequent state of the auxiliary instance, wherein The subsequent state of the secondary instance corresponds to the subsequent pre-error state of the primary instance.

10. The method according to claim 9, wherein: Pausing the auxiliary instance includes stopping the delivery of the input buffer to the auxiliary instance; and Continuing the auxiliary instance includes passing a single input from the buffered stream of inputs to the auxiliary instance.

11. The method according to any one of claims 1 to 10, wherein: Saving the current state of the auxiliary instance includes providing the current state of the auxiliary instance for inspection.

12. An electronic computer system comprising: one or more processors; and A memory storing one or more programs for execution by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 1-11.

13. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by an electronic computer system having one or more processors, cause the electronic computer system to perform the method according to any one of claims 1 to 11.

Images