Test as a service for cloud gaming

Abstract

Techniques for test as a service (TaaS) for video games are described. In one embodiment, a method includes operations for receiving a game application for testing for one or more bugs. The method also provides for executing a plurality of automated sessions of game sessions by an automated game testing module while implementing test inputs for the plurality of automated sessions, the test inputs including control inputs, game states, system parameters, and network parameters. The method further includes operations for detecting an occurrence of a bug during the executing the plurality of sessions to generate a snapshot file, the snapshot file including a portion of control inputs, game state data, and a video component associated with the occurrence of the bug.

Description

Technical Field

[0001] The present disclosure relates generally to video game development and, more particularly, to methods and systems for providing cloud testing of video games as a service. Background Art

[0002] Video game testing is part of the game development process, maintaining quality control over video games. One of the functions of game testing is to discover and document software defects (e.g., bugs) that could compromise the use of video games. In the past, due to the smaller scale and lower complexity of video games, game testing required relatively little manpower. Today, video game testing is a large-scale and expensive undertaking undertaken by developers and game publishers to ensure their game applications run smoothly. This is due to the sheer scope and complexity of today's video games and consumers' demand for near-perfect game action. Therefore, game testing plays a significant role in ensuring that released games run smoothly and are free of bugs, errors, or other defects. Some publishers may have dozens of quality assurance testers working on a particular game title at any given time, which can be very expensive. Furthermore, when game testers do identify a bug, developers are not always able to reproduce it due to variations in the game testers' gaming sessions and other factors.

[0003] It is in this context that those implementation plans emerged. Summary of the Invention

[0004] Embodiments of the present disclosure relate to methods and systems for playtest playback, automated playtesting, and testing as a service (TaaS).

[0005] In one embodiment, a method is provided. The method includes an operation for executing a video game for a player to play on a server of a cloud testing system, wherein the executing video game generates a video output. The method also includes an operation for receiving a control input sequence for interacting with the video game from a client device of the player, wherein a portion of the control input sequence causes an error to occur during the interaction with the video game. In addition, the method includes an operation for recording the portion of the control input sequence that causes the error, along with the video output and game state data generated by the video game, in a snapshot file generated by the player. In addition, the method includes an operation for processing the snapshot file generated by the player using a machine learning module, wherein the machine learning module extracts features from the control input sequence, the video output, and the game state for classification into an error detection model. In addition, the method includes an operation for using the error detection model to identify a set of test control inputs that are likely to reproduce the error through the video game.

[0006] In another embodiment, a method is provided. The method includes operations for generating player-generated snapshot files from a playthrough of a video game by one or more players, each of the player-generated snapshot files including a sequence of control inputs, game state data, and a video component associated with a portion of the playthrough of the video game in which an error occurred. The method further includes operations for processing the player-generated snapshot files using a machine learning module to generate a plurality of machine-learned control inputs that are in addition to the sequence of control inputs from the player-generated snapshot files that were used to reproduce the error. In addition, the method includes operations for executing a plurality of automated video game sessions while inputting the respective machine-learned control inputs, the executing of the plurality of automated video game sessions each producing respective game state data and respective video components, wherein the respective machine-learned control inputs, the respective game state data, and the respective video components are recorded in respective machine-generated snapshot files. Further, the method includes operations for processing the machine-generated snapshot files using the machine learning module to identify an error realization condition that can be used to identify a class of control input sequences that lead to the error, the class of control input sequences being in addition to the sequence of control inputs of the player-generated snapshot files.

[0007] In another embodiment, a method is provided. The method includes operations for receiving a game application for testing one or more errors at a server. The method further includes operations for executing an automated session of the game application by an automated game testing module while implementing test inputs for the automated session, the test inputs including control inputs, game state data, system parameters, and network parameters. In addition, the method provides for detecting an occurrence of an error by an error reporting program during the executing of the automated game session. Further, the method provides for generating a snapshot file including a portion of the control inputs, the game state data, and a video component associated with the occurrence of the error.

[0008] Other aspects of the disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrated by way of example in the several figures of the drawing. BRIEF DESCRIPTION OF DRAWINGS

[0009] The disclosure can best be understood by reference to the following description taken in conjunction with the accompanying drawings in which:

[0010] Figure 1 A conceptual diagram of error reporting and reproduction is shown in accordance with one embodiment.

[0011] Figure 2A conceptual diagram of an improved method and system for faithfully reproducing errors discovered by game testers is shown, according to one embodiment.

[0012] Figure 3 A conceptual diagram of an error reproduction module for reproducing errors identified by playtesters by processing snapshot files that capture the errors is shown, according to one embodiment.

[0013] Figure 4 A conceptual diagram of a snapshot log including multiple snapshot files is shown according to one embodiment.

[0014] Figure 5 Shown is how a cloud-based test server system according to one embodiment implements machine learning to generate new control inputs that are statistically correlated with causing previously identified errors or newly identified errors.

[0015] Figure 6 A conceptual diagram illustrating how a diff implementation module creates a diff package that is executed in conjunction with an automated game session at an automated game testing module, according to one embodiment.

[0016] Figure 7 A conceptual diagram of an automated game testing module executing multiple automated game sessions is shown, according to one embodiment.

[0017] Figure 8 A diagram illustrating the architecture of an automated game testing module that utilizes a distributed game engine to execute multiple automated game sessions, according to one embodiment.

[0018] Figure 9A and Figure 9B Conceptually illustrates how a machine learning module, an automated playtesting module, and a bug realization module may work together to identify a set of conditions that may realize a previously or newly identified bug, according to various embodiments.

[0019] Figure 10 A method for discovering new errors through chaos or shotgun testing is shown according to one embodiment.

[0020] Figure 11 A conceptual diagram illustrating automatic testing for previously identified bugs in a new build or version of a game, according to one embodiment.

[0021] Figure 12 Shown is how known misimplementation conditions for a first game can be applied to a second game as part of Testing as a Service (TaaS) according to one embodiment.

[0022] Figure 13 is a conceptual illustration of a platform that may be used for testing as a service (TaaS), according to one embodiment.

[0023] Figure 14 Components of an example device (eg, a TaaS server) that may be used to perform aspects of various embodiments of the present disclosure are shown. DETAILED DESCRIPTION

[0024] Embodiments of the present disclosure relate to methods and systems for enabling game developers to reproduce errors identified by game testers. Embodiments of the present disclosure also relate to methods for using machine learning algorithms to provide additional machine-learned control inputs that result in previously identified errors or unidentified errors. In addition, embodiments of the present disclosure relate to methods and systems for automating various parts of the testing process to test games under varying system-related conditions. In addition, embodiments of the present disclosure relate to allowing various aspects of game testing to be performed as a service. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without some or all of these specific details. In other instances, well-known processing operations have not been described in detail to avoid unnecessarily obscuring the present disclosure.

[0025] Game testing is an expensive and time-consuming process. Virtually all games that reach the commercial stage undergo numerous rounds of testing to ensure a certain level of functional and artistic quality. Testing plays a variety of roles in developing a video game application for release. Some of these include functional testing, compliance testing, compatibility testing, load testing, beta testing, multiplayer testing, and more. Generally speaking, the goal of game testing is to ensure that a video game runs smoothly and as expected on various hardware platforms. Therefore, one of the goals of game testing is to identify bugs in a video game that prevent it from running smoothly. Bugs can manifest in a variety of ways. Generally, a bug refers to a portion of a game or its associated code where unexpected results occur, whether related to game logic, game mechanics, glitches, artwork, system glitches, or other game or system responses that differ from what is expected, predicted, or anticipated.

[0026] Figure 1A conceptual diagram of error reporting and reproduction according to one embodiment is shown. A game tester 101 is shown using a testing device 100, such as a development kit, to play a segment 112 of a video game. The game tester 101 discovers three errors 106, 108, and 110 by playing the segment 112 of the video game. The game tester 101 is then shown providing an error report 102, which is delivered to a developer 104. The error report 102 can be provided through a defect tracking system and can specify the situation that produced the error (e.g., level or map coordinates) and the specific steps for reproducing the error (e.g., a set of control inputs). The developer 104 uses the error report 102 to reproduce the errors 106, 108, and 110 before attempting to fix the errors.

[0027] Developer 104, who may be an artist, programmer, or game designer, successfully reproduces error 106' on each of three attempts and reproduces error 108' on the second of three attempts. Developer 104 is unable to reproduce error 110. The inability to reproduce certain errors (e.g., error 110) and the intermittent ability to reproduce errors (e.g., error 108) may be due to a number of factors associated with game tester 101's game play. For example, game tester 101 and developer 104 may have different seed data or other game state data, or may have entered control input sequences in a significantly different timeframe. Additionally, there may be system differences associated with game tester 101's test device 100 (e.g., latency, jitter, CPU clock, GPU clock, memory usage, etc.) that can contribute to the generation of errors 108 and 110. These system differences make it difficult for developer 104 to track and simulate the reproduction of errors 108 and 110 using current methods. Furthermore, it may be the case that some playtesters report error 110 while other playtesters do not, leading to conflicting opinions as to whether error 110 exists and should be addressed. Improved methods and systems for providing faithful reproduction of errors are contemplated.

[0028] Figure 2 A conceptual diagram of an improved method and system for faithfully reproducing errors discovered by game testers 101 is shown, according to one embodiment. Figure 1 , Figure 21 is shown as having generated errors 106, 108, 110 during interaction with a segment 112 of a video game. Segment 112 may be part of a video game session 201 executing on a server system 200. That is, for example, video game session 201 may be hosted by system 204 having testing hardware 206 of server system 200. In some embodiments, testing hardware 206 may be proprietary.

[0029] System 204 is shown running an operating system 208 on which a video game application 210 for testing is executing. A game tester 101 interacts with a video game session 201 via a network 202. For example, the video game application 210 is expected to generate audio and video components at the server system 200 for delivery and presentation to the game tester 101. The server system 200 then receives control input from the game tester 101 via the network 202 to enable the game tester 101 to interact with the video game session 201. In other embodiments, the video game session 201 may be executed on a test device 100 local to the game tester 101. However, in any case, a set of raw test data 212 is generated as a result of the game tester 101 interacting with the video game application 210 during the video game session 201.

[0030] Raw test data 212 is shown as including a control input sequence 214, which can be any suitable data structure having control inputs as time-varying data elements. For example, control input sequence 214 is expected to record each of various control inputs entered on some input device of game tester 101, including button presses, keystrokes, mouse clicks, cursor movements, controller movements, joystick movements, voice commands, body movements, and other inputs made by game tester 101. Each of the data elements of control input sequence 214 can be time-stamped based on the time when the control inputs were recorded by video game application 201 and the time when they were entered at the controller in the real world. Raw test data 212 is also shown as including game state data 216, which can have any suitable data structure for capturing the game environment, including saved data, game values, map data, asset data, and other data. Game state data 216 is also expected to be time-stamped so that errors can be referenced to game state data 216 in time.

[0031] As the game tester 101 interacts with the video game application 201 to discover errors 106, 108, and 110, the system 204 generates a video component 218 as directed by the video game application 210. The video component 218 may include both an audio component and a video component in the form of a sequence of video frames. Additionally, as the game tester 101 interacts with the video game application 210, various network and system properties associated with the video game session 201 are captured in real time or near real time in network data 220 and system data 222, respectively, as part of the raw test data 212. It is contemplated that the network data 220 may include various parameters associated with the communication link between the server system 200 and the game tester 101, including, for example, latency, jitter, data rate capacity, packet loss, quality of service (QoS), throughput, and other network properties. It is contemplated that the system data 222 may include various parameters associated with the system 204 or test device 100 executing the video game application 210, including CPU clock, GPU clock, memory usage, frame rate, resolution, and other system properties. Both network data 220 and system data 222 are expected to be time-stamped so that errors can be temporally referenced to a specific time within network data 220 and / or system data 222 .

[0032] Seed data 224 and error log 226 are also shown as being included in raw test data 212. Seed data 224 refers to the initial data set that video game application 210 is used for introducing some variation or randomness into the video game. In some embodiments, seed data 224 can utilize a random number generator. In addition, error log 226 records the mark or flag that game tester 101 inputs for indicating the existence of an error. For example, when game tester 101 encounters each of errors 106, 108, and 110, they may indicate so much in error log 226. Error log 226 is also expected to be with timestamp, so that the time that each of errors 106, 108, and 110 occurs can be referenced to a specific window in each of control input sequence 214, game state data 216, video component 218, network data 220, and system data 222.

[0033] The snapshot generator 228 is configured to generate snapshot files 232 that are stored in the snapshot log 230. Each snapshot file 232 is intended to be associated with a particular bug and includes the control input sequence 214, the game state data 216, the video component 218, the network data 220, the system data 222, a portion of the seed data 224, and the bug-associated portion of the bug log 226. For example, each of the bugs 106, 108, and 110 is intended to be included within a snapshot file 232 generated by the snapshot generator 228. For example, when the game tester 101 indicates that there is a bug 106, the snapshot generator 228 is intended to access a window for each of the control input sequence 214, the game state data 216, the video component 218, the network data 220, the system data 222, and the seed data 224 corresponding to the timestamp of the bug 106 in the bug log 226 to generate a snapshot file 232 for the bug 106. The window can be between about 1 minute and 10 minutes before and after the bug 106 occurs. The snapshot generator 228 is intended to also generate snapshot files 232 for the bugs 108 and 110 to be stored in the snapshot log 230. In addition to the snapshot files 232 for the bugs 106, 108, and 110 discovered or produced by the game tester 101, the snapshot log 230 can be a database of snapshot files 232 associated with various additional bugs discovered or produced by other game testers or automated game testing agents.

[0034] The server system 200 is also intended to host a bug reproduction module 234 that interfaces with the device of the developer 104 over the network 202 to reproduce the bugs 106', 108', and 110'. The bug reproduction module 234 is intended to allow the developer 104 in a variety of ways to reproduce the bugs. Three non-limiting embodiments include a playback mode, an automatic reproduction mode, and a manual reproduction mode. In the playback mode, the snapshot files 232 are played back to the developer 104 such that the bugs 106', 108', and 110' are played back to the developer 104 as they were manifested to the game tester 101. Thus, the bugs 106, 108, and 110 are represented in the same manner as they were manifested to the game tester 101.

[0035] In the automatic reproduction mode, the error reproduction module 234 is expected to load the game state data 216 associated with the snapshot file 232 for execution of the video game application 210. The error reproduction module 234 can access the system 204, the test hardware 206, the operating system 208, and the video game application 210 for execution of the snapshot file 232 in the automatic reproduction mode. For example, the system 204 can load the game state data 216 to start the video game session 201 between about 5 seconds and about 10 minutes, or between about 10 seconds and 5 minutes, or between about 15 seconds and about 60 seconds before each of the errors 106, 108, and 110 occurs. Once the video game session 201 is started, the control input sequence 214 is automatically input into the video game application 210 based on the timestamps of each control input of the control input sequence 214. Thus, the control input sequence 214 is synchronized with the game state data 216, so that the video game session 201 is recreated in real time for the developer 104. In other embodiments, the error reproduction module 234 can be equipped with a development version of the dedicated playback system hardware, operating system, and video game application. In addition to replicating the control input sequence 214 during execution of the video game application 210, the network data 220, system data 222, and seed data 224 can also be copied, mirrored, or simulated to recreate the operating conditions during the window in which the errors 106, 108, and 110 occurred. In this way, the errors 106', 108', and 110' are reproduced for the developer, but the errors can theoretically be the same as the errors 106, 108, and 110 observed by the game tester 101.

[0036] In manual reproduction mode, the game state data 216 of the snapshot file 232 is loaded by the error reproduction module 234 for executing the video game application 210 in a state within the video game session 201 between about 5 seconds and about 10 minutes, or between about 10 seconds and 5 minutes, or between about 15 seconds and about 60 seconds before each of the errors 106, 108, and 110 occurred. However, instead of automatically entering the control input sequence 214, a textual or visual representation of the control input sequence 214 is provided to the developer 104 so that the developer can manually recreate the errors 106', 108', and 110'. In this way, the developer 104 causes or triggers the presentation of the errors 106', 108', and 110' and can try other configurations of control inputs that may or may not trigger the same errors. As a result, the developer 104 can better understand the parameters that lead to the errors. In any case, the error reproduction module enables the developer 104 to reproduce the errors 106 ′, 108 ′, and 110 ′ as identified by the game tester 101 so that the developer 104 can fix them.

[0037] Figure 3A conceptual diagram of an error reproduction module 234 according to one embodiment is shown, which is used to reproduce the error 106 identified by a game tester by processing a snapshot file 232 that captures the error 106. A snapshot file 232a "captures" the error 106 by including a portion of an error log 226a, a portion of a control input sequence 214a, a portion of game state data 216a, and a portion of a video component 218a that are associated with the time at which the error 106' occurred. Although not shown, system data, network data, and seed data are also expected to be included in the snapshot file 232a associated with the error 106'. In certain embodiments, each snapshot file 232a is associated with an instance of the error. For example, if error 106 is captured by a snapshot file 232a, errors 108 and 110 can be captured by corresponding snapshot files. However, when errors are closely associated with each other, such as when a first error causes a second error, or when a first error and a second error occur very close to each other in time, both the first error and the second error can be captured by one snapshot file.

[0038] In the illustrated embodiment, the error reproduction module 234 is expected to execute an error playback mode 302 to simply playback the error, execute an automatic playback mode 304 to automatically reproduce the same instance of the error, or execute a manual playback mode 306 to instruct the developer to manually reproduce the error. In any of the aforementioned modes, the error reproduction module 234 generates an error reproduction GUI 300 for the developer to interact with. The error reproduction GUI 300 is shown to include a plurality of panels for display, including a source code 308 panel, a game state 310 panel, a video component 312 panel, and a control input 314 panel. The video component 312 displays that the error 108 involves a football running across the field. If the error reproduction module 234 is in error playback mode 302, then as shown for the game tester, the video frame shown in the video component 312 is obtained directly from the video component 218a. Alternatively, if the automatic playback mode 304 or the manual playback mode 306 is adopted, the video component 312 may include a newly generated video frame due to executing the game application while inputting the control input sequence 214a. However, even in automatic reproduction mode 304 or in manual reproduction mode 306, error reproduction GUI 300 can display the video frames generated for display to the playtester (e.g., original frames) so that the developer can view the original frames and the newly generated frames side by side.

[0039] Source code 308 may include video game programming code containing instructions for operating the video game, including code for game mechanics, game logic, asset management, game physics, multiplayer mechanics, and metadata management. Source code 308 panel is envisioned to display the code that was executing during the occurrence of error 108. For example, source code 308 may include instructions to the game engine for causing a soccer ball to drop from a height of 5 units. Source code 308 panel is envisioned to scroll to the relevant portion of the game code when the error is reproduced.

[0040] Game state 310 is shown as including various game state values ​​associated with the soccer ball. Here, game state 310 shows that "Ball Bounce Coefficient" is set to "0" and "Ball Radius" is set to "6." The game state 310 panel enables developers to view various game state values ​​and game engine parameters that changed during the error reproduction.

[0041] In addition, the control input 314 panel enables the developer to view the control input sequence 214a that may have caused the error. Herein, a virtual representation of the controller used by the game tester is provided to present the control input sequence 214a as it was entered by the game tester. For example, the control input sequence 214a shows that the game tester entered the following sequence: "X-Down-Down-Triangle-Left_Joystick_Down". The same sequence can be virtually represented on the virtual controller in the same sequence and timing as the control input sequence 214a. For example, the virtual controller can press the "X" button, press the down button twice, press the triangle button, and tilt the left joystick downward.

[0042] Figure 4 A conceptual diagram of a snapshot log 230 including multiple snapshot files 232 is shown, according to one embodiment. The snapshot log 230 may include snapshot files 232 generated by one or more playtesters. Each snapshot file 232 in the snapshot log 230 includes a snapshot file identifier 401, an error identifier 400, a tag 402, game-related data 404, system data 406, and network data 408. The error identifier 400 can be synchronized with a bug tracking system so that errors with the same token are provided with the same error identifier 400. The tag 402 can be annotated by the playtester or developer and may include a description of the error type, the severity of the error, and a descriptor for categorizing the errors in the snapshot log 230. The game-related data 404 can be represented in a multidimensional space as a function of control inputs, video outputs, audio outputs, and game state data. The game state data itself can be represented in a multidimensional space as a function of various game state values, object coordinate values, time coordinates, levels, assets, abilities, seed data, and the like.

[0043] In addition, system data 406 includes time-dependent state data of the system executing the gaming application in which the associated error was identified. System data may include system performance data, such as CPU clock and usage, GPU clock and usage, memory speed and usage, frame rate, resolution, video compression codec and ratio, temperature indicator, and other system performance indicators. Network data 408 is expected to include time-dependent indicators of the communication channel between the game tester and the server system hosting the video game application. These indicators may specify latency, data rate capacity, jitter, and other communication properties.

[0044] Figure 5 The cloud-based testing server system 500, according to one embodiment, illustrates how machine learning can be implemented to generate new control inputs that are statistically correlated with causing previously identified or newly identified errors. The server system 500 includes a snapshot database 502, an automated playtesting module 504, a machine learning module 506, and an error implementation module 508. Multiple playtesters 501 are shown interacting with a video game to identify errors. When errors are identified, player-generated snapshot files 514 associated with those player-generated errors are recorded in the snapshot database 502. Player-generated snapshot files 514, including control inputs 516, are input into the machine learning module 506 via an application program interface 522. In some embodiments, the player-generated snapshot files 514 can be used as a training dataset 528 or ground truth. A feature extractor 524 analyzes the player-generated snapshot files 514 to determine values ​​in a feature space associated with the generation of a particular error. The feature extractor 524 can be configured to analyze features associated with a sequence of control inputs, the timing of control inputs, a previous history of control inputs, and the like. Additionally, the feature extractor 524 may analyze features associated with game state values, seed values, system parameters, and network parameters.

[0045] The machine learning module 506 uses the machine learning algorithm 526, the training dataset 528, and the player-generated snapshot files 514 to generate an error classification model 532. The error classification model 532 can be used to identify machine-learned control inputs 530 in addition to the player-generated control inputs 516. The machine-learned control inputs 530 are statistically predicted to cause errors. Any suitable machine learning algorithm 526 can be implemented to generate the error classification model 532, including but not limited to Bayesian networks, linear regression, decision trees, neural networks, k-means clustering, etc. The error classification model 532 provides statistically relevant combinations, permutations, and time-dependent sequences of control inputs that are predicted to cause player-generated errors or newly identified errors. The training dataset 528 and the player-generated snapshot files 514 can be used to assign a label to each set of incoming control inputs 516. The label can include "error" or "not an error."

[0046] In various embodiments, each set of machine-learned control inputs 530 is implemented by an automated game testing module 504. The automated game testing module 504 is configured to launch multiple game sessions to test multiple sets of machine-learned control inputs 514 for errors. The automated game testing module 504 includes a GUI module 534, a variance implementation module 536, a video game application 538, a game engine 540, an operating system 542, a game engine distribution module 544, and an error reporting program 546. The GUI module 534 enables developers to interface with the automated game testing module 504 and specify the conditions under which the automated game session is to be run. For example, the developer may decide to introduce variances (hereinafter referred to as "variances") in system or network parameters via the variance implementation module 536 when running the automated game session. In some embodiments, the machine learning module 506 may also provide a set of machine-learned variances that instruct the variance implementation module 536 on how to introduce variances during the automated game session. In other embodiments, the developer may choose to load test the video game application by introducing variances via the variance implementation module. For example, it is envisioned that developers can specify frame rate, resolution, CPU clock and usage, GPU clock and usage, and memory usage parameters to be simulated during an automated video game session. Additionally, developers can specify network differences such as latency, jitter, data rate capabilities, and other network properties to be simulated during an automated video game session.

[0047] Depending on the number of machine-learned control inputs 530 to be implemented, the differences to be introduced, and the granularity of the variations or differences between runs to be implemented, any suitable number of video game sessions can be launched in parallel. In any case, when an automated game session is run by the automated game testing module 504, the video game application 538 can be loaded with a particular game state that is immediately prior to the expected error occurrence, e.g., about 5 seconds to about 10 minutes, or about 10 seconds to about 5 minutes, or about 30 seconds to about 1 minute, before. When the video game application 538 is executed, the machine-learned control inputs 530 are input in a time-dependent manner. In addition, the automated game session can be executed while the differences are implemented via the difference implementation module 536. In this way, a set of machine-learned control inputs 514 can be used to implement multiple automated game sessions, where each session tests a different difference. For example, a first automated game session can implement a low level of jitter, while a second automated game session can implement a higher level of jitter. A third automated game session can overclock a GPU, while a fourth automated game session can underclock a similar GPU, and so on.

[0048] In addition, when the automated game sessions are executed, the error reporting program 546 can be layered on top of each game session for detecting the occurrence of errors. The error reporting program 546 can be predefined to automatically detect errors by the game states produced by the automated game sessions that are indicative of errors. In addition, the error reporting program 546 detects errors associated with game mechanics or rendering by using image analysis to identify images or portions of images that are indicative of the errors.

[0049] The results of the automated sessions, whether or not errors were reproduced, are recorded as machine-generated snapshot files 530. Based on the results of the error reporting program 546, the machine-generated snapshot files 518 are labeled as “error” or “not an error.” However, if the results of the error reporting program 546 are inconclusive or incomplete, or if the developer disagrees with the label, the developer can also annotate or change the label. The labeled machine-generated snapshot files 518 are then fed into the machine learning module 506 for training or updating the error classification model 532.

[0050] As the error classification model 532 is updated, a new set of machine-learned control inputs 530 can be generated and tested using the automated game testing module 504. The process of learning via the machine learning module 504, validating (e.g., monitoring) via the automated game testing module 504, and refining the error classification model 532 can be repeated until the error classification model 532 classifies a control input as predicted to cause an error or predicted not to cause an error with an acceptable level of confidence. That is, for example, the process of learning, validating, and model refinement can be repeated until the error classification model 532 is able to classify a new control input as either causing an error or not causing an error with a confidence interval of, for example, 99%. Furthermore, with each iteration of learning, validating, and refining, it is contemplated that the machine learning module 506 can generate multiple sets of machine-learned control inputs 530 that are statistically predicted to be more likely to cause the error. Conversely, the machine learning module 506 also improves at identifying sets of control inputs that are predicted not to cause an error.

[0051] While the machine learning module 506 has been described thus far as learning based on control inputs, such as those from player-generated snapshot files 514, the machine learning module 506 is also contemplated as learning based on system and network variances. For example, similar to generating machine-learned control inputs 530 based on error classification model 532, the machine learning module 506 can also generate machine-learned variances 503 that predict the occurrence of errors. Thus, the machine learning module 506 can detect patterns in system and network parameters associated with the occurrence of errors and can generate machine-learned variances 503 for the system and network parameters for testing via the automated game testing module 504. The automated game testing module 504 can be implemented via variance implementation module 536 so that the system and network conditions specified by the machine-learned variances 503 are simulated, emulated, or reproduced during the execution of an automated game session. The results of an error reporting program regarding whether an error occurred can be recorded in the machine-generated snapshot files 518 along with the machine-learned control inputs 530. For example, the error reporting program 546 can be used to label each run with a label of "error," "not an error," or "uncertain." Labelers can later resolve an autoplay session marked as "uncertain" to "error" or "not an error."

[0052] As with the machine learned control inputs 530, the results from the machine learned differences 503 can then be fed back into the machine learning module 506, which is used to update the error classification model 532. The cycle of learning, testing, and updating the error classification model 532 is used in one capacity to determine a set of system and network parameters that cause or nearly cause a particular error, or that collectively cause or nearly cause a set of errors.

[0053] In various embodiments, the machine learning module 506 is used in a capacity to learn, through experience, generalities in the types of control inputs, system parameters, and network parameters that lead to a particular error or set of errors. The generalities are represented in the error classification model 532 and can be expected to be extrapolated by the error implementation module 508. The error implementation module 508 is used to formalize or represent the generalities within the error classification model 532 as error implementation conditions 510, which include rules 548, categories 550, game scenarios 552, and system scenarios 554. The error implementation module 508 is also shown as including an error implementation library 512 having error implementation control inputs 556 and error implementation system parameters 520. In a sense, the error implementation module 508 is a platform for triggering or implementing a given error.

[0054] Similar to Figure 2 508 can replay or reproduce a given error for the developer to review. However, in addition, because the error implementation module 508 accesses the machine learning module 506, the snapshot database 502, the error implementation conditions 510, and the error implementation library 512, the error implementation module 508 can also show the developer a variety of other ways to cause the same error and certain limitations or boundaries on those conditions that cause the error. The error implementation module 508 is further expected to be able to identify and present newly discovered errors to the user. For example, when the machine learning module 506 generates machine-learned control inputs 530 and machine-learned differences 503, they can discover previously unidentified errors, at which point they are implemented by the automatic game testing module 504.

[0055] Error classification model 532 represents the generality associated with the occurrence of errors in various ways based on the model or machine learning algorithm 526. For example, the generality can be represented by weights, graph connectivity, clustering, distribution, probability and other relationships. Based on these, rules 548, categories 550, game contexts 552 and system contexts 554 can be constructed. Rules 548 define a set of conditions that lead to a specific error. For example, a rule such as "If X-down-down-triangle is entered when character A is at (x, y, z) on the map, an error will occur" can be included in rules 548. Categories 550 define the categories of conditions that lead to a specific error. For example, a category may limit the scope of control inputs that lead to an error.

[0056] Game context 552 and system context 554 also specify rules or categories related to situations in the video game, system, or network that cause errors. For example, there may be a game context 552 in which an AI character falls from the world when attempting to jump. Game context 552 can help identify errors that are not necessarily caused by player input. In addition, system context 554 describes the state of system and network parameters that lead to a certain error. For example, system context 554 may specify that a game session with latency exceeding 70 milliseconds, an overclocked CPU, and 60 FPS rendering will cause the player to fall from the virtual world. Of course, there may be slippage between each of rules 548, categories 550, game context 552, and system context 554, making one related to or dependent on the other.

[0057] It is also contemplated that the error implementation module 508 includes an error implementation library 512, which includes error implementation control inputs 556 and error implementation system parameters 520, which, when executed, will reproduce various errors. When a new build of the video game application 538 is being tested, the error implementation module 508 can be used to test the new build for the presence of errors that were previously identified by a tester or machine. In this way, the developer can automatically test for previously identified errors without the game tester 501 having to attempt to recreate the error from a previous build.

[0058] Figure 6 A conceptual diagram illustrates how the variance implementation module 536 creates a variance package 606 for execution in conjunction with an automated game session at the automated game testing module 504. Input variance 600 includes differences in the sequence and timing of control inputs sent to and executed by the automated game testing module 504 for the automated game session. For example, sequence 600a may be identical to the sequence of control inputs generated by a player identified by a game tester. Therefore, sequence 600a may serve as a control item expected to cause an error. Sequence 600b differs from sequence 600a in that the triangle input is shifted to an earlier time in time. Input variance 600 can help establish error implementation conditions 510. For example, if sequence 600b returns the same error as sequence 600a, the timing difference between sequences 600a and 600b can be determined to be an unspecified cause of the error. On the other hand, if sequence 600b does not return an error like sequence 600a, then, holding all else equal between sequences 600a and 600b, the timing of the input triangles can be determined to be the cause of the error.

[0059] Additionally, seed data variation 602 is shown as changing the seed data used to initialize an automated game session. Seed data 602a and seed data 602b differ, for example, in their first seed and second seed. Introducing variations in the seed data can help error implementation module 508 determine whether and how the seed data influenced the cause of the error. For example, if seed data 602a and seed data 602b return different error results, it can be inferred that the first or second seed, or both, had a causal relationship with the cause of the error. In a second iteration of difference implementation module 536, seed data variations can be introduced to isolate the seed that had a causal connection with the error.

[0060] Further, difference implementation module 536 is configured to introduce differences in system and network parameters in system variation 604 to be simulated during an automated test session. In system variation 604, the first column can represent latency, the second column can represent jitter, the third column can represent CPU clock, and the fourth column can represent GPU clock. System variation 604 can be used to mimic various real-world game conditions in which a player plays a video game with varying latency properties, jitter properties, CPU and GPU configurations, and other hardware and network variations. System variation 604 not only helps to simulate real-world game conditions, but can also point to whether these conditions had a causal relationship with the occurrence of the error. Difference implementation module 536 combines input variation 600, seed data variation 602, and system variation 604 to generate a plurality of difference packs. When an automated game session is launched by automated game testing module 504, operating system 542 can implement the set of conditions specified by difference pack 606 during the execution of the automated game session.

[0061] Figure 7A conceptual diagram of an automated game testing module 504 executing multiple automated game sessions 700 is shown, according to one embodiment. Each of the automated game sessions 700 is executed on a system 708 in communication with an operating system 542 operating a game engine 440. The game engine 540, in turn, executes a video game application 538 for the video game session 700. While the video game session 700 is running, control inputs 702, game state 704, and system parameters 706 are implemented in synchronization with the video game application 538. For example, the control inputs 702 may be implemented via input sockets of the game engine 540 during execution of the video game application 538. The control inputs 702 may be machine-learned control inputs 530 or player-generated control inputs 516, as modified by the variance implementation module 536. Additionally, the control inputs 702 may include control inputs from one or more virtual players of the multiplayer game. For example, a player-generated snapshot file 514 may include a sequence of control inputs 516 for each player during testing. The machine learning module 506 is operable to generate machine-learned control inputs 530 that change the control input for only one player or for more than one player. Thus, the automated game testing module 504 is intended to test for bugs associated with multi-player games and situations where certain types of player interactions lead to bugs.

[0062] Game state 704, such as seed data, can also be implemented via the game engine so that the video game application is initialized with predefined seed data. System parameters 706 (e.g., machine learning differences 503 modified by difference implementation module 536) can be implemented at the operating system 542 level or the system 708 level.

[0063] During each run of automated game session 700, an error reporting program 546 is executed at the operating system 542 level or the game engine 540 level. Error reporting program 546 is configured to automatically monitor and detect the occurrence of predefined errors or previously unidentified errors. Each run of automated game session 700 generates a corresponding automated game session report 710, which includes a machine-generated snapshot file associated with whether an error occurred. In this way, any number of automated game sessions 700 can be launched to better understand the conditions and boundaries that lead to the causes of errors. Understanding the conditions and their boundaries enables developers to more effectively resolve errors.

[0064] A gaming application (e.g., video game application 538) will typically require a game engine that serves a number of different functions. A typical game engine has many subcomponents for handling various aspects or features of the gaming environment and defining, for example, how it should behave in response to player input. Some of these subcomponents, or those that will be referred to as "game engine nodes," include timing nodes, physics nodes, artificial intelligence (AI) nodes, game logic nodes, game mechanics nodes, rendering nodes (which themselves may be subdivided into multiple nodes), map nodes, animation nodes, asset management nodes, network nodes, communication nodes, control input nodes, chat nodes, and other nodes. A rendering node can refer to various rendering functions (e.g., like or Typical functions of graphics APIs such as , including lens transformation, projection, clipping, lighting, shading, smoothing, etc. In addition, the game engine can provide additional services that are beneficial to the game process, such as social network interaction nodes, game help nodes, in-game resource rendering nodes, game process sharing or broadcasting nodes, etc.

[0065] In some game engine embodiments, the game engine can be provided by a single computing node (such as a computer, server, console or other device). In other embodiments, the computing node can be a virtual machine deployed by a server for the game engine, and is deployed as other virtual machines serving other game engines and game applications running there. Therefore, each in the game engine subcomponent is executed by a computing node (such as a server, virtual machine, desktop computer or console). This one-to-one or many-to-one architecture of game engine and computing node may not provide the performance required for the game engine, nor the efficiency required for the use of computing node, because the game engine may be elastic in its computing demand, rather than fixed. That is to say, the game engine may have processing, graphics, memory and network requirements different from any other given time in the same game at any given time.

[0066] For example, in a massively multiplayer online role-playing game (MMORPG), the computational requirements of the associated game engine may depend on the type of action occurring in the game (e.g., whether the players are engaging with each other), the number of players (e.g., whether there are currently 10 players, but there will be 100 players in the near future), and other factors. When resources are provided to a compute node of the game engine (e.g., within a server), the associated hardware may have a narrow window in which it operates as efficiently as needed and with the required performance. For example, when there are few players and they are not interacting in the MMORPG, the associated compute node may be underutilized, while when there are many players and they are engaging with each other, the associated compute node may perform poorly, resulting in reduced quality of service, lags, rendering quality issues, and errors.

[0067] These considerations are equally important in the context of automated playtesting, where hundreds, thousands, or more game sessions and associated game engines may be instantiated at once. For example, the computational requirements of the game engine for instantiating multiple automated game sessions will vary in a time-dependent manner depending on the game environment and the behavior of the objects therein, such as gathering resources at one time and battling AI characters at another. Furthermore, the type of computational requirements can vary depending on the time or game state. For example, the processing, graphics, memory, and network requirements of the game engine may vary between scenarios such as a player shooting a free throw and, after a missed free throw, the opposing team rebounding and pushing the ball up the court. Furthermore, automated game sessions can be used to automate multi-player game play by implementing control input from multiple players simultaneously during runtime. The presence of multi-player input also introduces further flexibility in the computational requirements of the corresponding game engine, for example, depending on the number of players and their level of interaction. Therefore, improved game engine architectures are contemplated for executing automated game sessions for the automated playtesting module 504.

[0068] Figure 8A diagram illustrating the architecture of an automated game testing module 504 utilizing a distributed game engine 800 to execute multiple automated game sessions 700, according to one embodiment, is shown. A video game application 538 executes on a distributed game engine 800 having multiple game engine nodes 800a and a game engine manager 800b. Each game engine node 800a is intended to serve as a subcomponent or portion of a subcomponent of the distributed game engine 800 as a whole. In some embodiments, game engine nodes 800a may be predefined based on the functionality they provide for the distributed game engine 800. For example, there may be game engine nodes 800a that are defined or dedicated to performing functions related to each of game logic, game mechanics, game timing, AI, physics, input mapping, scripting, networking, audio, graphics, animation, asset management, and in-game services. For example, for an automated game session 700a, game engine node 1 (GEN 1) may be dedicated to processing, for example, the game logic subcomponent / function, without processing any functions related to graphics or asset management. Meanwhile, game engine node 2 (GEN 2) may be defined to specifically handle AI, with GEN 3 defined for physics, GEN 4 for scripting, GEN 5 for graphics, GEN 6 for scripting, and GEN n for services such as live game broadcasting.

[0069] Each game engine node 800a is matched with a computing node within the server 802 of the rack 804 via the game engine manager 800b, depending on the computing requirements of each game engine node 800a. The game engine node 800a can communicate with the computing node via UDP, TCP, or some other protocol. The server 802 and the rack 804 can be located in a data center, etc. The game engine manager 800b serves as a bus between the game engine node 800a and the assigned computing node. The game engine manager 800b performs various tasks related to sending and receiving data, buffering, routing, threading, queuing, relaying, splicing, etc. In this way, the game engine manager 800b ensures that the operations of each game engine node 800a are executed by the desired computing node, and the resulting return values ​​or other results are delivered back to the appropriate game engine node 800a for implementation into the video game application 538. In addition, the game engine manager 800a manages the communication between the game engine nodes 800a to ensure the normal operation of the entire video game application.

[0070] like Figure 8 As shown in FIG, game engine node 1 GEN 1 is assigned to compute node 806, which is associated with a hardware CPU on server 1 located in rack 1. For example, a game engine manager may employ compute node 806 because of its computational requirements for processing game logic.

[0071] Meanwhile, automated game session 700b is also executed on a distributed game engine 800 having multiple game engine nodes GEN 1 through n, each assigned to handle a specific function or subcomponent of distributed game engine 800. The same applies to game session 700n. For example, similar to GEN 1 in automated game session 700a, the operations of GEN 1 in automated game session 700b and GEN 1 in automated game session 700n are both assigned to compute node 806 on server 1 in rack 1. Therefore, compute node 806 executes the processes requested by each of GEN 1, GEN 2, and GEN 3. GEN 2 for each of automated game sessions 700a, 700b, and 700n is assigned to compute node 808, which is a virtual machine VM 1 equipped with CPU 2, GPU 1, and 16GB of RAM. Game engine manager 800b may have identified VM 1 based on the anticipated needs of GEN 2, for example, to execute AI. In some embodiments, game engine manager 800b can itself request deployment of VM 1, or the request can be made by automated game testing module 504. In some embodiments, if the operation requested to be performed by a compute node is substantially similar for each game engine node assigned to the compute node, the compute node can potentially perform the requested operation once and return the result to each game engine node, rather than performing the requested operation once for each game engine node.

[0072] GEN 3 of each automated gaming session 700 is shown as being associated with compute node 812, which is associated with GPU 1, while GEN 4 is associated with compute 814, which is associated with GPU 2. GPU 1 and GPU 2 may have different configurations. For example, GPU 1 may have more cores, a higher clock speed, more memory, or a higher memory bandwidth than GPU 2. In this case, if GEN 3 requires a larger amount of computation or a more complex computation, then GEN 3's operations may have been assigned to GPU 1. GEN 4's operations may have been assigned to GPU 2 because GEN 4 may require fewer computations or less complex computations.

[0073] GEN 5 is shown as being assigned to compute node 810, which is a virtual machine VM 2 equipped with CPU 3, GPU 2, and 8 GB of RAM. GEN 5 can have been assigned to compute node 810 by game engine manager 800b to match the compute requirements of GEN 5. GEN 6 of automated game session 700 is shown as being assigned to containerized compute node 816, which includes containers 1 through N that interface with the host operating system on server 3 of rack 3. In particular, GEN 6 of automated game session 700a is assigned to container 1, GEN 6 of automated game session 700b is assigned to container 2, and GEN 6 of automated game session 700n is assigned to container N. Each of containers 1 through N is a unit of containment of software and dependencies, e.g., the unit includes software for processing operations of the associated game engine node. Each container runs on a container runtime that interfaces with the operating system of the server. In this sense, the operating system is virtualized for each container, while the hardware is virtualized for the virtual machine.

[0074] Figure 9A and Figure 9B The machine learning module 506, automated game testing module 504, and bug implementation module 508 are conceptually shown working together to identify a set of conditions under which a previously or newly identified bug can be implemented, in accordance with various embodiments. Bug implementation conditions refer to a range or category of conditions that are causally related to the occurrence of a given bug. Conditions refer to control inputs, system parameters, network parameters, and game states that, when causing these states to exist or coexist during execution of a game application, individually or collectively cause a given bug. Thus, bug implementation conditions can be conceptualized in a multi-dimensional space as a set of all combinations of conditions that cause or can cause a bug. Herein, bug 1 implementation conditions 900 would refer to a set of combinations of control inputs that, when executed under particular system and network parameters and under certain game states, cause or can cause bug 1. The machine learning module 506, automated game testing module 504, and bug implementation 508 are expected to work in concert to identify, bound, or approximate the implementation conditions of bug 1 to help developers understand the root cause of the bug, resulting in higher quality and more effective fixes. Bug 1 implementation conditions 900 can be formalized in bug implementation module 508 in terms of rules, categories, and contexts to convey to developers. For the sake of clarity, the following will be described with respect to machine-learned control inputs used to populate bug 1 implementation conditions 900. However, these principles apply with similar force to using machine-learned system and network parameters and game state data to populate the rest of bug 1 implementation conditions.

[0075] When one or more playtesters identify an error, the player-generated control inputs 902 are recorded in a snapshot file and processed by the machine learning module 506. The machine learning module 506 uses the error classification model to generate machine-learned control inputs 904, which are then implemented by the automated playtesting module 504 to detect the presence of Error 1. Here, a portion of the machine-learned control inputs 904 result in Error 1. The results are fed back into the machine learning module 504 to update the error classification model and generate a new set of machine-learned control inputs 906. Here, as might be expected, a greater portion of the second set of machine-learned generated control inputs 906 result in Error 1 compared to the first set of machine-learned control inputs 904. The learning and testing process is repeated for the machine-learned control inputs 908, and so on, until, for example, no new set of error-inducing control inputs is found. Error 1 implementation condition 900 is then processed by the error implementation module 508 to extract any rules, categories, or contexts attributable to Error 1 implementation condition 900.

[0076] exist Figure 9B In the example, the learning and testing process discovered a new error 2 during the analysis of error 1 implementation condition 900. For example, machine-learned control input 904 was tested by automated game testing module 504 and found to trigger both error 1 and error 2. In this case, in addition to or in parallel with assembling error 1 implementation condition 900, machine learning module 506, automated game testing module 504, and error implementation module 508 can also perform the same operations on error 2 implementation condition 901. Machine-learned control inputs 906 to 914 are generated by machine learning module 506 and tested by automated game testing module 904. The resulting range of error 2 implementation condition 901 is then formalized by error implementation module 508 into rules, categories, and scenarios, which will have those rules, categories, and scenarios for both error 1 and error 2.

[0077] Figure 10Shown is a method for discovering new errors by chaos or shotgun testing according to an embodiment.Except for the error definition error realization condition of known errors or player identification, the system and method proposed herein can also be expected to detect unidentified errors in an automatic manner.In some embodiments, chaos testing or shotgun testing is provided to find undiscovered errors.Herein, chaos testing module 1001 generates multiple groups of chaotic inputs 1002a to 1002g.Each of chaotic inputs 1002a to 1002g can comprise the input defined randomly, occasionally or chaotically.In other embodiments, chaotic inputs 1002a to 1002g can be known or knowable input sequences designed to strengthen game applications.In any case, in chaotic inputs 1002a to 1002g, chaotic input 1002c is shown as causing error 3 as detected by automatic game testing module 504. The automatic game testing module 504 relays the result to the machine learning module 506, which generates more targeted machine-learned inputs 1004a and 1004b. The learning and testing process continues until the scope of the error 3 implementation condition 1000 can be defined. In some embodiments, the chaos testing module 1001 can continue to generate chaotic inputs to find other ways to cause error 3 or other errors. Therefore, the system method proposed herein further enables automatic identification of previously unknown errors. The process of using the chaos testing module 1001, machine learning module 506, automatic game testing module 504 and error implementation module 508 to identify previously unknown errors can be provided to developers as a service (e.g., Test as a Service, TaaS), as discussed in more detail below.

[0078] Figure 11 A conceptual diagram illustrating automated testing for previously identified bugs in a new build or version of a game is shown. Assume that the circle representing bug implementation condition 1100 represents the bug implementation condition 1100 within the control input, game state, system, and data parameter space for a first build. Further assume that the bug implementation module has defined four categories of control inputs 1101a through 1101d that effectively overlap with most of the bug implementation conditions 1100 and trigger bugs in the first build. If each of control inputs 1101a through 1101d causes the bug again in the second build, it's likely that the bug has not been fixed. Alternatively, if only a portion of control inputs 1101a through 1101d causes the bug, it's likely that the bug has been partially fixed. Otherwise, if none of control inputs 1101a through 1101d cause the bug when implemented in the second build, it's likely that the bug has been fixed. Therefore, when a second build is constructed, previous bugs from the previous build can be automatically tested quickly and efficiently.

[0079] Figure 12 Shows how known misimplementation conditions for a first game can be applied to a second game as part of Testing as a Service (TaaS) according to one embodiment. Figure 12 In the example, the error implementation condition 1100 of the first game is known and can be reproduced using control inputs 1101a through 1101d. As part of the testing-as-a-service process, the control inputs 1101a through 1101d can be applied to a second game title, which can be created by a different developer than the first game title. Figure 12 , control inputs 1101a through 1101c are shown as partially or completely causing an error, while control input 1101d is shown as being outside of the error implementation conditions for Game Title 2 and therefore not causing the error.

[0080] Figure 13 is a conceptual illustration of a platform that can be used for Testing as a Service (TaaS), according to one embodiment. TaaS is intended to enable game developers to test video games for bugs and verify bug fixes without the need for a team of human game testers. Here, a developer 1300 simply drags and drops a game into a TaaS server 1303, which includes a first build of a game title 1302. The game title 1302 is executed as a video game application 538 running on an appropriate game engine 540. The game engine 540 can be a Figure 8 542. An appropriate operating system 540 is provided for the game engine 540. If the game title 1302 is intended to be executed on different game engines, different operating systems, or versions thereof, the automated game testing module 504 can initiate automated video game sessions of the video game application 538 for each type of game engine 540 and operating system 542 and the desired combination thereof. The automated game sessions can be run in parallel or in other ways.

[0081] The game title 1302 can be tested by the automated game testing module 504 in many ways. For example, TaaS can provide chaos testing of the video game application 538 via the chaos testing module 1001. Any errors identified through chaos testing will be recorded in the snapshot database 502. In addition, if the developer 1300 has previously identified an error and wants to understand the circumstances that led to the error, they can interact with the video game application 538 to manually cause the error when executing on TaaS. As a result of this interaction, a snapshot file will be generated, and the control inputs that caused the error will be recorded in the snapshot database 502. The machine learning module 506 can then generate new machine-learned control inputs 530 and machine-learned differences 503 that also lead to the error.

[0082] In addition, the video game application can be tested for error implementation conditions 510, as well as error implementation control inputs 556 and error implementation system parameters 520, of the error implementation module 508. The error implementation module 508 can provide various control inputs and system and network parameters that are known to cause errors in previous builds of the video game, for different video games having similar features to the game title 1302 or for different games having a common game engine. In addition, the TaaS server 1303 can communicate with one or more human game testers to discover errors, for which raw test data has been generated as snapshot files at the snapshot database 502.

[0083] Any errors identified in the manner described above are recorded in the snapshot database along with the control inputs 1302 (and system and network parameters) that caused the errors. These are shared with the developer 1300. Furthermore, these errors can be reproduced for the developer 1300 via the error reproduction module 234 in any of the playback mode 302, automatic reproduction mode 304, or manual reproduction mode 306. In this way, the developer can view any discovered errors and the source code, game state, control inputs, and video components associated with the errors in a GUI such as the error reproduction GUI 300.

[0084] The machine learning module 506 accesses the snapshot database 502 to generate machine-learned control inputs 530 and machine-learned differences 503 that are predicted to cause previously identified errors or new errors. These machine-learned control inputs 530 and machine-learned differences 503 are then tested in the automated game testing module 504 for the error reporting program 546 to detect whether a previously identified error or a previously unidentified error has occurred. The results of the automated game testing module 504 are again recorded in the snapshot database 502. Here, again, the machine-learned control inputs 530 and machine-learned differences 503 can be provided to the developer 1300 so that the developer can be informed of additional control inputs and system and network parameters that cause errors. In addition, the developer may be informed of new errors and the control inputs and system and network parameters that cause them.

[0085] The learning and testing process can be repeated to refine the error classification model 532 so that the error implementation module 508 can extract generalities associated with the cause of the error via the rules 548, categories 550, game contexts 552, and system contexts 554. The error implementation conditions 510 can then be provided to the developer 1300, which helps identify the conditions that led to the error. This enables the developer 1300 to better understand the root cause of the error, allowing for a higher-quality fix. After the developer 1300 attempts to fix the error in the second build of the game title 1304, the game title 1304 can be dropped into the TaaS server 1303 again for verification and testing. To see if the error has been fixed, certain control inputs 1302 known to have caused the error in the previous build (i.e., the first build) can be tested on the automated game testing module 504. Additional control inputs and system parameters can be generated from the error implementation module 508 that, while not necessarily tested to cause the error in the previous build, are expected to have the use of the rules 548, categories 550, game contexts 552, and system contexts 554. In this manner, developers 1300 are enabled to quickly test the operation of new builds with bug fixes.

[0086] Figure 14 Components of an example device 1400 that can be used to perform various aspects of various embodiments of the present disclosure are shown, such as server system 200, server system 500, one of server 802, or TaaS server 1300. The block diagram shows device 1400, which can be combined with or can be a personal computer, video game console, personal digital assistant, server, or other digital device suitable for practicing embodiments of the present disclosure. Device 1400 includes a central processing unit (CPU) 1402 for running software applications and optionally an operating system. CPU 1402 can be composed of one or more homogeneous or heterogeneous processing cores. For example, CPU 1402 is one or more general-purpose microprocessors with one or more processing cores. Other embodiments can also be implemented using one or more CPUs with a microprocessor architecture that is particularly suitable for highly parallel and computationally intensive applications such as automated game testing, machine learning operations, and error reproduction processes. Device 1400 can be local to the player playing the game segment (e.g., a game console) or remote from the player (e.g., a back-end server processor).

[0087] Memory 1404 stores applications and data used by CPU 1402. Storage 1406 provides non-volatile storage and other computer-readable media for applications and data and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROMs, DVD-ROMs, Blu-ray Discs, HD-DVDs, or other optical storage devices, as well as signal transmission and storage media. User input devices 1408 communicate user input from one or more users to device 1400. Examples of such devices may include a keyboard, mouse, joystick, touchpad, touch screen, still or video recorder / camera, tracking device for gesture recognition, and / or microphone. Network interface 1410 allows device 1400 to communicate with other computer systems via an electronic communications network and may include wired or wireless communications over local area networks and wide area networks such as the Internet. Audio processor 1412 is adapted to generate analog or digital audio output from instructions and / or data provided by CPU 1402, memory 1404, and / or storage 1406. The components of device 1400 , including CPU 1402 , memory 1404 , data storage device 1406 , user input device 1408 , network interface 1410 , and audio processor 1412 , are connected via one or more data buses 1422 .

[0088] Graphics subsystem 1420 is further connected to data bus 1422 and components of device 1400. Graphics subsystem 1420 includes a graphics processing unit (GPU) 1416 and graphics memory 1418. Graphics memory 1418 includes display memory (e.g., a frame buffer) that stores pixel data for each pixel of an output image. Graphics memory 1418 can be integrated into the same device as GPU 1416, connected to GPU 1416 as a separate device, and / or implemented within memory 1404. Pixel data can be provided directly to graphics memory 1418 from CPU 1402. Alternatively, CPU 1402 can provide data and / or instructions defining a desired output image to GPU 1416, which then generates pixel data for one or more output images based on the data and / or instructions. The data and / or instructions defining the desired output image can be stored in memory 1404 and / or graphics memory 1418. In an embodiment, GPU 1416 includes 3D rendering capabilities for generating pixel data for an output image based on instructions and data defining the geometry, lighting, shading, texturing, motion, and / or camera parameters of a scene. GPU 1416 may also include one or more programmable execution units capable of executing shader programs.

[0089] Graphics subsystem 1420 periodically outputs pixel data for an image from graphics memory 1418 for display on display device 1414. Display device 1414 may be any device capable of displaying visual information in response to signals from device 1400, including CRT, LCD, plasma, and OLED displays. Device 1400 may provide, for example, analog or digital signals to display device 1414.

[0090] It should be noted that access services delivered over a wide geographic area (such as providing access to games in the current implementation) often use cloud computing. Cloud computing is a type of computing in which dynamically scalable and often virtualized resources are provided as a service over the Internet. Users do not need to be experts in the technical infrastructure in the "cloud" that supports them. Cloud computing can be divided into different services, such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). Cloud computing services typically provide common applications (such as video games) that are accessed online from a web browser, while the software and data are stored on servers in the cloud. Based on the way the Internet is depicted in computer network diagrams, the term cloud is used as a metaphor for the Internet and an abstract concept for the complex infrastructure that it hides.

[0091] Most video games played on the Internet are operated via a connection to a game server. Typically, games use a dedicated server application that collects data from players and distributes it to other players. Users access remote services using a client device that includes at least a CPU, a display, and an I / O. The client device can be a PC, a mobile phone, a netbook, a PDA, etc. In one embodiment, the network identification client used on the game server adjusts the communication method used. In other cases, the client device uses a standard communication method (e.g., HTML) to access the application on the game server via the Internet.

[0092] Embodiments of the present disclosure may be practiced with various computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. The present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wired or wireless network.

[0093] It should be understood that a given video game or game application may be developed for a specific platform and a specific associated controller device. However, when such games are made available through the game cloud system presented herein, users may be using different controller devices to access the video game. For example, a game may have been developed for a game console and its associated controller, while a user may be accessing a cloud-based version of the game from a personal computer using a keyboard and mouse. In this case, an input parameter configuration may define a mapping of inputs generated from a user's available controller devices (in this case, a keyboard and mouse) to inputs acceptable for executing the video game.

[0094] In another example, the user can access the cloud gaming system via a tablet computing device, a touch screen smartphone or other touch screen driven device. In this case, the client device and the controller device are integrated together in the same device, where the input is provided by a detected touch screen input / gesture. For such a device, the input parameter configuration can define a specific touch screen input corresponding to the game input of the video game. For example, during the running of the video game, a button, a steering wheel or other type of input element may be displayed or covered to indicate the position on the touch screen that the user can touch to generate the game input. Gestures such as swiping in a specific direction or specific touch actions can also be detected as game input. In one embodiment, a guide indicating how to provide input by the touch screen in order to play the game can be provided to the user before starting to play the video game, for example, so as to adapt the user to operating controls on the touch screen.

[0095] In some embodiments, the client device serves as a connection point for the controller device. That is, the controller device communicates with the client device via a wireless or wired connection to transmit input from the controller device to the client device. The client device can process these inputs in turn and then transmit the input data to the cloud game server via a network (e.g., accessed via a local networking device such as a router). However, in other embodiments, the controller itself can be a networked device with the ability to directly communicate input to the cloud game server via the network without first communicating such input through the client device. For example, the controller can be connected to a local networking device (such as the above-mentioned router) to send data to the cloud game server and receive data from the cloud game server. Therefore, although the client device may still need to receive the video output from the cloud-based video game and render it on a local display, input latency can be reduced by allowing the controller to directly send input to the cloud game server over the network, thereby bypassing the client device.

[0096] In one embodiment, the networked controller and client device can be configured to send certain types of input directly from the controller to the cloud gaming server, and to send other types of input via the client device. For example, inputs whose detection does not rely on any additional hardware or processing other than the controller itself can be sent directly from the controller to the cloud gaming server via the network, thereby bypassing the client device. Such inputs can include button inputs, joystick inputs, embedded motion detection inputs (e.g., accelerometers, magnetometers, gyroscopes), etc. However, inputs that utilize additional hardware or require client devices to be processed can be sent by the client device to the cloud gaming server. These may include video or audio captured from the gaming environment, which can be processed by the client device before being sent to the cloud gaming server. In addition, the input from the motion detection hardware of the controller can be processed by the client device in conjunction with the captured video to detect the position and motion of the controller, which is then communicated to the cloud gaming server by the client device. It should be understood that the controller device according to various embodiments can also receive data (e.g., feedback data) from the client device or directly from the cloud gaming server.

[0097] It should be understood that the various embodiments defined herein may be combined or assembled into specific implementations using the various features disclosed herein. Therefore, the examples provided are merely some possible examples and are not intended to limit the various implementations that may be possible by combining various elements to define further implementations. In some examples, some implementations may include fewer elements without departing from the spirit of the disclosed or equivalent implementations.

[0098] Embodiments of the present disclosure may be practiced with various computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. Embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked over wired or wireless networks.

[0099] While the method operations are described in a particular order, it should be understood that other housekeeping operations may be performed between the operations, or the operations may be adjusted so that they occur at slightly different times, or the operations may be distributed in a system that allows processing operations to occur at various intervals associated with the processing so long as the processing of telemetry and game state data used to generate the modified game state is performed in the desired manner.

[0100] One or more embodiments may also be manufactured as computer-readable code on a computer-readable medium. A computer-readable medium is any data storage device that can store data that can then be read by a computer system. Examples of computer-readable media include hard drives, network attached storage devices (NAS), read-only memories, random access memories, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical and non-optical data storage devices. The computer-readable medium may include computer-readable tangible media distributed on a network-coupled computer system so that the computer-readable code is stored and executed in a distributed manner.

[0101] Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. The present embodiments are therefore to be considered as illustrative and not restrictive, and the embodiments are not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.

Claims

1. A test-as-a-service method for cloud gaming, comprising: executing a video game for a player to play on a server of the cloud testing system, wherein the execution of the video game generates a video output; receiving, from a client device of the player, a control input sequence for interacting with the video game, a portion of the control input sequence causing an error to occur during the interaction with the video game; recording said portion of a control input sequence that resulted in said error in a player-generated snapshot file along with said video output and game state data produced by said video game; processing the player-generated snapshot file using a machine learning module that extracts features from the control input sequence, the video output, and the game state data for classification into an error detection model; identifying, using the error detection model, a set of test control inputs that are likely to reproduce the error through the video game, The method further comprises: executing a plurality of automated video game sessions while respectively inputting the set of test control inputs for reproducing the error, the executing the plurality of automated video game sessions each generating corresponding game state data and corresponding video components, wherein the corresponding test control inputs, game state data, and video components are recorded in corresponding machine-generated snapshot files in a snapshot database of the cloud testing system, The method further comprises: using the machine learning module, performing classification of corresponding snapshot files based on whether the error is reproduced, and updating the error detection model based on the classification; extracting error realization conditions from the error detection model, the error realization conditions being able to be used to identify additional control inputs that cause the error, The error implementation condition includes one or more rules associated with the error, and the one or more rules can be used to identify a control input sequence category that causes the error, and the control input sequence category includes the control input sequence and additional control input sequences.

2. A method as claimed in claim 1, wherein the error implementation condition includes a game context associated with the error, and the game context can be used to identify one or more game state categories shared by the player-generated snapshot file and the machine-generated snapshot file associated with the occurrence of the error.

3. The method of claim 2, wherein the one or more game state categories specify one or more of a level, a stage, a mission, a mission, an in-game action, a map area, a character, an item, an ability, an in-game object, an interaction between in-game objects, an artificial intelligence (AI) character, a player-controlled character, or a structure.

4. A method as claimed in claim 1, wherein the error implementation condition includes a system context associated with the error, and the system context can be used to identify one or more system parameter categories shared by the player-generated snapshot file and the machine-generated snapshot file associated with the occurrence of the error.

5. The method of claim 4, wherein the system parameter category specifies one or more of a clock rate of a central processing unit (CPU), a clock rate of a graphics processing unit (CPU), a data rate capacity of a communication link, a latency of the communication link, a jitter or a compression rate associated with the communication link.

6. The method of claim 1 , wherein each of a plurality of automated video game sessions is executed on a distributed game engine, the distributed game engine comprising a plurality of functional units having operations performed by a corresponding plurality of computing nodes, wherein the operations performed by the corresponding plurality of computing nodes for the distributed game engine are managed by a distributed game engine manager.

7. The method of claim 1 , wherein the automated video game session is executed on a corresponding distributed game engine, each distributed game engine comprising a plurality of game engine nodes for performing a corresponding function, each of the game engine nodes communicating with a corresponding computing node to process operations for performing the corresponding function.

8. A computer system configured to implement the method of any preceding claim.

Images