Method for executing test as service (TaaS) and computer readable medium

Abstract

The invention relates to a method for executing a test as a service TaaS and a computer readable medium. A method for executing a test as a service TaaS includes receiving, at a server, a game application from a client for testing one or more errors; an automatic game testing module executes a plurality of automatic sessions of the game application program, corresponding testing input is achieved aiming at the automatic sessions, and the corresponding testing input comprises control input, game state data, system parameters and network parameters; detecting, by an error reporting program, an occurrence of an error during the plurality of automatic sessions executing the game application; and generating a snapshot file including a portion of the control input, the game state data, and a video component associated with the occurrence of the error.

Description

[0001] This application is a divisional application of the invention patent application with application number 201980089044.7, application date November 19, 2019, and invention title "Test as a Service for Cloud Gaming". Technical Field

[0002] This disclosure generally relates to video game development, and more specifically, to methods and systems for providing cloud testing of video games as a service. Background Technology

[0003] Video game testing is an integral part of the game development process, maintaining quality control over the game. One of the functions of game testing is to discover and document software defects (e.g., bugs) that could impair the use of a video game. In the past, due to the smaller size and lower complexity of video games, game testing required relatively little manpower. Today, video game testing is a large-scale and expensive undertaking for developers and publishers to ensure the smooth operation of their game applications. This is due to the vast scope and complexity of today's video games and consumers' demand for near-perfect gameplay. Therefore, game testing plays a significant role in ensuring that released games run smoothly without bugs, errors, or other defects. Some publishers may dedicate dozens of quality assurance testers to a specific game title at a given time, which can be extremely expensive. Furthermore, when game testers do identify bugs, developers are not always able to reproduce them due to differences in the testers' gameplay sessions and other factors.

[0004] Those implementation plans emerged in this context. Summary of the Invention

[0005] The embodiments disclosed herein relate to methods and systems for game test replay, automated game testing, and Test as a Service (TaaS).

[0006] In one implementation, a method for performing Test as a Service (TaaS) is provided. The method includes: receiving a game application from a client at a server for testing one or more errors; executing multiple automated sessions of the game application by an automated game testing module, while implementing corresponding test inputs for the multiple automated sessions, the corresponding test inputs including control inputs, game state data, system parameters, and network parameters; detecting the occurrence of errors by an error reporting program during the execution of the multiple automated sessions of the game application; and generating a snapshot file including a portion of the control inputs, the game state data, and a video component associated with the occurrence of the error.

[0007] In another embodiment, a computer-readable medium is provided having program instructions for performing a Test as a Service (TaaS), the program instructions including: program instructions for receiving, at a server, a game application for testing one or more errors from a client; program instructions for executing multiple automated sessions of the game application by an automated game testing module, while implementing corresponding test inputs for the multiple automated sessions, the corresponding test inputs including control inputs, game state data, system parameters, and network parameters; program instructions for detecting the occurrence of errors by an error reporting program during the execution of the multiple automated sessions of the game application; and program instructions for generating a snapshot file, the snapshot file including a portion of the control inputs, the game state data, and a video component associated with the occurrence of the error.

[0008] Other aspects of this disclosure will become apparent from the following detailed description taken in conjunction with the accompanying drawings, which illustrate the principles of this disclosure by way of example. Attached Figure Description

[0009] This disclosure can be best understood by referring to the following description in conjunction with the accompanying drawings, in which:

[0010] Figure 1 A conceptual diagram illustrating error reporting and reproduction based on one implementation scheme is shown.

[0011] Figure 2 A conceptual illustration is shown of an improved method and system for faithfully reproducing errors discovered by game testers, according to one implementation scheme.

[0012] Figure 3 A conceptual illustration of an error reproduction module according to one embodiment is shown, the error reproduction module being used to reproduce the error by processing a snapshot file that captures an error identified by a game tester.

[0013] Figure 4 A conceptual diagram of a snapshot log comprising multiple snapshot files according to one implementation scheme is shown.

[0014] Figure 5 This illustrates how a cloud-based test server system, according to one implementation, implements machine learning to generate new control inputs that are statistically correlated with leading to previously identified errors or newly identified errors.

[0015] Figure 6 A conceptual diagram is shown illustrating how a difference implementation module creates a difference package that is executed together with an automated game session at the automated game testing module, according to one implementation scheme.

[0016] Figure 7A conceptual illustration is shown of an automated game testing module executing multiple automated game sessions according to one implementation scheme.

[0017] Figure 8 A diagram illustrating the architecture of an automated game testing module that utilizes a distributed game engine to execute multiple automated game sessions, according to one implementation scheme.

[0018] Figure 9A and Figure 9B The concept illustrates how machine learning modules, automated game testing modules, and error implementation modules, according to various implementation schemes, can work together to identify a set of conditions that can implement previously or newly identified errors.

[0019] Figure 10 A method for discovering new errors through chaos or shotgun testing, according to one implementation, is shown.

[0020] Figure 11 The diagram illustrates a concept of automatically testing for previously identified bugs in a new build or version of a game, according to one implementation.

[0021] Figure 12 This demonstrates how, according to one implementation, as part of Test as a Service (TaaS), known bug implementation conditions for the first game can be applied to the second game.

[0022] Figure 13 This is a conceptual diagram of a platform that can be used to test TaaS (Ta as a Service) based on an implementation scheme.

[0023] Figure 14 Components of an example apparatus (e.g., a TaaS server) that can be used to carry out various embodiments of this disclosure are shown. Detailed Implementation

[0024] Embodiments of this disclosure relate to methods and systems for enabling game developers to reproduce errors identified by game testers. Embodiments of this disclosure also relate to methods for using machine learning algorithms to provide additional machine-learned control inputs that lead to previously identified or unidentified errors. Furthermore, embodiments of this disclosure relate to methods and systems for automating parts of the testing process to test games under varying system-related conditions. Additionally, embodiments of this disclosure relate to allowing aspects of game testing to be performed as a service. However, it will be apparent to those skilled in the art that this disclosure can be practiced without some or all of these specific details. In other instances, well-known processing operations have not been described in detail to avoid unnecessarily obscuring this disclosure.

[0025] Game testing is an expensive and time-consuming process. In fact, all games that reach the commercial stage undergo numerous rounds of testing to ensure a certain level of functional and artistic quality. Testing plays many roles in the development and release of video game applications. These include functional testing, compliance testing, compatibility testing, load testing, beta testing, multiplayer testing, and more. Overall, the goal of game testing is to ensure that video games run smoothly on various hardware platforms as expected. Therefore, one of the objectives of game testing is to identify bugs in video games that prevent them from running smoothly. Bugs can manifest in various ways. Typically, a bug refers to a part of the game and its associated code where unexpected results occur, regardless of whether these unexpected results are related to game logic, game mechanics, errors, illustrations, system malfunctions, or other game or system responses that differ from what is expected, anticipated, or anticipated.

[0026] Figure 1 A conceptual diagram illustrating bug reporting and reproduction according to one embodiment is shown. Game tester 101 is shown playing a segment 112 of a video game using a testing device 100, such as a development kit. Game tester 101 discovers three bugs 106, 108, and 110 by playing the video game segment 112. Game tester 101 is then shown providing bug report 102, which is delivered to developer 104. Bug report 102 can be provided via a defect tracking system and can specify the circumstances that caused the bug (e.g., level or map coordinates) and specific steps for reproducing the bug (e.g., a set of control inputs). Developer 104 uses bug report 102 to reproduce bugs 106, 108, and 110 before attempting to fix them.

[0027] Developer 104, who may be an artist, programmer, or game designer, successfully reproduced error 106' in each of three attempts and error 108' in the second of three attempts. Developer 104 was unable to reproduce error 110. The inability to reproduce certain errors (such as error 110) and the ability to reproduce errors intermittently (such as error 108) may be due to many factors associated with the game tester 101's gameplay. For example, game tester 101 and developer 104 may have different seed data or other game state data, or they may have entered control input sequences in very different ways over time. Additionally, there may be system differences associated with game tester 101's testing device 100 (e.g., latency, jitter, CPU clock, GPU clock, memory usage, etc.), which could contribute to the generation of errors 108 and 110. For developer 104, using current methods, these system differences would be difficult to track and mimic the reproduction of errors 108 and 110. Furthermore, it's possible that some game testers report bug 110 while others do not, leading to conflicting perceptions regarding the existence of bug 110 and whether it should be addressed. Improved methods and systems for providing accurate reproduction of bugs are envisioned.

[0028] Figure 2 A conceptual illustration is shown of an improved method and system for faithfully reproducing errors discovered by game tester 101, according to one embodiment. Similar to... Figure 1 , Figure 2 Game tester 101 is shown to have generated errors 106, 108, and 110 during interaction with segment 112 of the video game. Segment 112 may be part of a video game session 201 executed on server system 200. That is, for example, video game session 201 may be hosted by system 204 having test hardware 206 of server system 200. In some implementations, test hardware 206 may be proprietary.

[0029] System 204 is shown running operating system 208 on which the video game application 210 for testing is executed. Game tester 101 interacts with video game session 201 via network 202. For example, video game application 210 is intended to generate audio and video components at server system 200 for delivery and presentation to game tester 101. Server system 200 then receives control input from game tester 101 via network 202 to enable game tester 101 to interact with video game session 201. In other embodiments, video game session 201 may be executed on test device 100 local to game tester 101. However, in any case, a set of raw test data 212 is generated due to game tester 101's interaction with video game application 210 during video game session 201.

[0030] The raw test data 212 is shown to include a control input sequence 214, which can be any suitable data structure having control inputs as data elements that change over time. For example, the control input sequence 214 is intended to record each of various control inputs entered on some input device of the game tester 101, including button presses, keystrokes, mouse clicks, cursor movements, controller movements, joystick movements, voice commands, body movements, and other inputs made by the game tester 101. Each data element in the control input sequence 214 can be timestamped based on the time when the control inputs are recorded by the video game application 201 and the time when they are entered on a controller in the real world. The raw test data 212 is also shown to include game state data 216, which can have any suitable data structure for capturing the game environment, including saved data, game values, map data, asset data, and other data. The game state data 216 is also intended to be timestamped so that errors can be referenced to the game state data 216 over time.

[0031] When game tester 101 interacts with video game application 201 to discover errors 106, 108, and 110, system 204 generates video component 218 as instructed by video game application 210. Video component 218 may include both audio and video components in the form of a video frame sequence. Additionally, as game tester 101 interacts with video game application 210, various network and system properties associated with video game session 201 are captured in real-time or near real-time as part of raw test data 212 in network data 220 and system data 222, respectively. Network data 220 is expected to include various parameters associated with the communication link between server system 200 and game tester 101, including, for example, latency, jitter, data rate capacity, packet loss, quality of service (QoS), throughput, and other network properties. System data 222 is expected to include various parameters associated with system 204 or test device 100 executing video game application 210, including CPU clock, GPU clock, memory usage, frame rate, resolution, and other system properties. Both network data 220 and system data 222 are expected to be timestamped, so that errors can be time-referenced to a specific time within network data 220 and / or system data 222.

[0032] Seed data 224 and error log 226 are also shown as being included in the raw test data 212. Seed data 224 refers to the initial dataset used by the video game application 210 to introduce some variation or randomness into the video game. In some implementations, seed data 224 may utilize a random number generator. Additionally, error log 226 records markers or flags input by the game tester 101 to indicate the presence of errors. For example, when the game tester 101 encounters each of errors 106, 108, and 110, they may be indicated in error log 226. Error log 226 is also intended to be timestamped, such that the time of occurrence of each of errors 106, 108, and 110 can be referenced to a specific window within each of the control input sequence 214, game state data 216, video component 218, network data 220, and system data 222.

[0033] Snapshot generator 228 is configured to generate snapshot files 232 stored in snapshot log 230. Each snapshot file 232 is expected to be associated with a specific error and includes a portion of control input sequence 214, game state data 216, video component 218, network data 220, system data 222, seed data 224, and the error-related portion of error log 226. For example, each of errors 106, 108, and 110 is expected to be included in snapshot file 232 generated by snapshot generator 228. For example, when game tester 101 indicates the presence of error 106, snapshot generator 228 is expected to access windows corresponding to the timestamp of error 106 in error log 226 for each of the control input sequence 214, game state data 216, video component 218, network data 220, system data 222, and seed data 224 to generate snapshot file 232 for error 106. The window may be between approximately 1 minute and 10 minutes before and after error 106 occurs. Snapshot generator 228 is intended to also generate snapshot files 232 for errors 108 and 110 to be stored in snapshot log 230. In addition to snapshot files 232 for errors 106, 108, and 110 discovered or generated by game tester 101, snapshot log 230 may be a database of snapshot files 232 associated with various additional errors discovered or generated by other game testers or automated game testing agents.

[0034] Server system 200 is also intended to host error reproduction module 234, which interfaces with developer 104's device via network 202 to reproduce errors 106', 108', and 110'. Error reproduction module 234 is intended to allow developer 104 to reproduce errors in various ways. Three non-limiting embodiments include a playback mode, an automatic reproduction mode, and a manual reproduction mode. In playback mode, snapshot file 232 is played back to developer 104, such that errors 106', 108', and 110' are played back to developer 104 in the same way they are presented to game tester 101. Therefore, errors 106, 108, and 110 are represented in the same way they are presented to game tester 101.

[0035] In automatic reproduction mode, the error reproduction module 234 is intended to load game state data 216 associated with snapshot file 232 for execution of video game application 210. Error reproduction module 234 can access system 204, test hardware 206, operating system 208, and video game application 210 for execution of snapshot file 232 in automatic reproduction mode. For example, system 204 can load game state data 216 to start video game session 201 approximately 5 seconds to approximately 10 minutes, or approximately 10 seconds to 5 minutes, or approximately 15 seconds to approximately 60 seconds before each of errors 106, 108, and 110 occurs. Once video game session 201 is started, control input sequence 214 is automatically input into video game application 210 based on the timestamp of each control input. Therefore, control input sequence 214 is synchronized with game state data 216, enabling real-time recreation of video game session 201 for developer 104. In other embodiments, the error reproduction module 234 may be equipped with dedicated playback system hardware, an operating system, and a development version of the video game application. In addition to copying the control input sequence 214 during the execution of the video game application 210, network data 220, system data 222, and seed data 224 may also be copied, mirrored, or simulated to recreate the operating conditions during the windows in which errors 106, 108, and 110 occur. In this way, errors 106', 108', and 110' are regenerated for the developer, but these errors may theoretically be identical to those observed by the game tester 101.

[0036] In manual reproduction mode, game state data 216 from snapshot file 232 is loaded by error reproduction module 234 for use in executing video game application 210 in a state between approximately 5 seconds and approximately 10 minutes, or approximately 10 seconds and approximately 5 minutes, or approximately 15 seconds and approximately 60 seconds prior to the occurrence of each of errors 106, 108, and 110 within video game session 201. However, instead of automatically inputting control input sequences 214, a textual or visual representation of the control input sequences 214 is provided to developer 104, allowing the developer to manually recreate errors 106', 108', and 110'. In this way, developer 104 can cause or trigger the manifestation of errors 106', 108', and 110', and can try other configurations of control inputs that may or may not trigger the same errors. Therefore, developer 104 can better understand the parameters that caused the errors. In any case, the error reproduction module enables developer 104 to reproduce errors 106', 108', and 110' identified by game tester 101, allowing developer 104 to fix them.

[0037] Figure 3A conceptual illustration of an error reproduction module 234 according to one embodiment is shown, which is used to reproduce error 106 by processing a snapshot file 232a that captures an error 106 identified by a game tester. Snapshot file 232a “captures” error 106 by including a portion of an error log 226a associated with the time of error 106', a portion of a control input sequence 214a, a portion of game state data 216a, and a portion of a video component 218a. Although not shown, system data, network data, and seed data are also expected to be included in the snapshot file 232a associated with error 106'. In some embodiments, each snapshot file 232a is associated with one instance of an error. For example, if error 106 is captured by snapshot file 232a, then errors 108 and 110 can be captured by corresponding snapshot files. However, when errors are closely related to each other, such as when a first error leads to a second error, or when the first and second errors are very close in time, both the first and second errors can be captured by a single snapshot file.

[0038] In the illustrated implementation, the error reproduction module 234 is intended to execute error playback mode 302 to simply replay the error, execute automatic reproduction mode 304 to automatically reproduce the same instance of the error, or execute manual reproduction mode 306 to instruct the developer to manually reproduce the error. In any of the aforementioned modes, the error reproduction module 234 generates an error reproduction GUI 300 for the developer to interact with. The error reproduction GUI 300 is shown as including multiple panels for display, including a source code 308 panel, a game status 310 panel, a video component 312 panel, and a control input 314 panel. Video component 312 displays error 108 involving a soccer ball crossing a field. If the error reproduction module 234 is in error playback mode 302, the video frames shown in video component 312 are obtained directly from video component 218a, as shown to the game tester. Alternatively, if automatic reproduction mode 304 or manual reproduction mode 306 is used, video component 312 may include newly generated video frames due to the execution of the game application while inputting control input sequence 214a. However, even in automatic reproduction mode 304 or manual reproduction mode 306, the error reproduction GUI 300 can display the generated video frames (e.g., the original frames) intended for display to game testers, allowing developers to view the original frames and the newly generated frames side by side.

[0039] Source code 308 may include video game programming code containing instructions for operating the video game, including code for game mechanics, game logic, asset management, game physics, multiplayer mechanics, and metadata management. The Source Code 308 panel is envisioned as displaying the code being executed during error 108. For example, Source Code 308 may include instructions to the game engine for causing a soccer ball to fall from a height of 5 units. The Source Code 308 panel is intended to scroll to the relevant section of the game code when the error is reproduced.

[0040] Game state 310 is shown as including various game state values ​​associated with soccer. In this document, game state 310 shows that the "Ball Rebound Coefficient" is set to "0" and the "Ball Radius" is set to "6". The game state 310 panel allows developers to view various game state values ​​and game engine parameters that change during error reproduction.

[0041] Additionally, the control input panel 314 allows developers to view control input sequences 214a that may have caused errors. This document provides a virtual representation of the controller used by the game tester, presenting the control input sequence 214a as if it were entered by the game tester. For example, control input sequence 214a shows that the game tester entered the following sequence: “X-Down-Down-Triangle-Left Joystick-Down”. The same sequence can be virtually represented on the virtual controller with the same sequence and timing as control input sequence 214a. For example, the virtual controller could sequentially press the “X” button, press the down button twice, press the triangle button, and tilt the left joystick down.

[0042] Figure 4 A conceptual diagram of a snapshot log 230 comprising multiple snapshot files 232 according to one embodiment is shown. The snapshot log 230 may include snapshot files 232 generated by one or more game testers. Each snapshot file 232 in the snapshot log 230 includes a snapshot file identifier 401, an error identifier 400, a tag 402, game-related data 404, system data 406, and network data 408. The error identifier 400 may be synchronized with a defect tracking system such that errors with the same token are provided with the same error identifier 400. The tag 402 may be annotated by the game tester or developer and may include a description of the error type, the severity of the error, and a descriptor used to classify errors in the snapshot log 230. The game-related data 404 may be represented in a multidimensional space as a function of control inputs, video outputs, audio outputs, and game state data. The game state data itself may be represented in a multidimensional space as a function of various game state values, object coordinates, time coordinates, levels, assets, abilities, seed data, etc.

[0043] In addition, system data 406 includes time-dependent state data of the system executing the game application in which the associated error was identified. System data may include system performance data such as CPU clock and utilization, GPU clock and utilization, memory speed and utilization, frame rate, resolution, video compression codec and ratio, temperature indicators, and other system performance indicators. Network data 408 is expected to include time-dependent indicators of the communication channel between the game tester and the server system hosting the video game application. These indicators may specify latency, data rate capacity, jitter, and other communication characteristics.

[0044] Figure 5 This illustration shows how a cloud-based test server system 500, according to one embodiment, implements machine learning to generate new control inputs statistically associated with leading to previously identified errors or newly identified errors. The server system 500 includes a snapshot database 502, an automated game testing module 504, a machine learning module 506, and an error implementation module 508. Multiple game testers 501 are shown interacting with a video game to identify errors. When an error is identified, player-generated snapshot files 514 associated with those player-generated errors are recorded in the snapshot database 502. The player-generated snapshot files 514, including control inputs 516, are input to the machine learning module 506 via an application programming interface 522. In some embodiments, the player-generated snapshot files 514 can be used as a training dataset 528 or a benchmark truth. A feature extractor 524 analyzes the player-generated snapshot files 514 to determine values ​​in the feature space associated with the generation of a particular error. The feature extractor 524 can be defined as analyzing features associated with control input sequences, the temporal sequence of control inputs, the previous history of control inputs, etc. In addition, feature extractor 524 can analyze features associated with game state values, seed values, system parameters, and network parameters.

[0045] Machine learning module 506 uses machine learning algorithm 526, training dataset 528, and player-generated snapshot file 514 to generate error classification model 532. Error classification model 532 can be used to identify machine-learned control inputs 530 other than player-generated control inputs 516. Machine-learned control inputs 530 are statistically predicted to cause errors. Any suitable machine learning algorithm 526 can be implemented to generate error classification model 532, including but not limited to Bayesian networks, linear regression, decision trees, neural networks, k-means clustering, etc. Error classification model 532 provides statistically relevant combinations, permutations, and time-dependent sequences of control inputs that can be predicted to cause player-generated errors or newly identified errors. Training dataset 528 and player-generated snapshot file 514 can be used to assign labels to each group of incoming control inputs 516. Labels can include "error" or "not an error".

[0046] In various implementations, each set of machine-learned control inputs 530 is implemented by an automated game testing module 504. The automated game testing module 504 is configured to launch multiple game sessions to test whether errors occur in the multiple sets of machine-learned control inputs 514. The automated game testing module 504 includes a GUI module 534, a difference implementation module 536, a video game application 538, a game engine 540, an operating system 542, a game engine distribution module 544, and an error reporting program 546. The GUI module 534 enables developers to interface with the automated game testing module 504 and specify the conditions under which the automated game sessions should run. For example, developers can decide to introduce differences (hereinafter referred to as "differences") in system or network parameters via the difference implementation module 536 when running the automated game sessions. In some implementations, the machine learning module 506 may also provide a set of machine-learned differences that instruct the difference implementation module 536 on how to introduce differences during the automated game sessions. In other implementations, developers may choose to load test the video game application by introducing differences via the difference implementation module. For example, developers can expect to be able to specify parameters such as frame rate, resolution, CPU clock and utilization, GPU clock and utilization, and memory utilization to be simulated during an automated video game session. Additionally, developers can specify network variations, such as latency, jitter, data rate capacity, and other network properties to be simulated during the automated video game session.

[0047] Based on the machine-learned control inputs 530 to be implemented, the differences to be introduced, and the granularity of parameter changes or differences between runs, any suitable number of video game sessions can be launched in parallel. In any case, when an automated game session is run by the automated game testing module 504, the video game application 538 can be loaded with a specific game state immediately preceding the expected error, for example, approximately 5 seconds to approximately 10 minutes, or approximately 10 seconds to approximately 5 minutes, or approximately 30 seconds to approximately 1 minute prior. The machine-learned control inputs 530 are input in a time-dependent manner when the video game application 538 is executed. Additionally, automated game sessions can be executed while differences are implemented via the difference implementation module 536. In this way, multiple automated game sessions can be implemented using a set of machine-learned control inputs 514, where each session tests different differences. For example, a first automated game session might implement a low level of jitter, while a second automated game session might implement a higher level of jitter. A third automated game session might overclock the GPU, while a fourth automated game session might underclock a similar GPU, and so on.

[0048] Furthermore, when an automated game session is executed, the error reporting program 546 can be layered above each game session for detecting the occurrence of errors. The error reporting program 546 can be predefined to automatically detect errors based on game states that indicate errors generated by the automated game session. Additionally, the error reporting program 546 detects errors associated with game mechanics or rendering by using image analysis to identify images or parts of images indicating the errors.

[0049] The results of the automated session (regardless of whether the error is reproduced) are recorded as a machine-generated snapshot file 530. Based on the results of the error reporting program 546, the machine-generated snapshot file 518 is labeled as "error" or "not an error". However, if the results of the error reporting program 546 are uncertain or incomplete, or if the developer disagrees with the label, the developer can also annotate or change the label. The labeled machine-generated snapshot file 518 is then fed into the machine learning module 506 for training or updating the error classification model 532.

[0050] When the misclassification model 532 is updated, an automated game testing module 504 can be used to generate and test a new set of machine-learned control inputs 530. The process of learning via the machine learning module 504, validating (e.g., supervised) via the automated game testing module 504, and refining the misclassification model 532 can be repeated until the confidence level at which the misclassification model 532 classifies the control input as either predicting an error or predicting no error is acceptable to a certain extent. That is, for example, the process of learning, validating, and refining the model can be repeated until the misclassification model 532 can classify the new control input as either predicting an error or not with a 99% confidence interval. Furthermore, through each iteration of learning, validating, and refining, it is conceivable that the machine learning module 506 can generate multiple sets of machine-learned control inputs 530 that are statistically more likely to cause the error. Conversely, the machine learning module 506 is also improved in identifying those sets of control inputs that are predicted not to cause an error.

[0051] Although the machine learning module 506 has so far been described as learning based on control inputs, such as those from the player-generated snapshot file 514, it is also contemplated to learn based on system and network differences. For example, similar to generating machine-learned control inputs 530 based on the error classification model 532, the machine learning module 506 can also generate machine-learned differences 503 that predict errors. Therefore, the machine learning module 506 can detect patterns in system and network parameters related to the occurrence of errors and can generate machine-learned differences 503 for system and network parameters to be tested via the automated game testing module 504. The automated game testing module 504 can be implemented via a difference implementation module 536, such that the system and network conditions specified by the machine-learned differences 503 are simulated, mimicked, or reproduced during the execution of an automated game session. The results of an error reporting program regarding whether an error occurred can be recorded along with the machine-learned control inputs 530 in the machine-generated snapshot file 518. For example, an error reporting program 546 can be used to label each run with a label of "error," "not an error," or "uncertain." The labeler can later resolve automated game sessions marked "uncertain" to "error" or "not an error".

[0052] Similar to the machine-learned control input 530, the results from the machine-learned differences 503 can then be fed back to the machine learning module 506, which is used to update the error classification model 532. The cycle of learning, testing, and updating the error classification model 532 is designed to determine a set of system and network parameters that cause or approximately cause a specific error, or collectively cause or approximately cause a set of errors.

[0053] In various implementations, the machine learning module 506 has the capability to learn, through experience, the generalities of the types of control inputs, system parameters, and network parameters that lead to a particular error or a set of errors. These generalities are represented in the error classification model 532 and can be extrapolated by the error implementation module 508. The error implementation module 508 is used to formalize or represent the generalities within the error classification model 532 as error implementation conditions 510, which include rules 548, categories 550, game scenarios 552, and system scenarios 554. The error implementation module 508 is also shown to include an error implementation library 512 with error implementation control inputs 556 and error implementation system parameters 520. In a sense, the error implementation module 508 is a platform for triggering or implementing a given error.

[0054] Similar to Figure 2 The error reproduction module 234 and error implementation module 508 can replay or reproduce a given error for the developer to review. However, because error implementation module 508 accesses machine learning module 506, snapshot database 502, error implementation conditions 510, and error implementation library 512, it can also show the developer multiple other ways that lead to the same error, as well as certain limitations or boundaries regarding the conditions that cause the error. Error implementation module 508 is further intended to be able to identify and present newly discovered errors to the user. For example, when machine learning module 506 generates machine-learned control inputs 530 and machine-learned differences 503, they can discover previously unrecognized errors, as implemented by automated game testing module 504.

[0055] Error classification model 532, based on model or machine learning algorithm 526, represents the generalities associated with the occurrence of errors in various ways. For example, generalities can be represented by weights, graph connectivity, clustering, distribution, probability, and other relationships. Based on these, rules 548, categories 550, game scenarios 552, and system scenarios 554 can be constructed. Rule 548 defines the set of conditions that lead to a specific error. For example, a rule (e.g., "If X-down-down-triangle is entered when character A is at (x, y, z) on the map, an error will occur") can be included in rule 548. Category 550 defines the categories of conditions that lead to a specific error. For example, categories may limit the scope of control inputs that lead to errors.

[0056] Game Context 552 and System Context 554 further specify rules or categories related to situations in video games, systems, or networks that cause errors. For example, there might be a Game Context 552 where an AI character falls from the world when attempting to jump. Game Context 552 can help identify errors not necessarily caused by player input. Additionally, System Context 554 describes the state of system and network parameters that cause a particular error. For example, System Context 554 could specify that a game session with a wait time exceeding 70 milliseconds, CPU overclocking, and 60 FPS rendering will cause the player to fall from the virtual world. Of course, there may be slippage between each of Rule 548, Category 550, Game Context 552, and System Context 554, making one related to or dependent on another.

[0057] It is also anticipated that the error implementation module 508 includes an error implementation library 512, which includes error implementation control inputs 556 and error implementation system parameters 520, which, when executed, will reproduce various errors. When testing a new build of the video game application 538, the error implementation module 508 can be used to test the new build against existing errors previously identified by testers or machines. In this way, developers can automatically test previously identified errors, without the game tester 501 having to attempt to recreate errors from previous builds.

[0058] Figure 6 A conceptual diagram illustrates how the difference implementation module 536 creates a difference package 606 that is executed together with the automated game session at the automated game testing module 504. Input difference 600 includes differences in the sequence and timing of control inputs sent to and executed by the automated game testing module 504 for the automated game session. For example, sequence 600a could be the same as the control input sequence generated by a player identified by a game tester. Therefore, sequence 600a can be used as a control item expected to cause an error. Sequence 600b differs from sequence 600a in that the triangular input is shifted to an earlier time. Input difference 600 can help establish error implementation conditions 510. For example, if sequence 600b returns the same error as sequence 600a, the timing difference between sequence 600a and sequence 600b can be determined as an uncertain cause of the error. On the other hand, if sequence 600b does not return an error like sequence 600a, then the timing of the input triangle can be determined to be the cause of the error, provided that all other parameters are kept equal between sequences 600a and 600b.

[0059] Additionally, seed data difference 602 is shown as a change to the seed data used to initialize the automated game session. Seed data 602a and seed data 602b differ, for example, in their first and second seeds. Introducing differences in the seed data helps the error implementation module 508 determine whether and how the seed data affects the cause of the error. For example, if seed data 602a and seed data 602b return different error results, it can be inferred that the first or second seed, or both, are causally related to the cause of the error. In the second iteration of the difference implementation module 536, seed data changes can be introduced to isolate seeds that are causally associated with the error.

[0060] Furthermore, the difference implementation module 536 is configured to incorporate differences into system and network parameters in system difference 604 for simulation during an automated test session. In system difference 604, the first column may represent latency, the second column may represent jitter, the third column may represent the CPU clock, and the fourth column may represent the GPU clock. System difference 604 can be used to simulate various real-world game conditions, where players play video games with varying latency characteristics, jitter characteristics, CPU and GPU configurations, and other hardware and network variations. System difference 604 not only helps simulate real-world game conditions but can also indicate whether these conditions are causally related to the occurrence of errors. The difference implementation module 536 combines input difference 600, seed data difference 602, and system difference 604 to generate multiple difference packets. When an automated game session is initiated by the automated game test module 504, the operating system 542 can implement the set of conditions specified by difference packet 606 during the execution of the automated game session.

[0061] Figure 7A conceptual illustration is shown of an automated game testing module 504 executing multiple automated game sessions 700 according to one embodiment. Each of the automated game sessions 700 executes on a system 708 communicating with an operating system 542 that operates a game engine 440. The game engine 540, in turn, executes a video game application 538 for the video game sessions 700. While running the video game session 700, control inputs 702, game states 704, and system parameters 706 are implemented synchronously with the video game application 538. For example, control inputs 702 may be implemented via input jacks of the game engine 540 during the execution of the video game application 538. Control inputs 702 may be machine-learned control inputs 530 or player-generated control inputs 516, as modified by the difference implementation module 536. Alternatively, control inputs 702 may include control inputs from more than one virtual player in a multiplayer game. For example, a player-generated snapshot file 514 may include a sequence of control inputs 516 for each player during testing. Machine learning module 506 is operable to generate machine-learned control inputs 530, which modify control inputs for only one or more players. Therefore, automated game testing module 504 is intended to test errors associated with multiplayer games and scenarios where interactions with a particular type of player lead to mistakes.

[0062] For example, the game state 704 of the seed data can also be implemented via the game engine, allowing the video game application to be initialized with predefined seed data. System parameters 706 (e.g., machine learning differences 503 modified by the difference implementation module 536) can be implemented at the operating system level 542 or the system level 708.

[0063] During each run of the automated game session 700, an error reporting program 546 is executed at the operating system level 542 or the game engine level 540. The error reporting program 546 is configured to automatically monitor and detect the occurrence of predefined errors or previously unidentified errors. Each run of the automated game session 700 generates a corresponding automated game session report 710, which includes a machine-generated snapshot file associated with whether an error occurred. In this way, any number of automated game sessions 700 can be started to better understand the conditions and boundaries leading to the errors. Understanding the conditions and their boundaries enables developers to resolve errors more effectively.

[0064] Game applications (such as video game applications 538) typically require game engines that serve multiple different functionalities. A typical game engine has many sub-components to handle various aspects or characteristics of the game environment and define, for example, how it should behave in response to player input. Some of these sub-components, or those that will be referred to as "game engine nodes," include timing nodes, physics nodes, artificial intelligence (AI) nodes, game logic nodes, game mechanics nodes, rendering nodes (which themselves may be subdivided into multiple nodes), map nodes, animation nodes, asset management nodes, network nodes, communication nodes, control input nodes, chat nodes, and other nodes. Rendering nodes can refer to various rendering functions (e.g., typical functions of graphics APIs like DirectX® or OpenGL®), including camera transitions, projection, clipping, lighting, shading, smoothing, etc. In addition, game engines can provide additional services that benefit the gameplay experience, such as social network interaction nodes, game help nodes, in-game resource rendering nodes, and gameplay sharing or broadcasting nodes.

[0065] In some game engine implementations, the game engine may be served by a single compute node (e.g., a computer, server, console, or other device). In other implementations, the compute node may be a virtual machine deployed by a server for the game engine, as well as other virtual machines deployed to serve other game engines and game applications running there. Thus, each of the game engine sub-components is executed by a compute node (e.g., a server, virtual machine, desktop computer, or console). This one-to-one or many-to-one architecture of the game engine and compute nodes may not provide the performance required by the game engine, nor the efficiency needed for the use of compute nodes, because the game engine may be elastic rather than fixed in terms of its computational needs. That is, the game engine may have different processing, graphics, memory, and networking requirements at any given time than at any other given time within the same game.

[0066] For example, in a massively multiplayer online role-playing game (MMORPG), the computational demands of the associated game engine can depend on the type of action occurring in the game (e.g., whether players interact with each other), the number of players (e.g., currently there are 10 players, but in the near future there will be 100 players), and other factors. When resources are supplied to a compute node of the game engine (e.g., within a server), the associated hardware may have a narrow window in which it operates as efficiently as needed while maintaining the required performance. For example, when there are few players and they do not interact in the MMORPG, the associated compute node may be underutilized, while when there are many players and they interact with each other, the associated compute node may perform poorly, resulting in degraded service quality, lag, rendering quality issues, and errors.

[0067] These considerations are just as important as the context of automated game testing, in which hundreds or thousands of game sessions and associated game engines can be instantiated simultaneously. For example, the computational demands of the game engine for instantiating multiple automated game sessions will vary in a time-dependent manner depending on the game environment and the behavior of objects within it—for instance, gathering resources at one time and fighting AI characters at another. Furthermore, the type of computational demands can vary depending on time or game state. For example, the processing, graphics, memory, and network demands of the game engine might vary between scenarios where a player is taking a free throw, and after missing the throw, the opposing team rebounds and pushes the ball back onto the court. Moreover, automated game sessions can be used to automate multiplayer gameplay by simultaneously implementing control inputs from multiple players during runtime. The presence of multiplayer input also introduces further flexibility in the computational demands of the corresponding game engine, for example, depending on the number of players and their level of interaction. Therefore, an improved game engine architecture is anticipated to execute automated game sessions for the automated game testing module 504.

[0068] Figure 8 A diagram illustrating the architecture of an automated game testing module 504 that utilizes a distributed game engine 800 to execute multiple automated game sessions 700, according to one embodiment, is shown. A video game application 538 executes on a distributed game engine 800 having multiple game engine nodes 800a and a game engine manager 800b. Each game engine node 800a is intended to serve as a subcomponent or part of a subcomponent of the distributed game engine 800 as a whole. In some embodiments, game engine nodes 800a may be predefined based on the functionality they provide to the distributed game engine 800. For example, there may be game engine nodes 800a that are defined or dedicated to performing functions related to each of game logic, game mechanics, game timing, AI, physics, input mapping, scripting, networking, audio, graphics, animation, asset management, and in-game services. For example, for an automated game session 700a, game engine node 1 (GEN 1) may be dedicated to handling, for example, a game logic subcomponent / function, without handling any functions related to graphics or asset management. Meanwhile, Game Engine Node 2 (GEN 2) can be defined to handle AI, with GEN 3 defined for physics, GEN 4 for scripting, GEN 5 for graphics, GEN 6 for scripting, and GEN n for services such as real-time game broadcasting.

[0069] Each game engine node 800a is matched with a compute node within server 802 in rack 804 via game engine manager 800a, depending on the compute requirements of each game engine node 800a. Game engine nodes 800a can communicate with compute nodes via UDP, TCP, or some other protocol. Server 802 and rack 804 can be located in a data center, etc. Game engine manager 800b acts as a bus between game engine nodes 800a and the assigned compute nodes. Game engine manager 800b performs various tasks related to sending and receiving data, buffering, routing, threading, queuing, relaying, splicing, etc. In this way, game engine manager 800b ensures that the operations of each game engine node 800a are performed by the desired compute node, and that the returned values ​​or other results are delivered back to the appropriate game engine node 800a for implementation in video game application 538. Furthermore, game engine manager 800a manages communication between game engine nodes 800a to ensure the proper functioning of the entire video game application.

[0070] like Figure 8 As shown, game engine node 1 (GEN 1) is assigned to compute node 806, which is associated with a hardware CPU on server 1 located in rack 1. For example, the game engine manager may have adopted compute node 806 because of its computational needs for handling game logic.

[0071] Simultaneously, the automated game session 700b also executes on a distributed game engine 800 with multiple game engine nodes GEN 1 to n, each game engine node being assigned to handle a specific function or sub-component of the distributed game engine 800. The same applies to game session 700n. For example, similar to GEN 1 of automated game session 700a, the operations of GEN 1 of automated game session 700b and GEN 1 of automated game session 700n are assigned to compute node 806 of server 1 in rack 1. Therefore, compute node 806 executes the process requested by each of GEN 1, GEN 2, and GEN 3. GEN 2 for each of automated game sessions 700a, 700b, and 700n is assigned to compute node 808, which is a virtual machine VM 1 equipped with CPU 2, GPU 1, and 16 GB of RAM. Game engine manager 800b may have already identified VM1 based on the anticipated needs of GEN 2, which, for example, is used to execute AI. In some implementations, the game engine manager 800b can request the deployment of VM 1 itself, or the automated game testing module 504 can make the request. In some implementations, if the requested operation to be performed by the compute node is substantially similar for each game engine node assigned to the compute node, the compute node can potentially perform the requested operation once and return the result to each game engine node, rather than performing the requested operation once for each game engine node.

[0072] Each automated game session 700's GEN 3 is shown as associated with compute node 812, which is associated with GPU 1, while GEN 4 is associated with compute node 814, which is associated with GPU 2. GPU 1 and GPU 2 may have different configurations. For example, GPU 1 may have more cores, a higher clock speed, more memory, or higher memory bandwidth than GPU 2. In this case, if GEN 3 requires a larger amount of computation or more complex computation, the operation of GEN 3 may be assigned to GPU 1. The operation of GEN 4 may be assigned to GPU 2 because GEN 4 may require less computation or less complex computation.

[0073] GEN 5 is shown as being assigned to compute node 810, which is a virtual machine VM 2 equipped with CPU 3, GPU 2, and 8 GB RAM. GEN 5 may have been assigned to compute node 810 by game engine manager 800b to match the compute requirements of GEN 5. GEN 6 of automated game session 700 is shown as being assigned to containerized compute node 816, which includes containers 1 to N that interface with the host operating system on server 3 of rack 3. Specifically, GEN 6 of automated game session 700a is assigned to container 1, GEN 6 of automated game session 700b is assigned to container 2, and GEN 6 of automated game session 700n is assigned to container N. Each of containers 1 to N is a software and dependency containing unit, for example, the unit includes software for handling the operations of the associated game engine node. Each container runs on a container runtime that interfaces with the operating system of the server. In this sense, the operating system is virtualized for each container, while the hardware is virtualized for the virtual machine.

[0074] Figure 9A and Figure 9B This paper conceptually illustrates how a machine learning module 506, an automated game testing module 504, and an error implementation module 508, according to various implementation schemes, can work together to identify a set of conditions that can cause previously or newly identified errors. Error implementation conditions refer to the range or category of conditions that are causally related to the occurrence of a given error. Conditions refer to control inputs, system parameters, network parameters, and game states that, when these states are present or coexist during the execution of the game application, individually or collectively cause a given error. Therefore, error implementation conditions can be conceptualized in a multidimensional space as the set of all combinations of conditions that cause or may cause errors. In this paper, error 1 implementation condition 900 will refer to the set of combinations of control inputs that cause or may cause error 1 when executed under specific system and network parameters and in certain game states. Machine learning module 506, automated game testing module 504, and error implementation 508 are intended to work together to identify, limit, or approximate the implementation conditions of error 1 to help developers understand the root causes of errors, thereby achieving higher quality and more effective fixes. Error 1 implementation condition 900 can be formalized in error implementation module 508 in terms of rules, categories, and contexts to communicate with developers. For clarity, the following description concerns the machine-learned control inputs used to fill in Error 1 Implementation Condition 900. However, these principles apply with similar force to using machine-learned system and network parameters, as well as game state data, to assemble the rest of the Error 1 Implementation Condition.

[0075] When one or more game testers identify an error, the player-generated control input 902 is recorded in a snapshot file and processed by the machine learning module 506. The machine learning module 506 uses an error classification model to generate machine-learned control input 904, which is then implemented by the automated game testing module 504 to detect the presence of error 1. In this case, a subset of the machine-learned control input 904 leads to error 1. The results are fed back to the machine learning module 504 to update the error classification model and generate a new set of machine-learned control inputs 906. In this case, as can be expected, a larger proportion of the second set of machine-learned control inputs 906 will lead to error 1 compared to the first set of machine-learned control inputs 904. The learning and testing process is repeated for the machine-learned control inputs 908, and so on, until, for example, no new set of control inputs leading to the error is found. The error 1 implementation condition 900 is then processed by the error implementation module 508 to extract any rules, categories, or situations attributable to the error 1 implementation condition 900.

[0076] exist Figure 9B During the learning and testing process, a new error 2 is discovered during the analysis of error 1 fulfillment condition 900. For example, the machine-learned control input 904 is tested by the automated game testing module 504 and is found to trigger both error 1 and error 2. In this case, in addition to or in parallel with the aggregation of error 1 fulfillment condition 900, the machine learning module 506, the automated game testing module 504, and the error fulfillment module 508 can also perform the same operation on error 2 fulfillment condition 901. The machine-learned control inputs 906 to 914 are generated by the machine learning module 506 and tested by the automated game testing module 904. The resulting range of error 2 fulfillment condition 901 is then formalized by the error fulfillment module 508 into rules, categories, and contexts, which will have those rules, categories, and contexts for both error 1 and error 2.

[0077] Figure 10A method for discovering new errors through chaotic or shotgun testing according to one embodiment is illustrated. In addition to defining error implementation conditions for known or player-identified errors, the systems and methods presented herein are also contemplated for automatically detecting unidentified errors. In some embodiments, chaotic testing or shotgun testing is provided to discover undiscovered errors. In this document, a chaotic testing module 1001 generates multiple sets of chaotic inputs 1002a to 1002g. Each of the chaotic inputs 1002a to 1002g may include a set of inputs defined randomly, evenly, or chaotically. In other embodiments, the chaotic inputs 1002a to 1002g may be known or known sequences of inputs designed to enhance a game application. In any case, among the chaotic inputs 1002a to 1002g, chaotic input 1002c is shown as causing error 3 as detected by the automated game testing module 504. The automated game testing module 504 relays the results to the machine learning module 506, which generates more targeted machine-learned inputs 1004a and 1004b. The learning and testing process continues until the range of error 3 implementation condition 1000 can be defined. In some implementations, the chaos testing module 1001 can continue to generate chaotic inputs to find other ways that cause error 3 or other errors. Therefore, the system approach proposed herein further enables the automatic identification of previously unknown errors. The process of using the chaos testing module 1001, machine learning module 506, automated game testing module 504, and error implementation module 508 to identify previously unknown errors can be provided to developers as a service (e.g., Test as a Service, TaaS), as discussed in more detail below.

[0078] Figure 11 A conceptual diagram illustrating automated testing for previously identified bugs in a new build or version of a game is shown. It is assumed that the circles representing bug implementation conditions 1100 indicate bug implementation conditions 1100 in the control inputs, game state, system, and data parameter spaces of a first build. Further, it is assumed that the bug implementation module has defined four categories of control inputs 1101a to 1101d, which effectively overlap with most bug implementation conditions 1100 and trigger bugs in the first build. If each of the control inputs 1101a to 1101d again causes a bug in the second build, it is possible that the bug has not been fixed. Alternatively, if only a portion of the control inputs 1101a to 1101d cause the bug, it is possible that the bug has been partially fixed. Otherwise, if none of the control inputs 1101a to 1101d cause the bug when implemented in the second build, it is possible that the bug has been fixed. Therefore, when the second build is constructed, previous bugs from previous builds can be automatically tested in a fast and efficient manner.

[0079] Figure 12 This illustrates how, according to one implementation, as part of a Test-as-a-Service (TaaS) approach, known bug implementation conditions for a first game can be applied to a second game. Figure 12 In the first game, the error implementation condition 1100 is known and can be reproduced using control inputs 1101a to 1101d. As part of the test-as-a-service process, control inputs 1101a to 1101d can be applied to a second game title, which may be created by a different developer than the first game title. Figure 12 In this context, control inputs 1101a to 1101c are shown to partially or completely cause the error, while control input 1101d is shown to be outside the error implementation conditions of Game Title 2 and therefore does not cause the error.

[0080] Figure 13 This is a conceptual illustration of a platform that can be used for Test as a Service (TaaS) according to one implementation. TaaS is intended to enable game developers to test bugs in video games and verify bug fixes without requiring a team of human game testers. In this document, developer 1300 simply drags and drops the game onto TaaS server 1303, which includes a first internal version of the game title 1302. Game title 1302 is executed as a video game application 538 running on a suitable game engine 540. Game engine 540 can be... Figure 8 The distributed game engine is implemented as shown. A suitable operating system 540 is provided for the game engine 540. If it is intended to run the game title 1302 on different game engines, different operating systems, or their versions, the automated game testing module 504 can launch automated video game sessions of the video game application 538 correspondingly for each type of game engine 540 and operating system 542 and their required combinations. The automated game sessions can run in parallel or otherwise.

[0081] The game title 1302 can be tested in many ways by the automated game testing module 504. For example, TaaS can provide chaos testing for the video game application 538 via the chaos testing module 1001. Any errors identified by the chaos testing will be recorded in the snapshot database 502. Furthermore, if the developer 1300 has previously identified an error and wants to understand what caused it, they can interact with the video game application 538 during execution on TaaS to manually induce the error. As a result of this interaction, a snapshot file will be generated, and the control input that caused the error will be recorded in the snapshot database 502. The machine learning module 506 can then generate new machine-learned control inputs 530 and machine-learned differences 503 that also cause errors.

[0082] Furthermore, the video game application can be tested against the error implementation conditions 510, error implementation control inputs 556, and error implementation system parameters 520 of the error implementation module 508. The error implementation module 508 can provide various control inputs, as well as system and network parameters, for different video games with similar characteristics to game title 1302 or for different games with a common game engine, that are known to cause errors in previous internal versions of that video game. Additionally, the TaaS server 1303 can communicate with one or more human game testers to discover errors, the raw test data of which has been generated as snapshot files at snapshot database 502.

[0083] Any errors identified in the manner described above, along with the control input 1302 that caused the error (and system and network parameters), are recorded in the snapshot database. These will be shared with the developer 1300. Furthermore, these errors can be reproduced for the developer 1300 via the error reproduction module 234 in any of the following modes: playback mode 302, automatic reproduction mode 304, or manual reproduction mode 306. In this way, the developer can view any discovered errors, along with the associated source code, game state, control inputs, and video components, in a GUI such as the error reproduction GUI 300.

[0084] Machine learning module 506 accesses snapshot database 502 to generate machine-learned control inputs 530 and machine-learned differences 503 that predict whether previously identified errors or new errors will occur. These machine-learned control inputs 530 and machine-learned differences 503 are then tested in automated game testing module 504 for error reporting program 546 to detect whether previously identified or unidentified errors have occurred. The results of automated game testing module 504 are again recorded in snapshot database 502. Again, the machine-learned control inputs 530 and machine-learned differences 503 can be provided to developer 1300, allowing developers to be informed of additional control inputs leading to errors, as well as system and network parameters. Furthermore, developers may be informed of new errors and the control inputs, system and network parameters that led to them.

[0085] The learning and testing process can be repeated to refine the error classification model 532, allowing the error implementation module 508 to extract generalities associated with the cause of the error via rule 548, category 550, game context 552, and system context 554. The error implementation conditions 510 can then be provided to the developer 1300, which helps identify the conditions leading to the error. This allows the developer 1300 to better understand the root cause of the error, enabling the application of higher-quality fixes. After the developer 1300 attempts to fix the error in the second build of game title 1304, game title 1304 can be dragged and dropped again into the TaaS server 1303 for verification and testing. To see if the error has been fixed, certain control inputs 1302 known to have caused errors in the previous build (i.e., the first build) can be tested on the automated game testing module 504. Additional control inputs and system parameters can be generated from the error implementation module 508, which, while not necessarily tested to cause errors in the previous build, are expected to have the use of rule 548, category 550, game context 552, and system context 554. In this way, developers 1300 are able to quickly test the operation of new internal versions with bug fixes.

[0086] Figure 14 Components of an example device 1400, such as one of server system 200, server system 500, server 802, or TaaS server 1300, are shown for use in carrying out various embodiments of the present disclosure. The block diagram illustrates device 1400, which may be combined with or may be a personal computer, video game console, personal digital assistant, server, or other digital device suitable for practicing embodiments of the present disclosure. Device 1400 includes a central processing unit (CPU) 1402 for running software applications and optionally an operating system. CPU 1402 may consist of one or more homogeneous or heterogeneous processing cores. For example, CPU 1402 is one or more general-purpose microprocessors having one or more processing cores. Other embodiments may also be implemented using one or more CPUs with microprocessor architectures particularly suited to highly parallel and computationally intensive applications, such as automated game testing, machine learning operations, and error reproduction processes. Device 1400 may be local to the player playing a segment of the game (e.g., a game console) or remotely to the player (e.g., a back-end server processor).

[0087] Memory 1404 stores applications and data used by CPU 1402. Storage device 1406 provides non-volatile storage for applications and data and other computer-readable media, and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROMs, DVD-ROMs, Blu-ray discs, HD-DVDs, or other optical storage devices, as well as signal transmission and storage media. User input device 1408 conveys user input from one or more users to device 1400. Examples of such devices may include a keyboard, mouse, joystick, touchpad, touchscreen, still or video recorder / camera, tracking device for recognizing gestures, and / or microphone. Network interface 1410 allows device 1400 to communicate with other computer systems via electronic communication networks and may include wired or wireless communication over local area networks and wide area networks such as the Internet. Audio processor 1412 is adapted to generate analog or digital audio output from instructions and / or data provided by CPU 1402, memory 1404, and / or storage device 1406. The components of the device 1400, including a CPU 1402, a memory 1404, a data storage device 1406, a user input device 1408, a network interface 1410, and an audio processor 1412, are connected via one or more data buses 1422.

[0088] The graphics subsystem 1420 is further connected to the data bus 1422 and components of the device 1400. The graphics subsystem 1420 includes a graphics processing unit (GPU) 1416 and a graphics memory 1418. The graphics memory 1418 includes display memory (e.g., a frame buffer) for storing pixel data for each pixel of the output image. The graphics memory 1418 may be integrated into the same device as the GPU 1416, connected to the GPU 1416 as a separate device, and / or implemented within memory 1404. Pixel data may be provided directly from the CPU 1402 to the graphics memory 1418. Alternatively, the CPU 1402 may provide the GPU 1416 with data and / or instructions defining the desired output image, and the GPU 1416 may generate pixel data for one or more output images based on the data and / or instructions. The data and / or instructions defining the desired output image may be stored in memory 1404 and / or graphics memory 1418. In one implementation, GPU 1416 includes 3D rendering capabilities for generating pixel data for an output image based on instructions and data that define the geometry, lighting, shading, texturing, motion, and / or camera parameters of a scene. GPU 1416 may also include one or more programmable execution units capable of executing shader programs.

[0089] The graphics subsystem 1420 periodically outputs pixel data of an image from the graphics memory 1418 for display on the display device 1414. The display device 1414 can be any device capable of displaying visual information in response to signals from the device 1400, including CRT, LCD, plasma, and OLED displays. The device 1400 can provide, for example, analog or digital signals to the display device 1414.

[0090] It should be noted that access services delivered over vast geographical areas (such as providing access to games in current implementations) often utilize cloud computing. Cloud computing is a computing paradigm in which dynamically scalable and often virtualized resources are provided as a service over the internet. Users do not need to be experts in the technical infrastructure supporting their “cloud.” Cloud computing can be categorized into different services, such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). Cloud computing services typically provide commonly used applications (such as video games) online and accessible from a web browser, while the software and data are stored on servers in the cloud. Based on how the internet is depicted in computer network diagrams, the term cloud is used as a metaphor for the internet and is an abstract concept of the complex infrastructure it conceals.

[0091] Most video games played on the internet operate via a connection to a game server. Typically, the game uses a dedicated server application that collects data from players and distributes it to other players. Users access the remote service using client devices, which include at least a CPU, display, and I / O. Client devices can be PCs, mobile phones, netbooks, PDAs, etc. In one implementation, the network, operating on the game server, identifies the type of device used by the client and adjusts the communication method accordingly. In other cases, the client device uses standard communication methods (e.g., HTML) to access the application on the game server over the internet.

[0092] The embodiments of this disclosure can be practiced with various computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and so on. This disclosure can also be practiced in distributed computing environments, where tasks are performed by remote processing devices linked via wired or wireless networks.

[0093] It should be understood that a given video game or game application can be developed for a specific platform and a specific associated controller device. However, when such games are made available through the game cloud system presented herein, users may be using different controller devices to access the video game. For example, a game may have been developed for a game console and its associated controllers, while the user may be using a keyboard and mouse to access a cloud-based version of the game from a personal computer. In this case, input parameter configuration can define a mapping from inputs generated by the user's available controller device (in this case, a keyboard and mouse) to inputs acceptable for executing the video game.

[0094] In another example, a user can access the cloud gaming system via a tablet computing device, a touchscreen smartphone, or other touchscreen-driven device. In this case, the client device and controller device are integrated into the same device, where input is provided through detected touchscreen inputs / gestures. For such a device, input parameter configuration can define specific touchscreen inputs corresponding to the game inputs of the video game. For example, during the operation of the video game, buttons, steering wheels, or other types of input elements may be displayed or covered to indicate the location on the touchscreen that the user can touch to generate game inputs. Gestures such as swipes in a specific direction or specific touch actions can also be detected as game inputs. In one implementation, the user may be provided with instructions on how to provide inputs via the touchscreen to play the game before starting to play the video game, in order to familiarize the user with operating controls on the touchscreen.

[0095] In some implementations, the client device acts as a connection point for the controller device. That is, the controller device communicates with the client device via a wireless or wired connection to transmit input from the controller device to the client device. The client device can then process this input and transmit the input data to the cloud gaming server via a network (e.g., via a local networked device such as a router). However, in other implementations, the controller itself can be a networked device with the ability to transmit input directly to the cloud gaming server via the network, without first transmitting such input through the client device. For example, the controller can connect to a local networked device (e.g., the router mentioned above) to send and receive data from the cloud gaming server. Therefore, although the client device may still need to receive the video output from the cloud-based video game and render it on a local display, input latency can be reduced by allowing the controller to send input directly to the cloud gaming server via the network, thus bypassing the client device.

[0096] In one implementation, the networked controller and client device can be configured to send certain types of input directly from the controller to the cloud gaming server, and other types of input via the client device. For example, input whose detection does not rely on any additional hardware or processing outside the controller itself can be sent directly from the controller to the cloud gaming server via the network, bypassing the client device. Such inputs may include button inputs, joystick inputs, embedded motion detection inputs (e.g., accelerometers, magnetometers, gyroscopes), etc. However, inputs utilizing additional hardware or requiring processing by the client device can be sent to the cloud gaming server by the client device. These may include video or audio captured from the game environment, which can be processed by the client device before being sent to the cloud gaming server. Additionally, input from the controller's motion detection hardware can be processed by the client device in conjunction with the captured video to detect the controller's position and movement, which the client device then transmits to the cloud gaming server. It should be understood that the controller device according to various embodiments can also receive data (e.g., feedback data) from the client device or directly from the cloud gaming server.

[0097] It should be understood that the various implementation schemes defined herein can be combined or assembled into specific implementations using the various features disclosed herein. Therefore, the examples provided are merely possible examples and not a limitation on the various implementations possible by combining various elements to define more implementations. In some examples, some implementations may include fewer elements without departing from the spirit of the disclosed or equivalent implementations.

[0098] The embodiments of this disclosure can be practiced with various computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and so on. The embodiments of this disclosure can also be practiced in distributed computing environments where tasks are performed via remote processing devices based on wired or wireless network links.

[0099] Although the method operations are described in a specific order, it should be understood that other housekeeping operations may be performed between operations, or operations may be adjusted so that they occur at slightly different times, or operations may be distributed throughout the system. As long as the processing of telemetry and game state data for generating the modified game state is performed in the desired manner, the system allows processing operations to occur at various intervals associated with the processing.

[0100] One or more embodiments can also be made into computer-readable code on a computer-readable medium. A computer-readable medium is any data storage device capable of storing data that can subsequently be read by a computer system. Examples of computer-readable media include hard disk drives, network-attached storage devices (NAS), read-only memory, random access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tape, and other optical and non-optical data storage devices. The computer-readable medium may include computer-readable tangible media distributed across network-coupled computer systems, enabling the distributed storage and execution of computer-readable code.

[0101] Although the foregoing embodiments have been described in slightly more detail for the purpose of clarity, it will be apparent that certain variations and modifications may be practiced within the scope of the appended claims. Therefore, these embodiments are to be considered illustrative rather than restrictive, and are not limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.

Claims

1. A method for executing Test as a Service (TaaS), comprising: The server receives a game application from the client for testing one or more bugs. The automatic game testing module executes multiple automatic sessions of the game application and implements corresponding test inputs for the multiple automatic sessions. The corresponding test inputs include control inputs, game status data, system parameters, and network parameters. During multiple automated sessions of the game application, errors are detected by an error reporting program; and Generate a snapshot file, which includes a portion of the control input, the game state data, and a video component associated with the occurrence of the error.

2. The method according to claim 1, further comprising: At the error reproduction module, the portion of the control input, the game state data, and the video component associated with the occurrence of the error are processed to reproduce the error, and the reproduction of the error is configured to be sent for display by the client.

3. The method according to claim 1, further comprising: The snapshot file is processed using a machine learning module to generate multiple machine-learned control inputs, which are in addition to the portion of the control input used to reproduce the error; While inputting the corresponding machine-learned control input, the automatic game testing module is used to execute a second or more automatic sessions to generate corresponding game state data and corresponding video components, wherein the corresponding machine-learned control input, the corresponding game state data, and the corresponding video components are recorded in the corresponding machine-generated snapshot file; as well as The machine learning module is used to process the corresponding machine-generated snapshot file to identify error implementation conditions that can be used to identify additional control inputs that lead to the error, wherein the error implementation conditions are configured to be delivered to the client.

4. The method according to claim 1, wherein, The corresponding test inputs for the multiple automated sessions of the game application include a sequence of control inputs defined by a chaos testing module, or by a load testing module, or by the client, or by a previous snapshot file associated with a previous occurrence of the error identified when testing a previous version of the game application or when testing a different game application.

5. The method according to claim 1, wherein, The multiple automated sessions executing the game application also include introducing differences in the system and network parameters to reproduce the error.

6. A computer-readable medium having program instructions for performing a Test as a Service (TaaS), the program instructions comprising: Used to receive program instructions from the client at the server level for testing one or more bugs in a game application; The program instructions are used to execute multiple automatic sessions of the game application by the automatic game testing module, and to implement corresponding test inputs for the multiple automatic sessions. The corresponding test inputs include control inputs, game state data, system parameters and network parameters. Program instructions for detecting the occurrence of errors by an error reporting program during multiple automatic sessions of the game application being executed; as well as Program instructions for generating a snapshot file, the snapshot file including a portion of the control input, the game state data, and a video component associated with the occurrence of the error.

7. The computer-readable medium of claim 6, further comprising: The program instructions for reproducing the error are configured to process, at the error reproduction module, the portion of the control input, the game state data, and the video component associated with the occurrence of the error to reproduce the error, the reproduction of which is configured to be sent for display by the client.

8. The computer-readable medium of claim 6, further comprising: Program instructions for using a machine learning module to process the snapshot file to generate multiple machine-learned control inputs, the multiple machine-learned control inputs being in addition to the portion of the control input used to reproduce the error; Program instructions for using the automatic game testing module to execute a second plurality of automatic sessions while inputting corresponding machine-learned control inputs, to generate corresponding game state data and corresponding video components, wherein the corresponding machine-learned control inputs, the corresponding game state data and the corresponding video components are recorded in corresponding machine-generated snapshot files; as well as The machine learning module is used to process the corresponding machine-generated snapshot file to identify program instructions that can be used to identify error implementation conditions of additional control inputs that cause the error, wherein the error implementation conditions are configured to be delivered to the client.

9. The computer-readable medium according to claim 6, wherein, The corresponding test inputs for the multiple automated sessions of the game application include a sequence of control inputs defined by a chaos testing module, or by a load testing module, or by the client, or by a previous snapshot file associated with a previous occurrence of the error identified when testing a previous version of the game application or when testing a different game application.

10. The computer-readable medium according to claim 6, wherein, The execution of the multiple automatic sessions also includes introducing differences into the system parameters and network parameters to reproduce the error.

Images