Efficient game play training directed to artificial intelligence

Abstract

To improve testing method of a game.SOLUTION: A game play data model of a game application is provided from at least one server computing system executing a remote learning service to a client computing device. At least partially based on inference in which an actor component is generated by using a provided game play data model and based on an in-game result of artificial game play action executed by a local actor component, observation data are generated by a local actor component. Based on received observation data, a remote learning service corrects a game play data model, and provides the corrected game play data model to a local actor component so as to improve future artificial game play action.SELECTED DRAWING: Figure 4

Description

[Background technology]

[0001] background Consumer-grade Graphics Processing Units (GPUs), widespread broadband availability, and market forces have combined to produce games of considerable scope and complexity. Recent games are not only more complex than their predecessors, but also reflect fundamental changes in the way games are designed and played: simple linear indoor hierarchies have been replaced by vast, nearly photorealistic outdoor spaces, scripted sequences have been replaced by dynamic simulations, and proceduralism has enabled worlds of nearly infinite variety. Summary of the Invention [Problem to be solved by the invention]

[0002] Despite this dramatic shift in how games are played, the way games are tested has remained largely unchanged. Games are essentially simulations with complex and emergent interactions between systems within high-dimensional state spaces. This limits the usefulness of code-focused methods like unit testing. As a result, game testing is overwhelmingly a manual process, relying heavily on humans repeatedly playing games looking for defects. Unfortunately, the size of these teams can no longer keep up with the complexity of modern games, leading to delayed releases and poor product quality. [Means for solving the problem]

[0003] Overview of the embodiment Described herein are embodiments in which locally executed actor components are trained to perform real-time gameplay actions in a gaming application based on one or more gameplay data models generated by a remote learning service. The gameplay data models of the gaming application are provided to the client computing devices from one or more server computing systems executing the remote learning service. Observation data is generated by the local actor component based on in-game outcomes of artificial gameplay actions performed by the local actor component based at least in part on inferences generated by the actor component using the provided gameplay data model. Based on the received observation data, the remote learning service modifies the gameplay data model and provides the modified gameplay data model to the local actor component to improve future artificial gameplay actions. Modifying the gameplay data model based on the observation data may include, among other things, updating (e.g., in real-time) the gameplay data model with observation data generated locally by and received from the remote client computing device.

[0004] In certain embodiments, a method includes providing, from one or more server computing systems to a remote client computing device via a programmatic interface, a gameplay data model of a game application executing on the remote client computing device; receiving, from the remote client computing device via the programmatic interface, observational data generated from artificial gameplay actions performed by the actor component within the game application based at least in part on inferences generated by the actor component executing on the remote client computing device using the provided gameplay data model; and transmitting, from the one or more server computing systems to the remote client computing device, observational data generated from artificial gameplay actions performed by the actor component within the game application based at least in part on inferences generated by the actor component executing on the remote client computing device using the provided gameplay data model. The system may include modifying the gameplay data model based on the received observational data and providing the modified gameplay data model to a remote client computing device via a programmatic interface.

[0005] The method may further include the one or more server computing systems receiving, via the programmatic interface, control information mapping each of one or more output states of the game application to input variables of the actor component executing on the remote client computing device. The one or more output states of the game application may include one or more of the following group: a player reference position within the virtual environment of the game application, a position of an object relative to the player reference position within the virtual environment of the game application, a motion vector associated with the object relative to the player reference position within the virtual environment of the game application, geometry information regarding one or more aspects of the virtual environment of the game application, and / or one or more in-game reward indicators associated with gameplay of the game application.

[0006] The method may further include the one or more server computing systems receiving, via the programmatic interface, control information mapping each of the one or more input variables of the actor component to actions available to a human user of the game application.

[0007] Modifying the gameplay data model may be further based on additional observational data generated based on gameplay actions performed within the game application by a human user of the game application.

[0008] Modifying the gameplay data model based on the received additional observational data includes modifying the gameplay data model using deep learning artificial intelligence. The method may further include generating test data for the game application based on the artificial gameplay actions.

[0009] The method may further include aggregating the observational data at the remote client computing device before transmitting the observational data to the one or more server computing systems, e.g., in the form of batches of observational data. This may reduce data traffic in communications between the one or more server computing systems and the remote client computing device. Modifying the game play data model based on the received observational data may then be performed in response to the aggregation of the observational data satisfying at least one predefined criterion. The at least one criterion may include, e.g., at least one of a predetermined period of time, a predetermined amount of observational data (e.g., measured in bytes or other units of quantity), and an explicit request received at the remote client computing device, e.g., from the one or more server computing systems.

[0010] In particular embodiments, the server may include a network interface, one or more processors, and a memory storing a set of executable instructions that, when executed by the one or more processors, operate the one or more processors to generate a gameplay data model for the game application based at least in part on control information mapping input variables to each of one or more output states of the game application, provide the generated gameplay data model via a programmatic interface to an actor component executing on a remote client computing device, and cause the actor component to execute an action within the game application based on inferences generated by the actor component using the generated gameplay data model. The game application may further include having the actor component receive, via the programmatic interface, observation data generated from artificial game play actions performed by the game client, modify the generated game play data model based on the received observation data, and provide the modified game play data model to the actor component via the programmatic interface for use by the actor component in performing additional artificial game play actions within the game application.

[0011] The remote client computing device may run an instance of a game application, and the observation data is generated from artificial game play actions performed by actor components within the instance of the game application run by the remote client computing device.

[0012] The set of executable instructions may further operate the one or more processors to receive, via the programmatic interface, control information mapping each of one or more output states of the game application to input variables of an actor component executing on a remote client computing device. The one or more output states of the game application may include one or more of the following group: a player reference position within the virtual environment of the game application, a position of an object relative to the player reference position within the virtual environment of the game application, a motion vector associated with the object relative to the player reference position within the virtual environment of the game application, geometry information regarding one or more aspects of the virtual environment of the game application, and / or one or more in-game reward indicators associated with gameplay of the game application.

[0013] The set of executable instructions may further operate the one or more processors to receive, via the programmatic interface, control information mapping each of the one or more input variables of the actor component to actions available to a human user of the game application.

[0014] The set of executable instructions may further operate the one or more processors to receive, via the programmatic interface, additional observational data generated from gameplay actions performed within the game application by a human user of the game application, wherein modifying the gameplay data model is further based on the received additional observational data.

[0015] Modifying the gameplay data model based on the received additional observational data may include modifying the gameplay data model using deep learning artificial intelligence.

[0016] In particular embodiments, a client method may include an actor component executed by one or more processors receiving a gameplay data model of the game application from a machine learning component executing on one or more remote server computing systems via a programmatic interface; the one or more processors executing an instance of the game application; providing to the machine learning component via the programmatic interface observation data generated from artificial gameplay actions performed by the actor component within the executing instance of the game application based at least in part on inferences generated by the actor component using the gameplay data model; and receiving a modified gameplay data model based at least in part on the provided observation data from the machine learning component executing on the one or more remote server computing systems via the programmatic interface.

[0017] The client method may further include performing one or more additional artificial gameplay actions based at least in part on the additional inferences generated by the actor component using the modified gameplay data model.

[0018] The client method may further include generating test data for the game application based on the artificial game play actions.

[0019] The gameplay data model may be based at least in part on control information mapping each of one or more output states of the game application to input variables of the actor component. The one or more output states of the game application may include one or more of the following group: a player reference position within the virtual environment of the game application, a position of an object relative to the player reference position within the virtual environment of the game application, a motion vector associated with the object relative to the player reference position within the virtual environment of the game application, geometry information regarding one or more aspects of the virtual environment of the game application, and / or one or more in-game reward indicators associated with gameplay of the game application.

[0020] The gameplay data model may be based at least in part on control information that maps each of one or more output variables of an actor component to actions available to a human user of the game application.

[0021] The client method may further include generating additional observational data generated from gameplay actions performed within the game application by a human user of the game application such that the modified gameplay data model is further based on the additional observational data.

[0022] The present disclosure may be understood, and its numerous features and advantages made apparent to those skilled in the art by reference to the accompanying drawings, in which: The use of the same reference symbols in different drawings indicates similar or identical items. [Brief description of the drawings]

[0023] [Figure 1] 1 illustrates an exemplary networked game training system according to some embodiments. [Diagram 2] 1 illustrates another exemplary networked game training system according to some embodiments. [Diagram 3] 1 shows a schematic block diagram of a Gameplay Trainer (GT) system implemented in accordance with one or more embodiments. [Figure 4] FIG. 2 is a block flow diagram outlining an operational routine of the GT system according to one or more embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Detailed Description Embodiments of the technology described herein enable game application developers (also referred to as "game developers") to utilize AI (artificial intelligence) to train executable actor components that can play and test one or more game applications (e.g., video games or other simulations). For ease of reference, such technology, various embodiments thereof, may be referred to herein as a Gameplay Trainer (GT) system, which includes a software development kit (SDK) that a game developer may link to a game application, and the software development kit (SDK) that the game developer may use to train gameplay models associated with a particular game application. A game may utilize both a gameplay and a remote learning service. Thus, at a high level, the interaction between the game and the GT system would be similar to the interaction between a game and a human player. The game generates outputs that are sent to the GT system. The GT system evaluates how to respond to those outputs, returning artificial gameplay actions that the GT system would like to perform. The game then applies those actions and generates new outputs. The cycle repeats.

[0025] In certain embodiments, the Gameplay Trainer provides game developers with a remote learning service that works in conjunction with a locally running AI (artificial intelligence) actor component to play and test each of one or more game applications with a gameplay data model. In at least some embodiments, the gameplay data model used by the locally running actor component is generated by the GT system (via the GT learning service) based on observational data collected from artificial gameplay actions performed by the actor component within the game application. Thus, the GT system provides a solution tailored to various objectives of game development, including cost sensitivity, predictability, and ease of integration. Certain embodiments of the GT system provide a solution thereby enabling game developers to quickly integrate the GT system into game applications to generate useful gameplay data models.

[0026] In certain embodiments, such as to support popular frameworks and / or utilize common reference terms, the GT system may provide one or more application programming interfaces (APIs, as used herein, may refer to application programming interfaces or other suitable programmatic interfaces). In various embodiments, such APIs may be provided to allow game application developers to incorporate the use of platform-specific SDKs (software development kits) into their game applications.

[0027] The Gameplay Trainer system provides game developers with a solution that is useful, flexible, trainable, and allows progress toward goals in a game application beyond simply "winning." As one non-limiting example, a gameplay data model for a game application may enable the actor component of the GT system to determine one or more areas of the game world where a human player of the game application is likely unable to progress in the game world, i.e., where the human player is likely to become "stuck." As another non-limiting example, a gameplay data model may enable the actor component of the GT system to determine that one or more game world enemies are inappropriately empowered for their status in the game world, e.g., they are either stronger or weaker than they would be worthy of encountering in a role-playing game with a low or mid-level character. Thus, in certain embodiments, GT may emphasize the rapid development of multiple gameplay data models for a game application, each associated with one or more different goals in the game world. Alternatively, the GT system may develop one gameplay data model that includes or otherwise progresses toward multiple goals, such as goals identified via one or more user-specified parameters provided to the GT system and / or goals identified by the GT system itself.

[0028] Although the techniques and embodiments are described herein with respect to gaming applications, it will be appreciated that other embodiments may be utilized in conjunction with various simulation scenarios to generate behavioral data models and / or autonomous agents (e.g., self-driving cars, autonomous robotics, etc.) in other contexts. Although techniques and embodiments are described with respect to game application testing, such behavioral data models and / or automated agents in other contexts may be generated and / or utilized (e.g., automated playing agents or "bots" development, in-game autonomous companion characters, development of generative automated agents for playing with human players, etc.).

[0029] Embodiments of the GT system may provide various advantages to one or more developers who wish to test one or more aspects of their game applications. As one example, a developer may benefit from being able to remotely test a running game. In certain embodiments, the GT actor component runs locally with respect to a running instance of the game application that the GT system is testing. However, the GT system enables such testing via a connection to the game that traverses the public Internet, which implies bandwidth and latency limitations.

[0030] As another example, the GT system may reduce the computing resources utilized for such testing, which is typically a huge sacrifice for game developers. While the techniques described herein enable testing scaled to large numbers of individual instances of game applications, such techniques also enable the GT system to effectively train useful AI from a single instance.

[0031] As another example, the GT system may enable testing of one or more game applications through native support for one or more associated game engines (typically large code bases providing common underlying services such as game physics and graphical rendering). In various embodiments, one or more SDKs associated with the GT system may be provided for each of a variety of such game engines (e.g., Unity, Unreal, and pure C++) and compiled libraries of shared logic.

[0032] In certain embodiments, the GT system may generate an initial gameplay model of a game application based on one or more parameters defined for the game application. In general, the defined parameters typically provide three types of information to the GT actor components: observations (the game states that a player typically experiences), actions (logical interactions that a player may make in the game), and rewards (indications of how well or poorly the GT actor components are performing).

[0033] 1 illustrates an exemplary embodiment of a networked game training system 100. The networked game training system includes an instance of a GT actor component 105 running on a local computing system 110 that also runs a game application 115 that receives game play actions 128 from the GT actor component. The GT actor component 105 is communicatively coupled to a GT learning service 120 running on one or more remote servers 125 via one or more computer networks 101, such as the Internet or other intervening networks. In the illustrated embodiment, the GT actor component 105 generates and provides artificial game play actions 128 to the game application 115, receives observation data and in-game reward indicators 112 from the game application, and provides some or all of this information to the GT learning service 120 as observed experience data 130. The GT learning service 120 then uses the received observed experience data 115 to generate, refine, and / or adapt one or more gameplay models 135 associated with the game application 115 to improve both the overall gameplay and the individual artificial gameplay actions that the GT actor components provide to the game application. Provided at 5.

[0034] At various points in this process, the GT system 100 (through the GT learning services 120 and / or the GT actor component 105) may generate test data associated with the game application 115. In certain embodiments, a game developer associated with the game application 115 may specify one or more types and behaviors of such test data (e.g., through a programmatic interface of the GT learning services 120 and / or the GT actor component 105). In addition, in certain embodiments, the GT system may determine one or more aspects of the test data to generate, such as based on defined criteria stored by the GT system. In such embodiments, the defined criteria may relate to one or more types of game applications for which the game application 115 has been determined to meet the criteria. For example, a first set of defined criteria for the test data generated by the GT system may relate to two-dimensional platform game types, a second set to three-dimensional platform game types, a third set to racing game types, a fourth set to open world role-playing games, etc.

[0035] As mentioned above, in certain scenarios, the artificial gameplay actions of the GT actor component 105 may be based at least in part on one or more gameplay data models 135 generated by the GT system 100 based on one or more parameters defined for the game application 115. Such parameters may be provided (as a non-limiting example) by a developer of the game application 115 via a programmatic interface of the GT system 100. For example, an initial gameplay model may be based on control information that maps each of one or more output states of the game application 115 to input variables of the GT actor component 105 and / or control information that maps each of one or more input states of the game application to output variables of the GT actor component 105. In certain embodiments, such control information may map each of these input and / or output variables of the GT actor component 105 to observations or actions available to a human user of the game application. For example, the output variables of the GT actor component 105 may represent the movement of a virtual character in the game application, which output variables correspond to the movement of that virtual character by a physical input device utilized by the human user during gameplay. Such output variables of the GT actor component 105 may correspond to actions or observations available to a human user during such gameplay. In certain embodiments, the GT API allows game application developers to describe inputs and outputs using high-level primitives (e.g., "joysticks," "entities," etc.). The GT SDK then maps the high-level primitives to GT control information so that the API can be utilized without requiring game application developers to exercise expertise in machine learning.

[0036] In particular embodiments, the control information may include one or more output states of the game application 115 for use as input variables by the GT actor component 105. By way of non-limiting example, such output states may include a player reference position within the virtual environment of the game application 115, a position of an object relative to a player reference position within the virtual environment of the game application 115, a motion vector associated with an object relative to a player reference position within the virtual environment of the game application 115, geometry information regarding one or more aspects of the virtual environment of the game application 115, and / or a score or other in-game reward indicator associated with game play within the game application 115. In general, the control information may map any aspect of the game application observable by a human player to an input variable of the GT actor component 105.

[0037] Additionally, in some embodiments, the GT system 100 may receive observed experience data resulting from gameplay actions provided by one or more human players to the game application 115, such as to generate or modify one or more gameplay models associated with the game application 115 in a manner similar to that used to receive observed experience data 130 resulting from artificial gameplay actions of the GT actor component 105. As an example, the output variables of the GT actor component 105 may enable the GT actor component 105 to indicate an assistance state within the game application 115, such as when the GT actor component 105 encounters an obstacle in the game application 115 that it has been unable to overcome for a long period of time or after a predetermined number of attempts, the GT system 100 may initiate communication to prompt the one or more human players to provide one or more gameplay actions indicative of overcoming the obstacle. The observed experience data resulting from these human-provided gameplay actions is provided to the GT learning service 120, which then modifies the gameplay model 135 to enable the GT actor component 105 to overcome that and / or other obstacles the next time they are encountered.

[0038] One technique for training an AI that can play games is Reinforcement Learning (RL). In RL, developers reward for winning and penalize for losing, providing signals that the AI ​​uses later to autonomously learn increasingly optimal strategies. Unfortunately, while very impressive results have been demonstrated with RL, RL algorithms usually come with significant costs for developers in terms of time and computing resources, consuming large amounts of data (sample inefficiency), such as millions or billions of frames of data to train the player. Also, the results of these algorithms tend to be highly variable, and they utilize considerable knowledge in the domain, and tuning of hyperparameters, to achieve acceptable results.

[0039] To that end, certain embodiments utilize IL (Imitation Learning) machine learning techniques. IL machine learning techniques train an AI based on observing one or more human players play a game. Unlike RL, where the agent must find the optimal policy by itself, IL effectively replicates the behavior of human experts. Generative IL policies work well in scenarios that are similar, but not identical, to those captured in human demonstrations. This problem is especially acute in games, which are typically built as many variations (levels) on a small number of common themes (mechanics). An AI that can only learn specific variations but not the underlying themes will not be a very effective tool.

[0040] The GT system 100 uses observations that effectively generalize. For example, egocentric observations, where 3D information is expressed relative to the viewpoint of the GT actor component 105 rather than in absolute coordinates, can cause the GT learning service 120 to generate gameplay data models 135 that contain movements and goal policies that are unrelated to the training environment.

[0041] Thus, in various embodiments, the GT system 100 may decide to receive observed experience data 130 resulting from human-provided gameplay actions in order to refine the relevant gameplay model 135 based on one or more additional criteria, i.e., criteria other than confronting a challenging obstacle. By way of non-limiting example, the GT system 100 may receive such data for all sessions or a subset of sessions associated with one or more identified human players who have elected to provide such data; Such data about the application, and the like, may be received at regular or otherwise scheduled intervals.

[0042] In some embodiments, a developer may provide training data to the GT learning service by simply playing the game application 115 in real time, and the GT learning service 120 may modify one or more gameplay models 135 and update the GT actor components 105 accordingly. In this way, the GT system 100 provides interactive feedback to the developer regarding the quality of the GT learning service 120, allowing the developer to correct in a just-in-time manner if or when the GT system 100 encounters problems. In some embodiments, a developer may create as many concurrent instances of the GT actor components 105 as needed and have these instances play and test the game application at scale. Also, in certain embodiments, the GT system 100 may evaluate multiple gameplay data models 135 based on various reward criteria, determine which one or more gameplay data models 135 perform better with respect to these reward criteria in the game, and so on. The GT system 100 may then select the best-performing gameplay data model 135 as the gameplay data model to utilize for future inferences. Such automatic evaluation allows the GT system 100 to "refine" the trained gameplay data model.

[0043] Figure 2 illustrates an exemplary embodiment of another networked GT system 200. Similar to the networked system 100 of Figure 1, the networked system 200 includes an instance of a GT actor component 105 running on a local computing system 110. The local computing system 110 is also running a game application 115. The GT actor component 105 (which is communicatively coupled to a GT learning service 120 via computer network(s) 101) provides artificial game play actions 128 to the game application 115, receives observational data 112 from the game application 115, and provides the resulting observed experience data 130 to the GT learning service 120. Here, however, the game application 115 may receive the artificial game play actions 128 of the GT actor component 105, and may also receive other game play actions 260 of one or more human players 250 that perceive the display information 255 (which typically includes audio, visual, tactile, and / or other sense-detected information) generated by the game application 115 at various times and according to various criteria in the manner of a normal gaming session. Whether the game application 115 receives game play actions from the GT actor component 105, from the human player(s) 250, or both, the GT actor component 105 receives observational data 112 from the game application 115 and provides the resulting observed experience data 130 to the GT learning service 120.

[0044] As mentioned above, in certain embodiments and implementations, the GT actor component 105 may include a Gameplay Trainer SDK. The Gameplay Trainer SDK includes executable instructions and compiled libraries that a developer communicatively connects (or "links") to a game application 115 (such as by incorporating the SDK into the game application's program code) and APIs that the developer may use to enable programmatic interaction with one or more components of the GT system 200. In certain embodiments, the GT system 200 may include various SDKs to support each of several popular game development frameworks, such as Unity, Unreal, and C++ (e.g., for proprietary engines). Each of these SDKs provides the same functionality (collection of observations / actions, etc.). Although some implementations may provide a number of different capabilities (such as collection / submission, and on-device inference), in many cases the same functionality may be provided in an idiomatic way using language-specific and engine-specific bindings.

[0045] 3 illustrates a schematic block diagram of a GT system 300 implemented in accordance with one or more embodiments. In the illustrated embodiment, one or more remote GT servers 325 include a GT API 399, a storage facility 312, and a running instance of a GT learning service 360. A client computing system 310 runs an instance of a game application 315 and an instance of a GT actor component 305. Each of the GT server(s) 325 and client computing systems 310 may be fixed or mobile and may include instances of a variety of computing devices, such as, but not limited to, desktop or other computers (e.g., tablets, slates, etc.), database servers, network storage devices and other network appliances, smartphones and other mobile phones, consumer electronics, gaming systems, digital music player devices, portable gaming devices, PDAs, pagers, electronic organizers, Internet appliances, television-based systems (e.g., using set-top boxes and / or personal / digital video recorders), and various other consumer products equipped with appropriate communications capabilities.

[0046] As described herein, the GT learning service 360 ​​exchanges various information (e.g., authentication information, gameplay data models, observation data) with the GT actor component 305. In the illustrated embodiment, an embodiment of the GT learning service 360 ​​operates within a memory (not shown) of the remote GT server(s) 325, such as by executing software instructions of the GT learning service 360 ​​using one or more hardware processor(s) to perform at least some of the described techniques, in a manner that configures the remote GT server(s) 325 to perform automated operations that implement these described techniques. As part of such automated operations, the GT learning service 360 ​​may store and / or retrieve various types of data, including data structures in the storage facility 312.

[0047] The storage facility 312 stores various information used by the GT system 300 (and in particular the GT learning service 360) to generate and store the gameplay data models 334 as part of providing them to one or more client computing systems (e.g., the client computing system 310). Other information stored by the storage facility 312 includes developer information 338 (which may include access and project information for one or more game application developers), game application information 340 (which may include control information 341, gameplay observation data, analysis and / or evaluation of that gameplay observation data, and historical information for one or more particular game applications), game session information 342, and training data 336 (which may be used and stored by the GT learning service 360 ​​as part of generating one or more gameplay data models 334 and for other operations). In certain implementations, the storage facility 312 may be incorporated within the GT system 300 or may otherwise be operated directly by the GT system 300. In other implementations, some or all of the functionality provided by storage facility 312 may be provided by one or more third-party network-accessible storage service providers. Also, in certain embodiments, GT learning service 360 ​​includes logic for authenticating developers and tracking metadata about the developers' projects, such as by using and modifying aspects of developer information 338.

[0048] Interaction with the GT learning service 360 ​​(e.g., by game application developers and / or by client computing systems 310) occurs through a GT API 399. In the illustrated embodiment, the GT API 399 provides an access control facility 380 and a programmatic interface for passing gameplay data models 382 and observational data 384 between the remote GT server 345 and the client computing systems 310. In certain embodiments, use of the GT system 300 may be restricted by one or more revocable API keys 381, stored as part of developer information 338, etc., along with various other information regarding one or more projects associated with an identified developer. These keys may be used, for example, to index all developer submitted data, including observations and gameplay actions (generated by artificial or human players). In such an embodiment, the developer may only have access to the data they submitted and may further control various aspects of that data (e.g., forwarding and / or deletion). Thus, in certain embodiments, API requests may require the developer to provide a valid server-provided API key that can be used for authentication and throughout all interactions with the GT learning service. In this way, the GT system 300 ensures that API calls and collected data are associated with the calling developer.

[0049] Behind the developer-facing GT API 399, the GT learning service 360 ​​implements an Actor / Learner pattern in which the GT actor component 305 collects and generates gameplay observations for the GT learning service 360 ​​to convert them into gameplay actions by generating and updating / modifying one or more gameplay data models associated with the game application 315. In the illustrated embodiment, the GT actor component 305 and its MLP (machine learning platform) 302 may perform various interactions with the game application 315 to simulate the actions of a human player of the game application based on one or more local gameplay data models 306 received from the GT learning service 360 ​​via the GT API 399. For example, gameplay actions based on the local gameplay data models 306 are provided to the game application 315 via an action applicator 316. The action applicator 316 operates a controller module 318 to simulate the execution of these gameplay actions through the functions of a game controller that would normally be operated by the human player 250. These game play actions are reported to the MLP 302 via the action reporter 319. The game play actions themselves modify the simulation 320 via output variables (not shown) of the GT actor components 305, and additional environmental observations 322 are collected as a basis for generating local observation data 304.

[0050] In various embodiments, the GT actor component 305 may be included in and / or generated using the GT SDK (code and compiled libraries, not shown, that developers link into their game applications and utilize to interact via the GT API 399), and may perform on-device inference (e.g., using gameplay data models to generate one or more predictions of in-game behavior and outcomes as inferences 303) to collect and generate local observation data. The GT learning service 360 ​​provides the necessary infrastructure to ingest data from the SDK, store it via storage facility 312, train new gameplay data models 334, and return these gameplay data models to the GT actor component 305 via the GT API 399. In this way, the GT SDK provides the observation / action collection and on-device inference, and also acts as an adapter between the developer-facing API and the GT learning service 360. The GT learning service 360 ​​may also communicate with remote GT servers. The local gameplay data model 306 is executed by the local gameplay data model(s) 325, which uses various algorithms to train the gameplay data model. In particular embodiments, one or more of the model training algorithms may be based on the operation of a machine learning platform (MLP) 332, such as TensorFlow or other machine learning platform. Similarly, on-device inference may be performed using the MLP 302 (which may again include TensorFlow or other machine learning platform). The local gameplay data model 306 is retrieved from the GT learning service 360, and in particular embodiments, may include control information that describes how to map observations and actions to input and output variables of the gameplay data model.

[0051] The GT API 399 allows game application developers to define parameters that describe the logical inputs and outputs of a game application, as well as feedback about how well the AI ​​is doing at the time. Specifically, the GT system allows developers to define one or more observation parameters, one or more action parameters, and one or more reward parameters for each of one or more game applications (e.g., game application 315). The observation parameters (resulting in the generation of observation data) describe the game state that a player typically experiences and may include information such as the location of a visible enemy in a first-person shooter game, or the distance from the player's car to the wall of the track in a racing game. The action parameters describe the logical gameplay actions that a player performs in a game, such as jumping in a platform game, or the position of the steering wheel in a racing game. The reward parameters set one or more metrics to provide feedback about how well the GT actor component 305 is performing, and thus the output state of the game application depending on the gameplay actions. In certain scenarios and embodiments, such parameters may include numerical values ​​similar to how a player may earn points in a game application, but may also include other parameters. For example, reward parameters may include the average or maximum amount of damage a player may inflict during an encounter over a given period of time or during a particular game session, as well as a simple win / lose signal at the end of a game session.

[0052] This approach allows all relevant output to be sent at hundreds of kilobits per second (tens of megabits per second for 4K video), allowing the GT actor component 305 to focus on learning how to play the game application 315 without having to simultaneously solve complex computer vision problems. In the process, the game application developer is given more control over the data they share with the GT learning service 360. This approach works for standalone game clients and server-based game applications alike. Additionally, the GT system 300 operates without significant negative impact from the inevitable latency of tens of milliseconds (or more) between when a game sends an observation and when it receives an action. Given that game applications typically operate at more than 30 frames per second (less than 33.3 milliseconds per frame), which is well below the time it takes for a round trip between the server and the client computing system, such latency can be particularly problematic for alternative solutions that rely on server input for gameplay actions.

[0053] In certain embodiments, the GT system 300 may operate asynchronously between the remote GT server 325 and the client computing system 310, for example, generating observational data based on one frame, progressing the simulation, and then applying one or more gameplay actions several frames later. This work is leveraged by the GT system's actor / learner architecture, where the GT actor 305 rapidly translates gameplay observations into gameplay actions using a gameplay data model, and the GT learning service 360 ​​creates new gameplay data models based on the observation data, actions, and rewards generated by the actor component, as well as interactions with the game application 315. Separating the AI's behavior into these two components allows the actor component to translate gameplay observations into gameplay actions without incurring any latency longer than the time it takes to perform inference, even though the GT learning service 360 ​​leverages secure machine learning algorithms and significant amounts of computing resources.

[0054] This architecture has several advantages beyond avoiding the latency associated with traversing the public Internet. It naturally matches the asymmetry between training (which is very computationally intensive) and inference (which can be performed on a fraction of a single CPU). By batching and compressing experiences in the GT actor component 305, the GT system 300 can reduce the associated queries per second (QPS) by up to 30 times, as well as reduce bandwidth by up to 10 times.

[0055] Games are inherently interactive media. Unfortunately, traditional ML workflows are never interactive, with minutes or hours between the submission of data and the generation of a model based on that data. The GT system 300 addresses this problem by training gameplay data models in real time. In certain embodiments, as soon as the GT learning service 360 ​​receives observational data (whether based on actions taken by the GT actor component 305 with an existing gameplay data model or based on actions taken by one or more human players), the GT learning service 360 ​​starts training a model on that data, often building results from previous demonstrations and / or gameplay data models. Thanks to the compact representation of observations and actions, new gameplay data models can be generated in seconds. These gameplay data models can be constantly evaluated against demonstrations submitted by human players and provided to the GT actor component 305 only if the GT actor component outperforms the current gameplay data model in use.

[0056] Thus, as described above, the GT system 300 provides a fully real-time and interactive training experience to the game application developer. To train an AI, the game application developer can define and provide control information that maps each of one or more output states of the game application to input variables of an actor component running on a client computing device, and the game application developer can simply pick up a game controller and train the AI ​​by playing the game. After playing a few rounds of the game, the game application developer can put down the controller and watch the AI ​​play the game. If the AI ​​encounters a problematic state, the game application developer can simply pick up the controller, demonstrate the correct behavior, and then have the AI ​​operate again. The result is a highly experience-based and controllable service.

[0057] In order for the GT learning service 360 ​​to generate a gameplay data model, the GT actor component 305 periodically sends local observation data 304 (including gameplay observations, actions, and rewards) to the GT learning service. When learning from human gameplay demonstrations (e.g., via human players 250), this The observational data is derived from actions taken by humans while playing the game. In particular embodiments, this experience data may be generated by employees of the game development company, by consumer players, or a combination thereof.

[0058] After receiving a new batch of observational data, the GT API 399 generates new challenges 344. Each challenge 344 represents a request to generate a new gameplay data model. The challenges are provided to the GT Learning Service 360 ​​via a challenge queue 346. The GT Learning Service 360 ​​contains the necessary logic to combine and translate the gameplay actions, observational data, rewards, and specific parameters contained in the challenge into a format that the MLP 332 can understand and generate / evaluate the new gameplay data model. Such gameplay data models are designed to be disposable, and in certain embodiments, the state of the gameplay data model may be expressed as a restartable ML checkpoint.

[0059] Once the GT learning service 360 ​​has completed its work on the challenge, the correspondingly generated gameplay data model is stored as part of the gameplay data model 334. The GT learning service 360 ​​then provides the updated gameplay data model to the GT actor component 305 for use in on-device inference within the client computing system 310.

[0060] In at least some embodiments, the GT actor component 305 can perform inference on the order of milliseconds, such as to support real-time interaction with a developer's game. On the server side, the GT learning service 360 ​​may, in certain embodiments, convert the additional observational data into new gameplay data models in a period of time on the order of tens of seconds (e.g., less than 30 seconds).

[0061] The individual components and modules illustrated herein are provided as convenient examples for describing component-level structures and specific data flow operations, but it will be appreciated that in various embodiments other arrangements of specific components and modules may be used to implement the techniques presented herein.

[0062] Figure 4 is a block flow diagram outlining an operational routine 400 of the GT system, including both operation at a GT learning server 401 and operation at a client computing system 402. The client computing system 402 is running a game application and a GT actor component (such as the GT actor component 105 of Figures 1 and 2 or the GT actor component 305 of Figure 3) in accordance with one or more embodiments.

[0063] The routine begins at block 405, where the GT learning server 401 provides a gameplay data model 410 to the client computing system 402. The client computing system 402 receives the gameplay data model via the GT API 499 at block 415. As described in further detail herein, in certain embodiments, the gameplay data model 410 initially provided by the GT learning server 401 may be based on one or more parameters defined for the game application, and may include a combination of game observation parameters, game action parameters (e.g., control information), and game reward parameters. In addition, the initial gameplay data model 410 (and subsequent gameplay data models) may be based at least in part on observation data generated from one or more human players.

[0064] After the gameplay data model 410 is received at block 415, the routine proceeds to block 420, where the client computing system 402 generates observational data regarding gameplay using the gameplay data model. At block 425, the client computing system 402 provides the generated observational data 430 to the GT learning server 401 via the API 499.

[0065] In particular embodiments, the observational data and game play actions may be aggregated on the client computing system 402 (such as by a GT actor component, e.g., the GT actor component 305 of FIG. 3 ) until provided to the GT learning service 360 ​​in response to one or more criteria. For example, with reference to FIG. 3 , provision of local observational data 304 (including game play action information) may begin in response to an explicit request, after a predetermined period of time, after a predetermined amount of observational data has been generated, etc.

[0066] At block 435, the GT learning server 401 receives the observational data generated by the client computing system 402 using the game play data model 410, and the routine proceeds to block 440. At block 440, the GT learning server 410 modifies the game play data model 410 based on the newly received observational data 430.

[0067] In block 445, the GT learning server 401 provides the modified gameplay data model 450 to the client computing system 402 via the API 499. In block 455, the client computing system 402 receives the modified gameplay data model 450 and returns to block 420 to collect and generate additional observational data based on gameplay actions using the modified gameplay data model. Similarly, after providing the modified gameplay data model 450 in block 445, the GT learning server 401 returns to block 435 to receive updated observational data from the client computing system 402.

[0068] In some embodiments, some aspects of the techniques described above may be implemented by one or more processors of a processing system executing software. The software includes one or more sets of executable instructions stored or otherwise tangibly contained in a non-transitory computer-readable storage medium. The software may include these instructions and specific data that, when executed by one or more processors, cause the one or more processors to perform one or more aspects of the techniques described above. The non-transitory computer-readable storage medium may include, for example, solid-state storage devices such as magnetic or optical disk storage devices, flash memory, caches, Random Access Memory (RAM), or one or more other non-transitory memory elements. The executable instructions stored in the non-transitory computer-readable storage medium may be stored in source code, assembly language code, object code, or other instruction formats that are interpreted or executable by one or more processors.

[0069] A computer readable storage medium may include any storage medium or combination of storage media that can be accessed by a computer system to provide instructions and / or data to the computer system during use. Such storage media may include optical media (e.g., Compact Discs (CDs), Digital Versatile Discs (DVDs), Blu-Ray Disks), magnetic media (e.g., floppy disks, magnetic tapes, or magnetic hard drives), volatile memory (e.g., Random Access Memory (RAM) or cache), non-volatile memory (e.g., Read-Only Memory (ROM) or flash memory), or any combination of storage media that can be accessed by a computer system during use to provide instructions and / or data to the computer system. The computer readable storage medium may include, but is not limited to, Microelectromechanical System (MEMS) based storage media. The computer readable storage medium may be embedded in the computing system (e.g., system RAM or ROM), fixedly attached to the computing system (e.g., a magnetic hard drive), removably attached to the computing system (e.g., an optical disk or Universal Serial Bus (USB) based flash memory), or coupled to the computer system via a wired or wireless network (e.g., a Network Accessible Storage (NAS)). Storage).

[0070] It should be noted that not all of the above-described operations or elements are required in the overall description. Particular operations or parts of the device may not be required. In addition to the described operations and elements, one or more additional operations may be performed, and one or more additional components may be included. Furthermore, the order in which the operations are listed is not necessarily the order in which they are performed. Also, the concepts have been described with reference to specific embodiments. However, those skilled in the art will recognize that various modifications and changes can be made without departing from the scope of the present disclosure as set forth in the appended claims. Thus, the specification and drawings should be regarded as illustrative rather than strict, and all such modifications are intended to be within the scope of the present disclosure.

[0071] Benefits, other advantages, and solutions to problems have been described with reference to specific embodiments. However, any feature or features that may cause or make the benefits, advantages, or solutions more pronounced should not be construed as critical, necessary, or essential features of any or all claims. Furthermore, the specific embodiments disclosed herein are illustrative only, since the disclosed subject matter may be modified and practiced in equivalent manners with differences apparent to those skilled in the art having the benefit of the teachings herein. No limitations are to be placed on the details of construction or design shown herein, other than as described in the appended claims. It is therefore apparent that the specific embodiments disclosed herein may be modified or improved upon, and all such variations are deemed to be within the scope of the disclosed subject matter. Accordingly, the protection sought herein is as set forth in the appended claims.

Claims

1. Providing, from one or more server computing systems to remote client computing devices, a simulation data model of an application executing on the remote client computing devices; receiving observation data from the remote client computing device generated from simulated actions performed by the actor component within the application based at least in part on inferences generated by the actor component executing on the remote client computing device using the provided simulation data model; the one or more server computing systems modifying the simulation data model based on the received observation data; providing the modified simulation data model to the remote client computing device.

2. 2. The method of claim 1, further comprising: the one or more server computing systems receiving control information that maps each of one or more output states of the application to input variables of the actor component running on the remote client computing device.

3. 3. The method of claim 2, wherein the one or more output states of the application include one or more of the following group: a player reference position within a virtual environment of the application, a position of an object relative to the player reference position in the virtual environment of the application, a motion vector associated with an object relative to the player reference position in the virtual environment of the application, geometry information regarding one or more aspects of the virtual environment of the application, and / or one or more in-game reward indicators associated with a simulation of the application.

4. 2. The method of claim 1, further comprising: the one or more server computing systems receiving control information that maps each of one or more output variables of the actor component to an action available to a human user of the application.

5. The method of claim 1 , wherein modifying the simulation data model is further based on additional observational data generated based on simulated actions performed within the application by human users of the application.

6. The method of claim 5 , wherein modifying the simulation data model based on the additional observational data comprises modifying the simulation data model using deep learning artificial intelligence.

7. The method of claim 1 , further comprising generating test data for the application based on the simulated actions.

8. The method of claim 1 , further comprising: in response to receiving an aggregation of observational data that meets at least one predefined criterion, modifying the simulation data model based on the received observational data.

9. The method of claim 8 , wherein the at least one predefined criteria includes at least one of a predetermined time period, a predetermined amount of observed data, and an explicit request received at the remote client computing device.

10. A server, A network interface; one or more processors; and a memory storing a set of executable instructions, the set of executable instructions operating the one or more processors to: generating a simulation data model of the application based at least in part on control information mapping input variables to each of one or more output states of the application; providing the generated simulation data model to an actor component executing on a remote client computing device; receiving from the actor component observation data generated from simulated actions performed by the actor component within the application based on inferences generated by the actor component using the generated simulation data model; Modifying the generated simulation data model based on the received observation data; A server that causes the actor component to provide the modified simulation data model for use by the actor component in performing additional simulated actions within the application.

11. 11. The server of claim 10, wherein the remote client computing device executes an instance of the application, and the observational data is generated from simulated actions performed by the actor component within the instance of the application executed by the remote client computing device.

12. 11. The server of claim 10, wherein the set of executable instructions further operates the one or more processors to receive control information mapping each of one or more output states of the application to input variables of the actor component executing on the remote client computing device.

13. 13. The server of claim 12, wherein the one or more output states of the application include one or more of the following group: a player reference position within the virtual environment of the application, a position of an object relative to the player reference position within the virtual environment of the application, a motion vector associated with an object relative to the player reference position within the virtual environment of the application, geometry information regarding one or more aspects of the virtual environment of the application, and / or one or more in-game reward indicators associated with gameplay of the application.

14. 11. The server of claim 10, wherein the set of executable instructions further operates the one or more processors to receive control information mapping each of one or more output variables of the actor component to actions available to a human user of the application.

15. The server of any one of claims 10 to 14, wherein the set of executable instructions further operates the one or more processors to receive additional observational data generated from game play actions performed within the application by a human user of the application, and wherein modifying the simulation data model is further based on the received additional observational data.

16. The server of claim 15 , wherein modifying the simulation data model based on the received additional observational data comprises modifying the simulation data model using deep learning artificial intelligence.

17. an actor component executing on one or more processors receiving a simulation data model of the application from a machine learning component executing on one or more remote server computing systems; said one or more processors executing an instance of said application; providing the machine learning component with observation data generated from simulated actions performed by the actor component within the instance of the application based at least in part on inferences generated by the actor component using the simulation data model; and receiving a revised simulation data model from the machine learning component running on the one or more remote server computing systems, the revised simulation data model being based at least in part on the provided observational data.

18. 20. The method of claim 17, further comprising: performing one or more additional simulated actions based at least in part on additional inferences generated by the actor components using the modified simulation data model.

19. The method of claim 17 , further comprising generating test data for the application based on the simulated actions.

20. The method of claim 17 , wherein the simulation data model is based at least in part on control information that maps input variables of the actor component to each of one or more output states of the application.

21. 21. The method of claim 20, wherein the one or more output states of the application include one or more of the following group: a player reference position within the virtual environment of the application, a position of an object relative to the player reference position in the virtual environment of the application, a motion vector associated with an object relative to the player reference position in the virtual environment of the application, geometry information regarding one or more aspects of the virtual environment of the application, and / or one or more in-game reward indicators associated with gameplay of the application.

22. The method of claim 17 , wherein the simulation data model is based at least in part on control information that maps each of one or more output variables of the actor components to actions available to a human user of the application.

23. 20. The method of claim 17, further comprising generating additional observational data generated from game play actions performed within the application by a human user of the application, such that the modified simulation data model is further based on the additional observational data.

24. A computer system for carrying out the method of any one of claims 1 to 9 or 17 to 23.

25. A computer program storing executable instructions which, when executed by one or more processors, cause the one or more processors to perform a method according to any one of claims 1 to 9 or 17 to 23.

Images