Game development system and method of use thereof
By introducing a universal template into the game development system, the game logic and algorithm content are separated, the reinforcement learning concept is hidden, and the problems of high embedding cost and language limitation are solved, enabling universal application across multiple languages and games.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NETEASE (HANGZHOU) NETWORK CO LTD
- Filing Date
- 2022-09-20
- Publication Date
- 2026-07-14
AI Technical Summary
Embedding reinforcement learning in existing technologies is too costly and limited in applicable languages, resulting in high learning costs for game developers and making it difficult to apply it universally across different games.
A game development system is provided, including a game logic content module, an algorithm content module, and a general template. The system uses a control module to associate the features of the artificial intelligence agent module in the game logic content with the game logic, and calls the algorithm content to support the game output at the task level. It hides the concept of reinforcement learning and is suitable for multi-language development.
It reduces the learning cost for game developers, enables universal application across different languages and games, and lowers the difficulty of embedding reinforcement learning.
Smart Images
Figure CN115904360B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a game development system and its usage. Background Technology
[0002] In video games, artificial intelligence is primarily used to generate responses from non-player characters (NPCs) that resemble human intelligence. Since its inception, AI has been an indispensable part of games. AI implemented using reinforcement learning offers higher levels of competitiveness, more diverse styles, and a greater degree of automation.
[0003] In existing technologies, Unity's Machine Learning (ML) Agents solution, based on the Unity engine, provides a set of plugins for interfacing reinforcement learning algorithms with Unity-based games. Games developed using the Unity engine use the concept of Game Objects to represent entities within the game. ML-Agents, based on Unity's scripting mechanism, provides Agent objects that can be attached to specific Game Objects, thereby accessing or manipulating the Game Object's properties.
[0004] The aforementioned technical solutions still contain many concepts of reinforcement learning, which presents game developers with high learning costs and limitations in applicable languages. Summary of the Invention
[0005] In view of this, this application provides a game development system and its usage method to solve the problems of high cost and limited applicable languages when embedding reinforcement learning in the game development process.
[0006] The first aspect of this application provides a game development system, the system comprising: a first module providing game logic content, a second module providing algorithm content, and a general template for embedding reinforcement learning in the game;
[0007] The general template includes: at least one artificial intelligence agent module and a control module that provides control logic for the at least one artificial intelligence agent module;
[0008] The control module is used to respond to the access command issued by the first module, call each artificial intelligence agent module, and associate the relevant features of the control objects corresponding to each artificial intelligence agent module in the game logic content with the game logic. The relevant features include: the feature information of the control object and the feature information of the scene in which the control object is located.
[0009] In the process of associating relevant features with the game logic, each AI agent module calls the algorithm content provided by the second module according to the relevant features to support the game output at the task level of the control object corresponding to each AI agent module.
[0010] A second aspect of this application provides a method for using a game development system. The game development system provides a first module for providing game logic content, a second module for providing algorithm content, and a general template for embedding reinforcement learning in the game. The general template includes at least one artificial intelligence agent module and a control module providing control logic for the at least one artificial intelligence agent module. The method includes:
[0011] In response to the access command issued by the first module, each artificial intelligence agent module is invoked to associate the relevant features of the control objects corresponding to each artificial intelligence agent module in the game logic content with the game logic. The relevant features include: the feature information of the control object and the feature information of the scene in which the control object is located.
[0012] In the process of associating relevant features with the game logic, each AI agent module calls the algorithm content provided by the second module according to the relevant features to support the game output at the task level of the control object corresponding to each AI agent module.
[0013] A third aspect of this application provides a device for using a game development system, the device comprising: a response module and a calling module;
[0014] The response module is used to respond to the access command issued by the first module, call each artificial intelligence agent module, and associate the relevant features of the control object corresponding to each artificial intelligence agent module in the game logic content with the game logic. The relevant features include: the feature information of the control object and the feature information of the scene in which the control object is located.
[0015] The calling module is used to, during the process of associating relevant features with the game logic, call the algorithm content provided by the second module according to the relevant features to support the game output of the control object corresponding to each artificial intelligence agent module at the task level.
[0016] A fourth aspect of this application provides a computing device, including: a processor and a memory;
[0017] The memory stores computer-executed instructions;
[0018] The processor executes the computer execution instructions, causing the computing device to perform the method of using the game development system as described in the second aspect of the design above.
[0019] The fifth aspect of this application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the usage method of the game development system as described in the design of the second aspect above.
[0020] The technical solution provided in this application embodiment responds to an access command issued by a first module, invoking various artificial intelligence agent modules. It associates the relevant features of the control objects corresponding to each artificial intelligence agent module in the game logic content with the game logic. During this process, each artificial intelligence agent module, based on the relevant features, invokes the algorithm content provided by a second module to support the game output of the control objects corresponding to each artificial intelligence agent module at the task level. This technical solution provides an interface to the outside world from the control module to call the game logic content developed by game developers. Internally, the control module calls various artificial intelligence agent modules to assist in game generation. The reinforcement learning concept is invisible to game developers, reducing their learning cost when embedding game logic content. Providing an interface externally allows the corresponding interface to be called within the game logic content, making it suitable for games developed in multiple languages. Attached Figure Description
[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0022] Figure 1 A schematic diagram of the Unity ML-Agents embedding logic provided for existing technologies;
[0023] Figure 2 A schematic diagram of a game development system provided in an embodiment of this application;
[0024] Figure 3 A schematic diagram of a general template for embedding reinforcement learning in games, provided as an embodiment of this application. Figure 1 ;
[0025] Figure 4 A schematic diagram of a general template for embedding reinforcement learning in games, provided as an embodiment of this application. Figure 2;
[0026] Figure 5 A schematic diagram of a general template for embedding reinforcement learning in games, provided as an embodiment of this application. Figure 3 ;
[0027] Figure 6 This is a schematic diagram illustrating an embodiment of the usage method of the game development system provided in this application.
[0028] Figure 7 A schematic diagram of the device for using the game development system provided in this application embodiment;
[0029] Figure 8 A schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation
[0030] To enable those skilled in the art to better understand the technical solutions of this application, the application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. However, this application can be implemented in many other ways different from those described above. Therefore, based on the embodiments provided in this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.
[0031] It should be noted that the terms "first," "second," "third," etc., in the claims, specification, and drawings of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. Such data are interchangeable where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown or described herein. Furthermore, the terms "comprising," "having," and their variations are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.
[0032] First, let me explain the technical terms and background involved in this application:
[0033] Reinforcement learning (RL) is a fundamental learning method in machine learning. This method emphasizes how the learning interacts with the environment to maximize expected gains.
[0034] Markov decision process (MDP): A mathematical model of sequential decision-making, used to simulate stochastic policies and rewards achievable by an agent in environments where the system state exhibits Markov properties.
[0035] Non-player character (NPC): A character in a game that is not controlled by the player, such as a monster or an opponent in a single-player game.
[0036] In video games, artificial intelligence (AI) is primarily used to generate responses from non-player characters that resemble human intelligence. Since its inception, AI has been an indispensable component of games.
[0037] Compared to using behavior trees or state machines to achieve artificial intelligence, artificial intelligence achieved through reinforcement learning has a higher level of competitiveness and more diverse styles; at the same time, this production method has a higher degree of automation.
[0038] Among them, the behavior tree can be the guide that controls the NPC to perform behaviors such as walking, eating, and shopping, while the state machine can be similar to a complex behavior tree, a graph composed of behavioral relationships.
[0039] However, due to the significant differences between the production process and logical structure of artificial intelligence in traditional games and the concepts and modeling methods of Markov decision processes in reinforcement learning, introducing reinforcement learning methods into the game development process presents a series of problems, such as high learning costs, high customization requirements, and low reusability.
[0040] In existing technologies, the most primitive method is to directly embed reinforcement learning logic into the code. This method is strongly related to the game scenario being developed. That is, reinforcement learning logic needs to be embedded for each game, which is cumbersome and lacks universality.
[0041] Another approach is the (open-source) technology solution, Machine Learning (ML) Agents released by Unity. This solution, based on the Unity engine, provides a set of plugins for integrating reinforcement learning algorithms with Unity-based games. Its main idea is as follows: Figure 1 As shown.
[0042] Figure 1 A schematic diagram of the Unity ML-Agents embedding logic provided for existing technologies, specifically:
[0043] Games developed using the Unity engine use the concept of Game Objects to represent entities within the game. ML-Agents, based on Unity's scripting mechanism, provides Agent objects that can be attached to a specific game object, allowing users to access or manipulate the game object's properties.
[0044] Furthermore, the Agent provides four interfaces: 1# "OnEisodeBegin", 2# "CollectObservations", 3# "OnActionReceived", and 4# "Heuristic".
[0045] In one possible implementation, the OnEisodeBegin interface is called when the Agent's Episode begins, where Episode refers to a round, such as halftime in a football game or finishing a round of cards in a card game.
[0046] The CollectObservations interface is called every time the Agent needs to make a decision. It collects custom observations, such as how many cards are left in a hand in a card game and how many cards have been played.
[0047] The OnActionReceived interface is called every time the Agent receives an action. This function needs to implement operations on the game object and calculate the reward, such as playing a card in a game of Dou Dizhu and the win / loss mechanism.
[0048] The Heuristic interface implements some heuristic action acquisition methods, similar to the original behavior tree or flowchart rule action generator, that is, to obtain the action rule agent, which can be connected to a neural network or to a learnable module.
[0049] catch Figure 1 The existing technology shown has the following shortcomings:
[0050] First, because it is based on Unity, it only supports games developed using the Unity engine. It is not compatible with games written in other languages or languages like C# that do not use the Unity engine.
[0051] Secondly, the interfaces exposed at the Agent level contain many reinforcement learning concepts (such as Episode / Observation), which still pose a high cost for game developers; for technical personnel who perform algorithm embedding work, they need to have both reinforcement learning experience and game development experience.
[0052] This application addresses the aforementioned technical problems. The inventors have discovered that in existing technologies, the interfaces exposed at the Agent level require both game developers and algorithm engineers to learn, resulting in high learning costs. If a universal template for introducing reinforcement learning into games can be designed, logically dividing the system layer into two parts, the upper layer provides interface services for the game logic content developed by game developers and controls the logic of multiple AI agents at the lower layer. Each AI agent corresponds one-to-one with a game object controlled by the AI (hereinafter referred to as the control object). The lower-level AI agents specifically provide interfaces to associate the relevant features of the corresponding control objects in the game logic content with the game logic. Simultaneously, they call algorithm content to support the game output of each control object at the task level, thereby enabling game logic content written in any language to assist in game generation and avoiding the high reinforcement learning costs required for game developers.
[0053] The technical solution of this application will be described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.
[0054] Based on the problems existing in the above-mentioned prior art Figure 2 This is a schematic diagram of a game development system provided in an embodiment of this application, such as... Figure 2 As shown, the game development system includes: a first module 21 that provides game logic content, a second module 22 that provides algorithm content, and a general template 23 for embedding reinforcement learning in the game.
[0055] Furthermore, in Figure 2 On this basis, Figure 3 A schematic diagram of a general template for embedding reinforcement learning in games, provided as an embodiment of this application. Figure 1 ,like Figure 3 As shown, the general template 23 includes an upper layer and a lower layer.
[0056] The upper layer is provided for game developers, while the lower layer is provided for algorithm developers. The specific logic of the lower layer is not visible to users of the upper layer.
[0057] Optionally, the lower layer provides at least one artificial intelligence agent module, similar to Unity's ML-Agents, which includes the concept of AI Agent. This AI Agent provides similar agentStart, makeDecision, and agentEnd interfaces for algorithm developers to implement.
[0058] Furthermore, the upper layer includes a control module that provides control logic for at least one artificial intelligence agent module, namely the AIGamePlay concept. It provides three interfaces: gameStart, gameTick, and gameEnd, which can be easily integrated with the original game logic.
[0059] That is, the general template 23 includes: at least one artificial intelligence agent module and a control module that provides control logic for the at least one artificial intelligence agent module. Each artificial intelligence agent module corresponds one-to-one with a controlled object, which can be a non-player character (NPC) in a game, etc.
[0060] In terms of implementation, this general template 23 can be adapted to mainstream game development languages such as C++ / C# / Lua / Python. The upper layer only provides logical concepts, serving as middleware between game logic content and the lower layer. Furthermore, it encapsulates reinforcement learning concepts into the lower layer, making them invisible to game developers and reducing their learning cost for embedding. For algorithm developers, there is no need to worry about how game developers embed scripts into the game; they only need to focus on the implementation of a few specific functions in the lower layer (i.e., the interface functions in the subsequent AIAgent). In other words, the general template 23 can serve as an AI-Bridge.
[0061] Optionally, the game logic content is the game content developed by the game developers, which is based on the game's scenes, characters, and other content. This game logic content is placed in the first module 21 and called when used later. The algorithm content is the algorithm developed by the algorithm developers, which is obtained by the algorithm developers after modeling different specific tasks. This algorithm content is placed in the second module 22.
[0062] Furthermore, the execution steps of embedding reinforcement learning in the game using the general template 23 are as follows:
[0063] Step 1: In response to the access command issued by the first module 21, call each artificial intelligence agent module and associate the relevant characteristics of the control objects corresponding to each artificial intelligence agent module in the game logic with the game logic. The relevant characteristics include: the characteristic information of the control object and the characteristic information of the scene in which the control object is located.
[0064] Optionally, during the game generation process, the game logic content in the first module 21 initiates a call to the general template 23, that is, sends an access command to the general template 23. After the general template 23 receives the access command, it initiates a call to each artificial intelligence agent module, associating the relevant characteristics of the control objects corresponding to each artificial intelligence agent module in the game logic content with the game logic.
[0065] Since game logic can be divided into three parts—the beginning, the main logic, and the ending—it can be invoked in three phases.
[0066] Specifically, Figure 4 A schematic diagram of a general template for embedding reinforcement learning in games, provided as an embodiment of this application. Figure 2 ,like Figure 4 As shown, the Gameplay in the first module 21 includes: a start part, a main logic part, and an end part; the control module provides a game start interface, a game progression interface, and a game end interface; each artificial intelligence agent module provides a start interface (agentStart), a decision interface (makeDecision), and an end interface (agentEnd).
[0067] Furthermore, the access commands can also be divided into three types based on the game play: the first access command, the second access command, and the third access command.
[0068] 1. For the beginning of Gameplay: In response to the first access instruction in the first module 21 regarding the beginning of the game logic content, the game start interface is called to call the start interface in all AI agent modules and handle the initialization work of each AI agent module.
[0069] That is, it can be the control objects corresponding to each artificial intelligence agent module, as well as the attributes and layout of the scene where the control objects are located at the beginning of the game.
[0070] 2. For the main logic part of Gameplay: In response to the second access instruction of the main logic part of the game logic content in the first module 21, the game advancement interface is called to call the decision interface in all artificial intelligence agent modules to obtain the action output, skills and rewards of each artificial intelligence agent module.
[0071] That is, the actions, skills, and rewards of the control objects corresponding to each AI agent module and the scenes in which the control objects are located in the main logic of the game.
[0072] The methods for calling the game progression interface include:
[0073] Called during each loop of the main logic section, or called at a preset frequency.
[0074] 3. For the end of the gameplay: In response to the third access instruction regarding the end of the game logic content in the first module 21, the game end interface is called to call the end interface in all AI agent modules to handle the end of each AI agent module and subsequent cleanup work.
[0075] That is, it can be the attributes and layout of the control objects corresponding to each AI agent module and the scene in which the control objects are located at the end of the game.
[0076] Step 2: In the process of associating relevant features with the game logic, each AI agent module calls the algorithm content provided by the second module 22 according to the relevant features to support the game output of the control object corresponding to each AI agent module at the task level.
[0077] Optionally, in the three processes mentioned above that link relevant features to the game logic (the beginning of the game, the main logic of the game, and the end of the game), as game logic content is continuously added to the game logic, the corresponding artificial intelligence agent module needs to perform game output at the task level for the corresponding controlled object.
[0078] The game's output can be actions and rewards.
[0079] For each AI agent module, Figure 5 A schematic diagram of a general template for embedding reinforcement learning in games, provided as an embodiment of this application. Figure 3 ,like Figure 5 As shown, each AI agent module also provides a sendState interface, a doAction interface, and a sendReward interface.
[0080] In the process of associating game logic content with the game's overall logic, the above-mentioned status sending interface, action interface, and reward sending interface are implemented as follows:
[0081] 1. The status sending interface is used to input the relevant features of the control object corresponding to the artificial intelligence agent module in the game logic content into the relevant model in the algorithm content provided by the second module 22 after obtaining the relevant features.
[0082] Specifically, for the beginning, main logic, and end of the game, during the process of the control module calling the agentStart, makeDecision, and agentEnd interfaces of all AI agent modules, the relevant features of the control objects corresponding to the AI agent modules in the game logic content are obtained, the relevant models in the algorithm content provided by the second module 22 are called, and the relevant features are input into the relevant models.
[0083] In other words, the status sending interface implements the logic of constructing the input features of the relevant model in the algorithm content.
[0084] 2. Action interface, used to obtain the output action of the relevant model after the relevant features are input into the relevant model in the algorithm content.
[0085] Specifically, for the beginning, main logic, and end of the game, during the process of the control module calling the agentStart, makeDecision, and agentEnd interfaces of all AI agent modules, after the relevant features are input into the relevant models in the algorithm content, the relevant models in the algorithm content are called to obtain the relevant features as input for the output action.
[0086] In other words, it implements the logic of how the output values of the relevant model are mapped to the specific task level (which can be understood as how the discrete values (0, 1, 2...) output by the neural network are mapped to which skill released by a certain NPC in the game).
[0087] 3. The reward sending interface is used to obtain the reward associated with the output action after the relevant features are input into the relevant model in the algorithm content.
[0088] Specifically, for the beginning, main logic, and end of the game, during the process of the control module calling the agentStart, makeDecision, and agentEnd interfaces of all AI agent modules, after the relevant features are input into the relevant models in the algorithm content, the relevant models in the algorithm content are called to obtain the reward associated with the output action with the relevant features as input.
[0089] In other words, it implements the logic of how to calculate the reward value of the decision to output action A when the relevant model receives the current input S.
[0090] This application provides a game development system comprising: a first module providing game logic content, a second module providing algorithm content, and a general template for embedding reinforcement learning in the game. The general template includes: at least one artificial intelligence agent module and a control module providing control logic for the at least one artificial intelligence agent module. The control module, in response to an access command issued by the first module, calls each artificial intelligence agent module, associating relevant features of the control objects corresponding to each artificial intelligence agent module in the game logic content with the game logic. These relevant features include: feature information of the control object and feature information of the scene in which the control object is located. During the process of associating relevant features with the game logic, each artificial intelligence agent module, based on the relevant features, calls the algorithm content provided by the second module to support the game output at the task level for the control objects corresponding to each artificial intelligence agent module. This technical solution provides an interface to the outside world to call the game logic content developed by game developers. Internally, the control module calls each artificial intelligence agent module to assist in game generation. The reinforcement learning concept is invisible to game developers, reducing the learning cost for game developers when embedding game logic content. Providing an interface to the outside world allows the corresponding interface to be called within the game logic content, making it suitable for games developed in multiple languages.
[0091] Based on the aforementioned game development system (specifically, the game development system provides a first module for providing game logic content, a second module for providing algorithm content, and a general template for embedding reinforcement learning in the game, which includes: at least one artificial intelligence agent module and a control module for providing control logic for at least one artificial intelligence agent module), Figure 6 This is a schematic diagram illustrating an embodiment of the usage method of the game development system provided in this application. Figure 6 As shown, the method of using this system is applied to the general template in the above system, and the method includes the following steps:
[0092] Step 61: In response to the access command issued by the first module, call each artificial intelligence agent module and associate the relevant characteristics of the control objects corresponding to each artificial intelligence agent module in the game logic with the game logic.
[0093] Among them, the relevant characteristics of the controlled object include: the characteristic information of the controlled object and the characteristic information of the scene in which the controlled object is located.
[0094] In this step, when assisting in game generation, the game logic content in the first module initiates a call to the general template, that is, sends an access command to the general template. After the general template receives the access command, it initiates a call to each artificial intelligence agent module, associating the relevant characteristics of the control objects corresponding to each artificial intelligence agent module in the game logic content with the game logic.
[0095] Optionally, the process may include the following three parts:
[0096] First, in response to the first access instruction in the first module regarding the start of the game logic content, the game start interface is called to call the start interface in all AI agent modules and handle the initialization work of each AI agent module.
[0097] In one possible implementation, the game uses an AI Agent to correspond to a controlled object. For example, Table 1 shows the correspondence between AI Agents and the controlled objects in the game. (See Table 1.)
[0098] Table 1:
[0099]
[0100] That is, taking a basketball game as an example, NPC player A in the basketball game corresponds to AI Agent #1, NPC player B in the basketball game corresponds to AI Agent #2, and NPC player C in the basketball game corresponds to AI Agent #3.
[0101] Furthermore, the control module calls the start interface in all AI Agents to handle the initialization work of each artificial intelligence agent module, that is, to initialize NPC player A, NPC player B, and NPC player C according to the beginning part of the game logic content.
[0102] Specifically, Table 2 shows the correspondence between multiple AI Agents and the gameStart interface. (See Table 2 for details.)
[0103] Table 2:
[0104]
[0105] That is, it can be the attributes, layout, etc. of NPC player A, NPC player B, and NPC player C corresponding to each AI agent module at the start of the game.
[0106] Second, in response to the second access instruction of the main logic part of the game logic content in the first module, the game advancement interface is called to call the decision interface in all artificial intelligence agent modules to obtain the action output, skills and rewards of each artificial intelligence agent module.
[0107] In one possible implementation, Table 3 shows the correspondence between multiple AI Agents and tick interfaces. As shown in Table 3:
[0108] Table 3:
[0109]
[0110] That is, it can be the action output, skills and rewards of NPC player A, NPC player B and NPC player C corresponding to each artificial intelligence agent module in the main logic of the game.
[0111] Optionally, the game progression interface can be called in the following ways: during each loop of the main logic section, or at a preset frequency.
[0112] Optionally, within the loop of the main gameplay logic, the control module's tick interface can be configured to be called once per loop iteration or at fixed time intervals.
[0113] In one possible implementation, each cycle can be a round in a game. For example, in a basketball game, the first half is one round and the second half is another round; in a card game, a round is one round when all players have played all their cards.
[0114] Furthermore, at the beginning of each round, the control module's tick interface is called once to invoke the makeDecision interface of each AI Agent, thereby obtaining the action output, skills, and rewards of each AI Agent.
[0115] In another possible implementation, the fixed time interval can be every 20 minutes of the game, for example, the 0th minute, the 20th minute, the 40th minute, etc. in a basketball game; the 0th minute, the 3rd minute, the 6th minute, etc. in a Dou Dizhu game.
[0116] Furthermore, at the start of each moment, the control module's tick interface is called once to invoke the AI Agent's makeDecision interface, thereby obtaining the action output, skills, and rewards of each AI Agent.
[0117] Third, in response to the third access instruction regarding the end of the game logic content in the first module, the game end interface is called to invoke the end interface in all AI agent modules, and to handle the end of each AI agent module and subsequent cleanup work.
[0118] In one possible implementation, Table 4 shows the correspondence between multiple AI Agents and the gameEnd interface. As shown in Table 4:
[0119] Table 4:
[0120]
[0121] That is, it could be the attributes, layout, etc. of NPC player A, NPC player B, and NPC player C corresponding to each AI agent module at the end of the game.
[0122] Step 62: In the process of associating relevant features with the game logic, each AI agent module calls the algorithm content provided by the second module according to the relevant features to support the game output of the control object corresponding to each AI agent module at the task level.
[0123] In this step, during the process of associating the relevant characteristics of the control objects corresponding to each AI agent module in the game logic with the game logic, and the control module calling the game start interface, game progress interface, and game end interface in each AI agent module to associate the relevant characteristics of the control objects with the game logic, the state sending interface, action interface, and reward sending interface inside each AI agent module also call the algorithm content to cooperate with the game output of the control objects at the task level.
[0124] Game outputs include actions and rewards.
[0125] Specifically, the status sending interface, action interface, and reward sending interface are implemented as follows:
[0126] 1. After obtaining the relevant features of the control object corresponding to the artificial intelligence agent module in the game logic content, input the relevant features into the relevant model in the algorithm content provided by the second module;
[0127] The relevant features of the control object corresponding to the artificial intelligence agent module in the game logic are obtained, the relevant model in the algorithm content provided by the second module is called, and the relevant features are input into the relevant model.
[0128] In other words, the status sending interface implements the logic of constructing the input features of the relevant model in the algorithm content.
[0129] 2. After the relevant features are input into the relevant model in the algorithm, obtain the output action of the relevant model.
[0130] After inputting the relevant features into the relevant model in the algorithm, the relevant model in the algorithm is called to obtain the relevant features as input for the output action.
[0131] In other words, it implements the logic of how the output values of the relevant model are mapped to the specific task level (which can be understood as how the discrete values (0, 1, 2...) output by the neural network are mapped to which skill released by a certain NPC in the gameplay).
[0132] 3. After the relevant features are input into the relevant model in the algorithm, the reward associated with the output action is obtained.
[0133] After inputting the relevant features into the relevant model in the algorithm, the relevant model in the algorithm is called to obtain the reward associated with the output action, which is based on the relevant features as input.
[0134] In other words, it implements the logic of how to calculate the reward value of the decision to output action A when the relevant model receives the current input S.
[0135] The method for using the game development system provided in this application embodiment is applied to a general template in a game development system. This method, in response to an access command issued by a first module, calls various artificial intelligence agent modules, associating the relevant features of the control objects corresponding to each artificial intelligence agent module in the game logic content with the game logic. These relevant features include: characteristic information of the control object and characteristic information of the scene in which the control object is located. During the process of associating these relevant features with the game logic, each artificial intelligence agent module, based on these features, calls the algorithm content provided by a second module to support the game output at the task level for the control objects corresponding to each artificial intelligence agent module. The control module provides an interface to the outside world to call the game logic content developed by game developers. Internally, the control module calls various artificial intelligence agent modules to assist in game generation. The reinforcement learning concept is invisible to game developers, reducing the learning cost for game developers when embedding game logic content. Providing an interface to the outside world allows the corresponding interface to be called within the game logic content, making it suitable for games developed in multiple languages.
[0136] Based on the above method embodiments, Figure 7 This is a schematic diagram of the device used in the game development system provided in an embodiment of this application. Figure 7 As shown, the device includes: a response module 71 and a calling module 72;
[0137] The response module 71 is used to respond to the access command issued by the first module, call each artificial intelligence agent module, and associate the relevant features of the control objects corresponding to each artificial intelligence agent module in the game logic content with the game logic. The relevant features include: the feature information of the control object and the feature information of the scene in which the control object is located.
[0138] Module 72 is used to call the algorithm content provided by the second module based on the relevant features during the process of associating relevant features with the game logic of the game, so as to support the game output of the control object corresponding to each artificial intelligence agent module at the task level.
[0139] In one possible design of this application embodiment, the access command includes: a first access command, a second access command, and a third access command;
[0140] Response module 71 is specifically used for:
[0141] In response to the first access instruction in the first module regarding the start of the game logic content, the game start interface is called to call the start interface in all AI agent modules and handle the initialization work of each AI agent module;
[0142] In response to the second access instruction of the main logic part of the game logic content in the first module, the game advancement interface is called to call the decision interface in all AI agent modules to obtain the action output, skills and rewards of each AI agent module;
[0143] In response to the third access instruction regarding the end of the game logic content in the first module, the game end interface is invoked to call the end interfaces in all AI agent modules, handling the end of each AI agent module and subsequent cleanup work.
[0144] Optionally, the game progression interface can be invoked in the following ways:
[0145] Called during each loop of the main logic section;
[0146] or,
[0147] Call at a preset frequency.
[0148] In another possible design of this application embodiment, the response module is specifically used for:
[0149] After obtaining the relevant features of the control object corresponding to the artificial intelligence agent module in the game logic content, the relevant features are input into the relevant model in the algorithm content provided by the second module;
[0150] After the relevant features are input into the relevant model in the algorithm, the output action of the relevant model is obtained;
[0151] After the relevant features are input into the relevant model in the algorithm, the reward associated with the output action is obtained.
[0152] The device for using the game development system provided in this application embodiment can be used to execute the technical solution corresponding to the usage method of the game development system in the above embodiment. Its implementation principle and technical effect are similar, and will not be described again here.
[0153] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software through processing element calls; they can be fully implemented in hardware; or some modules can be implemented through processing element calls in software, while others are implemented in hardware. Moreover, these modules can be fully or partially integrated together, or implemented independently. The processing element mentioned here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through the integrated logic circuits in the hardware of the processor element or through software instructions.
[0154] Figure 8 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Figure 8 As shown, the computing device may include: a processor 81, a memory 82, and computer program instructions stored in the memory 82 and executable on the processor 81.
[0155] The computing device can be a general template for embedding reinforcement learning in games within a game development system.
[0156] Processor 81 executes computer execution instructions stored in memory 82, causing processor 81 to perform the scheme in the above embodiments. Processor 81 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0157] The memory 82 is connected to the processor 81 via the system bus and completes communication between them. The memory 82 is used to store computer program instructions.
[0158] The system bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus.
[0159] The computing device provided in this application embodiment can be used to execute the technical solution corresponding to the usage method of the game development system in the above embodiment. Its implementation principle and technical effect are similar, and will not be repeated here.
[0160] This application also provides a chip for executing instructions, which is used to execute the technical solution of the game development system usage method described in the above embodiments.
[0161] This application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on a computing device, the computing device performs the technical solution of the game development system usage method described in the above embodiments.
[0162] This application also provides a computer program product, including a computer program, which, when executed by a processor, is used to perform the technical solution of the game development system usage method described in the above embodiments.
[0163] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible by general-purpose or special-purpose computing devices.
[0164] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.
[0165] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.
Claims
1. A game development system, characterized in that, The system includes: a first module that provides game logic content, a second module that provides algorithm content, and a general template for embedding reinforcement learning in the game; The general template includes: at least one artificial intelligence agent module and a control module that provides control logic for the at least one artificial intelligence agent module; The control module is used to respond to the access command issued by the first module, call each artificial intelligence agent module, and associate the relevant features of the control objects corresponding to each artificial intelligence agent module in the game logic content with the game logic. The relevant features include: the feature information of the control object and the feature information of the scene in which the control object is located. In the process of associating relevant features with the game logic, each AI agent module calls the algorithm content provided by the second module according to the relevant features to support the game output at the task level of the control object corresponding to each AI agent module.
2. The system according to claim 1, characterized in that, The control module provides a game start interface, a game progression interface, and a game end interface. For each AI agent module, the AI agent module provides a start interface, a decision interface, and an end interface. The access instructions include: a first access instruction, a second access instruction, and a third access instruction. The step of responding to the access command issued by the first module by invoking each AI agent module and associating the relevant characteristics of the control objects corresponding to each AI agent module in the game logic with the game logic includes: In response to the first access instruction regarding the beginning of the game logic content in the first module, the game start interface is invoked to invoke the start interfaces in all AI agent modules and handle the initialization work of each AI agent module; In response to the second access instruction of the main logic part of the game logic content in the first module, the game advancement interface is called to call the decision interface in all artificial intelligence agent modules to obtain the action output, skills and rewards of each artificial intelligence agent module; In response to the third access instruction regarding the end of the game logic content in the first module, the game end interface is invoked to call the end interfaces in all AI agent modules, and to handle the end of each AI agent module and subsequent cleanup work.
3. The system according to claim 2, characterized in that, The methods for invoking the game advancement interface include: Called during each loop of the main logic section; or, Call at a preset frequency.
4. The system according to any one of claims 1-3, characterized in that, The game output includes actions and rewards; each AI agent module provides an interface for sending status, actions, and rewards. For each AI agent module, during the process of associating game logic content with the game's game logic, the sending status interface is used to input the relevant features into the relevant model in the algorithm content provided by the second module after obtaining the relevant features of the control object corresponding to the AI agent module in the game logic content. The action interface is used to obtain the output action of the relevant model after the relevant features are input into the relevant model in the algorithm content; The reward sending interface is used to obtain the reward associated with the output action after the relevant features are input into the relevant model in the algorithm content.
5. A method of using a game development system, characterized in that, The game development system provides a first module for providing game logic content, a second module for providing algorithmic content, and a general template for embedding reinforcement learning in the game. The general template includes at least one AI agent module and a control module providing control logic for the at least one AI agent module. The method includes: In response to the access command issued by the first module, each artificial intelligence agent module is invoked to associate the relevant features of the control objects corresponding to each artificial intelligence agent module in the game logic content with the game logic. The relevant features include: the feature information of the control object and the feature information of the scene in which the control object is located. In the process of associating relevant features with the game logic, each AI agent module calls the algorithm content provided by the second module according to the relevant features to support the game output at the task level of the control object corresponding to each AI agent module.
6. The method according to claim 5, characterized in that, The access instructions include: a first access instruction, a second access instruction, and a third access instruction; The step of responding to the access command issued by the first module by invoking each AI agent module and associating the relevant characteristics of the control objects corresponding to each AI agent module in the game logic with the game logic includes: In response to the first access instruction regarding the beginning of the game logic content in the first module, the game start interface is called to call the start interface in all AI agent modules and handle the initialization work of each AI agent module. In response to the second access instruction of the main logic part of the game logic content in the first module, the game advancement interface is called to call the decision interface in all artificial intelligence agent modules to obtain the action output, skills and rewards of each artificial intelligence agent module; In response to the third access instruction regarding the end of the game logic content in the first module, the game end interface is invoked to call the end interfaces in all AI agent modules, and to handle the end of each AI agent module and subsequent cleanup work.
7. The method according to claim 6, characterized in that, The ways to call the game advancement interface include: Called during each loop of the main logic section; or, Call at a preset frequency.
8. The method according to any one of claims 5-7, characterized in that, The game output includes actions and rewards; the step of calling the algorithm content provided by the second module to support the game output at the task level for the control objects corresponding to each artificial intelligence agent module based on the relevant features includes: After obtaining the relevant features of the control object corresponding to the artificial intelligence agent module in the game logic content, the relevant features are input into the relevant model in the algorithm content provided by the second module; After the relevant features are input into the relevant model in the algorithm content, the output action of the relevant model is obtained; After the relevant features are input into the relevant model in the algorithm content, the reward associated with the output action is obtained.
9. A computing device, characterized in that, include: Processor, memory, and computer program instructions stored in said memory and executable on the processor; When the processor executes the computer program instructions, it implements a method of using the game development system as described in any one of claims 5-8.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method of using the game development system as described in any one of claims 5-8.