Intelligent agent processing method and device

Abstract

The embodiment of the present application provides an agent processing method and device, the method comprising: performing L environmental interaction processes through a first agent model to obtain L experience data, each experience data comprising: previous state data, action data corresponding to the previous state data, reward data corresponding to the action data, and next state data corresponding to the action data, where L is an integer greater than or equal to 1. The first agent model is trained according to the multiple experience data to obtain a target agent. The target agent is controlled to automatically make action decisions in the game. The state data is processed by the agent model to obtain action data, and then the agent model is trained according to the state data, action data, and reward data, so that the agent model can learn according to real battle data to obtain a target agent, and then the game operation is determined based on the target agent, thereby effectively improving the accuracy of the game operation output by the agent.

Description

Technical Field

[0001] The embodiments of the present application relate to computer technology, and more particularly to an intelligent agent processing method and device. Background Art

[0002] With the continuous development of the gaming field, setting up intelligent agents in games to operate on behalf of users or to fight against users has become a very common operating method.

[0003] At present, the intelligent agents in the prior art usually use behavior trees or state machines to select game operations. For example, a specific game scene can be designed in advance, and then the intelligent agent performs the specified game actions based on the current game scene.

[0004] However, the implementation method of presetting the game scenes and corresponding game actions will result in very limited actions of the agent, thereby failing to guarantee the accuracy of the game operations selected by the agent. Summary of the invention

[0005] The embodiments of the present application provide an agent processing method and device to overcome the problem that the accuracy of the game operation selected by the agent cannot be guaranteed.

[0006] In a first aspect, an embodiment of the present application provides an agent processing method, comprising:

[0007] Performing L environmental interaction processes through the first agent model to obtain L experience data, each of which includes: previous state data, action data corresponding to the previous state data, reward data corresponding to the action data, and next state data corresponding to the action data, where L is an integer greater than or equal to 1;

[0008] Training the first agent model according to the plurality of experience data to obtain a target agent;

[0009] Controlling the target agent to automatically make action decisions within the game;

[0010] Among them, the environmental interaction processing includes: inputting the previous state data into the first intelligent agent model, the first intelligent agent model determines the action data corresponding to the previous state data, and feeds the action data back to the environmental engine to obtain the next state data corresponding to the action data and the reward data corresponding to the action data.

[0011] In a possible design, training the first agent model according to the plurality of experience data to obtain a target agent includes:

[0012] Obtaining previous state data, action data, reward data and next state data in the experience data;

[0013] According to the action network of the first agent model and the previous state data, obtaining action data currently output by the action network;

[0014] According to the value network of the first agent model, obtaining predicted reward data corresponding to the action data currently output by the action network;

[0015] Adjusting parameters of the first agent model according to the reward data in the experience data, the predicted reward data, the action data in the experience data, and the action data currently output by the action network;

[0016] For each of the experience data, the above-mentioned operation of adjusting the parameters of the first agent model is performed respectively until the training end condition of the first agent model is reached, thereby obtaining the target agent.

[0017] The adjusting the parameters of the first agent model according to the reward data in the experience data, the predicted reward data, the action data in the experience data and the action data currently output by the action network comprises:

[0018] Determine a strategy loss function value according to the reward data in the experience data, the predicted reward data, the action data in the experience data, and the action data currently output by the action network;

[0019] Determining a value loss function value based on the reward data in the experience data and the predicted reward data;

[0020] Determine the entropy loss function value according to the action data output by the current action network;

[0021] Summing the strategy loss function value, the value loss function value, and the entropy loss function value to obtain a target loss function value;

[0022] In one possible design, back propagation is performed based on the target loss function value to adjust the parameters of the first agent model.

[0023] In one possible design, the method further includes:

[0024] The previous state data is encoded to obtain encoding information corresponding to the previous state data.

[0025] In one possible design, the previous state data includes at least one of the following: numerical data, position data, and category data;

[0026] The encoding process of the previous state data to obtain encoding information corresponding to the previous state data includes:

[0027] Normalizing the numerical data to obtain coding information corresponding to the numerical data; and / or,

[0028] Performing one-bit effective encoding on the position data to obtain encoding information corresponding to the position data; and / or,

[0029] The type data is subjected to network initialization encoding, and feature data corresponding to the type data is learned to obtain encoding information corresponding to the type data.

[0030] In a possible design, feeding back the action data to an environment engine to obtain next state data corresponding to the action data includes:

[0031] Feeding the action data back to the environment engine, and determining the intermediate state data corresponding to the action data through the environment engine;

[0032] Inputting the intermediate state data into a second agent model to obtain action data output by the second agent model;

[0033] Determining, by the environment engine, state data corresponding to the action data output by the second agent model;

[0034] The state data corresponding to the action data output by the second agent model is determined as the next state data returned by the environment engine.

[0035] In one possible design, the second agent model and the first agent model are the same agent model;

[0036] Alternatively, the first agent model is the latest version of the agent model, and the second agent model is the historical version of the agent model.

[0037] In a possible design, controlling the target agent to automatically make action decisions in the game includes:

[0038] Inputting the state data to be processed in the game into the target agent, and obtaining the behavior data output by the target agent;

[0039] According to the behavior data output by the target intelligent agent, the target intelligent agent is controlled to automatically make action decisions corresponding to the behavior data in the game.

[0040] In a second aspect, an embodiment of the present application provides an intelligent agent processing device, including:

[0041] A processing module, configured to perform L environmental interaction processes through a first agent model to obtain L experience data, each of which includes: previous state data, action data corresponding to the previous state data, reward data corresponding to the action data, and next state data corresponding to the action data, where L is an integer greater than or equal to 1;

[0042] A training module, used for training the first agent model according to the plurality of experience data to obtain a target agent;

[0043] A control module, used to control the target agent to automatically make action decisions in the game;

[0044] Among them, the environmental interaction processing includes: inputting the previous state data into the first intelligent agent model, the first intelligent agent model determines the action data corresponding to the previous state data, and feeds the action data back to the environmental engine to obtain the next state data corresponding to the action data and the reward data corresponding to the action data.

[0045] In a possible design, the training module is specifically used for:

[0046] Obtaining previous state data, action data, reward data and next state data in the experience data;

[0047] According to the action network of the first agent model and the previous state data, obtaining action data currently output by the action network;

[0048] According to the value network of the first agent model, obtaining predicted reward data corresponding to the action data currently output by the action network;

[0049] Adjusting parameters of the first agent model according to the reward data in the experience data, the predicted reward data, the action data in the experience data, and the action data currently output by the action network;

[0050] For each of the experience data, the above-mentioned operation of adjusting the parameters of the first agent model is performed respectively until the training end condition of the first agent model is reached, thereby obtaining the target agent.

[0051] In a possible design, the training module is specifically used for:

[0052] Determine a strategy loss function value according to the reward data in the experience data, the predicted reward data, the action data in the experience data, and the action data currently output by the action network;

[0053] Determining a value loss function value based on the reward data in the experience data and the predicted reward data;

[0054] Determine an entropy loss function value according to the action data output by the current action network;

[0055] Summing the strategy loss function value, the value loss function value, and the entropy loss function value to obtain a target loss function value;

[0056] Back propagation is performed according to the target loss function value to adjust the parameters of the first agent model.

[0057] In one possible design, the processing module is further used for:

[0058] The previous state data is encoded to obtain encoding information corresponding to the previous state data.

[0059] In one possible design, the previous state data includes at least one of the following: numerical data, position data, and category data;

[0060] The processing module is specifically used for:

[0061] Normalizing the numerical data to obtain coding information corresponding to the numerical data; and / or,

[0062] Performing one-bit effective encoding on the position data to obtain encoding information corresponding to the position data; and / or,

[0063] The type data is subjected to network initialization encoding, and feature data corresponding to the type data is learned to obtain encoding information corresponding to the type data.

[0064] In a possible design, the processing module is specifically used for:

[0065] Feeding the action data back to the environment engine, and determining the intermediate state data corresponding to the action data through the environment engine;

[0066] Inputting the intermediate state data into a second agent model to obtain action data output by the second agent model;

[0067] Determining, by the environment engine, state data corresponding to the action data output by the second agent model;

[0068] The state data corresponding to the action data output by the second agent model is determined as the next state data returned by the environment engine.

[0069] In one possible design, the second agent model and the first agent model are the same agent model;

[0070] Alternatively, the first agent model is the latest version of the agent model, and the second agent model is the historical version of the agent model.

[0071] In a possible design, the control module is specifically used for:

[0072] Inputting the state data to be processed in the game into the target agent, and obtaining the behavior data output by the target agent;

[0073] According to the behavior data output by the target intelligent agent, the target intelligent agent is controlled to automatically make action decisions corresponding to the behavior data in the game.

[0074] In a third aspect, an embodiment of the present application provides an intelligent agent processing device, including:

[0075] Memory, used to store programs;

[0076] A processor is used to execute the program stored in the memory. When the program is executed, the processor is used to execute the method described in the first aspect and any one of the various possible designs of the first aspect.

[0077] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enables the computer to execute the method described in the first aspect and any one of the various possible designs of the first aspect.

[0078] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, it implements the method described in the first aspect and any one of the various possible designs of the first aspect.

[0079] The embodiment of the present application provides an agent processing method and device, the method comprising: performing L environmental interaction processing through a first agent model to obtain L experience data, each experience data comprising: the previous state data, the action data corresponding to the previous state data, the reward data corresponding to the action data, and the next state data corresponding to the action data, and L is an integer greater than or equal to 1. The first agent model is trained according to the multiple experience data to obtain a target agent. The target agent is controlled to automatically make action decisions in the game. Among them, the environmental interaction processing comprises: inputting the previous state data into the first agent model, the first agent model determines the action data corresponding to the previous state data, and feeds the action data back to the environment engine to obtain the next state data corresponding to the action data and the reward data corresponding to the action data. By processing the state data according to the agent model to obtain the action data, the agent model is trained according to the state data, the action data, and the reward data corresponding to the action data, so that the agent model can learn according to the real battle data, and the target agent is obtained after the training is completed, and the game operation is determined based on the target agent, so that the accuracy of the game operation output by the agent can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0081] Figure 1 A schematic diagram of the implementation of the intelligent agent provided in the embodiment of the present application;

[0082] Figure 2 A flowchart of an intelligent agent processing method provided in an embodiment of the present application;

[0083] Figure 3 The process of the intelligent agent processing method provided in the embodiment of the present application Figure 2 ;

[0084] Figure 4 A schematic diagram of status data provided by an embodiment of the present application;

[0085] Figure 5 A schematic diagram of a game state provided in an embodiment of the present application;

[0086] Figure 6 A schematic diagram of a single battle process provided in an embodiment of the present application;

[0087] Figure 7A schematic diagram for implementing the determination of next state data provided in an embodiment of the present application;

[0088] Figure 8 A schematic diagram of implementing multiple battles provided in an embodiment of the present application;

[0089] Fig. 9 A schematic diagram of implementing multi-threaded parallel processing provided in an embodiment of the present application;

[0090] Fig.10 A schematic diagram of the implementation of the empirical data provided in the embodiment of the present application;

[0091] Fig.11 A schematic diagram of the structure of an intelligent agent processing device provided in an embodiment of the present application;

[0092] Fig.12 A schematic diagram of the hardware structure of the intelligent agent processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0093] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0094] In order to better understand the technical solution of the present application, the background technology involved in the present application is firstly introduced in further detail:

[0095] With the continuous development of the gaming field, setting up intelligent agents in games to operate on behalf of users or to fight against users has become a very common operating method.

[0096] Among them, the intelligent agent can be understood as game artificial intelligence (AI). Traditional game AI mostly uses behavior trees or state machines. For example, a specific game scene can be designed in advance, and the intelligent agent can perform specified actions when encountering the scene. In early game AI-related work, the most common methods are planning methods such as tree search and its variants.

[0097] Among them, behavior trees or state machines are more traditional methods. For example, decision trees require manual writing of rules, have a single scenario, and are time-consuming. Game agents are relatively rigid and cannot engage in self-game.

[0098] Therefore, the game operation is selected based on the scenes and actions designed according to artificial experience, but the adapted game scenes are limited and not intelligent enough, and can hardly be applied to the self-game battle of the game, resulting in a lack of accuracy in the selection of the intelligent agent.

[0099] In addition, another important direction of game AI research is related to game theory. In many card games such as Texas Hold'em, Mahjong, and Landlord, the information of a game is not completely visible to the decision maker. When the game AI performs calculations, it needs to make some guesses about the opponent's state and strategy, which further increases the complexity of the problem. There is no way to directly deal with it using tree search methods.

[0100] Most turn-based games are information asymmetric games. Players on both sides cannot see each other's hidden information. The intelligent agent needs to deduce the enemy's hidden information through the opponent's actions and help us make decisions.

[0101] At present, the existing implementation solutions based on game theory can be, according to the idea of ​​reinforcement learning, the main research scope of game theory is multi-agent combat, for example, the Neural Fictitious Self Play (NFSP) method can combine reinforcement learning and game theory ideas, and the intelligent agent aims to learn the current strategy and the opponent's average strategy.

[0102] Regarding game theory solutions, the NFSP algorithm performs well in Landlord and Texas Hold'em. It can perform well by guessing the opponent's strategic habits through game theory and combining it with scene information. In turn-based self-games, the goal is to allow the game AI to learn how to use character skills through self-games between both teams. The game component is relatively low, and more learning of game strategies is required. Therefore, the learning efficiency of the NFSP algorithm agent is low, and it is difficult to learn generalized solutions.

[0103] In summary, in the current prior art, when the intelligent agent is implemented based on a behavior tree or a state machine, there is a problem that the output of the intelligent agent is not accurate. In addition, when the intelligent agent learns based on a game theory solution, there is a problem that the model lacks generalization. Based on the problems in the prior art, this application proposes the following technical ideas:

[0104] By constructing an intelligent agent model, and then processing the state data according to the intelligent agent model to obtain action data, the intelligent agent model is then trained based on the state data, action data, and reward data corresponding to the action data, so that the intelligent agent model can learn based on real battle data. After the training is completed, the target intelligent agent is obtained, and then the game operation is determined based on the target intelligent agent, thereby effectively improving the accuracy of the game operation output by the intelligent agent.

[0105] The following is an introduction to the agent processing method provided by the present application in conjunction with specific embodiments. Figure 1 The working process of the intelligent agent mentioned in this application is explained. Figure 1 A schematic diagram of the implementation of the intelligent agent provided in the embodiment of the present application.

[0106] like Figure 1 As shown, the agent can select corresponding game actions based on the game state, where the game state can be understood as the current game situation, for example, it can include health points, attack power, defense value, critical hit rate, action position, etc. The game action can be understood as a game operation, such as casting a skill, or performing a normal attack.

[0107] After selecting a game action, the agent can input the game action into the game environment, and the game environment determines the next game state based on the input game action. For example, if the current game positioning is an attack action against the enemy, the game environment can update the enemy's health value based on the game action and obtain the corresponding next game state.

[0108] Afterwards, the game environment can send the next game state to the agent, and the agent selects the next game operation based on the next game state. The environment engine then determines the next game state based on the new game operation, and repeats this process until the game ends or the game is completed.

[0109] Based on the above introduction, it can be determined that the intelligent agent can select game actions. The purpose of this application is to ensure that the intelligent agent can select actions with higher accuracy, or can be understood as more reasonable actions. Therefore, when training the intelligent agent, the intelligent agent can be trained with the purpose of selecting actions with better effects.

[0110] The following is an explanation of the implementation method of the training of the intelligent agent in conjunction with specific embodiments. It is worth noting that the execution subject of each embodiment in the present application can be, for example, a server, a processor, a microprocessor and other devices with data processing functions. This embodiment does not impose any restrictions on this and it can be selected according to actual needs as long as it has the data processing function.

[0111] Figure 2 A flowchart of the intelligent agent processing method provided in an embodiment of the present application.

[0112] like Figure 2 As shown, the method includes:

[0113] S201. Perform L environmental interaction processes through the first intelligent agent model to obtain L experience data, each experience data includes at least one of the following: previous state data, action data corresponding to the previous state data, reward data corresponding to the action data, and next state data corresponding to the action data, where L is an integer greater than or equal to 1.

[0114] In this embodiment, in order to train an intelligent agent that outputs reasonable actions, it is necessary to first build an intelligent agent model, wherein the intelligent agent model is a model that determines game actions based on the game state, and includes a series of processing logics within the intelligent agent model. This embodiment does not limit its internal implementation logic and specific implementation methods, and it can be selected according to actual needs, as long as the intelligent agent model can process according to the game state and output game actions.

[0115] In one possible implementation, the agent model in the embodiment of the present application may be a deep learning-based model, for example, it may be based on a proximal policy optimization (PPO) algorithm.

[0116] It is understandable that the currently determined intelligent agent model is a preliminary intelligent agent, and its accuracy and effectiveness cannot be guaranteed. Therefore, the intelligent agent model needs to be trained. The first intelligent agent model in this embodiment is the intelligent agent model that currently needs to be trained.

[0117] In one possible implementation, in order to ensure the effectiveness of training, the first intelligent agent model can be trained based on real combat data in this embodiment. Therefore, in this embodiment, L environmental interaction processes can be performed through the first intelligent agent model to obtain L experience data, where the L experience data are real combat data.

[0118] In one possible implementation, environmental interaction processing may include: inputting the previous state data into the first intelligent agent model, the first intelligent agent model determining the action data corresponding to the previous state data, feeding the action data back to the environmental engine, and obtaining the next state data corresponding to the action data and the reward data corresponding to the action data.

[0119] Among them, the previous state data (state) may, for example, include the health value, experience value, position, action position, survival status, buff (gain) information, etc. of each virtual character in the current game state, and the action data (action) may be a game action selected by the current first intelligent agent model, such as casting a skill of a virtual character, or casting a normal attack through a virtual character, etc. This embodiment does not limit the specific implementation method of the previous state data and action data, which can be selected according to actual needs, as long as the previous state data can indicate the current game state and the action data can indicate the game action.

[0120] And then the environment engine can return the next state data (state`) corresponding to the action data and the reward data (reward) corresponding to the action data.

[0121] It is understandable that after performing a certain game action in the game, the state of the game will change. For example, after an attack action, the life value of the enemy virtual object may change. For another example, after a treatment action, the life value of our virtual object may change. For another example, the corresponding game action will cause the transfer of buffs, the survival or death of virtual objects to change, etc. In short, game actions will cause changes in the game state, so the action data can correspond to the next state data.

[0122] In this embodiment, the next state data corresponding to the action data is not limited, which depends on the specific implementation of the game. In one possible implementation, the action data can be input into the game environment, and the action data is processed by the game environment to obtain the next state data corresponding to the action data, wherein the game environment can be understood as an environment engine, or a game engine, for example.

[0123] Alternatively, a specific processor may be used to process the action data, so as to obtain next state data corresponding to the action data, which is not particularly limited in this embodiment.

[0124] Also, in this embodiment, reward data corresponding to the action data can be obtained. It can be understood that the first agent model in this embodiment can be a model based on reinforcement learning, wherein reinforcement learning is a learning method in which the agent learns in a "trial and error" manner, and the rewards obtained through interaction with the environment guide the actions of the agent. The goal is to enable the agent to obtain the maximum reward, so the reward data can be used to guide the actions of the agent.

[0125] In this embodiment, reward data corresponding to the action data can be obtained. In one possible implementation, for example, the game environment can determine the corresponding reward data based on the current action data, or it can also be processed by a specific processor to determine the reward data corresponding to the action data. This embodiment does not limit this.

[0126] In the process of determining the reward data, for example, based on whether the action data corresponds to whether the enemy object is killed, or whether the damage amount corresponding to the action data is sufficient, or whether the action data leads to winning the game, etc., the corresponding reward data can be set according to these indicators in different dimensions to determine the reward data corresponding to the action data. The specific setting method and setting value of the reward data can be selected according to actual needs, and this embodiment does not limit this.

[0127] In this embodiment, when the model is allowed to learn based on real combat data, a large amount of data is inevitably required. Therefore, after obtaining the next state data, the next state data can be continuously input into the first intelligent agent model, and then the above operation is repeated.

[0128] That is to say, during the i-th environmental interaction processing in this embodiment, the previous state data input to the first intelligent agent model is the next state data in the i-1th experience data, that is to say, the obtained new state data is continuously input into the first intelligent agent model to obtain the corresponding action data, and then the above process is continuously executed in a loop.

[0129] Specifically, the following operations are repeatedly performed: inputting the previous state data into the first agent model to obtain the action data corresponding to the previous state data output by the first agent model; obtaining the next state data corresponding to the action data and the reward data corresponding to the action data, thereby obtaining multiple experience data.

[0130] Among them, the more times the execution is repeated, the more experience data will be obtained. The specific number of repeated executions L can be selected according to actual needs. For example, a preset number of times can be set. When the number of repetitions reaches the preset number, the execution is stopped to obtain multiple experience data; or a preset data volume can be set. When the amount of collected experience data reaches the preset data volume, the execution is stopped to obtain multiple experience data. This embodiment does not limit the specific conditions for ending the repetition, which can be selected according to actual needs.

[0131] S202. Train the first agent model according to multiple experience data to obtain a target agent.

[0132] After obtaining a plurality of experience data, the first intelligent agent model may be trained. In a possible implementation, the plurality of experience data obtained above may be used as training data to train the first intelligent agent model.

[0133] It is understandable that the process of training a model is similar to the process of applying the model. The difference is that during the training of the model, it is necessary to learn based on empirical data to adjust the parameters of the model so that the model can develop in a better direction.

[0134] In one possible implementation, during the training process of the first intelligent agent model, a loss function can be determined based on empirical data, and then the parameters of the first intelligent agent model can be adjusted based on the loss function to make the model develop in a more optimal direction, thereby realizing the training of the model.

[0135] After the training of the first agent model is completed, the target agent can be obtained, wherein the target agent is the agent required in this embodiment and can output a better action.

[0136] In an optional implementation, after obtaining the target intelligent agent, the game action can be selected based on the target intelligent agent. In a possible implementation, the state data to be processed can be input into the target intelligent agent, and the action data corresponding to the state data to be processed can be determined based on the output of the target intelligent agent.

[0137] S203, controlling the target intelligent agent to automatically make action decisions within the game.

[0138] After obtaining the target intelligent agent, in this embodiment, the target intelligent agent can be controlled to automatically make action decisions based on the status within the game. Because the target intelligent agent in this embodiment has undergone the training process introduced above, the accuracy of the action decisions made can be guaranteed.

[0139] In one possible implementation, for example, the status data to be processed in the current game can be input into the target intelligent agent, and the target intelligent agent can perform analysis based on the status data to output behavior data corresponding to the status data to be processed. Then, based on the behavior data, the target intelligent agent can be controlled to automatically execute the action decision corresponding to the behavior data in the game, thereby realizing automatic game selection and operation based on the target intelligent agent.

[0140] The agent processing method provided in the embodiment of the present application includes: performing L environmental interaction processing through the first agent model to obtain L experience data, each experience data includes: the previous state data, the action data corresponding to the previous state data, the reward data corresponding to the action data, and the next state data corresponding to the action data, and L is an integer greater than or equal to 1. The first agent model is trained according to multiple experience data to obtain a target agent. The target agent is controlled to automatically make action decisions in the game. Among them, the environmental interaction processing includes: inputting the previous state data into the first agent model, the first agent model determines the action data corresponding to the previous state data, and feeds the action data back to the environment engine to obtain the next state data corresponding to the action data and the reward data corresponding to the action data. By processing the previous state data according to the agent model to obtain the action data, and then training the agent model according to the previous state data, the action data and the reward data corresponding to the action data, so that the agent model can learn according to the real battle data, and the target agent is obtained after the training is completed. Then, the game operation is determined based on the target agent, so that the accuracy of the game operation output by the agent can be effectively improved.

[0141] Based on the above embodiments, Figures 3 to 10 The intelligent agent processing method provided by this application is further introduced in detail. Figure 3 The process of the intelligent agent processing method provided in the embodiment of the present application Figure 2 , Figure 4 A schematic diagram of the status data provided in the embodiment of the present application, Figure 5 A schematic diagram of a game state provided in an embodiment of the present application, Figure 6 A schematic diagram of a single battle process provided in an embodiment of the present application, Figure 7 A schematic diagram for implementing the determination of next state data provided in an embodiment of the present application, Figure 8 A schematic diagram of implementing multiple battles provided in an embodiment of the present application, Fig. 9 A schematic diagram of implementing multi-threaded parallel processing provided in an embodiment of the present application, Fig.10 A schematic diagram of the implementation of the empirical data provided in the embodiments of the present application.

[0142] like Figure 3 As shown, the method includes:

[0143] S301. Perform encoding processing on the previous state data to obtain encoding information corresponding to the previous state data.

[0144] It is understandable that when inputting the game status data into the intelligent agent model, the game status information should be input in a way that the neural network can understand. Common input methods include directly inputting each frame of the image. However, unlike real-time combat games, each frame has a very intuitive image meaning. In turn-based games, the images are rendered more as special effects animations. Therefore, for turn-based games, it is difficult for the neural network to perceive the meaning behind each frame of the image.

[0145] However, turn-based games can represent a lot of game information through encoding, so in this embodiment, the previous state data can be encoded to obtain the encoded information corresponding to the previous state data, wherein the previous state data can be the direct game state, and the encoded information can be information that can be understood by the neural network.

[0146] In a possible implementation, the previous state data may include at least one of the following: numerical data, location data, and category data.

[0147] The following describes the implementation methods of determining the encoding information corresponding to each different state data and each initial state data respectively:

[0148] For example, you can combine Figure 4 and Figure 5 To understand, first explain the numerical data, such as Figure 4 As shown, the numerical data can be understood as directly numerical data, such as health, mana, attack power, defense value, critical hit rate and other numerical data.

[0149] See also Figure 5 , assuming Figure 5 The game scene at a certain moment is shown in Figure 5 Assume that the current game scene is a battle between three enemy heroes and three our heroes, where the heroes can be understood as virtual objects.

[0150] like Figure 5 As shown, the positions of the enemy heroes are 501, 502, and 503 respectively. The enemy hero A is at position 501, the enemy hero B is at position 502, and the enemy hero C is at position 503.

[0151] Also, the positions of our heroes are shown as 504, 505, and 506 respectively. At position 504 is our hero D, at position 505 is our hero E, and at position 506 is our hero F.

[0152] Can be combined Figure 5 Understand numerical data such as Figure 5As shown, for example, the current enemy hero A has a health of 8800, a mana of 47, an attack power of 1300, a defense value of 760, and a critical strike rate of 12%. It can be seen that these status data are directly numerical values, which can be directly perceived by the neural network. Therefore, in this embodiment, when encoding the numerical data, in a possible implementation method, the numerical data can be normalized to obtain the encoding information corresponding to the numerical data.

[0153] Among them, the normalization process can make the neural network ignore the level information of the lineup and only focus on the ratio of numerical values, thus making the input more generalizable and easier to train and converge.

[0154] In one possible implementation, for a player vs. player (PVP) battle in a turn-based game, once the lineups of both sides are generated, the basic attributes of all heroes on the field are known, so the normalization process can, for example, be performed according to the maximum value of the attribute value among all heroes in the current lineup.

[0155] In the actual implementation process, the specific processing method of normalization can be selected according to actual needs, as long as it can ensure that the numerical data can be normalized within a certain range.

[0156] Next, the location data is explained, such as Figure 4 As shown, the position data can be understood as data indicating position information such as action position, surviving heroes, current action hero, etc.

[0157] See also Figure 5 , assuming that the enemy has three positions at the current moment, namely positions 501, 502, and 502, and our side has three positions, namely 504, 505, and 506. For example, if the action position at the current moment is position 501, then "the action position at the current moment is position 501" can be used as a position data, but the neural network may not be able to directly understand this type of data. Therefore, in this embodiment, the position data can be encoded.

[0158] In a possible implementation, the position data may be encoded with one bit of validity, thereby obtaining encoding information corresponding to the position data.

[0159] Among them, one-hot encoding mainly uses an N-bit state register to encode N states, each state has its own independent register bit, and only one bit is valid at any time.

[0160] For example, for different positions, operations of filling 0 and 1 are performed according to whether the hero is taking action or alive. For example, for each position, 1 can be used to indicate that the hero is alive, and 0 can be used to indicate that the hero is dead, so that the neural network can perceive that 1 represents survival and 0 represents death, thereby realizing the representation of surviving heroes; for another example, currently for each position, 1 is used to indicate that the hero is taking action, and 0 is used to indicate that the hero is not taking action, so that the neural network can perceive that 1 represents action and 0 represents no action, thereby realizing the representation of the action position, wherein the specific encoding method, and the meaning of the position data represented by the encoding method, can be selected according to actual needs, and this embodiment does not impose any special restrictions on this.

[0161] Finally, the category data is explained, such as Figure 4 As shown, the category data may include, for example, the hero's buff, the hero's skill identifier (ID), the hero identifier, and other category information.

[0162] Specifically, for data such as hero buffs, the information on the game screen is an animation or icon, and the program returns a buff_id (gain identifier). However, it is difficult for the neural network to perceive that the animation or buff_id corresponds to the actual meaning of the buff.

[0163] In this regard, the type information can be converted into position information, for example, letting the neural network know that the opponent's hero at position 501 is an output hero.

[0164] In a possible implementation, a more generalized approach is to encode the category data for network initialization, such as embedding initialization encoding. As the number of game matches increases, the intelligent body will learn the characteristic data of these category data, which is represented by the input vector. For example, the local xxx hero is the output hero, and the corresponding vector representing the output hero is learned, thereby obtaining the encoding information corresponding to the category data.

[0165] Therefore, in this embodiment, the numerical data and position data can be processed accordingly to obtain corresponding coding information, and the type data can be embedded in the initialization code and provided to the intelligent agent for learning, so as to learn and obtain the coding information corresponding to the type data.

[0166] In addition, it is worth mentioning that in turn-based PVP games, it is usually necessary to configure the lineup, for example, Figure 5 To understand, the enemy needs three heroes and our side also needs three heroes. Only after completing the lineup configuration can the game process be realized.

[0167] When determining the lineup corresponding to the first intelligent agent model, in one possible implementation method, for example, the configuration can be created manually, or the configuration can be created based on the popular online lineup, or the configuration of the lineup can be created based on a customized script, wherein the customized script can, for example, determine the configured lineup based on the hero's positioning and attribute growth value rules. This embodiment does not limit the internal logic of the customized script, which can be selected according to actual needs, as long as the lineup configuration can be implemented according to the hero's positioning and attribute growth value rules.

[0168] Through the above-mentioned lineup configuration implementation method, the problem of cumbersome and time-consuming configuration of lineup attributes in the self-game of the turn-based multi-lineup model is solved, and the rationality of the lineup can be guaranteed through the above-mentioned method, thereby providing good data quality for generalized turn-based intelligent agents to improve the effect of model learning.

[0169] S302: Input the coded information into the first agent model to obtain action data corresponding to the previous state data output by the first agent model.

[0170] After obtaining the encoding information corresponding to the previous state data, the encoding information can be input into the first intelligent agent model. The first intelligent agent model can effectively process the encoding information, thereby outputting the action data corresponding to the previous state data.

[0171] Therefore, in this embodiment, by inputting the coded information corresponding to the previous state data into the first agent model, the first agent model can effectively perceive the meaning of the previous state data for effective analysis, processing and learning.

[0172] S303: Determine the intermediate state data corresponding to the action data through the environment engine.

[0173] In this embodiment, after the first agent model outputs action data, it is necessary to determine the next game state corresponding to the action data, so as to subsequently determine new action data. For example, Figure 6 Understand the logic of a single battle in this embodiment.

[0174] like Figure 6 As shown in the figure, the logic of each single battle is: the first-generation model initializes the model parameters, the game environment first returns the initial state of the game to the first-generation model, the first-generation model selects the game action corresponding to the initial state according to the neural network, and then returns the action data of the selected game action to the game environment to obtain the next game state returned by the game environment, and then continues to select new game actions according to the next game state, and repeats until the data collection is completed or the game ends.

[0175] Understandably, Figure 6The next game state shown in the figure is both the next state data and the state data to be input into the first agent model, because in this embodiment, the next state data is continuously input into the first agent model to obtain multiple experience data.

[0176] Among them, each time before the game environment outputs the next game state, it determines whether the current game is over. If the current game is not over, the next game state is output. If the current game is over, a new game can be started, or data collection can be stopped.

[0177] Therefore, the first agent model in this embodiment needs to obtain the next game state returned by the game environment. At the same time, because the first agent model in this embodiment needs to play self-game, that is, the enemy played by the first agent model is also an agent model, so in fact the next game state returned by the game environment is obtained after being processed by the enemy's agent model. The implementation process of the game environment returning to the next game state is explained below.

[0178] First, see Figure 7 It is understood that the first agent model can output action data to the environment engine. In order to effectively distinguish, the action data output by the first agent model is referred to as the first action data. The environment engine in this embodiment can be the game environment introduced above, so that the environment engine determines the corresponding intermediate state data according to the current first action data of the first agent model. It can be understood that the intermediate state data in this embodiment is the state data directly generated by the first action data of the first agent model.

[0179] Combination Figure 5 To understand, for example, the first action data of the first intelligent agent model is to control our hero 504 to launch a normal attack on the enemy hero 502. The damage value of this normal attack is 1300. The intermediate state data corresponding to this first action data is the game state data after the health of the enemy hero 502 is reduced by 1300.

[0180] S304: Input the intermediate state data into the second agent model to obtain the action data output by the second agent model.

[0181] The game in this embodiment can be a turn-based game. In a turn-based game, our side and the enemy side take turns to act, one side takes one turn. In this embodiment, after the first agent model determines the action data, it is the turn of the second agent model of the enemy side to act. Figure 7 Therefore, the current state data, that is, the intermediate state data, needs to be input into the second agent model. When the second agent model acts, it will process according to the intermediate state data and output action data, such as Figure 7As shown, the second action data output by the second intelligent agent model can be obtained.

[0182] Among them, the second agent model is the enemy's agent model. Because the agent in this embodiment needs to play self-game, it can be understood that the second agent model and the first agent model actually have the same processing logic.

[0183] In a possible implementation, the second agent model and the first agent model are the same agent model, that is, the agent models of both parties are exactly the same.

[0184] Alternatively, the first agent model is the latest version of the agent model, and the second agent model is the historical version of the agent model.

[0185] For example, if the current intelligent agent model has been trained to the third generation, then the first intelligent agent model is, for example, a third-generation intelligent agent model, and the second intelligent agent model is, for example, a first-generation or second-generation intelligent agent model.

[0186] In one possible implementation, when loading the agent model for combat, each agent has an 80% chance of loading its latest model and a 20% chance of loading the previous model to enhance training stability, thereby ensuring that the agent as a whole evolves in a direction of increasing strength. This allows the agent to focus not only on whether it can defeat the current version of the model, but also on whether it is stronger than the opponent's previous version of the agent, ensuring that the model update is moving in a more optimal direction.

[0187] For example, it can prevent the problem that the third-generation intelligent agent can defeat the second-generation intelligent agent, the second-generation intelligent agent can defeat the first-generation intelligent agent, but the third-generation intelligent agent cannot defeat the first-generation intelligent agent, thus ensuring that the intelligent agent can evolve in a better direction.

[0188] For example, you can refer to Figure 8 To understand, Figure 8 What is illustrated in FIG. 1 is a self-game process involving multiple battles, in which our model is a first agent model, which may be, for example, an Nth generation model, and the enemy's model may be a second agent model, which may be, for example, an Mth generation model, wherein N and M may be equal, in which case the first agent model and the second agent model are both the latest version of the model, or N may be greater than M, in which case the first agent model is the latest version of the model, and the second agent model is a historical version of the model.

[0189] Also, see Figure 8What can be determined is that the first intelligent agent model and the second intelligent agent model can be used as the enemy and our side to simulate the battle, because the logic of the first intelligent agent model and the second intelligent agent model is the same, thus realizing the self-game of the intelligent agent model.

[0190] Specifically, the first intelligent agent model can act for one round, and the game environment determines the intermediate game environment based on the first action data of the first intelligent agent model. Then it is the turn of the second intelligent agent model, and the second intelligent agent model can determine the second action data of the second intelligent agent model based on the intermediate game environment, thus realizing a turn-based game.

[0191] S305. Determine, through the environment engine, the state data corresponding to the action data output by the second agent model.

[0192] After the second agent model outputs action data, see Figure 7 The environment engine can determine the corresponding state data according to the second action data of the second intelligent agent model. Its implementation method is similar to that of the first intelligent agent model introduced above, and will not be repeated here.

[0193] S306. Determine the state data corresponding to the action data output by the second agent model as the next state data returned by the environment engine.

[0194] After the second agent model's turn ends, it is the first agent model's turn to act. The game state after the second agent model acts is the next game state that needs to be returned to the first agent model.

[0195] If Figure 7 As shown, in this embodiment, the state data corresponding to the second action data output by the second agent model can be determined as the next state data returned by the game environment to the first agent model.

[0196] It can be understood that the process described above is a round process between the first agent model and the second agent model. In order to collect a large amount of data in this embodiment, multiple battles can be simulated by the first agent model and the second agent model, and each battle includes multiple rounds. For example, Figure 8 The first intelligent agent model and the second intelligent agent model shown can simulate 100 battles, after which the battle data generated by the first intelligent agent model can be stored in the experience pool, and the battle data generated by the second intelligent agent model can be stored in the experience pool for the model to learn, thereby obtaining a new generation of intelligent agent models.

[0197] At the same time, it is understandable that if a single-threaded battle simulation is used, the start of the second game often needs to wait until the end of the first game. The large amount of battle waiting time greatly reduces the efficiency of the agent's battle simulation. In a possible implementation method, Fig. 9 As shown, for example, a multi-threading method can be added at the same time to enable multiple agents to play games in parallel, and then the collected data is stored in a memory pool, so as to maximize the use of machine resources to effectively improve the efficiency of collecting combat data, thereby speeding up training.

[0198] S307: Obtain reward data corresponding to the action data.

[0199] At the same time, each action taken by the intelligent agent requires a reward to judge whether the action is a good strategy or a low-level strategy, so that the neural network can learn better and better. Therefore, in this embodiment, reward data corresponding to the action data can also be obtained. For example, the reward data can be determined by the game environment introduced above based on the action data.

[0200] In a possible implementation, in this embodiment, multiple rewards can be designed for turn-based games. Since both parties are playing a self-game, the winning and losing rewards are zero-sum. In addition, there are also kill and damage rewards, etc. Setting multiple rewards can create intelligent agents of various styles.

[0201] In turn-based games, if there are many rounds of battle, the conventional reward setting only focuses on winning or losing the game. The agent will explore the strategy that maximizes the probability of winning or losing. The agent may not be able to explore the optimal strategy and may fall into a local optimal solution.

[0202] If the agent is guided too much, such as setting too many kill and health rewards, the agent may get a lower winning rate due to the misleading rewards set by humans.

[0203] In this embodiment, for example, three types of agents can be provided for turn-based games. The first type only sets rewards for winning or losing, and gives a larger exploration rate and exploration time to allow the agent to fully learn. The second type of agent gives the agent a small amount of rewards for winning or losing and killing the enemy. At the same time, the rewards for winning or losing will change linearly according to the number of rounds of the battle. The shorter the number of rounds, the higher the reward. The optimal strategy is often not inclined to grind blood or heal each other. The third type of agent artificially sets a variety of guidance that conforms to people's cognition. In addition to winning or losing by killing, there are also changes in blood volume, other attribute changes, buff rewards, etc. The setting of multiple rewards creates agents with various turn-based styles.

[0204] The three types of rewards introduced above are three possible implementation methods. In the actual implementation process, the specific setting method of the reward can be selected according to actual needs. It can be understood that if you want the intelligent agent to output a certain action, you can just base it on the reward data with more actions. There are different emphases in the game, so this embodiment does not limit the specific implementation method of the reward function.

[0205] S308: Determine whether the end condition of data collection is met, if so, execute S309, if not, execute S301.

[0206] In this embodiment, after obtaining the next state data, it can be determined first whether the end condition of data collection is met to determine whether the above operation needs to be repeated later. The end condition of data collection can be, for example, the collection of a preset amount of data, or the completion of multiple battles of the model. This embodiment does not limit the end condition of data collection, which can be selected according to actual needs.

[0207] In one possible implementation, if not enough data is collected, the above-mentioned S301 operation is repeated, that is, the next state data is input into the first intelligent agent model, and then the new action data output by the first intelligent agent model is obtained, and the operation is repeated until the end condition of data collection is reached.

[0208] S309: Determine a plurality of experience data.

[0209] In another possible implementation, if the end condition of data collection is met, multiple experience data can be obtained, for example, all collected data are used as experience data, where each experience data includes: previous state data, action data corresponding to the previous state data, reward data corresponding to the action data, and next state data corresponding to the action data.

[0210] For example, you can combine Fig.10 Understanding empirical data, such as Fig.10 As shown, the experience data may include the previous state data, which may include, for example, health points, mana points, number of surviving heroes, character position action information, gain information, and other data representing the current game state.

[0211] The experience data may also include action data corresponding to the previous state data, and the experience data may also include the next game state after taking the action, such as the character being killed, the number of survivors decreasing, the health and technology being good, etc.

[0212] Furthermore, the experience data may also include reward data corresponding to the action data determined by the game environment. For example, the corresponding reward data may be determined based on whether the action data leads to killing an enemy object, the amount of damage caused, whether the game is won, etc.

[0213] In turn-based games, the previous state data in each piece of experience data can uniquely correspond to a game scenario. Similar state game scenarios, taking different actions will lead to different outcomes of the game. Therefore, the reward data is used as an indicator to weigh the impact of actions for model learning.

[0214] S310, obtaining the previous state data, action data, reward data and next state data in the experience data.

[0215] This embodiment trains the model based on the experience data. During the training process, for the currently processed experience data, the previous state data, action data, reward data and next state data in the experience data can be obtained.

[0216] S311. Obtain action data output by the current action network according to the action network of the first agent model and the previous state data.

[0217] It is worth noting that the intelligent agent model in this embodiment can be, for example, a PPO model, wherein the PPO model is an ACTOR-CRITIC (action-value) model, which contains two sub-networks, namely the ACTOR network and the CRITIC network, wherein the ACTOR network is used to select actions, and the CRITIC network is used to predict the reward data under the action.

[0218] Therefore, in a possible implementation, the action data output by the current action network can be obtained according to the action network and the previous state data of the first agent model.

[0219] S312. According to the value network of the first agent model, obtain the predicted reward data corresponding to the action data output by the current action network.

[0220] The intelligent agent model in this embodiment may also include a evaluation network (CRITIC network), wherein the CRITIC network may determine predicted reward data corresponding to the action data output by the current action network.

[0221] S313. Determine the strategy loss function value based on the reward data in the experience data, the predicted reward data, the action data in the experience data, and the action data output by the current action network.

[0222] After that, the loss function value can be determined based on the reward data in the experience data, the predicted reward data, the action data in the experience data, and the action data output by the current action network to adjust the parameters of the model. Among them, the loss function is used to evaluate the degree of difference between the predicted value and the true value of the model, and the reward data can be a parameter in the loss function.

[0223] In one possible implementation, in this embodiment, for example, three loss functions may be provided, namely, a policy loss function, a value loss function, and an entropy loss function. These three loss functions may together constitute the optimization objective of the model. For example, the loss function values ​​corresponding to each of the three loss functions may be determined based on the three loss functions, and then the parameters of the model may be adjusted according to the loss function values.

[0224] First, the policy loss function is introduced. In one possible implementation, the policy loss function value can be determined based on the reward data in the experience data, the predicted reward data, the action data in the experience data, and the action data output by the current action network.

[0225] Among them, the policy loss function value corresponding to the policy loss can satisfy the following formula 1, for example:

[0226]

[0227] Among them, L CLIP (θ) is the value of the strategy loss function, corresponding to policy loss, To obtain the expected operation, r t (θ) = p / p_old, where p is the current strategy probability output by the action network, and p_old is the strategy probability in the empirical data. is the advantage function: specifically, predicted reward data minus actual reward data, clip is the clipping operation, and ∈ is a 0-1 hyperparameter set by the user.

[0228] Among them, the role of the policy loss function in model optimization is to optimize the policy loss part, that is, to increase the probability of actions with high rewards and reduce the probability of actions with low rewards.

[0229] S314. Determine the value of the value loss function based on the reward data and the predicted reward data in the experience data.

[0230] In a possible implementation, the value loss function value corresponding to value loss may satisfy the following formula 2, for example:

[0231] The loss function of value loss can satisfy the following formula 2, for example:

[0232] L value =(rv) Formula 2

[0233] Among them, L value is the value loss function value, corresponding to value loss, r is the reward data in the experience data, and v is the predicted reward data.

[0234] Among them, the role of the value loss function in model optimization is: the value loss part represents the ability of the agent to predict the rewards that will be generated in the game after taking an action.

[0235] S315. Determine the entropy loss function value according to the action data output by the current action network.

[0236] Specifically, with the optimization of policy loss, whenever the agent encounters a similar situation, the agent will always take what it considers to be the optimal action, and this action is often not the optimal action in the early stage of training. Therefore, the entropy loss function in this embodiment introduces the concept of entropy, in which entropy is maximum when probability is uniformly distributed. By introducing the entropy loss function, the agent can expand high-reward actions while maintaining exploration.

[0237] Specifically, entropy is the amount of information contained in information, or it can be understood as the degree of uncertainty. The greater the degree of uncertainty, the greater the entropy.

[0238] For the output of the model, if the entropy of the output result is larger, it means that the output of the model is more average, which means that the current model is not sure which action is the best. When determining the action data based on the entropy, the model will have a certain probability of making random selections, so that the model can explore and avoid premature convergence and falling into local optimal problems.

[0239] The entropy loss function value corresponding to the entropy loss in this embodiment may satisfy the following formula 3, for example:

[0240]

[0241] Among them, H(X) is the entropy loss function value, corresponding to value loss, P(x i ) represents the xth i The probability of an action.

[0242] It is worth noting that the above S313-S315 introduces the implementation process of determining the three-part loss function value. This embodiment does not limit the implementation order of the above three steps, which can be selected according to actual needs.

[0243] S316. Sum the strategy loss function value, the value loss function value, and the entropy loss function value to obtain the target loss function value.

[0244] The three parts of the loss function value in this embodiment will play a certain role in the adjustment of the model parameters. Therefore, it is necessary to adjust the parameters of the model together according to the three parts of the loss function value. In one possible implementation method, the three parts of the loss function value introduced above can be summed to obtain the target loss function value, and then the model parameters are adjusted according to the target loss function value.

[0245] In an optional implementation, the operation on the three parts of the loss function value is not limited to the sum operation introduced above. For example, according to corresponding needs, certain mathematical processing can be performed on the three parts of the loss function value to obtain the target loss function value. This embodiment does not limit this and it can be selected according to actual needs, as long as the target loss function value is determined jointly based on the loss function values ​​of the three parts.

[0246] S317. Perform back propagation according to the target loss function value and adjust the parameters of the first agent model.

[0247] After obtaining the target loss function value, back propagation is performed according to the target loss function value to adjust the parameters of the first intelligent agent model. It can be understood that because the reward data is used when calculating the target loss function value, when back propagating according to the target loss function value to adjust the parameters of the first intelligent agent model, the parameters of the model are actually adjusted according to the guidance of the reward data.

[0248] It is understandable that the goal of adjusting model parameters is to reduce the loss function value, thereby reducing the gap between the true value and the expected value, so as to improve the accuracy and rationality of the model output.

[0249] S318: Determine whether the training end condition of the first agent model is met. If so, execute S319; if not, execute S310.

[0250] After the parameters of the first intelligent agent model are adjusted, it can be determined whether the training of the first intelligent agent model is completed, wherein the training end condition may be, for example, that the number of training rounds reaches a preset number of rounds, the accuracy of the intelligent agent model reaches a preset accuracy, the intelligent agent model converges, etc. The setting of the training end condition can be selected according to actual needs, and this embodiment does not limit this.

[0251] In one possible implementation, if the training has not yet ended, it is necessary to continue training the intelligent agent model. For example, the above steps of adjusting the model parameters can be repeated. Specifically, the remaining empirical data can be processed to obtain the target loss function value, and the above steps of adjusting the model parameters according to the target loss function value can be repeated to continuously adjust the model parameters until the first intelligent agent model reaches the training end condition.

[0252] S319, obtain the target intelligent agent.

[0253] In another possible implementation, if the training end condition of the first agent model is met, it means that the training of the first agent model is completed, and at this time, it can be determined that the target agent is obtained.

[0254] It is worth noting that the agent model in this embodiment is based on the PPO algorithm and is adapted and improved for turn-based games. The agent model includes the neural network design of turn-based games, the design of game actions, and the improvement of the training data set to self-game data. Invalid actions are shielded in the PPO model. The above improvements solve the problems of difficult training convergence and low-level strategies. For the characterized experience data stored in the memory pool, including the turn state, the selected action, and the reward of the corresponding state action, PPO can optimize the network that simulates the action reward and the network that outputs the action probability, so that the network can predict more accurate rewards and expand the probability of high-reward actions.

[0255] In addition, after the self-game training, the two intelligent agents need to have an evaluation scheme to evaluate the level of strategy. Usually, it is difficult to qualitatively and quantitatively evaluate the level of strategy in turn-based games. This embodiment provides several schemes for evaluating strategies:

[0256] For example, we can judge based on the reward settings, log video observation, evaluate the probability distribution of POLICY, the round number curve of the game, and the battle against the standard team. The overall strategy can be observed in the battle round number curve of the model. Since the winning and losing rewards change linearly according to the number of battle rounds, the number of rounds of multi-agent games should usually change from more to less, and finally converge to a fixed number of battle rounds. In addition, the specific strategy of the multi-agent is analyzed in the battle log and judged in the access game. The probability distribution of the network's strategy output can also judge the confidence level of learning.

[0257] S320, controlling the target intelligent agent to automatically make action decisions within the game.

[0258] After obtaining the target intelligent agent, the target intelligent agent can be controlled to automatically make action decisions in the game. The implementation method is similar to that introduced in S203 above and will not be repeated here.

[0259] In summary, the agent processing method provided in this embodiment collects the real self-game battle data of the agent model, and then trains the agent model according to the battle data to effectively ensure the accuracy and rationality of the game operation output by the target agent. In the process of collecting data, the state data is characterized and encoded to ensure that the state data can be input into the model in a way that the neural network can perceive, wherein reasonable characterization coding can ensure that the algorithm strategy of the agent model has better performance. At the same time, in this embodiment, various types of rewards can be set to obtain various types of agents to ensure that the agent model can be trained according to actual needs to enhance the flexibility and applicability of the agent model. In addition, by providing a variety of configuration lineup methods in this embodiment, the problem of cumbersome and time-consuming configuration lineup attributes can be effectively avoided. At the same time, a reasonable lineup can provide better data quality for generalized turn-based agents to improve the effect of model learning.

[0260] Fig.11 This is a schematic diagram of the structure of the intelligent agent processing device provided in the embodiment of the present application. Fig.11 As shown, the device 110 includes: a processing module 1101 and a training module 1102.

[0261] The processing module 1101 is used to perform L environmental interaction processes through the first agent model to obtain L experience data, each of which includes: previous state data, action data corresponding to the previous state data, reward data corresponding to the action data, and next state data corresponding to the action data, where L is an integer greater than or equal to 1;

[0262] A training module 1102 is used to train the first agent model according to the plurality of experience data to obtain a target agent;

[0263] A control module 1103 is used to control the target agent to automatically make action decisions in the game;

[0264] Among them, the environmental interaction processing includes: inputting the previous state data into the first intelligent agent model, the first intelligent agent model determines the action data corresponding to the previous state data, and feeds the action data back to the environmental engine to obtain the next state data corresponding to the action data and the reward data corresponding to the action data.

[0265] In a possible design, the training module 1102 is specifically used for:

[0266] Obtaining previous state data, action data, reward data and next state data in the experience data;

[0267] According to the action network of the first agent model and the previous state data, obtaining action data currently output by the action network;

[0268] According to the value network of the first agent model, obtaining predicted reward data corresponding to the action data currently output by the action network;

[0269] Adjusting parameters of the first agent model according to the reward data in the experience data, the predicted reward data, the action data in the experience data, and the action data currently output by the action network;

[0270] For each of the experience data, the above-mentioned operation of adjusting the parameters of the first agent model is performed respectively until the training end condition of the first agent model is reached, thereby obtaining the target agent.

[0271] In a possible design, the training module 1102 is specifically used for:

[0272] Determine a strategy loss function value according to the reward data in the experience data, the predicted reward data, the action data in the experience data, and the action data currently output by the action network;

[0273] Determining a value loss function value based on the reward data in the experience data and the predicted reward data;

[0274] Determine an entropy loss function value according to the action data output by the current action network;

[0275] Summing the strategy loss function value, the value loss function value, and the entropy loss function value to obtain a target loss function value;

[0276] Back propagation is performed according to the target loss function value to adjust the parameters of the first agent model.

[0277] In a possible design, the processing module 1101 is further used for:

[0278] The previous state data is encoded to obtain encoding information corresponding to the previous state data.

[0279] In one possible design, the previous state data includes at least one of the following: numerical data, position data, and category data;

[0280] The processing module 1101 is specifically used for:

[0281] Normalizing the numerical data to obtain coding information corresponding to the numerical data; and / or,

[0282] Performing one-bit effective encoding on the position data to obtain encoding information corresponding to the position data; and / or,

[0283] The type data is subjected to network initialization encoding, and feature data corresponding to the type data is learned to obtain encoding information corresponding to the type data.

[0284] In a possible design, the processing module 1101 is specifically used for:

[0285] Feeding the action data back to the environment engine, and determining the intermediate state data corresponding to the action data through the environment engine;

[0286] Inputting the intermediate state data into a second agent model to obtain action data output by the second agent model;

[0287] Determining, by the environment engine, state data corresponding to the action data output by the second agent model;

[0288] The state data corresponding to the action data output by the second agent model is determined as the next state data returned by the environment engine.

[0289] In one possible design, the second agent model and the first agent model are the same agent model;

[0290] Alternatively, the first agent model is the latest version of the agent model, and the second agent model is the historical version of the agent model.

[0291] In a possible design, the control module 1103 is specifically used for:

[0292] Inputting the state data to be processed in the game into the target agent, and obtaining the behavior data output by the target agent;

[0293] According to the behavior data output by the target intelligent agent, the target intelligent agent is controlled to automatically make action decisions corresponding to the behavior data in the game.

[0294] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.

[0295] Fig.12 A schematic diagram of the hardware structure of the intelligent agent processing device provided in the embodiment of the present application is shown in FIG. Fig.12 As shown, the intelligent agent processing device 120 of this embodiment includes: a processor 1201 and a memory 1202;

[0296] Memory 1202, used to store computer-executable instructions;

[0297] The processor 1201 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the agent processing method in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0298] Optionally, the memory 1202 may be independent or integrated with the processor 1201 .

[0299] When the memory 1202 is independently provided, the intelligent agent processing device further includes a bus 1203 for connecting the memory 1202 and the processor 1201 .

[0300] An embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the intelligent agent processing method executed by the intelligent agent processing device as described above is implemented.

[0301] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0302] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present application.

[0303] It should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented by a hardware processor or implemented by a combination of hardware and software modules in the processor.

[0304] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0305] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0306] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0307] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0308] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An agent processing method, characterized in that: include: Performing L environmental interaction processes through the first agent model to obtain L experience data, each of which includes: previous state data, action data corresponding to the previous state data, reward data corresponding to the action data, and next state data corresponding to the action data, where L is an integer greater than or equal to 1; Training the first agent model according to a plurality of experience data to obtain a target agent; Controlling the target agent to automatically make action decisions within the game; The environment interaction processing includes: inputting the previous state data into the first agent model, the first agent model determining the action data corresponding to the previous state data, feeding the action data back to the environment engine, obtaining the next state data corresponding to the action data and the reward data corresponding to the action data; Feeding back the action data to the environment engine to obtain next state data corresponding to the action data includes: Feeding the action data back to the environment engine, and determining the intermediate state data corresponding to the action data through the environment engine; Inputting the intermediate state data into a second agent model to obtain action data output by the second agent model; Determining, by the environment engine, state data corresponding to the action data output by the second agent model; Determining the state data corresponding to the action data output by the second agent model as the next state data returned by the environment engine; The second agent model and the first agent model are the same agent model; Alternatively, the first agent model is the latest version of the agent model, and the second agent model is the historical version of the agent model.

2. The method according to claim 1, characterized in that The first agent model is trained according to a plurality of experience data to obtain a target agent, including: Obtaining previous state data, action data, reward data and next state data in the experience data; According to the action network of the first agent model and the previous state data, obtaining action data currently output by the action network; According to the value network of the first agent model, obtaining predicted reward data corresponding to the action data currently output by the action network; Adjusting parameters of the first agent model according to the reward data in the experience data, the predicted reward data, the action data in the experience data, and the action data currently output by the action network; For each of the experience data, the above-mentioned operation of adjusting the parameters of the first agent model is performed respectively until the training end condition of the first agent model is reached, thereby obtaining the target agent.

3. The method according to claim 2, characterized in that The adjusting the parameters of the first agent model according to the reward data in the experience data, the predicted reward data, the action data in the experience data and the action data currently output by the action network comprises: Determine a strategy loss function value according to the reward data in the experience data, the predicted reward data, the action data in the experience data, and the action data currently output by the action network; Determining a value loss function value based on the reward data in the experience data and the predicted reward data; Determine an entropy loss function value according to the action data output by the current action network; Summing the strategy loss function value, the value loss function value, and the entropy loss function value to obtain a target loss function value; Back propagation is performed according to the target loss function value to adjust the parameters of the first agent model.

4. The method according to claim 1, characterized in that The method further comprises: The previous state data is encoded to obtain encoding information corresponding to the previous state data.

5. The method according to claim 4, characterized in that The previous state data includes at least one of the following: numerical data, position data, and category data; The encoding process of the previous state data to obtain encoding information corresponding to the previous state data includes: Normalizing the numerical data to obtain coding information corresponding to the numerical data; and / or, Performing one-bit effective encoding on the position data to obtain encoding information corresponding to the position data; and / or, The type data is subjected to network initialization encoding, and feature data corresponding to the type data is learned to obtain encoding information corresponding to the type data.

6. The method according to any one of claims 1 to 5, characterized in that: The controlling the target agent to automatically make action decisions in the game includes: Inputting the state data to be processed in the game into the target agent, and obtaining the behavior data output by the target agent; According to the behavior data output by the target intelligent agent, the target intelligent agent is controlled to automatically make action decisions corresponding to the behavior data in the game.

7. An intelligent agent processing device, characterized in that: include: A processing module, configured to perform L environmental interaction processes through a first agent model to obtain L experience data, each of which includes: previous state data, action data corresponding to the previous state data, reward data corresponding to the action data, and next state data corresponding to the action data, where L is an integer greater than or equal to 1; A training module, used for training the first agent model according to a plurality of experience data to obtain a target agent; A control module, used to control the target agent to automatically make action decisions in the game; The environment interaction processing includes: inputting the previous state data into the first agent model, the first agent model determining the action data corresponding to the previous state data, feeding the action data back to the environment engine, obtaining the next state data corresponding to the action data and the reward data corresponding to the action data; The processing module is specifically used to feed back the action data to the environment engine, and determine the intermediate state data corresponding to the action data through the environment engine; Inputting the intermediate state data into a second agent model to obtain action data output by the second agent model; Determining, by the environment engine, state data corresponding to the action data output by the second agent model; Determining the state data corresponding to the action data output by the second agent model as the next state data returned by the environment engine; The second agent model and the first agent model are the same agent model; Alternatively, the first agent model is the latest version of the agent model, and the second agent model is the historical version of the agent model.

8. An intelligent agent processing device, characterized in that: include: Memory, used to store programs; A processor, configured to execute the program stored in the memory; when the program is executed, the processor is configured to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The method comprises instructions which, when executed on a computer, cause the computer to perform the method as claimed in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images