An agent training method and device, an electronic device, and a storage medium

Abstract

The application provides an agent training method and device, electronic equipment and a storage medium. Different training rewards of different styles are set to train different agents. The trained agents can perform operations and corresponding behaviors of different styles, so that the model after overall training can generate agents of different styles, and the strategy strength of the agents is ensured on the basis of multiple styles.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more specifically, to a method, apparatus, electronic device, and storage medium for training intelligent agents. Background Technology

[0002] With the development of network technology, computer games and other computer-computer interaction applications can provide users with virtual scenes where they can manipulate virtual objects to perform operations for entertainment purposes. In scenarios such as game guidance, game testing, character hosting, or non-player character (NPC) control, the computer also needs to determine the operation to be performed on a virtual object and then control it.

[0003] Currently, some games often use artificial intelligence (AI) trained using self-play schemes to determine the actions that virtual objects might perform in the aforementioned scenarios. However, the cooperative strategies of current game AI trained using self-play schemes are often relatively simple and prone to overfitting, resulting in poor cooperation between the AI ​​and human players (or other game AIs).

[0004] Therefore, how to further improve current artificial intelligence (especially game AI) to satisfy the requirements of strategy strength while possessing strategy diversity is an urgent problem to be solved. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides an intelligent agent training method, apparatus, electronic device, and storage medium.

[0006] A first aspect of this invention provides an agent training method, the method comprising:

[0007] Extract environmental feature information for each agent in the simulated environment, the environmental feature information including image-like features and vector features;

[0008] The environmental characteristics of each agent are input into the decision model corresponding to the agent's behavioral style;

[0009] The decision model outputs the current decision action of each agent based on the input content.

[0010] Control each agent to execute corresponding decision-making actions and interact with the simulation environment;

[0011] The reward for each agent in the current simulation environment based on the decision action, as well as the new environmental feature information after executing the decision action, are obtained.

[0012] The environmental feature information, decision-making actions, and obtained rewards are spliced ​​together and saved as sample data;

[0013] Based on reinforcement learning algorithms, style decision models for different behaviors are trained using the saved sample data.

[0014] Optionally, the step of extracting environmental feature information for each agent in the simulated environment specifically includes:

[0015] Based on the position-related state of the agent in the simulated environment, image features are extracted.

[0016] Based on the state-related attributes of the agent in the simulated environment, vector features are extracted, and the vector features include information on the behavioral style of the corresponding agent.

[0017] Optionally, the decision model is a convolutional neural network model, which outputs the steps of each agent's current decision action based on the input content, specifically including:

[0018] After the vector features are processed through the first fully connected layer, the key factors in the vector features are enhanced by the self-attention module.

[0019] The image features are processed through five convolutional layers;

[0020] The processed vector features and image-like features are merged and then used to model the time series using an LSTM module;

[0021] The information processed by the LSTM module is then processed through the second fully connected layer and used as the output of the decision model.

[0022] The output of the decision model includes three branches: the main card decision action, the card-carrying decision action based on the main card decision action, and the evaluation value that evaluates the main card decision action and the card-carrying decision action.

[0023] Optionally, the step of training decision models for different behavioral styles using the saved sample data based on reinforcement learning algorithms specifically includes:

[0024] The decision model is updated and iterated using a proximal policy optimization algorithm until the preset model training termination condition is met.

[0025] The reward function shaping algorithm was used to train decision models with different behavioral styles.

[0026] The hyperparameters of the decision model are optimized using an automatic hyperparameter optimization algorithm.

[0027] Optionally, the step of training multiple decision models with different behavioral styles using the reward function shaping algorithm specifically includes:

[0028] Establish a basic reward model that is independent of behavioral style;

[0029] Based on the basic reward model, dense rewards corresponding to each behavioral style are determined according to the simulation environment for different behavioral styles.

[0030] The decision model is trained based on the aforementioned basic reward model and dense reward model.

[0031] Optionally, the step of optimizing the hyperparameters of the decision model using an automatic hyperparameter optimization algorithm specifically includes:

[0032] During training, multiple decision models are learned simultaneously, and the hyperparameters of the decision models are randomly adjusted during training.

[0033] In the same simulated environment, adversarial battles are conducted between agents of decision-making models trained with different hyperparameters.

[0034] Based on certain selection rules, hyperparameters with better performance are selected and updated onto the decision model that is being trained.

[0035] Optionally, the method includes:

[0036] The trained decision-making model is saved as a historical model and stored in the historical model library according to its corresponding behavioral style.

[0037] In the simulation environment, multiple agents with the same or different behavioral styles are randomly generated, and historical models with the same behavioral style are selected for each agent from the historical model library.

[0038] A second aspect of the present invention provides an agent training device, the device comprising:

[0039] A feature extraction unit is used to extract environmental feature information for each agent in the simulated environment, the environmental feature information including image-like features and vector features;

[0040] An action decision unit is used to input the environmental feature information of each agent into a decision model corresponding to the behavioral style of that agent. The decision model outputs the current decision action of each agent based on the input content.

[0041] The action interaction unit is used to control each agent to execute corresponding decision-making actions and interact with the simulation environment;

[0042] An environmental feedback unit is used to obtain the reward of each agent in the current simulation environment based on the decision action, as well as the new environmental feature information after the decision action is executed;

[0043] The sample storage unit is used to save the environmental feature information, decision-making actions, and obtained rewards as sample data by splicing them together.

[0044] The model training unit is used to train style decision models for different behaviors based on reinforcement learning algorithms and the saved sample data.

[0045] Optionally, the feature extraction unit is specifically used for:

[0046] Based on the position-related state of the agent in the simulated environment, image features are extracted.

[0047] Based on the state-related attributes of the agent in the simulated environment, vector features are extracted, and the vector features include information on the behavioral style of the corresponding agent.

[0048] Optionally, the decision model is a convolutional neural network model, and the action decision unit is specifically used for:

[0049] After the vector features are processed through the first fully connected layer, the key factors in the vector features are enhanced by the self-attention module.

[0050] The image features are processed through five convolutional layers;

[0051] The processed vector features and image-like features are merged and then used to model the time series using an LSTM module;

[0052] The information processed by the LSTM module is then processed through the second fully connected layer and used as the output of the decision model.

[0053] The output of the decision model includes three branches: the main card decision action, the card-carrying decision action based on the main card decision action, and the evaluation value that evaluates the main card decision action and the card-carrying decision action.

[0054] Optionally, the model training unit is specifically used for:

[0055] The decision model is updated and iterated using a proximal policy optimization algorithm until the preset model training termination condition is met.

[0056] The reward function shaping algorithm was used to train decision models with different behavioral styles.

[0057] The hyperparameters of the decision model are optimized using an automatic hyperparameter optimization algorithm.

[0058] Optionally, the method for training multiple decision models with different behavioral styles using the reward function shaping algorithm in the model training unit specifically includes:

[0059] Establish a basic reward model that is independent of behavioral style;

[0060] Based on the basic reward model, dense rewards corresponding to each behavioral style are determined according to the simulation environment for different behavioral styles.

[0061] The decision model is trained based on the aforementioned basic reward model and dense reward model.

[0062] Optionally, the method for training the decision model based on the basic reward model and the dense reward model by the model training unit specifically includes:

[0063] During training, multiple decision models are learned simultaneously, and the hyperparameters of the decision models are randomly adjusted during training.

[0064] In the same simulated environment, adversarial battles are conducted between agents of decision-making models trained with different hyperparameters.

[0065] Based on certain selection rules, hyperparameters with better performance are selected and updated onto the decision model that is being trained.

[0066] Optionally, the data storage unit is further configured to save the trained decision model as a historical model in the historical model library according to its corresponding behavioral style.

[0067] The device also includes a model deployment unit, which is used to randomly generate multiple agents with the same or different behavioral styles in the simulation environment, and select historical models with the same behavioral style for each agent from the historical model library.

[0068] A third aspect of the present invention provides an electronic device, characterized in that it comprises:

[0069] One or more processors; memory; one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to perform the method as described in the first aspect.

[0070] A fourth aspect of the present invention provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores program code that can be invoked by a processor to execute the method described in the first aspect.

[0071] In summary, this invention provides an agent training method, apparatus, electronic device, and storage medium. By setting different styles of training rewards, different agents can be trained. After training, the agents can perform different styles of operations and corresponding behaviors, so that the final trained model can generate agents with different styles, ensuring the policy strength of the agents on the basis of multiple styles. Attached Figure Description

[0072] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0073] Figure 1 This is a schematic diagram illustrating an application scenario of the intelligent agent training method and apparatus according to an embodiment of the present invention;

[0074] Figure 2 This is a flowchart of the intelligent agent training method according to an embodiment of the present invention;

[0075] Figure 3 This is a flowchart of an intelligent agent training method according to another embodiment of the present invention;

[0076] Figure 4 This is a functional block diagram of the intelligent agent training device according to an embodiment of the present invention;

[0077] Figure 5 This is a functional block diagram of an intelligent agent training device according to another embodiment of the present invention;

[0078] Figure 6 This is a structural block diagram of an electronic device for executing an agent training method according to an embodiment of the present invention.

[0079] Figure 7 This is a structural block diagram of a computer-readable storage medium for storing or carrying program code implementing the intelligent agent training method according to an embodiment of the present application.

[0080] icon:

[0081] AI server 100; simulation environment server 200; feature extraction unit 110; action decision unit 120; action interaction unit 130; environment feedback unit 140; data storage unit 150; model training unit 160; model deployment unit 170; electronic device 300; processor 310; memory 320; computer-readable storage medium 400; program code 410. Detailed Implementation

[0082] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make the invention more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of the invention. However, those skilled in the art will recognize that the technical solutions of the invention may be practiced with one or more of these specific details omitted, or other methods, components, apparatus, steps, etc., may be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of the invention.

[0083] Furthermore, the accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0084] Before providing a further detailed description of the embodiments of the present invention, the nouns and terms involved in the embodiments of the present invention will be explained, and the nouns and terms involved in the embodiments of the present invention shall be interpreted as follows.

[0085] (1) In response to, used to indicate the conditions or states on which the operation is performed depends. When the conditions or states on which the operation is performed are met, the one or more operations performed may be performed in real time or may have a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0086] (2) Based on, used to indicate the conditions or states on which the operation is performed depends. When the conditions or states on which it depends are met, one or more operations can be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order of execution of the multiple operations.

[0087] (3) Model training: Multi-class classification learning on the image dataset. This model can be constructed using deep learning frameworks such as TensorFlow and Torch, employing multiple layers of neural networks such as CNN to form a multi-class classification model. The model input is a three-channel or original-channel matrix formed by reading images using tools such as OpenCV. The model output is the multi-class probability, ultimately outputting the webpage category through algorithms such as softmax. During training, the model approximates the correct trend using objective functions such as cross-entropy.

[0088] (4) Neural Network (NN): Artificial Neural Network (ANN), also known as neural network or neural network-like network, is a mathematical or computational model in the fields of machine learning and cognitive science that imitates the structure and function of biological neural networks (the central nervous system of animals, especially the brain) and is used to estimate or approximate functions.

[0089] (5) Game Environment: This is the game environment displayed (or provided) by the application when it runs on the terminal. The game environment can be a simulation of the real world, a semi-simulated / semi-fictional three-dimensional environment, or a purely fictional three-dimensional environment. The game environment can be any of a two-dimensional game environment, a 2.5-dimensional game environment, or a three-dimensional game environment. The following embodiments use a three-dimensional game environment as an example, but this is not a limitation. Optionally, the game environment is also used for game environment battles between at least two virtual objects. Optionally, the game environment is also used for battles between at least two virtual objects through the rolling of a virtual sphere. Optionally, the game environment can also be, but is not limited to, shooting games, parkour games, racing games, multiplayer online battle arena (MOBA), racing games (RCG), and sports games (SPG). The trained game model provided in this application can be deployed in game servers corresponding to the aforementioned game scenarios to generate real-time game strategies, execute corresponding action information, simulate the operation of virtual users, and jointly complete different types of games in the game environment with actual users participating in the game.

[0090] (6) Action information: Taking the game user's first-person or third-person participation in speed competition, including racing, flying and other racing games as an example, action information refers to the operation instructions such as the directional keys that control the moving object as actions. For role-playing games, action information refers to virtual weapons that attack by firing bullets in the game environment, or virtual bows and arrows that fire arrows. Virtual objects in the game environment can pick up virtual guns and attack with the virtual guns they have picked up.

[0091] (7) Artificial Intelligence Agent (AI Agent): In cooperative games / cooperative events, a decision-maker controlled by computer artificial intelligence. In the following text, the abbreviation "intelligent agent" is sometimes used to refer to the AI ​​agent. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. For example, the AI ​​agent of this disclosure can cooperate or compete with real players in a manner similar to how real players control virtual objects. Based on the design principles and implementation methods of various intelligent machines, the AI ​​agent of this disclosure can understand different game strategies and cooperate and compete with real players using different strategies.

[0092] Action: Each agent performs an action. In the various embodiments of this disclosure below, the random variable of the action of the i-th agent is denoted as and the actual observed value of the action is denoted as ai. Without the denoting i, it signifies the concatenation of actions of all agents: A = [A1, A2, ..., An], a = [a1, a2, ..., an]. The action space of the i-th agent is denoted as Ai, which contains all possible actions of that agent. The action space of the entire population is A = A1 × A2 ... × An. The action spaces Ai and Aj of two agents may be the same or different.

[0093] Reward: The reward is a numerical value fed back to the agent by the environment. Let Ri be the random variable representing the reward of agent i, and ri be the actual observed value of the reward. Under the cooperative setting in the embodiments of this disclosure, R1 = R2 = ... = Rm. The reward at time t is jointly determined by the state St and the actions A = [A1, A2, ..., An] of all agents.

[0094] Population: A population is a typical multi-agent system. A population comprises multiple artificial intelligence agents. These agents share an environment and influence each other. For example, in some situations, an agent's action may change the environmental state, thus affecting the remaining agents. In the following text, the term "population" can refer to a collection of artificial intelligence agents. A population of artificial intelligence agents can be used to train another artificial intelligence agent to join the population, or any existing agent in the population, enabling that agent to cooperate with other agents in the population employing different strategies. In the embodiments of this disclosure, it is assumed that the population comprises n agents, with superscripts representing the agent's index (e.g., if the superscript is i, then i ranges from 1 to n), and the subscript t representing time.

[0095] An episode is the period from when an agent begins executing a task to a certain termination stage, such as when the task is completed (successfully or unsuccessfully), or when a certain amount of time has elapsed. For example, an episode can be when a robotic arm moves from its starting position to its final target position (a successful completion of the task). Another example is when an agent in a game is killed by an enemy, ending the task (a failed task).

[0096] The intelligent agent can be a user-controlled virtual object controlled through client operations, an artificial intelligence (AI) trained and set up for gameplay in the game environment, or a non-user virtual object (NPC / Non-Player Character) set up for interaction in the game environment. Optionally, the virtual object can be a virtual character competing in the game environment. Optionally, the number of virtual objects participating in the interaction in the game environment can be preset or dynamically determined based on the number of clients joining the interaction.

[0097] Example

[0098] Please see Figure 1 This embodiment provides a schematic diagram of an application scenario for an intelligent agent training method and device.

[0099] like Figure 1As shown, this invention provides an intelligent agent training method and apparatus, applied to an AI server 100 and a simulation environment server 200. The AI ​​server 100 is used to train the algorithm model, and the simulation environment server 200 is used to implement the simulation environment. In actual operation, the AI ​​server 100 connects to the simulation environment server 200. The simulation environment server 200 sends the current state of the simulation environment to the AI ​​server 100. The AI ​​server 100 extracts useful information from the simulation environment and transforms it into features that the model can recognize. These features are then passed to the decision model for prediction. Finally, the prediction result is sent to the simulation environment server 200 for specific execution.

[0100] It should be noted that the AI ​​server 100 and the simulation environment server 200 can be either cloud servers located in the cloud or local servers located on a local machine. The AI ​​server 100 and the simulation environment server 200 can be servers that are functionally differentiated but based on the same physical device, or servers that are functionally differentiated but based on different physical devices.

[0101] Based on the above, such as Figure 2 As shown, an agent training method is provided according to an embodiment of the present invention. The method includes:

[0102] Step S101: Extract environmental feature information for each agent in the simulated environment, including image-like features and vector features.

[0103] Among them, image-like features and vector features correspond to information of different dimensions in the simulated environment. Image-like features are mainly based on the image content corresponding to the location of the agent in the environment, while vector features are mainly based on attribute-related content in the environment.

[0104] Step S102: Input the environmental feature information of each agent into the decision model corresponding to the behavioral style of that agent.

[0105] In a simulation environment, multiple agents are usually randomly generated for simulation. Each agent has a pre-set behavioral style to generate corresponding sample data, which supports the subsequent training of decision models for different behavioral styles.

[0106] Step S103: The decision model outputs the current decision action of each agent based on the input content.

[0107] After inputting environmental feature information into the decision model, decision models with different behavioral styles will output decision actions of different styles.

[0108] Step S104: Control each agent to execute the corresponding decision-making action and interact with the simulation environment.

[0109] Each agent is controlled to execute decision actions that conform to its own behavioral style based on the output of the decision model, thereby enabling interaction with the simulated environment and other agents.

[0110] Step S105: Obtain the reward for each agent in the current simulation environment based on the decision action, as well as the new environmental feature information after executing the decision action.

[0111] After the interaction, the simulated environment changes accordingly, generating new environmental characteristics. Based on these changes, the reward for each agent after executing a decision-making action in the current simulated environment can be obtained.

[0112] Step S106: The environmental feature information, decision-making actions, and obtained rewards are spliced ​​together and saved as sample data.

[0113] Based on steps S101-S105 described above, the agent performs a decision-making action. This process generates a series of information, and the information that can be used for subsequent decision model training is saved as sample data. Repeating this process generates one sample data point per agent each time.

[0114] As a preferred embodiment of this example, the generated sample data can be saved to a Redis server. When the model needs to be trained later, the sample data can be retrieved from Redis for training.

[0115] Step S107: Based on the reinforcement learning algorithm, the style decision models for different behaviors are trained using the saved sample data.

[0116] Once a sufficient amount of sample data has been accumulated, the decision-making model can be trained and optimized using this data. Since the sample data is generated based on agents with different behavioral styles, different training rewards can be set for different behavioral styles, allowing for simultaneous training. By preserving the behavioral styles and employing reinforcement learning algorithms, the trained decision-making model can maintain sufficient policy strength.

[0117] The agent training method provided in this embodiment achieves training for different agents by setting different styles of training rewards. After training, the agents can perform different styles of operations and corresponding behaviors, so that the final trained model can produce agents with different styles, ensuring the policy strength of the agents on the basis of multiple styles.

[0118] like Figure 3 As shown, another embodiment of the intelligent agent training method of the present invention includes:

[0119] Step S201: Randomly generate multiple agents with the same or different behavioral styles in the simulation environment, and select a historical model with the same behavioral style for each agent from the historical model library.

[0120] The historical model library contains historical models of decision-making models with different behavioral styles. These historical models were learned and saved through previous training.

[0121] In this embodiment, during the training and optimization of the decision-making model, at the beginning of each round of optimization, multiple agents with the same or different behavioral styles are randomly generated in the simulation environment. Simultaneously, for different behavioral styles, a historical model corresponding to that style is selected from the historical model library and deployed onto the agent. For agents with the same behavioral style, different historical models of the same style may also be selected for deployment. This ensures that the style of the agents in each round has a certain degree of randomness, thereby generating sample data from more scenarios and providing support for subsequent training and optimization.

[0122] As a preferred embodiment of this example, in order to improve the efficiency of sample data production, multiple simulation environments can be deployed simultaneously. In different simulation environments, AI agents of different styles and the number of different styles are randomly shuffled, and self-play is run to produce sample data.

[0123] Step S202: Extract image features based on the position-related state of the agent in the simulated environment.

[0124] Step S203: Extract vector features based on the state related to the attributes of the agent in the simulation environment. The vector features include information on the behavioral style of the corresponding agent.

[0125] Taking a specific game scenario as an example, the image-like features have 5 channels. These include: the player's own position, the positions of all enemies, the positions of all skill-based entities on the field, the current safe zone and poison zone ranges, and the next round's safe zone and poison zone ranges. The vector features have a total of 525 dimensions, containing the detailed attributes of the current agent, as well as the basic attributes, skill attributes, and environmental attributes of each other agent. Additionally, the vector features incorporate model control elements reflecting the current agent's behavioral style.

[0126] Step S204: After processing the vector features through the first fully connected layer, the key factors in the vector features are enhanced by the self-attention module.

[0127] Step S205: The image-like features are processed through five convolutional layers.

[0128] Step S206: The processed vector features and image-like features are merged and then used to model the time series using the LSTM module.

[0129] Step S207: The information processed by the LSTM module is processed through the second fully connected layer and used as the output of the decision model.

[0130] The output of the decision model includes three branches: the main card decision action, the card-carrying decision action based on the main card decision action, and the evaluation value that evaluates the main card decision action and the card-carrying decision action.

[0131] Steps S204-S207 represent the process by which the decision model predicts behavior based on the input feature information. For vector features, the vectors described above are first processed into a high-dimensional vector, and then the self-attention module identifies which part of the information (or which agent) needs attention. For image-like features, they are first processed through a five-layer convolutional layer. The processed features are then used to model the temporal sequence using an LSTM, remembering what happened in chronological order. The output is then passed through a second fully connected layer. The output uses a cascaded approach: first predicting the main card's action, and then predicting the actions of the accompanying cards based on the main card's action.

[0132] Taking a specific game scenario as an example, the decision action of playing the main card corresponds to the game character releasing a skill, while the decision action of playing the secondary card corresponds to the direction of the skill release. When the decision model outputs, it uses a cascaded approach to first predict the specific skill release, and then predict the direction of the skill release.

[0133] Step S208: Control each agent to execute the corresponding decision-making action and interact with the simulation environment.

[0134] Step S209: Obtain the reward for each agent in the current simulation environment based on the decision action, as well as the new environmental feature information after executing the decision action.

[0135] In this embodiment, since it is necessary to train agents with different behavioral styles, different reward rules need to be set. The impact of different reward rules causes the decision model to converge according to different behavioral styles during training. The specific reward rule settings will be explained in detail in subsequent steps.

[0136] Step S210: The environmental feature information, decision-making actions, and obtained rewards are spliced ​​together and saved as sample data.

[0137] In this embodiment, the training method used is the reinforcement learning method PPO, and the corresponding sample data includes image feature information, vector feature information, generalized advantage estimation (GAE), reward information, and other information required for reinforcement learning training. When the training method is adjusted, the information contained in the sample data also needs to be adjusted accordingly based on the requirements of the training method being used.

[0138] On the other hand, to improve the efficiency of sample data generation, some unnecessary sample data can be deleted by setting some filtering rules. Taking a game scenario as an example, sample data generated by agents that have died in the current environment can be deleted.

[0139] Step S211: Use the proximal policy optimization algorithm to update and iterate the decision model until the preset model training termination condition is met.

[0140] As a preferred approach, the PPO algorithm was used during training. Surrogate loss and entropy loss were used to optimize the decision strategy of the decision model, and L2 loss was used as the value loss function to optimize the value, i.e., the evaluation value, thereby generating a more accurate decision strategy.

[0141] Step S212: Train multiple decision models with different behavioral styles using the reward function shaping algorithm.

[0142] The specific method of using the reward function shaping algorithm is as follows:

[0143] Establish a basic reward model that is independent of behavioral style;

[0144] Based on the basic reward model, dense rewards corresponding to each behavioral style are determined according to the simulation environment for different behavioral styles.

[0145] The decision model is trained based on the aforementioned basic reward model and dense reward model.

[0146] Taking a game scenario as an example, we first design a style-independent reward schema. After each round, we rank the AI ​​agents according to their death order, assigning each agent a different final reward. Then, we shape the reward function to design dense rewards of different styles; here, we take two styles as examples: conservative and aggressive. Let the ranking reward be R. r The reward for causing any damage is R. d The reward for being damaged by any skill is R. o :

[0147] For a conservative style, the reward function R(s) = R r -∑R o ×2+∑R d That is, the punishment for being harmed is twice the reward for causing the harm;

[0148] For the radical style, the reward function R(s) = R r -∑R o +∑R d ×2 means that the reward for causing damage is twice the penalty for being damaged.

[0149] Based on the above process, both types of agents with different styles will aim to extend their survival time. At the same time, the conservative-style agents will tend to reduce the damage they receive, while the aggressive-style agents will tend to actively cause damage.

[0150] Step S213: Optimize the hyperparameters of the decision model using an automatic hyperparameter optimization algorithm.

[0151] The specific method for using the hyperparameter automatic optimization algorithm is as follows:

[0152] During training, multiple decision models are learned simultaneously, and the hyperparameters of the decision models are randomly adjusted during training.

[0153] In the same simulated environment, adversarial battles are conducted between agents of decision-making models trained with different hyperparameters.

[0154] Based on certain selection rules, hyperparameters with better performance are selected and updated onto the decision model that is being trained.

[0155] In the above process, the training models with different styles of historical models are simulated to compete against each other, and the win rate and key indicators (such as survival time and damage caused) are statistically analyzed. The win rate is used as a metric for PBT to determine which historical models have better hyperparameters, and these are then updated to the model currently being trained.

[0156] In a preferred embodiment of the present invention, the above steps S211-S213 can be executed sequentially or synchronously, thereby improving the training efficiency of the decision model.

[0157] As a preferred embodiment of this example, when performing training optimization in steps S211-S213, multiple workers can perform the training task to accelerate the training speed and convergence speed.

[0158] Step S214: Save the trained decision model as a historical model in the historical model library according to its corresponding behavioral style.

[0159] The historical models saved in the historical model library can be deployed and used when step S201 is executed.

[0160] As a preferred embodiment of the present invention, a client with image display function can be connected to the simulation environment server 200 to display the action execution status and results of each agent in the simulation environment.

[0161] In summary, the agent training method provided in this embodiment achieves training for different agents by setting different styles of training rewards. The trained agents can then perform operations and corresponding behaviors in different styles, enabling the final trained model to generate agents with diverse styles, thus ensuring the policy strength of the agents across multiple styles. A multi-model parallel training approach is employed, allowing agents of various styles to generate training samples in a single environment. Reward shaping is used to design rewards specifically for each style of AI agent, and PBT training is applied to update the hyperparameters during multi-model training, ensuring the competitive strength of the agents across multiple styles.

[0162] like Figure 4 As shown, the intelligent agent training device provided by this invention includes:

[0163] Feature extraction unit 110 is used to extract environmental feature information of each agent in the simulated environment, the environmental feature information including image-like features and vector features;

[0164] Action decision unit 120 is used to input the environmental feature information of each agent into the decision model corresponding to the behavior style of the agent, and the decision model outputs the current decision action of each agent according to the input content;

[0165] The action interaction unit 130 is used to control each agent to execute corresponding decision actions and interact with the simulation environment;

[0166] The environmental feedback unit 140 is used to obtain the reward of each agent in the current simulation environment based on the decision action and the new environmental feature information after the decision action is executed;

[0167] The data storage unit 150 is used to save the environmental feature information, decision-making actions and obtained rewards as sample data after splicing them together.

[0168] The model training unit 160 is used to train style decision models for different behaviors based on reinforcement learning algorithms and the saved sample data.

[0169] like Figure 5 As shown, another embodiment of the present invention provides an intelligent agent training device, which, based on the previous embodiment, serves as a preferred embodiment of this invention:

[0170] The feature extraction unit 110 is specifically used for:

[0171] Based on the position-related state of the agent in the simulated environment, image features are extracted.

[0172] Based on the state-related attributes of the agent in the simulated environment, vector features are extracted, and the vector features include information on the behavioral style of the corresponding agent.

[0173] The decision model is a convolutional neural network model, and the action decision unit 120 is specifically used for:

[0174] After the vector features are processed through the first fully connected layer, the key factors in the vector features are enhanced by the self-attention module.

[0175] The image features are processed through five convolutional layers;

[0176] The processed vector features and image-like features are merged and then used to model the time series using an LSTM module;

[0177] The information processed by the LSTM module is then processed through the second fully connected layer and used as the output of the decision model.

[0178] The output of the decision model includes three branches: the main card decision action, the card-carrying decision action based on the main card decision action, and the evaluation value that evaluates the main card decision action and the card-carrying decision action.

[0179] The model training unit 160 is specifically used for:

[0180] The decision model is updated and iterated using a proximal policy optimization algorithm until the preset model training termination condition is met.

[0181] The reward function shaping algorithm was used to train decision models with different behavioral styles.

[0182] The hyperparameters of the decision model are optimized using an automatic hyperparameter optimization algorithm.

[0183] The method for training multiple decision models with different behavioral styles using the reward function shaping algorithm in the model training unit specifically includes:

[0184] Establish a basic reward model that is independent of behavioral style;

[0185] Based on the basic reward model, dense rewards corresponding to each behavioral style are determined according to the simulation environment for different behavioral styles.

[0186] The decision model is trained based on the aforementioned basic reward model and dense reward model.

[0187] The method for training a decision model based on the basic reward model and dense reward by the model training unit 160 specifically includes:

[0188] During training, multiple decision models are learned simultaneously, and the hyperparameters of the decision models are randomly adjusted during training.

[0189] In the same simulated environment, adversarial battles are conducted between agents of decision-making models trained with different hyperparameters.

[0190] Based on certain selection rules, hyperparameters with better performance are selected and updated onto the decision model that is being trained.

[0191] The data storage unit 150 is also used to save the trained decision model as a historical model and save it to the historical model library according to its corresponding behavioral style.

[0192] The device also includes a model deployment unit 170, which is used to randomly generate multiple agents with the same or different behavioral styles in the simulation environment, and select historical models with the same behavioral style for each agent from the historical model library.

[0193] The intelligent agent training device provided in this embodiment of the invention is used to implement the above-described intelligent agent training method. Therefore, the specific implementation method is the same as the above method and will not be described again here.

[0194] like Figure 6 The diagram shows a structural block diagram of an electronic device 300 provided in an embodiment of the present invention. This electronic device 300 can be a smartphone, tablet computer, e-reader, or other electronic device capable of running applications. The electronic device 300 in this application may include one or more of the following components: a processor 310, a memory 320, and one or more applications, wherein the one or more applications can be stored in the memory 320 and configured to be executed by one or more processors 310, and the one or more applications are configured to perform the methods described in the foregoing method embodiments.

[0195] Processor 310 may include one or more processing cores. Processor 310 connects to various parts within the electronic device 300 using various interfaces and lines, and performs various functions and processes data of the electronic device 300 by running or executing instructions, programs, code sets, or instruction sets stored in memory 320, and by calling data stored in memory 320. Optionally, processor 310 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 310 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 310 and may be implemented separately using a communication chip.

[0196] The memory 320 may include random access memory (RAM) or read-only memory (ROM). The memory 320 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 320 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below. The data storage area may also store data created by the terminal during use (such as phonebook data, audio and video data, chat log data, etc.).

[0197] like Figure 7 The diagram shows a structural block diagram of a computer-readable storage medium 400 provided in an embodiment of the present invention. The computer-readable medium stores program code 410, which can be called by a processor to execute the methods described in the above method embodiments.

[0198] The computer-readable storage medium 400 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 400 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 400 has storage space for program code 410 that performs any of the method steps described above. This program code 410 can be read from or written to one or more computer program products. The program code 410 may, for example, be compressed in a suitable form.

[0199] In summary, this invention provides an agent training method, apparatus, electronic device, and storage medium. By setting different styles of training rewards, different agents can be trained. After training, the agents can perform different styles of operations and corresponding behaviors, so that the final trained model can generate agents with different styles, ensuring the policy strength of the agents on the basis of multiple styles.

[0200] In the embodiments disclosed in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0201] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0202] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for training an intelligent agent, characterized in that, The method includes: Environmental feature information of each agent in the simulated environment is extracted. The environmental feature information includes image-like features and vector features. The image-like features are based on the image content corresponding to the location of the agent in the environment, and the vector features are based on the attribute-related content in the environment. The environmental feature information of each agent is input into the decision model corresponding to the agent's behavior style. Each agent has a pre-set behavior style. For different behavior styles, the historical model corresponding to the style is selected from the historical model library and deployed to the agent. At the same time, multiple simulation environments are deployed. Agents with different styles and the number of different styles are randomly shuffled in different simulation environments, and self-play is run to generate sample data. The decision model outputs the current decision action of each agent based on the input content. Control each agent to execute corresponding decision-making actions and interact with the simulation environment; The reward for each agent in the current simulation environment based on the decision action, as well as the new environmental feature information after executing the decision action, are obtained. The environmental feature information, decision-making actions, and obtained rewards are spliced ​​together and saved as sample data; Based on reinforcement learning algorithms, style decision models for different behaviors are trained using the saved sample data. The step of extracting environmental feature information of each agent in the simulated environment specifically includes: Based on the position-related state of the agent in the simulated environment, image features are extracted. Based on the state-related attributes of the agent in the simulated environment, vector features are extracted, and the vector features include information on the behavioral style of the corresponding agent. The decision model is a convolutional neural network model, which outputs the steps of each agent's current decision action based on the input content, specifically including: After the vector features are processed through the first fully connected layer, the key factors in the vector features are enhanced by the self-attention module. The image features are processed through five convolutional layers; The processed vector features and image-like features are merged and then used to model the time series using an LSTM module; The information processed by the LSTM module is then processed through the second fully connected layer and used as the output of the decision model. The output of the decision model includes three branches: the main card decision action, the card-carrying decision action based on the main card decision action, and the evaluation value for evaluating the main card decision action and the card-carrying decision action. The main card decision action corresponds to the release of skills by game characters in the game scene, and the card-carrying decision action corresponds to the direction of skill release in the game scene. The step of training decision models for different behavioral styles using the saved sample data based on the reinforcement learning algorithm specifically includes: The decision model is updated and iterated using a proximal policy optimization algorithm until the preset model training termination condition is met. The reward function shaping algorithm was used to train decision models with different behavioral styles. The hyperparameters of the decision model are optimized using an automatic hyperparameter optimization algorithm. The steps for training multiple decision models with different behavioral styles using the reward function shaping algorithm specifically include: Establish a basic reward model that is independent of behavioral style; Based on the basic reward model, dense rewards corresponding to each behavioral style are determined according to the simulation environment for different behavioral styles; the basic reward model is the reward strategy after the interaction with the simulation environment is completed, and the dense reward is the reward strategy for rewarding during the interaction with the simulation environment; The decision model is trained based on the aforementioned basic reward model and dense reward model.

2. The agent training method according to claim 1, characterized in that, The step of optimizing the hyperparameters of the decision model using an automatic hyperparameter optimization algorithm specifically includes: During training, multiple decision models are learned simultaneously, and the hyperparameters of the decision models are randomly adjusted during training. In the same simulated environment, adversarial battles are conducted between agents of decision-making models trained with different hyperparameters. Based on certain selection rules, hyperparameters with better performance are selected and updated onto the decision model that is being trained.

3. The agent training method according to claim 1 or 2, characterized in that, The method includes: The trained decision-making model is saved as a historical model and stored in the historical model library according to its corresponding behavioral style. In the simulation environment, multiple agents with the same or different behavioral styles are randomly generated, and historical models with the same behavioral style are selected for each agent from the historical model library.

4. An intelligent agent training device, characterized in that, The device includes: The feature extraction unit is used to extract environmental feature information for each agent in the simulated environment. The environmental feature information includes image-like features and vector features. The image-like features are based on the image content corresponding to the location of the agent in the environment, and the vector features are based on attribute-related content in the environment. An action decision unit is used to input the environmental feature information of each agent into a decision model corresponding to the agent's behavioral style. The decision model outputs the current decision action of each agent based on the input. Each agent has a pre-set behavioral style. For different behavioral styles, a historical model corresponding to the style is selected from the historical model library and deployed to the agent. At the same time, multiple simulation environments are deployed. Agents with different styles and the number of different styles are randomly shuffled in different simulation environments, and self-play is run to generate sample data. The action interaction unit is used to control each agent to execute corresponding decision-making actions and interact with the simulation environment; An environmental feedback unit is used to obtain the reward of each agent in the current simulation environment based on the decision action, as well as the new environmental feature information after the decision action is executed; The data storage unit is used to save the environmental feature information, decision-making actions, and obtained rewards as sample data by splicing them together. The model training unit is used to train style decision models for different behaviors based on reinforcement learning algorithms and the saved sample data. The feature extraction unit is specifically used for: Based on the position-related state of the agent in the simulated environment, image features are extracted. Based on the state-related attributes of the agent in the simulated environment, vector features are extracted, and the vector features include information on the behavioral style of the corresponding agent. The decision model is a convolutional neural network model, and the action decision unit is specifically used for: After the vector features are processed through the first fully connected layer, the key factors in the vector features are enhanced by the self-attention module. The image features are processed through five convolutional layers; The processed vector features and image-like features are merged and then used to model the time series using an LSTM module; The information processed by the LSTM module is then processed through the second fully connected layer and used as the output of the decision model. The output of the decision model includes three branches: the main card decision action, the card-carrying decision action based on the main card decision action, and the evaluation value for evaluating the main card decision action and the card-carrying decision action. The main card decision action corresponds to the release of skills by game characters in the game scene, and the card-carrying decision action corresponds to the direction of skill release in the game scene. The model training unit is specifically used for: The decision model is updated and iterated using a proximal policy optimization algorithm until the preset model training termination condition is met. The reward function shaping algorithm was used to train decision models with different behavioral styles. The hyperparameters of the decision model are optimized using an automatic hyperparameter optimization algorithm. The method for training decision models with multiple different behavioral styles using the reward function shaping algorithm specifically includes: Establish a basic reward model that is independent of behavioral style; Based on the basic reward model, dense rewards corresponding to each behavioral style are determined according to the simulation environment for different behavioral styles; the basic reward model is the reward strategy after the interaction with the simulation environment is completed, and the dense reward is the reward strategy for rewarding during the interaction with the simulation environment; The decision model is trained based on the aforementioned basic reward model and dense reward model.

5. An electronic device, characterized in that, include: One or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to perform the method as described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that can be invoked by a processor to execute the method as described in any one of claims 1-3.

Images