Game agent training method and apparatus, and electronic device and readable storage medium

Abstract

A game agent training method, comprising: acquiring game state data of a game match at the current moment; inputting the game state data into a first policy network and a value function network, so as to obtain an action probability distribution that is output by the first policy network and an evaluation value that is output by the value function network, wherein the action probability distribution is used for guiding a game agent to select an action having the maximum probability as a target action; after the game agent has executed the target action, acquiring the game state data of the game match at the current moment, and on the basis of the game state data of the game match at the current moment, determining a reward value of the game agent; on the basis of the reward value and the evaluation value, determining a first loss value, and on the basis of the first loss value, training the value function network; and on the basis of the action probability distribution, the reward value and the evaluation value, determining a second loss value, and on the basis of the second loss value, training the first policy network. The present disclosure can improve the team collaboration capability and behavior strategy capability of a game agent.

Description

Game agent training method, device, electronic device and readable storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application number 202410158100.0, filed on February 4, 2024, entitled “Training method, device, electronic device and readable storage medium for game intelligence agents”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of computer technology, and in particular to a method, device, electronic device, and computer-readable storage medium for training a game agent. Background Art

[0004] With the rapid development of machine learning, game agents (i.e., game robots) are widely used in competitive games. High-level and anthropomorphic game agents can significantly improve the playability and entertainment of games.

[0005] In the process of training game agents using reinforcement learning technology, the relevant technology controls the game agents through the behavior strategy network to perform corresponding actions to interact with the game environment, and calculates the reward value of the game agents in each state based on the preset reward function, thereby optimizing the behavior decision network according to the reward value.

[0006] However, the strategic style of gaming agents trained using these techniques is relatively simple. For games that require strong strategic coordination and teamwork, these agents often suffer from poor teamwork and behavioral strategy. For example, in a game with offensive and defensive roles, all agents trained using these techniques may only attack but not defend, or vice versa. Summary of the Invention

[0007] According to one aspect of the present disclosure, a method for training a game agent is provided, the method comprising: obtaining game state data at the current moment of a game match, the game state data including state data of game characters, the game characters being all virtual characters of the two opposing camps in the game match, the game characters including at least one game agent, and the camps of the game match including multiple character positions; inputting the game state data as input data into a first policy network and a value function network to be trained, obtaining an action probability distribution output by the first policy network, and an evaluation value output by the value function network; wherein the action probability distribution is used to guide the game agent to select the action with the highest probability as the target action, and the evaluation value is used to evaluate the action. The value is used to evaluate the value of the game status data; after the game agent performs the target action, the reward value of the game agent is obtained and determined based on the preset reward rules according to the game status data at the current moment of the game. The preset reward rules include reward rules corresponding to each role positioning. The reward rules corresponding to the role positioning are used to indicate the reward value that the game agent with the role positioning can obtain for the interactive events that occur during the game; according to the reward value and the evaluation value, a first loss value is determined, and the value function network is trained based on the first loss value; and according to the action probability distribution, the reward value and the evaluation value, a second loss value is determined, and the first strategy network is trained based on the second loss value.

[0008] According to one aspect of the present disclosure, a training device for a game agent is provided, the device comprising: an acquisition module for acquiring game status data at the current moment of a game match, the game status data comprising status data of game characters, the game characters being all virtual characters of the two opposing camps in the game match, the game characters comprising at least one game agent, and the camps of the game match comprising a plurality of character positions; a processing module for inputting the game status data as input data into a first policy network and a value function network to be trained, obtaining an action probability distribution output by the first policy network, and an evaluation value output by the value function network; wherein the action probability distribution is used to guide the game agent to select the action with the highest probability as the target action, and the evaluation value is used to evaluate the action. The value is used to evaluate the value of the game status data; the determination module is used to obtain and determine the reward value of the game agent based on the preset reward rules after the game agent performs the target action and the game status data at the current moment of the game. The preset reward rules include reward rules corresponding to each role positioning, and the reward rules corresponding to the role positioning are used to indicate the reward value that the game agent with the role positioning can obtain for the interactive events that occur during the game; the training module is used to determine the first loss value based on the reward value and the evaluation value, and train the value function network based on the first loss value; and, determine the second loss value based on the action probability distribution, the reward value and the evaluation value, and train the first strategy network based on the second loss value.

[0009] According to one aspect of the present disclosure, an electronic device is provided, which includes: a memory and a processor, and the memory and the processor are coupled; the memory is used to store one or more computer instructions; and the processor is used to execute one or more computer instructions to implement the training method of the game agent of any one of the above-mentioned first aspects.

[0010] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which one or more computer instructions are stored, characterized in that the instructions are executed by a processor to implement any of the above-mentioned methods for training a game agent.

[0011] According to one aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any of the above-mentioned methods for training a game agent when the computer program is executed by a processor.

[0012] Compared with the related art, the present disclosure has the following advantages:

[0013] The present disclosure provides a method for training a game agent. The method obtains game state data at the current moment of a game match. The game state data includes state data of game characters. Game characters are all virtual characters from the opposing camps in the game match. The game characters include at least one game agent, and each camp in the game match includes multiple character positions. The game state data is input into a first policy network and a value function network to be trained. The method obtains an action probability distribution output by the first policy network and an evaluation value output by the value function network. The action probability distribution guides the game agent to select the action with the highest probability as the target action, and the evaluation value is used to assess the value of the game state data. After the game agent executes the target action, a reward value for the game agent is obtained and determined based on the game state data at the current moment of the game match based on preset reward rules. The preset reward rules include reward rules corresponding to each character position. The reward rules corresponding to each character position indicate the reward value that the game agent with that character position can receive for interactive events that occur during the game match. A first loss value is determined based on the reward value and the evaluation value, and the value function network is trained based on the first loss value. And, determining a second loss value according to the action probability distribution, the reward value and the evaluation value, and training the first policy network based on the second loss value.

[0014] Compared with related technologies, the present invention sets different reward rules for game agents with different role positioning, which can enable game agents with different role positioning to cooperate in combat better; and by training game agents with different role positioning to adopt different behavioral strategies, the diversity of the role behaviors of game agents in the game is increased, thereby improving the team collaboration ability and behavioral strategy ability of the game agents. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] FIG1 is a flow chart of a method for training a gaming agent according to one embodiment of the present disclosure;

[0016] FIG2 is a schematic diagram of a global location distribution map and a local location distribution map provided by one embodiment of the present disclosure;

[0017] FIG3 is a schematic diagram of a first policy network and a value function network provided by one embodiment of the present disclosure;

[0018] FIG4 is a schematic diagram of a first policy network and a value function network provided by another embodiment of the present disclosure;

[0019] FIG5 is a schematic diagram of a process for generating a behavior strategy for a game agent according to one embodiment of the present disclosure;

[0020] FIG6 is a flow chart of a preset target selection rule provided by one embodiment of the present disclosure;

[0021] FIG7 is a schematic diagram of the structure of a training device for a gaming agent according to one embodiment of the present disclosure;

[0022] FIG8 is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present disclosure.

[0023] The above drawings illustrate specific embodiments of the present disclosure, which will be described in more detail below. These drawings and textual descriptions are not intended to limit the scope of the present disclosure in any way, but rather to illustrate the concepts of the present disclosure to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0024] To make the purposes, advantages, and features of the present disclosure more clear, the present disclosure is described clearly and completely below in conjunction with the accompanying drawings and specific embodiments. In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure. However, the described embodiments are only some of the embodiments of the present disclosure, not all of them. All other embodiments obtained by persons of ordinary skill in the art without inventive effort are within the scope of protection of the present disclosure.

[0025] It should be noted that, in the description of the present disclosure, the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance, or a specific order or precedence. For those skilled in the art, the specific meanings of the above terms in the present disclosure can be understood in specific circumstances. In addition, in the description of the present disclosure, unless otherwise specified, the term "plurality" refers to two or more. The term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. The terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] In order to facilitate understanding of the technical solutions of the present disclosure, relevant concepts that may be involved in the embodiments of the present disclosure are first introduced.

[0027] 1. AI

[0028] AI (Artificial Intelligence) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. It is a branch of computer science that involves making computers intelligent, that is, able to understand and respond to human instructions, and learn and adapt without explicit programming.

[0029] 2. Policy Network

[0030] A policy network is a neural network model that selects the optimal action based on the current state of the environment. In the field of reinforcement learning, policy networks are widely used to solve various decision-making problems. A policy network takes the current state of the environment as input and outputs a probability distribution representing the probability of selecting each possible action. Through training, the policy network learns strategies for selecting the optimal action under different environmental conditions. During training, the policy network interacts with the environment and updates its weights based on reward signals to optimize its ability to select the optimal action.

[0031] 3. Intelligent Agent

[0032] An agent is a system capable of perception, decision-making, and action, interacting and learning within its environment to achieve specific goals. An agent can be a physical entity, such as a robot, or a virtual entity, such as a computer program or a virtual character. An agent uses sensors to perceive environmental input, collects information from the environment, and uses internal decisions and rules to take corresponding actions, influencing the environment through the execution of these actions. The functions of an agent primarily include perception, decision-making, and action. Perception refers to the agent receiving and understanding information from the external environment, such as images, sounds, and text, through sensors. Decision-making refers to the process by which an agent selects the optimal action based on perceived information and internal rules and strategies. Action refers to the agent interacting with the environment by controlling its movements through actuators based on the results of its decisions.

[0033] 4. Game Agents

[0034] A game agent is an intelligent entity capable of making autonomous decisions and taking actions within a game environment. By perceiving the state and information of the game environment, such as maps, enemy locations, and items, the agent makes decisions based on set goals, rules, and strategies, selecting the optimal actions to achieve victory or complete specific tasks. Neural network-based game agents possess enhanced adaptability, intelligence, and challenge, enhancing the game's entertainment and interactivity. Within the game, agents can assume various roles, such as those of the opposing team, teammates, and non-player characters. The agent's behavior and degree of anthropomorphism within the game scene directly impact the game's difficulty, realism, and enjoyment.

[0035] 5. Reinforcement Learning

[0036] Reinforcement learning (RL) is a self-supervised learning method for solving sequential decision-making problems. It enables intelligent agents to learn autonomously without the need for manual collection of training data. In RL, an agent observes the current state of its environment, selects an action to influence the environment based on that state, and receives a reward signal to evaluate the effectiveness of that action. The agent's goal is to gradually learn an optimal policy through interaction with the environment, by trying different actions, and based on reward signals—that is, to be able to choose the best action given the environmental state. The core idea of ​​RL is based on the optimization of a value function or a policy function. The value function represents the expected reward from choosing an action in a given environmental state, while the policy function represents the probability of the agent choosing an action in a given environmental state. By continuously updating and optimizing these functions, the agent can gradually improve the quality of its decisions.

[0037] With the advancement of computer technology and the diversification of terminal functions, the demand for online gaming on smart devices such as mobile phones and computers is growing, and the variety and scale of online games are also increasing. Online games often have a certain number of players. For example, an 8v8 competitive game requires 16 players to start a game. This can lead to long wait times for players, which can cause them to lose interest in the game. To address this, game agents have emerged. Game agents are virtual entities that operate within the game scene and have the ability to make autonomous decisions and take actions. Therefore, when players are waiting for too long, game agents can be deployed into the game to start the game on behalf of the missing players.

[0038] In recent years, with the gradual maturity of artificial intelligence (AI) technology, gaming agents have been widely used in competitive games. Highly skilled and anthropomorphic gaming agents can significantly enhance the playability and entertainment value of games. With the rapid development of online games, an increasing number of game modes require players to possess high-level strategic thinking, teamwork, and real-time decision-making abilities. For example, capture the flag is a classic multiplayer game mode in which players are divided into two teams. The goal is to capture the opposing team's flag and bring it back to their territory, while also keeping their own flag from being captured. Therefore, gaming agents deployed in competitive games also need to possess strategic thinking, teamwork, and real-time decision-making abilities. However, gaming agents trained using traditional training methods often perform poorly in competitive games that require high strategic thinking, teamwork, and real-time decision-making abilities.

[0039] Below, the related technologies involved in this disclosure and the problems existing in the related technologies are explained:

[0040] Currently, traditional training methods for game agents mainly include rule design and reinforcement learning.

[0041] In terms of rule design, the following rules need to be defined: game rules, action rules, and strategy rules. Game rules define the basic rules of competitive games, including game objectives, victory conditions, and game time. Action rules describe the actions and movement patterns that game agents can take. Strategy rules include the agent's decision-making rules and target selection. For example, in a capture-the-flag game, the agent might prioritize attacking the enemy's virtual character or choose the shortest path to capture the enemy's flag. However, this rule design approach has several drawbacks: During implementation, it requires pre-setting a large number of behavior nodes and conditions for the agent, which can lead to a lack of diversity and flexibility in the agent's strategy. Furthermore, the rule design is prone to coupling, increasing the cost of later debugging. Furthermore, the behavior of the agent under this rule design approach is fixed and monotonous, making the agent less personified and easily identified by players, negatively impacting the player's gaming experience. Furthermore, since the game rules are designed by game planners themselves, it consumes significant manpower and makes updating the agent cumbersome.

[0042] As for reinforcement learning, it is a method based on learning through the interaction between game agents and the environment, and optimizing decision-making through reward signals. For example, in capture-the-flag gameplay, reinforcement learning can be used to train game agents to learn optimal strategies and tactics to maximize scores or protect the flag. However, current reinforcement learning methods have the following drawbacks: the game agents trained by them have poor team collaboration, understanding of team members' intentions, and ability to balance competition in competitive games. For example, in capture-the-flag games, game agents must collaborate to protect their own team's flag while also fighting against enemy characters and capturing the opposing team's flag. This involves balancing intra-team collaboration and inter-team competition, creating a complex game process. Reinforcement learning algorithms may face difficulties in managing this balance, resulting in poor performance in both collaborative and competitive situations. Furthermore, game agents trained by existing reinforcement learning schemes lack the ability to understand teammates' intentions. If this fails to grasp teammates' intentions in a timely manner, team members can easily learn strategies that are less collaborative, such as all trying to capture the flag or all trying to defend it.

[0043] In summary, for competitive games, the game agents obtained based on the above traditional training methods have technical problems such as poor team collaboration ability, poor behavioral strategy, weak degree of anthropomorphism, and easy recognition by players.

[0044] To address at least some of the aforementioned issues, the present disclosure provides a method for training a gaming agent, a corresponding device for training a gaming agent, an electronic device capable of implementing the method, and a computer-readable storage medium. The following examples provide detailed explanations of the method, device, electronic device, and computer-readable storage medium.

[0045] In order to make the purpose and technical solution of the present disclosure clearer and more intuitive, the method provided by the embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure. It is understood that the following embodiments can exist separately, and the following embodiments and features in the embodiments can be combined with each other when there is no conflict between the embodiments provided in the present disclosure. For the same or similar content, it will not be repeated in different embodiments. In addition, the step sequence in the following method embodiments is only an example and not a strict limitation. In some cases, the steps shown or described can be performed in a different order.

[0046] The present disclosure provides a method, device, electronic device and computer-readable storage medium for training a game agent. Specifically, the method for training a game agent of one embodiment of the present disclosure can be executed by a computer device, wherein the computer device can be a terminal or a server. The terminal can be a terminal device such as a smart phone, a tablet computer, a laptop computer, a touch screen, a game console, etc. The terminal can also include a client, which can be a game application client, a browser client carrying a game program, or an instant messaging client, etc. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and basic cloud computing services such as big data and artificial intelligence platforms.

[0047] Below, in conjunction with Figure 1, the training method of the game agent provided in one embodiment of the present disclosure is described. Figure 1 is a flow chart of the training method of the game agent provided in one embodiment of the present disclosure.

[0048] As shown in FIG1 , the method for training the game agent includes steps S10-S40:

[0049] S10. Obtain the game status data of the current moment of the game match. The game status data includes status data of game characters. The game characters are all virtual characters of the two opposing camps in the game match. The game characters include at least one game agent. Each camp in the game match includes multiple character positions.

[0050] The game state data at the current moment of a game match includes game character state data. The game character state data includes information reflecting the positional distribution of all game characters in the game match, basic game character state information, character positioning information, and radar charts. These are examples only and are not limited in the presently disclosed embodiments. The radar chart represents the game character's ability to perceive surrounding obstacles.

[0051] For example, let's take the capture the flag game as an example. Capture the flag is a classic multiplayer game mode in which characters are divided into two opposing camps. The goal is to capture the opposing camp's flag and bring it back to their base, while also protecting their own camp's flag from being captured. In capture the flag game, the positional distribution of all characters can be depicted in the form of global and local positional distribution maps, depicting the distribution of teammate and enemy characters in various areas of the game scene. Basic character status information includes information such as the position, health, speed, distance to each flag, and skills of each character in the opposing camp. A character's role positioning refers to the current role each character plays in the game. Specifically, role positioning indicates the responsibilities or division of labor of a character within their respective camp during the game. For example, in capture the flag game, role positioning includes flag-capturing, flag-guarding, and flag-defending roles. Therefore, the role positioning information of each character indicates which characters are designated as flag-capturing, flag-guarding, and flag-defending at the current moment in the game.

[0052] Below, in conjunction with Figure 2, taking an 8V8 (i.e., each of the two opposing camps has 8 game characters) capture the flag game as an example, the global position distribution map and the local position distribution map are exemplarily explained. Figure 2 is a schematic diagram of the global position distribution map and the local position distribution map provided by one embodiment of the present disclosure. The global position distribution map of the game characters in the game match, as shown in the left figure in Figure 2, the entire circular area in the global position distribution map represents the entire game scene, and the circular area on the left is divided into 8 equally divided areas. The global position distribution map illustrates the distribution of all game characters in the opposing camps in each area. The local position distribution map of the game characters in the game match, as shown in the right figure in Figure 2, the entire circular area of ​​the local position distribution map represents part of the game scene, and the circular area on the right is also divided into 8 equally divided areas. The local position distribution map illustrates the distribution of game characters in the opposing camps in each area of ​​the local game scene. As shown in the left image of Figure 2, in an 8v8 capture the flag game, the eight enemy characters are distributed in areas 1-8 in numbers 2, 1, 1, 1, 1, 1, 0, and 1, respectively. Furthermore, the eight friendly characters are distributed in areas 1-8 in numbers 1, 2, 0, 1, 2, 1, 1, and 0, respectively. As shown in the right image of Figure 2, the enemy characters are distributed in areas 1-8 in numbers 2, 1, 0, 0, 0, 0, 0, and 0, respectively. Furthermore, the friendly characters are distributed in areas 1-8 in numbers 0, 0, 0, 1, 0, 1, 0, and 0, respectively.

[0053] In the above description, the game characters are all virtual characters in the two opposing camps in the game, and the game characters include at least one game agent. For example, all or some of the game characters in each of the two opposing camps may be game agents, and the remaining game characters may be virtual characters controlled by real players. Alternatively, some or all of the game characters in either camp may be game agents, and the remaining game characters may be virtual characters controlled by real players. This is merely an example, and the present disclosure does not impose any limitation on this.

[0054] As mentioned above, each camp of the game includes multiple role positionings, and the role positioning is used to indicate the responsibilities or division of labor of the game characters in their respective camps during the game.

[0055] For example, in the aforementioned capture-the-flag game, there are three roles: flag-capturing, flag-guarding, and flag-defending. The flag-capturing role's role in the game is to choose a suitable path to capture the flag from the opposing camp's base and return it to their own camp's base. The flag-defending role's role in the game is to protect the flag captured by the flag-capturing role. The flag-defending role's role in the game is to protect their own camp's flag from being captured.

[0056] S20. The game state data is input as input data into the first policy network and the value function network to be trained, and the action probability distribution output by the first policy network and the evaluation value output by the value function network are obtained; wherein the action probability distribution is used to guide the game agent to select the action with the highest probability as the target action, and the evaluation value is used to evaluate the value of the game state data.

[0057] In the disclosed embodiments, the first policy network is the core component of training a gaming agent. After the gaming agent interacts with the game environment, it collects game state data from the current moment of the game match as input data, which is fed into the first policy network. The first policy network then outputs an action probability distribution. This action probability distribution guides the gaming agent in selecting the action with the highest probability as the target action. The gaming agent then executes this target action and interacts further with the game environment.

[0058] In the disclosed embodiments, the action probability distribution output by the first policy network refers to the probability distribution of each possible action selected by the game agent, given the game state data. This probability distribution can be represented as a discrete distribution or a continuous distribution, depending on the nature of the action space in the game environment. This is for illustrative purposes only and is not a limitation of the disclosed embodiments.

[0059] For example, in a discrete action space, actions are a finite number of discrete options. The policy network outputs a vector whose length is equal to the number of action types. Each element in the vector represents the probability of selecting the corresponding action. These probability values ​​are usually processed by a softmax function to ensure that these action probabilities are non-negative and sum to 1. For example, in a game, there are three actions to choose from: move left, move right, and stand still. The action probability distribution output by the first policy network may be [0.2, 0.6, 0.2], which means that at the current state of the game, the probability of choosing to move left is 0.2, the probability of choosing to move right is 0.6, and the probability of choosing to stand still is 0.2. Typically, the game agent chooses the action with the highest probability. In this example, the probability of the action "move right" is 0.6, which means that the game agent will choose "move right" as the target action.

[0060] The first policy network to be trained can be an initial, untrained first policy network, or a trained first policy network that has undergone multiple training cycles but has not yet achieved the training target. Once the first policy network reaches the preset training target after repeated iterative training, the training of the first policy network is considered complete and can be used in actual game play to control the game agent.

[0061] As mentioned above, the value function network is used to evaluate the game state data at the current moment of the game match and measure the value of the game state data.

[0062] In the embodiment of the present disclosure, the structure of the first policy network includes an input layer, a hidden layer, and an output layer. Among them, the input layer is used to receive input data (i.e., receive game state data), the hidden layer is used to process data and extract features, and the output layer is used to output policy recommendations (i.e., output action probability distribution). The structure of the value function network also includes an input layer, a hidden layer, and an output layer. Among them, the input layer is used to receive input data (i.e., receive game state data), the hidden layer is used to process data and extract features, and the output layer is used to output an evaluation value (i.e., output an evaluation value of the game state data at the current moment of the game match).

[0063] 3 , a schematic diagram of a first policy network and a value function network provided in an embodiment of the present disclosure is exemplarily described. FIG3 is a schematic diagram of a first policy network and a value function network provided in one embodiment of the present disclosure.

[0064] As shown in Figure 3, first policy network 301 is provided in an embodiment of the present disclosure. Its input data is game state data, and its output data is the action probability distribution. Value function network 302 is provided in an embodiment of the present disclosure. Its input data is game state data, and its output data is the evaluation value of the input data.

[0065] Furthermore, below, taking the capture the flag game as an example and combining FIG4 , the schematic diagram of the first strategy network and the value function network provided by the embodiment of the present disclosure is exemplarily described. FIG4 is a schematic diagram of the first strategy network and the value function network provided by another embodiment of the present disclosure.

[0066] As shown in Figure 4 , the network consists of a first policy network 401 and a value function network 402. The hidden layer of the first policy network 401 is composed of a cascade of multiple fully connected layers and a multi-head attention layer, while the hidden layer of the value function network 402 is composed of a cascade of multiple fully connected layers and a multi-head attention layer. The input data of the first policy network 401 and the value function network 402 is the game state data in the capture-the-flag game. This state data includes the game character's flag holding state, the game character's position state data, the game character's basic state information, the radar chart state, the game character's position, and the game character's movement speed. As shown in Figure 4 , each different type of state data in the game state data is input into a different network structure. For example, the flag state in the capture-the-flag game is input into one fully connected layer, the game character's position distribution state is input into another fully connected layer, and the game character's basic state information is input into the multi-head attention layer. The input of other state data is shown in Figure 4 and will not be repeated here. As shown in Figure 4 , the output data of the first policy network 401 is the action probability distribution, and the output data of the value function network 402 is the evaluation value of the input game state data.

[0067] As described above, the first policy network 401 and the value function network 402 shown in FIG4 are merely exemplary illustrations of the network structures of the first policy network and the value function network. The embodiments of the present disclosure do not impose any restrictions on the network structures of the first policy network and the value function network.

[0068] S30. After the game agent completes the target action, the game state data at the current moment of the game is obtained and the reward value of the game agent is determined based on the preset reward rules. The preset reward rules include reward rules corresponding to each role positioning. The reward rules corresponding to the role positioning are used to indicate the reward value that the game agent with the role positioning can obtain for the interactive events that occur during the game.

[0069] In the disclosed embodiment, based on the above steps, after the first policy network outputs the action probability distribution, the game agent selects the action with the highest probability as the target action. After the game agent executes the target action, the game state data for the current moment of the game match is obtained. Based on the game state data and preset reward rules, the reward value of the game agent is determined. The preset reward rules include reward rules corresponding to each character position. The reward rules corresponding to the character position are used to indicate the reward value that the game agent with that character position can receive for interactive events that occur during the game match.

[0070] In the disclosed embodiment, by setting different reward rules for game agents with different role positioning, the game agents can be made more focused on performing the tasks that their respective role positioning should undertake during the learning process. At the same time, with clear role positioning, the game agents can divide the work and cooperate with each other to improve the combat efficiency of the entire camp, avoiding the defects of using a single reward rule to cause the strategy style of the trained game agents to be relatively single and the teamwork to be low (for example, in a game with offensive and defensive role positioning, all the trained game agents can only attack but not defend, or only defend but not attack). For example, an offensive role may be motivated by the reward rule to cause damage more effectively, while a healing role will be motivated by the reward rule to maintain the health of teammates, so that the survivability and output capabilities of the entire camp can be improved.

[0071] Furthermore, by training game agents with different role roles according to their respective reward rules, agents with different role roles can better collaborate. For example, an agent with an attacking role role is incentivized to attack enemies, while a healing agent is incentivized to restore teammates' health. Together, these two agents can achieve better game results. Setting different reward rules can train agents with different role roles to adopt different behavioral strategies, increasing the diversity of the agents' role behaviors in the game, thereby improving their teamwork and behavioral strategy capabilities.

[0072] Below, taking the capture the flag game as an example, different reward rules for game agents with different role positioning are set as an example.

[0073] For example, in a capture-the-flag game, there are three roles: flag-capturing, flag-guarding, and flag-guarding. The flag-capturing role's primary task is to choose a suitable path to capture the enemy camp's flag and return it to the friendly camp's base. Therefore, compared to the other two roles, the flag-capturing role is specifically assigned flag-capturing and navigation rewards. Whether the game agent receives the navigation reward depends on the difference between the distance between the game agent's previous position and the target location and the distance between its current position and the target location. If the difference is greater than 0, it indicates that the game agent is approaching the target location and receives the navigation reward. Conversely, if the difference is less than 0, it indicates that the game agent is moving away from the target location and does not receive the navigation reward. It should be noted that for a game agent in the flag-capturing role, if the game agent has already captured the enemy camp's flag, the game agent's target location is the friendly camp's base. If the game agent has not yet captured the enemy camp's flag, the game agent's target location is the location of the enemy camp's flag.

[0074] Among them, the main task of the flag-guarding role is to protect the flag captured by the flag-grabbing role from being lost. Therefore, compared with the other two role positionings, a flag-holding reward is specially set up for the flag-guarding role, which means that the longer the flag-grabbing character holds the flag, the greater the reward for the flag-guarding role.

[0075] Among them, the main task of the flag-guarding character is to protect the flag of his own camp from being snatched. Therefore, if the flag is lost, the flag-guarding character will be given a certain penalty (i.e., negative points), and the longer the enemy camp game character holds the flag of his own camp, the greater the penalty; if the flag-guarding character can successfully take back his own flag from the enemy camp flag-grabbing character and return it to his own game character, the game intelligent body positioned as the flag-guarding character will be given a corresponding reward.

[0076] Below, using the game Capture the Flag as an example, Table 1 and a specific example illustrate setting different reward rules for game agents with different role positioning. For the example of Capture the Flag, the reward rules for the flag capturer, flag protector, and flag defender in the game can be found in Table 1.

[0077] Table 1

[0078] As shown in Table 1, corresponding reward values ​​are set for different event categories (such as game results, flag interaction events, and combat events) for each character. As shown in Table 1, game results include wins and losses; flag interaction events include capturing, navigating, holding, dropping, successfully capturing, and returning; and combat events include kills, deaths, and assists. Considering that the responsibilities of characters in different roles in the capture-the-flag game are primarily reflected in flag interaction events, the following details the reward value settings for different characters in the capture-the-flag game. For the capture-the-flag character, flag-capturing and navigation rewards are specifically set to encourage the game agent positioned as the capture character to capture more flags and approach the target location. The navigation reward for the capture character in Table 1 is 0.01*distance. This distance refers to the difference between the distance between the previous position of the capture character and the target location and the distance between the current position and the target location, when the capture character is approaching the target location. For the flag-guarding character, a flag-holding bonus (0.005 * the duration the character holds the flag) is set. This bonus indicates that the longer the flag-capturing character holds the flag, the greater the bonus to the flag-guarding character. For the flag-guarding character, a flag-returning bonus and flag-holding penalty (-0.005 * the duration the enemy character holds the flag) are set. This means that during training, the game agent positioned as the flag-guarding character is encouraged to perform more flag-returning actions and reduce the duration the enemy character holds the flag.

[0079] By setting differentiated reward rules based on different role positions, as shown in Table 1, the game agents trained in different role positions in the capture-the-flag game are more focused on executing the tasks assigned to each role position, thereby improving the teamwork and behavioral strategy capabilities of the game agents within the faction in the capture-the-flag game. Table 1 is merely an example of setting different reward rules for different role positions and is not limited in any way by the present disclosure.

[0080] S40. Determine a first loss value based on the reward value and the evaluation value, and train the value function network based on the first loss value; and determine a second loss value based on the action probability distribution, the reward value and the evaluation value, and train the first policy network based on the second loss value.

[0081] In an embodiment of the present disclosure, a first loss value is determined based on a preset loss function according to the reward value of the game agent and the evaluation value output by the value function network. The preset loss function can be used to calculate the difference between the reward value of the game agent and the evaluation value output by the value function network. The difference between the reward value of the game agent corresponding to each moment in the game and the evaluation value output by the value function network is used as the first loss value, and the value function network is trained based on the first loss value, that is, the parameters of the value function network are adjusted. By gradually reducing the first loss value and adjusting the parameters in the value function network during the iterative training process, a trained value function network can be obtained.

[0082] In the embodiment of the present disclosure, by comparing the action probability distribution most recently output by the first policy network (i.e., the new policy) and the action probability distribution last output (i.e., the old policy), as well as the reward value and the evaluation value output by the value function network, the loss value of the target action performed by the game agent, i.e., the second loss value, is calculated. Taking the Proximal Policy Optimization (PPO) algorithm as an example, the loss function used is a Surrogate Objective (an alternative objective, an objective function) with a clip, that is, an alternative objective function controlled by a truncation mechanism is used to limit the amplitude of each update and improve the stability of the algorithm. The second loss function can be referenced as follows: L_clip(θ)=E[min(r(θ)A,clip(r(θ),1-ε,1+ε)A)]. Among them, the value of L_clip(θ) is the second loss value, r(θ) is the ratio of the new strategy to the old strategy, θ represents the parameters of the first strategy network, A is the advantage function, A is used to calculate the difference between the action value function and the state value function, the action value function is used to evaluate the value of performing a specific action in a given state, which is the expected value of the long-term accumulated reward obtained by performing a specific action in a given state, the state value function is the output of the value function network, and ε is a hyperparameter used to control the truncation range of the r(θ) ratio. Subsequently, the first strategy network is trained according to the calculated second loss value to adjust the parameters of the first strategy network. The goal of training the first strategy network is to minimize the loss function L_clip(θ). In the PPO algorithm, the gradient descent method is usually used to update the parameters θ of the model. This is only an example, and the embodiments of the present disclosure do not impose any restrictions on this.

[0083] The disclosed embodiments provide a method for training a gaming agent. The method obtains game state data at the current moment of a game match. The game state data includes state data of game characters, which are all virtual characters from opposing factions in the game match. The game characters include at least one gaming agent, and each faction in the game match includes multiple character positions. The game state data is input into a first policy network and a value function network to be trained. The method obtains an action probability distribution output by the first policy network and an evaluation value output by the value function network. The action probability distribution guides the gaming agent to select the action with the highest probability as the target action, and the evaluation value is used to assess the value of the game state data. After the gaming agent executes the target action, a reward value for the gaming agent is obtained and determined based on the game state data at the current moment of the game match, based on preset reward rules. The preset reward rules include reward rules corresponding to each character position, and the reward rules corresponding to each character position indicate the reward value that the gaming agent with that character position can receive for interactive events occurring during the game match. A first loss value is determined based on the reward value and the evaluation value, and the value function network is trained based on the first loss value. And, determining a second loss value according to the action probability distribution, the reward value and the evaluation value, and training the first policy network based on the second loss value.

[0084] Compared to related technologies, this disclosure enables better collaboration between game agents with different role positions by setting different reward rules for them. Furthermore, by training game agents with different role positions to adopt different behavioral strategies, the diversity of the game agent's role behavior is increased, thereby improving the game agent's teamwork and behavioral strategy capabilities.

[0085] Based on the above embodiments, the training method of the game agent provided by the embodiment of the present disclosure is further explained below.

[0086] In an optional implementation, before step S20, the game agent winning training method provided by the embodiment of the present disclosure may further include step S50:

[0087] S50: Based on the preset role positioning allocation rules and the game status data, determine the role positioning of each game character.

[0088] As described above, the preset role positioning allocation rules are used to reallocate the role positioning of the game characters of the camp to which the game agent belongs according to the game status data during the game match.

[0089] In the disclosed embodiment, after the game agent performs the target action, the game state data (i.e., the latest game state data) at the current moment of the game match is obtained, wherein the latest game state data includes the state data of the game character. Based on the preset role positioning rules, the role positioning of the game character of the camp to which the game agent belongs is reallocated according to the latest game state data. Considering that the game state data changes at any time, by dynamically adjusting the role positioning of the game character of the camp to which the game agent belongs, the game agents with different role positionings and the game characters of other non-game agents (i.e., the game characters controlled by real players) can make more conducive actions to win. Moreover, by dynamically adjusting the role positioning, the game agents with different roles can make more appropriate decisions according to the current game state, improve the flexibility of the game agent's behavior strategy, optimize teamwork, and improve the winning rate of the game. For example, in the process of the game match, more game agents are needed to undertake the game task of restoring the health of teammates. At this time, the role positioning of other game agents positioned as non-therapeutic types can be dynamically adjusted to therapeutic types.

[0090] In an optional implementation, the game status data also includes status data of the game situation, the status data of the game situation includes status data of whether the target virtual items of the two hostile camps have been robbed, the character positioning includes the object-grabbing role, the object-protecting role and the object-guarding role, and the status data of the game character includes the position information of the game character in the game scene.

[0091] The game state data at the current moment of the game match also includes game status information. This game status information may include information reflecting the game progress of both teams, such as the current level and mission completion status of both teams. It may also include game map layout, game scene information, and the like. These are examples only and are not limited in any way by the presently disclosed embodiments.

[0092] For example, let's take the aforementioned capture the flag game as an example. In this game, the game status data may include status information of the flags of both camps (e.g., whether a flag has been captured, the number of captured flags, the number of uncaptured flags, the current position of the flags, whether either camp has won at the current moment, and the respective capture-the-flag scores of both camps. For example, a value of 1 indicates that the current capture-the-flag score of the own camp is greater than the enemy camp's, and a value of 0 indicates that the current capture-the-flag score of the own camp is less than or equal to the enemy camp's).

[0093] An optional implementation of step S50 includes steps S501-S502.

[0094] S501: Determine whether the target virtual item of the camp to which the game agent belongs has been snatched by a game character of the hostile camp. If so, locate a first number of game characters in the camp to which the game agent belongs that are closest to the target virtual item and determine them as the characters that will snatch the item. If not, locate a second number of game characters in the camp to which the game agent belongs that are closest to the target virtual item and determine them as the characters that will snatch the item, where the first number is greater than or equal to the second number and the first number is less than the number of characters in the camp to which the game agent belongs.

[0095] S502: Determine whether the camp to which the game agent belongs has captured the target virtual item of the enemy camp. If so, determine the role positions of all other game characters in the camp to which the game agent belongs, except for the game character that captured the target virtual item of the enemy camp, as guardian characters. If not, determine the role positions of all other game characters in the camp to which the game agent belongs, as capture characters.

[0096] For example, in an 8v8 capture-the-flag game, for example, the eight characters in the capture-the-flag camp (referred to as the "our camp") are dynamically repositioned. The "our camp" has three role positions: flag-capturing, flag-guarding, and flag-defending. (For example, using the target virtual object as a flag, the capture, guard, and defend roles in the role positions correspond to the capture, guard, and defend roles, respectively.) The current game state data for the game match includes game situation status data and character status data. The game situation status data includes whether the opposing camp's flag has been captured, and the character status data includes the character's position within the game scene. Specifically, flag-defending characters are first assigned. If the flag of the game agent's camp has not been captured, the two to three characters closest to the player's flag in the game agent's camp are selected as flag-guarding characters. If the flag of the player's camp has been captured, the three to four characters closest to the player's flag in the game agent's camp are selected as flag-defending characters. Next, assign flag-capturing characters. If the flag of the enemy camp to which the agent belongs has not yet been captured, all remaining characters in the agent's camp that have not been assigned a role position are selected as flag-capturing characters. If the flag of the enemy camp to which the agent belongs has been captured, all remaining characters in the agent's camp that have not been assigned a role position are selected as flag-guarding characters, excluding the flag-capturing character.

[0097] In the disclosed embodiment, based on the status data of whether the target virtual items of the two hostile camps have been robbed and the position information of the game characters in the game scene, the role positions of the game characters in the game camp are reallocated, which can optimize the flexibility of the game agent's tactical decision-making, enhance the teamwork ability of the game agent, adapt to changes in the battle situation, and ultimately improve the winning rate of the game.

[0098] In an optional implementation, after step S40, the method for training a game agent provided by the embodiment of the present disclosure may further include the following step S60:

[0099] S60: When a preset convergence condition is met, determine that the first strategy network training is completed.

[0100] Among them, the preset convergence conditions include any one of the following: the number of training times is greater than or equal to the preset training times threshold, the reward value of the game agent is greater than the preset reward threshold, and the difference in reward values ​​of game agents in different game games is less than or equal to the preset reward difference threshold.

[0101] In the embodiment of the present disclosure, when the first policy network satisfies any one of the above-mentioned preset convergence conditions, it can be considered that the first policy network has been trained and can be put into practical application, that is, used to make behavioral decisions for the game agent in an actual game match.

[0102] In an optional embodiment, the action type of the target action includes any one of the following: displacement, seizure, return, handover, and combat. The training method for the game agent provided in the embodiment of the present disclosure further includes the following steps S701-S702:

[0103] S701: When the action type of the target action is combat, determine a combat teammate and a combat enemy character for the game agent based on a preset target selection rule and the role positioning of the game agent. The preset target selection rule is used to assign one combat teammate and two combat enemy characters to the game agent, so that the game agent can fight against the two combat enemy characters in conjunction with the combat teammate.

[0104] S702. Input the target state data of the game agent, combat teammates, and combat hostile characters into the pre-trained second strategy network, and obtain the target combat action and target character of the game agent output by the second strategy network to control the game agent to perform the target combat action against the target character.

[0105] The target state data includes one or more of the state data of the game character data, which is only an example and is not limited in the embodiments of the present disclosure.

[0106] In the disclosed embodiment, the action type of the target action of the game agent output by the first policy network includes any one of the following: displacement, seizure, return, handover, and combat. In the case where the action type of the target action is displacement, seizure, return, or handover, the game agent directly executes the target action. In the case where the action type of the target action is combat, combat teammates and combat hostile characters are determined for the game agent based on the preset target selection rules and the role positioning of the game agent. Among them, the preset target selection rules are used to specify a combat teammate and two combat hostile characters for the game agent, so that the game agent can fight with the two combat hostile characters in conjunction with the combat teammate. Subsequently, the target state data of the game agent, the combat teammate, and the combat hostile character are input into the pre-trained second policy network to obtain the target combat action and target character of the game agent output by the second policy network, so as to control the game agent to perform the target combat action against the target character.

[0107] In the disclosed embodiment, combat teammates and combat hostile characters are determined for the game agent according to preset target selection rules and the role positioning of the game agent, and combat teammates and combat hostile characters are determined for the game agent according to preset target selection rules and the role positioning of the game agent. This can enhance the teamwork ability of the game agent, optimize tactical decision-making, and improve the survivability of the game agent.

[0108] In an optional implementation, the target state data includes skill information. A possible implementation of step S702 includes the following step S7021:

[0109] S7021. Input the target state data of the game agent, combat teammates, and combat hostile characters into the pre-trained second strategy network, and obtain the target skills and target characters of the game agent output by the second strategy network to control the game agent to perform the target skills against the target character.

[0110] In this disclosed embodiment, the output of the second policy network allows the game agent to more accurately select skills appropriate for the current situation and target character. Based on the input target state data, the second policy network can predict the optimal skills the game agent should perform, thereby improving the game agent's combat efficiency and damage output.

[0111] Exemplarily, step S7021 is exemplarily described in conjunction with FIG5 and a specific example. FIG5 is a flowchart of generating a game agent behavior strategy provided by one embodiment of the present disclosure.

[0112] For example, taking the capture the flag game as an example, the action types of the target action of the game agent output by the first strategy network 501 include displacement, capture the flag, return the flag, exchange the flag, and battle. When the action type of the target action is any one of displacement, capture the flag, return the flag, and exchange the flag, the game agent directly executes the target action. When the action type of the target action is battle, it is necessary to select a combat teammate and two combat hostile characters in combination with the role positioning of the game agent and the preset target selection rules. Subsequently, the target state data of the game agent, combat teammate, and combat hostile character are input into the pre-trained second strategy network 502 to obtain the target skills and target characters of the game agent output by the second strategy network 502 to control the game agent to perform the target skills against the target characters.

[0113] In an optional embodiment, the role positioning includes the object-grabbing role. A possible implementation of step S701 of "determining combat teammates and combat enemy characters for the game agent based on the preset target selection rules and the role positioning of the game agent" includes step S7011:

[0114] S7011: If the game agent is assigned to a looting role, the game agent's closest game character in the camp to which the game agent belongs is determined as a combat teammate. Furthermore, the two closest game characters in the enemy camp of the game agent's camp are determined as combat enemy characters.

[0115] In an optional embodiment, the role positioning includes the role of grabbing the object and the role of protecting the object. A possible implementation of step S701 of "determining the combat teammates and combat enemy roles for the game agent based on the preset target selection rules and the role positioning of the game agent" includes steps S7012-S7014:

[0116] S7012. When the role of the game agent is positioned as a guardian, determine whether the camp to which the game agent belongs has captured the target virtual item.

[0117] S7013. If not, the game character closest to the game agent in the camp to which the game agent belongs is determined as a combat teammate, and the two game characters closest to the game agent in the enemy camp of the camp to which the game agent belongs are determined as combat enemy characters.

[0118] S7014. If so, the game character in the camp to which the game agent belongs is positioned as the object-stealing character and is determined as a combat teammate, and the two game characters in the enemy camp of the camp to which the game agent belongs who are closest to the game character in the camp to which the game agent belongs who is positioned as the object-stealing character are determined as combat hostile characters.

[0119] In an optional embodiment, role positioning includes a guarding role. A possible implementation of step S701 of "determining combat teammates and combat enemy roles for the game agent based on preset target selection rules and the role positioning of the game agent" includes steps S7015-S7017:

[0120] S7015. When the role of the game agent is positioned as an object guarding role, determine whether the enemy camp of the camp to which the game agent belongs has captured the target virtual object.

[0121] S7016. If not, the game character closest to the game agent in the camp to which the game agent belongs is determined as a combat teammate, and the two game characters closest to the game agent in the enemy camp of the camp to which the game agent belongs are determined as combat enemy characters.

[0122] S7017: If yes, determine whether the distance between the game character in the enemy camp positioned as the looting character and the camp base of the enemy camp is less than a preset distance threshold. If yes, determine the game character in the camp to which the game agent belongs that is closest to the game agent as a combat teammate, and determine the game character in the enemy camp positioned as the looting character and the game character closest to the game agent as combat hostile characters. If no, determine the game character in the camp to which the game agent belongs that is closest to the game agent as a combat teammate, and determine the two game characters in the enemy camp to which the game agent belongs that are closest to the game agent as combat hostile characters.

[0123] 6 , taking the capture the flag game as an example, the flowchart of the preset target selection rule provided by the embodiment of the present disclosure is exemplarily described. FIG. 6 is a flowchart of the preset target selection rule provided by one embodiment of the present disclosure.

[0124] Taking the capture-the-flag game as an example, we'll illustrate how a game agent identifies one combat teammate and two combat enemy characters. As shown in Figure 6, first, the game agent's role positioning is input. If the game agent's role positioning is that of a capture-the-flag character, the game character closest to the game agent in the team to which it belongs is determined as a combat teammate. Furthermore, the two game characters closest to the game agent in the team's enemy team are determined as combat enemy characters.

[0125] As shown in Figure 6, when the game agent's character is positioned as a flag-guarding character, it is determined whether the camp to which the game agent belongs has captured the target virtual flag. If not, the game character closest to the game agent in the camp to which the game agent belongs is determined as a combat teammate. Furthermore, the two game characters closest to the game agent in the enemy camp of the game agent's camp are determined as combat enemy characters. If so, the game character in the camp to which the game agent belongs that is positioned as a flag-capturing character is determined as a combat teammate, and the two game characters in the enemy camp of the game agent's camp that are closest to the game character positioned as a flag-capturing character in the camp to which the game agent belongs are determined as combat enemy characters.

[0126] As shown in FIG6 , when the role of the game agent is positioned as a flag-guarding role, it is determined whether the enemy camp of the camp to which the game agent belongs has captured the target virtual flag.

[0127] If the enemy camp of the camp to which the game agent belongs fails to capture the target virtual flag, the game character closest to the game agent in the camp to which the game agent belongs will be determined as a combat teammate, and the two game characters closest to the game agent in the enemy camp of the camp to which the game agent belongs will be determined as combat enemy characters.

[0128] In the case where the enemy camp of the camp to which the game agent belongs has captured the target virtual flag, it is determined whether the distance between the game character whose role is positioned as the flag-grabbing character in the enemy camp and the camp base of the enemy camp is less than a preset distance threshold (such as 180 meters). If so, the game character closest to the game agent in the camp to which the game agent belongs is determined as a combat teammate, and the game character whose role is positioned as the flag-grabbing character in the enemy camp and another enemy character closest to the game agent are determined as combat hostile characters. If not, the game character closest to the game agent in the camp to which the game agent belongs is determined as a combat teammate, and the two game characters closest to the game agent in the enemy camp of the camp to which the game agent belongs are determined as combat hostile characters.

[0129] The following describes the training device for the game agent provided by the present disclosure. The training device for the game agent described below and the training method for the game agent described above can be referenced to each other.

[0130] FIG7 is a schematic diagram of the structure of a training device for a game agent according to one embodiment of the present disclosure. As shown in FIG7 , the training device for a game agent includes: an acquisition module 701 , a processing module 702 , a determination module 703 , and a training module 704 .

[0131] The acquisition module 701 is used to obtain the game state data of the current moment of the game, and the game state data includes the state data of the game characters. The game characters are all virtual characters of the two opposing camps in the game. The game characters include at least one game agent, and each camp of the game includes multiple character positions; the processing module 702 is used to input the game state data as input data to the first policy network and the value function network to be trained, and obtain the action probability distribution output by the first policy network and the evaluation value output by the value function network; wherein the action probability distribution is used to guide the game agent to select the action with the highest probability as the target action, and the evaluation value is used to evaluate the value of the game state data. Small; a determination module 703 is used to obtain and determine the reward value of the game agent based on the game status data at the current moment of the game game after the game agent performs the target action, based on the preset reward rules, the preset reward rules include reward rules corresponding to each role positioning, and the reward rules corresponding to the role positioning are used to indicate the reward value that the game agent with the role positioning can obtain for the interactive events that occur during the game game; a training module 704 is used to determine a first loss value based on the reward value and the evaluation value, and train the value function network based on the first loss value; and determine a second loss value based on the action probability distribution, the reward value and the evaluation value, and train the first strategy network based on the second loss value.

[0132] Compared with the existing technology, the training device of the game agent sets different reward rules for game agents with different role positioning, which can enable game agents with different role positioning to fight together better; and by training game agents with different role positioning to adopt different behavioral strategies, the diversity of the role behaviors of game agents in the game is increased, thereby improving the team collaboration ability and behavioral strategy ability of the game agents.

[0133] Optionally, the processing module 702 is further configured to determine the role positioning of each game character based on the game state data based on a preset role positioning allocation rule. Through the above-mentioned processing module, considering that the game state data is changing at any time, by dynamically adjusting the role positioning of the game character of the camp to which the game agent belongs, the game agents with different role positionings and the game characters of other non-game agents (i.e., game characters controlled by real players) can make more conducive behaviors to winning. Moreover, by dynamically adjusting the role positioning, the game agents with different roles can make more appropriate decisions according to the current game state, thereby improving the flexibility of the game agent's behavior strategy, optimizing team collaboration, and increasing the winning rate of the game.

[0134] Optionally, the game status data also includes the status data of the game situation, the status data of the game situation includes the status data of whether the target virtual items of the two hostile camps have been snatched, the role positioning includes the object-grabbing role, the object-protecting role and the object-guarding role, and the status data of the game character includes the position information of the game character in the game scene; the processing module 702 is specifically used to: determine whether the target virtual item of the camp to which the game agent belongs has been snatched by the game character of the hostile camp, if so, the role positioning of the first number of game characters closest to the target virtual item in the camp to which the game agent belongs is determined as the object-grabbing role; if not, the game agent is positioned as the object-grabbing role The role positioning of the second number of game characters closest to the target virtual item in the camp to which the game agent belongs is determined as the object-grabbing role, the first number is greater than or equal to the second number, and the first number is less than the number of characters in the camp to which the game agent belongs; it is determined whether the camp to which the game agent belongs has captured the target virtual item of the enemy camp. If so, the role positioning of the other game characters in the camp to which the game agent belongs, except for the game character that captured the target virtual item of the enemy camp, who are not assigned role positioning, are determined as the object-guarding role; if not, the role positioning of the other game characters in the camp to which the game agent belongs who are not assigned role positioning are determined as the object-grabbing role. Through the above-mentioned processing module, according to the status data of whether the target virtual item of the two enemy camps has been robbed and the position information of the game characters in the game scene, the role positioning of the game characters in the game camp is reallocated, which can optimize the flexibility of the game agent's tactical decision-making, enhance the teamwork ability of the game agent, adapt to changes in the battle situation, and ultimately improve the winning rate of the game.

[0135] Optionally, the training module 704 is further configured to determine that training of the first policy network is complete if a preset convergence condition is met; the preset convergence condition includes any one of the following: the number of training sessions is greater than or equal to a preset training number threshold, the reward value of the game agent is greater than a preset reward threshold, and the difference in reward values ​​between game agents in different game matches is less than or equal to a preset reward difference threshold. In this processing module, if the first policy network meets any of the above preset convergence conditions, the first policy network is considered to have been trained and can be used in practical applications, i.e., used to make behavioral decisions for the game agent in actual game matches.

[0136] Optionally, the action type of the target action includes any one of the following: displacement, capture, return, handover, and combat. The processing module 702 is further configured to: when the action type of the target action is combat, determine combat teammates and combat enemy characters for the game agent based on a preset target selection rule and the role positioning of the game agent; wherein the preset target selection rule is configured to specify one combat teammate and two combat enemy characters for the game agent, so that the game agent, in conjunction with the combat teammate, fights the two combat enemy characters; input the target state data of the game agent, the combat teammate, and the combat enemy characters into a pre-trained second policy network, and obtain the target combat action and target character of the game agent output by the second policy network, so as to control the game agent to execute the target combat action against the target character. In this processing module, combat teammates and combat enemy characters are determined for the game agent based on the preset target selection rule and the role positioning of the game agent, and combat teammates and combat enemy characters are determined for the game agent based on the preset target selection rule and the role positioning of the game agent. This can enhance the teamwork ability of the game agent, optimize tactical decision-making, and improve the survivability of the game agent.

[0137] Optionally, the target state data includes skill information; processing module 702 is specifically configured to input the target state data of the game agent, combat teammates, and combat enemy characters into a pre-trained second policy network, and obtain the target skills and target characters of the game agent output by the second policy network, thereby controlling the game agent to perform the target skills against the target characters. In this processing module, the output of the second policy network allows the game agent to more accurately select skills appropriate for the current situation and target character. Based on the input target state data, the second policy network can predict the optimal skills that the game agent should perform, thereby improving the game agent's combat efficiency and damage output.

[0138] Optionally, the role positioning includes the object-stealing role; the processing module 702 is specifically used to: when the role of the game agent is positioned as the object-stealing role, determine the game character closest to the game agent in the camp to which the game agent belongs as a combat teammate; and determine the two game characters closest to the game agent in the enemy camp of the camp to which the game agent belongs as combat enemy characters.

[0139] Optionally, role positioning includes an item-grabbing role and an item-protecting role; processing module 702 is specifically used to: when the role of the game agent is positioned as an item-protecting role, determine whether the camp to which the game agent belongs has captured the target virtual item; if not, determine the game character in the camp to which the game agent belongs that is closest to the game agent as a combat teammate, and determine the two game characters in the enemy camp of the camp to which the game agent belongs that are closest to the game agent as combat hostile characters; if so, determine the game character in the camp to which the game agent belongs that is positioned as an item-grabbing role as a combat teammate, and determine the two game characters in the enemy camp of the camp to which the game agent belongs that are closest to the game character positioned as an item-grabbing role as a combat hostile character.

[0140] Optionally, role positioning includes an object-guarding role; processing module 702 is specifically used to: when the role of the game agent is positioned as an object-guarding role, determine whether the enemy camp of the camp to which the game agent belongs has captured the target virtual item; if not, determine the game character in the camp to which the game agent belongs that is closest to the game agent as a combat teammate, and determine the two game characters in the enemy camp to which the game agent belongs that are closest to the game agent as combat hostile characters; if so, determine whether the distance between the game character in the enemy camp that is positioned as an object-grabbing character and the camp base of the enemy camp is less than a preset distance threshold; if so, determine the game character in the camp to which the game agent belongs that is closest to the game agent as a combat teammate, and determine the game character in the enemy camp that is positioned as an object-grabbing character and the game character closest to the game agent as combat hostile characters; if not, determine the game character in the camp to which the game agent belongs that is closest to the game agent as a combat teammate, and determine the two game characters in the enemy camp to which the game agent belongs that are closest to the game agent as combat hostile characters.

[0141] The game agent training device provided in this embodiment can be used to execute the technical solution of the above-mentioned game agent training method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0142] FIG8 is a schematic diagram of the hardware structure of an electronic device provided by one embodiment of the present disclosure. As shown in FIG8 , the electronic device 800 of this embodiment includes: a processor 801 and a memory 802;

[0143] The memory 802 is used to store computer-executable instructions; the processor 801 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the method for training a game agent in the above embodiment. For example:

[0144] Obtaining game state data at the current moment of a game match, the game state data including state data of game characters. Game characters are all virtual characters of the opposing camps in the game match, including at least one game agent, and each camp in the game match includes multiple character positions. The game state data is input as input data into a first policy network and a value function network to be trained, thereby obtaining an action probability distribution output by the first policy network and an evaluation value output by the value function network. The action probability distribution is used to guide the game agent to select the action with the highest probability as the target action, and the evaluation value is used to evaluate the value of the game state data. After the game agent executes the target action, obtaining and determining a reward value for the game agent based on preset reward rules based on the game state data at the current moment of the game match, the preset reward rules include reward rules corresponding to each character position, and the reward rules corresponding to the character position are used to indicate the reward value that the game agent of the character position can obtain for interactive events that occur during the game match. Determining a first loss value based on the reward value and the evaluation value, and training the value function network based on the first loss value. Finally, determining a second loss value based on the action probability distribution, the reward value, and the evaluation value, and training the first policy network based on the second loss value.

[0145] Compared with the existing technology, this application sets different reward rules for game agents with different role positioning, which can enable game agents with different role positioning to fight together better; and by training game agents with different role positioning to adopt different behavioral strategies, the diversity of the role behaviors of game agents in the game is increased, thereby improving the team collaboration ability and behavioral strategy ability of game agents.

[0146] Optionally, the game state data is used as input data and input into the first policy network and the value function network to be trained, and before obtaining the action probability distribution output by the first policy network and the evaluation value output by the value function network, it also includes: determining the role positioning of each game character according to the game state data based on the preset role positioning allocation rules. Through the above embodiment, considering that the game state data changes at any time, by dynamically adjusting the role positioning of the game character of the camp to which the game agent belongs, the game agents with different role positionings and the game characters of other non-game agents (i.e., game characters controlled by real players) can make more conducive to winning behaviors, and by dynamically adjusting the role positioning, the game agents of different roles can make more appropriate decisions according to the current game state, thereby improving the flexibility of the game agent's behavior strategy, optimizing teamwork, and improving the winning rate of the game.

[0147] Optionally, the game status data also includes status data of the game situation, the status data of the game situation includes status data of whether the target virtual items of the two hostile camps have been snatched away, the role positioning includes the role of grabbing the item, the role of protecting the item and the role of guarding the item, and the status data of the game character includes the position information of the game character in the game scene; based on the preset role positioning allocation rules, the role positioning of each game character is determined according to the game status data, including: judging whether the target virtual item of the camp to which the game intelligent agent belongs has been snatched away by the game character of the hostile camp, and if so, determining the role positioning of the first number of game characters in the camp to which the game intelligent agent belongs that are closest to the target virtual item as the ones that grab the item. object role; if not, the role positioning of the second number of game characters closest to the target virtual object in the camp to which the game intelligent body belongs are determined as object-grabbing roles, the first number is greater than or equal to the second number, and the first number is less than the number of characters in the camp to which the game intelligent body belongs; judge whether the camp to which the game intelligent body belongs has seized the target virtual object of the enemy camp. If so, the role positioning of other game characters in the camp to which the game intelligent body belongs, except for the game character that seized the target virtual object of the enemy camp, which have not been assigned role positioning, are determined as object-guarding roles; if not, the role positioning of other game characters in the camp to which the game intelligent body belongs, which have not been assigned role positioning, are determined as object-grabbing roles.

[0148] Through the above embodiments, the role positions of the game characters in the game camps are reallocated according to the status data of whether the target virtual items of the two hostile camps have been robbed and the position information of the game characters in the game scene. This can optimize the flexibility of the game agent's tactical decision-making, enhance the teamwork ability of the game agent, adapt to changes in the battle situation, and ultimately improve the winning rate of the game.

[0149] Optionally, after determining a first loss value based on the reward value and the evaluation value, and training the value function network based on the first loss value; and determining a second loss value based on the action probability distribution, the reward value, and the evaluation value, and training the first policy network based on the second loss value, the method further includes: determining that the training of the first policy network is complete if a preset convergence condition is met; wherein the preset convergence condition includes any one of the following: the number of trainings is greater than or equal to a preset training number threshold, the reward value of the game agent is greater than a preset reward threshold, and the difference in reward values ​​between game agents in different game matches is less than or equal to a preset reward difference threshold. In this embodiment, if the first policy network meets any of the above preset convergence conditions, the first policy network can be considered to have been trained and can be put into practical application, i.e., used to make behavioral decisions for the game agent in actual game matches.

[0150] Optionally, the action type of the target action includes any one of the following: displacement, capture, return, handover, and combat; the method further includes: when the action type of the target action is combat, determining a combat teammate and a combat hostile character for the game agent based on a preset target selection rule and the role positioning of the game agent; wherein the preset target selection rule is used to specify one combat teammate and two combat hostile characters for the game agent, so that the game agent, in conjunction with the combat teammate, fights the two combat hostile characters; inputting the target state data of the game agent, the combat teammate, and the combat hostile character into a pre-trained second policy network, obtaining the target combat action and target character of the game agent output by the second policy network, and controlling the game agent to execute the target combat action against the target character. In this embodiment, determining the combat teammate and combat hostile character for the game agent based on the preset target selection rule and the role positioning of the game agent, and determining the combat teammate and combat hostile character for the game agent based on the preset target selection rule and the role positioning of the game agent, can enhance the teamwork ability of the game agent, optimize tactical decision-making, and improve the survivability of the game agent.

[0151] Optionally, the target state data includes skill information; the target state data of the game agent, combat teammates, and combat hostile characters are input into a pre-trained second strategy network, and the target combat action and target character of the game agent are obtained from the second strategy network output, including: the target state data of the game agent, combat teammates, and combat hostile characters are input into a pre-trained second strategy network, and the target skills and target character of the game agent are obtained from the second strategy network output, so as to control the game agent to perform the target skills against the target character. In this embodiment, through the output of the second strategy network, the game agent can more accurately select the skills that are suitable for the current situation and the target character. The second strategy network can predict the optimal skills that the game agent should perform based on the input target state data, thereby improving the efficiency and damage output of the game agent's combat.

[0152] Optionally, the role positioning includes an item-stealing role; based on preset target selection rules and the role positioning of the game agent, combat teammates and combat hostile characters are determined for the game agent, including: when the role positioning of the game agent is an item-stealing role, the game character closest to the game agent in the camp to which the game agent belongs is determined as a combat teammate; and the two game characters closest to the game agent in the hostile camp of the camp to which the game agent belongs are determined as combat hostile characters.

[0153] Optionally, the role positioning includes an item-grabbing role and an item-protecting role; according to the preset target selection rules and the role positioning of the game agent, combat teammates and combat hostile characters are determined for the game agent, including: when the role of the game agent is positioned as an item-protecting role, judging whether the camp to which the game agent belongs has captured the target virtual item; if not, determining the game character in the camp to which the game agent belongs that is closest to the game agent as a combat teammate, and determining the two game characters in the enemy camp of the camp to which the game agent belongs that are closest to the game agent as combat hostile characters; if so, determining the game character in the camp to which the game agent belongs that is positioned as an item-grabbing role as a combat teammate, and determining the two game characters in the enemy camp of the camp to which the game agent belongs that are closest to the game character in the camp to which the game agent belongs that is positioned as an item-grabbing role as a combat teammate, and determining the two game characters in the enemy camp of the camp to which the game agent belongs that are closest to the game character in the camp to which the game agent belongs that is positioned as an item-grabbing role as a combat hostile character.

[0154] Optionally, role positioning includes an object-guarding role; according to preset target selection rules and the role positioning of the game agent, combat teammates and combat hostile roles are determined for the game agent, including: when the role positioning of the game agent is an object-guarding role, judging whether the hostile camp of the camp to which the game agent belongs has captured the target virtual item; if not, determining the game character closest to the game agent in the camp to which the game agent belongs as a combat teammate, and determining the two game characters closest to the game agent in the hostile camp of the camp to which the game agent belongs as combat hostile characters; if so, judging the role positioning in the hostile camp Whether the distance between the game character that is the object-stealing character and the camp base of the enemy camp is less than a preset distance threshold; if so, the game character that is closest to the game agent in the camp to which the game agent belongs is determined as a combat teammate, and the game character in the enemy camp that is positioned as the object-stealing character and the game character that is closest to the game agent are determined as combat hostile characters; if not, the game character that is closest to the game agent in the camp to which the game agent belongs is determined as a combat teammate, and the two game characters that are closest to the game agent in the enemy camp of the camp to which the game agent belongs are determined as combat hostile characters.

[0155] Optionally, the memory 802 may be independent or integrated with the processor 801 .

[0156] When the memory 802 is independently provided, the electronic device further includes a bus 803 for connecting the memory 802 and the processor 801 .

[0157] One embodiment of the present disclosure further provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, a technical solution corresponding to the method for training a game agent in any of the above embodiments is implemented as executed by the electronic device.

[0158] One embodiment of the present disclosure further provides a computer program product, which includes: a computer program, which is stored in a readable storage medium, and at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the technical solution corresponding to the training method of the game agent in any of the above embodiments.

[0159] Specifically, game state data at the current moment of a game match is obtained, the game state data including state data of game characters, which are all virtual characters of the opposing camps in the game match, including at least one game agent, and each camp in the game match includes multiple character positions; the game state data is input as input data into a first policy network and a value function network to be trained, thereby obtaining an action probability distribution output by the first policy network and an evaluation value output by the value function network; wherein the action probability distribution is used to guide the game agent to select the action with the highest probability as the target action, and the evaluation value is used to evaluate the value of the game state data; after the game agent executes the target action, a reward value for the game agent is obtained and determined based on preset reward rules based on the game state data at the current moment of the game match, wherein the preset reward rules include reward rules corresponding to each character position, and the reward rules corresponding to the character position are used to indicate the reward value that the game agent of the character position can obtain for interactive events that occur during the game match; a first loss value is determined based on the reward value and the evaluation value, and the value function network is trained based on the first loss value; and a second loss value is determined based on the action probability distribution, the reward value, and the evaluation value, and the first policy network is trained based on the second loss value.

[0160] Compared with the existing technology, this application sets different reward rules for game agents with different role positioning, which can enable game agents with different role positioning to fight together better; and by training game agents with different role positioning to adopt different behavioral strategies, the diversity of the role behaviors of game agents in the game is increased, thereby improving the team collaboration ability and behavioral strategy ability of game agents.

[0161] Optionally, the game state data is used as input data and input into the first policy network and the value function network to be trained, and before obtaining the action probability distribution output by the first policy network and the evaluation value output by the value function network, it also includes: determining the role positioning of each game character according to the game state data based on the preset role positioning allocation rules. Through the above embodiment, considering that the game state data changes at any time, by dynamically adjusting the role positioning of the game character of the camp to which the game agent belongs, the game agents with different role positionings and the game characters of other non-game agents (i.e., game characters controlled by real players) can make more conducive to winning behaviors, and by dynamically adjusting the role positioning, the game agents of different roles can make more appropriate decisions according to the current game state, thereby improving the flexibility of the game agent's behavior strategy, optimizing teamwork, and improving the winning rate of the game.

[0162] Optionally, the game status data also includes status data of the game situation, the status data of the game situation includes status data of whether the target virtual items of the two hostile camps have been snatched away, the role positioning includes the role of grabbing the item, the role of protecting the item and the role of guarding the item, and the status data of the game character includes the position information of the game character in the game scene; based on the preset role positioning allocation rules, the role positioning of each game character is determined according to the game status data, including: judging whether the target virtual item of the camp to which the game intelligent agent belongs has been snatched away by the game character of the hostile camp, and if so, determining the role positioning of the first number of game characters in the camp to which the game intelligent agent belongs that are closest to the target virtual item as the ones that grab the item. object role; if not, the role positioning of the second number of game characters closest to the target virtual object in the camp to which the game intelligent body belongs are determined as object-grabbing roles, the first number is greater than or equal to the second number, and the first number is less than the number of characters in the camp to which the game intelligent body belongs; judge whether the camp to which the game intelligent body belongs has seized the target virtual object of the enemy camp. If so, the role positioning of other game characters in the camp to which the game intelligent body belongs, except for the game character that seized the target virtual object of the enemy camp, which have not been assigned role positioning, are determined as object-guarding roles; if not, the role positioning of other game characters in the camp to which the game intelligent body belongs, which have not been assigned role positioning, are determined as object-grabbing roles.

[0163] Through the above embodiments, the role positions of the game characters in the game camps are reallocated according to the status data of whether the target virtual items of the two hostile camps have been robbed and the position information of the game characters in the game scene. This can optimize the flexibility of the game agent's tactical decision-making, enhance the teamwork ability of the game agent, adapt to changes in the battle situation, and ultimately improve the winning rate of the game.

[0164] Optionally, after determining a first loss value based on the reward value and the evaluation value, and training the value function network based on the first loss value; and determining a second loss value based on the action probability distribution, the reward value, and the evaluation value, and training the first policy network based on the second loss value, the method further includes: determining that the training of the first policy network is complete if a preset convergence condition is met; wherein the preset convergence condition includes any one of the following: the number of trainings is greater than or equal to a preset training number threshold, the reward value of the game agent is greater than a preset reward threshold, and the difference in reward values ​​between game agents in different game matches is less than or equal to a preset reward difference threshold. In this embodiment, if the first policy network meets any of the above preset convergence conditions, the first policy network can be considered to have been trained and can be put into practical application, i.e., used to make behavioral decisions for the game agent in actual game matches.

[0165] Optionally, the action type of the target action includes any one of the following: displacement, capture, return, handover, and combat; the method further includes: when the action type of the target action is combat, determining a combat teammate and a combat hostile character for the game agent based on a preset target selection rule and the role positioning of the game agent; wherein the preset target selection rule is used to specify one combat teammate and two combat hostile characters for the game agent, so that the game agent, in conjunction with the combat teammate, fights the two combat hostile characters; inputting the target state data of the game agent, the combat teammate, and the combat hostile character into a pre-trained second policy network, obtaining the target combat action and target character of the game agent output by the second policy network, and controlling the game agent to execute the target combat action against the target character. In this embodiment, determining the combat teammate and combat hostile character for the game agent based on the preset target selection rule and the role positioning of the game agent, and determining the combat teammate and combat hostile character for the game agent based on the preset target selection rule and the role positioning of the game agent, can enhance the teamwork ability of the game agent, optimize tactical decision-making, and improve the survivability of the game agent.

[0166] Optionally, the target state data includes skill information; the target state data of the game agent, combat teammates, and combat hostile characters are input into a pre-trained second strategy network, and the target combat action and target character of the game agent are obtained from the second strategy network output, including: the target state data of the game agent, combat teammates, and combat hostile characters are input into a pre-trained second strategy network, and the target skills and target character of the game agent are obtained from the second strategy network output, so as to control the game agent to perform the target skills against the target character. In this embodiment, through the output of the second strategy network, the game agent can more accurately select the skills that are suitable for the current situation and the target character. The second strategy network can predict the optimal skills that the game agent should perform based on the input target state data, thereby improving the efficiency and damage output of the game agent's combat.

[0167] Optionally, the role positioning includes an item-stealing role; based on preset target selection rules and the role positioning of the game agent, combat teammates and combat hostile characters are determined for the game agent, including: when the role positioning of the game agent is an item-stealing role, the game character closest to the game agent in the camp to which the game agent belongs is determined as a combat teammate; and the two game characters closest to the game agent in the hostile camp of the camp to which the game agent belongs are determined as combat hostile characters.

[0168] Optionally, the role positioning includes an item-grabbing role and an item-protecting role; according to the preset target selection rules and the role positioning of the game agent, combat teammates and combat hostile characters are determined for the game agent, including: when the role of the game agent is positioned as an item-protecting role, judging whether the camp to which the game agent belongs has captured the target virtual item; if not, determining the game character in the camp to which the game agent belongs that is closest to the game agent as a combat teammate, and determining the two game characters in the enemy camp of the camp to which the game agent belongs that are closest to the game agent as combat hostile characters; if so, determining the game character in the camp to which the game agent belongs that is positioned as an item-grabbing role as a combat teammate, and determining the two game characters in the enemy camp of the camp to which the game agent belongs that are closest to the game character in the camp to which the game agent belongs that is positioned as an item-grabbing role as a combat teammate, and determining the two game characters in the enemy camp of the camp to which the game agent belongs that are closest to the game character in the camp to which the game agent belongs that is positioned as an item-grabbing role as a combat hostile character.

[0169] Optionally, role positioning includes an object-guarding role; according to preset target selection rules and the role positioning of the game agent, combat teammates and combat hostile roles are determined for the game agent, including: when the role positioning of the game agent is an object-guarding role, judging whether the hostile camp of the camp to which the game agent belongs has captured the target virtual item; if not, determining the game character closest to the game agent in the camp to which the game agent belongs as a combat teammate, and determining the two game characters closest to the game agent in the hostile camp of the camp to which the game agent belongs as combat hostile characters; if so, judging the role positioning in the hostile camp Whether the distance between the game character that is the object-stealing character and the camp base of the enemy camp is less than a preset distance threshold; if so, the game character that is closest to the game agent in the camp to which the game agent belongs is determined as a combat teammate, and the game character in the enemy camp that is positioned as the object-stealing character and the game character that is closest to the game agent are determined as combat hostile characters; if not, the game character that is closest to the game agent in the camp to which the game agent belongs is determined as a combat teammate, and the two game characters that are closest to the game agent in the enemy camp of the camp to which the game agent belongs are determined as combat hostile characters.

[0170] Although the present disclosure is disclosed as above in terms of preferred embodiments, it is not intended to limit the present disclosure. Any person skilled in the art may make possible changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the scope of protection of the present disclosure shall be based on the scope defined by the claims of the present disclosure.

[0171] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or module, which can be electrical, mechanical or other forms.

[0172] The integrated modules implemented in the form of software function modules can be stored in a computer-readable storage medium. The software function modules stored in a storage medium include a number of instructions for causing an electronic device (which can be a personal computer, server, or network device, etc.) or a processor to perform some of the steps of the methods of various embodiments of the present disclosure.

[0173] It should be understood that the processor described above may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), or application-specific integrated circuits (ASICs). A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0174] The memory may include a high-speed RAM memory, and may also include non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk.

[0175] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, the buses in the figures of this disclosure are not limited to just one bus or just one type of bus.

[0176] The storage medium may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0177] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A method for training a game agent, the method comprising: Obtaining game state data of a current moment in a game match, the game state data including state data of game characters, the game characters being all virtual characters of two opposing camps in the game match, the game characters including at least one game agent, and each camp in the game match including multiple character positions; The game state data is input as input data to a first policy network and a value function network to be trained, and an action probability distribution output by the first policy network and an evaluation value output by the value function network are obtained; wherein the action probability distribution is used to guide the game agent to select the action with the highest probability as the target action, and the evaluation value is used to evaluate the value of the game state data; After the game agent completes the target action, a reward value for the game agent is obtained and determined based on game state data at the current moment of the game match and preset reward rules, wherein the preset reward rules include reward rules corresponding to each of the character positions, and the reward rules corresponding to the character positions are used to indicate the reward value that the game agent at the character position can obtain for an interaction event occurring during the game match; Determine a first loss value based on the reward value and the evaluation value, and train the value function network based on the first loss value; and determine a second loss value based on the action probability distribution, the reward value and the evaluation value, and train the first policy network based on the second loss value.

2. The method according to claim 1, wherein Before inputting the game state data as input data into the first policy network and the value function network to be trained, and obtaining the action probability distribution output by the first policy network and the evaluation value output by the value function network, the method further includes: Based on the preset role positioning allocation rules, the role positioning of each of the game characters is determined according to the game status data.

3. The method according to claim 2, wherein: The game status data also includes game situation status data, including whether the target virtual item of the two opposing camps has been snatched; the character positioning includes the object-grabbing role, the object-protecting role, and the object-guarding role; and the game character status data includes the position information of the game character in the game scene; The determining the role positioning of each of the game characters based on the preset role positioning allocation rules and according to the game status data includes: Determine whether the target virtual item of the camp to which the game agent belongs has been snatched by a game character of the hostile camp; if so, locate a first number of game characters in the camp to which the game agent belongs that are closest to the target virtual item and determine them as the characters that snatch the item; if not, locate a second number of game characters in the camp to which the game agent belongs that are closest to the target virtual item and determine them as the characters that snatch the item, where the first number is greater than or equal to the second number, and the first number is less than the number of characters in the camp to which the game agent belongs; Determine whether the camp to which the game agent belongs has seized the target virtual item of the hostile camp. If so, determine the role positioning of other game characters in the camp to which the game agent belongs who have not been assigned role positions, except for the game character that has seized the target virtual item of the hostile camp, as the object guarding role; if not, determine the role positioning of other game characters in the camp to which the game agent belongs who have not been assigned role positions, as the object seizing role.

4. The method according to claim 1, wherein After determining a first loss value based on the reward value and the evaluation value, and training the value function network based on the first loss value; and determining a second loss value based on the action probability distribution, the reward value, and the evaluation value, and training the first policy network based on the second loss value, the method further includes: When a preset convergence condition is met, determining that the first strategy network training is completed; Among them, the preset convergence conditions include any one of the following: the number of training times is greater than or equal to the preset training times threshold, the reward value of the game agent is greater than the preset reward threshold and the difference in reward values between game agents in different game matches is less than or equal to the preset reward difference threshold.

5. The method according to claim 1, wherein The target action type includes any one of the following: displacement, seizure, return, handover, and combat; the method further includes: When the action type of the target action is combat, determining a combat teammate and a combat hostile character for the game agent according to a preset target selection rule and the role positioning of the game agent; wherein the preset target selection rule is used to designate one combat teammate and two combat hostile characters for the game agent, so that the game agent joins forces with the combat teammate to fight against the two combat hostile characters; The target state data of the game agent, the combat teammates, and the combat hostile character are input into the pre-trained second strategy network, and the target combat action and target character of the game agent output by the second strategy network are obtained to control the game agent to perform the target combat action against the target character.

6. The method according to claim 5, wherein: The target state data includes skill information; the target state data of the game agent, the combat teammate, and the combat hostile character are input into a pre-trained second policy network to obtain the target combat action and target character of the game agent output by the second policy network, including: The target state data of the game agent, the combat teammates, and the combat hostile character are input into the pre-trained second strategy network, and the target skills and target character of the game agent output by the second strategy network are obtained to control the game agent to perform the target skills against the target character.

7. The method according to claim 5, wherein: The role positioning includes the object-grabbing role; and determining combat teammates and combat hostile roles for the game agent based on the preset target selection rules and the role positioning of the game agent, including: When the role of the game agent is positioned as a looting role, the game character closest to the game agent in the camp to which the game agent belongs is determined as a combat teammate; and the two game characters closest to the game agent in the enemy camp of the camp to which the game agent belongs are determined as combat enemy characters.

8. The method according to claim 5, wherein The role positioning includes the role of grabbing objects and the role of protecting objects; The step of determining combat teammates and combat hostile characters for the game agent based on preset target selection rules and the role positioning of the game agent includes: In the case where the role of the game agent is positioned as a guardian role, determining whether the camp to which the game agent belongs has captured the target virtual item; If not, the game character closest to the game agent in the camp to which the game agent belongs is determined as the combat teammate, and the two game characters closest to the game agent in the enemy camp of the camp to which the game agent belongs are determined as combat enemy characters; If so, the game character in the camp to which the game agent belongs is positioned as the object-stealing character and is determined as a combat teammate, and the two game characters in the enemy camp of the camp to which the game agent belongs that are closest to the game character in the camp to which the game agent belongs is positioned as the object-stealing character are determined as combat hostile characters.

9. The method according to claim 5, wherein: The role positioning includes a guarding role; and determining combat teammates and combat hostile roles for the game agent based on preset target selection rules and the role positioning of the game agent, including: When the role of the game agent is positioned as a guarding role, determining whether the enemy camp of the camp to which the game agent belongs has captured the target virtual item; If not, the game character closest to the game agent in the camp to which the game agent belongs is determined as the combat teammate, and the two game characters closest to the game agent in the enemy camp of the camp to which the game agent belongs are determined as combat enemy characters; If so, determine whether the distance between the game character positioned as the object-stealing character in the hostile camp and the camp base of the hostile camp is less than a preset distance threshold; if so, determine the game character in the camp to which the game agent belongs that is closest to the game agent as the combat teammate, and determine the game character in the hostile camp that is positioned as the object-stealing character and the game character closest to the game agent as combat hostile characters; if not, determine the game character in the camp to which the game agent belongs that is closest to the game agent as the combat teammate, and determine the two game characters closest to the game agent in the hostile camp of the camp to which the game agent belongs as combat hostile characters.

10. A device for training a game agent, comprising: an acquisition module configured to execute acquisition of game state data at a current moment of a game match, wherein the game state data includes state data of game characters, wherein the game characters are all virtual characters of two opposing camps in the game match, wherein the game characters include at least one game agent, and each camp in the game match includes multiple character positions; a processing module configured to input the game state data as input data into a first policy network and a value function network to be trained, and obtain an action probability distribution output by the first policy network and an evaluation value output by the value function network; wherein the action probability distribution is used to guide the game agent to select an action with the highest probability as a target action, and the evaluation value is used to evaluate the value of the game state data; a determination module configured to obtain and determine a reward value for the game agent based on preset reward rules after the game agent completes the target action, the reward value being obtained based on game state data at the current moment of the game match, the preset reward rules including reward rules corresponding to each of the character positions, the reward rules corresponding to the character positions being used to indicate a reward value that the game agent at the character position can obtain for an interaction event occurring during the game match; A training module is configured to determine a first loss value based on the reward value and the evaluation value, and train the value function network based on the first loss value; and determine a second loss value based on the action probability distribution, the reward value and the evaluation value, and train the first policy network based on the second loss value.

11. An electronic device, comprising: A processor and a memory, wherein the memory is used to store a data processing program. After the electronic device is powered on and the program is run by the processor, the training method for a game agent as described in any one of claims 1 to 9 is executed.

12. A computer-readable storage medium storing a data processing program, wherein the program is executed by a processor to perform the method for training a game agent according to any one of claims 1 to 9.

Images