Agent training method and device, electronic equipment, and storage medium

Abstract

The present disclosure provides an agent training method and device, an electronic device, and a storage medium, and belongs to the technical field of agent training. The method comprises the following steps: determining a first policy model of a main agent and a second policy model of a training agent, the first policy model being a model in a historical version pool of the main agent, and the second policy model being a model in a historical version pool of the training agent; updating the first policy model based on interaction data of the first policy model and the second policy model to obtain a third policy model, and updating the second policy model based on the interaction data of the first policy model and the second policy model to obtain a fourth policy model; adding the third policy model to the historical version pool of the main agent, and adding the fourth policy model to the historical version pool of the training agent. The agent training method and device, the electronic device, and the storage medium provided by the present disclosure can meet the actual requirements of policy variability and complex environment in asymmetric tasks.

Description

Technical Field

[0001] This disclosure belongs to the field of intelligent agent training technology, and more specifically, relates to an intelligent agent training method and apparatus, electronic device, and storage medium. Background Technology

[0002] In asymmetric tasks, such as offense and defense, agents need to quickly adapt to the opponent's changing strategies. The core of agent training lies in the agent's ability to adapt to dynamic and complex environments and handle the significant differences in task requirements between offense and defense. However, traditional methods, lacking strategy diversity and dynamic update mechanisms, struggle to cover complex combinations of adversarial strategies, causing agents to easily fall into "local optima" strategy patterns and failing to meet the practical needs of asymmetric tasks with changing strategies and complex environments. Summary of the Invention

[0003] The purpose of this disclosure is to provide an agent training method, apparatus, electronic device, and storage medium to meet the practical needs of asymmetric tasks with varied strategies and complex environments.

[0004] A first aspect of this disclosure provides an agent training method, comprising:

[0005] Determine the first policy model of the main agent and the second policy model of the training agent. The first policy model is a model in the historical version pool of the main agent, and the second policy model is a model in the historical version pool of the training agent.

[0006] The first strategy model is updated based on the interaction data between the first strategy model and the second strategy model to obtain the third strategy model, and the second strategy model is updated based on the interaction data between the first strategy model and the second strategy model to obtain the fourth strategy model.

[0007] The third strategy model is added to the historical version pool of the main agent, and the fourth strategy model is added to the historical version pool of the training agent.

[0008] A second aspect of this disclosure provides an intelligent agent training device, comprising:

[0009] The strategy model determination module is used to determine the first strategy model of the main agent and the second strategy model of the training agent. The first strategy model is a model in the historical version pool of the main agent, and the second strategy model is a model in the historical version pool of the training agent.

[0010] The strategy model update module is used to update the first strategy model based on the interaction data between the first strategy model and the second strategy model to obtain a third strategy model, and to update the second strategy model based on the interaction data between the first strategy model and the second strategy model to obtain a fourth strategy model.

[0011] The historical version pool update module is used to add the third strategy model to the historical version pool of the main agent and the fourth strategy model to the historical version pool of the training agent.

[0012] A third aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the above-described agent training method.

[0013] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described agent training method.

[0014] The beneficial effects of the agent training method, apparatus, electronic device, and storage medium provided in this disclosure are as follows: Firstly, by utilizing the historical version pools of the master agent and the training agent to determine the initial policy model, this disclosure lays a rich and diverse foundation for the training process, greatly enhancing the complexity and diversity of training. Secondly, updating based on data generated by the interaction of the two models effectively optimizes the policy according to actual feedback, enabling the agent to continuously improve its policy level through continuous adversarial learning and achieve iterative growth. Finally, re-incorporating the updated model into the historical version pool not only expands the types of policies in the pool, creating more diverse adversarial scenarios for subsequent training and effectively avoiding the dilemma of local optima, but also promotes the continuous evolution of the agent through the recycling of new models. This significantly enhances the agent's adaptability and competitiveness in complex tasks, promotes its continuous development and improvement in dynamic environments, and meets the practical needs of variable policies and complex environments in asymmetric tasks. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart illustrating an agent training method provided in an embodiment of this disclosure;

[0017] Figure 2 A schematic diagram of the training principle of an agent alliance provided in an embodiment of this disclosure;

[0018] Figure 3 A flowchart of agent alliance training provided in one embodiment of this disclosure;

[0019] Figure 4 This is a structural block diagram of an intelligent agent training device provided in an embodiment of the present disclosure;

[0020] Figure 5 This is a schematic block diagram of an electronic device provided according to an embodiment of the present disclosure. Detailed Implementation

[0021] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, so as to provide a thorough understanding of the embodiments of this disclosure. However, those skilled in the art will understand that this disclosure may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this disclosure with unnecessary detail.

[0022] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.

[0023] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating an agent training method according to an embodiment of the present disclosure. The method includes:

[0024] S101: Determine the first policy model of the main agent and the second policy model of the training agent. The first policy model is a model in the historical version pool of the main agent, and the second policy model is a model in the historical version pool of the training agent.

[0025] In this embodiment, a first policy model is selected from the historical version pool of the master agent, and a second policy model is selected from the historical version pool of the training agent. The historical version pools of the master and training agents store policy models of the agents at different training stages, providing diverse starting points for training. The initial models of the master and training agents can be obtained through pre-training. By selecting from the historical version pools, previous training results can be utilized, providing different policy foundations for subsequent training and increasing the diversity and complexity of training.

[0026] In this embodiment, the main agent includes a pre-trained model and a historical version pool. During initialization, the main agent loads a pre-trained model to ensure the initial policy possesses basic capabilities. The pre-trained model has been trained in relevant scenarios using other methods, aiming to provide a stable initial policy for subsequent alternating training. The historical version pool is constructed by saving the policy model version of the main agent after each training round. It is designed as a queue or storage list structure, recording the policy evolution process of the main agent in chronological order. The diversity of the historical version pool provides the training agent module with rich opponent selection options, helping the training agent discover policy vulnerabilities in the main agent during training and further optimize the policy.

[0027] Similar to the main agent module, the training agent also includes a pre-trained model and a historical version pool. During system initialization, the training agent loads a pre-trained model that provides basic adversarial strategies, ensuring the training agent can generate an effective adversarial environment during early training with the main agent. As training progresses, the training agent's strategy model is continuously saved and updated in the historical version pool, providing the main agent with diverse opponent choices. This dynamic adjustment of opponent strategies allows the main agent to gradually adapt to complex adversarial combinations.

[0028] S102: The first strategy model is updated based on the interaction data between the first strategy model and the second strategy model to obtain the third strategy model, and the second strategy model is updated based on the interaction data between the first strategy model and the second strategy model to obtain the fourth strategy model.

[0029] In this embodiment, the main agent and the training agent interact based on a selected first policy model and a second policy model. During the interaction, the first policy model and the second policy model can take actions in a specific task environment according to their respective policies and obtain corresponding feedback (such as rewards, state changes, etc.). This interaction data can serve as the basis for model updates.

[0030] In this embodiment, deep reinforcement learning algorithms can be used to update the first policy model based on the collected interaction data to obtain a third policy model, and the second policy model can be updated to obtain a fourth policy model. For example, the model parameters can be adjusted based on the reward signals in the interaction data to optimize the policy, enabling the agent to achieve better performance in subsequent interactions. This process is an iterative optimization process, gradually improving the agent's policy capabilities through continuous interaction and learning from the opponent.

[0031] S103: Add the third strategy model to the master agent's historical version pool, and add the fourth strategy model to the training agent's historical version pool.

[0032] In this embodiment, the updated third policy model is added to the master agent's historical version pool, and the fourth policy model is added to the training agent's historical version pool. This enriches the content of the historical version pool, providing more diverse opponent choices for subsequent training. As training progresses, the models in the historical version pool will continuously increase and be updated, allowing the agent to face more different policy combinations in subsequent training, avoiding getting trapped in local optima. Simultaneously, new policy models can be selected again as the starting point or opponent for training, thereby promoting the agent's continuous evolution.

[0033] As can be seen from the above, this embodiment determines the initial policy model by utilizing the historical version pools of the main agent and the training agent, laying a rich and diverse foundation for the training process and greatly enhancing the complexity and diversity of training. Secondly, updating based on the data generated by the interaction between the two models effectively optimizes the policy based on actual feedback, enabling the agent to continuously improve its policy level through ongoing adversarial learning and achieving iterative growth. Finally, re-incorporating the updated model into the historical version pool not only expands the types of policies in the pool, creating more diverse adversarial scenarios for subsequent training and effectively avoiding the dilemma of local optima, but also promotes the continuous evolution of the agent through the recycling of the new model. This significantly enhances the agent's adaptability and competitiveness in complex tasks, promoting its continuous development and improvement in dynamic environments, and meeting the practical needs of variable policies and complex environments in asymmetric tasks.

[0034] In one embodiment of this disclosure, determining a first policy model for the main agent and a second policy model for the training agent includes:

[0035] In response to the absolute value of the difference between the strength scores of the main agent and the training agent being less than or equal to a first value, the strategy model corresponding to the main agent is taken as the first strategy model, and the strategy model of the training agent is taken as the second strategy model.

[0036] In this embodiment, by comparing the performance scores of the main agent and the training agent, when the absolute value of the difference between the two is less than or equal to a first value, it indicates that their abilities are relatively similar. In this case, the policy model currently corresponding to the main agent is selected as the first policy model, and the policy model of the training agent is selected as the second policy model. The purpose is to ensure that the ability gap between the two agents is within a certain range in the early stage or specific phase of training, making the training process more challenging and effective. If the difference in ability between the two is too large, it may lead to poor training results; if one is too dominant, the other will find it difficult to obtain effective learning feedback.

[0037] Choosing models of similar strength for interaction allows both sides to learn from each other and optimize their strategies in a relatively balanced competitive environment, promoting the mutual improvement of their strategies.

[0038] For example, in a simulated military confrontation training scenario, if the difference in strength scores between the main agent and the training agent is within a set range, their strategy models can better expose their respective strengths and weaknesses during the interaction process, thereby enabling targeted improvements based on the interaction data.

[0039] As can be seen from the above, this embodiment selects strategy models of similar strength for training based on the difference in strength scores between the main agent and the training agent, creating a relatively fair and moderately challenging training environment. This helps both agents fully utilize their respective capabilities in the competition, avoiding one being excessively suppressed or the other unable to effectively train due to a significant difference in strength. In such an environment, both can learn and improve more efficiently from each other's strategy responses, jointly promoting strategy optimization and enhancement, and strengthening their adaptability and coping capabilities in complex task scenarios.

[0040] In one embodiment of this disclosure, it further includes:

[0041] Based on the target probability, a policy model is randomly selected from the historical version pool of the training agent as the second policy model.

[0042] In this embodiment, a target probability can be set, and a policy model can be randomly selected from the historical version pool of the training agent as the second policy model based on the target probability. This avoids the limitations of traditional fixed or single-rule selection of the opponent's policy model.

[0043] For example, suppose a training scenario simulating cybersecurity attack and defense is implemented, in which the main agent is responsible for attacking, and the training agent is responsible for defending. The training agent's historical version pool stores various defense strategy models obtained from different training stages in the past.

[0044] Setting the target probability to 0.3, this embodiment determines whether to perform random selection before each training round based on this probability. For example, in a certain training round, random selection is determined by comparing the generated random number with the target probability. An early strategy model with a relatively conservative defense architecture is selected from the pool of historical versions of the training agent as the second strategy model. In the process of adversarial training against this special strategy model, the main agent discovers new attack vulnerabilities and countermeasures, rather than being limited to common defense strategy patterns. After multiple such random selection training sessions, the main agent can quickly and effectively respond to various complex and ever-changing network defense strategies, significantly improving its adaptability and robustness in network security attack and defense tasks.

[0045] As can be seen from the above, this embodiment, by introducing randomness, enables the agent to encounter more diverse training strategies during the training process. If the agent always selects opponent strategies according to a fixed pattern during training, it is prone to overfitting to specific types of strategies. However, a model that randomly selects training strategies allows the agent to be exposed to strategies of different styles and characteristics, thereby broadening its range of coping strategies and enhancing its adaptability and robustness in complex and changing environments.

[0046] In one embodiment of this disclosure, a third strategy model is obtained by updating the first strategy model based on the interaction data of the first strategy model and the second strategy model, and a fourth strategy model is obtained by updating the second strategy model based on the interaction data of the first strategy model and the second strategy model; including:

[0047] Obtain the interaction data between the first strategy model and the second strategy model;

[0048] The policy parameters of the first policy model are updated based on the deep reinforcement learning algorithm to obtain the third policy model;

[0049] The policy parameters of the second policy model are updated based on the deep reinforcement learning algorithm to obtain the fourth policy model.

[0050] In this embodiment, it is necessary to acquire a large amount of data generated during the interaction between the first and second policy models. This data covers key elements such as the agent's state information, actions taken, and corresponding rewards during task execution, forming an experience pool for subsequent model updates.

[0051] The policy parameters of the first policy model are updated using deep reinforcement learning algorithms (such as DQN and PPO). Deep reinforcement learning algorithms can judge the quality of a policy based on reward signals in the experience pool. For example, if the action taken by the first policy model in a certain interaction scenario yields a high reward, the deep reinforcement learning algorithm may tend to strengthen the policy parameters corresponding to that action; conversely, if the reward is low, it will adjust the parameters to encourage the model to make better decisions in subsequent similar scenarios, thus obtaining an optimized third policy model.

[0052] Similar to the update of the first strategy model, the second strategy model's policy parameters are adjusted using the same interaction data, also based on deep reinforcement learning algorithms. By analyzing various feedback information in the interaction data, the model's decision-making mechanism is continuously optimized, making it perform better in subsequent adversarial or cooperative scenarios, ultimately resulting in the fourth strategy model.

[0053] For example, suppose in a robot soccer training scenario, the first strategy model of the main agent (attacker) is initially set to prioritize passing the ball to the area near the opponent's goal and looking for shooting opportunities, while the second strategy model of the training agent (defender) focuses on marking key players of the attacking side and blocking dangerous areas.

[0054] In one interaction, the primary agent controls a robot player to dribble forward according to its strategy model. When encountering a training agent's defense, it chooses to pass the ball to a teammate. However, due to the training agent's tight defense, the pass is intercepted. The primary agent receives a lower reward but generates interaction data containing the current game state (such as player position, ball position, etc.), the action taken (passing), and the reward. The training agent successfully intercepts the pass, receives a higher reward, and also generates corresponding interaction data.

[0055] After storing these interaction data in an experience pool, the first policy model of the main agent is updated using a deep reinforcement learning algorithm. If the algorithm analyzes a failed pass, it will appropriately reduce the weight of the policy parameter for choosing that pass method in similar scenarios, while potentially increasing the weight of the policy parameter for attempting a breakthrough or changing the passing route. After multiple iterations, a third policy model is obtained.

[0056] For the training agent, based on the same interaction data, the deep reinforcement learning algorithm will enhance its strategy parameters for successful defense in that area, such as optimizing the selection of marking players and adjusting their positions, resulting in a fourth strategy model. With continuous training and model updates, the performance of both the main agent and the training agent in robot soccer matches will gradually improve. The main agent's offensive strategies will become more flexible and varied, while the training agent's defense will become more rigorous and efficient.

[0057] As can be seen from the above, this embodiment, by acquiring rich data generated from the interaction of the two policy models, can comprehensively capture the agent's behavior and feedback in different situations. By using deep reinforcement learning algorithms to update policy parameters, the agent is prompted to continuously optimize its own policy. In this process, both the master agent and the training agent can continuously improve their coping abilities, enhance the effectiveness and adaptability of the policy, avoid falling into fixed patterns, and thus improve the overall training effect and the agent's performance.

[0058] In one embodiment of this disclosure, it further includes:

[0059] The strength scores of the main agent and the training agent are updated based on the interaction data of the first and second strategy models.

[0060] In this embodiment, the performance scores of the main agent and the training agent are updated using a large amount of data generated during the interaction between the first and second strategy models. This interaction data includes various performance details of the agents in task execution, such as the actions taken in different states and their results, rewards obtained, and other information.

[0061] By comprehensively analyzing this data, we can determine the effectiveness of the agent's strategy and the improvement or decline in its actual capabilities. For example, if the main agent frequently achieves its goals and obtains high rewards in its interactions with the training agent, it indicates that its strategy is performing well in the current adversarial situation, and its performance score can be improved accordingly; conversely, if it frequently fails or performs poorly, its performance score will decrease.

[0062] The updated ability rating dynamically reflects the agent's capability status, thus influencing the subsequent training process. The opponent selection strategy can be adjusted based on the updated rating, ensuring the agent always faces opponents with appropriate challenges, maintaining the effectiveness and efficiency of training, promoting continuous strategy optimization and capability improvement, and avoiding poor training results due to prolonged exposure to opponents that are too strong or too weak.

[0063] In one embodiment of this disclosure, updating the performance ratings of the main agent and the training agent based on the interaction data of the first policy model and the second policy model includes:

[0064] The performance ratings of the main AI agent and the training AI agent are updated based on the first formula.

[0065] The first formula is:

[0066]

[0067] in, This is represented by the updated performance score, which intuitively reflects the change in the agent's ability assessment after a round of interactive training. This represents the agent's performance rating before the update, reflecting its previous capabilities. This represents the step size parameter for rating updates; its magnitude determines the extent to which the rating is adjusted based on the interaction results each time. A larger value results in a more significant adjustment to the score, while a smaller value results in a relatively gradual adjustment, thus playing a role in regulating the sensitivity of score changes. This represents the interaction result between the main agent and the training agent. In practical applications, if the main agent wins the interaction, It can take the value 1, and 0 if it fails. In case of a tie or other specific situations (such as task timeout), it can have a corresponding agreed value (such as 0.5). This represents the expected win rate of the main AI agent or the training AI agent.

[0068] In one embodiment of this disclosure, the expected win rate of the main agent or the training agent is calculated based on a second formula;

[0069] The second formula is:

[0070]

[0071] in, This represents the opponent's strength rating. This indicates a rating of one's own ability.

[0072] In this embodiment, the expected win rate of the main agent or the training agent utilizes the characteristics of an exponential function. This means that the greater the difference in skill ratings, the more significant the change in the expected win rate. This allows for a more reasonable reflection of the impact of the skill comparison on the outcome during the rating update process. For example, when two agents are of similar skill, their expected win rate is close to 0.5, while when the skill gap is large, the stronger agent's expected win rate approaches 1, and the weaker agent's approaches 0. In this way, after each interaction, the actual interaction result and the skill-based expected win rate are comprehensively considered, and the agent's skill rating is accurately updated using the first formula. This allows the skill rating to dynamically and accurately reflect the agent's ability development and relative strength changes during training, providing a scientific basis for subsequent training strategy adjustments (such as opponent selection and training difficulty adaptation).

[0073] For example, the scenario is set as follows:

[0074] Red (Main Agent): Responsible for attack, current strength rating is 1500.

[0075] Blue team (sparring agent): Responsible for defense; the historical version pool contains three historical versions:

[0076] Blue Team 1: Strength rating is 1450.

[0077] Blue Team 2: Strength rating is 1500.

[0078] Blue Team 3: Strength rating is 1550.

[0079] Opponent selection strategy settings:

[0080] 1. Prioritize opponents with similar strength ratings: Choose opponents whose strength ratings differ from the red side's by ±50.

[0081] 2. Random selection strategy: There is a 20% probability of randomly selecting an opponent from the blue team's historical version pool.

[0082] Training parameters:

[0083] 𝐾=16 (score update step size parameter).

[0084] Each round of battle collects 10 sets of data (status, actions, rewards, next status).

[0085] Strategy updates are sampled from the experience pool, which has a capacity of 100.

[0086] Training process:

[0087] 1. Opponent selection phase

[0088] The Red team's current strength rating is 1500. The Blue team's AI agent that meets the criteria of having a similar rating is Blue Team 2 (rated 1500).

[0089] However, based on a 20% probability of random selection, the opponent was ultimately randomly selected as Blue Team 3 (score 1550).

[0090] 2. Multi-round battle data collection phase

[0091] The red team and the blue team will play multiple rounds of battles, totaling 10 games, collecting battle data (status, actions, rewards, next state) and storing it in the experience pool.

[0092] The purpose of the experience pool is to store interaction data from multiple rounds of battles, providing training samples for subsequent strategy optimization.

[0093] 3. Strategy update phase (sampling from the experience pool)

[0094] The red team randomly samples 32 battle data points from the experience pool (batch size can be adjusted) and uses a deep reinforcement learning algorithm (such as the PPO algorithm or the DQN algorithm) to optimize the strategy parameters.

[0095] After the strategy optimization is completed, the updated Red team strategy version is stored in the Red team's historical version pool.

[0096] Meanwhile, Blue Team 3's strategy is also updated by sampling data from its experience pool, and the optimized strategy is stored in Blue Team's historical version pool.

[0097] 4. Rating Update Phase

[0098] The winning percentage is calculated based on the win-loss statistics of 10 games. For example, if the red side wins 6 games, then the actual match result is 0.6.

[0099] Based on the updated strength rating formula, calculate the expected win rate (r) of red and blue sides in match 3:

[0100]

[0101] The red team's strength rating has been updated to:

[0102]

[0103] Blue Team 3's strength rating has been updated to:

[0104]

[0105] 5. Historical version pool update phase

[0106] The red team's update strategy is stored in the red team's historical version pool for the blue team to choose from later.

[0107] The blue team's update strategy for version 3 is stored in the blue team's historical version pool for the red team to choose from later.

[0108] 6. Dynamic adjustment of opponent selection strategy

[0109] Based on the results of the matches, if the red side's winning rate continues to increase, the probability of randomly selecting historical versions can be increased (e.g., from 20% to 30%) to increase the diversity of opponents and avoid overfitting.

[0110] As can be seen from the above, this embodiment, by precisely quantifying interaction results and expected win rates, can dynamically and accurately reflect the development and relative strength changes of the agent's capabilities, providing a solid basis for adjusting training strategies. Regarding opponent selection, opponents can be reasonably matched based on strength scores, ensuring the challenge and effectiveness of training. Simultaneously, during training, the random selection probability can be adjusted in a timely manner according to score changes, increasing opponent diversity, avoiding overfitting, and promoting continuous strategy optimization by the agent.

[0111] Next, as Figure 2 As shown, this embodiment can enter an alternating training mechanism, selecting the main agent and the training partner agent alternately for training, allowing their strategies to be continuously optimized in a dynamic adversarial environment. Every certain number of rounds, this embodiment stores the latest strategies of both the main agent and the training partner agent in a historical version pool to ensure the diversity and quality of the strategy pool. After multiple rounds of alternating training, the main agent's strategy is gradually optimized, enabling it to achieve stronger adversarial adaptability and strategic flexibility in complex asymmetric task environments.

[0112] For example, such as Figure 3 As shown, agent alliance training for asymmetric tasks:

[0113] Input: Initial main agent Red, initial sparring agent Blue, opponent selection strategy;

[0114] Output: The trained main agent Red;

[0115] Import the main agent Red;

[0116] Import the AI ​​assistant Blue;

[0117] Initialize the main agent's policy pool Red_Pool and add the current version of Red to Red_Pool;

[0118] Initialize the training agent policy pool Blue_Pool and add the current version of Blue to Blue_Pool;

[0119] Define the number of training rounds N

[0120] For each iteration, i ranges from 1 to N;

[0121] Main agent training phase:

[0122] Based on the opponent selection strategy, select a sparring agent (blue_opponent) from the Blue_Pool;

[0123] Using blue_opponent as the environment, it interacts with the main agent Red to collect battle data;

[0124] Update the policy parameters of the main agent Red based on the collected data;

[0125] Add the updated Red to Red_Pool;

[0126] Training phase of the sparring agent:

[0127] Based on the opponent selection strategy, select a primary agent (red_opponent) from the Red_Pool;

[0128] Using red_opponent as the environment, it interacts with the training agent Blue to collect battle data;

[0129] The strategy parameters of the training agent Blue are updated based on the collected data;

[0130] Add the updated Blue to Blue_Pool;

[0131] Rating update:

[0132] Update the strength ratings of the main AI agent Red and the sparring AI agent Blue based on the battle results;

[0133] Regularly save historical intelligent agents:

[0134] If the current round i is a saved round, add the current versions of Red and Blue to their respective historical strategy pools;

[0135] Dynamically adjust opponent selection strategy:

[0136] Adjust the parameters of the opponent selection strategy (Selection_Strategy) based on the current battle results;

[0137] endfor

[0138] Return to the trained main agent Red.

[0139] In this embodiment, alternating training between the main agent and the training agent is implemented in an asymmetric task environment. Mechanisms such as opponent selection strategy, strength rating updates, and historical strategy pool management enable the agent to continuously optimize its strategies in diverse opponent environments. This embodiment achieves co-evolution of the main agent and the training agent through real-time collection of combat data and strategy adjustments, thereby enhancing the main agent's adaptability and adversarial capabilities. Updating the rating after each training round, dynamically adjusting the opponent selection scheme, and periodically saving strategy versions ensure the diversity and challenge of strategies, thus improving the agent's performance in complex task scenarios.

[0140] In this embodiment, an innovative alternating training method and opponent selection strategy address the problem that traditional symmetric task training methods are difficult to apply to asymmetric tasks. Specifically:

[0141] First, increase the diversity of strategies to enhance the adaptability of the main agent.

[0142] Because the primary agent and the training agent in asymmetric tasks have different objectives, traditional self-play methods struggle to provide sufficient policy diversity, easily leading to convergence on a single policy and insufficient adaptability. This embodiment introduces a historical policy pool and a dynamic opponent selection scheme, enabling the primary agent to face diverse opponents during training, thereby improving its adaptability to different policies. This design avoids the problem of policies getting trapped in local optima, making the agent more robust.

[0143] Secondly, the opponent selection and strategy update mechanism based on strength rating optimizes the agent's combat capabilities.

[0144] This embodiment updates the strength rating after each battle, ensuring that each agent's rating accurately reflects its combat capability and providing quantifiable training feedback. The opponent selection scheme based on strength rating dynamically chooses opponents according to the agent's current level, ensuring the main agent encounters suitable challenging opponents at each stage, gradually improving its combat capability. Compared to traditional training methods with fixed opponents, this embodiment makes the agent's strategy optimization process more efficient.

[0145] Secondly, dynamically adjust the opponent's selection strategy to prevent overfitting.

[0146] In traditional training, agents may overfit to fixed opponents, leading to a decrease in competitiveness. This embodiment incorporates random selection and dynamic adjustment mechanisms into the opponent selection scheme, enabling flexible adjustment of selection probabilities based on current match results. Thus, when the agent's win rate against a particular opponent is too high, the probability of random selection can be increased, ensuring the agent encounters opponents with diverse strategies. This prevents overfitting and further enhances strategy diversity and challenge.

[0147] Finally, the management of the historical strategy pool (i.e., the historical version pool) ensures continuous optimization of training.

[0148] In each training round, this embodiment periodically stores the latest version of the agent's policy into a historical policy pool, ensuring the quality and diversity of the opponent pool. Compared to the traditional static policy pool, the dynamic policy pool management method of this embodiment can provide continuously evolving opponent policies, offering a richer environment for the training process of the main agent. This not only improves training effectiveness but also ensures that the agent continues to evolve in complex asymmetric task environments.

[0149] Corresponding to the agent training method in the above embodiments, Figure 4 This is a structural block diagram of an intelligent agent training device provided according to an embodiment of the present disclosure. For ease of explanation, only the parts relevant to the embodiment of the present disclosure are shown. References Figure 4 The intelligent agent training device 20 includes: a policy model determination module 21, a policy model update module 22, and a historical version pool update module 23.

[0150] Among them, the strategy model determination module 21 is used to determine the first strategy model of the main agent and the second strategy model of the training agent. The first strategy model is the model in the historical version pool of the main agent, and the second strategy model is the model in the historical version pool of the training agent.

[0151] The strategy model update module 22 is used to update the first strategy model based on the interaction data of the first strategy model and the second strategy model to obtain the third strategy model, and to update the second strategy model based on the interaction data of the first strategy model and the second strategy model to obtain the fourth strategy model.

[0152] The historical version pool update module 23 is used to add the third strategy model to the main agent's historical version pool and the fourth strategy model to the training agent's historical version pool.

[0153] In one embodiment of this disclosure, the strategy model determination module 21 is further configured to:

[0154] In response to the absolute value of the difference between the strength scores of the main agent and the training agent being less than or equal to a first value, the strategy model corresponding to the main agent is taken as the first strategy model, and the strategy model of the training agent is taken as the second strategy model.

[0155] In one embodiment of this disclosure, the strategy model determination module 21 is further configured to:

[0156] Based on the target probability, a policy model is randomly selected from the historical version pool of the training agent as the second policy model.

[0157] In one embodiment of this disclosure, the policy model update module 22 is specifically used for:

[0158] Obtain the interaction data between the first strategy model and the second strategy model;

[0159] The policy parameters of the first policy model are updated based on the deep reinforcement learning algorithm to obtain the third policy model;

[0160] The policy parameters of the second policy model are updated based on the deep reinforcement learning algorithm to obtain the fourth policy model.

[0161] In one embodiment of this disclosure, the intelligent agent training device 20 further includes: a strength scoring module, which is specifically used for:

[0162] The strength scores of the main agent and the training agent are updated based on the interaction data of the first and second strategy models.

[0163] In one embodiment of this disclosure, the strength scoring module is further used for:

[0164] The performance ratings of the main AI agent and the training AI agent are updated based on the first formula.

[0165] The first formula is:

[0166]

[0167] in, This represents the updated strength rating. This represents the strength rating before the update. This is represented as the step size parameter for score updates. This represents the interaction result between the main agent and the training agent. This represents the expected win rate of the main AI agent or the training AI agent.

[0168] In one embodiment of this disclosure, the strength scoring module is further used for:

[0169] The expected win rate of the main AI agent or the training AI agent is calculated based on the second formula.

[0170] The second formula is:

[0171]

[0172] in, This represents the opponent's strength rating. This indicates a rating of one's own ability.

[0173] See Figure 5 , Figure 5 This is a schematic block diagram of an electronic device provided according to an embodiment of the present disclosure. Figure 5 The electronic device 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to perform the functions of the modules in the aforementioned device embodiments, for example... Figure 4 The functions of modules 21 to 23 are shown.

[0174] It should be understood that, in the embodiments of this disclosure, the processor 301 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0175] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0176] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store device type information.

[0177] In specific implementations, the processor 301, input device 302, and output device 303 described in the embodiments of this disclosure can execute the implementation methods described in the first and second embodiments of the intelligent agent training method provided in the embodiments of this disclosure, or they can execute the implementation methods of the electronic devices described in the embodiments of this disclosure, which will not be repeated here.

[0178] In another embodiment of this disclosure, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. Alternatively, the computer program can instruct related hardware to implement these processes. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0179] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the electronic device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0180] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0181] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0182] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces or units, or it may be an electrical, mechanical, or other form of connection.

[0183] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this disclosure, depending on actual needs.

[0184] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0185] The above are merely specific embodiments of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this disclosure, and these modifications or substitutions should all be covered within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. An agent training method, characterized by, Comprising: In response to the absolute value of the strength score difference between the main agent and the sparring agent being less than or equal to a first value, the strategy model corresponding to the main agent is taken as a first strategy model, and the strategy model of the sparring agent is taken as a second strategy model; the first strategy model is a model in the main agent historical version pool, and the second strategy model is a model in the sparring agent historical version pool; Obtain interaction data of the first strategy model and the second strategy model; Based on the deep reinforcement learning algorithm, the policy parameters of the first strategy model are updated using the interaction data to obtain a third strategy model; Based on the deep reinforcement learning algorithm, the policy parameters of the second strategy model are updated using the interaction data to obtain a fourth strategy model; The third strategy model is added to the main agent historical version pool, and the fourth strategy model is added to the sparring agent historical version pool; Also comprising: Set a target probability, and based on the target probability, decide whether to randomly select a strategy model from the sparring agent historical version pool as a second strategy model; Also comprising: Update the strength scores of the main agent and the sparring agent based on the interaction data of the first strategy model and the second strategy model; Wherein, the strength scores of the main agent and the sparring agent are updated based on a first formula; The first formula is: wherein, represents the updated strength score, represents the pre-update strength score, represents the step parameter for score updating, represents the interaction result of the main agent and the sparring agent, represents the expected win rate of the main agent or the sparring agent; Calculate the expected win rate of the main agent or the sparring agent based on a second formula; The second formula is: wherein, represents the strength score of the opponent, represents the strength score of the self.

2. An agent training apparatus characterized by comprising: Comprising: The strategy model determination module is configured to, in response to the absolute value of the strength score difference between the main agent and the sparring agent being less than or equal to a first value, take the strategy model corresponding to the main agent as a first strategy model, and take the strategy model of the sparring agent as a second strategy model; the first strategy model is a model in the main agent historical version pool, and the second strategy model is a model in the sparring agent historical version pool; The strategy model update module is configured to obtain interaction data of the first strategy model and the second strategy model; based on the deep reinforcement learning algorithm, the policy parameters of the first strategy model are updated using the interaction data to obtain a third strategy model; based on the deep reinforcement learning algorithm, the policy parameters of the second strategy model are updated using the interaction data to obtain a fourth strategy model; The historical version pool update module is configured to add the third strategy model to the main agent historical version pool, and add the fourth strategy model to the sparring agent historical version pool; The strategy model determination module is further configured to set a target probability, and based on the target probability, decide whether to randomly select a strategy model from the sparring agent historical version pool as a second strategy model; The strength score module is specifically configured to update the strength scores of the main agent and the sparring agent based on the interaction data of the first strategy model and the second strategy model; Wherein, the strength scores of the main agent and the sparring agent are updated based on a first formula; The first formula is: wherein, represents the updated strength score, represents the pre-update strength score, represents the step parameter for score updating, represents the interaction result of the main agent and the sparring agent, represents the expected win rate of the main agent or the sparring agent; Calculate the expected win rate of the main agent or the sparring agent based on a second formula; The second formula is: wherein, represents the strength score of the opponent, represents the strength score of the self.

3. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, The processor, when executing the computer program, implements the steps of the method of claim 1.

4. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 3. The computer program, when executed by the processor, implements the steps of the method of claim 1.

Images