Intelligent agent training method and device, electronic equipment and storage medium

Abstract

The invention provides an agent training method and device, electronic equipment and a storage medium, and belongs to the technical field of agent training.The method comprises the steps that a first strategy model of a main agent and a second strategy model of a partner training agent are determined, the first strategy model is a model in a main agent historical version pool, and the second strategy model is a model in a partner training agent historical version pool; the second strategy model is a model in a training partner agent historical version pool; updating the first strategy model based on the interaction data of the first strategy model and the second strategy model to obtain a third strategy model, and updating the second strategy model based on the interaction data of the first strategy model and the second strategy model to obtain a fourth strategy model; and adding the third strategy model into the main agent historical version pool, and adding the fourth strategy model into the partner training agent historical version pool. According to the agent training method and device, the electronic equipment and the storage medium provided by the invention, the actual requirements of variable strategies and complex environments in asymmetric tasks can be met.

Description

Technical Field

[0001] The present disclosure belongs to the field of intelligent agent training technology, and more specifically, relates to an intelligent agent training method and device, electronic equipment, and storage medium. Background Art

[0002] In asymmetric tasks, such as offensive and defensive tasks, the agent needs to quickly adapt to the opponent's changing strategies. The core of agent training is that the agent can adapt to dynamic and complex environments and handle significant differences in the requirements of offensive and defensive tasks. However, traditional methods lack strategy diversity and dynamic update mechanisms, making it difficult to cover complex adversarial strategy combinations, causing the agent to easily fall into the "local optimal" strategy mode, making it difficult to meet the actual needs of asymmetric tasks with changing strategies and complex environments. Summary of the invention

[0003] The purpose of the present disclosure is to provide an intelligent agent training method and device, electronic device, and storage medium to meet the actual needs of changing strategies and complex environments in asymmetric tasks.

[0004] A first aspect of the embodiments of the present disclosure provides an agent training method, comprising: Determine a first strategy model of the master agent and a second strategy model of the sparring agent, wherein the first strategy model is a model in the master agent historical version pool, and the second strategy model is a model in the sparring agent historical version pool; Based on the interaction data between the first policy model and the second policy model, the first policy model is updated to obtain a third policy model; based on the interaction data between the first policy model and the second policy model, the second policy model is updated to obtain a fourth policy model; The third strategy model is added to the historical version pool of the main agent, and the fourth strategy model is added to the historical version pool of the sparring agent.

[0005] A second aspect of the embodiments of the present disclosure provides an intelligent agent training device, comprising: A strategy model determination module, used to determine a first strategy model of a master agent and a second strategy model of a sparring agent, wherein the first strategy model is a model in a historical version pool of the master agent, and the second strategy model is a model in a historical version pool of the sparring agent; a policy model updating module, configured to update the first policy model based on the interaction data between the first policy model and the second policy model to obtain a third policy model, and to update the second policy model based on the interaction data between the first policy model and the second policy model to obtain a fourth policy model; The historical version pool update module is used to add the third strategy model to the historical version pool of the main agent and add the fourth strategy model to the historical version pool of the sparring agent.

[0006] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the above-mentioned intelligent agent training method when executing the computer program.

[0007] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned intelligent agent training method are implemented.

[0008] The beneficial effects of the agent training method and device, electronic device, and storage medium provided by the embodiments of the present disclosure are: the embodiments of the present disclosure use the historical version pool of the main agent and the sparring agent to determine the starting strategy model, which lays a rich and diverse foundation for the training process and greatly improves the complexity and diversity of the training. Secondly, by updating the data generated by the interaction of the two models, the strategy can be effectively optimized according to actual feedback, so that the agent can continuously improve the strategy level in continuous confrontation and learning, and achieve iterative growth. Finally, the updated model is reincorporated into the historical version pool, which not only expands the types of strategies in the pool, creates more diverse confrontation scenarios for subsequent training, and effectively avoids the dilemma of local optimality, but also the recycling of new models promotes the continuous evolution of the agent, which effectively enhances the adaptability and competitiveness of the agent in complex tasks, promotes its continuous development and improvement in a dynamic environment, and meets the actual needs of changing strategies and complex environments in asymmetric tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0010] Figure 1 A flowchart of an intelligent agent training method provided by an embodiment of the present disclosure; Figure 2 A schematic diagram of the training principle of an intelligent agent alliance provided in an embodiment of the present disclosure; Figure 3 A flow chart of intelligent agent alliance training provided in one embodiment of the present disclosure; Figure 4A structural block diagram of an intelligent agent training device provided in one embodiment of the present disclosure; Figure 5 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0011] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0012] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0013] Please refer to Figure 1 , Figure 1 A flow chart of an agent training method provided in one embodiment of the present disclosure, the method comprising: S101: Determine a first strategy model of the main agent and a second strategy model of the sparring agent, wherein the first strategy model is a model in the historical version pool of the main agent, and the second strategy model is a model in the historical version pool of the sparring agent.

[0014] In this embodiment, the first strategy model is selected from the historical version pool of the main agent, and the second strategy model is selected from the historical version pool of the sparring agent. The historical version pool of the main agent and the historical version pool of the sparring agent store the strategy models of the agents at different training stages, providing a diverse starting point for training. The initial models of the main agent and the sparring agent can be obtained through pre-training. By selecting from the historical version pool, the previous training results can be utilized, and different strategy foundations can be provided for subsequent training, increasing the diversity and complexity of training.

[0015] In this embodiment, the main agent includes a pre-trained model of the main agent and a historical version pool. The main agent loads a pre-trained model at the time of initialization to ensure that the initial strategy has certain basic capabilities. The pre-trained model was previously trained by other methods in related scenarios, and its purpose is to provide a stable initial strategy for subsequent alternating training. The historical version pool is constructed by saving the strategy model version of the main agent after each round of training, and is designed as a queue or storage list structure to record the strategy evolution process of the main agent in chronological order. The diversity of the historical version pool can provide a wealth of opponent selection options for the sparring agent module, which helps to discover the strategy loopholes of the main agent during the sparring agent training and further optimize the strategy.

[0016] Similar to the main agent module, the sparring agent also contains the pre-trained model and historical version pool of the sparring agent. The sparring agent also loads a pre-trained model when the system is initialized. The model provides basic adversarial strategies to ensure that the sparring agent can generate an effective adversarial environment in the early training with the main agent. As the training progresses, the strategy model of the sparring agent is continuously saved and updated in the historical version pool, providing the main agent with a variety of opponent options. This way of dynamically adjusting the opponent's strategy enables the main agent to gradually adapt to complex adversarial combinations.

[0017] S102: Based on the interaction data between the first policy model and the second policy model, the first policy model is updated to obtain a third policy model; based on the interaction data between the first policy model and the second policy model, the second policy model is updated to obtain a fourth policy model.

[0018] In this embodiment, the main agent and the sparring agent interact based on the selected first strategy model and second strategy model. During the interaction, the first strategy model and the second strategy model can take actions in a specific task environment according to their respective strategies and obtain corresponding feedback (such as rewards, state changes, etc.). These interaction data can be used as the basis for model updates.

[0019] In this embodiment, the first strategy model can be updated based on the collected interaction data to obtain the third strategy model, and the second strategy model can be updated to obtain the fourth strategy model through deep reinforcement learning algorithm technology. For example, the parameters of the model can be adjusted according to the reward signal in the interaction data to optimize the strategy so that the agent can achieve better performance in subsequent interactions. This process is an iterative optimization process, which gradually improves the strategy ability of the agent by continuously interacting and learning with the opponent.

[0020] S103: Add the third strategy model to the historical version pool of the main agent, and add the fourth strategy model to the historical version pool of the sparring agent.

[0021] In this embodiment, the updated third strategy model is added to the historical version pool of the main agent, and the fourth strategy model is added to the historical version pool of the sparring agent. This can enrich the content of the historical version pool and provide more diverse opponent options for subsequent training. As the training proceeds, the models in the historical version pool will continue to increase and update, so that the agent can face more different strategy combinations in subsequent training and avoid falling into local optimal solutions. At the same time, the new strategy model can be selected again as the starting point or opponent of training, thereby promoting the continuous evolution of the agent.

[0022] From the above, it can be concluded that this embodiment determines the starting strategy model by utilizing the historical version pool of the main agent and the sparring agent, which lays a rich and diverse foundation for the training process and greatly improves the complexity and diversity of training. Secondly, by updating based on the data generated by the interaction of the two models, the strategy can be effectively optimized according to actual feedback, so that the agent can continuously improve the strategy level in continuous confrontation and learning, and achieve iterative growth. Finally, the updated model is reincorporated into the historical version pool, which not only expands the types of strategies in the pool, creates more diverse confrontation scenarios for subsequent training, and effectively avoids the dilemma of local optimality, but also the recycling of new models promotes the continuous evolution of the agent, which effectively enhances the adaptability and competitiveness of the agent in complex tasks, promotes its continuous development and improvement in a dynamic environment, and meets the actual needs of changing strategies and complex environments in asymmetric tasks.

[0023] In one embodiment of the present disclosure, determining a first strategy model of a master agent and a second strategy model of a sparring agent includes: In response to the absolute value of the difference in strength scores between the main agent and the sparring agent being less than or equal to a first value, the strategy model corresponding to the main agent is used as the first strategy model, and the strategy model of the sparring agent is used as the second strategy model.

[0024] In this embodiment, by comparing the strength scores of the main agent and the sparring agent, when the absolute value of the difference between the two is less than or equal to the first value, it indicates that the strengths of the two are relatively close. In this case, the strategy model currently corresponding to the main agent is selected as the first strategy model, and the strategy model of the sparring agent is selected as the second strategy model. The purpose is to ensure that in the early stage or a specific stage of training, the ability gap between the two agents is within a certain range, making the training process more challenging and effective. If the strength difference between the two parties is too large, it may lead to poor training results. One party is too strong, which makes it difficult for the other party to obtain effective learning feedback.

[0025] Selecting models of similar strength to interact can allow both parties to learn from each other and optimize strategies in a relatively balanced confrontation environment, promoting the mutual improvement of both parties' strategies.

[0026] For example, in an intelligent agent training scenario simulating military confrontation, if the difference in strength scores between the main intelligent agent and the sparring intelligent agent is within a set range, their strategy models can better expose their respective strengths and weaknesses during the interaction process, thereby making targeted improvements based on the interaction data.

[0027] From the above, it can be concluded that this embodiment creates a relatively fair and moderately challenging training environment by selecting strategy models with similar strengths for training based on the difference in strength scores between the main agent and the sparring agent. This helps the two agents to give full play to their respective abilities in the confrontation, avoiding excessive suppression of one party or inability of the other party to be effectively trained due to the disparity in strength. In such an environment, both parties can learn and improve from each other's strategic responses more efficiently, jointly promote the optimization and improvement of strategies, and enhance their adaptability and response capabilities in complex task scenarios.

[0028] In one embodiment of the present disclosure, it also includes: Based on the target probability, a policy model is randomly selected from the historical version pool of the sparring agent as the second policy model.

[0029] In this embodiment, a target probability can be set, and a strategy model can be randomly selected from the historical version pool of the sparring agent based on the target probability as the second strategy model. This can avoid the limitations of traditional fixed or single rule selection of opponent strategy models.

[0030] For example, suppose that in an intelligent agent training scenario simulating network security attack and defense, the main intelligent agent is responsible for attacking and the sparring intelligent agent is responsible for defending. The historical version pool of the sparring intelligent agent stores a variety of defense strategy models obtained from training at different stages in the past.

[0031] The target probability is set to 0.3. Before each round of training begins, this embodiment can decide whether to make a random selection based on this probability. For example, in a certain round of training, a random selection is decided by comparing the random number generation with the target probability. A strategy model that adopted a more conservative defense architecture in the early days was selected from the historical version pool of the sparring agent as the second strategy model. In the process of confronting this special strategy model, the main agent discovered new attack breakthrough points and response methods, rather than being limited to common defense strategy patterns. After many such random selection trainings, the main agent can respond quickly and effectively when faced with various complex and changeable network defense strategies, significantly improving its adaptability and robustness in network security attack and defense tasks.

[0032] From the above, it can be concluded that this embodiment, by introducing randomness, can enable the main agent to face more diverse sparring strategies during the training process. During the agent training process, if the opponent strategy is always selected according to a fixed pattern, the main agent is prone to over-adaptation to a specific type of strategy, that is, overfitting. The random selection of the sparring strategy model can allow the main agent to be exposed to strategies of different styles and characteristics, thereby broadening the range of its response strategies and enhancing its adaptability and robustness in complex and changing environments.

[0033] In one embodiment of the present disclosure, the first policy model is updated based on the interaction data between the first policy model and the second policy model to obtain a third policy model, and the second policy model is updated based on the interaction data between the first policy model and the second policy model to obtain a fourth policy model; including: Acquire interaction data between the first strategy model and the second strategy model; Update the policy parameters of the first policy model based on the deep reinforcement learning algorithm to obtain a third policy model; Based on the deep reinforcement learning algorithm, the strategy parameters of the second strategy model are updated to obtain a fourth strategy model.

[0034] In this embodiment, a large amount of data generated during the interaction between the first strategy model and the second strategy model needs to be obtained. These data cover key elements such as the state information of the agent when performing tasks, the actions taken, and the corresponding rewards obtained, and constitute the experience pool for subsequent model updates.

[0035] The policy parameters of the first policy model are updated through deep reinforcement learning algorithms (such as DQN algorithm and PPO algorithm). The deep reinforcement learning algorithm can judge the pros and cons of the strategy based on the reward signal in the experience pool. For example, if the action taken by the first policy model in a certain interaction scenario obtains a higher reward, the deep reinforcement learning algorithm may tend to strengthen the policy parameters corresponding to the action; conversely, if the reward is low, the parameters will be adjusted to prompt the model to make better decisions in subsequent similar scenarios, thereby obtaining an optimized third policy model.

[0036] Similar to the update of the first strategy model, the strategy parameters of the second strategy model are adjusted using the same interaction data based on the deep reinforcement learning algorithm. By analyzing various feedback information in the interaction data, the decision-making mechanism of the model is continuously optimized to make it perform better in subsequent confrontation or cooperation, and finally the fourth strategy model is obtained.

[0037] For example, assume that in an agent training scenario simulating a robot soccer game, the first strategy model of the main agent (offensive side) is initially set to prioritize passing the ball to the area near the opponent's goal and look for shooting opportunities, while the second strategy model of the sparring agent (defensive side) focuses on marking the key players of the offensive side and blocking dangerous areas.

[0038] In one interaction, the main agent controlled the robot player to dribble the ball according to its strategy model. When it encountered the defense of the sparring agent, it chose to pass the ball to its teammates. However, due to the tight defense of the sparring agent, the pass was intercepted and the main agent received a lower reward. At the same time, interaction data including the current game status (such as player position, ball position, etc.), the action taken (pass) and the reward were generated. The sparring agent successfully intercepted the pass and received a higher reward, which also generated corresponding interaction data.

[0039] After storing these interaction data in the experience pool, the deep reinforcement learning algorithm is used to update the first strategy model of the main agent. The algorithm analyzes that the pass failed this time, and appropriately reduces the weight of the strategy parameter for choosing this pass method in similar scenarios. At the same time, it may increase the weight of the strategy parameter for trying to break through or change the pass route. After multiple iterative updates, the third strategy model is obtained.

[0040] For the sparring agent, based on the same interaction data, the deep reinforcement learning algorithm will strengthen its strategic parameters for successful defense in the area, such as optimizing the selection of the defensive player and the position adjustment, and obtain the fourth strategy model. With continuous training and model updates, the performance of the main agent and the sparring agent in the robot football game scene will gradually improve. The offensive strategy of the main agent will be more flexible and varied, and the defense of the sparring agent will be more rigorous and efficient.

[0041] From the above, it can be concluded that this embodiment can fully capture the behavior and feedback of the agent in different situations by acquiring rich data generated by the interaction of the two-party strategy models. The deep reinforcement learning algorithm is used to update the strategy parameters, prompting the agent to continuously optimize its own strategy. In this process, the main agent and the sparring agent can continuously improve their coping ability, enhance the effectiveness and adaptability of the strategy, avoid falling into a fixed mode, and thus improve the overall training effect and the performance of the agent.

[0042] In one embodiment of the present disclosure, it also includes: The strength scores of the main agent and the sparring agent are updated based on the interaction data of the first strategy model and the second strategy model.

[0043] In this embodiment, the strength scores of the main agent and the sparring agent are updated through a large amount of data generated during the interaction between the first strategy model and the second strategy model. These interaction data contain various performance details of the agents in task execution, such as the actions taken in different states and their results, rewards obtained, and other information.

[0044] Through comprehensive analysis of these data, we can judge the effectiveness of the agent's strategy and the improvement or decline of its actual ability. For example, if the main agent frequently achieves goals and obtains high rewards in the interaction with the sparring agent, it means that its strategy performs well in the current confrontation, and its strength score can be improved accordingly; conversely, if it often fails or performs poorly, its strength score will be reduced.

[0045] The update of the strength score can dynamically reflect the ability status of the agent, which in turn affects the subsequent training process. The opponent selection strategy can be adjusted according to the updated score to ensure that the agent always faces moderately challenging opponents, maintain the effectiveness and efficiency of training, promote the agent to continuously optimize the strategy and improve its ability, and avoid poor training results due to long-term exposure to too strong or too weak opponents.

[0046] In one embodiment of the present disclosure, updating the strength scores of the main agent and the sparring agent based on the interaction data of the first strategy model and the second strategy model includes: Update the strength scores of the main agent and the sparring agent based on the first formula; The first formula is:

[0047] in, It is represented as the updated strength score, which intuitively reflects the change in the ability evaluation of the agent after a round of interactive training; It is represented as the strength score before the update, reflecting the previous ability level of the agent; It is represented as the step size parameter of the score update, and its size determines the magnitude of the score adjustment each time according to the interaction result. If the value is large, the score adjustment will be more significant, otherwise it will be relatively gentle, which plays a role in regulating the sensitivity of score changes; It represents the interaction result between the main agent and the sparring agent. In practical applications, if the main agent wins the interaction, It can take the value of 1, 0 for failure, and a corresponding agreed value (such as 0.5) in case of a tie or other special circumstances (such as task timeout). Represents the expected winning rate of the master agent or the sparring agent.

[0048] In one embodiment of the present disclosure, the expected winning rate of the master agent or the sparring agent is calculated based on the second formula; The second formula is:

[0049] in, It is the strength score of the opponent. Indicates your own strength rating.

[0050] In this embodiment, the expected winning rate of the main agent or the sparring agent utilizes the characteristics of the exponential function, so that the greater the difference in strength scores, the more obvious the change in the expected winning rate, thereby more reasonably reflecting the impact of the strength comparison of both parties on the results during the score update process. For example, when the strength of the two agents is similar, their expected winning rate is close to 0.5, and when the strength gap is large, the expected winning rate of the stronger one will approach 1, and the weaker one will approach 0. In this way, after each interaction, the actual interaction results and the expected winning rate based on strength are comprehensively considered, and the strength score of the agent is accurately updated using the first formula, so that the strength score can dynamically and accurately reflect the ability development and relative strength changes of the agent during the training process, and provide a scientific basis for subsequent training strategy adjustments (such as opponent selection, training difficulty adaptation, etc.).

[0051] Example, scene setting: Red Team (Main Agent): Responsible for attacking, current strength score is 1500.

[0052] Blue team (sparring agent): responsible for defense. The historical version pool contains three historical versions: Blue Team 1: Strength rating is 1450.

[0053] Blue Team 2: Strength rating is 1500.

[0054] Blue Team 3: Strength rating is 1550.

[0055] Opponent selection scheme settings: 1. Priority is given to opponents with similar strength scores: choose opponents whose strength score is within ±50 of the red team.

[0056] 2. Random selection strategy: There is a 20% probability of randomly selecting an opponent from the blue side's historical version pool.

[0057] Training parameters: 𝐾=16 (score update step size parameter).

[0058] 10 sets of data (state, action, reward, next state) are collected in each round of battle.

[0059] Strategy updates are sampled from the experience pool, which has a capacity of 100.

[0060] Training process: 1. Opponent selection phase The current strength score of the Red team is 1500, and the Blue team agent that meets the score proximity condition is Blue Team 2 (score 1500).

[0061] However, according to the 20% probability of random selection, Blue Team 3 (score 1550) was finally randomly selected as the opponent.

[0062] 2. Multi-round battle data collection stage The Red Team and the Blue Team 3 play multiple rounds of battles, a total of 10 rounds, collecting battle data (status, actions, rewards, next state) and storing them in the experience pool.

[0063] The role of the experience pool is to store interaction data from multiple rounds of battles and provide training samples for subsequent strategy optimization.

[0064] 3. Strategy update phase (sampling from the experience pool) The red team randomly samples 32 battle data from the experience pool (the batch size is adjustable) and uses a deep reinforcement learning algorithm (such as the PPO algorithm or the DQN algorithm) to optimize the strategy parameters.

[0065] After the strategy optimization is completed, the updated Red Team strategy version is stored in the Red Team historical version pool.

[0066] At the same time, Blue Team 3's strategy also samples data from its experience pool for update, and the optimized strategy is stored in the Blue Team's historical version pool.

[0067] 4. Rating update stage The winning rate is calculated based on the winning and losing statistics of 10 games. For example, if the red team wins 6 games, the actual game result 𝑆=0.6.

[0068] According to the strength score update formula, calculate the expected winning rate 𝐸 of the red team and the blue team 3:

[0069] The Red team's power rating is updated to:

[0070] The strength rating of Blue Team 3 is updated to:

[0071] 5. Historical version pool update phase The red team's update strategy is stored in the red team's historical version pool for subsequent selection by the blue team.

[0072] The update strategy of Blue Team 3 is stored in the Blue Team historical version pool for subsequent selection by the Red Team.

[0073] 6. Dynamic adjustment of opponent selection strategy According to the results of the game, if the red team's winning rate continues to increase, the probability of randomly selecting historical versions can be increased (for example, from 20% to 30%) to increase the diversity of opponents and avoid overfitting.

[0074] It can be concluded from the above that this embodiment can dynamically and accurately reflect the development of intelligent agent capabilities and changes in relative strength by accurately quantifying the interaction results and expected winning rates, providing a solid basis for adjusting training strategies. In terms of opponent selection, opponents can be reasonably matched based on strength scores to ensure the challenge and effectiveness of training. At the same time, during the training process, the probability of random selection can be adjusted in a timely manner according to the changes in scores, increasing the diversity of opponents, avoiding overfitting, and promoting the agent to continuously optimize its strategy.

[0075] Next, if Figure 2 As shown, this embodiment can enter the alternating training mechanism, by selecting the main agent and the sparring agent for training in turn, so that the strategies of both parties are continuously optimized in a changing confrontation environment. At regular intervals, this embodiment stores the latest strategies of the main agent and the sparring agent into the historical version pool to ensure the diversity and quality of the strategy pool. After multiple rounds of alternating training, the strategy of the main agent is gradually optimized, so that it can obtain stronger confrontation adaptability and strategy flexibility in a complex asymmetric task environment.

[0076] For example, Figure 3 As shown, agent alliance training for asymmetric tasks: Input: initial master agent Red, initial sparring agent Blue, opponent selection strategy Selection_Strategy; Output: The trained main agent Red; Import the main agent Red; Import the sparring agent Blue; Initialize the main agent strategy pool Red_Pool and add the current version of Red to Red_Pool; Initialize the sparring agent strategy pool Blue_Pool and add the current version of Blue to Blue_Pool; Define the number of training rounds N for each iteration i from 1 to N do; Main agent training phase: Select a sparring agent blue_opponent from Blue_Pool according to the opponent selection scheme Selection_Strategy; Use blue_opponent as the environment to interact with the main agent Red to collect battle data; Update the strategy parameters of the main agent Red based on the collected data; Add the updated Red to Red_Pool; Accompanying agent training phase: Select a master agent red_opponent from Red_Pool according to the opponent selection scheme Selection_Strategy; Use red_opponent as the environment to interact with the sparring agent Blue to collect battle data; Update the strategy parameters of the sparring agent Blue based on the collected data; Add the updated Blue to Blue_Pool; Rating Update: Update the strength scores of the main agent Red and the sparring agent Blue based on the battle results; Periodically save historical agents: If the current round i is a save round, add the current versions of Red and Blue to their respective historical strategy pools; Dynamically adjust opponent selection scheme: Adjust the parameters of the opponent selection strategy Selection_Strategy according to the current battle results; endfor Returns the trained main agent Red.

[0077] In this embodiment, by implementing alternating training of the main agent and the sparring agent in an asymmetric task environment, the embodiment uses mechanisms such as opponent selection strategy, strength score update, and historical strategy pool management to enable the agent to continuously optimize the strategy in a diverse opponent environment. This embodiment realizes the co-evolution of the main agent and the sparring agent through real-time collection of battle data and strategy adjustment to enhance the adaptability and confrontation ability of the main agent. After each round of training, the score is updated, the opponent selection scheme is dynamically adjusted, and the strategy version is saved regularly, which ensures the diversity and challenge of the strategy, thereby improving the performance of the agent in complex task scenarios.

[0078] In this embodiment, the problem that the traditional symmetric task training method is difficult to apply to asymmetric tasks is solved through an innovative alternating training method and opponent selection strategy. Specifically: First, improve strategy diversity and enhance the adaptability of the main intelligent agent.

[0079] Since the main agent and the sparring agent in the asymmetric task have different task objectives, the traditional self-playing method is difficult to provide sufficient strategy diversity, which easily leads to the convergence of the agent on a single strategy and insufficient adaptability. This embodiment introduces a historical strategy pool and a dynamic opponent selection scheme to enable the main agent to face a variety of opponents during training, thereby improving its adaptability to different strategies. This design avoids the problem of the strategy falling into the local optimum and makes the agent more robust.

[0080] Secondly, the opponent selection and strategy update mechanism based on strength score is used to optimize the agent's confrontation ability.

[0081] This embodiment updates the strength score after the battle, so that the score of each agent can truly reflect its confrontation ability and provide quantitative training feedback. The opponent selection scheme based on the strength score can dynamically select opponents according to the current level of the agent, ensuring that the main agent can encounter suitable challenging opponents at each stage and gradually improve its confrontation ability. Compared with the traditional fixed opponent training method, this embodiment makes the strategy optimization process of the agent more efficient.

[0082] Again, dynamically adjust the opponent selection plan to prevent strategy overfitting.

[0083] In traditional training, the agent may overfit to a fixed opponent, resulting in a decrease in adversarial resistance. This embodiment adds random selection and dynamic adjustment mechanisms to the opponent selection scheme, which can flexibly adjust the selection probability according to the current battle results. In this way, when the main agent has a high winning rate against a certain opponent, the probability of random selection can be increased to ensure that the main agent is exposed to more opponents with different strategies, thereby preventing overfitting and further enhancing the diversity and challenge of the strategy.

[0084] Finally, the management of the historical strategy pool (i.e., historical version pool) ensures continuous optimization of training.

[0085] In each round of training, this embodiment ensures the quality and diversity of the opponent pool by regularly storing the latest version of the agent strategy into the historical strategy pool. Compared with the traditional static strategy pool, the dynamic strategy pool management method of this embodiment can provide continuously evolving opponent strategies and provide a richer environment for the training process of the main agent. This not only improves the training effect, but also ensures the continuous evolution of the agent in a complex asymmetric task environment.

[0086] Corresponding to the agent training method in the above embodiment, Figure 4 This is a structural block diagram of an intelligent agent training device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 4 The intelligent agent training device 20 includes: a strategy model determination module 21, a strategy model update module 22, and a historical version pool update module 23.

[0087] Among them, the strategy model determination module 21 is used to determine the first strategy model of the main agent and the second strategy model of the sparring agent, the first strategy model is a model in the historical version pool of the main agent, and the second strategy model is a model in the historical version pool of the sparring agent; A policy model updating module 22, configured to update the first policy model based on the interaction data between the first policy model and the second policy model to obtain a third policy model, and to update the second policy model based on the interaction data between the first policy model and the second policy model to obtain a fourth policy model; The historical version pool update module 23 is used to add the third strategy model to the historical version pool of the main agent and add the fourth strategy model to the historical version pool of the sparring agent.

[0088] In one embodiment of the present disclosure, the policy model determination module 21 is further configured to: In response to the absolute value of the difference in strength scores between the main agent and the sparring agent being less than or equal to a first value, the strategy model corresponding to the main agent is used as the first strategy model, and the strategy model of the sparring agent is used as the second strategy model.

[0089] In one embodiment of the present disclosure, the policy model determination module 21 is further configured to: Based on the target probability, a policy model is randomly selected from the historical version pool of the sparring agent as the second policy model.

[0090] In one embodiment of the present disclosure, the policy model updating module 22 is specifically used to: Acquire interaction data between the first strategy model and the second strategy model; Update the policy parameters of the first policy model based on the deep reinforcement learning algorithm to obtain a third policy model; Based on the deep reinforcement learning algorithm, the strategy parameters of the second strategy model are updated to obtain a fourth strategy model.

[0091] In one embodiment of the present disclosure, the agent training device 20 further includes: a strength scoring module, which is specifically used to: The strength scores of the main agent and the sparring agent are updated based on the interaction data of the first strategy model and the second strategy model.

[0092] In one embodiment of the present disclosure, the strength scoring module is further used to: Update the strength scores of the main agent and the sparring agent based on the first formula; The first formula is:

[0093] in, Represents the updated strength score, It is the strength score before the update. It is represented as the step size parameter of score update, It represents the interaction result between the main agent and the sparring agent. Represents the expected winning rate of the master agent or the sparring agent.

[0094] In one embodiment of the present disclosure, the strength scoring module is further used to: Calculate the expected winning rate of the main agent or the sparring agent based on the second formula; The second formula is:

[0095] in, It is the strength score of the opponent. Indicates your own strength rating.

[0096] See also Figure 5 , Figure 5 Schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Figure 5 The electronic device 300 in the embodiment shown may include: one or more processors 301, one or more input devices 302, one or more output devices 303 and one or more memories 304. The processors 301, input devices 302, output devices 303 and memories 304 communicate with each other via a communication bus 305. The memory 304 is used to store computer programs, which include program instructions. The processor 301 is used to execute the program instructions stored in the memory 304. The processor 301 is configured to call the program instructions to execute the functions of the modules in the above-mentioned device embodiments, such as Figure 4 The functions of modules 21 to 23 are shown.

[0097] It should be understood that in the embodiment of the present disclosure, the processor 301 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0098] The input device 302 may include a touch panel, a fingerprint collection sensor (for collecting the user's fingerprint information and fingerprint direction information), a microphone, etc., and the output device 303 may include a display (LCD, etc.), a speaker, etc.

[0099] The memory 304 may include a read-only memory and a random access memory, and provide instructions and data to the processor 301. A portion of the memory 304 may also include a non-volatile random access memory. For example, the memory 304 may also store information about the device type.

[0100] In a specific implementation, the processor 301, input device 302, and output device 303 described in the embodiments of the present disclosure can execute the implementation methods described in the first and second embodiments of the intelligent agent training method provided in the embodiments of the present disclosure, and can also execute the implementation methods of the electronic device described in the embodiments of the present disclosure, which will not be repeated here.

[0101] In another embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by the processor, all or part of the processes in the above-mentioned embodiment method are implemented, and the computer program can also be completed by instructing the relevant hardware through the computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0102] The computer-readable storage medium may be an internal storage unit of the electronic device of any of the aforementioned embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of the electronic device and an external storage device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0103] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0104] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the electronic devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0105] In the several embodiments provided in the present application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces or units, or it can be an electrical, mechanical or other form of connection.

[0106] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present disclosure.

[0107] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0108] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present disclosure, and these modifications or replacements should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for training an intelligent agent, characterized in that: include: Determine a first strategy model of the master agent and a second strategy model of the sparring agent, wherein the first strategy model is a model in the master agent historical version pool, and the second strategy model is a model in the sparring agent historical version pool; Based on the interaction data between the first policy model and the second policy model, the first policy model is updated to obtain a third policy model; based on the interaction data between the first policy model and the second policy model, the second policy model is updated to obtain a fourth policy model; The third strategy model is added to the historical version pool of the main agent, and the fourth strategy model is added to the historical version pool of the sparring agent.

2. The agent training method according to claim 1, characterized in that: The step of determining the first strategy model of the master agent and the second strategy model of the sparring agent includes: In response to the absolute value of the difference in strength scores between the main agent and the sparring agent being less than or equal to a first value, the strategy model corresponding to the main agent is used as the first strategy model, and the strategy model of the sparring agent is used as the second strategy model.

3. The agent training method according to claim 2, characterized in that: Also includes: Based on the target probability, a policy model is randomly selected from the historical version pool of the sparring agent as the second policy model.

4. The agent training method according to claim 1, characterized in that: The method of updating the first policy model based on the interaction data between the first policy model and the second policy model to obtain a third policy model, and updating the second policy model based on the interaction data between the first policy model and the second policy model to obtain a fourth policy model comprises: Acquire interaction data between the first strategy model and the second strategy model; Update the strategy parameters of the first strategy model based on a deep reinforcement learning algorithm to obtain a third strategy model; Based on the deep reinforcement learning algorithm, the strategy parameters of the second strategy model are updated to obtain a fourth strategy model.

5. The agent training method according to claim 1, characterized in that: Also includes: The strength scores of the main agent and the sparring agent are updated based on the interaction data of the first strategy model and the second strategy model.

6. The agent training method according to claim 5, characterized in that: The updating of the strength scores of the master agent and the sparring agent based on the interaction data of the first strategy model and the second strategy model includes: Update the strength scores of the master agent and the sparring agent based on the first formula; The first formula is: in, Represents the updated strength score, It is the strength score before the update. It is represented as the step size parameter of score update, It represents the interaction result between the main agent and the sparring agent. Represents the expected winning rate of the master agent or the sparring agent.

7. The agent training method according to claim 6, characterized in that: Calculate the expected winning rate of the master agent or the sparring agent based on the second formula; The second formula is: in, It is the strength score of the opponent. Indicates your own strength rating.

8. An intelligent agent training device, characterized in that: include: A strategy model determination module, used to determine a first strategy model of a master agent and a second strategy model of a sparring agent, wherein the first strategy model is a model in a historical version pool of the master agent, and the second strategy model is a model in a historical version pool of the sparring agent; a policy model updating module, configured to update the first policy model based on the interaction data between the first policy model and the second policy model to obtain a third policy model, and to update the second policy model based on the interaction data between the first policy model and the second policy model to obtain a fourth policy model; The historical version pool update module is used to add the third strategy model to the historical version pool of the main agent and add the fourth strategy model to the historical version pool of the sparring agent.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images