Generalized agent generation method in cooperation and competition scene
By building a diversity strategy pool and turn memory library, optimizing updates and perspective alignment to transform policy network parameters, we generate agents that can flexibly respond to dynamic changes in collaboration and competition, solving the problem of insufficient generalization capabilities of agents in new tasks and environments, and significantly improving adaptability and generalization performance.
Patent Information
- Application Number
- CN202411713024.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-27
- Publication Date
- 2025-03-04
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
When integrating into the new multi-agent system, agents face the challenge of adapting to new task environments and teammates and opponents who have never interacted, and the existing technology has insufficient generalization ability in new tasks and environments.
A generalizable agent generation method in the collaboration and competition scenario is proposed. By building a diversity policy pool, building a round memory bank, optimizing and updating policy network parameters, perspective alignment conversion and hypernetwork generation policy network parameters, agiles that can flexibly cope with dynamic changes in collaboration and competition.
It significantly improves the adaptability of the agent in a dynamic environment and generalization performance in different evaluation scenarios, and can effectively collaborate and compete with unknown teammates and opponents.
Smart Images
Figure CN119558343A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of generalization of multi-agent systems, and in particular to a method for generating generalizable agents in collaborative and competitive scenarios. Background Art
[0002] After being trained in specific tasks and fixed environments, agents face many challenges when integrating into new multi-agent systems (Multi-Agent Systems, MAS). During the integration process, agents must adapt to new task environments and teammates and opponents with whom they have never interacted, which requires them to demonstrate extremely high flexibility and learning capabilities to quickly master effective strategies in complex and changing dynamic environments. The development of this technology is of great significance to multiple fields. For example, in the field of automated control, it can enable agents to seamlessly integrate into robot or drone swarms to perform complex tasks; in the gaming industry, this technology can significantly improve the intelligence and adaptability of artificial intelligence. Given the complexity of directly addressing this challenge, researchers have proposed two simplification strategies in order to gradually resolve this problem.
[0003] Zero-shot learning (ZSL) is a machine learning technique that allows agents to effectively perform tasks without being trained on specific tasks or environments. In the field of multi-agent reinforcement learning (MARL), ZSL significantly improves the adaptability and generalization of MARL. Collaboration between agents is the key to achieving generalization in different scenarios and tasks. However, current ZSL-based MARL research mainly focuses on cooperation within a group of agents. In practical applications, agents often need to collaborate with unknown teammates to complete tasks together, which requires them to be able to effectively cooperate with unfamiliar agents.
[0004] Learners have limited ability to control their teammates. Another research direction to enhance the generalization ability of agents is ad hoc teamwork (AHT). This approach aims to enable agents to seamlessly adapt to various teammates in a zero-shot setting. In an interactive environment, the trained agent is called a learner, which is characterized by continuously acquiring and applying relevant knowledge to improve its strategy during the interaction process. The AHT problem poses several key challenges. First, learners who are new to the team may face uncertainty due to their lack of experience in working with teammates. In order to quickly adapt to new teammates, learners need to adjust their strategies and roles. Second, since learners have limited control over the environment or influence on their teammates, learners need to show high flexibility and resilience to adapt to the dynamic changes of the environment and teammates. In addition, although all agents work together to achieve a team goal, some agents may still pursue individual goals that do not conflict with the collective task. Although the AHT method has achieved impressive results in training agents to cooperate with unfamiliar teammates to complete specific tasks, its generalization ability in new tasks and environments needs to be further enhanced.
[0005] In order to solve the above technical problems, enhance the adaptability of intelligent agents in the complex and dynamic real world, and give them true generalization capabilities, it is urgent to propose a generalizable agent generation method in collaborative and competitive scenarios. Summary of the invention
[0006] To solve the above technical problems, the present invention proposes a generalizable agent generation method in collaboration and competition scenarios. The method sets an agent that adapts to collaboration-competition scenarios (Agent Collaboration-Competition Adaptation, ACCA). The goal is to ensure that the agent can flexibly respond to the dynamic changes of collaboration and competition in unforeseen scenarios and tasks. Even when the strategies and numbers of teammates and opponents are unknown, the agent can still establish an effective cooperative relationship with teammates and form an effective competitive strategy against opponents, so as to solve the problems existing in the above-mentioned prior art.
[0007] To achieve the above object, the present invention provides a method for generating a generalizable intelligent agent in a collaborative and competitive scenario, comprising the following steps:
[0008] Constructing a diverse strategy pool, wherein the diverse strategy pool includes multiple sub-environments and multi-agent joint strategies;
[0009] Select a sub-environment, extract the strategy network parameters of the team agent, and assign them to the learner;
[0010] The multi-agent reinforcement learning algorithm is used to optimize and update the learner's strategy network parameters to obtain the learner's first strategy network parameters;
[0011] Constructing a round memory library, the round memory library including observation data of teammates and opponents, and performing perspective alignment conversion on the observation data of teammates and opponents;
[0012] In the optimization and updating process, based on the learner's observation features, the observation data that best matches the learner is retrieved from the round memory bank;
[0013] After position encoding the retrieved observation data, the learner's second strategy network parameters are generated based on the hypernetwork;
[0014] Combine the first strategy network parameters and the second strategy network parameters to obtain the optimized learner strategy network parameters and complete the agent generation.
[0015] Optionally, it also includes using a multi-agent reinforcement learning algorithm to optimize and update the strategy network parameters of teammates, and adding the optimized and updated strategy network parameters of teammates to the diversity strategy pool.
[0016] Optionally, the process of performing perspective alignment conversion on the observation data of the teammates and opponents includes: each teammate and opponent has an independent perspective alignment encoder, and the third-person observation of the teammates and opponents is mapped to the first-person based on the perspective alignment encoder, and the converted observation mapping is stored in the round memory library.
[0017] Optionally, the process of obtaining optimized learner strategy network parameters by combining the first strategy network parameters and the second strategy network parameters includes:
[0018] The first strategy network parameters and the second strategy network parameters, as well as the corresponding scaling factors, are combined to obtain optimized learner strategy network parameters.
[0019] The present invention also provides a generalizable agent generation system in a collaborative and competitive scenario, which is used to implement a generalizable agent generation method in a collaborative and competitive scenario, comprising: a diversity population module, a first network parameter updating module, a second network parameter updating module and an agent generation module;
[0020] The diversity population module is used to construct a diversity strategy pool, which includes multiple sub-environments and multi-agent joint strategies;
[0021] The first network parameter updating module is used to optimize and update the learner's strategy network parameters in the diversity strategy pool using a multi-agent reinforcement learning algorithm to obtain the learner's first strategy network parameters;
[0022] The second network parameter updating module is used to construct a round memory library. During the optimization and updating process, based on the learner's observation characteristics, the observation data that best matches the learner is retrieved from the round memory library, and the observation data is position-encoded, and the second strategy network parameters of the learner are generated based on the hypernetwork.
[0023] The agent generation module is used to combine the first strategy network parameters and the second strategy network parameters to obtain optimized learner strategy network parameters and complete agent generation.
[0024] Optionally, the diversity population module is also used to select a sub-environment, extract the strategy network parameters of the team agent, and assign them to the current learner.
[0025] Optionally, the first network parameter updating module is also used to optimize and update the strategy network parameters of teammates using a multi-agent reinforcement learning algorithm, and add the optimized and updated strategy network parameters of teammates to the diversity strategy pool.
[0026] Optionally, the second network parameter updating module includes:
[0027] A memory library construction unit, used to construct a round memory library, wherein the round memory library includes observation data of teammates and opponents;
[0028] A perspective conversion unit, used for performing perspective alignment conversion on the observation data of the teammates and opponents;
[0029] A data retrieval unit, used for retrieving observation data that best matches the learner from the round memory bank based on the learner's observation features;
[0030] The encoding integration unit is used to perform position encoding processing on the retrieved observation data, and then generate the learner's second strategy network parameters based on the hypernetwork.
[0031] The present invention also provides an electronic device, comprising: a memory and a processor; the memory is used to store programs; the processor is used to execute the programs to implement the various steps of the generalizable agent generation method in collaborative and competitive scenarios.
[0032] The present invention also provides a readable storage medium on which a computer program is stored. When the computer program is executed by a processor, each step of the method for generating a generalizable intelligent agent in cooperative and competitive scenarios is implemented.
[0033] Compared with the prior art, the present invention has the following advantages and technical effects:
[0034] The present invention constructs a round memory library, including observation data of teammates and opponents, to help the intelligent agent quickly build models; and performs perspective alignment conversion on the observation data of teammates and opponents; in the optimization and updating process, based on the learner's observation characteristics, the observation data that best matches the learner is retrieved from the round memory library; after position encoding processing of the retrieved observation data, the learner's second strategy network parameters are generated based on the hypernetwork; the first strategy network parameters and the second strategy network parameters are combined to obtain the optimized learner strategy network parameters to complete the generation of the intelligent agent. The present invention integrates the behaviors and attributes of all intelligent agents in the multi-agent system and explicitly models other intelligent agents. This modeling method not only improves the learner's adaptability in a dynamic environment, but also significantly enhances its generalization performance in different evaluation scenarios, and can effectively collaborate and compete with unknown teammates and opponents in the current environment, showing great potential in promoting the research of generalizable intelligent agents. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0036] Figure 1 Schematic diagrams comparing the generalization performance verification settings of existing intelligent agents in embodiments of the present invention and the intelligent agent settings proposed in this embodiment, wherein (a) is a schematic diagram of the generalization performance verification of multi-agents for zero-shot learning, (b) is a schematic diagram of the generalization performance verification of agents for temporary teamwork, and (c) is a schematic diagram of the generalization performance verification of the intelligent agents proposed in this embodiment;
[0037] Figure 2 An algorithm framework for multi-agent retrieval and strategy dynamic generation according to an embodiment of the present invention. DETAILED DESCRIPTION
[0038] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0039] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0040] Embodiment 1
[0041] In the prior art, there is a CSP method based on a decoupled solution. This method uses a detection module to analyze limited pre-collected turn data to characterize and classify new teammates, thereby improving the effect of multi-agent control. This method focuses on how agents collaborate with unknown teammates to complete tasks, and focuses on the diversity of teammate strategies and cooperation modes. However, these methods often fail to fully consider the changes in the environment and tasks, so their generalization ability is insufficient when facing new environments and tasks.
[0042] The ODITS method introduces a technique based on information regularization, which automatically approximates the hidden state of the global encoder as the hidden state of the local encoder. RPM stores and sorts the teammate strategies based on the rewards of the training round, improves the diversity of the agent group strategy, and enables the agent to choose a variety of behavioral strategies during training, thereby enhancing the generalization ability of the agent strategy and the ability to interact with unknown agents. However, these methods only implicitly simulate the behavior of teammates and rely too much on the knowledge obtained from the training data to infer the behavior and strategy of other agents. Therefore, when facing new teammates, learners may tend to cooperate with familiar teammates and ignore the interaction with new teammates.
[0043] In order to solve the above problems, Figure 1-Figure 2 As shown, this embodiment provides a method for generating a generalizable intelligent agent in a collaborative and competitive scenario, comprising the following steps:
[0044] Constructing a diverse strategy pool, wherein the diverse strategy pool includes multiple sub-environments and multi-agent joint strategies;
[0045] Select a sub-environment, extract the strategy network parameters of the team agent, and assign them to the learner;
[0046] The multi-agent reinforcement learning algorithm is used to optimize and update the learner's strategy network parameters to obtain the learner's first strategy network parameters;
[0047] Constructing a round memory library, the round memory library including observation data of teammates and opponents, and performing perspective alignment conversion on the observation data of teammates and opponents;
[0048] In the optimization and updating process, based on the learner's observation features, the observation data that best matches the learner is retrieved from the round memory bank;
[0049] After position encoding the retrieved observation data, the learner's second strategy network parameters are generated based on the hypernetwork;
[0050] Combine the first strategy network parameters and the second strategy network parameters to obtain the optimized learner strategy network parameters and complete the agent generation.
[0051] It is extremely challenging to generalize an agent that can perform specific tasks in a training environment to a new multi-agent system (Multi-Agent System, MAS), which requires the agent to not only adapt to changing environments and tasks, but also interact effectively with unfamiliar teammates and opponents. To meet this challenge, this embodiment proposes two simplified scenario settings: one is multi-agent reinforcement learning (Multi-Agent Reinforcement Learning, MARL) for zero-shot learning (ZSL), and the other is ad hoc teamwork (Ad Hoc Teamwork, AHT). Based on these studies, this embodiment proposes a generalizable agent generation method in collaborative and competitive scenarios.
[0052] Furthermore, the multi-agent reinforcement learning algorithm adopted in this embodiment adopts the Multi-Retrieval and Dynamic Generation (MRDG) method, which effectively models teammates and opponents based on the agent behavior trajectory. It includes a position encoder for adapting to different team sizes and a hypernetwork module to enhance the learning and adaptability of the agent. In addition, this embodiment also designs a perspective alignment module to ensure that the learning agent is consistent with the observation perspective of the retrieved teammates and opponents. This embodiment has been extensively tested in the SMAC, Overcooked and Melting Pot experimental scenarios. The comparative analysis results show that the MRDG method outperforms the existing baseline methods in new environments and tasks, and significantly improves the robust collaboration and competitiveness of the agent with unseen teammates and opponents.
[0053] The goal of this embodiment is to design an agent that can adapt to changes in different environments, tasks, and teams, and quickly achieve collaboration and competition with teammates and opponents. The policy network parameters of the agent are expressed as θ = <θ marl ,θ hype >.
[0054] In the initial stage of training, this embodiment randomly selects a sub-environment in the diverse team group and determines the corresponding task optimization target.
[0055] Next, the policy network parameters of the team agent are extracted from this sub-environment and assigned to the current group of agents.
[0056] In this framework, the strategies of learners and team members are jointly optimized through a multi-agent reinforcement learning algorithm. The purpose of this example is to obtain a strategy network parameter θ marlThe learner is the first policy network parameter of the learner.
[0057] Throughout the optimization process, the observations and actions of the team agent are transformed through perspective alignment and stored in the episode memory; this alignment ensures that the observation perspective of the team agent remains consistent with that of the learner.
[0058] Subsequently, this embodiment uses the learner's observation features after viewpoint alignment to retrieve the observation-action pairs of the team agent that best matches the learner from the round memory to explicitly model the team behavior.
[0059] The retrieved action value is processed by position encoding and then input into the hypernetwork, which generates the learner’s strategy parameter θ hype , which is the learner’s second strategy network parameter.
[0060] Finally, the trained team members’ strategy parameters are stored back in the corresponding sub-environment in the diverse team population.
[0061] Furthermore, in order to enable learners to effectively cooperate with diverse team members, this embodiment designs a diversity strategy pool (DPP), which contains multiple sub-environments and multi-agent joint strategies. In these sub-environments, learners will interact with different teammates, and their rewards are affected by the learner strategy and the teammate joint strategy. The Q value of each strategy is optimized by minimizing the regression loss. In order to maintain the diversity of strategies, this embodiment randomly samples sub-environments and joint strategies before the start of each round, and adds the updated teammate strategy to the corresponding storage slot at the end of the round. Such a dynamic interactive storage mechanism ensures the diversity of multi-agent trajectories, thereby enhancing the learner's ability to cooperate in diverse teams.
[0062] Furthermore, this embodiment establishes a round memory library containing multi-step trajectories to collect data from teammates and opponents to help the agent quickly build models. The observation data is processed by the view alignment encoder, and the most similar action sequence is retrieved from the memory library using the retrieval network to predict the next action of teammates and opponents. In addition, this embodiment introduces position encoding to mark the position and order of teammates or opponents, and a hypernetwork to integrate the action and position information of teammates and opponents into the learner's strategy network to improve the generalization ability of the model and cope with the challenges brought by partial observability.
[0063] Furthermore, during the training process, in order to enable the learner to match the observations of other agents, the present embodiment uses a perspective alignment (VA) encoder to map the third-person observations of teammates and opponents to the first-person. Each teammate or opponent has an independent VA encoder to map their observations and store them in the round memory. The learner's observations are also encoded by the same VA encoder, and then the L2 loss is used to ensure that the observations of teammates or opponents are as close as possible to the learner's observations, with this as the goal to optimize the parameters of the VA encoder. In this way, the observations of team members are converted to the learner's perspective. During reasoning, the learner's observations are only extracted through the VA encoder to extract features without perspective conversion, thereby achieving the sharing of VA encoders between team members.
[0064] Furthermore, during the training process, the learner's policy network parameters are divided into two parts: one part is obtained through standard multi-agent reinforcement learning and interaction with teammates or opponents, and the other part is generated by a super network, which uses the retrieved action values and position encodings of the team agents to generate parameters. This design helps learners quickly retrieve and model teammates or opponents during testing, accelerating strategy convergence. In order to enhance the learner's adaptability to new teammates, this embodiment adopts a reparameterization method to adjust the initial degree of network parameters by combining the learner's policy network parameters and the reparameterization initial parameters, as well as the corresponding scaling factors.
[0065] The agent generated in this embodiment is called an agent adapted to collaborative-competitive scenarios (ACCA). This embodiment integrates the behaviors and attributes of all agents in a multi-agent system and explicitly models other agents. This modeling method not only improves the learner's adaptability in a dynamic environment, but also significantly enhances its generalization performance in different evaluation scenarios, and can effectively collaborate and compete with unknown teammates and opponents in the current environment, showing great potential in promoting the research of generalizable agents.
[0066] This embodiment also provides a generalizable agent generation system in a collaborative and competitive scenario, which is used to implement a generalizable agent generation method in a collaborative and competitive scenario, including: a diversity population module, a first network parameter updating module, a second network parameter updating module and an agent generation module;
[0067] The diversity population module is used to construct a diversity strategy pool, which includes multiple sub-environments and multi-agent joint strategies;
[0068] The first network parameter updating module is used to optimize and update the learner's strategy network parameters in the diversity strategy pool using a multi-agent reinforcement learning algorithm to obtain the learner's first strategy network parameters;
[0069] The second network parameter updating module is used to construct a round memory library. In the optimization and updating process, based on the learner's observation characteristics, the observation data that best matches the learner is retrieved from the round memory library, and the observation data is position-encoded, and the learner's second strategy network parameters are generated based on the hypernetwork.
[0070] The intelligent agent generation module is used to combine the first strategy network parameters and the second strategy network parameters to obtain the optimized learner strategy network parameters and complete the intelligent agent generation.
[0071] Optionally, the diversity population module is also used to select a sub-environment, extract the strategy network parameters of the team agent, and assign them to the current learner.
[0072] It is feasible that the first network parameter updating module is also used to optimize and update the strategy network parameters of teammates using a multi-agent reinforcement learning algorithm, and add the optimized and updated strategy network parameters of teammates to the diversity strategy pool.
[0073] The second network parameter updating module may include:
[0074] A memory library construction unit, used to construct a round memory library, wherein the round memory library includes observation data of teammates and opponents;
[0075] A perspective conversion unit, used for performing perspective alignment conversion on the observation data of the teammates and opponents;
[0076] A data retrieval unit, configured to retrieve observation data that best matches the learner from the round memory bank based on the learner's observation features;
[0077] The encoding integration unit is used to perform position encoding processing on the retrieved observation data, and then generate the learner's second strategy network parameters based on the hypernetwork.
[0078] Embodiment 2
[0079] This embodiment also provides an electronic device, including: a memory and a processor; the memory is used to store programs; the processor is used to execute the programs to implement various steps of the generalizable agent generation method in collaborative and competitive scenarios.
[0080] Embodiment 3
[0081] This embodiment also provides a readable storage medium on which a computer program is stored. When the computer program is executed by a processor, each step of the method for generating a generalizable intelligent agent in cooperative and competitive scenarios is implemented.
[0082] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for generating generalizable intelligent agents in cooperative and competitive scenarios, characterized in that: The following steps are involved: Constructing a diverse strategy pool, wherein the diverse strategy pool includes multiple sub-environments and multi-agent joint strategies; Select a sub-environment, extract the strategy network parameters of the team agent, and assign them to the learner; The multi-agent reinforcement learning algorithm is used to optimize and update the learner's strategy network parameters to obtain the learner's first strategy network parameters; Constructing a round memory library, the round memory library including observation data of teammates and opponents, and performing perspective alignment conversion on the observation data of teammates and opponents; In the optimization and updating process, based on the learner's observation features, the observation data that best matches the learner is retrieved from the round memory bank; After position encoding the retrieved observation data, the learner's second strategy network parameters are generated based on the hypernetwork; Combine the first strategy network parameters and the second strategy network parameters to obtain the optimized learner strategy network parameters and complete the agent generation.
2. The method according to claim 1, characterized in that It also includes using a multi-agent reinforcement learning algorithm to optimize and update the strategy network parameters of teammates, and adding the optimized and updated strategy network parameters of teammates to the diversity strategy pool.
3. The method according to claim 1, characterized in that The process of performing perspective alignment conversion on the observation data of the teammates and opponents includes: each teammate and opponent has an independent perspective alignment encoder, and the third-person observation of the teammates and opponents is mapped to the first-person based on the perspective alignment encoder, and the converted observation mapping is stored in the round memory library.
4. The method according to claim 1, characterized in that: The process of obtaining optimized learner strategy network parameters by combining the first strategy network parameters and the second strategy network parameters includes: The first strategy network parameters and the second strategy network parameters, as well as the corresponding scaling factors, are combined to obtain optimized learner strategy network parameters.
5. A generalizable agent generation system in cooperative and competitive scenarios, characterized in that: A generalizable agent generation method for implementing cooperation and competition scenarios, comprising: a diversity population module, a first network parameter updating module, a second network parameter updating module and an agent generation module; The diversity population module is used to construct a diversity strategy pool, which includes multiple seed environments and multi-agent joint strategies; The first network parameter updating module is used to optimize and update the learner's strategy network parameters in the diversity strategy pool using a multi-agent reinforcement learning algorithm to obtain the learner's first strategy network parameters; The second network parameter updating module is used to construct a round memory library. During the optimization and updating process, based on the learner's observation characteristics, the observation data that best matches the learner is retrieved from the round memory library, and the observation data is position-encoded, and the second strategy network parameters of the learner are generated based on the hypernetwork. The agent generation module is used to combine the first strategy network parameters and the second strategy network parameters to obtain optimized learner strategy network parameters and complete agent generation.
6. The system according to claim 5, characterized in that The diversity population module is also used to select a sub-environment, extract the strategy network parameters of the team agent, and assign them to the current learner.
7. The system according to claim 5, characterized in that The first network parameter updating module is also used to optimize and update the strategy network parameters of teammates using a multi-agent reinforcement learning algorithm, and add the optimized and updated strategy network parameters of teammates to the diversity strategy pool.
8. The system according to claim 5, characterized in that The second network parameter updating module includes: A memory library construction unit, used to construct a round memory library, wherein the round memory library includes observation data of teammates and opponents; A perspective conversion unit, used for performing perspective alignment conversion on the observation data of the teammates and opponents; A data retrieval unit, configured to retrieve observation data that best matches the learner from the round memory bank based on the learner's observation features; The encoding integration unit is used to perform position encoding processing on the retrieved observation data, and then generate the learner's second strategy network parameters based on the hypernetwork.
9. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; the processor is used to execute the programs to implement the various steps of the method for generating a generalizable intelligent agent in a collaborative and competitive scenario as described in any one of claims 1-4.
10. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the method for generating a generalizable intelligent agent in a cooperative and competitive scenario as described in any one of claims 1 to 4 is implemented.