Multi-agent parameter sharing method and multi-agent system
By combining time-varying contrastive sequence encoding and graph attention mechanism network connection in a multi-agent system, dynamic cooperative relationships between agents are realized, solving the problems of insufficient policy diversity and flexibility in existing technologies, and improving the system's cooperative ability and task processing complexity.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGDONG LAB OF ARTIFICIAL INTELLIGENCE & DIGITAL ECONOMY (SZ)
- Filing Date
- 2023-05-16
- Publication Date
- 2026-05-22
AI Technical Summary
In existing collaborative multi-agent reinforcement learning modeling, the full parameter sharing mechanism limits the policy diversity and flexibility of agents, making it impossible to establish reliable collaborative relationships based on real-time environmental needs, leading to task failure.
By encoding time-varying comparative sequences of multiple agents, using graph attention network connections to combine abstract features, and selecting matching Q-networks for parameter sharing, agents can choose partners according to environmental changes, thus achieving dynamic collaborative relationships.
It improves the collaborative ability and overall capability limit of multi-agent systems, enabling them to handle more complex tasks without being limited by the complexity of experimental scenarios, and exhibits better dynamics.
Smart Images
Figure CN116663606B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a method for sharing parameters among multiple agents and a multi-agent system. Background Technology
[0002] With societal progress and technological advancements, the application of intelligent agents in the field of artificial intelligence is becoming increasingly widespread. An intelligent agent is an individual system that perceives its environment and interacts with it using existing knowledge or its own iterative strategies to complete a designated task. Multiple such agents can form a multi-agent system (MAS), which can address the limitations of a single agent's capabilities. Furthermore, with the deepening development of artificial intelligence, reinforcement learning techniques are frequently applied to agent modeling to enhance the agent's perception, learning, and decision-making abilities. In addition, collaboration among multiple agents can raise the overall system's capability ceiling, enabling the system to handle more complex tasks. Therefore, collaborative multi-agent reinforcement learning modeling has emerged.
[0003] In cooperative multi-agent reinforcement learning modeling, a full parameter sharing mechanism is often used, where all agents in a cooperative multi-agent system share a single policy network (or Q-network). However, this mechanism only provides a fixed cooperation pattern for multiple agents. This not only limits the exploration of diverse policies by the agents but also prevents them from flexibly changing their cooperation patterns according to real-time environmental requirements. Consequently, the agents are unable to establish and improve complementary and reliable cooperative relationships based on the real-time needs of the environment, leading to task failure. Summary of the Invention
[0004] This application provides a multi-agent parameter sharing method and a multi-agent system, which can improve the cooperation ability of multiple agents, is not limited by the complexity of the experimental scenario, and has better dynamics. The technical solution is as follows:
[0005] Firstly, a method for sharing parameters among multiple agents is provided, the method comprising:
[0006] Encode time-varying contrastive sequences of multiple agents to determine the first abstract features of the agents; use a graph attention network to connect and combine the first abstract features of the agents to obtain the second abstract features of the agents; the second abstract features are higher-level abstract features compared to the first abstract features; select a matching first Q network for each agent based on the first abstract features; agents with the same matching first Q network share parameters; select a matching second Q network for each agent based on the second abstract features; agents with the same matching second Q network share parameters.
[0007] In some possible implementations, when the first agent is in an interactive state with the environment, if the first update condition is not met at the current moment, the time-varying comparison sequence of the first agent is collected into the time-varying comparison sequence set; the first agent can be any one of multiple agents; if the first update condition is met at the current moment, the elements in the time-varying comparison sequence set are encoded to obtain the first abstract feature of the first agent.
[0008] In some possible implementations, elements from the time-varying comparison sequence set are input into a gated recurrent unit to obtain the first temporal feature corresponding to the time-varying comparison sequence; the first temporal feature is input into an encoder network to obtain a hidden Gaussian distribution; the hidden Gaussian distribution is sampled to obtain the first abstract feature of the first agent; and the time-varying comparison sequence set is cleared.
[0009] In some possible implementations, when the first agent is in an interactive state with the environment, if the second update condition is met at the current moment, a graph attention mechanism network is used to connect and combine the first abstract features of multiple agents to obtain the second abstract features of the first agent.
[0010] In some possible implementations, after inputting the first temporal features into the encoder network to obtain the hidden Gaussian distribution, the method further includes: collecting the hidden Gaussian distribution into a set of hidden Gaussian distributions; using a graph attention mechanism network to connect and combine the first abstract features of multiple agents to obtain the second abstract features of the first agent, including: inputting the elements in the set of hidden Gaussian distributions into a gated recurrent unit to obtain the second temporal features of the first agent; inputting the second temporal features into the graph attention mechanism network to generate a value distribution; sampling the value distribution using a reinforcement learning method to obtain the second abstract features of the first agent; and clearing the set of hidden Gaussian distributions.
[0011] In some possible implementations, a first Q network matching the first abstract feature of the first agent is selected from the first Q network pool to obtain the first Q network of the first agent; the first agent can be any one of multiple agents.
[0012] In some possible implementations, a second Q network matching the second abstract feature of the first agent is selected from the second Q network pool to obtain the second Q network of the first agent; the first agent is any one of the plurality of agents.
[0013] In some possible implementations, the method determines whether a preset network training trigger condition is met. If the network training trigger condition is met, the neural network is trained and updated. If the network training trigger condition is not met, the following steps are executed: encoding the time-varying comparison sequences of multiple agents to determine the first abstract features of the multiple agents; using the second abstract features of the multiple agents as the difference discrimination criterion to group the multiple agents, and obtaining the second grouping result, the method further includes: if a preset loop termination condition is not met, returning to the execution step: encoding the time-varying comparison sequences of multiple agents to determine the first abstract features of the multiple agents; if the preset loop termination condition is met, returning to the execution step: determining whether the preset network training trigger condition is met.
[0014] The first aspect of this application provides a multi-agent parameter sharing method. This method encodes time-varying comparative sequences of multiple agents to determine first abstract features for each agent. A graph attention network is then used to connect and combine these first abstract features to obtain second abstract features for each agent. Based on the first abstract features, a matching first Q-network is selected for each agent, with agents sharing the same first Q-network. Similarly, based on the second abstract features, a matching second Q-network is selected for each agent, with agents sharing the same second Q-network. Firstly, this method provides diverse strategy choices for multiple agents, enabling more accurate identification of differences between agents. Secondly, the Q-network is not fixed during parameter sharing; agents can choose appropriate partners based on environmental changes. Furthermore, the maximum number of sampleable types for the first and second abstract features is much smaller than the number of agents. Therefore, the parameter sharing mechanism proposed in this application is less affected by an increase in the number of agents, thus improving the reliability of the Q-network and enhancing the cooperation among multiple agents. Furthermore, in this method, the second abstract feature is obtained by connecting and combining the first abstract feature. That is, there is a correlation between the first and second abstract features. Therefore, there is also a collaborative relationship between the groups obtained based on the first and second abstract features, which improves the collaborative ability of multi-agents and thus increases the upper limit of the overall system's capabilities, enabling the system to handle more complex tasks. Moreover, this method is not limited by the complexity of the experimental scenario when sharing parameters, thus exhibiting better dynamism.
[0015] Secondly, a multi-agent system is provided, which includes the multiple agents provided in the first aspect.
[0016] Thirdly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the above-described multi-agent parameter sharing method.
[0017] It is understood that the beneficial effects of the second and third aspects mentioned above can be found in the relevant descriptions in the first aspect above, and will not be repeated here. Attached Figure Description
[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a schematic diagram illustrating different parameter sharing mechanisms provided in the embodiments of this application;
[0020] Figure 2 This is a flowchart of a multi-agent parameter sharing method provided in an embodiment of this application;
[0021] Figure 3 This is a flowchart of yet another example of a multi-agent parameter sharing method provided in the embodiments of this application;
[0022] Figure 4 This is a schematic diagram of the structure of an example of a multi-agent parameter sharing method provided in an embodiment of this application;
[0023] Figure 5 This is a code diagram illustrating an example of a multi-agent parameter sharing method provided in an embodiment of this application;
[0024] Figure 6 This is a schematic diagram of the structure of the multi-agent parameter sharing device provided in the embodiments of this application;
[0025] Figure 7 This is a schematic diagram of the structure of the multi-agent system provided in the embodiments of this application. Detailed Implementation
[0026] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.
[0027] It should be understood that "multiple" as mentioned in this application refers to two or more. In the description of this application, unless otherwise stated, " / " indicates "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist, for example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, to facilitate a clear description of the technical solutions of this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and that "first," "second," etc., do not necessarily imply differences.
[0028] To better understand the embodiments of this application, the terms or concepts that may be involved in the embodiments are explained below.
[0029] 1. Learning rate
[0030] The learning rate of a neural network refers to the rate at which the network learns from data. It is a crucial hyperparameter in supervised learning and deep learning, determining whether and when the objective function converges to a local minimum. A suitable learning rate allows the objective function to converge to a local minimum within a reasonable timeframe. In gradient descent, the learning rate guides us in adjusting the network weights using the gradient of the loss function.
[0031] 2. Experience Replay Buffer
[0032] In reinforcement learning, a replay cache is a data structure used to store past experiences. A replay cache is a cache used to store the actions an agent performs and the observations made in the environment. It helps the agent learn from past experiences to better perform the current task.
[0033] 3. Target network
[0034] In reinforcement learning, the target network is a neural network used to calculate the Q-value. It is a separate network from the main network, used to compute the target Q-value. The parameters of the target network are periodically updated from the parameters of the main network. The target network can make training more stable because it reduces fluctuations in the target value.
[0035] 4. Interaction cycle (episode)
[0036] An interaction cycle is one complete interaction cycle between an agent and its environment.
[0037] 5. Gated Recurrent Units (GRUs)
[0038] GRU is a gating mechanism in recurrent neural networks, introduced by Kyunghyun Cho et al. in 2014. It is similar to Long Short-Term Memory (LSTM) networks with forget gates. Compared to LSTM, GRU has a simpler gating mechanism, consisting of only update and reset gates. The internal structure of GRU comprises three parts: a reset gate, an update gate, and candidate hidden states. The reset gate controls how much of the previous hidden state is forgotten in relation to the current input; the update gate controls how much of the previous hidden state is updated in relation to the current input; and the candidate hidden state is determined jointly by the current input and the previous hidden state.
[0039] 6. Graph Attention Network (GAT)
[0040] GAT is a type of Graph Neural Network (GNN). GNNs use an attention mechanism to learn representations of nodes in a graph. GAT introduces a multi-head attention mechanism to enrich model capacity and stabilize the learning process. Each attention head has its own parameters, and their outputs can be combined in two ways: by connecting or averaging multiple graph attention layers. Graph attention networks include K-head graph attention networks (K-GAT).
[0041] 7. Attention Mechanism
[0042] Attention mechanisms are techniques that mimic cognitive attention, boosting the weights of certain parts of the input data to a neural network while weakening the weights of others. This allows the network to focus its attention on the most important small subset of the data. Attention mechanisms can be applied to various deep learning models, such as image classification and natural language processing.
[0043] 8. Batch processing
[0044] Multiple cycles (i.e. multiple episodes) in which the batch processing agent fully interacts with the environment.
[0045] Before providing a detailed explanation of the embodiments of this application, the application scenarios of these embodiments will be described first.
[0046] This application applies to multi-agent systems, which include multiple agents. In many scenarios, these agents need to collaborate to complete a common task. Therefore, reinforcement learning modeling of collaborative multi-agent systems is necessary. During this reinforcement learning modeling process, if the agents do not share parameters, each agent uses its own Q-network, such as... Figure 1 As shown in Figure b, without parameter sharing, the collaborative ability between agents is limited, as is the complexity of the tasks they can perform, and they may even be unable to collaborate at all. Therefore, agents can increase the overall system's capability ceiling by sharing Q-network parameters, enabling the system to handle more complex tasks.
[0047] In related technologies, the Q-network parameter sharing mechanisms in collaborative multi-agent reinforcement learning modeling include the following:
[0048] 1) Full parameter sharing
[0049] like Figure 1 As shown in Figure a, the full parameter sharing mechanism means that all agents share a single policy network. This mechanism only provides agents with a fixed cooperation pattern, thus limiting their ability to explore diverse policies and preventing them from flexibly changing their cooperation patterns according to real-time environmental requirements. Consequently, agents are unable to establish and improve complementary and reliable cooperative relationships based on real-time environmental needs, leading to task failure.
[0050] 2) Partial parameter sharing (Celebrating diversity in shared multi-agent reinforcement learning, CDS)
[0051] like Figure 1 As shown in Figure c, the CDS mechanism divides the agent's policy network into shared and independent parts. The independent part of the policy network introduced by each agent is fixed, so the computational cost of the independent part is very sensitive to changes in the number of agents. Although the data sharing mechanism introduces a regularization term to guide agents to choose to share with other agents more, how to balance the relationship between the shared and independent parts is severely affected by the environment.
[0052] 3) Static Selective Parameter Sharing Machine (SePS)
[0053] like Figure 1As shown in Figure d, the SePS mechanism clusters agents based on empirical trajectories and fixed encoding, and shares parameters among agents in the same group. However, this parameter sharing mechanism has two problems: firstly, the effectiveness of grouping agents through clustering depends on the complexity of the experimental scenario and can only be implemented during centralized training, which severely limits the dynamism of the parameter sharing mechanism; secondly, no cooperative relationship is established between different groups of agents.
[0054] Therefore, this application provides a method for sharing parameters among multiple agents, which can effectively improve the diversity of agent strategies and promote reliable complementary cooperation among agents based on the dynamic requirements of the environment.
[0055] The following is a detailed explanation of a multi-agent parameter sharing method provided in the embodiments of this application.
[0056] Figure 2 This is a flowchart of a multi-agent parameter sharing method provided in an embodiment of this application. See also... Figure 2 The system controls the interaction between each agent in the multi-agent system and the environment according to an interaction cycle. During the interaction, within the current interaction cycle (episode), steps S101 to S104 are executed. It can be understood that steps S101 to S104 are repeated in other interaction cycles.
[0057] S101: Encode the time-varying comparison sequence of multiple agents to determine the first abstract feature of the multiple agents.
[0058] Time-varying contrastive sequences, also known as word contrastive sequences, are introduced by agents to define abstract differences between them. The low-level abstract features obtained by encoding based on the agent's time-varying contrastive sequences are called first abstract features (also known as low-level abstract concepts).
[0059] Specifically, taking any one of the multiple intelligent agents (referred to as the first intelligent agent) as an example, the process of obtaining the first abstract feature of the first intelligent agent is explained:
[0060] A. Initialize the first abstract feature of the first agent. Specifically, the first abstract feature of the first agent can be randomly determined as the first initial abstract feature of the first agent. For example, the first initial abstract feature is {0, 1, 2}.
[0061] B. During the interaction between the first intelligent agent and the environment, within the current interaction period (episode), it is determined in real time whether the first update condition is met at the current moment; if the first update condition is not met at the current moment, then step C is executed; if the first update condition is met at the current moment, then step D is executed.
[0062] The first update condition is the condition for updating the first abstract feature. It can be a time condition, which is optional. The first update condition is determined according to the first preset period. For example, the first update condition can be: the current time is the start time of the first preset period, and the current time is not the start time of the current interaction period.
[0063] C. Use the first initial abstract feature as the first abstract feature of the first agent, and collect the time-varying comparison sequence of the first agent into the time-varying comparison sequence set.
[0064] D. Encode the elements in the time-varying comparison sequence set to obtain the first abstract feature of the first agent.
[0065] Specifically, step D can be implemented through the following process: The first agent inputs the elements in the time-varying contrast sequence set into the gated recurrent unit to obtain the first temporal feature corresponding to the time-varying contrast sequence. The first temporal feature is input into the encoder network to obtain the hidden layer Gaussian distribution (also known as the hidden space distribution). The hidden layer Gaussian distribution is sampled to obtain the first abstract feature of the first agent.
[0066] Optionally, the hidden layer Gaussian distribution can be sampled using the argmax function to obtain the maximum value of the hidden layer Gaussian distribution, and the obtained maximum value can be determined as the first abstract feature of the first agent.
[0067] S102: The graph attention mechanism network is used to connect and combine the first abstract features of multiple agents to obtain the second abstract features of multiple agents; the second abstract features are higher-level abstract features compared to the first abstract features.
[0068] The second abstract feature (also known as the high-level abstract concept) is obtained by connecting and combining the first abstract features of multiple agents by mimicking the connection mechanism of different projection areas in the human brain.
[0069] Specifically, taking any one of the multiple intelligent agents as an example, the process of obtaining the second abstract feature of the first intelligent agent is explained as follows:
[0070] A. Initialize the second abstract feature of the first agent. Specifically, the second abstract feature of the first agent can be randomly determined as the second initial abstract feature of the first agent. For example, the second initial abstract feature is {0, 1, 1}.
[0071] B. During the interaction between the first intelligent agent and the environment, within the current interaction period (episode), it is determined in real time whether the second update condition is met at the current moment; if the second update condition is not met at the current moment, then step C is executed; if the second update condition is met at the current moment, then step D is executed.
[0072] The second update condition is the condition for updating the second abstract feature. It can be a time condition or optional. The second update condition is determined according to the second preset period. For example, the second update condition is: the current time is the start time of the second preset period, and the current time is not the start time of the current interaction period.
[0073] Optionally, the second preset period of the second update condition is an integer multiple of the first preset period for satisfying the first update condition. For example, if the time when the first update condition is satisfied is 7, then the time when the second update condition is satisfied can be 14, 21, 28, etc. It should be noted that the second preset period is an integer multiple of the first preset period, meaning that when the second update condition is satisfied, multiple hidden Gaussian distributions can be generated. For example, if the second preset period is 14 and the first preset period is 7, then at the current time 14, two hidden Gaussian distributions can be generated. The unit of the period can be microseconds, milliseconds, etc.
[0074] C. Use the second initial abstract feature as the second abstract feature of the first agent, and collect the time-varying comparison sequence of the first agent into the time-varying comparison sequence set.
[0075] D. Input the first temporal feature into the encoder network to obtain the hidden layer Gaussian distribution.
[0076] E. Collect the hidden Gaussian distributions into a set of hidden Gaussian distributions. Input the elements in the set of hidden Gaussian distributions into a gated recurrent unit to obtain the second temporal features of the first agent.
[0077] F. Input the second temporal features into the graph attention mechanism network to generate the value distribution.
[0078] G. Use reinforcement learning methods to sample the value distribution to obtain the second abstract feature of the first agent.
[0079] S103: Select a matching first Q network for multiple agents based on the first abstract feature; wherein, agents with the same matching first Q network share parameters.
[0080] Optionally, network pools can be pre-established and trained incrementally. These pools include a first Q-network pool and a second Q-network pool. The first Q-network pool contains multiple Q-networks suitable for first abstract features (i.e., low-level abstract concepts); the second Q-network pool contains multiple Q-networks suitable for second abstract features (i.e., high-level abstract concepts). The first Q-network consists of an encoder network that generates the first abstract features and a decoder network that trains the first abstract features. The parameters of the first Q-network can be obtained by acquiring the encoder network parameters and the decoder network parameters.
[0081] In a multi-agent system, each agent selects a corresponding Q-network from a first Q-network pool based on its first abstract feature. Agents with the same selected Q-network belong to the same group. This process of grouping multiple agents is called the first grouping result.
[0082] For example, the first agent selects a first Q-network from the first Q-network pool that matches its first abstract feature, thus obtaining the first agent's first Q-network. The second agent selects a first Q-network from the first Q-network pool that matches its first abstract feature, thus obtaining the second agent's first Q-network. If the first agent's first Q-network is the same as the second agent's Q-network, then the first agent and the second agent belong to the same group. Taking a multi-agent system with 8 agents and a maximum sample size of 4 for the first abstract feature as an example, the grouping process of the multi-agent system is illustrated. The first abstract features of the 8 agents are, for example, {0, 1, 3, 0, 2, 1, 3, 2}. The first agent selects Q-network 0, the second agent selects Q-network 1, the third agent selects Q-network 3, the fourth agent selects Q-network 0, the fifth agent selects Q-network 2, the sixth agent selects Q-network 1, the seventh agent selects Q-network 3, and the eighth agent selects Q-network 2. Agents with the same Q-network number are selected to share Q-network parameters. Specifically, the first and fourth agents share Q-network parameters, the second and sixth agents share Q-network parameters, the third and seventh agents share Q-network parameters, and the fifth and eighth agents share Q-network parameters.
[0083] S104: Select matching second Q networks for multiple agents based on the second abstract features; wherein, agents with the same matching second Q network share parameters.
[0084] The second Q-network consists of a graph attention network that generates the second abstract features and a hybrid network that trains the second abstract features. The parameters of the second Q-network are obtained by acquiring the parameters of the graph attention network and the hybrid network. The second Q-network is initialized by initializing the maximum number of sampleable second abstract features of the first agent, which is equal to the number of acquired second Q-network parameters.
[0085] In a multi-agent system, each agent selects a second Q-network from a second Q-network pool that matches the second abstract feature of the first agent based on its second abstract feature, thus obtaining the second Q-network of the first agent. Agents with the same second Q-network among multiple agents are grouped together to obtain the second grouping result.
[0086] Optionally, before performing the above process, the relevant parameters and networks can be initialized. The initialized parameters and networks may include, for example, the maximum number of samples that can be taken for the first abstract feature, the maximum number of samples that can be taken for the second abstract feature, a first Q-network, and a second Q-network. Here, the maximum number of samples that can be taken for the first abstract feature is the number of first Q-networks obtained, and the maximum number of samples that can be taken for the second abstract feature is the number of second Q-networks obtained.
[0087] Taking multiple agents with first abstract features of {0, 0, 0, 0, 1, 1, 1, 1} and a maximum sample size of 2 as an example, the first, second, third, and fourth agents choose Q-network 0, therefore, they share the parameters of Q-network 0. The fifth, sixth, seventh, and eighth agents choose Q-network 1; the fifth, sixth, seventh, and eighth agents share the parameters of Q-network 1.
[0088] The multi-agent parameter sharing method provided in this application encodes time-varying comparison sequences of multiple agents to determine first abstract features of the agents. A graph attention network is then used to connect and combine these first abstract features to obtain second abstract features. Based on the first abstract features, a matching first Q-network is selected for each agent, with agents sharing the same first Q-network. Similarly, based on the second abstract features, a matching second Q-network is selected for each agent, with agents sharing the same second Q-network. Firstly, this method provides diverse strategy choices for multiple agents, enabling more accurate identification of differences between agents. Secondly, the Q-network is not fixed during parameter sharing; agents can choose appropriate partners based on environmental changes. Furthermore, the maximum number of sampleable types for the first and second abstract features is much smaller than the number of agents. Therefore, the parameter sharing mechanism proposed in this application is less affected by an increase in the number of agents, thus improving the reliability of the Q-network and consequently enhancing the cooperation among multiple agents. Furthermore, in this method, the second abstract feature is obtained by connecting and combining the first abstract feature. That is, there is a correlation between the first and second abstract features. Therefore, there is also a collaborative relationship between the groups obtained based on the first and second abstract features, which improves the collaborative ability of multi-agents and thus increases the upper limit of the overall system's capabilities, enabling the system to handle more complex tasks. Moreover, this method is not limited by the complexity of the experimental scenario when sharing parameters, thus exhibiting better dynamism.
[0089] In one embodiment, before performing step S101 above, the method further includes:
[0090] S105. Determine whether the preset network training trigger conditions are met; if the network training conditions are met, proceed to step S106; if not, proceed to step S101.
[0091] Meeting the network training conditions allows for multiple preset cycles of complete interaction between the agent and the environment. The preset cycles can be 7, 14, 20, etc.
[0092] S106. Train and update the neural network.
[0093] In one embodiment, after step S104, the method further includes:
[0094] S107. Determine the preset loop termination condition; if it is not met, return to step S101 above; if it is met, return to step S105 above.
[0095] The process of a multi-agent parameter sharing method provided in this application embodiment will be further explained below with reference to the accompanying drawings and examples.
[0096] Figure 3 This is a flowchart of an example of a multi-agent parameter sharing method provided in an embodiment of this application. See also... Figure 3 The method includes the following steps:
[0097] S201: Initialize the maximum number of samples for the first abstract feature, the maximum number of samples for the second abstract feature, the first update condition, the second update condition, the encoder network, the decoder network, the graph attention network, and the hybrid network.
[0098] For example, the maximum number of samples that can be taken for the first abstract feature, k, is 4; the maximum number of samples that can be taken for the second abstract feature, m, is 2; the first update condition, U, is 7; and the second update condition, V, is 17. It should be noted that k is an integer multiple of m, and V is an integer multiple of U.
[0099] The encoder and decoder networks are used to generate the first Q-network parameters, while the graph attention network and hybrid network are used to generate the second Q-network parameters.
[0100] S202: Determine if the algorithm has reached the maximum number of execution steps. If not, execute S203; otherwise, the program stops.
[0101] For example, the maximum number of execution steps for the algorithm can be set to 2 trillion steps, 5 trillion steps, etc. The program terminates when the algorithm reaches the maximum number of execution steps.
[0102] S203: Determine whether the preset network training trigger conditions are met. If not, execute S204; if yes, execute S212.
[0103] For example, the preset network training trigger condition is 32, which means that the agent interacts with the environment for 32 complete cycles.
[0104] S204: Initialize the complete interaction cycle between the agent and the environment, initialize the first abstract feature, and initialize the second abstract feature.
[0105] When initializing the interaction cycle, the first initial abstract feature and the second initial abstract feature of the first agent are randomly determined. The initial observation value and the initial global state value of the environment for each agent are then obtained and placed into the experience replay cache. For example, with 8 agents, the first initial abstract features of the 8 agents are {0, 1, 3, 0, 2, 1, 3, 2}, and the second initial abstract features of the 8 agents are {0, 0, 0, 1, 0, 1, 1, 1}.
[0106] S205: Determine whether the first update condition is met. If it is met, proceed to S207; otherwise, proceed to S206.
[0107] The first update condition is determined based on a first preset period. The first update condition is satisfied when the interaction period is the first preset period. For example, if the first preset period is 7, then the first update condition is satisfied when the interaction period is 7. The first update condition is not satisfied when the interaction period is not a multiple of 7; that is, the first update condition is satisfied even when the interaction period is an integer multiple of the first preset period.
[0108] S206: Determine whether the second update condition is met. If it is met, then execute S209.
[0109] The second update condition is determined based on a second preset period. The second update condition is satisfied when the interaction period is the second preset period, which is an integer multiple of the first preset period. For example, if the first preset period is 7, and the second preset period is twice the first preset period (i.e., 14), then the second update condition is satisfied when the interaction period reaches 14. If the interaction period is not a multiple of 14, then the second update condition is not satisfied.
[0110] S207: Update the first abstract feature.
[0111] The first agent inputs elements from the time-varying comparison sequence set into a gated recurrent unit to obtain the first temporal feature corresponding to the time-varying comparison sequence. The first temporal feature is then input into an encoder network to obtain a hidden Gaussian distribution. Sampling of this hidden Gaussian distribution yields the first abstract feature of the first agent. In other words, the interaction period starts from 0. When the interaction period satisfies the first update condition 7, the first agent inputs elements from the time-varying comparison sequence set collected over these 7 time points into the gated recurrent unit to obtain the first temporal feature hu corresponding to the time-varying comparison sequence. Then, the first temporal feature hu is input into the encoder network to obtain a hidden Gaussian distribution z. This hidden Gaussian distribution z is placed into a set Ze, and sampling of z yields the first abstract feature of the first agent. For example, when the interaction period satisfies the first update condition 7, the updated first abstract feature of the first agent is {0, 1, 1, 3, 2, 2, 3, 0}.
[0112] S208: Select the first Q network from the first Q network pool that matches the first abstract feature of the first agent to obtain the first Q network of the first agent.
[0113] Taking 8 agents as an example, when the interaction period is less than the first preset period of 7, the first abstract feature is not updated. At this time, the agents need to be grouped using the first initial abstract feature as {0, 1, 3, 0, 2, 1, 3, 2}, and the corresponding Q network is selected. Agents in the same group can share Q network parameters. That is, the first agent and the fourth agent are in the same group and can share Q network parameters; the second agent and the sixth agent are in the same group and can share Q network parameters; the third agent and the seventh agent are in the same group and can share Q network parameters; the fifth agent and the eighth agent are in the same group and can share Q network parameters.
[0114] Optionally, when the interaction period equals the first preset period, the first abstract feature has already been updated. In this case, the updated first abstract feature needs to be grouped, the corresponding Q-network selected, and the agents in the same group share the Q-network parameters. The time-varying comparison sequence set is then cleared. It should be noted that when the interaction period is an integer multiple of the first preset period, the time-varying comparison sequence is re-collected to obtain the updated first abstract feature. Subsequently, the updated first abstract feature is used for grouping, the corresponding Q-network selected, and the agents in the same group share the Q-network parameters.
[0115] Optionally, when the interaction period is not an integer multiple of the first preset period, the first abstract feature is the first abstract feature from the last update.
[0116] S209: Update the second abstract feature.
[0117] The first agent inputs the first temporal feature into the encoder network to obtain the hidden Gaussian distribution. This hidden Gaussian distribution is then collected into a set of hidden Gaussian distributions. Elements from this set are input into a gated recurrent unit to obtain the second temporal feature of the first agent. This second temporal feature is then input into a graph attention mechanism network to generate a value distribution. Reinforcement learning is used to sample the value distribution, resulting in the second abstract feature of the first agent. In other words, when the second update condition is met, the collected set of hidden Gaussian distributions Ze is input into the gated recurrent unit to obtain the second temporal feature hv of the first agent. This second temporal feature is then input into the graph attention mechanism network to further extract the second temporal feature hv, generating a value distribution. Reinforcement learning is used to sample the value distribution, resulting in the second abstract feature of the first agent. For example, when the interaction period meets the second preset period 14, the updated first abstract feature of the first agent is {0, 0, 0, 0, 1, 1, 1, 1}.
[0118] It should be noted that the first update condition is also met when the second update condition is met; that is, the first abstract feature is updated at the same time when the second abstract feature is updated.
[0119] S210: Select a second Q network from the second Q network pool that matches the second abstract feature of the first agent to obtain the second Q network of the first agent.
[0120] Taking 8 agents as an example, when the interaction period is less than the second preset period of 14, the second abstract feature is not updated. At this time, the second initial abstract feature is used to group the agents into groups of {0, 0, 0, 1, 0, 1, 1, 1}, and the corresponding Q network is selected. Agents in the same group share Q network parameters. That is, the first, second, third, and fifth agents are in the same group and can share Q network parameters; the fourth, sixth, seventh, and eighth agents are in the same group and can share Q network parameters.
[0121] Optionally, when the interaction period equals the second preset period, the second abstract feature has already been updated. In this case, the updated second abstract feature needs to be grouped, the corresponding Q-network selected, and agents in the same group share Q-network parameters. The hidden layer Gaussian distribution set is then cleared. It should be noted that when the interaction period is an integer multiple of the second preset period, the updated second abstract feature is obtained based on the newly collected hidden layer Gaussian distribution set. Subsequent grouping is based on the updated second abstract feature, the corresponding Q-network selected, and agents in the same group share Q-network parameters.
[0122] Figure 4This is a schematic diagram of the structure of an example of a multi-agent parameter sharing method provided in an embodiment of this application, as shown below. Figure 4 As shown, Figure 4 Figure (a) illustrates the process of obtaining the first abstract feature. The first agent inputs elements from the time-varying comparison sequence set into a gated recurrent unit (GRU) to obtain the first temporal feature corresponding to the time-varying comparison sequence. The first temporal feature is then input into an encoder network to obtain a hidden Gaussian distribution. The hidden Gaussian distribution is sampled to obtain the first abstract feature of the first agent. Figure 4 Figure (b) in the diagram illustrates the process of obtaining the second abstract feature. The first agent inputs the first temporal feature into the encoder network to obtain the hidden Gaussian distribution. Then, the hidden Gaussian distribution is collected into a set of hidden Gaussian distributions. The elements in the set of hidden Gaussian distributions are input into a gated recurrent unit (GRU) to obtain the second temporal feature of the first agent. The second temporal feature is then input into a graph attention mechanism network (e.g., k-head) to generate a value distribution. The value distribution is sampled using a reinforcement learning method to obtain the second abstract feature of the first agent. Figure 4 Figure (c) in the figure illustrates parameter sharing, where Q-network parameters are shared through the Q-network of the first abstract feature and the Q-network of the second abstract feature.
[0123] Optionally, when the interaction period is not an integer multiple of the second preset period, the second abstract feature is the second abstract feature from the last update.
[0124] S211: Obtain the next observation value and the global environmental state value.
[0125] The agent can obtain the next observation and the next global state of the environment. It should be noted that the agent will obtain new observations and new global state of the environment in each interaction cycle.
[0126] It should be noted that within an interaction cycle, the multi-agent system collects the initial observations, the initial global state of the environment, the first abstract feature, the second abstract feature, the next observation, the next global state of the environment, the action value of each agent, and the global reward value of each action, and puts them into a set, namely the interaction trajectory data. When the environment terminates, this set is put into the experience replay cache for subsequent network training.
[0127] S212: Determine whether the multi-agent is completely dead or the environment has terminated. If not, return to S205; if yes, return to S203.
[0128] S213: Train and update the network, then return to S202.
[0129] The multi-agent system retrieves interaction trajectory data collected from multiple interaction cycles from the experience replay cache, and trains and updates the network based on the collected interaction data.
[0130] 1) Calculate the intrinsic reward of the sampling results.
[0131] The intrinsic reward of the sampling results can be calculated using formula (1):
[0132]
[0133] Cu i This represents the number of different first abstract features corresponding to the second abstract feature sampled by the agent, where i represents the second abstract feature sampled by the agent. Cv i K represents the number of second abstract features with sampling result i for the agent, and K is the maximum number of first abstract features that can be sampled.
[0134] Taking the first abstract feature as {1, 1, 2, 3, 2, 2, 3, 0} and the second abstract feature as {0, 1, 1, 0, 1, 0, 1, 1} as an example, when the agent's second abstract feature is 0, the agent's first abstract feature is {1, 3, 2}, and the number of different first abstract features is 3, i.e., Cu. 0 =3; When the agent's second abstract feature is 1, the agent's first abstract feature is {1, 2, 3, 3, 0}, and the number of different first abstract features is 4, i.e., Cu 1 =4; the quantity when the second abstract feature is 0 is 3, i.e., Cv 0 =3, the quantity when the second abstract feature is 1 is 5, that is, Cv 1 =5, then according to formula (1), the intrinsic reward ri of the sampling result can be calculated as max{3,4}-[3-4+5-4]=4.
[0135] By calculating the intrinsic reward of the sampling results, the cooperation among multiple agents is enhanced.
[0136] 2) Using reinforcement learning methods, train the Q-network of the first abstract feature using the intrinsic reward and the collected global reward.
[0137] The Q-network with the first abstract feature is
[0138] 3) Using reinforcement learning methods, train the Q-network of the second abstract feature using the intrinsic reward and the collected global reward.
[0139] The Q-network for the second abstract feature is
[0140] 4) Use reinforcement learning methods to train the policy network of the multi-agent system using the collected global rewards.
[0141] The agent's policy network is
[0142] 5) Determine if the target network parameters need to be updated. If not, skip this step; otherwise, update them.
[0143] Figure 5 This is a code diagram illustrating an example of a multi-agent parameter sharing method provided in an embodiment of this application. For example... Figure 5 As shown.
[0144] The multi-agent parameter sharing method provided in this application encodes time-varying comparison sequences of multiple agents to determine first abstract features of the agents. A graph attention network is then used to connect and combine these first abstract features to obtain second abstract features. Based on the first abstract features, a matching first Q-network is selected for each agent, with agents sharing the same first Q-network. Similarly, based on the second abstract features, a matching second Q-network is selected for each agent, with agents sharing the same second Q-network. Firstly, this method provides diverse policy choices for multiple agents, enabling more accurate identification of differences between agents. Secondly, the Q-network is not fixed during parameter sharing; agents can choose appropriate partners based on environmental changes. Furthermore, the maximum number of sampleable types for the first and second abstract features is much smaller than the number of agents. Therefore, the parameter sharing mechanism proposed in this application is less affected by an increase in the number of agents, thereby improving the reliability of the policy network and ultimately enhancing the cooperation among multiple agents. Furthermore, in this method, the second abstract feature is obtained by connecting and combining the first abstract feature. That is, there is a correlation between the first and second abstract features. Therefore, there is also a collaborative relationship between the groups obtained based on the first and second abstract features, which improves the collaborative ability of multi-agents and thus increases the upper limit of the overall system's capabilities, enabling the system to handle more complex tasks. Moreover, this method is not limited by the complexity of the experimental scenario when sharing parameters, thus exhibiting better dynamism.
[0145] Figure 6 A schematic diagram of a multi-agent parameter sharing device provided in this application is shown. The device 600 includes:
[0146] The first abstract feature module 601 is used to encode the time-varying comparison sequence of multiple agents and determine the first abstract features of the multiple agents.
[0147] Second abstract feature module 602: used to connect and combine the first abstract features of multiple agents using a graph attention mechanism network to obtain the second abstract features of multiple agents; the second abstract features are higher-level abstract features compared to the first abstract features.
[0148] Parameter sharing module 603: used to select a matching first Q network for multiple agents based on the first abstract feature; wherein, agents with the same matching first Q network share parameters.
[0149] Parameter sharing module 603: is also used to select matching second Q networks for multiple agents based on the second abstract features; wherein, agents with the same matching second Q network share parameters.
[0150] In some embodiments, the first abstract feature module 601 is further configured to, when the first agent is in an interactive state with the environment, collect the time-varying comparison sequence of the first agent into a time-varying comparison sequence set if the first update condition is not met at the current time; the first agent is any one of multiple agents; if the first update condition is met at the current time, encode the elements in the time-varying comparison sequence set to obtain the first abstract feature of the first agent.
[0151] In some embodiments, the first abstract feature module 601 is further configured to input elements in the time-varying comparison sequence set into a gated recurrent unit to obtain a first temporal feature corresponding to the time-varying comparison sequence; input the first temporal feature into an encoder network to obtain a hidden layer Gaussian distribution; sample the hidden layer Gaussian distribution to obtain a first abstract feature of the first agent; and clear the time-varying comparison sequence set.
[0152] In some embodiments, the second abstract feature module 602 is further used to, when the first agent is in an interactive state with the environment, if the second update condition is met at the current moment, use a graph attention mechanism network to connect and combine the first abstract features of multiple agents to obtain the second abstract features of the first agent.
[0153] In some embodiments, after the second abstract feature module 602 inputs the first temporal features into the encoder network to obtain the hidden Gaussian distribution, the method further includes: collecting the hidden Gaussian distribution into a set of hidden Gaussian distributions; using a graph attention mechanism network to connect and combine the first abstract features of multiple agents to obtain the second abstract features of the first agent, including: inputting the elements in the set of hidden Gaussian distributions into a gated recurrent unit to obtain the second temporal features of the first agent; inputting the second temporal features into a graph attention mechanism network to generate a value distribution; sampling the value distribution using a reinforcement learning method to obtain the second abstract features of the first agent; and clearing the set of hidden Gaussian distributions.
[0154] In some embodiments, the parameter sharing module 603 is further configured to select a first Q network that matches the first abstract feature of the first agent from the first Q network pool to obtain the first Q network of the first agent; the first agent is any one of a plurality of agents.
[0155] In some embodiments, the parameter sharing module 603 is further configured to select a second Q network from the second Q network pool that matches the second abstract feature of the first agent, thereby obtaining the second Q network of the first agent; the first agent is any one of a plurality of agents.
[0156] In some embodiments, the first abstract feature module 601 is further configured to determine whether a preset network training trigger condition is met; if the network training trigger condition is met, then the neural network is trained and updated; if the network training trigger condition is not met, then the following steps are performed: encoding the time-varying comparison sequence of multiple agents to determine the first abstract features of multiple agents; using the second abstract features of multiple agents as the difference discrimination criterion to group multiple agents, and after obtaining the second grouping result, the method further includes: if the preset loop termination condition is not met, then returning to the execution step: encoding the time-varying comparison sequence of multiple agents to determine the first abstract features of multiple agents; if the preset loop termination condition is met, then returning to the execution step: determining whether the preset network training trigger condition is met.
[0157] The specific manner in which the device 600 executes the multi-agent parameter sharing method and the beneficial effects thereof can be found in the relevant descriptions in the method embodiments, and will not be repeated here.
[0158] Figure 7 This is a schematic diagram of the structure of a multi-agent system provided in an embodiment of this application. Figure 7 As shown, the multi-agent system 700 includes: a processor 710, a memory 720, and a computer program 721 stored in the memory 720 and executable on the processor 710. When the processor 710 executes the computer program 721, it implements the steps in the multi-agent parameter sharing method in the above embodiments.
[0159] Those skilled in the art will understand that Figure 7 This is merely an example of a multi-agent system 700 and does not constitute a limitation on the multi-agent system 700. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.
[0160] The processor 710 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.
[0161] In some embodiments, memory 720 may be an internal storage unit of the multi-agent system 700, such as a hard disk or memory of the multi-agent system 700. In other embodiments, memory 720 may be an external storage device of the multi-agent system 700, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the multi-agent system 700. Furthermore, memory 720 may include both internal storage units and external storage devices of the multi-agent system 700. Memory 720 is used to store the operating system, applications, boot loader, data, and other programs. Memory 720 may also be used to temporarily store data that has been output or will be output.
[0162] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the various method embodiments described above.
[0163] This application provides a computer program product that, when run on a computer, causes the computer to perform the steps described in the various method embodiments above.
[0164] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above method embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate form. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographing device / terminal device, a recording medium, a computer memory, ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage devices. The computer-readable storage medium mentioned in this application can be a non-volatile storage medium; in other words, it can be a non-transient storage medium.
[0165] It should be understood that all or part of the steps of the above embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented in whole or in part as a computer program product. The computer program product includes one or more computer instructions. The computer instructions can be stored in the above-described computer-readable storage medium.
[0166] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0167] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0168] In the embodiments provided in this application, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0169] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0170] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A method for sharing parameters among multiple agents, characterized in that, Applied to a multi-agent system, the method includes: Encode the time-varying comparison sequence of the plurality of agents to determine the first abstract feature of the plurality of agents, wherein the time-varying comparison sequence is used to indicate the difference in abstract features among the plurality of agents; A graph attention mechanism network is used to connect and combine the first abstract features of the multiple agents to obtain the second abstract features of the multiple agents; the second abstract features are higher-level abstract features compared to the first abstract features. Based on the first abstract feature, a matching first Q network is selected for each of the plurality of agents; wherein, agents with the same matching first Q network share parameters. Based on the second abstract feature, a matching second Q network is selected for each of the plurality of agents; wherein, agents with the same matching second Q network share parameters. The process of encoding the time-varying comparison sequences of the multiple agents to determine the first abstract feature of the multiple agents includes: When the first agent is in an interactive state with the environment, if the first update condition is met at the current moment, the elements in the time-varying comparison sequence set are input into the gated recurrent unit to obtain the first temporal feature corresponding to the time-varying comparison sequence; the first temporal feature is input into the encoder network to obtain the hidden layer Gaussian distribution; the hidden layer Gaussian distribution is sampled to obtain the first abstract feature of the first agent; the time-varying comparison sequence set contains the time-varying comparison sequence of the first agent; the first agent is any one of the plurality of agents; The method of using a graph attention mechanism network to connect and combine the first abstract features of the multiple agents to obtain the second abstract features of the multiple agents includes: The hidden layer Gaussian distributions are collected into a set of hidden layer Gaussian distributions; The elements in the set of hidden layer Gaussian distributions are input into the gated recurrent unit to obtain the second temporal features of the first agent; The second temporal feature is input into the graph attention mechanism network to generate a value distribution; The value distribution is sampled using reinforcement learning methods to obtain the second abstract features of the first agent.
2. The method as described in claim 1, characterized in that, The process of encoding the time-varying comparison sequences of the multiple agents to determine the first abstract feature of the multiple agents includes: When the first agent is in an interactive state with the environment, if the first update condition is not met at the current moment, the time-varying comparison sequence of the first agent is collected into the time-varying comparison sequence set.
3. The method according to claim 2, characterized in that, After obtaining the first abstract feature of the first intelligent agent, the method further includes: Clear the set of time-varying comparison sequences.
4. The method according to claim 3, characterized in that, The method of using a graph attention mechanism network to connect and combine the first abstract features of the multiple agents to obtain the second abstract features of the multiple agents includes: When the first agent is in an interactive state with the environment, if the second update condition is met at the current moment, the first abstract features of the multiple agents are connected and combined using a graph attention mechanism network to obtain the second abstract features of the first agent.
5. The method according to claim 4, characterized in that, The first update condition is determined based on a first preset period, and the second update condition is determined based on a second preset period, wherein the second preset period is an integer multiple of the first preset period; After obtaining the second abstract feature of the first agent, the method further includes: Clear the set of hidden Gaussian distributions.
6. The method according to claim 1, characterized in that, The step of selecting a matching first Q-network for each of the plurality of agents based on the first abstract feature includes: A first Q network matching the first abstract feature of the first agent is selected from the first Q network pool to obtain the first Q network of the first agent; the first agent is any one of the plurality of agents.
7. The method according to claim 1, characterized in that, The step of selecting a matching second Q-network for each of the plurality of agents based on the second abstract feature includes: In the second Q-network pool, a second Q-network that matches the second abstract feature of the first agent is selected to obtain the second Q-network of the first agent; the first agent is any one of the plurality of agents.
8. The method according to any one of claims 1 to 7, characterized in that, Before encoding the time-varying comparison sequences of the plurality of agents to determine the first abstract features of the plurality of agents, the method further includes: Determine whether the preset network training trigger conditions are met; If the network training triggering condition is met, then the neural network is trained and updated; If the network training triggering condition is not met, then the following steps are performed: Encode the time-varying contrast sequence of the multiple agents to determine the first abstract feature of the multiple agents; After grouping the multiple agents using the second abstract features as the difference discrimination criterion to obtain the second grouping result, the method further includes: If the preset loop termination condition is not met, return to the execution step: encode the time-varying comparison sequence of the multiple agents to determine the first abstract feature of the multiple agents; If the preset loop termination condition is met, return to the execution step: determine whether the preset network training trigger condition is met.
9. A multi-agent system, characterized in that, The multi-agent system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the method as described in any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 8.