A multi-agent reinforcement learning method based on a hierarchical topology of zero-order optimization
By introducing a hierarchical topology and a residual zero-order gradient estimator into the multi-agent system, the problems of high communication burden and low efficiency in existing multi-agent reinforcement learning are solved, and efficient policy training and global information acquisition are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2024-02-22
- Publication Date
- 2026-07-31
AI Technical Summary
Existing multi-agent reinforcement learning methods suffer from high communication burden and low efficiency, especially in fully connected, centralized, and distributed topologies where they suffer from bandwidth, dimensionality explosion, and communication delay issues.
A hierarchical topology is adopted, dividing agents into high-level and low-level categories. A three-stage communication structure is designed, consisting of intra-group aggregation, inter-group communication, and intra-group sharing. The residual zero-order gradient estimator is used to update the policy network parameters, reducing the communication requirements for global information acquisition.
It improved communication efficiency, reduced communication pressure, enhanced convergence performance and training results, and reduced the number of communications.
Smart Images
Figure CN118014000B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of multi-agent reinforcement learning, and mainly relates to a multi-agent reinforcement learning method based on a hierarchical topology with zero-order optimization. Background Technology
[0002] Resource allocation has wide applications in multi-agent systems, such as information resource scheduling and network communication resource scheduling and optimization among agents. However, as the number of agents increases, the dimensionality of the joint state and action space grows exponentially, leading to a dimensionality explosion problem when directly applying existing reinforcement learning algorithms to multi-agent systems. Applying single-agent reinforcement learning algorithms to each agent in a multi-agent system results in poor convergence and unsatisfactory training effects. Therefore, extending reinforcement learning to multi-agent systems presents significant research challenges. Currently, designing corresponding multi-agent reinforcement learning methods based on the network topology of the multi-agent system is one solution for multi-agent reinforcement learning.
[0003] Zero-order optimization is a current optimization method for solving multi-agent reinforcement learning. Depending on the topology, multi-agent reinforcement learning methods based on zero-order optimization can be divided into three types: fully connected, centralized, and distributed. In a fully connected topology, each agent in the system exchanges zero-order gradient information with all other agents, thereby collecting global zero-order gradient information and using this global information to learn its own policy to achieve the optimal policy. In a centralized topology, a central agent or controller gathers the zero-order gradient information of all other agents, processes it, and then shares it with all agents to obtain global information for policy learning. In a distributed topology, agents form a communication graph, and each agent only exchanges zero-order gradient information with its neighboring agents during each communication.
[0004] However, the aforementioned zero-order optimization multi-agent reinforcement learning methods all have corresponding problems. In a fully connected topology, since communication with all other agents is required, the communication capabilities of the agents are highly demanding. As the number of agents increases, bandwidth issues arise, resulting in significant communication overhead. In a centralized topology, all zero-order gradient information is concentrated at the central point, leading to dimensionality explosion and capacity bottlenecks at the central node as the scale increases. In a distributed topology, since each agent only communicates with its neighbors, the agent needs to communicate with its neighbors multiple times to obtain global zero-order gradient information, resulting in significant overall communication overhead. Furthermore, the topology has poor ability to cope with communication delays.
[0005] To address the issues raised above, hierarchical topologies have become a research hotspot in multi-agent reinforcement learning in recent years. One method for implementing hierarchical topologies is to divide agents into high-level and low-level classes and design different communication tasks for each class. Using hierarchical topologies for zero-order gradient information exchange requires only a few high-level agents to possess strong communication capabilities, and by distributing global information across multiple agents, it achieves better communication efficiency and robustness. Summary of the Invention
[0006] The purpose of this invention is to address the shortcomings of existing topologies by providing a multi-agent reinforcement learning method based on zero-order optimization in a hierarchical topology, thus solving the problems of high communication burden and low efficiency in existing zero-order optimization multi-agent reinforcement learning methods under existing topologies.
[0007] To address the above problems, the present invention provides the following technical solution:
[0008] This invention first provides a multi-agent reinforcement learning method based on a hierarchical topology with zero-order optimization, comprising the following steps:
[0009] Step 1: Pre-group the agents in the environment. The total number of agents in the environment is N. After pre-grouping, all agents are divided into M groups. The agents in the m-th group are represented by G. m = {…,i,…} represents the group, where i represents the agent belonging to that group; the number of agents in the m-th group is denoted by C. m This means that each group consists of a group leader and several group members, and different groups exchange information through the group leader.
[0010] Step 2: Based on the multi-agent reinforcement learning problem, propose the objective function to be optimized, and apply the zero-order optimization method to obtain the corresponding local objective function estimate based on the parameters of the agent policy network;
[0011] Step 3: Based on the grouping in Step 1 and the local objective function estimate obtained in Step 2, construct the hierarchical topology method and perform information interaction between agents to complete the estimation of the global objective function;
[0012] Step 4: Based on the global objective function estimated in Step 3, apply the residual zero-order gradient estimator to estimate the gradient value of the objective function, and update the parameters of the agent policy network through the estimated gradient value of the objective function to complete one training of the agent policy network.
[0013] Step 5: Repeat steps 2-4 to iteratively train the agent policy network. When the number of training iterations reaches the preset value, end the training of the agent policy network and solve the objective function based on the trained agent policy network to achieve multi-agent reinforcement learning.
[0014] The present invention also provides a system for implementing the above-described multi-agent reinforcement learning method, comprising:
[0015] The agent grouping module is used to group agents in the environment;
[0016] The local objective function estimation module is used to propose the objective function to be optimized and apply the zero-order optimization method to obtain the corresponding local objective function estimate based on the parameters of the agent policy network.
[0017] The global objective function estimation module is used to construct a hierarchical topology method based on the grouping of agents by the agent grouping module and the local objective function estimation value obtained by the local objective function estimation module, and to carry out information interaction between agents, thereby completing the estimation of the global objective function.
[0018] The agent policy network training module is used to estimate the gradient value of the objective function by applying the residual zero-order gradient estimator, and update the parameters of the agent policy network by the estimated objective function gradient value, and iteratively train the agent policy network.
[0019] The objective function solving module is used to solve the objective function through the trained agent policy network, thereby realizing multi-agent reinforcement learning.
[0020] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0021] (1) In the modeling of multi-agent reinforcement learning problems, this invention does not require each agent to obtain global information. Each agent can make decisions based on its own observations. The policy training and global information acquisition are carried out in a hierarchical topology.
[0022] (2) Based on the properties of the zero-order optimization method and the hierarchical topology structure, this invention proposes a three-segment hierarchical topology communication structure. Specifically, this invention divides agents into high-level agents and low-level agents, and designs a three-segment hierarchical topology communication structure based on zero-order optimization, which includes intra-group aggregation, inter-group communication, and intra-group sharing. The global objective function is estimated, and the agent policy network is updated using the residual zero-order gradient estimator.
[0023] (3) The advantage of this invention compared to other topologies is that it has multiple advanced agents, requiring only that the advanced agents have strong communication capabilities, while also sharing all the communication pressure and ensuring the communication efficiency of the method. At the same time, the method can obtain the global objective function estimate in one communication, which also has advantages in convergence performance. Attached Figure Description
[0024] Figure 1 This is a schematic diagram illustrating the principle of the method of the present invention.
[0025] Figure 2 This is a schematic diagram illustrating the principle of the hierarchical topological communication structure in the method of the present invention.
[0026] Figure 3 This is a schematic diagram of the multi-agent connection topology in an embodiment of the method of the present invention.
[0027] Figure 4 This is a schematic diagram illustrating the change of the objective function with the training cycle in an embodiment of the method of the present invention.
[0028] Figure 5 This is a schematic diagram illustrating the change in the number of exchanges with the training cycle in an embodiment of the method of the present invention. Detailed Implementation
[0029] The embodiments of the method of the present invention will be described in detail below with reference to the accompanying drawings.
[0030] This invention proposes a multi-agent reinforcement learning method based on a hierarchical topology with zero-order optimization. First, all agents in the environment are pre-grouped and a hierarchical topology communication structure is constructed. Then, the agents interact with the environment to obtain local objective function information. After that, the local objective functions are aggregated and processed through the hierarchical topology communication structure. Finally, the agents calculate the zero-order gradient by using the residual zero-order gradient estimator to obtain the processed global objective function estimate information, and use the gradient ascent method to update the parameters of their respective policy networks.
[0031] In a specific embodiment of the present invention, in a multi-agent resource scheduling and allocation environment, the present invention proposes a multi-agent reinforcement learning method based on a hierarchical topology with zero-order optimization. The principle is as follows: Figure 1 As shown, the specific steps include:
[0032] Step 1: Pre-group agents in the resource allocation environment:
[0033] In a resource allocation environment, agents are divided into two categories: high-level agents and low-level agents. After grouping, each group contains one high-level agent. The total number of agents in the environment is N. After pre-grouping, all agents are divided into M groups. The agents in the m-th group are denoted by G. m= {…,i,…} represents the group, where i represents the agent belonging to that group. The number of agents in the m-th group is denoted by C. m According to the definition, we have:
[0034]
[0035] The diagram after grouping is as follows Figure 2 As shown.
[0036] Step 2: The agent interacts with the resource allocation environment to obtain local objective function information, which includes the following sub-steps:
[0037] Step 2.1: Establish a multi-agent reinforcement learning problem model under a resource allocation environment:
[0038] In a resource scheduling and allocation environment, the resource transfer relationships between agents are represented using an undirected connected graph. There are N agents in the environment. i Let S represent the set of neighbors of agent i. The joint state space of the environment is S, representing the amount of resources owned and demanded by all agents in the environment. The joint action space of the agents is A, representing the set of resource transfer actions for each agent. Taking a single agent as an example, the state space of agent i is S. i The state of each agent is denoted as s. i (t)=[m i (t),d i (t)] T , where m i (t) and d i (t) represents the amount of resources that agent i possesses and the amount of resources it requires at time t. The action space is A. i The action of each agent is denoted as a. ij (t)∈[0,1], representing the transmission of data from the i-th agent to its neighbor j∈N. i The percentage of resources available. The formula for updating agent i's local resources and demand is: and d i (t)=A i sin(ω i t+φ i )+ω i,t A i and φ i Follows a uniform distribution, ω i =2π, ω i,t It follows a Gaussian distribution. At each time t, agent i receives a reward r based on the amount of resources it possesses locally. i Specifically: when m i When (t)≥0, r i (t) = 0, when m iWhen (t) < 0, r i (t)=-m i (t) 2 The policy of agent i is represented as follows: Among them o i In this environment and s i They are equal. The policy network parameters of agent i are θ. i Depending on the initial environmental state s0, the goal of agent i is to maximize its cumulative reward, represented by the value function V:
[0039]
[0040] Where π = π i It is the joint policy of all agents, γ is the discount factor, T is the total time length in one environmental interaction, s0 is the initial state, and r i,t Let be the reward value obtained by agent i at time t. The objective function of agent i is expressed as: Where ρ0 is the initial state distribution function. The ultimate goal is to achieve a balance in the amount of resources available to all agents in the environment.
[0041]
[0042] Where parameters
[0043] Step 2.2: Establish the objective function based on zero-order optimization:
[0044] In the zero-order optimization method, the parameters of the agent policy network are perturbed by a smoothing parameter δ, which is used to obtain the local objective function estimate J over a training epoch. i (θ+δu,ξ), where ξ is the noise generated by the agent while exploring the environment. In a resource allocation environment, each agent allocates its resources to its neighboring agents according to the current policy network and receives a reward from the environment based on the amount of resources it currently possesses.
[0045] Each time the agent interacts with the environment, it obtains a local estimate of the objective function J. i (θ+δu,ξ).
[0046] Step 3: Consider the implementation of hierarchical topology and construct the hierarchical topology communication structure, which includes the following sub-steps:
[0047] Step 3.1: Aggregation of local objective functions within the group:
[0048] In each training epoch k, the advanced agent in each group aggregates the local objective function estimates J obtained in step 2.2 from the agents within its group. i(θ+δu,ξ), obtain the aggregate value
[0049] Step 3.2: Information exchange between groups:
[0050] The higher-level agents in each group will exchange aggregated values and then update their own group's aggregated value:
[0051]
[0052] Step 3.3: Information sharing within the group:
[0053] Each group's high-level agent processes the aggregated value obtained in step 3.2 after communication and shares the processed value with the agents within its group. Specifically:
[0054] Aggregate values Divide by the number of agents C in the group m That is, the estimated global objective function received by each agent in the group is...
[0055] in This represents the estimated global objective function value obtained by agent i in the k-th training cycle.
[0056] Step 4: Each agent updates the policy network parameters using the global objective function estimate obtained in Step 3.3, including the following sub-steps:
[0057] Step 4.1: Using the residual zero-order gradient estimator, calculate the zero-order gradient estimate for this training period using the global objective function estimate obtained in this training period and the global objective function estimate from the previous training period. The specific form of the residual zero-order gradient estimator is as follows:
[0058]
[0059] Step 4.2: Update the parameters of the policy network using the gradient ascent method, specifically in the following form:
[0060]
[0061] Where α is the learning rate. It is the global objective function estimate obtained by each agent in step 3.3.
[0062] Step 5: Using the hierarchical multi-agent reinforcement learning method based on zero-order optimization designed in Steps 1-4, perform k training cycles in the environment to complete the training of the agent policy network.
[0063] The present invention also provides a system for implementing the above-described multi-agent reinforcement learning method, comprising:
[0064] The agent grouping module is used to group agents in the environment;
[0065] The local objective function estimation module is used to propose the objective function to be optimized and apply the zero-order optimization method to obtain the corresponding local objective function estimate based on the parameters of the agent policy network.
[0066] The global objective function estimation module is used to construct a hierarchical topology method based on the grouping of agents by the agent grouping module and the local objective function estimation value obtained by the local objective function estimation module, and to carry out information interaction between agents, thereby completing the estimation of the global objective function.
[0067] The agent policy network training module is used to estimate the gradient value of the objective function by applying the residual zero-order gradient estimator, and update the parameters of the agent policy network by the estimated objective function gradient value, and iteratively train the agent policy network.
[0068] The objective function solving module is used to solve the objective function through the trained agent policy network, thereby realizing multi-agent reinforcement learning.
[0069] The effectiveness of the present invention will be verified below with reference to a specific implementation.
[0070] In a multi-agent resource allocation environment, the number of agents is set to N=16, such as Figure 3 As shown, through a connected topology graph, each agent will transfer a portion of its resources to its neighboring agents. During the training phase, the time interval for each training cycle is set to T = 25, the smoothing parameter to δ = 0.01, the learning rate to α = 0.0001, and the discount factor to γ = 0.75. Through steps 1-5 above, the agents are pre-grouped into 4 groups, and then the hierarchical multi-agent reinforcement learning method based on zero-order optimization of this invention and the existing distributed multi-agent reinforcement learning method based on zero-order optimization are executed respectively, and compared. The global objective function value, i.e., the cumulative reward value, is as follows: Figure 4 As shown in the figure, the blue and black lines represent the curves of the global objective function value changing with the number of training epochs under hierarchical and distributed topologies, respectively. The number of exchanges is as follows: Figure 5 As shown in the figure, the orange and blue lines represent the curves of the number of interactions as a function of the number of training epochs under hierarchical and distributed topologies, respectively. It can be seen that the hierarchical multi-agent reinforcement learning method based on zero-order optimization proposed in this invention can converge, and its convergence performance and interaction efficiency are superior to the corresponding distributed method.
[0071] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not limiting. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the scope of protection of the present invention.
Claims
1. A multi-agent reinforcement learning method based on a hierarchical topology with zero-order optimization, characterized in that, Includes the following steps: Step 1: Group the agents in the multi-agent system in the resource allocation scenario, where the total number of agents is... After pre-grouping, all agents are divided into Group, No. The intelligent agents in the group use It means that, among them Represents the agents belonging to this group; the first The number of agents in a group This means that each group consists of a group leader and several group members, and different groups exchange information through the group leader. Step 2: Define the global objective function to be optimized for the multi-agent reinforcement learning problem in the resource allocation scenario, and apply the zero-order optimization method to obtain the estimated value of the corresponding local objective function based on the parameters of the agent policy network; Step 3: Construct a hierarchical topology method based on the grouping in Step 1 and the local objective function estimate obtained in Step 2, and estimate the global objective function through information interaction between agents; Step 3 specifically includes the following sub-steps: Step 3.1: Based on the local objective function estimate of each agent obtained in Step 2 based on zero-order optimization, and based on the grouping in Step 1, aggregate the local objective functions within each group to obtain the... Group intelligent agent Aggregate value of each training cycle ;in, No. The local objective function for each training cycle. It is the number of training cycles; Step 3.2: Based on the aggregated objective function value of each group obtained in Step 3.1, and using a hierarchical communication topology, information is exchanged between groups, with the group leader as the communication partner, thereby updating the aggregated value of each group: ； Step 3.3: Based on the aggregated objective function values of each group obtained in Step 3.2, and using a hierarchical communication topology, information is shared within the group. The group leader shares the aggregated objective function values obtained after communication with the agents within the group. The agents thus obtain an estimate of the global objective function, denoted as... ; In step 3.3, the group leader shares the aggregated objective function value obtained after the exchange with the agents within the group. Specifically, this involves sharing the aggregated value... Divide by the number of agents in the group The estimated global objective function received by each agent in the group is , ; Step 4: Based on the global objective function estimated in Step 3, apply the residual zero-order gradient estimator to estimate the gradient value of the objective function, and update the parameters of the agent policy network through the estimated gradient value of the objective function to complete one training of the agent policy network. Step 5: Repeat steps 2-4 to iteratively train the agent policy network. When the number of training iterations reaches the preset value, end the training of the agent policy network and solve the global objective function based on the trained agent policy network to achieve multi-agent reinforcement learning.
2. The multi-agent reinforcement learning method according to claim 1, characterized in that, Step 2 specifically includes the following sub-steps: Step 2.1: Establish a multi-agent reinforcement learning problem model and define the state space of the environment. The action space, the agent's reward, the agent's policy, and the objective function value to be optimized are all considered. The final optimization objective is expressed in the following form: ； ； ； in, It is an intelligent agent Parameters of the policy network This indicates that the policy network parameters of the agent are... The global objective function value at that time. Represents intelligent agents The local objective function value, For intelligent agents The initial state space, It is a joint strategy of all agents. It is a discount factor. Represents intelligent agents exist The reward value obtained at any time, It is the total time length of a single environmental interaction; Step 2.2: Utilizing the properties and methods related to zero-order optimization, based on Step 2.1, establish an objective function based on zero-order optimization. The parameters of the agent policy network are determined according to the smoothing parameters. A perturbation is applied Used to obtain local objective function estimates ,in It is the noise generated by the intelligent agent when exploring the environment.
3. The multi-agent reinforcement learning method according to claim 1, characterized in that, In step 4, each agent updates the policy network parameters using the global objective function estimate obtained in step 3.3, including the following sub-steps: Step 4.1: Using the residual zero-order gradient estimator, calculate the zero-order gradient estimate for this training period using the global objective function estimate obtained in this training period and the global objective function estimate from the previous training period; Step 4.2: Update the parameters of the policy network using the gradient ascent method, specifically in the following form: ； in, It is the learning rate. It is the global objective function estimate obtained by each agent in step 3.
3.
4. The multi-agent reinforcement learning method according to claim 3, characterized in that, The specific form of the residual zero-order gradient estimator is as follows: ； in, For smoothing parameters, The estimated global objective function obtained in this training cycle This is the estimated global objective function value from the previous training cycle. It is the number of training cycles.
5. A system for implementing the multi-agent reinforcement learning method of claim 1, characterized in that, include: The agent grouping module is used to group agents in a multi-agent system in a resource allocation scenario. The local objective function estimation module is used to propose the objective function to be optimized based on the definition of the multi-agent reinforcement learning problem in the resource allocation scenario, and apply the zero-order optimization method to obtain the corresponding local objective function estimate based on the parameters of the agent policy network. The global objective function estimation module is used to construct a hierarchical topology method based on the grouping of agents by the agent grouping module and the local objective function estimation value obtained by the local objective function estimation module, and to carry out information interaction between agents, thereby completing the estimation of the global objective function. The agent policy network training module is used to estimate the gradient value of the objective function by applying the residual zero-order gradient estimator, and update the parameters of the agent policy network by the estimated objective function gradient value, and iteratively train the agent policy network. The objective function solving module is used to solve the global objective function through the trained agent policy network, thereby realizing multi-agent reinforcement learning.