A dynamic hierarchical multi-agent control method based on reinforcement learning

Abstract

The present invention discloses a dynamic hierarchical multi-agent control method based on reinforcement learning. First, the number of levels of the hierarchical structure is determined according to demand, the agents are divided into several subgroups, and representatives responsible for communication are selected in each subgroup. According to the determined reinforcement learning algorithm, a dynamic hierarchical multi-agent strategy optimization model is constructed, and the dynamic hierarchical multi-agent strategy optimization model is trained based on a reward function, and the trained dynamic hierarchical multi-agent strategy optimization model is used to generate a multi-agent control scheme, and information interaction and strategy update are performed according to environmental changes to achieve dynamic adjustment of the multi-agent. The scheme of the present invention dynamically adjusts the agent hierarchy, and performs hierarchical strategy generation and fusion according to the hierarchy. While ensuring the system collaborative decision-making and learning effect, it reduces the communication load, reduces the network bandwidth requirement, and improves the scalability and robustness of the system.

Description

Technical Field

[0001] The present invention belongs to the field of multi-agent control based on reinforcement learning, and specifically relates to a dynamic hierarchical multi-agent control method based on reinforcement learning. Background Art

[0002] In recent years, multi-agent systems (MAS) have been widely used in scenarios such as drone swarms, autonomous driving fleets, robot teams, and traffic management. For example, in drone swarms, MAS can be used to perform tasks such as search and rescue, post-disaster reconstruction, farmland monitoring, and border patrols. The core challenge of MAS is that it requires agents to make autonomous decisions and collaborate in dynamic, complex, and unpredictable environments. In this process, factors such as the number of agents, the complexity of tasks, and the uncertainty of the environment will greatly increase the complexity of the system.

[0003] The application of reinforcement learning (RL) in MAS has become a research hotspot because it allows agents to gradually learn the optimal strategy in a complex environment through trial and error. The application of RL in MAS faces two major challenges: first, the efficiency and convergence of the learning process; second, how to effectively collaborate and share information among multiple agents. In large-scale systems, the number of agents is huge, and learning complex multi-agent strategies requires a lot of computing resources and time, so improving learning efficiency is crucial; large-scale multi-agent systems require information sharing and collaborative decision-making. In this process, the communication burden is extremely high. Relying solely on global communication will lead to problems such as bandwidth overload and increased latency. Therefore, it is necessary to reduce the communication load while ensuring collaboration.

[0004] Previous studies have proposed centralized reinforcement learning, decentralized reinforcement learning, distributed reinforcement learning, and hierarchical reinforcement learning to solve these problems, but they all have their own limitations. The centralized reinforcement learning method concentrates the state and action information of all agents into a central controller or model for processing, which is prone to insufficient computing and memory resources in large-scale systems. In addition, the existence of a central controller means that the system has the risk of single point failure. Decentralized reinforcement learning allows each agent to independently learn its own strategy based on its own observations. For example, Kaiqing Zhang et al.'s article "Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents", which was included in the PMLR collection, proposed an Actor-Critic algorithm with function approximation suitable for large-scale multi-agent reinforcement learning problems to solve the scalability problem of traditional centralized reinforcement learning. However, decentralized reinforcement learning is prone to sub-optimization phenomena, making it difficult to effectively share and integrate information in complex tasks, resulting in low collaboration efficiency. Distributed RL methods organize agents into a distributed network structure, where agents can share information and collaborate through limited communication. However, it relies on communication between agents. In large-scale systems, as the number of agents increases, the communication overhead increases exponentially, resulting in bandwidth bottlenecks and latency problems. Hierarchical reinforcement learning decomposes tasks into subtasks at different levels, so that each level can focus on specific subtasks. For example, the article "Hierarchical multi-agent reinforcement learning for cooperative tasks with sparse rewards in continuous domain" published in the journal "Neural Computing & Applications" divides the architecture into two levels: high-level and low-level, which effectively solves the sparse reward problem while improving the scalability of multi-agent systems. However, most methods assume that the hierarchical structure is pre-set, lacks flexibility, and is difficult to adapt to dynamically changing environments. Moreover, in a multi-level structure, rewards and penalties need to be reasonably distributed to different levels, which also faces the challenges of communication and credit allocation.

[0005] In summary, the multi-agent control framework based on reinforcement learning has been widely used in unmanned swarms. By optimizing task allocation, path planning, and resource allocation, it has effectively improved the group's collaboration ability and execution efficiency. However, there is still a lot of room for improvement in the performance and scale of unmanned swarms. Future research needs to further focus on improving the adaptability and scalability of the multi-agent control framework to cope with larger-scale heterogeneous unmanned swarms and more complex and dynamic task environments. Summary of the invention

[0006] In view of the above problems, the object of the present invention is to provide a dynamic hierarchical multi-agent control method based on reinforcement learning.

[0007] The specific technical solution for achieving the purpose of the present invention is:

[0008] A dynamic hierarchical multi-agent control method based on reinforcement learning includes the following steps:

[0009] Step 1: Determine the number of levels of the hierarchy according to the requirements, divide the agents into several subgroups, and select representatives responsible for communication in each subgroup;

[0010] Step 2: According to the determined reinforcement learning algorithm, a dynamic hierarchical multi-agent strategy optimization model is constructed, and the dynamic hierarchical multi-agent strategy optimization model is trained based on the reward function;

[0011] Step 3: Use the trained dynamic hierarchical multi-agent strategy optimization model to generate a multi-agent control plan, and perform information interaction and strategy update according to environmental changes to achieve dynamic adjustment of the multi-agent.

[0012] Compared with the prior art, the present invention has the following beneficial effects:

[0013] (1) The solution of the present invention divides the intelligent agents into multiple levels and constructs a strategy generation network based on each level. At the same time, the intelligent agents in the level can also be dynamically adjusted according to needs and status. The dynamic construction and adjustment mechanism of the hierarchical structure is innovatively introduced, so that the system can adapt to changes in the environment and tasks.

[0014] (2) The solution of the present invention proposes a new reward decomposition and credit allocation mechanism, and provides a new solution to the problem of individual contribution evaluation in large-scale systems, which helps the system to achieve multi-level strategy integration and achieve the optimal solution for the entire system.

[0015] The present invention is further described below in conjunction with specific implementation modes. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1It is a flow chart of the dynamic hierarchical multi-agent control method based on reinforcement learning of the present invention.

[0017] Figure 2 Schematic diagram of a dynamic hierarchical multi-agent control framework based on reinforcement learning in an embodiment of the present invention.

[0018] Figure 3 This is an overall flow chart of applying a dynamic hierarchical multi-agent reinforcement learning framework to perform actual tasks in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] Example

[0020] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an kind" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0022] Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values ​​of the parts and steps set forth in these embodiments do not limit the scope of the present application. At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to the actual proportional relationship. The technology, method and equipment known to those of ordinary skill in the relevant field may not be discussed in detail, but in appropriate cases, the technology, method and equipment should be considered as a part of the authorization specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so that once a certain item is defined in an accompanying drawing, it does not need to be further discussed in subsequent drawings.

[0023] Combination Figures 1 to 3 , a dynamic hierarchical multi-agent control method based on reinforcement learning, comprising the following steps:

[0024] Step 1: Determine the number of levels of the hierarchy according to the requirements, divide the agents into several subgroups, and select representatives responsible for communication in each subgroup:

[0025] Based on factors such as the agent's geographical location, task similarity, or communication topology, the agents are dynamically divided into several subgroups (Clusters); within each subgroup, the agents are dynamically adjusted according to needs, such as subtasks, including the merging and splitting of subgroups:

[0026] In the agent collection middle, represents the kth subgroup, which contains several agents, among which Represents the serial numbers of all agents contained in the i-th subgroup;

[0027] When subgroups and A parameter value of the agent in exceeds the set threshold t, for example, the minimum similarity of the agent is higher than the specific threshold t, that is, When Merge into a new subgroup ;

[0028] When the agents in a subgroup need to be split into two or more subgroups for some reason, the split can be achieved by following the steps below: When a parameter value of an agent in exceeds a set threshold, the distance between agents or the difference in tasks are determined by clustering algorithm, for example . Then assign the subgroup The intelligent agent in

[0029] More specifically, in actual operation, during training or testing, if the environment (task area, communication conditions, task content, etc.) changes, the dynamic hierarchy adjustment machine will be triggered to dynamically merge small subgroups or split overly large subgroups as needed, adjust the leaders of subgroups, increase or decrease levels, etc., to achieve dynamic adjustment of the intelligent body.

[0030] When it is necessary to select a representative responsible for communication, based on the coordinates of each agent, the cluster center coordinates in each subgroup are determined based on the K-means++ clustering algorithm. The agent with the closest Euler distance to the cluster center is taken as the representative responsible for communication in the sub-group, that is, the initial sub-group representative.

[0031] Step 2: According to the determined reinforcement learning algorithm, a dynamic hierarchical multi-agent strategy optimization model is constructed, and the dynamic hierarchical multi-agent strategy optimization model is trained based on the reward function:

[0032] This scheme adopts the idea of ​​hierarchical reinforcement learning. Each level of the intelligent agent or group of intelligent agents builds a strategy network to generate hierarchical strategies based on its own local observations and limited communication with neighbors.

[0033] The dynamic hierarchical multi-agent strategy optimization model includes a global strategy network, a sub-group strategy network and a single agent strategy network;

[0034] Among them, the policy network at the single agent level can be relatively simple, built based on DQN, PPO or DNN algorithms, focusing on rapid response and real-time decision-making; the global level policy network and the sub-group level policy network are built using collaborative reinforcement learning algorithms to facilitate better integration of information.

[0035] The policy network gradient is updated as follows:

[0036]

[0037] in, are network parameters, is the learning rate, is the objective function.

[0038] Policy network at the agent level Based on its own local observations and limited communication with neighbors, it learns the optimal strategy, where the agents within the subgroup learn subgroup-level strategies by sharing information and collaborating to optimize the overall performance of the subgroup. The representatives of the subgroup (such as the subgroup leader) communicate and collaborate with each other to learn global strategies to ensure that the goals of the entire system are achieved.

[0039] Multi-level strategy fusion is divided into two parts: strategy feature extraction and feature fusion. Taking the strategy feature extraction at the individual level as an example, since the action space at the subgroup level may not directly overlap with the action space of the individual, we use a neural network layer. The strategy network at the single agent level takes the strategy probability distribution of the individual as input to obtain the strategy features of the individual:

[0040]

[0041] in, represents the strategic features of the i-th agent, FFN represents a layer of forward neural network, logits( ) represents the action probability distribution of the strategy network of the ith agent, Represents the environmental information observed at time i, including but not limited to the picture of the current environment, current time, light intensity, etc.; Represents the action of agent i at time i;

[0042] The subgroup level The policy network takes the policy probability distribution of the sub-group as input to obtain the policy characteristics of the sub-group;

[0043]

[0044]

[0045] in represents the action probability distribution of the policy network of subgroup c, Represents the feature weight of the i-th agent;

[0046] The global policy network Take the global strategy probability distribution as input to obtain the strategy characteristics of the sub-group;

[0047] Among them, the values ​​of the aggregated features in the subgroup-level policy network and the global-level policy network are:

[0048]

[0049] represents the feature weight of the c-th subgroup, They represent the aggregate features of all agents within the cth subgroup and the aggregate features of all internal subgroups in the global layer, respectively.

[0050] In addition, the simple weighted feature fusion can be changed to feature fusion using GNN. Finally, the individual policy network is The subgroup strategy network is , the global layer policy network is At each time step, policy networks at different levels obtain environmental observation information and feature information of low-level policy networks respectively, and then obtain action output through the policy network of this layer.

[0051] The global policy network, the subgroup policy network and the single agent policy network are trained based on the objective function and the reward function to achieve parameter update of the multi-level policy network;

[0052] In the multi-level policy gradient algorithm, the gradient of the global layer is related to the sub-group, and the gradient of the sub-group is related to the intelligent agent within the sub-group.

[0053] The global layer is updated as follows:

[0054]

[0055] Similarly, the subgroup layer is updated as follows:

[0056]

[0057] in is the strategy weight of the subgroup, is the policy weight at the global level. are the parameters of the global policy network, sub-group policy network and single agent network respectively. are the objective functions of the global policy network, subgroup policy network, and single agent network, respectively. In a multi-level framework, policy fusion helps integrate learning signals from individual, subgroup, and global layers, ensuring that policies at all levels are updated toward a common goal. The expression of is calculated by the reward function. Specifically, let The corresponding policy network The parameters (i.e. Represents the above , Represents the above ), is the value of the reward function at time t ( Represents the following ), the maximum time step of an action sequence is T, then the objective function is:

[0058]

[0059] The reward function is:

[0060]

[0061] in, express , are the objective functions of the global policy network, the subgroup policy network, and the single agent policy network, respectively. The corresponding policy network Parameters, express , that is, the value of the reward function in the global policy network, the sub-group policy network, and the single agent policy network, T represents the maximum time step of an action sequence;

[0062] N is the total number of agents, K is the total number of sub-populations, represents the kth subgroup, for Global rewards at all times, for The reward of the kth subgroup at time, for The reward obtained by agent i at this moment, For subgroups The weight in the global reward, is the agent i in the subgroup The reward weight in It is a reward related only to the global goal. is only and subgroup Relevant rewards.

[0063] The subgroup Weight in the global reward Update and confirm based on the principles of contribution, fairness and dynamism;

[0064] Contribution principle: The agents and subgroups that have a greater impact on the task results have higher corresponding weights; when a task becomes more urgent or important, the weights of the relevant subgroups and agents should increase accordingly; when resources are limited, rewards should be allocated first to agents or subgroups with high resource utilization efficiency.

[0065] Fairness principle: Avoid overly biased reward distribution towards certain agents to ensure the overall collaboration of the system. Therefore, we use a clipping mechanism similar to the PPO algorithm to set the upper and lower bounds of the overall weight of the system. That is, each time the weight is updated, the weight before and after the update satisfies:

[0066]

[0067]

[0068] in, Pre-update subpopulation The weight in the global reward, represents the weight of the subgroup after update, Represents the subgroup to which agent i belongs before updating The reward weight in represents the reward weight of agent i after update, Represents the maximum allowed change in weight;

[0069] Dynamic principle: On the one hand, the time-weighted average method is used to smooth the fluctuation of individual rewards and prevent the rewards from being too high or too low in the short term. On the other hand, the strategy can be updated every u time steps according to the absolute advantage function:

[0070]

[0071] This can increase the reward weight of the subgroup with higher than average rewards over a period of time and suppress the rewards for poor performance; similarly, the reward weight of each agent in the subgroup can be updated according to the following formula:

[0072]

[0073] represents the sub-group at time t The weight in the global reward, For all subgroups in The average reward at the moment, For subgroups Rewards, is the learning rate, u represents the time step of the update interval, is the subgroup to which agent i belongs at time t The reward weight in For the entire sub-group of internal agents The average reward at the moment, For agent i in Moment of reward.

[0074] Based on the above content, information exchange and strategy update are carried out within the cluster; the agents share information within the sub-groups, such as status, intention, etc. The sub-group leaders exchange key information and coordinate the global strategy; then multi-level strategy fusion is carried out, the strategy gradient is updated, and data such as task completion, learning speed, communication overhead, and reward distribution fairness are recorded at the same time, and hyperparameters such as learning rate, reward weight, and communication frequency are adjusted.

[0075] Step 3: Use the trained dynamic hierarchical multi-agent strategy optimization model to generate a multi-agent control plan, and perform information interaction and strategy update according to environmental changes to achieve dynamic adjustment of the multi-agent.

[0076] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0077] Step 1: Determine the number of levels of the hierarchy according to the requirements, divide the agents into several subgroups, and select representatives responsible for communication in each subgroup;

[0078] Step 2: According to the determined reinforcement learning algorithm, a dynamic hierarchical multi-agent strategy optimization model is constructed, and the dynamic hierarchical multi-agent strategy optimization model is trained based on the reward function;

[0079] Step 3: Use the trained dynamic hierarchical multi-agent strategy optimization model to generate a multi-agent control plan, and perform information interaction and strategy update according to environmental changes to achieve dynamic adjustment of the multi-agent.

[0080] A computer storable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0081] Step 1: Determine the number of levels of the hierarchy according to the requirements, divide the agents into several subgroups, and select representatives responsible for communication in each subgroup;

[0082] Step 2: According to the determined reinforcement learning algorithm, a dynamic hierarchical multi-agent strategy optimization model is constructed, and the dynamic hierarchical multi-agent strategy optimization model is trained based on the reward function;

[0083] Step 3: Use the trained dynamic hierarchical multi-agent strategy optimization model to generate a multi-agent control plan, and perform information interaction and strategy update according to environmental changes to achieve dynamic adjustment of the multi-agent.

[0084] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A dynamic hierarchical multi-agent control method based on reinforcement learning, characterized in that: The following steps are involved: Step 1: Determine the number of levels of the hierarchy according to the requirements, divide the agents into several subgroups, and select representatives responsible for communication in each subgroup; Step 2: According to the determined reinforcement learning algorithm, a dynamic hierarchical multi-agent strategy optimization model is constructed, and the dynamic hierarchical multi-agent strategy optimization model is trained based on the reward function: The dynamic hierarchical multi-agent strategy optimization model is constructed based on a reinforcement learning algorithm, including a global strategy network, a sub-group strategy network, and a single agent strategy network; At each time step, the policy networks at different levels obtain the environment observation information and the feature information of the lower-level policy network respectively, and then obtain the action output through the policy network of this layer; The strategy network at the single agent level is constructed based on DQN, PPO or DNN algorithm; the strategy network at the global level and the strategy network at the subgroup level are constructed using a collaborative reinforcement learning algorithm; The policy network at the single agent level Taking the individual's strategy probability distribution as input, we can get the individual's strategy characteristics: ； in, represents the strategic features of the i-th agent, FFN represents a layer of forward neural network, logits( ) represents the action probability distribution of the strategy network of the ith agent, represents the environmental information observed at time i, Represents the action of agent i at time i; The subgroup level The policy network takes the policy probability distribution of the sub-group as input to obtain the policy characteristics of the sub-group; ； ； in represents the action probability distribution of the policy network of subgroup c, Represents the feature weight of the i-th agent; The global policy network Take the global strategy probability distribution as input to obtain the strategy characteristics of the sub-group; Among them, the values ​​of the aggregated features in the subgroup-level policy network and the global-level policy network are: ； represents the feature weight of the c-th subgroup, They represent the aggregate features of all agents within the cth subgroup and the aggregate features of all internal subgroups in the global layer respectively; Step 3: Use the trained dynamic hierarchical multi-agent strategy optimization model to generate a multi-agent control plan, and perform information interaction and strategy update according to environmental changes to achieve dynamic adjustment of the multi-agent.

2. The dynamic hierarchical multi-agent control method based on reinforcement learning according to claim 1 is characterized in that: In step 1, the agents are divided into several subgroups according to actual needs, specifically: Based on the geographical location, task similarity or communication topology of the agents, the agents are dynamically divided into several subgroups; within each subgroup, the agents are dynamically adjusted according to the needs, including the merging and splitting of subgroups: In the agent collection middle, represents the kth subgroup, where Represents the serial numbers of all agents contained in the i-th subgroup; When subgroups and If a parameter value of the agent in exceeds the set threshold, then Merge into a new subgroup ; When subgroups When a parameter value of an agent in exceeds the set threshold, the sub-group center is determined and the sub-group is reassigned. The intelligent agent in .

3. The dynamic hierarchical multi-agent control method based on reinforcement learning according to claim 1 is characterized in that: The step 1 of selecting a representative in each subgroup who is responsible for communication is as follows: Based on the coordinates of each agent, the cluster center coordinates in each subgroup are determined based on the clustering algorithm. The agent with the closest Euler distance to the cluster center is taken as the representative responsible for communication in the sub-group, that is, the initial sub-group representative.

4. The dynamic hierarchical multi-agent control method based on reinforcement learning according to claim 1 is characterized in that: The global policy network, the subgroup policy network and the single agent policy network are trained based on the objective function and the reward function to achieve parameter update of the multi-level policy network; The objective function is: ； The reward function is: ； in, express , are the objective functions of the global policy network, the subgroup policy network, and the single agent policy network, respectively. is the corresponding policy network Parameters, express , that is, the value of the reward function in the global policy network, the sub-group policy network, and the single agent policy network, T represents the maximum time step of an action sequence; N is the total number of agents, K is the total number of sub-populations, represents the kth subgroup, for Global rewards at all times, for The reward of the kth subgroup at the moment, for The reward obtained by agent i at this moment, For subgroups The weight in the global reward, is the agent i in the subgroup The reward weight in It is a reward related only to the global goal. is only and subgroup Relevant rewards.

5. The dynamic hierarchical multi-agent control method based on reinforcement learning according to claim 4 is characterized in that: The subgroup Weight in the global reward Update and confirm based on the principles of contribution, fairness and dynamism; ； ； in, Pre-update subpopulation The weight in the global reward, represents the weight of the subgroup after update, Represents the subgroup to which agent i belongs before updating The reward weight in represents the reward weight of agent i after update, Represents the maximum allowed change in weight; ； ； represents the sub-group at time t The weight in the global reward, For all subgroups in The average reward at the moment, For subgroups Rewards, is the learning rate, u represents the time step of the update interval, is the subgroup to which agent i belongs at time t The reward weight in For the entire sub-group of internal agents The average reward at the moment, For agent i in Moment of reward.

6. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

7. A computer storable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images