Multi-agent reinforcement learning system

Abstract

The present invention discloses a multi-agent reinforcement learning system, including a local coordination control subsystem and a global coordination control subsystem; the local coordination control subsystem includes a biologically inspired behavior module, a cluster behavior control module and a local agent strategy pool, the biologically inspired behavior module is used to realize the coordinated control of the obstacle avoidance, navigation and multiple operations of the agent, the cluster behavior control module is used to realize the cluster behavior control of the agent, and the local agent strategy pool is used to store the agent processing strategy; the global coordination control subsystem includes a global data update module, a global arbitration module and a global agent strategy pool, the global data update module is used to push the global strategy to the agents in the global and obtain the accumulated strategies of the agents, the global arbitration module is used to determine the best strategy, and the global agent strategy pool is used to store the best strategy. The above scheme provides a bionic intelligent reinforcement learning framework, which can improve the learning rate and performance of the system.

Description

Technical Field

[0001] The present invention relates to the field of multi-agent technology, and in particular to a multi-agent reinforcement learning system. Background Art

[0002] Reinforcement Learning, as a method of machine learning, is often used to solve sequential decision problems. It models sequential decision problems as an external environment, takes actions based on observations of the external environment, and obtains feedback signals from the external environment. The agent measures the quality of the action based on this feedback signal, summarizes experience, and gradually learns to obtain the optimal decision sequence.

[0003] Multi-agent systems have attracted considerable attention in the development of drone swarms because these systems can be modeled and learned in a more decentralized manner. Multi-agent systems can gather information from the environment and use it more effectively, especially in more dynamic environments. This approach is easy to implement and enables swarms of agents such as drones to fly more efficiently and perform intended tasks. Multi-agent systems improve their efficiency in learning about the environment by leveraging onboard sensors. While multi-agent systems can perform tasks more efficiently, enabling these systems to collaborate becomes more complex, so introducing reinforcement learning to multi-agent systems is now a hot topic of research.

[0004] Existing reinforcement learning methods in multi-agent systems still have the following shortcomings: determining the best strategy requires knowledge about the environment and goals; training agents takes more time and responses to new environments may be delayed; when applied in more complex environments, the learning rate requirements of the agents become very high. Summary of the invention

[0005] In view of the above problems, the present invention is proposed to provide a multi-agent reinforcement learning system that overcomes the above problems or at least partially solves the above problems.

[0006] According to a multi-agent reinforcement learning system provided by the present invention, the multi-agent reinforcement learning system includes a local coordination control subsystem and a global coordination control subsystem;

[0007] The local coordination control subsystem includes a biologically inspired behavior module, a cluster behavior control module and a local agent strategy pool, wherein the biologically inspired behavior module is used to realize the coordinated control of multiple operations within the agent, the cluster behavior control module is used to realize the cluster behavior control of the agent, including coordination, alignment and consistency with other agents, and the local agent strategy pool is used to store the agent processing strategy;

[0008] The global coordination control subsystem includes a global data update module, a global arbitration module and a global agent strategy pool, wherein the global data update module is used to push global strategies to the agents within the global system and obtain the accumulated strategies of multiple agents within the global system, the global arbitration module is used to execute a reinforcement learning algorithm to determine the best strategy from the local agent strategy pool, and the global agent strategy pool is used to store the best strategy.

[0009] In some embodiments, a ground control module is also included to obtain the position of the intelligent body and obtain environmental information.

[0010] In some embodiments, the bio-inspired behavior module in the local coordination control subsystem is configured to:

[0011] Based on the Boids model in group motion, the movement trends of multiple agents are controlled and the cohesion or separation strategies between agents are determined.

[0012] In some implementations, determining a cohesion or separation strategy between agents includes:

[0013] Detect the positions of all agents within a certain range, calculate the center of mass, and then generate a separation velocity away from the center of mass;

[0014] Detect the speed of all agents within a certain range, calculate the average speed, and then generate a parallel speed in the same direction as the average speed;

[0015] Detect the positions of all agents within a certain range, calculate the center of mass, and then generate a cohesive velocity pointing to the center of mass.

[0016] In some implementations, determining the cohesion or separation strategy between agents further includes:

[0017] When the distance between two agents is less than the collision radius, the separation steering behavior is performed; when the distance between two agents is greater than or equal to the collision radius, the cohesion steering behavior is performed.

[0018] In some implementations, the global arbitration module in the global coordination control subsystem is configured to:

[0019] An actor-critic algorithm is used to learn and determine the optimal strategy from the local agent strategy pool.

[0020] In some embodiments, using an actor-critic algorithm to learn and determine an optimal strategy from the local agent strategy pool includes:

[0021] The participant is used to fit the agent's policy function, which is responsible for generating actions and interacting with the environment; the critic is responsible for fitting the value function, which is used to evaluate the participant's performance and guide the participant's actions in the next stage.

[0022] In some embodiments, a proximal policy optimization (PPO) algorithm based on the actor-critic algorithm is used to construct an actor action decision neural network and a critic action evaluation neural network;

[0023] Taking the local agent strategy pool as input, the optimal strategy is determined based on the participant action decision neural network and the critic action evaluation neural network.

[0024] In some implementations, the agent redetermines its own strategy based on the decision result of the global arbitration module in the global coordination control subsystem, and simultaneously identifies adjacent agents on the path and determines the strategy to be executed.

[0025] In some embodiments, the intelligent agent is a drone.

[0026] As can be seen from the above, the two-level hybrid control framework disclosed in the embodiment of the present invention uses a two-level control strategy. Local control uses AI-based bio-inspired steering behavior control to perform lower-level maneuvers such as clustering and collision avoidance, and the final optimal strategy is specified by the global control through the PPO algorithm, thereby improving the learning rate and performance under continuous monitoring.

[0027] Furthermore, the multi-agent reinforcement learning system disclosed in the present invention can help learn the environment faster by combining the two subsystems and updating the algorithm of advanced knowledge by using the experience of the local strategy pool. In addition, the algorithm model given is trained and verified based on the local strategy and the construction of the neural network model. The results show that the bionic intelligent reinforcement learning performs well with less training times.

[0028] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0030] Figure 1A schematic diagram of the architecture of a multi-agent reinforcement learning system according to an embodiment of the present invention is shown;

[0031] Figure 2 A schematic structural diagram of a drone according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0032] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0033] Figure 1 The multi-agent reinforcement learning system according to one embodiment of the present invention is shown, including a local coordination control subsystem and a global coordination control subsystem;

[0034] The local coordination control subsystem includes a biologically inspired behavior module, a cluster behavior control module and a local agent strategy pool, wherein the biologically inspired behavior module is used to realize the coordinated control of multiple operations within the agent, the cluster behavior control module is used to realize the cluster behavior control of the agent, including coordination, alignment and consistency with other agents, and the local agent strategy pool is used to store the agent processing strategy;

[0035] The global coordination control subsystem includes a global data update module, a global arbitration module and a global agent strategy pool, wherein the global data update module is used to push global strategies to the agents within the global system and obtain the accumulated strategies of multiple agents within the global system, the global arbitration module is used to execute a reinforcement learning algorithm to determine the best strategy from the local agent strategy pool, and the global agent strategy pool is used to store the best strategy.

[0036] In summary, according to this embodiment, a two-level control framework for a multi-agent group is implemented, in which a two-level control strategy is set. Local control uses AI-based bio-inspired steering behavior control to perform lower-level maneuvers such as clustering and collision avoidance, and the final optimal strategy is specified by global control through the PPO algorithm, thereby improving the learning rate and performance while continuously monitoring the application; and by creating a framework environment, the performance of the agent can be trained and verified.

[0037] Further, combined with Figure 1It can be seen that the above embodiment realizes the hybridization of bio-inspired steering behavior methods and multi-agent reinforcement learning. The first layer of the two layers of the framework is responsible for managing and executing low-level control activities, such as obstacle avoidance, coordinated control with other drones, cluster behavior, etc., while the high-level control in the second layer is used to help drones and other intelligent agents make decisions for continuous monitoring.

[0038] In some embodiments, a ground control module is also included to obtain the flight position of the intelligent body and obtain the flight environment information.

[0039] In some embodiments, the bio-inspired behavior module in the local coordination control subsystem is configured to:

[0040] Based on the Boids model in group motion, the movement trends of multiple agents are controlled and the cohesion or separation strategies between agents are determined.

[0041] In 1986, Craig Reynolds created a simulation of the collective behavior of birds, called the "flock-like model." This model includes three rules that can control the interaction between individuals, resulting in flock-like behavior.

[0042] In some embodiments, determining a cohesion or separation strategy between agents includes:

[0043] Detect the positions of all agents within a certain range, calculate the center of mass, and then generate a separation velocity away from the center of mass;

[0044] Detect the speed of all agents within a certain range, calculate the average speed, and then generate a parallel speed in the same direction as the average speed;

[0045] Detect the positions of all agents within a certain range, calculate the center of mass, and then generate a cohesive velocity pointing to the center of mass.

[0046] In some embodiments, determining the cohesion or separation strategy between agents further comprises:

[0047] When the distance between two agents is less than the collision radius, the separation steering behavior is performed; when the distance between two agents is greater than or equal to the collision radius, the cohesion steering behavior is performed.

[0048] According to the above embodiment, the set of n agents is , for each agent, Ai(t)={hi(t),vi(t),vm}, hi(t) is the position of the agent at time t, vi(t) and vm are the current speed and maximum speed of the agent, respectively. Nearby Agents It is defined as the area around the agent where it can detect its neighbors, where D represents the distance between two agents, r represents the perception radius, and Ф represents the viewing angle. The specific steps of the algorithm are as follows:

[0049] Step 1: First, the algorithm needs to know the neighboring agents Ai within the scanned sensor radius r;

[0050] Step 2: Before further evaluation, the length of the detection box Dbl should be fixed. According to the Reynolds model, the length of the detection box is proportional to the agent's vi(t). The length of the detection box is determined as ,in is the minimum detection box length, Gs and Sl are the reference speed and scanning length respectively.

[0051] Step 3: The number of nearby agents Dbl that fall within the sensing radius Φ after calibration is identified, and each nearby agent is marked for subsequent consideration. The intersection point of the closest agent is calculated by representing the agent as a circle (Cx, Cy). The calculation method is as follows:

[0052] (1)

[0053] here, Where n2,n1,e2,e1 are the north and east coordinates of the agent.

[0054] Step 4: Agents that fall on the intersection box will be marked and considered further, and other agents will be discarded. Agents with low positive ratings are given priority to cooperative functions in group behavior.

[0055] Step 5: Agent Research Use Separation or cohesion When the distance between two agents is less than the collision radius, the separation steering behavior is utilized. If this is not the case, the cohesion steering behavior is applied. The separation and cohesion steering behaviors are calculated as follows:

[0056] (2)

[0057] (3)

[0058] here, is the current agent, is a nearby agent. γ is a constant, (γ>1) is a cohesive steering behavior, and is a separation steering behavior.

[0059] Step 6: and The strategy is updated to the local accumulated experience El(n) in the local strategy pool Lapl.

[0060] It should be noted that the above algorithm is particularly suitable for updating global strategies in unknown dynamic environments. The agent attempts to redistribute work based on the decision of the global arbitration module in the global coordination control subsystem. When performing the assigned tasks, the role of the agent is to identify the neighboring agents that obtain the path and calculate the separation or cohesion strategy.

[0061] In some embodiments, the global arbitration module in the global coordination control subsystem is configured to:

[0062] An actor-critic algorithm is used to learn and determine the optimal strategy from the local agent strategy pool.

[0063] It should be noted that the actor-critic algorithm is a reinforcement learning algorithm that consists of two main parts: the actor and the critic. The core idea of ​​this algorithm is that the actor is responsible for interacting with the environment, taking the current state of the environment and determining the best action to take, while the critic plays the role of strategy evaluation, evaluating the quality of the strategy by taking the state of the environment and taking actions, and returning a score to describe how good the state is for a certain action. In other words, the actor explores multiple behaviors like a child, while the critic, like a parent, criticizes bad behavior and rewards good behavior. This algorithm learns both the policy function and the state-action value function, and updates its parameters through gradient ascent, making this method very effective when facing high-dimensional continuous state and action spaces.

[0064] In an embodiment of the present invention, the main steps of the above algorithm include:

[0065] The global arbitrator draws from the local agent strategy pool Update strategy, where It is the local strategy obtained by a single agent, and is calculated and trained to make global decisions. Specifically, a participant-critic method PPO is used to calculate the training of learning the optimal strategy.

[0066] The PPO algorithm consists of two approximations: the actor μ(s) (given observations O(S) and predicting the probability of taking an action in the upper part of the state space S) and the critic V(s) (taking observations O(S) and predicting the expected long-term reward). After training, the optimal policy is stored in the actor μ(s). Based on this, the algorithm is formulated by the following steps:

[0067] Step 1: Random parameter value θ μ ,θ Vare assigned to the participants μ(s) and the critics V(s).

[0068] Step 2: The experience E(n) is calculated according to the following empirical sequence:

[0069] (4)

[0070] here, Indicates the observation state. express Actions taken, Indicates the next state, Represents the agent from Move to Rewards received when It represents the time step of the current N experiences.

[0071] Step 3: Calculate the advantage function using the advantage estimation method

[0072] (5)

[0073] here, , λ represents the smoothing factor.

[0074] Step 4: Calculate the return value:

[0075] (6)

[0076] Step 5: Learn the mini-batch set Kn in E(n);

[0077] Step 6: Repeat steps 2 and 4 until the agent reaches the desired goal.

[0078] In some embodiments, using an actor-critic algorithm to learn and determine an optimal strategy from the local agent strategy pool includes:

[0079] The participant is used to fit the agent's policy function, which is responsible for generating actions and interacting with the environment; the critic is responsible for fitting the value function, which is used to evaluate the participant's performance and guide the participant's actions in the next stage.

[0080] In some embodiments, a proximal policy optimization PPO algorithm based on the actor-critic algorithm is used to construct an actor action decision neural network and a critic action evaluation neural network;

[0081] Taking the local agent strategy pool as input, the optimal strategy is determined based on the participant action decision neural network and the critic action evaluation neural network.

[0082] In some embodiments, the agent redetermines its own strategy according to the decision result of the global arbitration module in the global coordination control subsystem, and simultaneously identifies adjacent agents on the path and determines the strategy to be executed.

[0083] In some embodiments, the agent is a drone.

[0084] It should be noted that:

[0085] The algorithm and display provided herein are not inherently related to any particular computer, virtual device or other equipment. Various general-purpose devices can also be used together with the teachings based on this. According to the above description, it is obvious to construct the structure required for this type of device. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific language is for the purpose of disclosing the best mode of the present invention.

[0086] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0087] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this method of disclosure should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim.

[0088] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and further may be divided into a plurality of submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device so disclosed may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0089] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present invention and to form different embodiments.

[0090] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components of the drone tracking device according to an embodiment of the present invention. The present invention may also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program for implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0091] Figure 2 A schematic diagram of the structure of an embodiment of a drone of the present invention is shown, and the specific embodiment of the present invention does not limit the specific structure of the drone.

[0092] like Figure 2 As shown, the UAV (including the flight control system) may include: a processor (processor) 202 , a communications interface (Communications Interface) 204 , a memory (memory) 206 , and a communication bus 208 .

[0093] The processor 202 , the communication interface 204 , and the memory 206 communicate with each other via a communication bus 208 . The communication interface 204 is used to communicate with other devices such as a client or other server network elements. The processor 202 is used to execute a program 210 .

[0094] Specifically, the program 210 may include program codes, which include computer operation instructions.

[0095] The processor 202 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the drone may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0096] The memory 206 is used to store the program 210. The memory 206 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0097] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

Claims

1. A multi-agent reinforcement learning system, characterized in that: The multi-agent reinforcement learning system includes a local coordination control subsystem and a global coordination control subsystem; The local coordination control subsystem includes a biologically inspired behavior module, a cluster behavior control module and a local agent strategy pool, wherein the biologically inspired behavior module is used to realize the coordinated control of multiple operations within the agent, the cluster behavior control module is used to realize the cluster behavior control of the agent, including coordination, alignment and consistency with other agents, and the local agent strategy pool is used to store the agent processing strategy; The global coordination control subsystem includes a global data update module, a global arbitration module and a global agent strategy pool, wherein the global data update module is used to push the global strategy to the agents in the global system and obtain the strategies accumulated by multiple agents in the global system, the global arbitration module is used to execute the reinforcement learning algorithm to determine the best strategy from the local agent strategy pool, and the global agent strategy pool is used to store the best strategy; The bio-inspired behavior module in the local coordination control subsystem is configured to: control the movement trends of multiple agents based on the Boids model in group movement, and determine the cohesion or separation strategy between the agents; The global arbitration module in the global coordination control subsystem is configured to: learn and determine the optimal strategy from the local agent strategy pool using an actor-critic algorithm; Among them, the participant is used to fit the agent's strategy function, responsible for generating actions and interacting with the environment; the critic is responsible for fitting the value function, which is used to evaluate the participant's performance and guide the participant's actions in the next stage.

2. The multi-agent reinforcement learning system according to claim 1, characterized in that: It also includes a ground control module for obtaining the position of the intelligent body and obtaining environmental information.

3. The multi-agent reinforcement learning system according to claim 1, characterized in that: Strategies for determining cohesion or separation between agents include: Detect the positions of all agents within a certain range, calculate the center of mass, and then generate a separation velocity away from the center of mass; Detect the speed of all agents within a certain range, calculate the average speed, and then generate a parallel speed in the same direction as the average speed; Detect the positions of all agents within a certain range, calculate the center of mass, and then generate a cohesive velocity pointing to the center of mass.

4. The multi-agent reinforcement learning system according to claim 3, characterized in that: Determining the cohesion or separation strategy between agents further includes: When the distance between two agents is less than the collision radius, the separation steering behavior is performed; when the distance between two agents is greater than or equal to the collision radius, the cohesion steering behavior is performed.

5. The multi-agent reinforcement learning system according to claim 1, characterized in that: Using an actor-critic algorithm to learn and determine the optimal strategy from the local agent strategy pool further includes: Based on the proximal policy optimization PPO algorithm of the actor-critic algorithm, the actor action decision neural network and the critic action evaluation neural network are constructed; Taking the local agent strategy pool as input, the optimal strategy is determined based on the participant action decision neural network and the critic action evaluation neural network.

6. The multi-agent reinforcement learning system according to claim 1, characterized in that: The intelligent agent redetermines its own strategy according to the decision result of the global arbitration module in the global coordination control subsystem, and simultaneously identifies the adjacent intelligent agents on the path and determines the strategy to be executed.

7. The multi-agent reinforcement learning system according to any one of claims 1 to 5, characterized in that: The intelligent agent is a drone.

Images