A multi-agent reinforcement learning method for formation encirclement

Abstract

The present invention is a multi-agent reinforcement learning method for formation encirclement, which is particularly suitable for collision avoidance requirements and simple closed encirclement trajectories, and includes the following steps: a) determining the state space, action space and reward function of each agent in the reinforcement learning environment; b) designing the actor network and judge network structure of each agent; c) determining the action of each agent through the actor network based on its current state, and then interacting with the environment to obtain the reward value of the action and the next state of the agent; d) determining the state and action value functions through the judge network based on the actions, current and next states of all agents, and then designing the loss function; e) repeating steps c) and d) according to the back propagation algorithm to update the weights in the actor and judge networks until the expected requirements are met. This method is simple and reliable, does not rely on the system model, and can quickly realize the formation encirclement task in a complex environment.

Description

Technical Field

[0001] The invention relates to a multi-agent reinforcement learning method for formation surrounding. Background Art

[0002] Formation encirclement, also known as collaborative encirclement, is a control technology in which multiple unmanned systems surround a target in a certain formation. It is currently widely used in collaborative patrols, target navigation, resource positioning and autonomous rescue. At present, most formation encirclement design methods are based on traditional system models, such as (Chen Yangyang; Wei Ping; Method for tracking unknown targets by distributed formation spherical encirclement based on relative position, Chinese patent, authorized, 2017-05-03, ZL201710303014.4.). At the same time, once formation encirclement is added with collision avoidance / obstacle avoidance, traditional controller design becomes very difficult. The reason is that the control law easily causes the unmanned system to fall into a local equilibrium point and cannot continue to complete the formation encirclement of the target.

[0003] Reinforcement learning aims to allow the agent to explore and learn a specific unknown environment. By setting the reward function and updating the value function, the agent is guided to learn to make decisions and make the corresponding optimal actions for a specific state, so as to obtain the maximum reward. Compared with previous control methods, this method does not require a specific and accurate model. That is, as long as the environment and the agent are set, the ideal results can be obtained through training. Reinforcement learning methods have made some progress in formation encirclement, but the encirclement is still limited to static circular encirclement, for example (VRKonda and JNTsitsiklis, "Actor-critic algorithms," in Advances in neural information processing systems, pp. 1008–1014, 2000.). In practice, whether it is underwater unmanned robots tracking and collaboratively collecting seabed biological groups, or unmanned aerial vehicles collaboratively encircling targets, it may be a non-circular encirclement (such as an ellipse), and the unmanned system is required to move around the target rather than being stationary. At present, there is still a lack of formation encirclement algorithms that can achieve collision avoidance and parameterize simple encirclement trajectories at the same time. Summary of the invention

[0004] Technical problem: The purpose of the present invention is to provide a multi-agent reinforcement learning method for formation encirclement, which is simple and reliable and suitable for formation encirclement tasks in obstacle environments.

[0005] Technical solution: The present invention is a formation encirclement multi-agent reinforcement learning method, which is particularly suitable for collision avoidance requirements and simple closed encirclement trajectories. In order to train the desired reinforcement learning formation encirclement algorithm, the action space, state space and reward value function in the multi-agent reinforcement learning environment must be set first. Next, the reinforcement learning formation encirclement algorithm can be trained through the state, action and reward value of the agent provided by the multi-agent reinforcement learning environment. The core idea is as follows: First, the state value of the agent in the reinforcement learning environment is initialized. Then an actor neural network is designed with the state value of the agent as input and the action value as output. The initialized state value is input into the neural network to obtain the action value. At this time, a little noise is added to obtain the final action value, and then the action and state value are input into the reinforcement learning environment to obtain the state value at the next moment and the reward value under the state action. Then a critic neural network with action value, state value and reward value as input and action value function as output is designed, and the action value, state value and reward value are input into the network to obtain the action value function. The obtained action value function is combined with the reward value to construct the loss function of the actor neural network and the critic neural network respectively. The weight of the neural network is updated by back propagation calculation. The updated neural network repeats the above steps and the training process continuously, and finally a result that converges to the optimal value is obtained. In this way, the training and learning process of this formation encirclement multi-agent reinforcement learning method is completed.

[0006] Specifically:

[0007] The present invention is a multi-agent reinforcement learning method for formation encirclement. No system model is required. The encirclement trajectory can be a general simple closed curve and can achieve collision avoidance. It is particularly suitable for collision avoidance requirements and simple closed encirclement trajectories. The method comprises the following steps:

[0008] a) Determine the action space, state space, and reward function for each agent in the reinforcement learning environment.

[0009] b) Design the actor network and critic network structure of each agent;

[0010] c) The current state of each agent determines its action through the actor network, and then interacts with the environment to obtain the reward value of the action and the next state of the agent;

[0011] d) The action-state value function is determined by the actions, current and next states of all agents through the critic network, and then the loss function is designed;

[0012] e) Repeat steps c) and d) according to the back-propagation algorithm to update the weights in the actor and critic networks until the desired requirements are met.

[0013] Wherein said step a) includes the following:

[0014] a1) The state and actions of the agent in the reinforcement learning environment are determined by its dynamics;

[0015] a2) Set the dynamics of the target and the position of the obstacle, and calculate the distance and polar angle from the agent to the target, the relative distance and relative polar angle between the agent and its neighbors, and the relative distance from the agent to external obstacles;

[0016] a3) The reward function in the reinforcement learning environment is determined by the task.

[0017] Wherein said step b) includes the following situation:

[0018] b1) Design the number of fully connected layers of the actor network for each agent, the number of neurons in each layer, and the activation function used.

[0019] b2) Design the number of fully connected layers of the critic network for each agent, the number of neurons in each layer, and the activation function used.

[0020] Wherein said step c) includes the following situation:

[0021] c1) Input the current state of the agent into the actor neural network to determine the action of each agent;

[0022] c2) Input the action into the reinforcement learning environment to obtain the reward value of the action and the next state of the agent.

[0023] Wherein said step d) includes the following situation:

[0024] d1) Input the actions, current and next state of all agents into the judge network to determine the action-state value function;

[0025] d2) Design the loss function of the critic network from the action-state value function and the reward function.

[0026] d3) Design the loss function of the actor network from the action-state value function and the reward function.

[0027] Wherein said step e) includes the following situation:

[0028] e1) Update the weights in the actor network according to the back-propagation algorithm until the expected requirements are met;

[0029] e2) Update the weights in the judge network according to the back-propagation algorithm until the expected requirements are met.

[0030] Beneficial effects: This method is simple and effective, does not rely on the system model, and is suitable for formation encirclement tasks in obstacle environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Schematic diagram of a multi-agent reinforcement learning environment surrounded by formations;

[0032] Figure 2 It is a schematic diagram of the position and angle information of the intelligent body of the elliptical trajectory in the relative coordinate system;

[0033] Figure 3 It is a schematic diagram of the structure of the actor neural network;

[0034] Figure 4 This is a schematic diagram of the critic neural network structure;

[0035] Figure 5 To design a flow chart.

[0036] In the above figure: where p ri Represents the coordinates of the ith agent in the relative coordinate system; represents the polar angle of the ith agent in the relative coordinate system; It represents the polar angle difference between the ith agent and the adjacent agent in the relative coordinate system; T represents the target position, i.e., the origin of the relative coordinate system; C represents the intersection of the target and the agent's line on the elliptical trajectory; FC represents the fully connected layer in the neural network; ReLU represents the linear rectification function in the activation function; tanh represents the hyperbolic tangent function in the activation function. DETAILED DESCRIPTION

[0037] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0038] Figure 5 It is a design flow chart of the present invention, which consists of steps P1-P5, and each step is described as follows:

[0039] 1) Step P1

[0040] Figure 1 The multi-agent reinforcement learning environment surrounded by the formation shown in the figure, the specific implementation steps of the reinforcement learning environment are as follows:

[0041] Step 1: State space s: the position information of the i-th agent in the inertial coordinate system speed And control input in and At the same time, the initial values ​​of position and speed are set. The training process is carried out in the form of scenes, that is, at the end of each scene, the environment will return the robot state value to the initial state.

[0042] Action space u: Each agent is subject to two forces f in the x and y directionsx and f y , set the upper limit of the force according to the actual driving ability of the intelligent agent.

[0043] Step 2: Set the position of the obstacle through external devices (such as cameras) O (t) = [x O (t),y O (t)] T and the target location The position p of each agent i and the target position p T Calculate the relative position to the target Then the relative position to the target p Ti Calculate the distance to the target ρ iT (t)=||p iT ||. With the target as the center and the target's velocity direction as the x direction, calculate the polar angle corresponding to the relative position to the target By polar angle φ i (t) The angle difference between adjacent agents is calculated by the following formula:

[0044] Δφ i =φ i -φ i-1 ,

[0045] Where φ0 represents φ n and The relative position between the agent and its neighbors Calculate the relative distance ρ between them ij (t)=||p ij ||. The relative position of the agent and the obstacle Calculate the relative distance ρ between them iO (t)=||p iO ||.

[0046] Step 3: Design the agent’s reward function.

[0047] The total reward function for each agent is as follows:

[0048] r total =k1r ρi (t)+k2r φi (t)+k3r ωi (t)+k4r ic (t),

[0049] Where k1, k2, k3 and k4 represent the weights of each reward function. The ellipse surrounds the reward r ρi (t), formation reward r φi (t), angular velocity reward r ai(t) and collision avoidance reward r ci (t) are as follows:

[0050] (1) Encirclement reward function:

[0051] r ρi (t) = -(ρ iT (t)-ρ * (θ i (t))) 2 ,

[0052] where ρ * (θ i ) is the parameterized expected distance, θ i is the orbit parameter. For example, if the enclosing trajectory is a circle, then ρ * (θ i )=R, where R is the radius of the circle; if the enclosing trajectory is an ellipse, then ρ * (θ i )=((a cosθ i ) 2 +(b sinθ i ) 2 ) 1 / 2 , where a is the length of the major semi-axis and b is the length of the minor semi-axis. This reward function drives each agent to enclose the target with a desired simple closed curve.

[0053] (2) Formation reward function:

[0054] r φi (t) = -(Δφ i (t)-φ * ) 2 ,

[0055] in represents the expected angle difference, and n is the number of agents. This reward function is designed to make the agents evenly distributed on the given track.

[0056] (2) Angular velocity reward function:

[0057] r ai (t)=-((φ i (t+1)-φ i (t))-w * ) 2 ,

[0058] where ω * Represents the expected angular velocity. This reward function is designed to make the agent rotate at a constant speed on a given track.

[0059] (3) Collision avoidance reward function:

[0060]

[0061] Where ∈ is the minimum safe distance, is the safe distance. This reward function is used to prevent the agent from collision during movement.

[0062] 2) Step P2

[0063] Step P2 designs the actor network and judge network structure of each agent respectively. The design is implemented in the following steps:

[0064] Step 1: Design Figure 3 The actor neural network shown in the figure consists of three fully connected layers. The number of neurons in the first layer is 128, and the fully connected layer is followed by a linear rectifier function as the activation function; the number of neurons in the second layer is 64, and the fully connected layer is also followed by a linear rectifier function as the activation function; the number of neurons in the last layer is the number of dimensions of the action value, and the fully connected layer is followed by a hyperbolic tangent function as the activation function. Each agent needs to have an actor neural network, with the input being the state value of the corresponding agent and the output being the action value of the corresponding agent, where each agent only uses the information observed by itself.

[0065] Step 2: Design Figure 4 The critic network shown in the figure consists of four fully connected layers. The number of neurons in the first layer is 256, and the fully connected layer is followed by a linear rectifier function as the activation function; the number of neurons in the second layer is 128, but the input is composed of the result obtained from the first fully connected layer and the action values ​​of all agents, and the fully connected layer is followed by a linear rectifier function as the activation function; the number of neurons in the third layer is 64, and the fully connected layer is followed by a linear rectifier function as the activation function; the number of neurons in the last layer is 1, but no activation function is connected, because this action value function must have a negative number (indicating that the action generates negative feedback). Similarly, each agent also has a critic neural network, the input is the state value of all agents, and the output is the action-state value function that evaluates the quality of the action, because the information of all agents is used.

[0066] 3) Step P3

[0067] Step P3 inputs the current state of each agent into the corresponding actor network to obtain the action of the agent, and interacts with the environment to obtain the corresponding reward value and the next state. The design is implemented in the following steps:

[0068] Step 1: Input the state value of the agent into the actor neural network designed in P2 to obtain the action value of each agent.

[0069] Step 2: Add appropriate noise (here we use ordinary Gaussian distribution) to the action value obtained in the first step, so that the agent can explore a wider space. Then bring the action value and state value into the reinforcement learning environment to interactively obtain the state at the next moment and the reward value under the action state.

[0070] 4) Step P4

[0071] Step P4 inputs the actions of all agents, the current state and the next state obtained in P3 into the critic network to determine the action-state value function, and designs the loss functions of the critic network and the actor network respectively based on the obtained action-state value function and the reward function designed in P1.

[0072] Step 1: Input the current state values ​​of all agents and the action values, next state values, and reward values ​​of all agents obtained in P3 into the critic neural network designed in P2 to obtain the action-state value function under the state action.

[0073] Step 2: Design the loss function of the critic network based on the obtained action-state value function and the reward function designed in P1. In order to stabilize the update of the network, the corresponding target network is added to the actor and critic networks. The target network has the same structure, but the update method is different. And the update method can be divided into hard update (the target network parameters are updated to the same parameters as the corresponding network after a specified number of training episodes, and are not updated at other times) and soft update as follows:

[0074]

[0075] That is, it is updated every step, but the update is slower. Q ,θ Q′ ,θ μ ,θ μ′ They represent the weights of the critic network, critic target network, actor network, and actor target network respectively, and τ is the update rate.

[0076] In order to ensure independent and distributed sampling during training, it is also necessary to establish an experience playback cache D = (x t ,a t ,R t ,x t+1 ), where x t ={o1(t),...,o N (t)} represents the state value of all agents at the current moment, and o i (t) represents the observation value of the ith agent, a t represents the action value of all agents at the current moment, R t represents the reward value of all agents at the current moment, xt+1 Represents the state value of all agents at the next moment.

[0077] First, we use the target actor network and the critic network to obtain the realistic action-state value function:

[0078]

[0079] where r i represents the reward value of the ith agent, It means that the action value obtained by the target actor network is brought into the action value function obtained by the critic network. Then the TD-error is obtained by using the actual action-state value function as the loss function of the critic network:

[0080]

[0081] Where S is the number of samples selected in the experience pool, and Q is the action-state value function of the target.

[0082] Step 3: Design the loss function of the actor network based on the obtained action-state value function and the reward function designed in P1. The gradient used to update the actor network is obtained based on the target action-state value function:

[0083]

[0084] 5) Step P5

[0085] Step P5 uses the loss function obtained in P4 according to the back propagation algorithm to update the weights in the judge network and the actor network respectively until the expected requirements are met.

[0086] Step 1: Update the weights in the critic network using the loss function obtained in P4 according to the back propagation algorithm until the desired requirements are met. The weights in the critic network are updated by applying the back propagation algorithm (based on the gradient descent method) to the loss function of the critic network obtained in the first step of P4, and the target network of the corresponding critic network is updated using the soft update method. Repeat the above steps and the training process of the neural network after updating the weights, and finally a result that converges approximately to the optimal value can be obtained.

[0087] Step 2: Similarly, according to the back-propagation algorithm, the weights in the actor network are updated using the loss function obtained in P4 until the desired requirements are met. The weights in the actor network are updated by applying the back-propagation algorithm to the loss function of the actor network obtained in the second step of P4, and the target network of the corresponding actor network is updated using the soft update method. The neural network with updated weights is repeatedly trained through the above steps, and finally a result that converges approximately to the optimal value can be obtained. Thus, the training and learning process of this formation encirclement multi-agent reinforcement learning method is completed.

Claims

1. A multi-agent reinforcement learning method for formation encirclement, characterized in that: The method comprises the following steps: a) Determine the action space u and state space s of each agent in the reinforcement learning environment, and obtain the obstacle, target point and position information of each agent through the peripheral device to design the formation encirclement reward function; b) Use a fully connected neural network to design the actor network and critic network structure for each agent; c) Based on the current state of each agent, the agent action is determined through the actor network, and the agent interacts with the environment. The reward function is used to obtain the formation surrounding reward of the action, and then the next state of the agent is obtained; d) Determine the action-state value function through the critic network based on the actions, current and next states of all agents, and design the loss functions of the actor network and critic network respectively; e) Repeat steps c) and d) according to the back propagation algorithm to update the weights in the actor and judge networks until the desired requirements are met; Wherein said step a) includes the following: a1) The state space s of each agent is determined by the position, velocity and control input information of each agent in the reinforcement learning environment, and the action space u of the agent is determined by the force conditions of the agent; a2) Set the obstacle position and target position through the peripheral device, and calculate the distance and polar angle from the agent to the target, the relative distance and relative polar angle from the agent to the neighbor, and the relative distance from the agent to the obstacle according to the relative position of each agent to the target and obstacle; a3) Based on the information obtained in a2), the encirclement reward function, formation reward function, angular velocity reward function and collision avoidance reward function are designed for each agent, and then the total reward of each agent is designed; Wherein said step b) includes the following situation: b1) Design the number of fully connected layers of the actor network of each agent, the number of neurons in each layer, and the activation function of the network; b2) Design the number of fully connected layers of the critic network of each agent, the number of neurons in each layer, and the activation function of the network; Wherein said step c) includes the following situation: c1) Input the current state of the agent into the actor neural network to determine the action of each agent; c2) Input the action into the reinforcement learning environment to obtain the reward value of the action and the next state of the agent and the reward value under the action state; Wherein said step d) includes the following situation: d1) Input the actions, current and next states of all agents into the critic network to determine the action-state value function; d2) Set corresponding target networks for the actor and critic networks respectively, and design the loss function of the critic network based on the action-state value function and reward function; d3) Design the loss function of the actor network from the action-state value function and reward function; Wherein said step e) includes the following situation: e1) Update the weights in the actor network according to the back-propagation algorithm and the loss function in d3) until the desired requirements are met; e2) Update the weights in the critic network according to the back-propagation algorithm and the loss function in d2) until the expected value is reached.

Images