An intelligent obstacle avoidance method for drone swarm based on reinforcement learning

Abstract

The present invention provides a method for intelligent obstacle avoidance of drone swarm based on reinforcement learning, which realizes intelligent obstacle avoidance of drone swarm by combining swarm algorithm and single-agent reinforcement learning algorithm. The method controls drones to perform swarm movement and track the navigator. The reinforcement learning algorithm controls the navigator in the swarm to complete intelligent obstacle avoidance and provide navigation information for the swarm drones. The detection information of the drone swarm on the environment provides environmental status information for the virtual navigator. The present invention can effectively improve the obstacle avoidance performance and consistency performance of drone swarm.

Description

Technical Field

[0001] The present invention relates to the field of multi-UAV cooperative control, and in particular to an intelligent obstacle avoidance method for UAV clusters based on reinforcement learning. Background Art

[0002] Biological clustering behavior is a common phenomenon in nature. It is an innate survival ability of social biological groups to adapt to the living environment after long-term evolution, such as bird flocks and fish schools. In 1986, Reynolds proposed three heuristic cluster control rules: cohesion, separation and alignment. On this basis, many scholars have proposed a variety of cluster models, among which the most representative one is the cluster control algorithm proposed by Olfati-Saber. In recent years, due to the increasing autonomy and intelligence of drones, their application value in military and civilian fields has attracted more and more attention and attention. Drone clusters have attracted the interest of many researchers because they can complete more difficult and dangerous tasks.

[0003] The control input of the UAV swarm control algorithm for the UAVs in the swarm mainly includes obstacle avoidance control, swarm control and tracking the leader. The leader is a virtual agent that tracks the planned trajectory. Obstacle avoidance control is to enable the swarm to avoid obstacles. This swarm obstacle avoidance method is prone to fall into local optimality. Summary of the invention

[0004] The purpose of the present invention is to provide a method for intelligent obstacle avoidance of drone clusters in a multi-obstacle environment, to improve the obstacle avoidance performance and consistency performance of drone clusters, and to enhance the adaptability of drone clusters to the environment.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] The invention discloses an intelligent obstacle avoidance method for a swarm of unmanned aerial vehicles. The unmanned aerial vehicles in the swarm keep moving in a swarm and track a virtual navigator. The unmanned aerial vehicles do not have the obstacle avoidance capability, but transmit the detection information of the environment to the virtual navigator. The virtual navigator is trained by a reinforcement learning algorithm. The virtual navigator provides navigation guidance information for the swarm of unmanned aerial vehicles, thereby realizing intelligent obstacle avoidance for the swarm of unmanned aerial vehicles.

[0007] Specifically, the method comprises the following steps:

[0008] (1) Set up a control algorithm for the virtual navigator in the UAV cluster, with a speed limit of 10m / s to 100m / s. Set a target point for the virtual navigator to provide navigation information for the cluster. Use a reinforcement learning algorithm to control the virtual navigator. The detection information of the UAV on the environment provides state information for the virtual navigator, which serves as the state input when training the virtual navigator. The output is the speed vector of the virtual navigator, which serves as the action space of the reinforcement learning algorithm. The position feature of the cluster is replaced by the center position of the cluster. Based on this, the state space is designed, including the relative position of the virtual navigator and the cluster, the relative position between the virtual navigator and the target, and the relative position between the cluster and the obstacle.

[0009] (2) The UAV cluster tracks the virtual navigator, completes obstacle avoidance, provides environmental information to the navigator, maintains the distance between drones, and realizes intelligent obstacle avoidance of the UAV cluster.

[0010] In step (1), the action space is the velocity vector of the virtual navigator, and considering the speed limit and motion limit, the output of the designed network contains [V, α, β], where the angle [α, β] is limited to [-πrad, πrad], thereby obtaining the velocity vector of the virtual navigator [Rcosα, Rsinαsinβ, Rsinαcosα];

[0011] The reward function is designed as follows:

[0012] R=R obstacle +R leader +R center

[0013] The reward function consists of three parts: R leader Used to reward the virtual navigator for approaching the destination, R center It is used to reward the cluster to reduce the distance between it and the virtual leader. obstacle Used to reward clusters for avoiding obstacles;

[0014] Design a neural network, which includes a Critic network and an Actor network. The Critic network first uses an LSTM layer to process the input, then uses two fully connected layers, and uses the Tanh layer as their activation function. The Actor network consists of an LSTM layer, a fully connected layer, and a Tanh layer. The output of the network is the mean of a normal distribution. The variance matrix of this distribution is C=c3I, where I is the identity matrix and c3 is a positive constant.

[0015] According to the reward function, the loss function is constructed as follows:

[0016]

[0017] In step (2), the cluster control algorithm consists of two parts: one is to control the UAVs in the cluster to maintain the distance between them, and the other is to control the cluster to track the virtual navigator. Specifically, the input of the UAV is as follows:

[0018]

[0019] In the formula, α agent represents any agent in the cluster, γ agent represents the virtual navigator, represents the interaction term between α agents, Represents the navigation feedback item of the virtual navigator to the cluster.

[0020] The beneficial effects of the present invention are:

[0021] 1. The present invention uses a cluster control algorithm to control the UAV cluster and track the virtual navigator, and uses a reinforcement learning algorithm PPO to control the virtual navigator, which can maintain the distance between the drones and track the virtual navigator.

[0022] 2. The present invention uses the reinforcement learning algorithm PPO to train the virtual navigator, designs the state space and action space of the reinforcement learning algorithm, and designs a reward function that can improve the adaptability and obstacle avoidance ability of the virtual navigator, so that the virtual navigator in the drone cluster can have learning and exploration capabilities.

[0023] 3. The present invention combines the cluster algorithm with the single-agent reinforcement learning algorithm to realize the intelligent obstacle avoidance of the drone cluster. The method controls the drones to perform cluster movement and track the navigator. The reinforcement learning algorithm controls the navigator in the cluster to complete intelligent obstacle avoidance and provide navigation information for the cluster drones. The detection information of the drone cluster on the environment provides the virtual navigator with environmental status information. The present invention can effectively improve the obstacle avoidance performance and consistency performance of the drone cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0025] Figure 1 It is a schematic diagram of the principle of the method of the embodiment of the present invention;

[0026] Figure 2 A schematic diagram of the structure of a network in an embodiment of the present invention;

[0027] Figure 3 It is a schematic diagram of a scenario of an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0029] like Figure 1-3 As shown, a UAV swarm intelligent obstacle avoidance method based on reinforcement learning includes the following steps:

[0030] Step 1: Cluster control method

[0031] In three-dimensional space, the motion model of the agent is as follows:

[0032]

[0033] where q i , p i , and u i ∈R n Represent the position, speed and control input vector of the i-th agent respectively. Each agent can only communicate with neighboring individuals within its communication range. The set of neighboring individuals of agent i at time t is expressed as follows:

[0034]

[0035] Where ||·|| represents the Euler distance, and r represents the maximum interaction radius between agents. The distance between each agent and its neighboring individuals in the cluster needs to satisfy the following constraints:

[0036]

[0037] Where d is a positive constant representing the minimum allowed distance between agents, and d≤r.

[0038] In an obstacle environment, the input of each agent in the multi-agent system is shown in formula (4).

[0039]

[0040] Among them, α and γ represent three types of agents. α agent represents any agent in the cluster, and γ agent represents the virtual navigator. represents the interaction term between α agents, Represents the navigation feedback item of the virtual navigator to the cluster. The specific definitions are as follows:

[0041]

[0042]

[0043] Aggregate agents in a multi-agent system and consists of two parts. The first part sets the distance between agents to the desired distance, and the second part makes the speed of an agent consistent with that of its neighboring agents. The expression of the first part is as follows:

[0044]

[0045]

[0046]

[0047] Where H,∈ α , and is a positive constant, the value of H is usually greater than, introducing ρ H (q i ) can effectively prevent cluster fragmentation. When the distance between agents increases, ρ H (q i ) will also increase rapidly.

[0048] The second component is ρ h (z) is an impulse function as follows:

[0049]

[0050] is to enable the agent to track the virtual navigator, and is a positive constant, q γ and p γ denote the position and velocity of the virtual navigator, σ1(q i -q γ ) and Q h The following are expressed:

[0051]

[0052]

[0053] Q h The purpose is to reduce the height difference between the agent and the navigator.

[0054] In order to apply the multi-agent swarm algorithm to drone swarms, it is necessary to establish the relationship between the drone motion model and the agent motion model. The simplified dynamic model of a fixed-wing drone is shown below.

[0055]

[0056] where [x i ,y i ,h i ],V i , ψ i ,and They represent the position, speed, heading angle, and altitude change rate of the i-th UAV respectively. and Represent the speed, heading and altitude input commands of the corresponding autopilot. ψ , τ v and are positive time constants corresponding to the input command heading angle, velocity and altitude. These four parameters are preconfigured positive constants depending on the state of the autopilot.

[0057] The motion of the drone in three-dimensional space, its position, velocity and control vector are given by q i =[x i ,y i ,h i ] T , and u i =[u x，i ,u y，i ,u h，i ] T Given by u i ,get:

[0058]

[0059]

[0060] Substituting equations (14) and (15) into the simplified UAV model (13), we can obtain:

[0061]

[0062]

[0063]

[0064] Step 2: Design of reinforcement learning PPO

[0065] The optimization objective function of the PPO algorithm Actor network is expressed as:

[0066]

[0067] Where c1, c2 are coefficients, S represents entropy, Represents the squared error loss L CLIP (θ) represents in is the generalized advantage estimate.

[0068] The optimization objective function of the Critic network is expressed as:

[0069]

[0070] The PPO algorithm has two networks, namely the Critic network and the Actor network. Figure 1 As shown in Figure 2, the Critic network structure first uses an LSTM layer to process the input, then uses two fully connected layers (FC) and uses the Tanh layer as their activation function.

[0071] The Actor network consists of a neural network and a normal distribution. The Actor network consists of an LSTM layer, an FC layer, and a TanH layer. The output of the network is the mean of a normal distribution. The variance matrix of this distribution is C=c3I, where I is the identity matrix and c3 is a small positive constant. The behavior is obtained from this distribution. The output of the Actor is mainly to obtain the velocity vector of the navigator. For this purpose, the output of the Actor network needs to be designed. Considering the projection of the radius of the ball on the three-dimensional axis as the velocity vector, it is assumed that the output of the Actor network is the radius R of the ball and two angles (α, β). α is the angle between the radius of the ball and the z-axis, and β is the angle between the projection of the radius on the xy plane and the x-axis. The velocity vector [Rcosα, Rsinα sinβ, Rsinαcosα] is obtained. Considering the movement restrictions of the drone, R is limited to a certain range and the angle is limited to [-πrad, πrad]. The mean of R and (α, β) output by the Actor network both use TanH as the activation function.

[0072] Step 3: Design of intelligent obstacle avoidance method for drone swarm

[0073] The PPO algorithm is used to train and control the virtual navigator, and the distance between the aircraft and the navigator are maintained based on the cluster control algorithm. The navigator is a particle composed of a position vector and a velocity vector, such as Figure 2 As shown. The navigator controlled by the reinforcement learning algorithm has three main purposes: one is to make the virtual navigator approach the target area, the second is to enable the cluster to avoid obstacles, and the third is to enable the cluster to follow the virtual navigator and reduce the distance between it and the navigator. Based on these purposes, the reward function is constructed as follows:

[0074] R=R obstacle +R leader +R center (twenty one)

[0075] Where R leader is to reward the navigator for being close to the destination, R center is to reward the cluster center for closing the distance with the leader, R obstacleThe purpose is to reward the cluster center for avoiding obstacles. Based on the purpose of the algorithm and the composition of the reward function, the state space of the algorithm includes the position of the navigator, the center position of the drone cluster and the distance vector between it and the obstacle, and the action space of the algorithm is the speed of the navigator. In order to simplify the obstacle avoidance process, the obstacles are simplified to spherical obstacles.

[0076] The intelligent obstacle avoidance algorithm for drone swarms is composed of two algorithms. One is the swarm control algorithm introduced in step 1, that is, the input of each drone is The second is the reinforcement learning algorithm, in which the navigator integrates the environmental detection information of the swarm drones to complete obstacle avoidance and path planning.

[0077] In summary, the virtual navigator in the intelligent obstacle avoidance method of the drone swarm of the present invention has the ability to learn and explore, so that the drone swarm has better obstacle avoidance and navigation capabilities in unknown and complex environments. The drone swarm needs to detect the environment and maintain the distance between the drones and track the navigator based on the cluster control algorithm. The virtual navigator avoids obstacles and pre-plans the path by integrating the information detected by the cluster. The interaction between drones in the cluster is relatively simple, so that the drone swarm has better consistency when avoiding obstacles.

Claims

1. A UAV swarm intelligent obstacle avoidance method based on reinforcement learning, characterized in that: The drones in the cluster keep moving in cluster and track the virtual navigator. The drones do not have the ability to avoid obstacles. They transmit the detection information of the environment to the virtual navigator. The virtual navigator is trained using the reinforcement learning algorithm. The virtual navigator provides navigation guidance information for the cluster drones to realize intelligent obstacle avoidance of the drone cluster. The following steps are included: (1) Set up a control algorithm for the virtual navigator in the UAV cluster, with a speed limit of 10m / s to 100m / s. Set a target point for the virtual navigator to provide navigation information for the cluster. Use a reinforcement learning algorithm to control the virtual navigator. The detection information of the UAV on the environment provides state information for the virtual navigator, which is used as the state input when training the virtual navigator. The output is the velocity vector of the virtual navigator, which is used as the action space of the reinforcement learning algorithm. The position feature of the cluster is replaced by the center position of the cluster. Based on this, the state space is designed, including the relative position of the virtual navigator and the cluster, the relative position between the virtual navigator and the target, and the relative position between the cluster and the obstacle. Among them, the action space is the velocity vector of the virtual navigator. The reward function is designed as follows: R=R obstacle +R leader +R center The reward function consists of three parts: R leader Used to reward the virtual navigator for approaching the destination, R center It is used to reward the cluster to reduce the distance between it and the virtual leader. obstacle Used to reward clusters for avoiding obstacles; Design a neural network, which includes a Critic network and an Actor network. The Critic network first uses an LSTM layer to process the input, then uses two fully connected layers, and uses the Tanh layer as their activation function. The Actor network consists of an LSTM layer, a fully connected layer, and a Tanh layer. The output of the network is the mean of a normal distribution. The variance matrix of this distribution is C=c3I, where I is the identity matrix and c3 is a positive constant. (2) The UAV cluster tracks the virtual navigator, completes obstacle avoidance, provides environmental information to the navigator, maintains the distance between drones, and realizes intelligent obstacle avoidance of the UAV cluster.

2. The method for intelligent obstacle avoidance of drone swarm based on reinforcement learning according to claim 1 is characterized in that: In step (2), the cluster control algorithm consists of two parts: one is to control the UAVs in the cluster to maintain the distance between them, and the other is to control the cluster to track the virtual navigator. Specifically, the input of the UAV is as follows: In the formula, α agent represents any agent in the cluster, γ agent represents the virtual navigator, represents the interaction term between α agents, Represents the navigation feedback item of the virtual navigator to the cluster.

Images