A Collaborative Obstacle Avoidance Method for UAV Swarms Based on Deep Reinforcement Learning

Abstract

This invention discloses a collaborative obstacle avoidance method for UAV swarms based on deep reinforcement learning. The invention uses the obstacle avoidance results of each UAV in the swarm as the initial obstacle avoidance result, and then trains the entire swarm as a whole using deep reinforcement learning to obtain a more complete result. The reward function in this invention includes the distance between the UAV swarm and the target position after obstacle avoidance, the degree of formation recovery after obstacle avoidance, and the difference between the obstacle avoidance direction of the UAV that determined the initial obstacle avoidance direction and its initial obstacle avoidance direction, referred to as the difference degree between the two obstacle avoidance decisions. This invention continuously fits the globally optimal obstacle avoidance strategy and executes the best obstacle avoidance action at each stage, thereby maximizing the overall obstacle avoidance performance of the UAV swarm.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a collaborative obstacle avoidance method for drone swarms based on deep reinforcement learning. Background Technology

[0002] Reinforcement learning algorithms are algorithms that enable intelligent agents to maximize the reward value they receive when interacting with complex and uncertain environments.

[0003] Deep reinforcement learning is an end-to-end reinforcement learning method that does not require manual feature design and only requires input states to enable the system to directly output actions; it typically uses neural networks to fit the value function or policy network.

[0004] Markov decision processes are mathematical models of sequential decision-making used to simulate stochastic policies and rewards that agents can implement in environments where the system state has Markov properties.

[0005] Q-learning refers to an agent interacting with its environment and receiving rewards, with the goal of using the best policy to select actions to maximize rewards. During the learning process, the agent updates its Q-learning table; at the end of the round, the task is completed and the goal is achieved.

[0006] DQN stands for Deep Q-Network. To make the agent training process more stable, a replay buffer is introduced, which can remember the behaviors experienced by the agent. Then, random samples in the replay buffer are used for training, which can reduce the correlation between the agent's experiences and help the agent learn from a wider range of experiences.

[0007] The application of unmanned aerial vehicle (UAV) technology has played a significant role in civilian fields such as exploration, firefighting, and disaster relief, as well as other areas. UAVs offer advantages such as low personnel casualties and low cost, and can replace manned aircraft in performing harsh, dangerous, and monotonous tasks. However, individual UAVs suffer from low efficiency and poor robustness due to limitations in computing power and battery energy. In contrast, UAV swarms, with their cooperative control and complementary capabilities, greatly improve efficiency and robustness in mission execution.

[0008] However, most current research on UAV swarm missions employs rule-based methods, lacking intelligent autonomous decision-making capabilities, making it difficult for swarms to adapt to the complex and ever-changing battlefield environment. In recent years, reinforcement learning has made significant breakthroughs in solving complex sequential problems. Applying reinforcement learning to collaborative obstacle avoidance decision-making in UAV swarms can improve the flexibility and intelligence of obstacle avoidance decisions.

[0009] Currently, the most common methods for drone swarm cooperative formation are the leader-follower method and the particle swarm optimization method. The leader-follower method employs a wingman-follower cooperative formation approach, offering advantages such as simple mathematical analysis, ease of maintaining formation, and low communication pressure. The particle swarm optimization method determines the next position by comprehensively considering current position, social cognition, and individual cognition, boasting advantages such as fast search speed, high efficiency, and simple algorithm. The artificial potential field method is a commonly used algorithm for obstacle avoidance in drone swarms. This algorithm treats the area of ​​object movement as a potential field. The gravitational potential field is directly proportional to the distance; the greater the distance, the greater the potential energy of the gravitational potential field between objects. Conversely, the repulsive potential field is inversely proportional to the distance; the smaller the distance, the smaller the potential energy of the gravitational potential field between objects. This method has the advantages of simple mathematical calculation and ease of analysis. However, this method, which artificially sets distance thresholds, leads to avoidance measures being taken when obstacle avoidance is not required, increasing the distance the drones need to detour.

[0010] The ALFPSO algorithm is a cooperative obstacle avoidance algorithm for UAV swarms based on the artificial potential field method and the LFPSO algorithm. The LFPSO algorithm combines the leader-follower method and particle swarm optimization, assuming that each particle only has social cognition, not individual cognition. Social cognition represents the globally optimal solution, i.e., the lead aircraft in the leader-follower formation method. For example... Figure 1 As shown, the technical implementation of the LFPSO algorithm is as follows: First, the UAV flight area is initialized. Gravitational and repulsive fields are set around each UAV and obstacles within the flight area. The initial positions of the lead UAV and the target UAV are set, while the initial positions of the wingmen are randomly generated within the flight area. The lead UAV starts from its initial position and flies towards the target UAV under the influence of the gravitational potential field of the target point. The wingmen move towards the lead UAV under the influence of the lead UAV's gravitational field. During flight, both the lead UAV and the wingmen are subjected to repulsive forces from obstacles, thus avoiding them. The UAV swarm, through the combined force of the gravitational and repulsive forces, continuously approaches the target point while avoiding obstacles, ultimately completing the obstacle avoidance mission.

[0011] The artificial potential field method has inherent flaws. During obstacle avoidance by drones, it struggles to reach the target point and is prone to oscillations when navigating narrow passages between obstacles. This method applies a repulsive force to the agent by calculating whether the current relative distance between the agent and the obstacle is less than a set safety threshold, without pre-judging whether a collision will occur. When the distance is less than the safety threshold, an avoidance tactic may be taken even if a collision doesn't happen, increasing the distance the agent has to travel to avoid obstacles. Furthermore, the pilot-follower method lacks real-time online path planning and autonomous obstacle avoidance for the pilot drone, resulting in suboptimal drone positions after swarm obstacle avoidance and poor formation recovery. The particle swarm optimization algorithm is poor at handling discrete optimization problems and is prone to getting trapped in local optima. Summary of the Invention

[0012] To address the problems existing in the prior art, the present invention aims to provide a collaborative obstacle avoidance method for UAV swarms based on deep reinforcement learning. This invention uses the obstacle avoidance results of each UAV in the swarm as the initial obstacle avoidance result, and then trains the entire swarm as a whole using deep reinforcement learning to obtain a more complete result. The reward function in this invention includes the distance between the UAV swarm and the target position after obstacle avoidance, the degree of formation recovery after obstacle avoidance, and the difference between the obstacle avoidance direction of the UAV that determined the initial obstacle avoidance direction and its initial obstacle avoidance direction, referred to as the difference degree between the two obstacle avoidance decisions.

[0013] The technical solution of this invention is as follows:

[0014] A method for cooperative obstacle avoidance in a drone swarm based on deep reinforcement learning, comprising the following steps:

[0015] 1) The status input information is obtained from the external environment through the onboard sensors of each UAV in the UAV swarm and sent to the server; the status input information includes the formation status of the UAV swarm and the onboard sensor information;

[0016] 2) The server establishes a Markov model of the interaction between the UAV cluster and the obstacle avoidance environment based on the state input information, and constructs the state space and action space in the Markov model, including the discretization of the continuous state space of the UAV cluster, the state space of the UAV cluster, and the action space of the UAV cluster.

[0017] 3) The server constructs the state transition probability function, reward function, and discount factor in the Markov model, including the UAV swarm state transition function and the distance-reward function R. c1 Recovery reward function R c2 and obstacle avoidance selection reward function R c3 ; Reward function R c =∈1R1+∈2R2+∈3R3+∈4R4, R1=W R1 R c1 R3 = W R2 R c2 R4 = W R3 R c3 W R1 W R2 W R3 R² is the penalty factor, R2 is the penalty value for drone collisions in the cluster, and ∈1, ∈2, ∈3, ∈4 are coefficients; according to Difference settings for obstacle avoidance selection reward function R c3 ;

[0018] 4) The server, based on a Markov model, uses deep reinforcement learning to train each UAV in the UAV cluster to perform autonomous obstacle avoidance tasks in the current environment, thereby obtaining a set of obstacle avoidance matrices. in, This includes the state transition probabilities obtained by drone i performing actions in action space A under each state space S. That is, the obstacle avoidance value corresponding to drone i; based on the obstacle avoidance matrix set The obstacle avoidance values ​​corresponding to each drone can be used to identify drones with non-fixed obstacle avoidance directions.

[0019] 5) The server, based on a Markov model, uses deep reinforcement learning to train the UAV swarm to perform autonomous obstacle avoidance decision-making in the current environment, thereby obtaining a swarm obstacle avoidance matrix. in, It includes the state transition probability obtained by the drone swarm as a whole node performing actions in the action space in the j-th state space;

[0020] 6) The server is based on the result obtained in step 4). Step 5) The difference yields the obstacle avoidance choice reward function R. c3 Based on the settings in step 3), the drones with non-fixed obstacle avoidance directions determined in step 4) are trained on autonomous obstacle avoidance decision-making tasks using deep reinforcement learning methods to obtain the obstacle avoidance direction corresponding to each drone at the next moment.

[0021] 7) The server sends the obstacle avoidance direction of each drone in the drone cluster to the corresponding drone at the next moment.

[0022] Furthermore, the obstacle avoidance selection reward function in, This represents the initial obstacle avoidance probability when the i-th drone in the cluster performs action j. Let M represent the obstacle avoidance probability of the cluster executing action j, M represent the total number of drones in the cluster, and N represent the total number of action spaces.

[0023] Furthermore, the distance reward function The recovery reward function Where uD is the target location. Let n be the position of drone i at time c, and n be the total number of drones. U represents the position of the centroid of the UAV swarm at time c. i Let u0 represent the initial position of drone i, and u0 represent the initial position of the centroid of the drone cluster.

[0024] Furthermore, in step 6), the training method for the UAV with the non-fixed obstacle avoidance direction determined in step 4) to perform autonomous obstacle avoidance decision-making is as follows:

[0025] 61) Select two deep neural networks with the same structure, one as the main network, used to determine the state D at the current time c. c and the action taken A c Estimate the Q value, i.e., Q(D) c A c ;θ i ), θ i One is the parameter of the current main network; the other is the parameter of the target network, based on the target state D at the next time step c+1. c+1 and historical parameters θ - The recursive formula for calculating the Q-value of the target state-action sequence is as follows:

[0026]

[0027] Where R c Let γ be the reward value at the current time c, and γ be the discount factor; set the step size to k and the empty replay buffer F, initialize the environment, and acquire the observation data O0. in It is the initial position of the i-th drone in the cluster. It is the initial velocity of the i-th drone in the cluster. is the initial yaw angle of the i-th UAV in the cluster, M(P)0 is the initial lateral obstacle avoidance direction matrix of the UAV cluster; initialize the sequence D0={O0} and preprocess the sequence φ0=φ(D0);

[0028] 62) After running k steps according to the set step size, replace the parameters in the target network with the parameters in the main network, i.e., θ. i →θ - ;

[0029] 63) An Epsilon-greedy strategy is used for action selection. When the drone's obstacle avoidance probability is ε, an action is randomly selected from the action space; when the drone's obstacle avoidance probability is 1-ε, the action with the largest Q value, i.e., P, is selected from the action space. max =argmax A Q(φ(D c ), A; θ); Calculate the optimal action value function of the drone swarm.

[0030] 64) Calculate the root mean square error between the current estimated Q-value and the target Q-value of the main network. As the loss function value, gradient descent is used to adjust the parameter θ. i Perform the update;

[0031] 65) Perform the action and obtain observation data O c+1 And a reward value R c Update the drone swarm policy table; set D c+1 ={D c A c O c+1} and perform preprocessing φ c+1 =φ(D c+1 ); transfer state transition data (φ) c A c R c , φ c+1 ) to cache F.

[0032] Furthermore, the formation status includes: the position of each drone in the drone swarm, the speed of the drone, the distance of the drone from the target point, and the average distance of each drone in the drone swarm from the target point; the airborne sensor information includes: the position of the obstacle, the length and width of the obstacle.

[0033] A server is characterized by comprising a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the methods described above.

[0034] A computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the steps of the above-described method.

[0035] Typically, intelligent obstacle avoidance methods for drone swarms treat each drone as an autonomous decision-making node. However, some drones in the swarm, due to their specific positions within the formation, can perform obstacle avoidance flight in more than one direction. However, this multiple obstacle avoidance direction can affect the formation's recovery efficiency, potentially causing the swarm to fall short of the optimal distance to the target point after re-establishing formation, thus preventing the drone swarm from achieving optimal obstacle avoidance performance. Therefore, performing secondary obstacle avoidance decision-making on the drone swarm can help it achieve better obstacle avoidance performance. Secondary obstacle avoidance decision-making refers to re-determining the obstacle avoidance strategies of drones capable of performing obstacle avoidance flight in more than one direction through the overall obstacle avoidance decision-making of the swarm, thereby achieving optimal efficiency in the swarm's obstacle avoidance mission.

[0036] like Figure 2 As shown, each drone in the cluster can be viewed as an autonomous decision-making node, and each node can be modeled as a Markov decision process. The process by which the environment sends a state S0 and a feedback reward value R0 to each node, and the node makes a corresponding action A0, is called the autonomous obstacle avoidance strategy π0. The initial obstacle avoidance probability is obtained through training. When the initial obstacle avoidance probability or At this time, the drone is considered to have a fixed obstacle avoidance direction, which is converted into an initial obstacle avoidance value. And store the initial obstacle avoidance matrix. When the initial obstacle avoidance probability At this time, the drone is considered to have more than one obstacle avoidance direction, and will execute the swarm secondary obstacle avoidance strategy π1 to obtain the best obstacle avoidance action A. t Secondary obstacle avoidance by UAV swarms can optimize the efficiency of swarm obstacle avoidance tasks. This efficiency includes: the distance R1 between the UAV swarm and the target position after obstacle avoidance; the formation recovery degree R2 after obstacle avoidance; and the difference between the obstacle avoidance direction of the UAV that determined its initial obstacle avoidance direction and its initial obstacle avoidance direction after the secondary obstacle avoidance decision, referred to as the difference degree R3 between the two obstacle avoidance decisions. The secondary obstacle avoidance process of UAV swarms can be modeled as a Markov decision process, where the reward values ​​include: the distance R1 from the swarm to the endpoint, the formation recovery degree R2, and the difference degree R3 between the two obstacle avoidance decisions. This is achieved through the interaction between the environment and the UAV swarm: the environment sends the state environment S1 and feedback reward values ​​R1 and R2 to the swarm, and the swarm makes corresponding actions A. t .

[0037] like Figure 3 As shown, the drone swarm is treated as a unified intelligent agent that interacts with the environment. The current state s, the action a, and network parameters θ are input into the Q-network to obtain the Q-value Q(s, a; θ). The selection of action a employs a greedy strategy, either choosing the action with the highest Q-value or selecting a random action. Simultaneously, the reward value r for the next state s′ and the action a for state s are placed into a replay buffer. Then, s′, a′, and θ are sampled in batches from the buffer pool. - The input is fed into the target Q-network, based on max′ a Q(s′,a′;θ - The target value is calculated, and the loss is calculated between it and the current Q value Q(s, a; θ). Backpropagation is used to update the parameters θ, making the current Q value closer to the target Q value. Every C steps, the parameters of the current Q network are copied to the target Q network; the obstacle avoidance strategy π1 is continuously fitted and updated. In other words, the drone swarm executes the best action argmax through interaction with the environment. a Q(s, a; θ), where θ is the Q-learning update parameter, is the state at the current time step s. After selecting action a from the current state s, the next state s′ is obtained; or a random action (s, a) is selected from the replay buffer D and entered into the target Q-network, and the stored state transition data (s, a, r; s′) is stored in the replay buffer D; a small batch of state transition data s′ is randomly sampled from the replay buffer D and added to the target Q-network, with max′ set. a Q(s′,a′;θ -The algorithm calculates the DQN loss and performs a gradient descent step on θ, replicating the target Q network every C steps. It continuously fits the state-action value function to update the obstacle avoidance strategy π1, and finally obtains the globally optimal action to obtain the maximum reward value, which is the obstacle avoidance action of the drone swarm with the highest overall efficiency.

[0038] The advantages of this invention are as follows:

[0039] This application focuses on the obstacle avoidance performance of UAV swarms, specifically the distance UAVs travel to the target point after obstacle avoidance and the recovery rate of the swarm formation. It designs a UAV reinforcement learning-based obstacle avoidance method. By jointly determining the final obstacle avoidance strategy through individual UAV obstacle avoidance strategies and the swarm's obstacle avoidance strategy, this method addresses the problem of over-reliance on the lead UAV and the lack of UAV intelligence in the leader-follower approach. Furthermore, based on a deep Q-learning network, the obstacle avoidance strategy of the UAV swarm at each stage continuously fits to the globally optimal strategy, executing the best obstacle avoidance action at each stage, thus maximizing the overall obstacle avoidance performance of the UAV swarm. Attached Figure Description

[0040] Figure 1 This is a flowchart of the implementation scheme of the existing technology.

[0041] Figure 2 This is a flowchart of the secondary obstacle avoidance process for a drone swarm based on deep reinforcement learning.

[0042] Figure 3 A diagram illustrating the DQN update.

[0043] Figure 4 A schematic diagram illustrating obstacle avoidance in a drone swarm formation.

[0044] Figure 5 The flowchart is for an example. Detailed Implementation

[0045] The present invention will now be described in further detail with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0046] The implementation process of the embodiments of the present invention is as follows: Figure 5 As shown, the specific steps are as follows:

[0047] Step 1: Obtain status input from the external environment through the UAV's onboard sensors: including the UAV swarm formation status and onboard sensor information, and send it to the server.

[0048] A diagram illustrating obstacle avoidance in drone swarm formations is shown below. Figure 4As shown: This is based on a two-dimensional UAV swarm obstacle avoidance environment model, where the UAV swarm formation state depends on the formation form. In this embodiment, a triangular formation of 10 UAVs is used. The formation state includes: the position of the UAVs in the formation, the speed of the UAVs in the formation, the distance of each UAV from the target point, and the average distance of the UAV swarm from the target point. The UAV swarm moves in the scene, and the onboard radar obtains the position information of the UAVs, the onboard speedometer obtains the speed information of the UAVs, and the onboard infrared sensor collects obstacle information, i.e., the obstacle tree state includes: the position of the obstacle, the length and width of the obstacle. In this embodiment, the obstacle is represented as a two-dimensional tree, and its position is directly in front of the UAV swarm.

[0049] Step 2: Construct the state space and action space in the Markov model, including: discretization of the continuous state space of the UAV swarm, the state space of the UAV swarm, and the action space of the UAV swarm.

[0050] Establish a Markov model of the interaction between the UAV swarm and the current obstacle avoidance environment. Based on the action decision-making process of the UAV swarm obstacle avoidance, model the five-tuple (S, A, R, p, γ) of the Markov decision process.

[0051] Construct the state space of the drone swarm, S, as follows:

[0052] S=(s i =(u i v i , ψ i ), M(P));

[0053] Where u i v represents the current position of the i-th drone. i Let ψ represent the current velocity of the i-th drone in the chosen action direction. i Let represent the yaw angle of the i-th UAV, and M(P) represent the matrix of the UAV swarm's lateral obstacle avoidance direction, where the values ​​range from [0, 1].

[0054] The Gaussian membership function is selected to define the continuous UAV swarm state space S = (s i =(u i v i Discretize M(P) and choose the Gaussian membership function as:

[0055]

[0056] Where U represents the number of states of the UAV swarm in continuous space, P represents the number of states of the UAV swarm after mapping to discrete space, and c ij , σ ijThese represent the center and width of the Gaussian membership function, respectively; using the selected Gaussian membership function, for the UAV swarm state S... i Use the Gaussian function f j (S i Calculate the state S of the drone swarm i The membership degree value is selected from the discrete state corresponding to the maximum membership degree, which is represented as: The discrete set of drone swarm states is

[0057] Construct the drone swarm action space, where drone swarm action space A is:

[0058]

[0059] Treating each drone in the swarm as a point mass, each drone moves forward in a straight line at a constant speed in space, and the yaw angle ψ is changed... i To execute the actions of a drone swarm on a two-dimensional plane, via ψ i The action in the action space is implemented by changing the velocity offset of the forward flight speed. In this specific implementation example, only... and ψ i The trigonometric function values ​​serve as the action space for the drone swarm, representing the drone swarm's movement to the left and right to avoid obstacles.

[0060] Step 3: Construct the state transition probability function, reward function, and discount factor in the Markov model, including: the drone swarm state transition function, distance reward function, resilience reward function, and obstacle avoidance selection reward function.

[0061] The state transition probability function describes the probability that a drone swarm will choose a certain action from its current state to transition to the next state in an environmental model. The value of the transition probability is determined by the obstacle avoidance probability. Represents obstacle avoidance probability. The range of values ​​is This represents the probability that the drone will move to the left, with a value ranging from [0, 1]. This represents the probability of the drone moving to the right, with values ​​ranging from [0, 1], and is the sum of the probabilities of moving to the left and right.

[0062] The reward function R describes the state of the drone swarm at the current moment, representing the immediate feedback obtained after selecting a certain action, and signifies the reward for the corresponding state-action pair. Based on the drone swarm's location and the target's location, the distance reward function can be defined as:

[0063]

[0064] Where uD is the target location. Let be the position of UAV i at time c. Based on the initial centroid position of the UAV cluster, the initial UAV position, the current centroid position of the UAV cluster, and the current UAV position, the restoring reward function can be defined as:

[0065]

[0066] in This represents the position of UAV i at time c. U represents the position of the centroid of the UAV swarm at time c. i Let ui represent the initial position of UAV i, and u0 represent the initial position of the UAV swarm's centroid; based on the initial obstacle avoidance matrix in the UAV swarm's state space. Obstacle avoidance matrix at the same time as drone swarm

[0067] The obstacle avoidance selection reward function is the initial set of obstacle avoidance matrices trained on an autonomous obstacle avoidance task with policy π0. pass Obstacle avoidance probability matrix for each drone and cluster obstacle avoidance probability matrix The difference is set as the obstacle avoidance selection reward function, which can be defined as follows:

[0068]

[0069] in Let represent the initial obstacle avoidance probability of the i-th UAV in the cluster performing action j during autonomous obstacle avoidance task training with strategy π0. Let M represent the obstacle avoidance probability of the cluster executing action j, M represent the total number of drones in the cluster, and N represent the total number of action spaces.

[0070] Based on the above returns, the return function R c Defined as:

[0071] R1 = W R1 R c1

[0072] R2 = -15

[0073] R3 = W R2 R c2

[0074] R4 = W R3 R c3

[0075] R c =∈1R1+∈2R2+∈3R3+∈4R4

[0076] Among them W R1 W R2 W R3 R² is the penalty factor, which represents the penalty value for collisions between drones in the cluster, including collisions between drones in the cluster and obstacles in the current environment, as well as collisions between drones in the cluster and other drones in the cluster. ∈1, ∈2, ∈3, and ∈4 are coefficients, and R... c This represents the reward value after the drone cluster performs an action at time c.

[0077] The discount factor γ is used to describe the cumulative impact of the obstacle avoidance action decision at a given moment on the immediate future reward function during the obstacle avoidance process of a drone swarm.

[0078] Step 4: Each drone in the drone swarm performs autonomous obstacle avoidance training under policy π0 in an environment with the current obstacle tree and puts the initial obstacle avoidance values ​​into the initial obstacle avoidance matrix set.

[0079] Each drone in the drone swarm is treated as a node, and each node performs autonomous obstacle avoidance training with policy π0 in the current environment. Each node completes the obstacle avoidance task and obtains an initial obstacle avoidance probability matrix through autonomous obstacle avoidance decisions. Where i represents the number of drone sorties. This includes the state transition probabilities obtained by drone i performing actions in action space A under each state space S. That is, the initial obstacle avoidance value. Figure 4 The first two and last two columns of the drone swarm from left to right have relatively fixed obstacle avoidance directions, while the other drones in the middle can perform obstacle avoidance flight in more than one direction.

[0080] Treating each drone in the cluster as an autonomous node, an initial obstacle avoidance value set is obtained through intelligent obstacle avoidance methods. And recorded in the initial obstacle avoidance matrix i represents the number of drone sorties. The initial obstacle avoidance probability is composed of the current state and the state transition probability of the currently selected action. It consists of left obstacle avoidance confidence and right obstacle avoidance confidence; where i is the number of drone flights. When drone i has an initial obstacle avoidance probability... At that time, the initial obstacle avoidance direction of drone i is left; when the initial obstacle avoidance probability of drone i is... At that time, the initial obstacle avoidance direction of UAV i is to the right.

[0081] Step 5: Train the UAV swarm to perform autonomous obstacle avoidance decision-making based on strategy π1 in an environment with current obstacles, and add the swarm obstacle avoidance values ​​to the swarm obstacle avoidance matrix set.

[0082] The entire drone swarm is treated as a single node, which performs autonomous obstacle avoidance training with policy π1 in the current environment. Through autonomous obstacle avoidance decisions, the drone swarm completes the obstacle avoidance task and obtains a set of swarm obstacle avoidance probability matrices. It includes the state transition probabilities obtained by the drone swarm as a whole node performing actions in the action space in each state space, where The state transition probability is obtained by the drone swarm as a whole node performing actions in the j-th state space.

[0083] Cluster obstacle avoidance probability matrix The probability range is the same as the initial obstacle avoidance probability of the drone in step 4. When the cluster obstacle avoidance probability is (0.6-1, 0-0.4), the obstacle avoidance direction of the drone cluster node is to the left. When the cluster obstacle avoidance probability is (0-0.4, 0.6-1), the obstacle avoidance direction of the drone cluster node is to the right.

[0084] Step 6: Training the drone autonomous obstacle avoidance decision-making task of strategy π2 for drones in the drone swarm that do not have a fixed obstacle avoidance direction.

[0085] The training process of the drone execution strategy π2 in a drone swarm with non-fixed obstacle avoidance direction can be modeled as a deep Q-learning network, considering the drone swarm state space and action space in step 2, and the hybrid design reward function in step 3. Two deep neural network structures are used, one as the main network, used to adjust the current state D. c and the action taken A c Estimate the Q value, i.e., Q(D) c A c ;θ i ), θ i One is the parameter of the current main network; the other is the parameter of the target network, based on the target state D. c+1 and historical parameters θ - The recursive formula for calculating the Q-value of the target state-action sequence is as follows:

[0086]

[0087] Where R c Let be the reward value at the current moment, and γ be the discount factor. Set the step size to k and the empty replay buffer F, initialize the environment, and acquire the observation data O0. in It is the initial position of the i-th drone in the cluster. It is the initial velocity of the i-th drone in the cluster. Let be the initial yaw angle of the i-th UAV in the cluster, and M(P)0 be the initial lateral obstacle avoidance direction matrix of the UAV cluster. Initialize the sequence D0 = {O0} and preprocess the sequence φ0 = φ(D0);

[0088] After running k steps according to the set step size, replace the parameters in the target network with the parameters from the main network, i.e., θ. i →θ - The action space A of the drone swarm s The obstacle avoidance direction is left or right. An Epsilon-greedy strategy is used for action selection: when there is a probability of ε, an action is randomly selected from the action space; when there is a probability of 1-ε, the action with the largest Q value (P) is selected from the action space. max =argmax A Q(φ(D c The optimal action value function of a drone swarm is defined as:

[0089]

[0090]

[0091] According to the Bellman equation for Q-values, in neural network training, the mean squared error between the current main network's estimated Q-value and the target Q-value is defined as the loss function:

[0092]

[0093] During the iteration process, gradient descent is used to adjust the parameter θ. i Perform the update;

[0094] Perform actions and obtain observation data O c+1 And a reward value R c Update the drone swarm policy table. Set D c+1 ={D c A c O c+1} and perform preprocessing φ c+1 =φ(D c+1 ); transfer state transition data (φ) c A c R c , φ c+1 The training process involves randomly collecting small batches of quadruples from the replay cache F. When training the target network is required, the training speed is accelerated by randomly collecting small batches of quadruples from the replay cache F.

[0095] Complete the training process in step 5 according to the set number of steps k. When the deep Q-network model obtains a relatively convergent result, end the training and record the model parameters; otherwise, continue iterating the training process in step 5. Change the initialized model parameters and train multiple times in the obstacle environment model in step 1 to select the best model parameters so that the drone swarm achieves the best obstacle avoidance performance. This completes the obstacle avoidance task of the drone swarm.

[0096] Step 7: The server sends the obstacle avoidance direction of each drone in the drone cluster to the corresponding drone at the next moment.

[0097] This invention utilizes deep reinforcement learning in the training of drone swarm obstacle avoidance tasks, performing multiple training sessions with a single drone as a node and the drone swarm as a whole as a node. Strategy π0 involves single-node training for obstacle avoidance by each drone in the swarm; strategy π1 involves training the drone swarm as a single node.

[0098] The obstacle avoidance scenarios applicable to drones of this invention include two-dimensional and three-dimensional scenarios.

[0099] In this invention, the action space for reinforcement learning can be set to three directions, not limited to the two directions described. The three directions are... Move left, stay still, or move right.

[0100] The problem in this invention can be modeled in a continuous action space, not limited to a discrete action space.

[0101] The drone swarm training method described in this paper is illustrated using the deep Q-network method as an example. It can be replaced by optimized methods such as the actor-commentator method, the asynchronous advantage actor-commentator method, or the generative adversarial network model.

[0102] Although specific embodiments of the invention have been disclosed for illustrative purposes to aid in understanding and implementing the invention, those skilled in the art will understand that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, the invention should not be limited to the content disclosed in the preferred embodiments, and the scope of protection claimed by the invention is defined by the claims.

Claims

1. A method for cooperative obstacle avoidance in a drone swarm based on deep reinforcement learning, comprising the following steps: 1) The status input information is obtained from the external environment through the onboard sensors of each UAV in the UAV swarm and sent to the server; the status input information includes the formation status of the UAV swarm and the onboard sensor information; 2) The server establishes a Markov model of the interaction between the UAV cluster and the obstacle avoidance environment based on the state input information, and constructs the state space and action space in the Markov model, including the discretization of the continuous state space of the UAV cluster, the state space of the UAV cluster, and the action space of the UAV cluster. 3) The server constructs the state transition probability function, reward function, and discount factor in the Markov model, including the UAV swarm state transition function and the distance-reward function R. c1 Recovery reward function R c2 and obstacle avoidance selection reward function R c3 ; Reward function R c =∈1R1+∈2R2+∈3R3+∈4R4, R1=W R1 R c1 R3 = W R2 R c2 R4 = W R3 R c3 W R1 W R2 W R3 R² is the penalty factor, R2 is the penalty value for drone collisions in the cluster, and ∈1, ∈2, ∈3, ∈4 are coefficients; according to Difference settings for obstacle avoidance selection reward function R c3 ; 4) The server, based on a Markov model, uses deep reinforcement learning to train each UAV in the UAV cluster to perform autonomous obstacle avoidance tasks in the current environment, thereby obtaining a set of obstacle avoidance matrices. in, This includes the state transition probabilities obtained by drone i performing actions in action space A under each state space S. That is, the obstacle avoidance value corresponding to drone i; based on the obstacle avoidance matrix set The obstacle avoidance values ​​corresponding to each drone in the data can be used to identify drones with non-fixed obstacle avoidance directions; 5) The server, based on a Markov model, uses deep reinforcement learning to train the UAV swarm to perform autonomous obstacle avoidance decision-making in the current environment, thereby obtaining a swarm obstacle avoidance matrix. in, It includes the state transition probability obtained by the drone swarm as a whole node performing actions in the action space in the j-th state space; 6) The server, according to the result obtained in step 4), Step 5) The difference yields the obstacle avoidance choice reward function R. c3 Based on the settings in step 3), the drones with non-fixed obstacle avoidance directions determined in step 4) are trained on autonomous obstacle avoidance decision-making tasks using deep reinforcement learning methods to obtain the obstacle avoidance direction corresponding to each drone at the next moment. 7) The server sends the obstacle avoidance direction of each drone in the drone cluster to the corresponding drone at the next moment.

2. The method according to claim 1, characterized in that, The obstacle avoidance selection reward function in, This represents the initial obstacle avoidance probability when the i-th drone in the cluster performs action j. Let M represent the obstacle avoidance probability of the cluster executing action j, M represent the total number of drones in the cluster, and N represent the total number of action spaces.

3. The method according to claim 1 or 2, characterized in that, The distance reward function The recovery reward function Among them, u D For the target location, Let n be the position of drone i at time c, and n be the total number of drones. U represents the position of the centroid of the UAV swarm at time c. i Let u0 represent the initial position of drone i, and u0 represent the initial position of the centroid of the drone cluster.

4. The method according to claim 1, characterized in that, In step 6), the training method for the UAV with the non-fixed obstacle avoidance direction determined in step 4) to perform autonomous obstacle avoidance decision-making is as follows: 61) Select two deep neural networks with the same structure, one as the main network, used to determine the state D at the current time c. c and the action taken A c Estimate the Q value, i.e., Q(D) c A c ;θ i ), θ i One is the parameter of the current main network; the other is the parameter of the target network, based on the target state D at the next time step c+1. c+1 and historical parameters θ - The recursive formula for calculating the Q-value of the target state-action sequence is as follows: Where R c Let γ be the reward value at the current time c, and γ be the discount factor; set the step size to k and the empty replay buffer F, initialize the environment, and acquire the observation data O0. in It is the initial position of the i-th drone in the cluster. It is the initial velocity of the i-th drone in the cluster. is the initial yaw angle of the i-th UAV in the cluster, M(P)0 is the initial lateral obstacle avoidance direction matrix of the UAV cluster; initialize the sequence D0={O0} and preprocess the sequence φ0=φ(D0); 62) After running k steps according to the set step size, replace the parameters in the target network with the parameters in the main network, i.e., θ. i →θ - ; 63) An Epsilon-greedy strategy is used for action selection. When the drone's obstacle avoidance probability is ε, an action is randomly selected from the action space; when the drone's obstacle avoidance probability is 1-ε, the action with the largest Q value, i.e., P, is selected from the action space. max =argmax A Q(φ(D c ), A; θ); Calculate the optimal action value function of the drone swarm. 64) Calculate the root mean square error between the current estimated Q-value and the target Q-value of the main network. As the loss function value, gradient descent is used to adjust the parameter θ. i Perform the update; 65) Perform the action and obtain observation data O c+1 And a reward value R c Update the drone swarm policy table; set D c+1 ={D c A c O c+1 } and perform preprocessing φ c+1 =φ(D c+1 ); transfer state transition data (φ) c A c R c φ c+1 ) to cache F.

5. The method according to claim 1, 2, or 3, characterized in that, The formation status includes: the position of each drone in the drone cluster, the speed of the drone, the distance of the drone from the target point, and the average distance of each drone in the drone cluster from the target point; the airborne sensor information includes: the position of the obstacle, the length and width of the obstacle.

6. A server, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing each step of the method of any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images