A quadrotor formation obstacle avoidance control method based on improved DDPG algorithm

Abstract

The application discloses a quad-rotor formation obstacle avoidance control method based on an improved DDPG algorithm, and belongs to the technical field of unmanned aerial vehicle formation control. The method adopts an improved DDPG reinforcement learning algorithm to plan an obstacle avoidance path of the quad-rotor. When the improved DDPG reinforcement learning algorithm is executed, the priority weight of each quadruple experience in the experience replay pool is initialized. After the quadruple experience in the experience replay pool is sampled and trained, the priority weight of each quadruple experience is recalculated based on a TD error, and a Sum_Tree structure is updated. The method effectively alleviates the training instability caused by hyperparameter sensitivity, the misleading of policy updating caused by overestimation of Q values, and the problem that key experiences are not sufficiently learned, accelerates the convergence speed of the quad-rotor formation obstacle avoidance training, and improves the obstacle avoidance effect.

Description

Technical Field

[0001] This invention pertains to unmanned aerial vehicle (UAV) formation control methods, specifically involving a quadrotor formation obstacle avoidance control method based on an improved DDPG algorithm. Background Technology

[0002] Deep Deterministic Policy Gradient (DDPG) algorithms are widely used in tasks involving continuous action spaces, such as robot control and path planning. Traditional DDPG algorithms generate actions by constructing a policy network (Actor) and a value network (Critic), while also incorporating an experience replay mechanism and a dual-network architecture to stabilize the training process. For example, in robot control scenarios, DDPG algorithms can output continuous action commands based on the robot's current state, such as controlling the rotation angle of a robotic arm joint.

[0003] However, the traditional DDPG algorithm has some significant drawbacks:

[0004] First, it is extremely sensitive to hyperparameters; different hyperparameter settings often lead to drastically different training results. This means that in practical applications, a significant amount of time and computing resources are required for hyperparameter tuning, increasing the complexity and cost of the application. For example, in robotic arm trajectory planning tasks, even small changes in hyperparameters such as the learning rate and discount factor can cause huge differences in the quality of the trajectory planned by the robotic arm, or even prevent the completion of the expected task.

[0005] Secondly, the DDPG algorithm suffers from Q-value overestimation. Because the Critic network's estimation of action value is biased, it can mislead the update direction of the Actor network, making the policy optimization process unstable. For example, in path planning tasks in complex environments, Q-value overestimation may cause the robot to choose a path that appears to offer high returns but is not actually optimal, ultimately getting stuck in a local optimum and failing to find the globally optimal path.

[0006] Third, traditional experience replay mechanisms assign the same priority to all experiences, and random sampling means that key experiences (such as those with high TD errors) are not fully utilized, reducing learning efficiency. In robot obstacle avoidance scenarios, if some key experiences that reflect successful obstacle avoidance or collision hazards are not learned in a priority manner, the robot may need more time to master effective obstacle avoidance strategies. Summary of the Invention

[0007] To address the aforementioned shortcomings in existing technologies, the quadrotor formation obstacle avoidance control method based on the improved DDPG algorithm provided by this invention enhances the quality of initial experience by introducing an experience collection strategy based on human thought. It employs an experience replay technique based on Sum_Tree to prioritize the use of key experiences with high TD errors. Combined with a dual-network architecture design and OU random noise optimization, this effectively alleviates the training instability caused by hyperparameter sensitivity, the misleading effect of Q-value overestimation on policy updates, and the problem of insufficient learning of key experiences. This accelerates the convergence speed of quadrotor formation obstacle avoidance training and improves obstacle avoidance performance.

[0008] To achieve the aforementioned objectives, the technical solution adopted by this invention is: a quadrotor formation obstacle avoidance control method based on an improved DDPG algorithm, comprising:

[0009] Determine whether the distance between each quadcopter in the formation and the obstacle is greater than a preset distance threshold;

[0010] If so, the Leader-Follower cooperative algorithm is used to control the quadcopter formation for obstacle avoidance;

[0011] If not, use the improved DDPG reinforcement learning algorithm to plan the obstacle avoidance path for the quadcopter;

[0012] All quadrotors in the formation share an experience replay pool based on the Sum_Tree structure, and each quadrotor trains an independent improved DDPG reinforcement learning algorithm based on the data in the shared experience replay pool to control the obstacle avoidance path of each quadrotor.

[0013] When executing the improved DDPG reinforcement learning algorithm, the priority weights of each quadruple experience in the experience replay pool are initialized. After training based on the quadruple experiences in the experience replay pool, the priority weights of each quadruple experience are recalculated based on the TD error, and the Sum_Tree structure is updated.

[0014] Furthermore, in the Sum_Tree-based experience replay pool, each quadruple experience and its priority weight are stored in the leaf node of the Sum_Tree structure, and the internal nodes of the Sum_Tree structure only store the sum of the priority weights of the two child nodes.

[0015] Furthermore, the priority weight of the quadruple experience is the TD error corresponding to the quadruple experience. It is represented as:

[0016]

[0017] In the formula, This indicates the current state of the Critic value network. Take action The predicted Q value, This represents the feature state extraction function. This represents the parameters of the current Critic value network. This represents the immediate reward obtained during a state transition. Indicates the discount factor. This represents the target Actor action network's response to the next state. The generated optimal action, This represents the parameters of the target Critic value network, and the subscript p indicates the index of the quadruplet experience sample in the experience replay pool.

[0018] The probability of the quadruple experience being sampled for:

[0019]

[0020] In the formula, and Let represent the priority correction values ​​for the p-th quadruple experience and the q-th quadruple experience, respectively. The priority parameter is represented by the subscripts p and q, which represent the indexes of the quadruplet experience samples in the experience replay pool, and p, q = 1 to m, where m represents the number of quadruplet experience samples in the experience replay pool.

[0021] Furthermore, during the process of executing the improved DDPG reinforcement learning algorithm to control the obstacle avoidance path of the quadcopter, when determining the action to be executed based on the output value of the current Actor action network, in order to... Add random noise with a probability of , and with The probability of directly executing the action output by the Actor action network;

[0022] The differential equation for random noise is expressed as:

[0023]

[0024] In the formula, and Let these represent the rate of mean reversion and volatility, respectively. This represents the mean. express The observed values, Representing the Wiener process, This represents the state value of random noise at time t.

[0025] Furthermore, the improved DDPG reinforcement learning algorithm includes a Critic value network and an Actor action network:

[0026] The Critic value network comprises a first input layer, three first hidden layers, and a first output layer connected in sequence; wherein, the first input layer includes an action space and a state space obtained by collecting four-tuple experiences from the experience replay pool, the activation function of each neuron in each first hidden layer is a ReLU function, and the activation function of the first output layer is a linear activation function;

[0027] The Actor action network comprises a second input layer, three second hidden layers, and a second output layer connected in sequence. The second input layer is a state space obtained by collecting four-tuple experiences from the experience replay pool. The activation function of each neuron in each second hidden layer is the ReLU function. The second output layer comprises three neurons, and the activation function of each neuron is the Tanh function.

[0028] Furthermore, the loss function of the Critic value network for:

[0029]

[0030] In the formula, This represents the weight of the p-th four-tuple experience in the experience replay pool. Indicates the target Q value. This indicates that the parameters of the current Critic value network are... Calculate state under the condition Next action The predicted Q-value; where the weights , and Let p and q represent the probabilities that the p-th and q-th quadruples are empirically sampled, respectively. This represents the sampling weight coefficient.

[0031] Furthermore, in the improved DDPG reinforcement learning algorithm:

[0032] No. The state space of a quadcopter is , Indicates the first The distance between the quadcopter and the nearest obstacle. Indicates the distance to the first The angle of the quadcopter at its closest point, Indicates along the first The velocity along the x-axis of the quadrotor's body coordinate system, the velocity along the y-axis of the body coordinate system, and the velocity along the z-axis of the body coordinate system;

[0033] No. The motion space of the quadcopter is the Actor motion network that sets the current state of the quadcopter. Given the input, the output follows the first... The velocity along the x-axis of the quadcopter's body coordinate system and the velocity along the y-axis of the body coordinate system .

[0034] Furthermore, the distance Represented as:

[0035]

[0036] In the formula, This indicates the first point in the point cloud data collected by the lidar. The point at the th The position of the quadcopter in the coordinate system. This represents the x-axis coordinate in the body coordinate system. This represents the y-axis coordinate in the body coordinate system. This represents the z-axis coordinate in the body coordinate system;

[0037] The angle for:

[0038] In the formula, For a sign function, when When the value is greater than 0, the sign function outputs 1; when... When equal to 0, the sign function outputs 0; when When the value is less than 0, the sign function outputs -1.

[0039] Furthermore, the reward function of the improved DDPG reinforcement learning algorithm for:

[0040]

[0041] In the formula, Indicates the first The distance between the quadcopter and the nearest obstacle. Indicates the distance to the first The angle of the quadcopter at its closest point, Indicates the dangerous obstacle avoidance distance of a quadcopter. This represents the dynamic offset distance for stable obstacle avoidance. This represents the obstacle avoidance distance taken over by the improved DDPG reinforcement learning algorithm. The reward function, which considers all time and distance factors for obstacle avoidance by each quadcopter, is expressed as:

[0042]

[0043] In the formula, This indicates the flight path of the quadcopter obstacle avoidance system in the formation. This indicates the time taken for the quadcopters in the formation to avoid obstacles. Control parameters representing travel costs Control parameters representing time costs , , This indicates the time when obstacle avoidance begins for the i-th quadrotor. This indicates the time when the obstacle avoidance operation of the i-th quadcopter ends. This represents the speed at which the i-th quadcopter avoids obstacles.

[0044] The beneficial effects of this invention are as follows:

[0045] (1) Improving initial training efficiency through experience collection strategies based on human thought

[0046] This invention uses a human-controlled quadcopter to collect high-quality initial experience, replacing the traditional random strategy for experience collection. This solves the problem of low-quality initial experience and slow training convergence caused by a large amount of meaningless data in the traditional DDPG, enabling the algorithm to learn quickly based on effective experience in the early stages of training and shortening the overall training cycle.

[0047] (2) Sum_Tree experience replay technology improves the utilization rate of key experiences

[0048] This invention uses TD error as a priority indicator and prioritizes the sampling of high-value experiences through the Sum_Tree structure, and dynamically updates the priority. This overcomes the shortcomings of traditional random sampling of experience replay, which leads to the easy discarding or low utilization of key experiences (such as obstacle avoidance failure, danger approach, and other scenario data). It strengthens the learning of high-value experiences and improves the learning efficiency and robustness of obstacle avoidance strategies.

[0049] (3) The effectiveness of OU random noise in enhancing environmental exploration

[0050] This invention introduces Ornstein-Uhlenbeck noise with Markov characteristics to replace traditional Gaussian noise, solving the problem of insufficient environmental exploration and easy getting trapped in local optima caused by the strong determinism of traditional DDPG. The continuity and mean regression characteristics of the OU noise used in this invention are more suitable for quadcopter inertial systems, which can guide them to explore safe paths more comprehensively during obstacle avoidance and improve the optimization ability of global optimal strategy.

[0051] (4) The multi-agent framework balances data sharing and communication overhead.

[0052] This invention adopts the DDPG reinforcement learning architecture of "shared experience pool + independent algorithm", which shares experience while maintaining the independence of individual algorithms. It avoids the problems of "low data utilization due to completely independent experience pool" or "excessive communication volume due to completely shared algorithm" in traditional multi-agent schemes. While improving the overall training speed of the formation, it reduces the coupling and communication burden between quadcopters and is suitable for large-scale formation obstacle avoidance scenarios.

[0053] (5) Improved model performance through optimized network structure and loss function

[0054] This invention designs a specific Critic / Actor network structure and introduces sample priority weights into the loss function, which solves the problems of insufficient fitting accuracy of quadrotor state-action space by traditional network structures and poor training stability caused by the loss function not distinguishing the importance of experience. This makes value assessment more accurate, action generation more in line with obstacle avoidance requirements, and improves the smoothness and accuracy of formation obstacle avoidance. Attached Figure Description

[0055] Figure 1 The flowchart of the quadrotor formation obstacle avoidance control method based on the improved DDPG algorithm provided by the present invention is shown.

[0056] Figure 2 A schematic diagram illustrating the experience collection strategy based on human thought provided by this invention.

[0057] Figure 3 A schematic diagram of the Critic value network provided by this invention.

[0058] Figure 4 This is a schematic diagram of the Actor action network provided by the present invention.

[0059] Figure 5 This is a schematic diagram showing the positional relationship between the obstacle and the quadcopter provided by the present invention.

[0060] Figure 6 This is a schematic diagram showing the distance relationship between the quadcopter and the obstacle provided by the present invention. Detailed Implementation

[0061] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0062] This invention provides a quadrotor formation obstacle avoidance control method based on an improved DDPG algorithm. (See also...) Figure 1 ,include:

[0063] Determine whether the distance between each quadcopter in the formation and the obstacle is greater than a preset distance threshold;

[0064] If so, the Leader-Follower cooperative algorithm is used to control the quadcopter formation for obstacle avoidance;

[0065] If not, use the improved DDPG reinforcement learning algorithm to plan the obstacle avoidance path for the quadcopter;

[0066] All quadrotors in the formation share an experience replay pool based on the Sum_Tree structure, and each quadrotor trains an independent improved DDPG reinforcement learning algorithm based on the data in the shared experience replay pool to control the obstacle avoidance path of each quadrotor.

[0067] In this embodiment, the distance threshold is the obstacle avoidance distance. This represents the obstacle avoidance distance taken over by the improved DDPG reinforcement learning algorithm.

[0068] In this embodiment, the traditional DDPG algorithm generates a large amount of quadruple empirical data when interacting with the environment. Since this data is linearly arranged over time, directly training the neural network with this data may lead to unstable training results and slow convergence. Therefore, the concept of an experience replay pool is introduced. Sampled data is placed in the experience replay pool, and then randomly sampled from it during each training iteration. This method allows for repeated sampling of historical data, avoiding the situation where data is discarded after a single chronological sample, thus enabling better results to be trained with fewer samples.

[0069] However, the experience replay strategy has a significant drawback: it samples randomly from the experience replay pool, and the pool has a limited capacity. If the experience replay array is full, the oldest data in the pool will be discarded. This means that some important and valuable data in the experience replay pool will be discarded before they have been sampled. Therefore, the quadcopter may have difficulty learning from this valuable data, resulting in longer training time and slower convergence.

[0070] To address this issue, this invention proposes an experience replay technique based on a Sum_Tree structure. In this embodiment, in the Sum_Tree-based experience replay pool, each quadruple experience and its priority weight are stored in the leaf nodes of the Sum_Tree structure, and the internal nodes of the Sum_Tree structure only store the sum of the priority weights of the two child nodes.

[0071] Furthermore, due to the varying TD errors of different samples in the experience replay pool, the effect of backpropagation on the neural network also differs. A larger TD error leads to a greater effect of backpropagation; conversely, a smaller TD error has less impact on the backward gradient calculation of the neural network. In the DDPG reinforcement learning algorithm, the TD error represents the difference between the target value calculated by the target Critic value network and the current value calculated by the current Critic value network.

[0072] Based on this, in this embodiment, the priority weight of the quadruplet experience is used as the TD error corresponding to the quadruplet experience. It is represented as:

[0073]

[0074] In the formula, This indicates the current state of the Critic value network. Take action The predicted Q value, This represents the feature state extraction function, which represents extracting the original state. Convert it into a feature vector that the network can process. This represents the parameters of the current Critic value network. This represents the immediate reward obtained during a state transition. Indicates the discount factor. This represents the target Actor action network's response to the next state. The generated optimal action, This represents the parameters of the target Critic value network, and the subscript p represents the index of the quadruplet experience sample in the experience replay pool.

[0075] Therefore, TD error The magnitude of the error also indicates the degree to which the quadrotor-like intelligent agent still needs to learn; the larger the error, the greater the quaternary empirical value it represents. The higher the priority, the more worthwhile it is to be learned.

[0076] Based on the Sum_Tree structure, the storage rule for quadruplet experiences is used to obtain the probability of quadruplet experiences being sampled. for:

[0077]

[0078] In the formula, and Let represent the priority correction values ​​for the p-th quadruple experience and the q-th quadruple experience, respectively. The priority parameter is represented by the subscripts p and q, which represent the indexes of the quadruplet experience samples in the experience replay pool, and p, q = 1 to m, where m represents the number of quadruplet experience samples in the experience replay pool.

[0079] In this embodiment, by constructing the experience replay pool with the above structure, samples with high TD errors are more likely to be sampled, which is beneficial to the backpropagation of neural network errors.

[0080] In this embodiment, before training the improved DDPG reinforcement learning algorithm, in order to improve the quality of the previously collected quadruple experiences, an experience collection strategy based on human thought is adopted, such as... Figure 2 As shown, this strategy mainly involves humans using the interface provided by the simulation algorithm to manually control the interaction between the quadcopter and the environment in the early stages, thereby obtaining a sufficient number of high-quality quadruple experiences collected by human thought control, that is, the number of quadruple experience values ​​obtained is equal to the number of human experience entries.

[0081] In the traditional DDPG reinforcement learning algorithm, since it can only execute a certain action based on the output value of the current Actor action network each time, DDPG does not explore the environment enough and thus cannot find the optimal action, thus getting stuck in a local optimum.

[0082] In this embodiment, random noise is added to the improved DDPG reinforcement learning algorithm to make the training results as globally optimal as possible. Specifically, during the process of controlling the obstacle avoidance path of the quadcopter using the improved DDPG reinforcement learning algorithm, when determining the action to be executed based on the output value of the current Actor action network, the following steps are taken: Add random noise with a probability of , and with The probability of directly executing the action output by the Actor action network;

[0083] The differential equation for random noise is expressed as:

[0084]

[0085] In the formula, and Let these represent the rate of mean reversion and volatility, respectively. This represents the mean. express The observed values, Representing the Wiener process, This represents the state value of random noise at time t.

[0086] The random noise added in this embodiment is suitable for inertial systems such as quadcopters and robotic arms, especially when the time discretization granularity is small. It exhibits Markov characteristics, meaning that the noise in later steps is influenced by the noise in previous steps. Therefore, unlike Gaussian noise, which is completely random, it explores forward or backward around the mean. Thus, this noise is more suitable than Gaussian noise for reinforcement learning algorithms like DDPG to explore the environment.

[0087] In this embodiment, the improved DDPG reinforcement learning algorithm includes a Critic value network and an Actor action network:

[0088] like Figure 3 As shown, the Critic value network includes a first input layer, three first hidden layers, and a first output layer connected in sequence. The first input layer includes an action space and a state space obtained from the experience of four-tuples collected from the experience replay pool. The activation function of each neuron in each first hidden layer is a ReLU function, and the activation function of the first output layer is a linear activation function.

[0089] Furthermore, each first hidden layer contains 200 neurons, and the activation function of each neuron is the ReLU function; based on the above Critic value network structure, the input actions and states can be converted into their corresponding Q values.

[0090] In this embodiment, based on the above-described Sum_Tree-based experience replay pool design, the loss function of the Critic value network is... Optimized to:

[0091]

[0092] In the formula, This represents the weight of the p-th four-tuple experience in the experience replay pool. Indicates the target Q value. This indicates that the parameters of the current Critic value network are... Calculate state under the condition Next action The predicted Q-value; where the weights , and Let p and q represent the probabilities that the p-th and q-th quadruples are empirically sampled, respectively. This represents the sampling weight coefficient.

[0093] like Figure 4As shown, the Actor Action Network includes a second input layer, three second hidden layers, and a second output layer connected in sequence. The second input layer is a state space obtained by collecting four-tuple experiences from the experience replay pool. The activation function of each neuron in each second hidden layer is the ReLU function. The second output layer includes three neurons, and the activation function of each neuron is the Tanh function.

[0094] Furthermore, each second hidden layer also contains 200 neurons, and the activation function of each neuron is the ReLU function; based on the above Actor action network structure, the Actor action network can obtain the corresponding action by the input state.

[0095] In this embodiment, in the improved DDPG reinforcement learning algorithm:

[0096] No. The state space of a quadcopter is , Indicates the first The distance between the quadcopter and the nearest obstacle. Indicates the distance to the first The angle of the quadcopter at its closest point, Indicates along the first The velocity along the x-axis of the quadrotor's body coordinate system, the velocity along the y-axis of the body coordinate system, and the velocity along the z-axis of the body coordinate system;

[0097] No. The motion space of the quadcopter is the Actor motion network that sets the current state of the quadcopter. Given the input, the output follows the first... The velocity along the x-axis of the quadcopter's body coordinate system and the velocity along the y-axis of the body coordinate system .

[0098] Specifically, in the design of the state space, to achieve collision avoidance between quadrotors in the formation and between quadrotors and obstacles, the quadrotors need to collect surrounding environmental information in real time to achieve obstacle avoidance. This paper uses a lidar on each quadrotor to scan the surrounding environment in real time, thereby obtaining the first obstacle avoidance information in the environment. Distance from the quadcopter to the obstacle The obstacle relative to the first Orientation information of the quadcopter and the The speed value of the quadcopter A lidar sensor was used to scan for obstacle information. The sensor's horizontal scanning radius was set to [-150°, 150°], the vertical scanning radius to [-40°, 40°], and the scanning distance to 4.5 meters.

[0099] This includes calculating the distance and azimuth information to the nearest obstacle (or other quadrotor) to the quadrotor. For the... The distance of the quadcopter to the nearest obstacle (or other quadcopter). Represented as:

[0100]

[0101] In the formula, This indicates the first point in the point cloud data collected by the lidar. The point at the th The position of the quadcopter in the airframe coordinate system. This represents the x-axis coordinate in the body coordinate system. This represents the y-axis coordinate in the body coordinate system. This represents the z-axis coordinate in the body coordinate system.

[0102] like Figure 5 As shown, for the distance to the th The angle information of the nearest point of the quadcopter is used to calculate the distance between that point and the first... The angle between the line connecting point O in the quadcopter's body coordinate system and the x-axis of the body coordinate system is obtained. for:

[0103] In the formula, For a sign function, when When the value is greater than 0, the sign function outputs 1; when... When equal to 0, the sign function outputs 0; when When the value is less than 0, the sign function outputs -1.

[0104] Specifically, in the process of obstacle avoidance in the action state space, for quadrotor formations, the action space considering motion along the x-axis and y-axis of the body coordinate system in this embodiment is difficult to achieve with discrete actions. In order to ensure the effect of obstacle avoidance and the continuity of the flight trajectory during obstacle avoidance, the Actor action network in the improved DDPG reinforcement learning algorithm based on Sum_Tree is used to process the current state of the quadrotor. Given the input, the output follows the first... The velocity along the x-axis of the quadcopter's body coordinate system and the velocity along the y-axis of the body coordinate system This makes the first quadcopter in the formation A quadcopter can avoid tree trunks perpendicular to the ground and other quadcopters in formation.

[0105] In this embodiment, the reinforcement learning algorithm learns based on feedback from the reward function; therefore, the design of the reward function significantly impacts the final effect of the reinforcement learning. For example, the distance relationship between the quadcopters and obstacles in the formation... Figure 6 As shown; based on this distance relationship, the criterion for judging whether the quadcopter's obstacle avoidance is successful is whether the angle between the quadcopter's flight direction and the obstacle is greater than 90°, that is: .

[0106] In this embodiment, the reward function is designed by considering the following two factors:

[0107] (1) Distance factors between each quadcopter in the formation or between a quadcopter and an obstacle

[0108] In designing this reward function, three aspects were considered: a. The distance between each quadcopter in the formation and the obstacle cannot be too close, that is: Otherwise, the quadcopter risks colliding with obstacles; b. The distance between each quadcopter and the obstacle cannot be too far, that is: Otherwise, it will deviate from the quadcopter's original route, causing the quadcopter to take a longer route. Therefore, the best way is to ensure that the quadcopter maintains a suitable distance from obstacles; c. When the quadcopter maintains a suitable distance from obstacles, that is: The obstacle avoidance direction of the quadcopter is also crucial. If the obstacle is to the right front of the quadcopter, the obstacle avoidance direction should be to the left front; otherwise, the obstacle avoidance distance may be excessive, or the quadcopter may even collide directly with the obstacle. Based on this, the reward function for the first part is designed as follows:

[0109]

[0110] (2) The time and distance factors for obstacle avoidance for each quadcopter in the formation

[0111] After a quadcopter in the formation successfully avoids an obstacle, it needs to be rewarded for the time and distance it took to achieve this obstacle avoidance. The shorter the time and distance traveled during obstacle avoidance, the higher the reward. This encourages the quadcopters to find better obstacle avoidance paths and achieve obstacle avoidance more quickly. Based on this, the second part of the reward is designed as follows:

[0112]

[0113] In the formula, Control parameters representing travel costs Control parameters representing time costs and These represent the flight distance and time taken for the quadcopter obstacle avoidance system in the formation, respectively:

[0114]

[0115]

[0116] In the formula, This indicates the time when obstacle avoidance begins for the i-th quadrotor. This indicates the time when the obstacle avoidance operation of the i-th quadrotor ends. This represents the speed at which the i-th quadcopter avoids obstacles.

[0117] In summary, the reward function of the improved DDPG reinforcement learning algorithm in this embodiment is obtained. for:

[0118]

[0119] In the formula, Indicates the first The distance between the quadcopter and the nearest obstacle. Indicates the distance to the first The angle of the quadcopter at its closest point, Indicates the dangerous obstacle avoidance distance of the quadcopter. This represents the dynamic offset distance for stable obstacle avoidance. This represents the obstacle avoidance distance taken over by the improved DDPG reinforcement learning algorithm. This represents the reward function that takes into account all time and distance factors used for obstacle avoidance by each quadcopter.

[0120] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

[0121] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A quadrotor formation obstacle avoidance control method based on an improved DDPG algorithm, characterized in that, include: Determine whether the distance between each quadcopter in the formation and the obstacle is greater than a preset distance threshold; If so, the Leader-Follower cooperative algorithm is used to control the quadcopter formation for obstacle avoidance; If not, use the improved DDPG reinforcement learning algorithm to plan the obstacle avoidance path for the quadcopter; All quadrotors in the formation share an experience replay pool based on the Sum_Tree structure, and each quadrotor trains an independent improved DDPG reinforcement learning algorithm based on the data in the shared experience replay pool to control the obstacle avoidance path of each quadrotor. When executing the improved DDPG reinforcement learning algorithm, the priority weights of each quadruple experience in the experience replay pool are initialized. After training based on the quadruple experiences in the experience replay pool, the priority weights of each quadruple experience are recalculated based on the TD error, and the Sum_Tree structure is updated. In the Sum_Tree-based experience replay pool, each quadruple experience and its priority weight are stored in the leaf node of the Sum_Tree structure, and the internal nodes of the Sum_Tree structure only store the sum of the priority weights of the two child nodes. The priority weight of the quadruple experience is the TD error corresponding to the quadruple experience. It is represented as: In the formula, This indicates the current state of the Critic value network. Take action The predicted Q value, This represents the feature state extraction function. This represents the parameters of the current Critic value network. This represents the immediate reward obtained during a state transition. Indicates the discount factor. This represents the target Actor action network's response to the next state. The generated optimal action, This represents the parameters of the target Critic value network, and the subscript p indicates the index of the quadruplet experience sample in the experience replay pool. The probability of the quadruple experience being sampled for: In the formula, and Let represent the priority correction values ​​for the p-th quadruple experience and the q-th quadruple experience, respectively. The priority parameter is represented by the subscripts p and q, which represent the indexes of the quadruplet experience samples in the experience replay pool, and p, q = 1 to m, where m represents the number of quadruplet experience samples in the experience replay pool.

2. The quadrotor formation obstacle avoidance control method based on the improved DDPG algorithm according to claim 1, characterized in that, During the execution of the improved DDPG reinforcement learning algorithm to control the obstacle avoidance path of the quadcopter, the action to be executed is determined based on the output value of the current Actor action network. Add random noise with a probability of , and with The probability of directly executing the action output by the Actor action network; The differential equation for random noise is expressed as: In the formula, and Let these represent the rate of mean reversion and volatility, respectively. This represents the mean. express The observed values, Representing the Wiener process, This represents the state value of random noise at time t.

3. The quadrotor formation obstacle avoidance control method based on the improved DDPG algorithm according to claim 1, characterized in that, The improved DDPG reinforcement learning algorithm includes a Critic value network and an Actor action network: The Critic value network comprises a first input layer, three first hidden layers, and a first output layer connected in sequence; wherein, the first input layer includes an action space and a state space obtained by collecting four-tuple experiences from the experience replay pool, the activation function of each neuron in each first hidden layer is a ReLU function, and the activation function of the first output layer is a linear activation function; The Actor action network comprises a second input layer, three second hidden layers, and a second output layer connected in sequence. The second input layer is a state space obtained by collecting four-tuple experiences from the experience replay pool. The activation function of each neuron in each second hidden layer is the ReLU function. The second output layer comprises three neurons, and the activation function of each neuron is the Tanh function.

4. The quadrotor formation obstacle avoidance control method based on the improved DDPG algorithm according to claim 3, characterized in that, The loss function of the Critic value network for: In the formula, This represents the weight of the p-th four-tuple experience in the experience replay pool. Indicates the target Q value. This indicates that the parameters of the current Critic value network are... Calculate state under the condition Next action The predicted Q-value; where the weights , and Let p and q represent the probabilities that the p-th and q-th quadruples are empirically sampled, respectively. This represents the sampling weight coefficient.

5. The quadrotor formation obstacle avoidance control method based on the improved DDPG algorithm according to claim 1, characterized in that, In the improved DDPG reinforcement learning algorithm: No. The state space of a quadcopter is , Indicates the first The distance between the quadcopter and the nearest obstacle. Indicates the distance to the first The angle of the quadcopter at its closest point, Indicates along the first The velocity along the x-axis of the quadrotor's body coordinate system, the velocity along the y-axis of the body coordinate system, and the velocity along the z-axis of the body coordinate system; No. The motion space of the quadcopter is the Actor motion network that sets the current state of the quadcopter. Given the input, the output follows the first... The velocity along the x-axis of the quadcopter's body coordinate system and the velocity along the y-axis of the body coordinate system .

6. The quadrotor formation obstacle avoidance control method based on the improved DDPG algorithm according to claim 5, characterized in that, The distance Represented as: In the formula, This indicates the first point in the point cloud data collected by the lidar. The point at the th The position of the quadcopter in the coordinate system. This represents the x-axis coordinate in the body coordinate system. This represents the y-axis coordinate in the body coordinate system. This represents the z-axis coordinate in the body coordinate system; The angle for: In the formula, For a sign function, when When the value is greater than 0, the sign function outputs 1; when... When equal to 0, the sign function outputs 0; when When the value is less than 0, the sign function outputs -1.

7. The quadrotor formation obstacle avoidance control method based on the improved DDPG algorithm according to claim 1, characterized in that, The reward function of the improved DDPG reinforcement learning algorithm for: In the formula, Indicates the first The distance between the quadcopter and the nearest obstacle. Indicates the distance to the first The angle of the quadcopter at its closest point, Indicates the dangerous obstacle avoidance distance of a quadcopter. This represents the dynamic offset distance for stable obstacle avoidance. This represents the obstacle avoidance distance taken over by the improved DDPG reinforcement learning algorithm. The reward function, which considers all time and distance factors for obstacle avoidance by each quadcopter, is expressed as: In the formula, This indicates the flight path of the quadcopter obstacle avoidance system in the formation. This indicates the time taken for the quadcopters in the formation to avoid obstacles. Control parameters representing travel costs Control parameters representing time costs , , This indicates the time when obstacle avoidance begins for the i-th quadrotor. This indicates the time when the obstacle avoidance operation of the i-th quadrotor ends. This represents the speed at which the i-th quadcopter avoids obstacles.

Images