Cooperative obstacle avoidance control method, device and equipment for multiple UAVs based on PPO algorithm

Abstract

The present application relates to a method, device and equipment for cooperative obstacle avoidance control of multiple UAVs based on the PPO algorithm, the method comprising: constructing an intelligent agent, deploying an intelligent agent on each UAV, the intelligent agent comprising a policy network and a value network; using a chained PPO training framework to conduct multiple rounds of training on the UAVs in the formation to obtain the optimal strategy for the path planning and obstacle avoidance tasks of each UAV; each UAV in the formation executes its own optimal strategy for path planning and obstacle avoidance tasks to achieve cooperative obstacle avoidance control of multiple UAVs. During the training process, except for the UAV currently in training, other UAVs maintain a fixed strategy to effectively maintain the stability of the training environment; the chained training framework uses the local observation state of each UAV to estimate the amount of action, and under the guidance of heuristic information and reward functions, it can maintain a certain formation stability to reach the target point under the premise of avoiding obstacles, thereby improving the effect of quickly reaching the destination.

Description

Technical Field

[0001] The present application relates to the technical field of multi-aircraft cooperative control, and in particular to a method, device and equipment for cooperative obstacle avoidance control of multiple UAVs based on a PPO algorithm. Background Art

[0002] In today's world, drone technology is developing rapidly and its application areas are becoming increasingly extensive. Especially in complex environments, such as forests and urban canyons, the autonomous navigation and path planning capabilities of multiple drones are particularly important. In such environments, drones need to have a high degree of autonomy and flexibility to avoid obstacles while maintaining an efficient flight path when performing missions.

[0003] In this context, with the rapid development of artificial intelligence technology, control methods based on deep reinforcement learning (DRL) have attracted widespread attention. Motion planning of multiple UAVs shuttling through dense obstacle environments based on deep reinforcement learning algorithms has become a research hotspot.

[0004] Existing research results mainly focus on two-dimensional plane motion planning in the process of multi-UAV swarm formation obstacle avoidance. At the same time, the effect of reinforcement learning training for multi-UAV swarms with continuous motion is not ideal, and the training in a continuous three-dimensional multi-UAV swarm environment is unstable, which makes the effectiveness of multi-UAV formation mission execution low. Summary of the invention

[0005] Based on this, it is necessary to provide a multi-UAV collaborative obstacle avoidance control method, device and equipment based on the PPO algorithm to address the above technical problems. This method solves the current problem that the model is difficult to establish under the multi-UAV obstacle avoidance task and the traditional reinforcement learning is difficult to converge.

[0006] A multi-UAV cooperative obstacle avoidance control method based on PPO algorithm, the method comprising:

[0007] Build an intelligent agent and deploy one on each drone. The intelligent agent includes a policy network and a value network.

[0008] The chained PPO training framework is used to conduct multiple rounds of training on the UAVs in the formation to obtain the optimal strategy for path planning and obstacle avoidance tasks of each UAV; the chained PPO training framework adopts a distributed training method, and the currently trained UAV adopts the PPO algorithm, adding heuristic information as a guide, inputting the current UAV local observation state into the policy network and the value function network, the policy network outputs the action value to the training environment to update the state, and the value network outputs the Q value to evaluate the current state; the non-currently trained UAVs in the formation adopt a fixed strategy based on their respective local observation states to output their respective action amounts to the training environment to update the state; the training environment is a three-dimensional continuous space for multiple UAVs to move.

[0009] Each UAV in the formation executes its own optimal strategy for path planning and obstacle avoidance tasks, realizing multi-UAV collaborative obstacle avoidance control.

[0010] Among them, adding heuristic information as a guide includes: using heuristic speed to superimpose on the amount of action to improve the training speed and effect; among them, the heuristic speed is:

[0011] ;

[0012] in, is the heuristic speed, is a small positive coefficient, is the current position of the training drone. is the current training drone target point position, is the matrix composed of the position vectors of the drone group at this time, It is the matrix composed of the position vectors of the target points of the drone group.

[0013] In one embodiment, an artificial potential field method is used in the first round of training to simulate the obstacle avoidance shuttle motion control strategy of the non-currently trained UAVs in the formation.

[0014] In one embodiment, a chained PPO training framework is used to perform multiple rounds of training on the drones in the formation to obtain the optimal strategy for path planning and obstacle avoidance tasks of each drone; the chained PPO training framework adopts a distributed training method, the drone currently being trained adopts the PPO algorithm, adds heuristic information as a guide, inputs the local observation state of the current drone into the policy network and the value function network, the policy network outputs the action value to the training environment to update the state, and the value network outputs the Q value to evaluate the current state; the drones not currently being trained in the formation adopt a fixed strategy based on their respective local observation states to output their respective action amounts to the training environment to update the state; the training environment is a three-dimensional continuous space where multiple drones move, including:

[0015] Set the first drone in the formation as the current training drone.

[0016] The local observation state of the currently trained UAV is used as the input of the policy network of the currently trained UAV, and heuristic information is added as a guide. The policy network outputs an action value, and the action is applied to the UAV to interact with the training environment. Through interaction, experience is collected and multiple types of rewards are accumulated over a long period of time. The policy gradient of each batch is calculated, and then gradient clipping is performed and the policy network parameters are updated.

[0017] The local observation state of the currently trained UAV is input into the value network of the currently trained UAV, and the Q value is output to evaluate the current state; the minimum value function estimate of each sample taking different actions under different states is calculated, the value function of the target point is updated, and then the value network parameters are updated with the loss function.

[0018] The non-currently trained UAVs in the formation output their own action quantities to the training environment based on their respective local observation states and fixed strategies to update the UAV states.

[0019] After the currently trained UAV is trained, the UAV loads the trained model and then selects the UAV closest to the currently trained UAV in the formation as the currently trained UAV. The training continues until all UAVs in the formation are traversed and the next round of training is entered until the preset termination condition is met. The optimal strategy for the path planning and obstacle avoidance tasks of each UAV is obtained.

[0020] In one embodiment, the multiple different types of rewards are rewards for reaching the target point, collision penalties, path costs, and formation stability rewards.

[0021] The rewards for reaching the target point are:

[0022] ;

[0023] in, Rewards for reaching the target point, is the trajectory point information of the current training drone, is the target point vector of the current training drone.

[0024] The collision penalty is:

[0025] ;

[0026] ;

[0027] ;

[0028] in, is the number of sampling points along a single step; is the total collision penalty, For other drones in the same group; and Respectively represent i The collision penalty between the current UAV and static obstacles and dynamic obstacles at each sampling moment.

[0029] The path cost is:

[0030] ;

[0031] in, is the path cost of the current step, m is a positive coefficient, They are the speeds of the current drone's x, y, and z axes, respectively.

[0032] The formation stability bonus is:

[0033] ;

[0034] ;

[0035] in, To maintain the formation's rewards, For sparse rewards, is the current drone position, The position of the drone expected by the formation at this time, is the current speed of the drone, is the speed of the formation leader, n For the range A real number.

[0036] A multi-UAV cooperative obstacle avoidance control device based on PPO algorithm, the device comprising:

[0037] The intelligent agent building module is used to build an intelligent agent. An intelligent agent is deployed on each drone. The intelligent agent includes a policy network and a value network.

[0038] The optimal strategy determination module for path planning and obstacle avoidance tasks is used to conduct multiple rounds of training on the UAVs in the formation using the chained PPO training framework to obtain the optimal strategy for path planning and obstacle avoidance tasks for each UAV; the chained PPO training framework adopts a distributed training method, and the currently trained UAV adopts the PPO algorithm, adding heuristic information as a guide, inputting the current UAV local observation state into the policy network and the value function network, the policy network outputs the action value to the training environment to update the state, and the value network outputs the Q value to evaluate the current state; the non-currently trained UAVs in the formation adopt a fixed strategy based on their respective local observation states to output their respective action amounts to the training environment to update the state; the training environment is a three-dimensional continuous space for the movement of multiple UAVs.

[0039] The multi-UAV collaborative obstacle avoidance control module is used for each UAV in the formation to execute its own optimal strategy for path planning and obstacle avoidance tasks, thereby realizing multi-UAV collaborative obstacle avoidance control.

[0040] Among them, the optimal strategy determination module for path planning and obstacle avoidance tasks is also used to superimpose the heuristic speed on the action amount to improve the training speed and effect; wherein the heuristic speed is:

[0041] ;

[0042] in, is the heuristic speed, is a small positive coefficient, is the current position of the training drone. is the current training drone target point position, is the matrix composed of the position vectors of the drone group at this time, It is the matrix composed of the position vectors of the target points of the drone group.

[0043] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0044] The beneficial effects of this method are:

[0045] The above-mentioned method, device and equipment for cooperative obstacle avoidance control of multiple UAVs based on the PPO algorithm include: constructing an intelligent agent, deploying an intelligent agent on each UAV, and the intelligent agent includes a policy network and a value network; using a chain PPO training framework to conduct multiple rounds of training on the UAVs in the formation to obtain the optimal strategy for the path planning and obstacle avoidance tasks of each UAV; each UAV in the formation executes its own optimal strategy for path planning and obstacle avoidance tasks to achieve cooperative obstacle avoidance control of multiple UAVs. During the training process, except for the UAV currently in training, other UAVs maintain a fixed strategy to effectively maintain the stability of the training environment; the chain training framework uses the local observation state of each UAV to estimate the amount of action, and under the guidance of heuristic information and reward functions, it can maintain a certain formation stability to reach the target point under the premise of avoiding obstacles, thereby improving the effect of quickly reaching the destination. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 1 is a flow chart of a multi-UAV cooperative obstacle avoidance control method based on a PPO algorithm in one embodiment;

[0047] Figure 2 A schematic diagram of a chain-based deep reinforcement learning training framework in one embodiment;

[0048] Figure 3 A schematic diagram of a chain training process in an embodiment;

[0049] Figure 4 A reward change curve diagram in another embodiment;

[0050] Figure 5 This is a diagram showing the obstacle avoidance effect of a single machine in another embodiment;

[0051] Figure 6This is another example of a multi-machine obstacle avoidance effect diagram.

[0052] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] In one embodiment, Figure 1 As shown, a multi-UAV cooperative obstacle avoidance control method based on the PPO algorithm is provided, and the method includes the following steps:

[0055] Step 100: Build an intelligent agent and deploy an intelligent agent on each drone. The intelligent agent includes a policy network and a value network.

[0056] Specifically, the value network is responsible for estimating the value of taking different actions under different states, that is, estimating the value function. Collect a batch of data from the playback buffer, calculate the minimum value function estimate of each sample taking different actions under different states, update the value function to the target point, and then use the loss function to update the value network parameters. The action selection here is obtained by relying on the policy network.

[0057] The policy network is responsible for learning a policy, that is, outputting an action that is most likely to bring high value under a given state through the policy network. Collect a batch of data from the playback buffer, calculate the policy gradient for each batch, perform gradient clipping, and update the policy network parameters.

[0058] Policy Network and Value Network are two core concepts in Deep Reinforcement Learning (DRL). Generally, a fully connected network is used. The structure of the policy network is , is the state space dimension, is the action space dimension; the value network structure is , For the state space dimension, the value network generally outputs a value to judge the value of the current state.

[0059] Step 102: Use the chained PPO training framework to conduct multiple rounds of training on the UAVs in the formation to obtain the optimal strategy for path planning and obstacle avoidance tasks of each UAV; the chained PPO training framework adopts a distributed training method, the UAV currently being trained adopts the PPO algorithm, adds heuristic information as a guide, inputs the local observation state of the current UAV into the policy network and the value function network, the policy network outputs the action value to the training environment to update the state, and the value network outputs the Q value to evaluate the current state; the non-currently trained UAVs in the formation adopt a fixed strategy based on their respective local observation states to output their respective action amounts to the training environment to update the state; the training environment is a three-dimensional continuous space for the movement of multiple UAVs.

[0060] Among them, adding heuristic information as a guide includes: using heuristic speed to superimpose on the amount of action to improve the training speed and effect; among them, the expression of heuristic speed is:

[0061] ;

[0062] in, is the heuristic speed, is a small positive coefficient, For the current training drone (id is ) At this time, is the current training drone target point position, is the matrix composed of the position vectors of the drone group at this time, It is the matrix composed of the position vectors of the target points of the drone group.

[0063] Specifically, an improved deep reinforcement learning algorithm is used to realize action decision-making. In traditional methods, path planning is usually based on pre-defined map information, which performs poorly in dynamic environments. However, the deep reinforcement learning algorithm can learn the optimal action strategy in the process of continuous interaction with the environment, enabling drones to flexibly cope with various complex environments and obstacles.

[0064] At present, most of the research results on obstacle avoidance of multi-UAV swarm formations focus on two-dimensional plane motion planning, and the effect of reinforcement learning training for continuous motion of multiple machines is not ideal. The distributed reinforcement learning training under the chain training framework effectively solves the problem of unstable training in continuous three-dimensional multi-machine environment. The three-dimensional continuous space is used as the task environment, that is, the motion space of the UAV is three-dimensional, which greatly improves the freedom of UAV exploration. At the same time, the dense jungle task of multi-UAV formation obstacle avoidance is designed to increase the complexity of the environment and show the practicality of the algorithm.

[0065] Heuristic information is used as a control quantity to guide the UAV to fly to the target point, which effectively improves the convergence speed of the algorithm and improves the effect of quickly reaching the destination.

[0066] Deep reinforcement learning strategy training with heuristic actions: Due to the large three-dimensional continuous space, if no heuristic information is given, there are relatively few positive samples that the drone can reach the target point. Sparse positive samples are difficult to reproduce, and training is difficult. At the same time, due to the large collision penalty, the drone will learn hesitant decisions during training, and it is difficult to obtain a better strategy model. The following formula is used as a heuristic speed superimposed on the action amount, which greatly improves the training speed and effect.

[0067] The numerator on the right side of the heuristic speed expression is the vector pointing to the target point from the current drone, and the denominator is the norm of the vector matrix of all drones pointing to their respective target points. When a drone is too far ahead of other drones, the heuristic speed value is smaller, increasing the exploration degree. When it lags behind, the larger heuristic speed guides the drone to keep up with the formation.

[0068] Partial observation state, that is, the agent can only obtain partial information about the current state of the environment, rather than a full picture of the complete state. This situation is common in many application scenarios, such as self-driving cars can only perceive objects within a certain range around them, or robots can only see a limited field of view in front of them in a maze. Local observation increases the complexity of the agent's learning strategy because it needs to infer the global state from limited information and make decisions. This also enables the trained agent to have strong generalization and decision-making capabilities so that it can still effectively complete tasks even when information is incomplete.

[0069] The PPO (Proximal Policy Optimization) algorithm is a widely used algorithm in the field of reinforcement learning. It improves the quality of decision-making by optimizing the policy network. By improving the PPO algorithm, the autonomous navigation capability of multiple drones in complex environments can be further enhanced.

[0070] The chained PPO training framework adopts a distributed training method, which mainly solves the problem that the PPO algorithm is difficult to converge in a complex continuous environment with multiple agents. When each individual in the multi-agent performs independent and continuous actions, it will have a great impact on the overall state. When multiple agents are trained at the same time, the environment will be particularly unstable, and it is difficult to train a multi-agent strategy model in a multi-dimensional continuous motion space and state space. Using a distributed training method, in each training, the strategies of other drones that are not currently trained are relatively fixed, so that the current strategy adapts to the strategies of other drones and achieves the relative optimality of individuals.

[0071] In the training environment, after receiving the state information, the deep reinforcement learning method generates an action value through a random or deterministic strategy. The action will act on the unmanned vehicle and interact with the surrounding environment, mainly completing path planning and obstacle avoidance functions, and collecting experience and rewards through interaction. The algorithm will store the experience data in the playback buffer and regularly collect batch data from it for learning.

[0072] In deep reinforcement learning training, learning is done through path planning and obstacle avoidance, and interaction with the environment, which includes randomly selecting a series of actions and observing the feedback from the environment. The selection of these actions is based on the value function method, which aims to evaluate the potential value of each action to guide decision-making. The value function can help evaluate the long-term accumulation of multiple different types of rewards that can be brought by taking an action in the current state, thereby helping the agent make the best action decision. During the training process, by constantly interacting with the environment and adjusting the estimate of the value function based on feedback, the agent gradually improves its decision-making ability, allowing the agent to act more intelligently and efficiently in complex environments.

[0073] Step 104: Each UAV in the formation executes its own optimal strategy for path planning and obstacle avoidance tasks to achieve multi-UAV collaborative obstacle avoidance control.

[0074] In the above-mentioned multi-UAV collaborative obstacle avoidance control method based on the PPO algorithm, the method includes: constructing an intelligent agent, deploying an intelligent agent on each UAV, and the intelligent agent includes a policy network and a value network; using a chain PPO training framework to conduct multiple rounds of training on the UAVs in the formation to obtain the optimal strategy for the path planning and obstacle avoidance tasks of each UAV; each UAV in the formation executes its own optimal strategy for path planning and obstacle avoidance tasks to achieve multi-UAV collaborative obstacle avoidance control. During the training process, except for the UAV currently in training, other UAVs maintain a fixed strategy to effectively maintain the stability of the training environment; the chain training framework uses the local observation state of each UAV to estimate the amount of action, and under the guidance of heuristic information and reward functions, it can maintain a certain formation stability to reach the target point under the premise of avoiding obstacles, thereby improving the effect of quickly reaching the destination.

[0075] In one embodiment, an artificial potential field method is used in the first round of training to simulate the obstacle avoidance shuttle motion control strategy of the non-currently trained UAVs in the formation.

[0076] In one embodiment, step 102 includes: using the first drone in the formation as the currently trained drone; using the local observation state of the currently trained drone as the input of the policy network of the currently trained drone, and adding heuristic information as a guide, the policy network outputs an action value, and after the action is applied to the drone, it interacts with the training environment, collects experience and accumulates multiple types of rewards over a long period of time through interaction, calculates the policy gradient of each batch, and then performs gradient clipping and updates the policy network parameters; inputs the local observation state of the currently trained drone into the value network of the currently trained drone, outputs the Q value to evaluate the current state; calculates each sample in The minimum value function estimates of different actions taken under different states are used to update the value function of the target point, and then the loss function is used to update the value network parameters; the non-currently trained UAVs in the formation output their own action amounts to the training environment based on their respective local observation states and fixed strategies to update the UAV state; after the currently trained UAV is trained, the trained model is loaded on the UAV, and then the UAV closest to the currently trained UAV in the formation is selected as the currently trained UAV, and training continues until all UAVs in the formation are traversed, and then the next round of training is entered until the preset termination conditions are met, and the optimal strategy for the path planning and obstacle avoidance tasks of each UAV is obtained.

[0077] Specifically, based on the chain-based deep reinforcement learning training framework such as Figure 2 As shown in the figure, based on the chain deep reinforcement learning training framework, except for the drone currently in training, other drones maintain a fixed strategy during the training process, that is, the strategy will not participate in the training adjustment, which can effectively maintain the stability of the training environment. The drone currently in training uses the PPO algorithm, adds heuristic information as a guide, inputs the local observation state of the current drone into the policy network and the value function network, and the policy network outputs the action amount to the training environment to update the state, and the value function network outputs the Q value to evaluate the current state. Other drones not currently in training use existing strategies based on their respective states to output their respective action amounts to the environment for updating.

[0078] Since a single training only updates the current reinforcement learning network, it is difficult to achieve inter-machine collaboration, and multiple rounds of training are required to make the strategies adapt to each other. In the first round of training, the artificial potential field method is used to simulate the obstacle avoidance shuttle motion control strategy of other drones that are not currently being trained.

[0079] After selecting the first UAV for training, the nearest neighbor UAV in the formation is selected as the next UAV for training. The training framework is as follows: Figure 3 shown.

[0080] The arrow points to the next training drone. After training a drone, the trained model is loaded for the drone for subsequent training.

[0081] In deep reinforcement learning training, learning is done through path planning and obstacle avoidance, and interaction with the environment, which includes randomly selecting a series of actions and observing the feedback from the environment. The selection of these actions is based on the value function method, which aims to evaluate the potential value of each action to guide decision-making. The value function can help evaluate the long-term accumulation of multiple different types of rewards that can be brought by taking an action in the current state, thereby helping the agent make the best action decision. During the training process, by constantly interacting with the environment and adjusting the estimate of the value function based on feedback, the agent gradually improves its decision-making ability, allowing the agent to act more intelligently and efficiently in complex environments.

[0082] In one embodiment, the multiple different types of rewards are rewards for reaching the target point, collision penalties, path costs, and formation stability rewards.

[0083] The expression for reaching the target point reward is:

[0084] ;

[0085] in, Rewards for reaching the target point, is the track point information of the current training drone (id is i) (i.e. the position of the drone after the state is updated under the control of the previous action). is the target point vector of the current training drone (id is i).

[0086] Collision penalty: Since the task obstacles are divided into dynamic obstacles and static obstacles, the collision penalty is designed into two parts. The expression of the collision penalty is:

[0087] ;

[0088] ;

[0089] ;

[0090] in, is the number of sampling points along the way for a single step, used to calculate the collision penalty along the way; is the total collision penalty (the total collision penalty is the sum of the collision penalties of all sampling points), For other drones in the same group; and Respectively represent i The collision penalty between the current UAV and static obstacles and dynamic obstacles at each sampling moment. When the distance between the UAV and the obstacle is in different intervals, different discrete values ​​are selected as penalties to indicate the degree of danger at different distances.

[0091] In order for the drone trajectory to meet the mission requirements and be optimal, the path cost of each step needs to be designed during training to avoid unnecessary detours. The expression of the path cost is:

[0092] ;

[0093] in, is the path cost of the current step, m is a positive coefficient, They are the speeds of the current drone's x, y, and z axes, respectively.

[0094] The expression of formation stability reward is:

[0095] ;

[0096] ;

[0097] in, To reward the drone for maintaining the formation, the inverse of the distance between the drone’s current position and the expected position is calculated as the reward for maintaining the formation. The closer the drone is to the expected position, the greater the reward. For sparse rewards, As a sparse reward, the training drone is encouraged to have a speed similar to that of the virtual leader to maintain the stability of the formation drone. is the current drone position, The position of the drone expected by the formation at this time, is the current speed of the drone, is the speed of the formation leader, n For the range A real number.

[0098] UAV formations have attracted widespread attention due to their superior functionality and flexibility in performing complex and varied tasks such as search and rescue, resource exploration, reconnaissance and surveillance. To maintain the stability of the formation, a virtual leader-follow mode can be used for stability control. The leader-follow mode is a common control strategy that uses a virtual drone located at the center of the formation as the leader, and other drones as followers to maintain formation flight. Since the action constraint is set to an absolute value of the speed in each direction less than 0.4m / s, the virtual leader drone is designed to reach the vicinity of the target point at a speed of 0.4m / s, and the current virtual drone position is obtained by vector transformation to obtain the expected position of the current training drone. In order to maintain the stability of the formation when multiple drones are clustered, a reward function is designed to make the drones prefer to walk on the preset trajectory.

[0099] Specifically, reward function design is the core of model training. Reward function design is the core link in reinforcement learning, which directly defines the agent's behavioral goals and learning direction. A well-designed reward function can guide the agent to learn effectively to achieve the task, while an improper design may lead to low learning efficiency or the agent's behavior deviating from the expected goal. Therefore, when designing the reward function, it is necessary to carefully consider the specific requirements of the task, ensure that the reward signal can accurately reflect the quality of the agent's behavior, and continuously adjust and optimize in practical applications to improve the learning effect.

[0100] In this application, in order to make the swarm shuttle through dense forest obstacles to reach the target point, it is necessary to design the following reward functions, first of all, the reward for reaching the target point, the collision penalty (with static obstacle forests and dynamic obstacles other drones), the path cost and the time cost. By adjusting the parameters, the effect will be better.

[0101] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of the steps, and the steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequentially, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0102] In a verification embodiment, the software environment requirement for the simulation experiment is: install pytorch in a python3.8 environment. Tests were conducted in an experimental environment, and the experimental results show that the training difficulty of adding heuristic information is small, the convergence speed is fast, and the reward is not easy to oscillate. Under the condition of approximate path cost, the PPO-based reinforcement learning algorithm has higher risk resistance than the traditional artificial potential field algorithm, and can maintain a certain stability of the formation. This method has certain feasibility. If more ideal results are required, various parameters need to be adjusted during training. In the simulation test, static obstacles in the forest and target point positions are randomly generated. The drone swarm uses this training framework for distributed training. The reward change curve is shown in the figure. Figure 4 As shown in the figure, when the reward curve is stable, the optimal (average reward maximum) model is selected for testing. The single machine obstacle avoidance test effect diagram is as follows Figure 5 As shown in the figure, the effect of multi-machine obstacle avoidance test is as follows Figure 6 shown.

[0103] In one embodiment, a multi-UAV cooperative obstacle avoidance control device based on a PPO algorithm is provided, the device comprising: an agent building module, an optimal strategy determination module for path planning and obstacle avoidance tasks, and a multi-UAV cooperative obstacle avoidance control module, wherein:

[0104] The intelligent agent building module is used to build an intelligent agent. An intelligent agent is deployed on each drone. The intelligent agent includes a policy network and a value network.

[0105] The optimal strategy determination module for path planning and obstacle avoidance tasks is used to conduct multiple rounds of training on the UAVs in the formation using the chained PPO training framework to obtain the optimal strategy for path planning and obstacle avoidance tasks for each UAV; the chained PPO training framework adopts a distributed training method, and the currently trained UAV adopts the PPO algorithm, adding heuristic information as a guide, inputting the current UAV local observation state into the policy network and the value function network, the policy network outputs the action value to the training environment to update the state, and the value network outputs the Q value to evaluate the current state; the non-currently trained UAVs in the formation adopt a fixed strategy based on their respective local observation states to output their respective action amounts to the training environment to update the state; the training environment is a three-dimensional continuous space for the movement of multiple UAVs.

[0106] The multi-UAV collaborative obstacle avoidance control module is used for each UAV in the formation to execute its own optimal strategy for path planning and obstacle avoidance tasks, thereby realizing multi-UAV collaborative obstacle avoidance control.

[0107] Among them, the optimal strategy determination module for path planning and obstacle avoidance tasks is also used to superimpose the heuristic speed on the action amount to improve the training speed and effect; the heuristic speed is obtained by calculating the above heuristic speed.

[0108] In one embodiment, in the optimal strategy determination module for path planning and obstacle avoidance tasks, an artificial potential field method is used in the first round of training to simulate the obstacle avoidance shuttle motion control strategy of non-currently trained UAVs in the formation.

[0109] In one embodiment, the optimal strategy determination module for the path planning and obstacle avoidance tasks is also used to use the first drone in the formation as the currently trained drone; use the local observation state of the currently trained drone as the input of the policy network of the currently trained drone, and add heuristic information as a guide, the policy network outputs an action value, and after the action is applied to the drone, it interacts with the training environment, collects experience and accumulates multiple types of rewards over a long period of time through interaction, calculates the policy gradient of each batch, and then performs gradient clipping and updates the policy network parameters; inputs the local observation state of the currently trained drone into the value network of the currently trained drone, and outputs a Q value to evaluate the current state; Calculate the minimum function estimate of each sample taking different actions under different states, update the value function of the target point, and then use the loss function to update the value network parameters; the non-currently trained drones in the formation output their own action amounts to the training environment based on their respective local observation states and fixed strategies to update the drone state; after the currently trained drone is trained, the drone loads the trained model, and then selects the drone in the formation that is closest to the currently trained drone as the currently trained drone, and continues training until all drones in the formation are traversed, and enters the next round of training until the preset termination conditions are met, and the optimal strategy for path planning and obstacle avoidance tasks for each drone is obtained.

[0110] In one embodiment, the multiple different types of rewards are respectively a target point arrival reward, a collision penalty, a path cost, and a formation stability reward. The target point arrival reward is shown in the expression of the target point arrival reward; the collision penalty is shown in the expression of the collision penalty; the path cost is shown in the expression of the path cost; and the formation stability reward is shown in the expression of the formation stability reward.

[0111] For the specific definition of the multi-UAV collaborative obstacle avoidance control device based on the PPO algorithm, please refer to the definition of the multi-UAV collaborative obstacle avoidance control method based on the PPO algorithm above, which will not be repeated here. Each module in the above-mentioned multi-UAV collaborative obstacle avoidance control device based on the PPO algorithm can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0112] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multi-UAV collaborative obstacle avoidance control method based on the PPO algorithm is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0113] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0114] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above method embodiment when executing the computer program.

[0115] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0116] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A multi-UAV cooperative obstacle avoidance control method based on PPO algorithm, characterized in that: The method comprises: Building an intelligent agent, deploying an intelligent agent on each drone, wherein the intelligent agent includes a policy network and a value network; The chained PPO training framework is used to conduct multiple rounds of training on the drones in the formation to obtain the optimal strategy for path planning and obstacle avoidance tasks of each drone; the chained PPO training framework adopts a distributed training method, the drone currently being trained adopts the PPO algorithm, adds heuristic information as a guide, inputs the local observation state of the current drone into the policy network and the value function network, the policy network outputs the action value to the training environment to update the state, and the value network outputs the Q value to evaluate the current state; the drones not currently being trained in the formation adopt a fixed strategy based on their respective local observation states to output their respective action amounts to the training environment to update the state; the training environment is a three-dimensional continuous space for the movement of multiple drones; Each UAV in the formation executes its own optimal strategy for path planning and obstacle avoidance tasks, achieving multi-UAV collaborative obstacle avoidance control; Among them, heuristic information is added as a guide, including: The heuristic speed is superimposed on the action volume to improve the training speed and effect; wherein the heuristic speed is: in, is the heuristic speed, is a small positive coefficient, For the current training drone (id is ) At this time, is the current training drone target point position, is the matrix composed of the position vectors of the drone group at this time, The matrix composed of the position vectors of the target points of the drone group; Among them, the chained PPO training framework is used to conduct multiple rounds of training on the drones in the formation to obtain the optimal strategy for path planning and obstacle avoidance tasks for each drone, including: Set the first drone in the formation as the current training drone; The local observation state of the currently trained drone is used as the input of the policy network of the currently trained drone, and heuristic information is added as a guide. The policy network outputs an action value, and the action is applied to the drone to interact with the training environment. Through the interaction, experience is collected and multiple types of rewards are accumulated over a long period of time. The policy gradient of each batch is calculated, and then the gradient is clipped and the policy network parameters are updated; Input the local observation state of the currently trained drone into the value network of the currently trained drone, and output the Q value to evaluate the current state; calculate the minimum value function estimate of each sample taking different actions under different states, update the value function of the target point, and then use the loss function to update the value network parameters; The non-currently trained UAVs in the formation output their own action quantities to the training environment based on their own local observation states and fixed strategies to update the UAV states; After the currently trained UAV is trained, the UAV loads the trained model and then selects the UAV closest to the currently trained UAV in the formation as the currently trained UAV. The training continues until all UAVs in the formation are traversed and the next round of training is entered until the preset termination condition is met. The optimal strategy for the path planning and obstacle avoidance tasks of each UAV is obtained.

2. The multi-UAV cooperative obstacle avoidance control method based on the PPO algorithm according to claim 1 is characterized in that: In the first round of training, the artificial potential field method was used to simulate the obstacle avoidance and shuttle motion control strategy of non-currently trained UAVs in the formation.

3. The multi-UAV cooperative obstacle avoidance control method based on the PPO algorithm according to claim 1 is characterized in that: There are many different types of rewards, including rewards for reaching the target point, collision penalties, path costs, and formation stability rewards; The reward for reaching the target point is: in, Rewards for reaching the target point, is the trajectory point information of the current training drone, is the target point vector of the current training drone; The collision penalty is: in, is the number of sampling points along a single step; is the total collision penalty, For other drones in the same group; and Respectively represent i The collision penalty between the current drone and static obstacles and dynamic obstacles at each sampling moment; The path cost is: in, is the path cost of the current step, m is a positive coefficient, They are the current x, y, and z axis speeds of the drone respectively; The formation stability reward is: in, To maintain the formation's rewards, For sparse rewards, is the current drone position, The position of the drone expected by the formation at this time, is the current speed of the drone, is the speed of the formation leader, n For the range A real number.

4. A multi-UAV cooperative obstacle avoidance control device based on PPO algorithm, characterized in that: The device comprises: An intelligent agent building module is used to build an intelligent agent, and deploy an intelligent agent on each drone, wherein the intelligent agent includes a policy network and a value network; The optimal strategy determination module for path planning and obstacle avoidance tasks is used to use a chained PPO training framework to perform multiple rounds of training on the drones in the formation to obtain the optimal strategy for path planning and obstacle avoidance tasks for each drone; the chained PPO training framework adopts a distributed training method, the drone currently being trained adopts the PPO algorithm, adds heuristic information as a guide, inputs the local observation state of the current drone into the policy network and the value function network, the policy network outputs the action value to the training environment to update the state, and the value network outputs the Q value to evaluate the current state; the drones not currently being trained in the formation adopt a fixed strategy based on their respective local observation states to output their respective action amounts to the training environment to update the state; the training environment is a three-dimensional continuous space for the movement of multiple drones; The multi-UAV collaborative obstacle avoidance control module is used for each UAV in the formation to execute its own optimal strategy for path planning and obstacle avoidance tasks, thus realizing multi-UAV collaborative obstacle avoidance control; Among them, the optimal strategy determination module for path planning and obstacle avoidance tasks is also used to superimpose the heuristic speed on the action amount to improve the training speed and effect; wherein the heuristic speed is: in, is the heuristic speed, is a small positive coefficient, For the current training drone (id is ) At this time, is the current training drone target point position, is the matrix composed of the position vectors of the drone group at this time, The matrix composed of the position vectors of the target points of the drone group; The optimal strategy determination module for the path planning and obstacle avoidance tasks is also used to use the first UAV in the formation as the currently trained UAV; use the local observation state of the currently trained UAV as the input of the policy network of the currently trained UAV, and add heuristic information as a guide. The policy network outputs an action value, and the action is applied to the UAV to interact with the training environment, collect experience and accumulate multiple types of rewards over a long period of time through interaction, calculate the policy gradient of each batch, and then perform gradient clipping and update the policy network parameters; input the local observation state of the currently trained UAV into the value network of the currently trained UAV, output the Q value to evaluate the current state; calculate each The minimum value function estimate of each sample taking different actions in different states is used to update the value function of the target point, and then the loss function is used to update the value network parameters; the non-currently trained UAVs in the formation output their own action amounts to the training environment based on their respective local observation states and fixed strategies to update the UAV state; after the currently trained UAV is trained, the trained model is loaded on the UAV, and then the UAV closest to the currently trained UAV in the formation is selected as the currently trained UAV, and training continues until all UAVs in the formation are traversed, and then the next round of training is entered until the preset termination conditions are met, and the optimal strategy for path planning and obstacle avoidance tasks of each UAV is obtained.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the multi-UAV collaborative obstacle avoidance control method based on the PPO algorithm described in any one of claims 1 to 3 is implemented.

Images