Multi-uav cooperative path planning method and system, computing module and storage medium

Abstract

The application relates to the field of unmanned aerial vehicle control and artificial intelligence, in particular to a multi-unmanned aerial vehicle cooperative path planning method and system, a computing module and a storage medium, the method comprising environment perception and state collection, non-expert strategy guidance, CVAE strategy distillation, disturbance network exploration enhancement, multi-agent reinforcement learning cooperation, path execution and iterative optimization; through combination of non-expert strategy guidance and individualized distillation technology, learning efficiency and strategy quality are improved.

Description

Technical Field

[0001] This application relates to the fields of unmanned aerial vehicle (UAV) control and artificial intelligence, and in particular to methods, systems, computing modules and storage media for multi-UAV cooperative path planning. Background Technology

[0002] Currently, multi-UAV systems are widely used in collaborative missions in complex and dynamic environments, such as search and rescue, logistics delivery, environmental monitoring, and communication. However, achieving efficient path planning and collaborative control of multiple UAVs still faces many challenges, including obstacle avoidance, energy conservation, adaptability to dynamic environments, and efficiency in task allocation and coordination.

[0003] Existing path planning methods mainly include heuristic algorithms, rule-based algorithms, and learning-based algorithms. Heuristic algorithms (such as genetic algorithms and particle swarm optimization) find the optimal path by simulating natural evolution and group behavior. These algorithms perform well in various scenarios and can achieve high path planning efficiency. However, due to the large amount of searching and computation required, their computational cost is high, limiting their application on resource-constrained UAV platforms.

[0004] Rule-based path planning methods (such as Artificial Potential Field (APF) and distributed control) provide a reliable theoretical foundation and initial solution space, significantly improving path planning efficiency and reducing computation time. However, these methods are prone to getting trapped in local optima, lack flexibility, and are difficult to adapt to the real-time adjustment requirements in complex dynamic environments.

[0005] Learning-based algorithms, such as path planning methods utilizing reinforcement learning (RL), achieve autonomous learning and path optimization through interaction with the environment, exhibiting strong adaptability and generalization capabilities. In particular, multi-agent reinforcement learning (MARL) algorithms can improve path planning efficiency and task coordination through collaborative learning among multiple UAVs. However, with increasing environmental complexity, the state space of multi-agent systems grows exponentially, posing a significant challenge to the application of MARL in multi-UAV path planning tasks. The main challenge lies in low sampling efficiency, as agents require extensive interaction with the environment to converge to the optimal strategy. This reliance on numerous interactions makes this process impractical in the real world, where data collection is often limited by scarce resources.

[0006] To address the shortcomings of the aforementioned methods, policy distillation has been introduced in recent years as an effective way to improve the computational efficiency of MARL. Zhang et al. proposed a non-expert policy-enhanced UAV obstacle avoidance method, utilizing APF to generate non-expert policies to guide the agent in the initial learning phase, thus avoiding the inefficiency of random exploration. Qiu et al. further developed a two-stage MARL algorithm assisted by non-expert policies, where the trajectory generated by APF is used for pre-training, and the non-expert policy serves as prior guidance for online learning, significantly improving sample efficiency. Although these methods improve the sample efficiency of MARL to some extent, they often rely on the direct guidance of non-expert policies, ignoring the potential distributional differences between non-expert policies and the target policy, as well as the inherent diversity of policies. This direct guidance may lead to the agent's over-reliance on non-expert policies, thereby limiting its exploration capabilities and ultimately affecting the optimality and robustness of the final policy. Therefore, how to effectively utilize non-expert policy guidance while mitigating its negative impact on the agent's exploration capabilities remains a key research challenge.

[0007] In summary, existing technologies for multi-UAV collaborative path planning still suffer from problems such as high computational cost, susceptibility to local optima, lack of flexibility, and insufficient adaptability to dynamic environments. There is an urgent need for a new path planning method that can comprehensively utilize the advantages of rule-based and learning methods to improve the collaborative efficiency and robustness of multi-UAV systems in complex environments. Summary of the Invention

[0008] To address the aforementioned technical issues, this application provides a multi-UAV cooperative path planning method, system, computing module, and storage medium.

[0009] Firstly, this application provides a multi-UAV cooperative path planning method: The multi-UAV cooperative path planning method mainly includes the following steps: S1. Environmental perception and status acquisition: Acquire observation information, which includes at least one of the following: the position and speed of the UAV, the static / dynamic status of obstacles within the perception range, the coordinates of the target area, and the relative position and motion status of other UAVs in the neighborhood; S2. Non-expert strategy guidance: Based on the artificial potential field (APF) model, calculate the resultant force of the UAV under the combined action of gravity (pointing towards the target area), repulsion (moving away from obstacles), and interaction forces between agents (avoiding UAV collisions) to generate initial actions as the basic path guidance for non-expert strategies. S3. CVAE Strategy Distillation: Construct a Conditional Variational Autoencoder (CVAE), input the observation information from step S1 into the CVAE encoder and map it into latent variables. The decoder then reconstructs the reconstructed action, which serves as a personalized initial strategy to adapt to multi-UAV path planning tasks, realizing the knowledge extraction and flexible transfer of non-expert strategies. S4. Enhanced Perturbation Network Exploration: Using the reconstruction action in step S3 and the observation information in step S1 as input, a perturbation vector is generated through the perturbation network, and an optimized action is generated based on the perturbation vector to enhance the path exploration capability; S5. Multi-agent reinforcement learning collaboration: A centralized training with distributed execution framework (CTDE) is adopted, which uses a centralized critique network to evaluate the global action value and combines distillation loss and reinforcement learning loss to optimize the participant's policy; S6. Path Execution and Iterative Optimization: Each UAV executes actions in a distributed manner based on the optimized strategy, updates its own position and environmental status, and iteratively executes steps S1 to S5 until all UAVs complete the collaborative path planning for the target area.

[0010] Optionally, in step S4, generating the optimization action based on the perturbation vector includes generating the optimization action according to the following formula:

[0011] In the formula, Indicates an optimized action; Indicates a refactoring action; , representing the hyperparameter controlling the amplitude of the disturbance; , represents the perturbation vector. It is the action space dimension.

[0012] Optionally, in step S5, the distillation loss L d Calculate using the following formula:

[0013] In the formula, E o~D Indicates in observation information The expected value on the corresponding distribution D; KL(||) represents the KL divergence; Indicates the participants' strategies; This indicates the perturbation network output strategy; The overall policy loss combines the reinforcement learning objective with the distillation loss and is calculated using the following formula:

[0014] In the formula, L policy Let L represent the overall policy loss; (1-λ) represent the weights controlling the reinforcement learning loss, and λ represent the control distillation loss L. d Weights; E s~D This represents the mathematical expectation of the distribution D corresponding to the environmental state s; This represents a network of critics used to evaluate actions performed in environment state s. a i The higher the value, the better the path. The parameters representing the critic network; π i Indicates the first i A policy network for individual agents; o i Indicates the first i Observational information of each agent; Indicates the first i Individual agent policy network π i Parameters; Among them, weight λ Dynamically adjust using the following formula:

[0015] In the formula, The number of times the policy network is updated. This represents the total number of training sets.

[0016] Optionally, in step S2, calculating the resultant force of the UAV under the combined action of gravity, repulsion, and inter-agent interaction forces includes using the following formula:

[0017] In the formula, Indicates resultant force; Represents gravity; Indicates repulsive force; This represents the interaction forces between intelligent agents.

[0018] Optionally, in step S2, generating the initial action includes: Calculate the direction vector of the unit composite force based on the resultant force:

[0019] In the formula, It is the direction vector of the unit composite force, and its direction is parallel to... The direction is consistent with that of the system's velocity; Initial motion is generated based on the direction vector of the unit composite force.

[0020] In the formula, Indicates the maximum speed. Indicates the initial action.

[0021] Secondly, this application provides a multi-UAV cooperative path planning system; the system includes modules for performing the method in the first aspect or any possible implementation of the first aspect: This multi-UAV collaborative path planning system includes: An environmental perception module is used to collect observation information, which includes at least one of the following: the position and speed of the UAV, the static / dynamic state of obstacles within the perception range, the coordinates of the target area, and the relative position and motion state of other UAVs in the neighborhood. The non-expert strategy calculation module is used to calculate the resultant force of the UAV under the combined action of gravity, repulsion and inter-agent interaction forces based on the artificial potential field (APF) model, and generate the initial action as the basic path guidance for the non-expert strategy. The CVAE policy distillation module is used to input observation information into the CVAE encoder and map it into latent variables. The reconstructed action is obtained by the decoder and serves as a personalized initial policy to adapt to multi-UAV path planning tasks, realizing the knowledge extraction and flexible transfer of non-expert policies. The perturbation network exploration module is used to generate perturbation vectors through a perturbation network, taking reconstructed actions and observation information as input, and generating optimized actions based on the perturbation vectors to enhance path exploration capabilities. The multi-agent reinforcement learning collaborative module is used to adopt a centralized training and distributed execution CTDE framework, utilize a centralized criticism network to evaluate the global action value, and combine distillation loss and reinforcement learning loss to optimize the participant's policy. The path execution and iteration module executes actions in a distributed manner based on the optimized strategy, updates its own position and environmental status, until all drones complete the collaborative path planning for the target area.

[0022] Thirdly, this application provides a UAV computing module, including a processor, a memory, and a communication bus. The communication bus is used to realize a communication connection between the processor and the memory. The processor is used to execute a computer program stored in the memory to implement the method described in any of the preceding claims.

[0023] Fourthly, this application also provides a computer-readable storage medium storing a computer program; the computer program can be executed by a processor to implement the method described above.

[0024] Fifthly, this application also provides a computer program product, including a computer program that can be executed by a processor to implement the method described above.

[0025] This application includes at least the following beneficial technical effects: 1. Based on the resultant force calculation of APF attraction / repulsion / interaction forces, the initial path of the UAV is equipped with the foundation for obstacle avoidance, target movement, and collision avoidance, ensuring that the path can be traversed in multi-UAV collaboration as much as possible. Furthermore, the rigid non-expert strategy is distilled through CVAE to transform it into personalized guidance that can be adapted to different scenarios, avoiding path rigidity caused by differences in tasks among multiple UAVs (such as target priority and obstacle distribution). At the same time, the dynamic adjustment and optimization of CVAE actions by perturbation network is combined with the perturbation vector generated by dual input of observation information and reconstructed actions, allowing the UAV to actively explore the possibility of shorter paths and better collaboration on a safe basis, overcoming the problem that APF is prone to getting trapped in local optima (such as detour obstacle avoidance and collaboration conflict).

[0026] 2. In addition, a centralized critique network based on the CTDE framework breaks through the limitations of local observation by a single UAV, evaluates the value of multiple UAV actions from a global perspective, solves the problem of individual optimization in multi-UAV collaboration, achieves global optimization, and realizes the efficiency of the overall path.

[0027] 3. The weight λ decays with training, which allows the distillation loss and reinforcement learning loss to maintain a dynamic balance. This ensures that the path planning relies on APF to ensure safety in the early stages and allows for exploration to reach the optimal value in the later stages. Attached Figure Description

[0028] Figure 1 A schematic diagram of a multi-UAV cooperative path planning algorithm framework provided in this application embodiment; Figure 2 This is a schematic diagram of a multi-UAV cooperative path planning method provided in an embodiment of this application; Figure 3 A schematic diagram comparing the learning curves of different path planning methods provided in this application embodiment; Figure 4 A schematic diagram comparing the learning curves of another different path planning method provided in an embodiment of this application; Figure 5 A schematic diagram comparing the inference time of different algorithm models provided in an embodiment of this application; Figure 6 This application provides a schematic diagram comparing the success rates of different models under different numbers of obstacles, as part of an embodiment of the present application. Figure 7 This application provides a schematic diagram comparing the success rates of different models with different numbers of drones in an embodiment of the present application. Figure 8A schematic diagram comparing the success rates (SR) of the MADDPG variant provided in this application embodiment with the original MADDPG under different numbers of drones (left) and obstacles (right); Figure 9 This application provides a schematic diagram of the structure of a multi-UAV cooperative path planning system. Figure 10 This is a schematic diagram of a drone computing module structure provided in an embodiment of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0030] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items. The term “exemplary” means “serving as an example, embodiment, or illustration,” and any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments. The terms “first” and “second” are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as “first” or “second” may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, “a plurality” means two or more.

[0031] Current multi-agent reinforcement learning-based collaborative path planning methods for multiple unmanned aerial vehicles (UAVs) suffer from high computational costs and a tendency to get trapped in local optima. This invention aims to generate high-quality guidance policies using a rule-driven teacher model and, through policy distillation, enable each agent to learn personalized and efficient policies tailored to its specific task.

[0032] To better understand this invention, the technical terms used in this invention are explained below: Unmanned Aerial Vehicles (UAVs): These are physical carriers that perform path planning tasks; they are hardware entities that actually participate in environmental interaction, navigation, obstacle avoidance, and task coordination. In this embodiment, each UAV is explicitly regarded as an independent intelligent agent, and its physical attributes such as position, speed, and perception range are the basic parameters for path planning.

[0033] Agent: An abstract representation of a drone from an algorithmic perspective, and a core concept in the Multi-Agent Reinforcement Learning (MARL) framework. Each drone is modeled as an agent in the algorithm, possessing the ability to perceive the environment (acquire observation information), generate actions (plan paths), and optimize policies through learning.

[0034] Actor: In multi-agent reinforcement learning, the "actor" module is the specific component that generates actions for the agent.

[0035] At the physical level, drones are abstracted into intelligent agents through algorithms. These intelligent agents generate actions through internal participants (actor modules), ultimately driving the drones to complete path planning tasks.

[0036] The multi-UAV cooperative path planning algorithm proposed in this embodiment can achieve efficient cooperative path planning for multiple UAVs in complex dynamic environments. The algorithm framework is as follows: Figure 1 As shown, it includes an APF-based non-expert policy generation module, a CVAE-based personalized policy distillation module, a perturbation network exploration enhancement module, and a multi-agent reinforcement learning collaborative module.

[0037] The non-expert policy generation module of APF provides a safety net for initial path guidance for multiple drones, avoiding the inefficiency of random exploration in the early stages of reinforcement learning. It calculates the resultant force of the three types of forces acting on the drone using an artificial potential field (APF), transforming it into initial action commands (such as velocity direction) to form the basic path of the non-expert policy. As a teacher policy, it provides a structured initial solution space for subsequent learning, addressing the problem of blind and collision-prone exploration in the early stages of reinforcement learning.

[0038] The personalized policy distillation module based on CVAE is used to transform the rigid policy of APF into transferable and personalized guidance signals, avoiding the agent's over-reliance on non-expert policies. The Conditional Variational Autoencoder (CVAE) extracts knowledge from the policy generated by the APF. The encoder maps the UAV's observation information into latent variables, capturing the core logic of the APF policy. The decoder reconstructs the APF actions based on the latent variables, generating personalized initial policies adapted to different UAV scenarios. As a policy converter, it retains the safety logic of the APF while enabling flexible knowledge transfer through latent variables, solving the problem of rigid APF policies and difficulty in adapting to complex dynamic environments.

[0039] The Perturbation Network Exploration Enhancement Module introduces strategic perturbations into the personalized policies generated by CVAE, overcoming the local optima limitations of non-expert policies. The Perturbation Network takes the actions generated by CVAE and the current observations as input to generate perturbation vectors to optimize the actions. As an exploration engine, it allows the agent to try better paths safely, addressing the problem of APF easily getting trapped in local optima.

[0040] The multi-agent reinforcement learning collaborative module achieves global collaborative optimization of multiple UAVs through centralized training and distributed execution of the CTDE framework. It utilizes a shared critic network to evaluate the global action value of all UAVs (e.g., whether multi-UAV congestion exists, and whether the overall path is efficient) to optimize policy loss. Each UAV independently generates actions based on local observations and personalized policies, avoiding the impact of communication latency on real-time performance. As the collaborative hub, it balances individual optimality with global optimality, addressing the problem of individual path optimization but overall inefficiency in multi-UAV collaboration.

[0041] This algorithmic framework optimizes multi-UAV path planning based on the following path: 1. APF provides a safety foundation → 2. CVAE extracts personalized knowledge → 3. Perturbation network enhances exploration → 4. MARL achieves global collaboration, ultimately generating multi-UAV paths. The core innovation of this framework lies in combining the efficiency of the rule-based APF method with the adaptability of the learning-based MARL method, while simultaneously addressing the pain points of over-reliance on non-expert strategies and inefficient exploration through CVAE and perturbation networks.

[0042] This invention investigates the cooperative path planning problem for a multi-UAV system in a three-dimensional bounded airspace. Consider a multi-UAV cooperative path planning system environment consisting of U homogeneous UAVs, O obstacles, and a target area. The objective of the multi-UAV cooperative path planning task is to generate optimal paths for the UAVs from their respective starting positions to the shared target area, while ensuring obstacle avoidance and safe operation throughout the process. Each UAV performs path planning and obstacle avoidance by detecting its surrounding environment. The UAV's position, velocity, and perception range are denoted as follows: and ,in Furthermore, obstacles in the mission area can be static or dynamic, and their center locations are indicated as follows: ,in To simplify calculations, both drones and obstacles are treated as point masses, with their shape and size ignored. Therefore, the dynamics of each drone i are modeled as particle dynamics.

[0043] This invention proposes a Personalized Policy Distillation Multi-Agent Actor-Critic algorithm (PPDAC), a novel framework that combines non-expert policy guidance with personalized distillation techniques to improve the efficiency and effectiveness of multi-UAV cooperative path planning. The method consists of several key modules, each addressing specific challenges of Multi-Agent Reinforcement Learning (MARL) in UAV path planning. These modules include: (1) non-expert policy generation based on the Artificial Potential Field (APF) method; (2) policy distillation using a Conditional Variational Autoencoder (CVAE); (3) a Perturbation Network (PN) for augmented exploration; and (4) a multi-agent actor-critic framework with personalized learning.

[0044] refer to Figure 2 The multi-UAV cooperative path planning method provided in this embodiment mainly includes the following steps: S1. Environmental Perception and State Acquisition: Acquire observation information, which includes at least one of the following: the UAV's position, speed, static / dynamic state of obstacles within the perception range, target area coordinates, and relative position and motion state of other UAVs in the neighborhood; S2. Non-Expert Strategy Guidance: Based on the Artificial Potential Field (APF) model, calculate the resultant force of the UAV under the combined action of gravity, repulsion, and inter-agent interaction forces to generate initial actions as the basic path guidance for non-expert strategies; In multi-agent reinforcement learning, an excessively large state-action space often leads to low exploration efficiency in the early stages of learning. To alleviate this problem, this embodiment of the invention employs the APF method to generate non-expert policies, providing structured initial guidance to the agents. APF is a mature path planning technique that simulates gravity pulling the agent towards the target region and repulsion pushing it away from obstacles.

[0045] Specifically, in step S2, calculating the resultant force of the UAV under the combined action of gravity, repulsion, and inter-agent interaction forces includes using the following formula:

[0046] In the formula, Indicates resultant force; Represents gravity; Indicates repulsive force; This represents the interaction forces between intelligent agents.

[0047] In step S2, generating the initial action specifically includes: Calculate the direction vector of the unit composite force based on the resultant force:

[0048] In the formula, It is the direction vector of the unit composite force, and its direction is parallel to... The direction is consistent with that of the system's velocity; Initial motion is generated based on the direction vector of the unit composite force.

[0049] In the formula, Indicates the maximum speed. Indicates the initial action.

[0050] S3. CVAE Policy Distillation: Construct a Conditional Variational Autoencoder (CVAE), input the observation information from step S1 into the CVAE encoder and map it into latent variables. The reconstructed action is obtained through the decoder and serves as a personalized initial strategy to adapt to multi-UAV path planning tasks, realizing the knowledge extraction and flexible transfer of non-expert strategies. To reduce the potential over-reliance on non-expert policies that may result in excessive agent dependence and to improve the model's exploration and optimization capabilities, this invention proposes a trainable shared embedding function based on a Conditional Variational Autoencoder (CVAE). This function captures the latent distribution of non-expert policies generated by the APF and transforms it into flexible guidance signals that support personalized learning. By learning rule-based observation embeddings, CVAE enables the agent to extract latent representational knowledge from non-expert policies while retaining the flexibility to adaptively improve its policies. Mapping to latent variables This variable is used to reconstruct the actions generated by the APF. To obtain the reconstruction action CVAE contains an encoder. A decoder A conditional prior The training objective is to minimize the following loss function:

[0051] In the formula, L CVAE This represents the total loss of the CVAE, which includes the reconstruction loss and the KL divergence regularization term, used to optimize the encoder and decoder of the CVAE. By minimizing this loss, the CVAE can accurately learn the action logic of non-expert policies while ensuring the flexibility of knowledge transfer.

[0052] Reconstruction Loss represents the reconstruction error of the generative model and measures the generation distribution. With real data The degree of matching. The negative sign indicates that the loss function must maximize the action reconstruction probability by minimizing the negative log-likelihood, ensuring that the action output by the decoder is consistent with the non-expert action.

[0053] The expected value (i.e., average loss) of the distribution of latent variables generated by the encoder is denoted by z; z represents the latent variable, which is the compressed knowledge carrier extracted by CVAE from the input and contains the core logic of non-expert strategies (such as execution decisions). This represents the probability distribution of the encoder, with the input being observation information o and non-expert actions. The output is the distribution of the latent variable z. ϕ (Network parameters of the encoder). This represents the probability distribution of the decoder, with inputs of latent variable z and observation information o, and aims to output the initial action. Distribution ( (Network parameters for the decoder).

[0054] This represents the KL divergence regularizer, used to constrain the reasonableness of the distribution of latent variables and measure the difference between two probability distributions (the smaller the value, the closer the distributions). This represents the conditional prior distribution, such as that based on observation information. o The standard normal distribution represents the ideal distribution that a latent variable should have under the current observations.

[0055] In this embodiment, the initial action generated by APF through attraction and repulsion is a reliable basic strategy, but it is rigid and prone to getting trapped in local optima. To address this, the reconstruction loss forces the decoder to generate an action highly similar to the initial action under the input of latent variable z and observation information o, ensuring that CVAE extracts the core obstacle avoidance and cooperative logic of APF; for example, adjusting the direction to the right when the obstacle is on the left.

[0056] However, if only reconstruction loss is used, CVAE may overfit the specific actions of APF and fail to cope with new scenarios. Therefore, the latent variable distribution generated by the encoder constrained by KL divergence regularization is close to the prior distribution, making the latent variable z an abstract policy feature rather than a replication of specific actions. For example, z can represent the abstract logic of obstacle avoidance priority > target approach priority, rather than a specific speed value, thus adapting to different obstacle distribution scenarios. This avoids CVAE rote memorization of APF actions and enhances the generalization ability of knowledge.

[0057] S4. Perturbation Network Exploration Enhancement: Taking the reconstruction action in step S3 and the observation information in step S1 as input, a perturbation vector is generated through the perturbation network, and an optimization action is generated based on the perturbation vector, thereby enhancing the path exploration capability; To reduce the suboptimal nature of non-expert strategies, this embodiment introduces a perturbation network. This network can strategically improve the reconstruction actions generated by CVAE, thereby enhancing exploration and enabling the agent to discover better strategies. Specifically, in step S4, generating optimized actions based on the perturbation vector includes generating optimized actions according to the following formula:

[0058] In the formula, Indicates an optimized action; Indicates a refactoring action; , representing the hyperparameter controlling the amplitude of the disturbance; , represents the perturbation vector. It is the action space dimension.

[0059] Optionally, the δ generated by the perturbation network is not a random perturbation, but a strategic improvement based on current observations. When it is observed that the APF policy may get stuck in a local optimum (due to interference from multiple dynamic obstacles or path conflicts between multiple drones), δ will output a targeted adjustment (such as a slight change in direction to find a shorter path, or an adjustment in speed to avoid neighboring drones). The perturbation amplitude is controlled by hyperparameters to avoid excessive adjustment that could compromise basic safety, preventing the drone from directly colliding with obstacles due to perturbations. Through dynamic control of hyperparameters, exploration is encouraged in the early stages of training to quickly find potential better paths, while convergence and stability are achieved in the later stages to avoid meaningless adjustments, ultimately generating an optimized path that is safe in obstacle avoidance, highly efficient in collaboration, and adaptable to the scenario.

[0060] In an optional embodiment of this application, the perturbation vector can be determined in the following way: The input to the perturbation network is the fusion feature of the current observation and the CVAE reconstruction action, and the specific processing is as follows: The current observation includes local environmental information of the UAV, such as its own 3D coordinates, velocity, the position and motion state of obstacles within the perception range (static obstacle coordinates / dynamic obstacle coordinates + velocity), target area coordinates, and the relative positions of other UAVs in the vicinity (to avoid cooperative collisions). This information is quantized into a dimension... d o vector O i .

[0061] CVAE reconstructs the motion, providing the UAV's three-dimensional velocity commands (such as...). ), dimension d a (Consistent with the dimension of action space).

[0062] observation vector O i With refactoring action splicing into dimensions fused feature vector , as input to the perturbation network.

[0063] The perturbation network uses a lightweight fully connected neural network to adapt to the resource-constrained characteristics of drones, and its structure is as follows: Input layer: Receives the fused feature vector X (dimension X) ).

[0064] Hidden layers: Set 2 to 3 fully connected layers, with the number of neurons decreasing layer by layer (e.g., 256 in the first layer and 128 in the second layer), and use ReLU as the activation function (to enhance the ability to extract non-linear features).

[0065] Output layer: Number of neurons is (Consistent with the action space dimension), the activation function is Tanh (which limits the output value to the range of [-1,1] to avoid excessive perturbation that could lead to policy instability).

[0066] The network is trained using gradient descent, with the goal of maximizing the expected cumulative reward. The loss function is defined as:

[0067] In the formula, D represents the distribution of observation-action pairs. It is a strategy based on perturbation actions. As a discount factor, T The round length is given. The target's perturbation trajectory is approximated using Monte Carlo sampling.

[0068] To ensure stability and prevent excessive bias, a weighted penalty term is added to the loss function. The total loss is:

[0069] In the formula, Represents the perturbation vector The squared L2 norm is used to limit the amplitude of the disturbance.

[0070] S5. Multi-agent reinforcement learning collaboration: Adopting a centralized training and distributed execution CTDE framework, utilizing a centralized criticism network to evaluate the global action value, and combining distillation loss and reinforcement learning loss to optimize participant policies; To integrate policy distillation and reinforcement exploration into a cohesive MARL framework, this embodiment proposes a framework based on the centralized training distributed execution (CTDE) paradigm. In this approach, each UAV is treated as an independent agent, and the combination of CVAE-based policy distillation and perturbation networks ensures that UAVs can develop personalized, high-quality policies while also benefiting from non-expert guidance.

[0071] Specifically, for each intelligent agent i There are two critic networks and Their parameters are respectively and To incorporate personalized learning, this embodiment of the invention introduces a distillation loss, which incorporates the participant's strategy... With perturbation network output strategy Alignment:

[0072] In the formula, E o~D Indicates in observation information The expected value on the corresponding distribution D; KL(||) represents the KL divergence; Indicates the participants' strategies; This indicates the perturbation network output strategy; The overall policy loss combines the reinforcement learning objective with the distillation loss and is calculated using the following formula:

[0073] In the formula, L policy L represents the overall policy loss; (1-λ) represents the weights controlling the reinforcement learning loss, and λ represents the control distillation loss L. d Weights; E s~D This represents the mathematical expectation of the distribution D corresponding to the environmental state s; This represents a network of critics used to evaluate actions performed in environment state s. a i The higher the value, the better the path. The parameters representing the critic network; a i Indicates the first i The action output by the intelligent agent; π i Indicates the first i A policy network for individual agents; o i Indicates the first i Observational information of each agent; Indicates the first i Individual agent policy network πi Parameters; The weight λ is dynamically adjusted using the following formula:

[0074] In the formula, The number of times the policy network is updated. This represents the total number of training sets.

[0075] It should be noted that the environmental state 's' refers to the objectively existing complete set of environmental information, encompassing all elements related to multi-UAV path planning, including but not limited to the positions and speeds of all UAVs; the positions and motion states of all obstacles (static / dynamic); the coordinates of the target area; and environmental constraints (terrain boundaries). Essentially, it represents the global, true state of the environment, theoretically containing all the information needed for decision-making. However, a single UAV cannot directly acquire the complete environmental state (limited by its perception range and communication capabilities). Observational information, on the other hand, is localized information acquired by a single UAV through its own sensors and local communication. It represents a partial perception of the environmental state 's', specifically including: its own position and speed; obstacle information within its perception range; the states of other UAVs in the vicinity (obtained through communication module interaction); and the relative position of the target area (calculated based on its own position). Essentially, it represents the UAV's localized perception of the environment, limited by sensor accuracy, perception range, and communication distance, and may contain incomplete information or noise (such as failure to perceive distant obstacles).

[0076] For example, environmental conditions s The environmental state s contains the global coordinates of all 35 obstacles, but the observation information of a certain drone may only contain the locations of 10 obstacles within its perception range (limited by the sensor detection distance); for example, environmental state s contains the global coordinates of all 11 drones, but the observation information of a certain drone may only contain the relative positions of 3 neighboring drones within its communication range.

[0077] It should be understood that the optimized actions output by the perturbation network serve as exploration action samples for the actor network. These actions interact with the environment to generate feedback (e.g., whether a collision occurs, or whether the target is approached), and are then input into the centralized critique network. The critique network evaluates the global value of this action (e.g., whether it improves overall collaborative efficiency) based on global observations (the states and actions of all UAVs), and optimizes the actor network parameters through the overall policy loss function. This allows MARL to learn the exploration logic of the perturbation network and gradually internalize strategic perturbations as part of its own strategy. A well-trained actor network can directly output actions that integrate the basic CVAE logic and the perturbation exploration logic (without additional intervention from the perturbation network), achieving a transition from relying on external perturbations to autonomously generating optimized actions.

[0078] The optimized actions of the perturbation network serve as high-quality learning samples for the participant policies. The participant policies align with the optimized actions of the perturbation network through loss distillation and reinforcement learning, gradually learning and internalizing these optimization logics. Ultimately, the participant policies can autonomously generate perturbation-optimized actions. In the Multi-Agent Reinforcement Learning (MARL) framework, the policies output by the participant networks are used to generate the final execution actions of the UAV. Its core objective is to generate safe and globally optimal cooperative path actions by learning from environmental feedback and policy distillation knowledge, directly determining the UAV's flight decisions.

[0079] S6. Path Execution and Iterative Optimization: Each UAV executes actions in a distributed manner based on the optimized strategy, updates its own position and environmental status, and iteratively executes steps S1 to S5 until all UAVs complete the collaborative path planning for the target area.

[0080] After performing a maneuver, the drone updates its own position and environmental observations in real time, repeating the closed loop of "observation → action generation → execution → update" (until the termination conditions are met, such as reaching the target area or reaching the required number of iterations), ultimately forming an optimized path that is safe in obstacle avoidance, highly efficient in collaboration, and adaptable to dynamic environments.

[0081] The beneficial effects of this invention are as follows: Compared with existing UAV cooperative path planning schemes, the technology proposed in this invention improves learning efficiency and policy quality by combining non-expert policy guidance and personalized distillation technology. This scheme offers two main advantages: 1. High efficiency: By utilizing non-expert policies generated by an artificial potential field (APF) as initial guidance, the inefficiency of random exploration is significantly reduced, accelerating the convergence of the agent in the early learning stages. 2. Improved sample efficiency: Through the policy distillation mechanism, the agent can efficiently extract knowledge from non-expert policies, reducing dependence on environmental interactions and significantly improving sample utilization. 3. Enhanced exploration capability: The perturbation network provides the agent with exploration capabilities beyond the initial guidance, encouraging the agent to discover better solutions in the policy space and overcoming the limitations of non-expert policies.

[0082] To verify the algorithm's performance, this invention was tested in a constructed simulation environment and compared with several advanced algorithms to demonstrate the performance advantages of the proposed method. In this embodiment, various cooperative planning tasks involving multiple drones are simulated in a three-dimensional environment containing multiple homogeneous drones, several homogeneous obstacles, and a single target. To simplify the experimental setup, these drones, obstacles, and the target are represented as spheres. (Drone radius...) obstacle radius and target radius The values ​​were set to 0.02, 0.12, and 0.13 respectively. All drones shared a common objective and were randomly initialized within a defined environment. The drones moved in three-dimensional space by adjusting forces along three axes. The environment boundary was defined as a 2×2×2 cubic region, with each axis ranging from -1 to 1. Furthermore, the environment was divided into two modes depending on whether the obstacle was stationary or moving. In stationary mode, the obstacle did not move. In moving mode, the proportion of moving obstacles could be dynamically adjusted; the proportions tested in this invention included 0.3, 0.5, 0.7, and 1.0. The speed of the moving obstacle was randomized within the range of [-0.05, 0.05].

[0083] Experiments were conducted to compare the method of this invention with the traditional strategy distillation method (MATD3-BC) and the baseline method (MATD3) to verify the effectiveness of the PPDAC algorithm of this invention. Figure 3 The learning curves for Average Reward (AR) and Success Rate (SR) are presented, obtained by testing the model on 10,000 training segments. The results show that the proposed method consistently maintains higher AR and SR values ​​throughout the training process, outperforming MATD3 and MATD3-BC. Furthermore, PPDAC exhibits faster convergence and superior policy optimization capabilities, maintaining its advantage throughout the training process.

[0084] The performance of the PPDAC algorithm of this invention in multi-UAV path planning tasks was compared and compared with baseline methods (such as MATD3, MATD3-BC, CTFDE, MADDPG, Dec-TD3 and MASAC). Figure 4 The performance of different methods in trajectory planning under 50% obstacle motion conditions is demonstrated. As shown in the figure, the proposed solution performs excellently, with smooth trajectory planning and effective obstacle avoidance, exhibiting superior environmental adaptability. MATD3-BC, CTFDE, and MATD3 generate relatively smooth trajectories, but some unnecessary detours are observed on certain paths, indicating that these methods may face optimization challenges when handling dynamic obstacles, resulting in slightly longer paths. In contrast, the trajectories generated by MADDPG and Dec-TD3 exhibit abrupt changes near obstacles, indicating unstable path planning that may reduce overall efficiency. Finally, MASAC deviates significantly from the optimal path, reflecting its poor adaptability.

[0085] The time efficiency of this invention and the benchmark method was compared in two scenarios. Since collisions during UAV flight can cause the algorithm to terminate prematurely, using the total trajectory inference time as the standard for efficiency evaluation is not feasible. Therefore, single-step inference time (SIT) is used as the key indicator for measuring algorithm efficiency. Experimental results are as follows: Figure 5As shown, the average inference time of the method of this invention is 5.27 milliseconds, only slightly lower than the fastest baseline methods, such as MATD3 (4.75 milliseconds) and MADDPG (4.92 milliseconds), with a difference of less than 1 millisecond. Meanwhile, it significantly outperforms less efficient methods such as MASAC (7.53 milliseconds) and CTFDE (14.03 milliseconds). It is noteworthy that the inference time of CTFDE is almost three times that of the method of this invention, and its error curve fluctuates significantly, indicating poor performance consistency. In contrast, the error curve of the method of this invention is much smaller, reflecting a more stable inference time, which is particularly important in dynamic environments.

[0086] To systematically evaluate the model's performance and stability, this invention considered different numbers of drones and obstacles, and conducted ablation experiments in a constructed simulation scenario. All experiments were conducted with a 0% dynamic obstacle ratio. This invention fixed the number of drones at 5, and varied the number of obstacles, setting them to 15, 20, 25, 30, and 35 respectively. Figure 6 The success rate is demonstrated under different numbers of obstacles. As the number of obstacles increases, the algorithm of this invention maintains a higher success rate (SR) compared to the baseline algorithm. This invention also analyzes the impact of different numbers of drones on the performance of the proposed method. The number of obstacles is fixed at 25, and the number of drones is set to 3, 5, 7, 9, and 11. Figure 7 The performance of the algorithm presented in this invention is demonstrated under different numbers of drones, exhibiting excellent stability and adaptability. Furthermore, to verify the robustness and transferability of the proposed method, this invention combines the MADDPG algorithm with a personalized policy distillation method, and compares and analyzes it with the original MADDPG under different numbers of drones and obstacles. Figure 8 As shown, although the success rate (SR) of both methods decreases with increasing task complexity, the algorithm proposed in this invention consistently outperforms the original MADDPG in all scenarios. Furthermore, the performance gap widens further with increasing complexity. The results demonstrate that the algorithm proposed in this invention significantly enhances the scalability, robustness, and adaptability of MADDPG when handling complex multi-UAV cooperative tasks.

[0087] Based on the above method embodiments, this embodiment also provides a multi-UAV cooperative path planning system, which can be used to implement the steps of the above method.

[0088] refer to Figure 9 The multi-UAV collaborative path planning system includes: The environmental perception module 91 is used to collect observation information, which includes at least one of the following: the position and speed of the UAV, the static / dynamic state of obstacles within the perception range, the coordinates of the target area, and the relative position and motion state of other UAVs in the neighborhood. The non-expert strategy calculation module 92 is used to calculate the resultant force of the UAV under the combined action of gravity, repulsion and inter-agent interaction forces based on the artificial potential field APF model, and generate the initial action as the basic path guidance for the non-expert strategy. The CVAE strategy distillation module 93 is used to input observation information into the CVAE encoder and map it into latent variables. The reconstructed action is obtained by the decoder and serves as a personalized initial strategy to adapt to multi-UAV path planning tasks, realizing the knowledge extraction and flexible transfer of non-expert strategies. The perturbation network exploration module 94 is used to generate perturbation vectors through a perturbation network by taking reconstructed action and observation information as input, and to generate optimized actions based on the perturbation vectors to enhance path exploration capabilities. The multi-agent reinforcement learning collaborative module 95 is used to adopt a centralized training and distributed execution CTDE framework, utilize a centralized criticism network to evaluate the global action value, and combine distillation loss and reinforcement learning loss to optimize the participant's policy. The path execution and iteration module 96 executes actions in a distributed manner based on the optimized strategy, updates its own position and environmental status, until all UAVs complete the collaborative path planning of the target area.

[0089] Various variations and specific examples of the methods provided in the above embodiments are also applicable to the system of this embodiment. Through the foregoing detailed description of the methods, those skilled in the art can clearly understand the implementation method of the system in this embodiment. For the sake of brevity, they will not be described in detail here.

[0090] To better execute the above method, embodiments of this application also provide a UAV computing module, such as... Figure 10 As shown, the UAV computing module includes a processor 101, a memory 102, and a communication bus 103 for enabling communication between the processor 101 and the memory 102.

[0091] The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for at least one function, and instructions for implementing the methods provided in the above embodiments; the data storage area may store data involved in the methods provided in the above embodiments.

[0092] Optionally, the memory may be a read-only memory (ROM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), optical disc (including compact disc read-only memory (CD-ROM), compressed optical disc, laser disc, digital versatile optical disc, Blu-ray disc, etc.), magnetic disk storage medium, or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited to these. The memory exists independently and is connected to the processor via a communication bus, or the memory is integrated with the processor.

[0093] A processor may include one or more processing cores. The processor executes instructions, programs, code sets, or instruction sets stored in memory, and calls data stored in memory to perform various functions and process data as described in this application. The processor may be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), controller, microcontroller, and microprocessor. It is understood that, for different devices, the electronic devices used to implement the above-described processor functions may also be other types, and the embodiments of this application do not specifically limit this.

[0094] The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0095] In an optional embodiment, the UAV computing module may also include a communication interface (not shown) for communication with other devices.

[0096] This application provides a computer-readable storage medium, including, for example, various media capable of storing program code such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. This computer-readable storage medium stores a computer program that can be loaded by a processor and execute the methods of the above embodiments.

[0097] This application also provides a computer program product comprising a computer program tangibly embodied on a readable medium thereof, the computer program containing program code for performing any of the methods described in this application, the computer program being downloadable and installable over a network, and / or installed from a removable medium (such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc.).

[0098] The above description of the embodiments is only used to provide a detailed introduction to the technical solutions of this application. However, the description of the above embodiments is only for the purpose of helping to understand the methods and core ideas of this application, and should not be construed as a limitation of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. A multi-UAV cooperative path planning method, characterized in that, The method includes: S1. Environmental perception and status acquisition: Acquire observation information, which includes at least one of the following: the position and speed of the UAV, the static / dynamic status of obstacles within the perception range, the coordinates of the target area, and the relative position and motion status of other UAVs in the neighborhood; S2. Non-expert strategy guidance: Based on the artificial potential field (APF) model, calculate the resultant force of the UAV under the combined action of gravity, repulsion and inter-agent interaction forces to generate initial actions as the basic path guidance for non-expert strategies. S3. CVAE Strategy Distillation: Construct a Conditional Variational Autoencoder (CVAE), input the observation information from step S1 into the CVAE encoder and map it into latent variables. The reconstructed action is obtained through the decoder and serves as a personalized initial strategy for adapting to multi-UAV path planning tasks. S4. Enhanced Perturbation Network Exploration: Using the reconstruction action in step S3 and the observation information in step S1 as input, a perturbation vector is generated through the perturbation network, and an optimized action is generated based on the perturbation vector to enhance the path exploration capability; The optimization action generated based on the perturbation vector includes generating the optimization action according to the following formula: In the formula, Indicates an optimized action; Indicates a refactoring action; , representing the hyperparameter controlling the amplitude of the disturbance; , represents the perturbation vector. It is the dimension of action space; S5. Multi-agent reinforcement learning collaboration: Adopting a centralized training and distributed execution CTDE framework, utilizing a centralized criticism network to evaluate the global action value, and combining distillation loss and reinforcement learning loss to optimize participant policies; Distillation loss L d Calculate using the following formula: In the formula, E o~D Indicates in observation information The expected value on the corresponding distribution D; KL(||) represents the KL divergence; Indicates the participants' strategies; This indicates the perturbation network output strategy; The overall policy loss combines the reinforcement learning loss and the distillation loss, and is calculated using the following formula: In the formula, L policy Let L represent the overall policy loss; (1-λ) represent the weights controlling the reinforcement learning loss, and λ represent the control distillation loss L. d Weights; E s~D This represents the mathematical expectation of the distribution D corresponding to the environmental state s; This represents a network of critics used to evaluate actions performed in environment state s. a i The higher the value, the better the path. The parameters representing the critic network; π i Indicates the first i A policy network for individual agents; o i Indicates the first i Observational information of each agent; Indicates the first i Individual agent policy network π i Parameters; The weight λ is dynamically adjusted using the following formula: In the formula, The number of times the policy network is updated. This represents the total number of training sets. S6. Path Execution and Iterative Optimization: Each UAV performs actions in a distributed manner based on the optimized participant strategy, updates its own position and environmental state, and iteratively executes steps S1 to S5 until all UAVs complete the collaborative path planning for the target area.

2. The method as described in claim 1, characterized in that, In step S2, the resultant force of the UAV under the combined action of gravity, repulsion, and inter-agent interaction forces is calculated using the following formula: In the formula, Indicates resultant force; Represents gravity; Indicates repulsive force; This represents the interaction forces between intelligent agents.

3. The method as described in claim 2, characterized in that, In step S2, generating the initial action includes: Calculate the direction vector of the unit composite force based on the resultant force: In the formula, It is the direction vector of the unit composite force, and its direction is parallel to... The direction is consistent with that of the system's velocity; Initial motion is generated based on the direction vector of the unit composite force. In the formula, Indicates the maximum speed. Indicates the initial action.

4. A multi-UAV cooperative path planning system, characterized in that, Based on the multi-UAV cooperative path planning method as described in any one of claims 1-3, the system includes: An environmental perception module is used to collect observation information, which includes at least one of the following: the position and speed of the UAV, the static / dynamic state of obstacles within the perception range, the coordinates of the target area, and the relative position and motion state of other UAVs in the neighborhood. The non-expert strategy calculation module is used to calculate the resultant force of the UAV under the combined action of gravity, repulsion and inter-agent interaction forces based on the artificial potential field (APF) model, and generate the initial action as the basic path guidance for the non-expert strategy. The CVAE strategy distillation module is used to input observation information into the CVAE encoder and map it into latent variables. The decoder then reconstructs the reconstructed action, which serves as a personalized initial strategy to adapt to multi-UAV path planning tasks. The perturbation network exploration module is used to generate perturbation vectors through a perturbation network, taking reconstructed actions and observation information as input, and generating optimized actions based on the perturbation vectors to enhance path exploration capabilities. The multi-agent reinforcement learning collaborative module is used to adopt a centralized training and distributed execution CTDE framework, utilize a centralized criticism network to evaluate the global action value, and combine distillation loss and reinforcement learning loss to optimize the participant's policy. The path execution and iteration module executes actions in a distributed manner based on the optimized strategy, updates its own position and environmental status, until all drones complete the collaborative path planning for the target area.

5. A UAV computing module, characterized in that, It includes a processor, a memory, and a communication bus, wherein the communication bus is used to realize the communication connection between the processor and the memory, and the processor is used to execute a computer program stored in the memory to implement the multi-UAV cooperative path planning method as described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program; the computer program can be executed by a processor to implement the multi-UAV cooperative path planning method as described in any one of claims 1-3.

7. A computer program product, characterized in that, Includes a computer program that can be executed by a processor to implement the multi-UAV cooperative path planning method as described in any one of claims 1-3.

Images