Unmanned aerial vehicle cluster safe path planning reinforcement learning method and device

Abstract

This application relates to a reinforcement learning method and apparatus for safe path planning in unmanned aerial vehicle (UAV) swarms. The method includes: inputting UAV swarm state, mission objective, and environmental observation information into a deep reinforcement learning strategy in a continuous action space to generate a global guidance quantity; constructing a set of explicit safety constraints between UAVs and between UAVs and obstacles; using graph structure modeling and a GNN adaptive mechanism to represent UAVs / obstacles with a dynamic graph; predicting CBF parameters using neighborhood and relative velocity information; and performing safety projection correction on the guidance quantity based on constraints and CBF parameters to obtain the actual control command for the corresponding safe path. After command execution, feedback information such as state deviation and constraint tension is collected to adjust the reinforcement learning strategy, iterating cyclically to continuously output an optimized path, achieving collision-free planning throughout the swarm. This method enables stable and reliable safety protection in complex dynamic environments.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) path planning technology, and in particular to a reinforcement learning method and apparatus for safe path planning of UAV swarms. Background Technology

[0002] In applications such as warehouse inspection, emergency search and rescue, low-altitude logistics, and formation performances, drone swarms need to simultaneously complete tasks such as multi-target arrival, formation maintenance, and coordinated passage through narrow areas within a three-dimensional airspace. The core of these tasks lies in generating and executing a set of collision-free, controllable, and landable navigation trajectories in real time, under conditions of multiple drones being coupled together and complex and potentially dynamically changing environmental obstacles, while simultaneously considering global metrics such as completion time, total range, energy consumption, and computational overhead. Existing multi-drone safe path planning technologies can generally be categorized into three types: centralized planning, distributed local coordination, and learning-driven strategies. However, under the "safety-critical + real-time + continuous control" problem setting of drone swarm collision avoidance navigation, significant bottlenecks and engineering implementation obstacles still exist.

[0003] Traditional centralized global planning can theoretically obtain better global paths or scheduling schemes, but its computational complexity increases rapidly with the number of UAVs, task points, and obstacle complexity, making it difficult to meet the real-time requirements of UAV flight control cycles. At the same time, this type of method relies heavily on static or quasi-static modeling assumptions. When sudden dynamic obstacles, communication delays leading to inconsistent states, or deviations at the execution end occur, the global plan generated offline or semi-online often needs to be replanned frequently, further increasing the computational burden and even causing the failure problem of "global plan updates lagging behind environmental changes".

[0004] Distributed / local obstacle avoidance methods have a certain degree of scalability, but their decisions are usually aimed at local feasibility and instantaneous collision avoidance, lacking overall planning for global task efficiency and group coordination. They are prone to oscillations in high-density scenarios, stagnation caused by mutual yielding, or getting stuck in local optima in complex obstacle layouts, resulting in overall range redundancy and increased completion time. Furthermore, they lack stable global guidance for typical conflict situations in three-dimensional space such as "vertical crossing" and "meeting in narrow passages".

[0005] In recent years, learning-based multi-UAV navigation methods have shown potential in adapting to dynamic environments. However, existing solutions often suffer from two prominent problems: First, end-to-end strategies, when lacking explicit safety constraints, struggle to provide verifiable constraints on collision risks, leading to uncontrollable risks in safety-critical scenarios. Second, some methods still rely on manually set safety constraints or control obstacle function parameters. When faced with changes in the number of UAVs, differences in obstacle configurations, sensor noise, and execution disturbances, these parameters are difficult to adapt, resulting in a significant decline in success rate and safety when exceeding the training distribution or scaling up.

[0006] Furthermore, existing engineering implementations generally suffer from a systemic defect of "hierarchical fragmentation": global planning, trajectory / waypoint tracking, and local obstacle avoidance are usually pieced together from different algorithm modules, lacking a unified closed-loop coordination mechanism and consistent optimization objectives, resulting in mutual constraints between upper-level planning and lower-level execution. While relying solely on global planning can provide a feasible path in a static sense, it is difficult to quickly correct execution deviations and dynamic risks. While relying solely on local obstacle avoidance can avoid collisions in the short term, it lacks stable guidance for global arrival and group efficiency, manifesting as severe detours, decreased goal achievement rate, and efficiency losses caused by repeated avoidance in dense interactions. Summary of the Invention

[0007] Therefore, it is necessary to provide a reinforcement learning method and device for safe path planning of UAV swarms that can significantly reduce collision risks and achieve stable and reliable safety protection in complex dynamic environments, in order to address the above-mentioned technical problems.

[0008] A reinforcement learning method for safe path planning in unmanned aerial vehicle (UAV) swarms, the method comprising:

[0009] Step 1: Collect the group status, mission target information and environmental observation information of the drone swarm. The group status includes the motion status of each drone and the relative status of neighboring drones within the perception radius. The environmental observation information includes the relative status of static and dynamic obstacles.

[0010] Step 2: Input the group state, mission target information and environmental observation information of the UAV swarm into a deep reinforcement learning strategy based on continuous action space, and generate global path guidance information for mission efficiency online. The global path guidance information is at least one of the following: short field global path point, reference velocity or acceleration sequence as guidance quantity.

[0011] Step 3: Construct a set of explicit safety constraints between UAVs and between UAVs and obstacles based on environmental observation information and interaction information in the group state;

[0012] Step 4: Introduce a graph structure interactive modeling and graph neural network-driven parameter adaptation mechanism to construct a dynamic graph of the UAV and obstacles, and use neighborhood topology, relative position and relative velocity information to predict the relevant parameters of the control obstacle function;

[0013] Step 5: Based on the explicit safety constraint set, the guided quantity is corrected by safety projection using the predicted control obstacle function to obtain the actual control command; the actual control command corresponds to the safe path planning result of the UAV swarm.

[0014] Step 6: Send the actual control commands to the UAV for execution, and at the same time collect information on the state deviation, safety constraint tension and correction magnitude of the UAV during execution and feed it back to the deep reinforcement learning strategy based on continuous action space;

[0015] Step 7: Adaptively adjust the deep reinforcement learning strategy based on the continuous action space according to the feedback information, repeat steps 1 to 6, and continuously output the optimized safe path planning results to achieve collision-free safe path planning for the entire drone swarm.

[0016] A reinforcement learning device for safe path planning in unmanned aerial vehicle (UAV) swarms, the device comprising:

[0017] The cluster and environmental information acquisition module is used to collect the group status, mission target information and environmental observation information of the UAV cluster. The group status includes the motion status of each UAV and the relative status of neighboring UAVs within the perception radius. The environmental observation information includes the relative status of static and dynamic obstacles.

[0018] The deep reinforcement learning global path guidance module is used to input the group state, task target information and environmental observation information of the UAV swarm into a deep reinforcement learning strategy based on continuous action space, and generate global path guidance information oriented towards task efficiency online. The global path guidance information is at least one of short field-of-view global path points, reference velocity or acceleration sequence as guidance quantity.

[0019] The safety constraint set construction module is used to construct explicit safety constraint sets between UAVs and between UAVs and obstacles based on environmental observation information and interaction information in the group state;

[0020] The graph neural network parameter prediction module is used to introduce graph structure interactive modeling and graph neural network driven parameter adaptation mechanism to construct a dynamic graph of UAV and obstacles, and use neighborhood topology, relative position and relative velocity information to predict the relevant parameters of the control obstacle function.

[0021] The guidance quantity safety projection correction module is used to perform safety projection correction on the guidance quantity based on the explicit safety constraint set and using a predicted control obstacle function to obtain the actual control command; the actual control command corresponds to the safe path planning result of the UAV cluster.

[0022] The execution and feedback information acquisition module is used to send the actual control commands to the UAV for execution, and at the same time collect the state deviation, safety constraint tension and correction magnitude information of the UAV during the execution process and feed it back to the deep reinforcement learning strategy based on continuous action space.

[0023] The reinforcement learning strategy adaptive optimization module is used to adaptively adjust the deep reinforcement learning strategy based on the continuous action space according to the feedback information, and continuously output the optimized safe path planning results to achieve collision-free safe path planning for the entire drone swarm.

[0024] The aforementioned reinforcement learning method and apparatus for safe path planning in UAV swarms employs a two-layer architecture: a high-level dual-delay deep deterministic policy gradient algorithm for global guidance generation and a low-level CBF (Critical Flow Factor) layer for local safety correction. This architecture enables information interaction and closed-loop instruction correction between global planning and local obstacle avoidance, overcoming the efficiency loss caused by the fragmented operation of modules in traditional technologies. The high-level dual-delay deep deterministic policy gradient algorithm takes swarm state, target information, neighborhood interactions, and obstacle observations as inputs, outputting executable global guidance quantities. By maximizing cumulative rewards, it learns efficient global planning behavior, providing rolling global path guidance for the swarm. The low-level CBF safety filter layer takes the output of the high-level layer as the nominal intent and performs minimum deviation projection correction on the nominal control when safety constraints are activated, ensuring collision-free operation in dynamic environments and complex interactions. Furthermore, the closed-loop coordination of "policy generation—safety projection—state feedback" ensures a smooth and efficient global path. This approach enables high-level strategies to generate fewer strongly corrective guidance outputs, reducing redundant actions and significantly improving cluster task completion efficiency while ensuring safety. The architecture balances global guidance quality with online real-time performance. High-level planning is completed through neural network forward inference, while the underlying CBF safety filtering can run in a distributed parallel manner, avoiding the high computational overhead of traditional centralized planning. It maintains stable performance even in scenarios with increased drone numbers and more complex obstacles, demonstrating good engineering scalability and deployment feasibility. Furthermore, the GNN-CBF module in the solution autonomously outputs adaptive CBF parameters by learning the interaction patterns between drones and the environment, overcoming the limitations of traditional manually preset CBF parameters. It maintains a high planning success rate even with drone scales and environmental configurations exceeding the training range, and explicit safety constraints significantly reduce collision risks, achieving stable and reliable safety protection in complex dynamic environments. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating a reinforcement learning method for safe path planning in a drone swarm, as shown in one embodiment.

[0026] Figure 2 This is a flowchart of a safe path planning method for a drone swarm in one embodiment. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0028] In one embodiment, such as Figure 1 As shown, a reinforcement learning method for safe path planning in UAV swarms is provided, including the following steps:

[0029] Step 1: Collect the group status, mission target information and environmental observation information of the UAV swarm. The group status includes the motion status of each UAV and the relative status of neighboring UAVs within the perception radius. The environmental observation information includes the relative status of static and dynamic obstacles.

[0030] The drone's own motion status includes position, speed, heading / attitude, etc.; the relative status of neighboring drones includes relative position, relative speed, etc.; the relative status of obstacles includes the position and size of static obstacles, and the position, speed, and trajectory of dynamic obstacles, etc. The above information is collected by state perception modules such as the nose vision sensor and the fuselage sensor (radar / camera) to provide a data foundation for subsequent path planning.

[0031] Step 2: Input the group state, mission target information and environmental observation information of the UAV swarm into a deep reinforcement learning strategy based on continuous action space, and generate global path guidance information for mission efficiency online. The global path guidance information is at least one of short-field global path points, reference velocity or acceleration sequence as guidance quantity.

[0032] To address swarm task efficiency metrics (such as minimizing maximum completion time, reducing total range, and minimizing congestion and energy consumption), the navigation process of UAV swarms in a 3D environment is modeled as a Markov decision process: the high-level policy network takes the swarm state, target information, and local observations as inputs and outputs the global guidance quantity for each UAV. The guidance quantity includes at least one of the following: reference velocity vector / desired acceleration (continuous action); reference heading angle and velocity amplitude; short-term trajectory segment parameters or sparse sub-target (waypoint) sequence within the planned field of view.

[0033] A dual-delay deep deterministic policy gradient algorithm is adopted to learn the global guidance policy. By using a dual-Q network and delayed policy updates, overestimation is suppressed and the stability of continuous control is improved. This enables the policy to quickly generate executable global guidance under dynamic obstacles and strong coupling interaction among multiple drones. At the same time, the high-level policy no longer relies on global replanning for centralized constraint solving, thereby reducing the computational burden caused by the growth of UAV scale and improving real-time response capability and scalability.

[0034] Step 3: Construct a set of explicit safety constraints between UAVs and between UAVs and obstacles based on environmental observation information and interaction information in the group state.

[0035] An explicit set of safety constraints is constructed based on the safety distance requirements between drones and between drones and obstacles. These explicit constraints ensure that drones do not collide during flight, providing a basis for subsequent safety corrections.

[0036] Step 4: Introduce graph structure interactive modeling and graph neural network driven parameter adaptation mechanism to construct a dynamic graph of UAV and obstacles, and use neighborhood topology, relative position and relative velocity information to predict the relevant parameters of the control obstacle function.

[0037] The parameter adaptation mechanism driven by graph neural networks (GNN) can overcome the limitations of traditional control obstacle function parameters that rely on manual presets. By capturing the interaction between the UAV and the environment through dynamic graph modeling, it can achieve online adaptive adjustment of CBF parameters. It can adjust the "strength" of safety constraints online according to environmental changes, and improve the stability and generalization ability under different densities, obstacle configurations and cluster sizes.

[0038] Step 5: Based on the explicit safety constraint set, the guiding quantity is corrected by safety projection using the predicted control obstacle function to obtain the actual control command; the actual control command corresponds to the safe path planning result of the UAV swarm.

[0039] The system integrates environmental perception information and group interaction information in real time to construct safe distance constraints between UAVs and between UAVs and obstacles, along with their corresponding control obstacle functions. Within each control cycle, the nominal guidance / control output of the higher-level strategy is corrected online. This correction follows the principle of "maintaining the original control intent as much as possible while satisfying safety constraints": that is, using the higher-level output as the target, it solves a minimum deviation optimization problem (or equivalent projection problem) with CBF constraints to obtain actual control commands that satisfy the safety constraints, which are then issued and executed. This addresses dynamic obstacles, dense interactions, and execution deviations, ensuring collision-free operation throughout the entire process.

[0040] Step 6: Send the actual control command to the UAV for execution, and at the same time collect information on the state deviation, safety constraint tension and correction magnitude of the UAV during execution and feed it back to the deep reinforcement learning strategy based on continuous action space.

[0041] State deviation includes the deviation between the actual motion state of the UAV and the global guidance quantity; safety constraint tension refers to the relevant parameters of the activation degree of safety constraints; correction magnitude refers to the adjustment magnitude of the local obstacle avoidance correction layer on the nominal guidance quantity. The above feedback information is collected through the interaction between the communication module and the flight control unit to provide a basis for the adaptive adjustment of the high-level strategy.

[0042] Step 7: Adaptively adjust the deep reinforcement learning strategy based on the continuous action space according to the feedback information, repeat steps 1 to 6, and continuously output the optimized safe path planning results to achieve collision-free safe path planning for the entire drone swarm.

[0043] The high-level reinforcement learning strategy receives feedback information, perceives the tension of underlying security constraints, and adaptively adjusts the decision direction to make the subsequently generated global guidance quantity more in line with actual operational needs, reducing unnecessary deceleration, repeated avoidance and oscillation detours, and forming a closed-loop collaboration of "global guidance - execution tracking - security projection - state feedback".

[0044] In the aforementioned reinforcement learning method for safe path planning of UAV swarms, this application utilizes a two-layer architecture of "high-level TD3 reinforcement learning global guidance generation + low-level CBF local safety correction" to achieve information interaction and closed-loop instruction correction between global planning and local obstacle avoidance. This breaks down the efficiency losses caused by the fragmented and independent operation of "global planning, trajectory tracking, and local obstacle avoidance" in traditional technologies at the system level. On one hand, the high-level TD3-based policy network takes swarm state, target information, neighborhood interaction, and obstacle observation as inputs and directly outputs executable global guidance quantities. By maximizing cumulative rewards, the policy automatically learns global planning behaviors that are "faster to the target, less detour, and smoother interaction," thus providing UAV swarms with rolling global path guidance oriented towards task efficiency. On the other hand, the low-level CBF safety filtering layer uses the output of the high-level layer as the nominal intent. When UAV-UAV and UAV-obstacle safety constraints are activated, it performs minimum deviation projection correction on the nominal control, ensuring collision-free operation even under dynamic obstacles, dense interactions, and execution disturbances. The closed-loop collaboration of "strategy generation - security projection - state feedback" makes the high-level strategy tend to generate fewer guidance outputs that trigger strong corrections in long-term operation, thereby reducing unnecessary deceleration, repeated avoidance and oscillating detours, and ultimately significantly improving the cluster's completion time, passage efficiency and overall task completion rate while ensuring safety.

[0045] Meanwhile, compared to traditional global planning schemes that rely on centralized constraint solving, this method completes high-level planning through forward inference of the TD3 policy. Online computation mainly consists of forward computation of the neural network and a small amount of post-processing, avoiding the global optimization overhead that increases sharply with the number of drones and environmental complexity. At the same time, the low-level CBF safety filtering is corrected online mainly through local neighborhood constraints, which can run in parallel under a distributed execution framework. Through this layered decomposition of "learning-driven global guided generation + explicit constraint local filtering", this scheme retains the ability to coordinate task efficiency at the global level while possessing real-time response characteristics close to local methods. It can maintain stable planning and execution performance in scenarios with increased drone numbers, more intensive interactions, and more complex obstacles, demonstrating good engineering scalability and deployment feasibility.

[0046] Furthermore, the GNN-CBF module in this method learns the interaction patterns between the UAV and the environment, and can autonomously output adaptive CBF parameters, overcoming the limitations of manually preset parameters in traditional CBF methods. Even in scenarios where the number of UAVs exceeds the training range, the planning success rate of this method remains extremely high, showing only a slight decline, demonstrating excellent cross-scale generalization ability; while traditional CBF schemes experience significant performance degradation in this scenario due to parameter incompatibility. Simultaneously, the explicit safety constraints of the GNN-CBF module significantly reduce collision risk compared to end-to-end pure learning strategies, achieving stable and reliable safety protection in complex dynamic environments.

[0047] In one embodiment, the group state, mission objective information, and environmental observation information of the UAV swarm are input into a deep reinforcement learning strategy based on a continuous action space to generate global path guidance information oriented towards mission efficiency online, including:

[0048] The group state, mission objective information, and environmental observation information of the UAV swarm are input into a deep reinforcement learning strategy based on a continuous action space. The global path planning problem of the UAV swarm is modeled as a Markov decision process of continuous state-continuous action. A dual-delay deep deterministic policy gradient algorithm is used to learn the global guidance strategy, and executable global path guidance information is generated online under dynamic environment and multi-machine interaction conditions.

[0049] Specifically, modeling the navigation process as a Markov decision process can fully consider the temporal correlation and uncertainty of the state. The dual-delay deep deterministic policy gradient algorithm effectively suppresses the overestimation problem through the dual-Q network and the delayed policy update mechanism, improves the stability and reliability of continuous control, and ensures that the generated global path guidance information meets both the task efficiency requirements and has good executability.

[0050] In one embodiment, the global path planning problem of a drone swarm is modeled as a Markov decision process of continuous state-continuous action, including:

[0051] Define the output format of states, actions, and policies:

[0052] At any moment , for the The observation of the drone is defined as

[0053] ;

[0054] in, For its own motion state, For the mission objective, The set of relative states of neighboring drones within the sensing radius. This is the set of relative states of the perceived obstacles; for ease of network input, A fixed number of nearest neighbor truncation, rasterization, or graph structure encoding can be used;

[0055] Action (Global Guiding Quantity) Definition:

[0056] The dual-delay deep deterministic policy gradient algorithm is oriented towards a continuous action space, allowing the output of the higher-level policy to be...

[0057] ;

[0058] Under the semantics of global path planning Instead of directly representing the low-level input of the motor / attitude, it represents global path guidance, preferably any one (or a combination thereof):

[0059] The global path point for short field of view is:

[0060] ;

[0061] in, for t The next global sub-target within the limited field of vision at any given time. For the first i A drone in t The current position at any given moment. This is a correction amount;

[0062] Reference velocity / acceleration guidance is as follows: or ,in, For the first i A drone in t Reference speed guidance amount at any given moment. For the first i A drone in t The nominal acceleration guide quantity at any given moment;

[0063] When the action quantity adopts the form of short-field global path points, the global path is obtained by rolling stitching: sub-objectives are generated in each decision cycle. And after executing several steps, it is updated to form a sequence, that is, the global path guidance information is:

[0064] ;

[0065] When the motion quantity is expressed as a reference velocity / acceleration, the global path is:

[0066] ;

[0067] in, Indicates the first i A drone in t The actual execution acceleration at any given moment Indicates the current speed. Indicates the time step.

[0068] In one embodiment, the global path planning problem of a drone swarm is modeled as a Markov decision process of continuous state-continuous action, with the reward set as follows:

[0069] ;

[0070] in, and For risk penalty function, Indicates the drone has reached the The distance to each obstacle, Indicates the conditions for reaching the target. This represents the penalty coefficient for approaching the target. The penalty coefficient represents the range of motion. This represents the penalty coefficient for conflicts between drones. This represents the penalty coefficient for drone-obstacle collisions. This represents the reward coefficient for achieving the goal. Represented as the first j A drone in t The current position at any given moment.

[0071] Specifically, this reward function enables the strategy to learn a comprehensive planning preference during the training phase, namely "global approach to the target, reducing energy consumption / control amplitude, and avoiding potential conflicts." The target approach penalty encourages the drone to quickly approach the target, the action amplitude penalty suppresses excessive control actions to reduce energy consumption, the conflict penalty avoids the risk of collisions between drones and between drones and obstacles, and the target achievement reward incentivizes the drone to complete the task. It should be noted that the true hard safety is guaranteed by the underlying CBF. The risk term in this reward function is mainly to enable the high-level strategy to learn the planning habit of "less triggering of strong CBF corrections," thereby improving efficiency and stability.

[0072] In one embodiment, a dual-delay deep deterministic policy gradient algorithm is used to learn the global guidance policy, including:

[0073] Use two Function approximator Configure the corresponding network. By replaying samples based on experience Construct target actions; employ dual Calculate the target by taking the minimum. The update methods for the comment network, action network, and target network are set. The output of the action network is used as the global guide. The output of the action network in each cycle is calculated according to the rolling decision method. The global path guidance information is obtained by dynamic integration or sub-target concatenation.

[0074] Specifically, two Q-function approximators are used. and using the target network For samples in experience playback Construct target actions with a smooth target strategy:

[0075] ;

[0076] The target Q-value employs a "minimum of both Qs" approach to suppress overestimation.

[0077] ;

[0078] in This is the discount factor.

[0079] By storing historical interaction data through an experience replay mechanism and randomly sampling samples for training, the correlation between samples can be broken, thus improving training stability. The method of minimizing the double Q function effectively suppresses the problem of Q-value overestimation, making the value assessment more accurate. The addition of truncated random noise to the target action enhances the policy's exploration ability and robustness.

[0080] In one embodiment, the update methods for the comment network, action network, and target network are configured, including:

[0081] The update method for the comment network is set as follows:

[0082] ;

[0083] in, For the first i A drone in t Observations of time, For the first i The drone in t The output of each decision cycle For the first i The drone in t The target Q value for each decision cycle;

[0084] The dual-delay deep deterministic policy gradient algorithm does not update the action network at every step, but rather every [time / period]. Update once per cycle (delayed update), and maximize Q using a deterministic policy gradient form:

[0085] ;

[0086] Configure the action network update method as follows:

[0087] ;

[0088] in, For parameters Action network;

[0089] The update method for the target network is set as follows:

[0090] ;

[0091] in, This is the soft update coefficient. Indicates the first Online network parameters of a comment network, Indicates the first k The target network parameters for the comment network.

[0092] In one embodiment, the output of the action network for each cycle is calculated according to the rolling decision method, including:

[0093] Action network output This represents global guidance (sub-objective / reference velocity / nominal control), whose optimization objective is determined by cumulative reward.

[0094] ;

[0095] The decision is equivalent to learning a global planning strategy within the training distribution that "reaches the goal faster, the path is shorter, and the interaction is more stable";

[0096] The output of the action network for each cycle is calculated based on the rolling decision-making method.

[0097] ;

[0098] in, For the first i The drone in t Observations for each decision-making cycle For parameters Action network.

[0099] The global guidance sequence is obtained through dynamic integration or sub-objective splicing. The "global boot sequence" is then handed over to the underlying CBF for hard constraint safety projection to form the final executable trajectory.

[0100] Specifically, the rolling decision-making approach does not require generating a complete global path all at once. Instead, it generates the current local guidance quantity in each decision cycle, and forms the global path by piecing together segments or through dynamic integration. This allows for rapid adaptation to dynamic environmental changes and execution deviations, improving the real-time performance and flexibility of path planning. Figure 2 The flowchart shows that the output will be sent as a global command to the local obstacle avoidance correction layer for safety correction.

[0101] In one embodiment, the explicit safety constraint set includes inter-UAV constraints and UAV-obstacle constraints; the inter-UAV constraints are... ,in, For the safe radius of the drone, for t Time of the first i The location of the drone for t Time of the first k The location of the drone; the constraint between the drone and the obstacle is ,in, For the first j The center position of each obstacle For the first j The radius of the obstacle.

[0102] Specifically, the above constraint definitions clarify the safe distance requirements between drones and between drones and obstacles, providing a foundation for the construction of the CBF function and ensuring that drones maintain a safe distance from other drones and obstacles during flight, thus avoiding collisions.

[0103] To quantify the safe state, two types of barrier functions are constructed:

[0104] Inter-drone obstacle function: ;

[0105] Drone - Obstacle Function: ;

[0106] Safe operation must meet the following requirements ( α For a function of class K, take ).

[0107] In one embodiment, a parameter adaptation mechanism driven by graph structure interaction modeling and graph neural networks is introduced to construct a dynamic graph of the drone and obstacles. The relevant parameters of the obstacle control function are predicted using neighborhood topology, relative position, and relative velocity information, including:

[0108] Modeling the drone swarm system as a dynamic graph The node features are those of drone nodes. , For the first i The location of the drone For the first i The movement information of the drone For the target location, For node types; edge features are constructed based on perceptual radius, if Then, edges are generated, and the characteristics of the generated edges are: , For relative velocity, For the first i The location of the drone For the first k The location of the drone;

[0109] The dynamic graph is input into a GNN for message generation and feature updating through an information aggregation process, outputting a high-dimensional embedding of the UAV node, including:

[0110] Message generation: ( It is a multilayer perceptron (MLP). (the set of neighbors of a node).

[0111] Feature update: ( (for MLP).

[0112] After aggregation, a high-dimensional embedding of the drone node is output, which can characterize the interaction state between the surrounding drones and obstacles;

[0113] Based on the high-dimensional embedding, the CBF-related parameters are output through the GNN.

[0114] ;

[0115] in, To control the amount of correction, For adaptive K-type function parameters, To track target weights, This represents the control barrier function for the output.

[0116] Specifically, GNN can effectively capture the complex interaction between UAVs and the environment through dynamic graph modeling, extract high-dimensional features through information aggregation, and then accurately predict CBF-related parameters, achieving adaptive adjustment of safety constraint parameters. This breaks through the limitations of manually preset parameters in traditional CBF methods and improves the adaptability of the method in different scenarios.

[0117] In one embodiment, the loss function of the GNN is set as follows:

[0118] ;

[0119] ;

[0120] ;

[0121] ;

[0122] in, Indicates safety loss, Indicates tracking loss. Indicates appropriate loss. t Indicates the time step index; or Indexes representing drones or obstacles, For the safe radius of the drone, and For the target location, for t Time of the first i The location of the drone for t Time of the first k The location of the drone Indicates the first A drone in The actual execution acceleration at any given moment.

[0123] It should be understood that, although Figure 1The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0124] In one embodiment, a reinforcement learning device for safe path planning in a drone swarm includes:

[0125] The cluster and environmental information acquisition module is used to collect the group status, mission target information and environmental observation information of the UAV cluster. The group status includes the motion status of each UAV and the relative status of neighboring UAVs within the perception radius. The environmental observation information includes the relative status of static and dynamic obstacles.

[0126] The deep reinforcement learning global path guidance module is used to input the group state, task target information and environmental observation information of the UAV swarm into a deep reinforcement learning strategy based on continuous action space, and generate global path guidance information oriented towards task efficiency online. The global path guidance information is at least one of short field-of-view global path points, reference velocity or acceleration sequence as guidance quantity.

[0127] The safety constraint set construction module is used to construct explicit safety constraint sets between UAVs and between UAVs and obstacles based on environmental observation information and interaction information in the group state;

[0128] The graph neural network parameter prediction module is used to introduce graph structure interactive modeling and graph neural network driven parameter adaptation mechanism to construct a dynamic graph of UAV and obstacles, and use neighborhood topology, relative position and relative velocity information to predict the relevant parameters of the control obstacle function.

[0129] The guidance quantity safety projection correction module is used to perform safety projection correction on the guidance quantity based on the explicit safety constraint set and using a predicted control obstacle function to obtain the actual control command; the actual control command corresponds to the safe path planning result of the UAV cluster.

[0130] The execution and feedback information acquisition module is used to send the actual control commands to the UAV for execution, and at the same time collect the state deviation, safety constraint tension and correction magnitude information of the UAV during the execution process and feed it back to the deep reinforcement learning strategy based on continuous action space.

[0131] The reinforcement learning strategy adaptive optimization module is used to adaptively adjust the deep reinforcement learning strategy based on the continuous action space according to the feedback information, and continuously output the optimized safe path planning results to achieve collision-free safe path planning for the entire drone swarm.

[0132] Specific limitations regarding the reinforcement learning device for safe path planning in UAV swarms can be found in the limitations of the reinforcement learning method for safe path planning in UAV swarms described above, and will not be repeated here. Each module in the aforementioned reinforcement learning device for safe path planning in UAV swarms can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0133] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0134] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A reinforcement learning method for safe path planning in unmanned aerial vehicle (UAV) swarms, characterized in that, The method includes: Step 1: Collect the group status, mission target information and environmental observation information of the UAV swarm. The group status includes the motion status of each UAV and the relative status of neighboring UAVs within the perception radius. The environmental observation information includes the relative status of static and dynamic obstacles. Step 2: Input the group status, mission target information and environmental observation information of the UAV swarm into a deep reinforcement learning strategy based on continuous action space, and generate global path guidance information for mission efficiency online. The global path guidance information is at least one of short-field global path points, reference velocity or acceleration sequence as guidance quantity. Step 3: Construct a set of explicit safety constraints between UAVs and between UAVs and obstacles based on environmental observation information and interaction information in the group state; Step 4: Introduce graph structure interaction modeling and graph neural network-driven parameter adaptation mechanisms to construct a dynamic graph of the UAV and obstacles. Utilize neighborhood topology, relative position, and relative velocity information to predict relevant parameters of the obstacle control function, including: Modeling the drone swarm system as a dynamic graph The node features are those of drone nodes. , For the first i The location of the drone For the first i The movement information of the drone For the target location, For node types; edge features are constructed based on perceptual radius, if Then, edges are generated, and the characteristics of the generated edges are: , For relative velocity, For the first i The location of the drone For the first k The location of the drone; The dynamic graph is input into a GNN for message generation and feature updating through an information aggregation process, and a high-dimensional embedding of the UAV node is output; the high-dimensional embedding represents the interaction state between the surrounding UAV and obstacles. Based on the high-dimensional embedding, the CBF-related parameters are output through the GNN as follows: in, To control the amount of correction, For adaptive K-class function parameters, To track target weights, This represents the control barrier function for the output; Step 5: Based on the explicit safety constraint set, the guiding quantity is corrected by safety projection using the predicted control obstacle function to obtain the actual control command; the actual control command corresponds to the safe path planning result of the UAV swarm. Step 6: Send the actual control command to the UAV for execution, and at the same time collect information on the state deviation, safety constraint tension and correction magnitude of the UAV during execution and feed it back to the deep reinforcement learning strategy based on continuous action space; Step 7: Adaptively adjust the deep reinforcement learning strategy based on the continuous action space according to the feedback information, repeat steps 1 to 6, and continuously output the optimized safe path planning results to achieve collision-free safe path planning for the entire drone swarm.

2. The method according to claim 1, characterized in that, The group state, mission objective information, and environmental observation information of the UAV swarm are input into a deep reinforcement learning strategy based on a continuous action space to generate global path guidance information online, oriented towards mission efficiency, including: The group state, mission objective information, and environmental observation information of the UAV swarm are input into a deep reinforcement learning strategy based on a continuous action space. The global path planning problem of the UAV swarm is modeled as a Markov decision process of continuous state-continuous action. A dual-delay deep deterministic policy gradient algorithm is used to learn the global guidance strategy, and executable global path guidance information is generated online under dynamic environment and multi-machine interaction conditions.

3. The method according to claim 2, characterized in that, The global path planning problem of drone swarms is modeled as a Markov decision process of continuous state-continuous action, including: At any moment , for the The observation of the drone is defined as in For its own motion state, For the mission objective, The set of relative states of neighboring drones within the sensing radius. The set of relative states of the perceived obstacles; Define time The global path point for short field of view is: in, for t The next global sub-target within the limited field of vision at any given time. For the first i A drone in t The current position at any given moment. This is a correction amount; Reference velocity / acceleration guidance is as follows: ,in, For the first i A drone in t Reference speed guidance amount at any given moment. For the first i A drone in t The nominal acceleration guide quantity at any given moment; When the motion quantity is expressed using a short-view global pathpoint, the global path guidance information is as follows: ； When the motion quantity is expressed as a reference velocity / acceleration, the global path is: in, Indicates the first i A drone in t The actual execution acceleration at any given moment Indicates the current speed. Indicates the time step.

4. The method according to claim 3, characterized in that, The method further includes: The global path planning problem of drone swarms is modeled as a Markov decision process of continuous state-continuous action, with the reward set as... in, and For risk penalty function, Indicates the drone has reached the The distance to each obstacle, Indicates the conditions for reaching the target. This represents the penalty coefficient for approaching the target. The penalty coefficient represents the range of motion. This represents the penalty coefficient for conflicts between drones. This represents the penalty coefficient for drone-obstacle collisions. This represents the reward coefficient for achieving the goal. Represented as the first j A drone in t The current position at any given moment.

5. The method according to claim 2, characterized in that, A dual-delay deep deterministic policy gradient algorithm is used to learn the global guidance policy, including: Use two Function approximator Configure the corresponding network. By replaying samples based on experience Construct target actions; employ dual Calculate the target by taking the minimum. The update methods for the comment network, action network, and target network are set. The output of the action network is used as the global guide. The output of the action network in each cycle is calculated according to the rolling decision method. The global path guidance information is obtained by dynamic integration or sub-target concatenation.

6. The method according to claim 5, characterized in that, Configure the update methods for the comment network, action network, and target network, including: The update method for the comment network is set as follows: in, For the first i A drone in t Observations of time, For the first i The drone in t The output of each decision cycle For the first i The drone in t The target Q value for each decision cycle; Configure the action network update method as follows: in, For parameters Action network; The update method for the target network is set as follows: in, This is the soft update coefficient. Indicates the first Online network parameters of a comment network, Indicates the first The target network parameters for the comment network.

7. The method according to claim 5, characterized in that, The output of the action network for each cycle is calculated based on the rolling decision-making method, including: The output of the action network for each cycle is calculated based on the rolling decision-making method. in, For the first i The drone in t Observations for each decision-making cycle For parameters Action network.

8. The method according to claim 1, characterized in that, The explicit safety constraint set includes inter-UAV constraints and UAV-obstacle constraints; the inter-UAV constraints are... ,in, For the safe radius of the drone, for t Time of the first i The location of the drone for t Time of the first k The location of the drone; the constraint between the drone and the obstacle is ,in, For the first j The center position of each obstacle For the first j The radius of the obstacle.

9. The method according to claim 1, characterized in that, The loss function of the GNN is set as follows: in, Indicates safety loss, Indicates tracking loss. Indicates appropriate loss. t Indicates the time step index; or Indexes representing drones or obstacles, For the safe radius of the drone, and For the target location, for t Time of the first i The location of the drone for t Time of the first k The location of the drone Indicates the first A drone in The actual execution acceleration at any given moment.

10. A reinforcement learning device for safe path planning in unmanned aerial vehicle (UAV) swarms, characterized in that, The device includes: The cluster and environmental information acquisition module is used to collect the group status, mission target information and environmental observation information of the UAV cluster. The group status includes the motion status of each UAV and the relative status of neighboring UAVs within the perception radius. The environmental observation information includes the relative status of static and dynamic obstacles. The deep reinforcement learning global path guidance module is used to input the group state, task target information and environmental observation information of the UAV swarm into a deep reinforcement learning strategy based on continuous action space, and generate global path guidance information oriented towards task efficiency online. The global path guidance information is at least one of short field-of-view global path points, reference velocity or acceleration sequence as guidance quantity. The safety constraint set construction module is used to construct explicit safety constraint sets between UAVs and between UAVs and obstacles based on environmental observation information and interaction information in the group state; The graph neural network parameter prediction module introduces graph structure interaction modeling and graph neural network-driven parameter adaptation mechanisms. It constructs a dynamic graph of the UAV and obstacles, utilizing neighborhood topology, relative position, and relative velocity information to predict relevant parameters of the obstacle control function, including: Modeling the drone swarm system as a dynamic graph The node features are those of drone nodes. , For the first i The location of the drone For the first i The movement information of the drone For the target location, For node types; edge features are constructed based on perceptual radius, if Then, edges are generated, and the characteristics of the generated edges are: , For relative velocity, For the first i The location of the drone For the first k The location of the drone; The dynamic graph is input into a GNN for message generation and feature updating through an information aggregation process, and a high-dimensional embedding of the UAV node is output; the high-dimensional embedding represents the interaction state between the surrounding UAV and obstacles. Based on the high-dimensional embedding, the CBF-related parameters are output through the GNN as follows: in, To control the amount of correction, For adaptive K-class function parameters, To track target weights, This represents the control barrier function for the output; The guidance quantity safety projection correction module is used to perform safety projection correction on the guidance quantity based on the explicit safety constraint set and using a predicted control obstacle function to obtain the actual control command; the actual control command corresponds to the safe path planning result of the UAV cluster. The execution and feedback information acquisition module is used to send the actual control commands to the UAV for execution, and at the same time collect the state deviation, safety constraint tension and correction magnitude information of the UAV during the execution process and feed it back to the deep reinforcement learning strategy based on continuous action space. The reinforcement learning strategy adaptive optimization module is used to adaptively adjust the deep reinforcement learning strategy based on the continuous action space according to the feedback information, and continuously output the optimized safe path planning results to achieve collision-free safe path planning for the entire drone swarm.

Images