A priori guided timing fusion multi-unmanned aerial vehicle cooperative obstacle avoidance route planning method

Abstract

The application discloses a priori guided time sequence fusion multi-unmanned aerial vehicle cooperative obstacle avoidance route planning method, which comprises the following steps: 1) multi-unmanned aerial vehicle route planning; 2) POMDP model establishment; 3) priori guided time sequence fusion value decomposition algorithm PGL-QMIX; and 4) Q value structure optimization under state-action time sequence fusion. The application uses A* offline generated reference path as weak priori, and fuses local observation and priori information through a double-layer time sequence network, models local dynamics and group cooperation, and thus improves stability and cooperation of joint value estimation. The application constructs a local priori segment construction and time sequence fusion mechanism, which uses the reference path generated by the A* algorithm in the static environment, extracts only the locally visible path segment as heuristic priori input in the execution process, and thus obtains global geometric trend information without relying on complete global information. Simulation results show that the application can improve the feasibility and cooperation of multi-unmanned aerial vehicle route planning.

Description

Technical Field

[0008] This invention relates to a priori-guided, time-series fusion-based multi-UAV cooperative obstacle avoidance route planning method, belonging to the field of computer communication technology. Background Technology

[0009] With the rapid development of the low-altitude economy, unmanned aerial vehicles (UAVs), as aerial platforms with high mobility and flexible scheduling capabilities, have been widely used in data acquisition, mobile communication, and other mission scenarios. Compared with single-UAV systems, multi-UAV cooperative systems have significant advantages in terms of mission coverage, execution efficiency, and robustness. To overcome these shortcomings, multi-UAV cooperative obstacle avoidance route planning methods have gradually become a research hotspot.

[0010] Existing research has proposed various traditional algorithms, among which the A* algorithm based on heuristic search has been widely used due to its high efficiency and good scalability. In recent years, Deep Reinforcement Learning (DRL) has been widely applied in route planning tasks. Deep Q-Network (DQN) can autonomously learn strategies through interaction with the environment, approximating the optimal solution in unknown environments, thus compensating for the shortcomings of traditional methods in terms of environmental adaptability. Currently, a threshold-based Deep Q-Network (T-DQN) algorithm has been proposed, which effectively improves the convergence speed of obstacle avoidance strategies by filtering experience replay samples through thresholding. Currently, a dual-network architecture is designed by fusing Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) to extract spatiotemporal features in UAV obstacle avoidance tasks. However, when these methods are extended to multi-UAV collaborative tasks, they often have difficulty accurately distinguishing whether the observed changes originate from environmental disturbances or adjustments to other UAV strategies, and the strategy learning process is therefore easily affected by environmental non-stationarity.

[0011] To enhance the stability and generalization ability of multi-agent policies, Value-Decomposition Networks (VDNs) employ an additive structure, combining the value functions of individuals based on local observations. The Q-decomposition Multi-agent Independent eXtension (QMIX) algorithm introduces a hybrid network structure that satisfies monotonicity constraints and utilizes the hybrid network to generate weighted parameters, thereby enhancing the collaborative capabilities among agents. To further mitigate the overestimation problem, Feng et al., addressing the difficulty of balancing communication performance and flight safety for UAVs in low-altitude mixed obstacle environments, proposed an obstacle avoidance communication model based on collision probability maps. They combined user scheduling optimization with Multi-Agent Deep Deterministic Policy Gradient (MADDPG) to achieve collaborative planning. Furthermore, Guan et al., for multi-UAV collaborative optimization in disaster emergency communication scenarios, jointly modeled with K-clustering and Multi-Agent Proximal Policy Optimization (MAPPO) algorithms, achieving faster policy convergence and higher system throughput.

[0012] Furthermore, some studies have attempted to combine prior path information with reinforcement learning to accelerate training convergence. Currently, prior paths generated by A* and conflict priority rules are embedded into the DQN framework; however, this method relies too heavily on static prior knowledge, limiting the policy's exploration ability and generalization performance in unknown environments. Current research utilizes the A* algorithm to plan the global trajectory offline, and then fuses local trajectory fragments with real-time observations into the deep reinforcement learning network online, guiding the policy to converge stably to the global objective. These works demonstrate that the organic integration of prior knowledge and learning optimization helps accelerate convergence and improve the robustness of online decision-making.

[0013] In summary, existing multi-agent deep reinforcement learning methods have significant shortcomings in multi-UAV autonomous cooperative obstacle avoidance and navigation tasks: on the one hand, relying solely on local observations makes it difficult to effectively capture global state features, resulting in insufficient accuracy in value estimation and low learning efficiency; on the other hand, in unknown environments, frequent collisions generate a large number of negative samples, reducing the effectiveness of initial training data and significantly delaying the stable convergence process of the strategy. Summary of the Invention

[0014] The purpose of this invention is to address the shortcomings and deficiencies of existing technologies by proposing a priori-guided, time-series fusion-based multi-UAV cooperative obstacle avoidance route planning method. This method utilizes A* offline-generated reference paths as weak priors and fuses local observations and prior information through a two-layer time-series network to model local dynamics and group cooperation, thereby improving the stability and coordination of joint value estimation. Simulation results show that this invention can reduce path conflicts while maintaining real-time performance, improving the feasibility and coordination of multi-UAV route planning.

[0015] The technical solution adopted by this invention to solve its technical problem is: a priori-guided time-series fusion multi-UAV cooperative obstacle avoidance route planning method, which includes the following steps: Step 1: Multi-UAV route planning.

[0016] A group of drones is set up obstacle collection in Indicates the number of drones. The number of obstacles in the environment. (Drone) From the starting point Starting from the action space Select an action, and its position changes over time. Finally reached the target point And generate a trajectory Its trajectory length is Where t represents the discrete decision time step, T represents the maximum number of decision steps per round in the mission, and the set of trajectories for all drones is: .

[0017] In multi-UAV cooperative trajectory planning, path length is typically used as the primary performance indicator for UAVs to complete their tasks. Therefore, the objective function of this invention is to minimize the total path length of the entire UAV swarm in a given environment while avoiding collisions. Specifically: (1) To ensure the feasibility of the route and the security of the system, each drone The following constraints must be met: A path length constraint, limiting the path length of each drone to no more than the maximum flight distance. For safety constraints, ensure that the Euclidean distance between the drone and other drones and obstacles is never less than the minimum safe distance at any given time. To avoid collisions; initial and final position constraints ensure that each drone departs from a designated starting point and accurately arrives at its target location. Violation of any of these constraints constitutes mission failure.

[0018] Step 2: Establish the POMDP model.

[0019] 2-1) Observational Space Modeling Within the POMDP framework, the global state of the environment Visible only during the intensive training phase; during the execution phase, each agent relies on local observations. Independent decision-making. Regarding drones. Its local observation It consists of its own state and environmental information within the perceptual domain. Local observation. By radius The sensor in Obtained within a three-dimensional local grid, it only reflects obstacle and neighboring device information within the perceptible range. The specific definition is as follows: (2) in, for The current position at any given moment; For the target location; For time t, the drone Distance to target; For time t, the drone and drones Euclidean distance ( ); For time t, the drone Distance to the nearest obstacle. During the training phase, A* paths are generated offline on the grid; during the execution phase, each agent can only access local path segments within the observation range and does not access the global grid.

[0020] 2-2) Action Space Modeling In multi-UAV cooperative obstacle avoidance planning, the action space determines the set of UAV behaviors and directly affects the efficiency and executability of policy learning. This invention employs a discrete action space. Includes a set of six-connected discrete actions and dwell actions: During execution, boundary crossing detection and obstacle feasibility assessment are performed to ensure the safety of action decisions.

[0021] Using a discrete action space has two advantages: First, it is consistent with the connectivity of the 3D grid environment and the offline A* reference path, which can achieve a natural mapping between the global reference trajectory and the action sequence, avoiding the problem of inconsistency between the prior path and the actual execution; Second, discretization reduces the complexity of the policy network output dimension and the joint action space, thereby improving training stability and sample efficiency.

[0022] Furthermore, to improve the continuity and smoothness of the trajectory, this invention introduces a B-spline trajectory smoothing method in the post-processing stage, converting the discrete path node sequence into a continuous trajectory. Specifically, a cubic B-spline is used, represented as follows: (3) in, These are control points obtained by fitting discrete path nodes. The basis functions are cubic B-spline functions. It should be noted that this smoothing process is only used for trajectory display and execution reference; it does not participate in reinforcement learning training, nor does it change the definition of the discrete action space.

[0023] 2-3) Reward Function Modeling To achieve a unified optimization that minimizes path length and balances safety constraints, this invention designs a reward function within a reinforcement learning framework that combines potential-based shaping with global prior guidance. This design comprehensively considers factors such as task completion, path advancement, collision penalties, and prior path alignment, ensuring long-term policy convergence while enabling the UAV to achieve a balance between autonomous exploration and global guidance.

[0024] Define the team's average distance to the target as: (4) in Indicates the first A drone at all times The Euclidean distance to the target point. Simultaneously, the safety constraints between obstacles and the drone are expressed as soft constraints. (5) in , These represent the minimum distance between the obstacle and the drone. This indicates taking the non-negative part.

[0025] To enhance the global guidance trend of UAVs, a reference path based on the single-agent A* algorithm is introduced. And define the deviation of the current UAV from the reference path as: (6) Its change Reflects the degree of fit between the drone and the reference trajectory: when At that time, the drone is approaching the prior path. Consider the potential function formed by the target distance and the reference alignment: (7) in At that moment The instant reward is defined as: (8) in, This represents a small penalty at each step, used to suppress redundant actions and promote convergence; and These are the weights for obstacles and inter-machine safety constraints, respectively. The discount factor. Potential function difference. The densely packed shaping elements incur a small penalty when deviating from the global path, while the drone receives a cumulative positive reward when it re-fits the reference trajectory. This mechanism maintains both exploration diversity and the long-term guiding trend towards the global path. and The rewards and penalties are respectively for reaching the target and for collision termination events; among them, and These are the indicator functions for reaching the target and colliding with it, respectively. The value is 10 when the corresponding event occurs, and 0 otherwise.

[0026] Based on the aforementioned reward function, each drone can autonomously plan and collaboratively complete tasks while avoiding collisions. Each drone, according to its state... With action Receive the corresponding instant rewards The system's cumulative discount reward is defined as (9) This objective is consistent with the optimization objective of minimizing the overall trajectory length as expressed in Equation 1, guiding the UAV swarm to converge to the global objective quickly and stably while satisfying the constraints.

[0027] Step 3: Prior-guided temporal fusion value decomposition algorithm PGL-QMIX.

[0028] For the A drone is deployed to calculate its shortest path offline on a static map. And at runtime, based on the current position With perception radius Extract locally visible segments: (11) For local path segments This invention comprehensively evaluates the importance of each reference node, and for each node... Construct a comprehensive scoring function. The specific definition is as follows: (12) in, This represents the distance between the node and the nearest obstacle. This represents the minimum distance between the node and other drones. and These represent the distances from the node to the current drone position and the target point, respectively. It comprehensively measures the importance of each node in the current segment; the higher the value, the greater the reference value of the node at the current moment. At the individual level, local observation After being concatenated with the reference bias vector, the data is input into a prior LSTM network. Through input gates, forget gates, and output gates, the temporal information of local observations and the reference trend is dynamically integrated to extract individual temporal features. The calculation process is as follows: (13) in, For current drones exist Input at any time and for The hidden state and unit state output at any time. This is the weight matrix. For bias. Forget Gate Determine whether to retain the cell state from the previous time step. and hidden state Input gate Update the cell state to the current time. And generate candidate cell states. Output gate Output the hidden state at the current moment. In the middle. Its network structure is as follows: Figure 3 As shown. Step 4: Q-value structure optimization under state-action timing fusion. In multi-UAV cooperative navigation, the environment and individual strategies evolve over time, and inter-entity interactions exhibit significant non-stationarity. Relying solely on individual-level temporal modeling is insufficient to reflect the dynamic dependencies of group cooperative behavior, easily leading to unstable joint value assessments. Therefore, this invention introduces a system-level LSTM into the PGL-QMIX framework to generate time-dependent nonlinear weights and biases, adaptively adjusting the contribution of each individual's Q-value to the joint Q, thereby enhancing the expressiveness and stability of the joint strategy for non-stationary interactions. This invention uses prior embedding LSTM to obtain individual temporal features. Regarding the first For each drone, its individual action value function is defined as: (14) in For immediate team rewards, and The parameters of the evaluation network and the target network are respectively used. The temporal value of the individual Q-function under local observation and prior guidance is used to evaluate the value of the individual's actions. It should be noted that the A* prior only provides global geometric trend information and does not directly guarantee conflict-free cooperation among multiple UAVs; the relevant dynamic conflict mitigation and joint obstacle avoidance are jointly implemented by the time-varying weights of the system-level LSTM and its monotonic hybrid network. First, the temporal features of all agents are collected at time step t, and an input vector reflecting the global cooperative behavior is constructed. The temporal representations of all individuals are concatenated into... System-level LSTM integration in the time dimension Inter-entity dependencies and output hidden states: (15) in These are the parameters for the system-level LSTM. Subsequently, the output layer of this LSTM architecture directly generates the mixed weights and biases: (16) (17) because make sure Furthermore, it can adaptively map the Q-values ​​of each drone to a weighted coefficient and a global bias based on the current state of each drone. Finally, a joint Q-value function satisfying monotonic value decomposition is constructed. This is to achieve dynamic integration and value assessment of the overall strategy. (18) in Meanwhile, to more effectively optimize joint Q-value estimation, the system layer optimizes the joint action value function by minimizing the temporal difference (TD) error. This allows it to gradually approach the true optimal action value. The specific loss function is defined as follows: (19) in, express Target Q value at any given time: (20) The network parameters are synchronized at fixed steps. Through this optimization process, the system layer can dynamically correct the estimation error of the joint Q in non-stationary environments, maintain the continuous convergence of the global value function, and thus enable the multi-UAV swarm to achieve stable collaborative decision-making over long time scales. Beneficial effects: 1. This invention introduces a two-level temporal modeling mechanism into the QMIX value decomposition framework, and then integrates global reference and local observation through prior LSTM and hybrid LSTM, thereby realizing the fusion of global guidance information and local temporal features, which effectively improves the foresight and stability of policy learning. 2. This invention constructs a mechanism for the construction and temporal fusion of local prior fragments. This mechanism utilizes reference paths generated in a static environment using the A* algorithm, extracting only locally visible path fragments as heuristic prior inputs during execution. This allows for the acquisition of global geometric trend information without relying on complete global information. While maintaining autonomous exploration capabilities, it significantly improves convergence efficiency and generalization performance in unknown environments. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the multi-UAV route planning system model of the present invention.

[0030] Figure 2 This is a schematic diagram of the PGL-QMIX system model of the present invention.

[0031] Figure 3 This is a schematic diagram of the prior LSTM system model of the present invention.

[0032] Figure 4 This invention is a schematic diagram of a grid simulation scene setup.

[0033] Figure 5 This invention provides a comparative diagram of the reward values ​​of seven algorithms in different scenarios.

[0034] Figure 6 This invention presents a schematic diagram comparing the success rates of seven algorithms in different scenarios.

[0035] Figure 7 For the present invention 20 3 A schematic diagram of the seven algorithm paths in the scenario (seed=42).

[0036] Figure 8 For the present invention 30 3 A schematic diagram of the seven algorithm paths in the scenario (seed=42). Detailed Implementation

[0037] The invention will now be described in further detail with reference to the accompanying drawings.

[0038] 1) Multi-UAV route planning like Figure 1 As shown in the figure, the initial positions of multiple drones, the target position, and the distribution of obstacles are illustrated. Consider a group of drones. obstacle collection in Indicates the number of drones. The number of obstacles in the environment. (Drone) From the starting point Starting from the action space Select an action, and its position changes over time. Finally reached the target point And generate a trajectory Its trajectory length is Where t represents the discrete decision time step, T represents the maximum number of decision steps per round in the mission, and the set of trajectories for all drones is: .

[0039] In multi-UAV cooperative trajectory planning, path length is typically used as the primary performance indicator for UAVs to complete their tasks. Therefore, the objective function of this invention is to minimize the total path length of the entire UAV swarm in a given environment while avoiding collisions. Specifically: (1) To ensure the feasibility of the route and the security of the system, each drone The following constraints must be met: A path length constraint, limiting the path length of each drone to no more than the maximum flight distance. For safety constraints, ensure that the Euclidean distance between the drone and other drones and obstacles is never less than the minimum safe distance at any given time. To avoid collisions; initial and final position constraints ensure that each drone departs from a designated starting point and accurately arrives at its target location. Violation of any of these constraints constitutes mission failure.

[0040] 2) Establishment of the POMDP model Because multi-UAV autonomous cooperative obstacle avoidance navigation tasks are state-dependent and cannot fully observe all information in the environment, this problem can be modeled as a partially observable Markov decision process. The POMDP model can be represented as a tuple: in For the global state space; It is the collection of all actions that drones can perform. It is a state transition function, representing the state transition. Under these circumstances, the drones will carry out joint operations. Then, the system transitions to state. The probability of this is denoted as: ; The drone takes action as a reward function. The shared global rewards obtained; Represents the observation space; the first A drone at any time The observation is recorded as The observation function is Indicates that the drone has taken action. Transfer to Time to obtain observation The probability of. It is a discount factor used to balance the impact of short-term and long-term rewards.

[0041] 2-1) Observational Space Modeling Within the POMDP framework, the global state of the environment Visible only during the intensive training phase; during the execution phase, each agent relies on local observations. Independent decision-making. Regarding drones. Its local observation It consists of its own state and environmental information within the perceptual domain. Local observation. By radius The sensor in Obtained within a three-dimensional local grid, it only reflects obstacle and neighboring device information within the perceptible range. The specific definition is as follows: (2) in, for The current position at any given moment; For the target location; For time t, the drone Distance to target; For time t, the drone and drones Euclidean distance ( ); For time t, the drone Distance to the nearest obstacle. During the training phase, A* paths are generated offline on the grid; during the execution phase, each agent can only access local path segments within the observation range and does not access the global grid.

[0042] 2-2) Action Space Modeling In multi-UAV cooperative obstacle avoidance planning, the action space determines the set of UAV behaviors and directly affects the efficiency and executability of policy learning. This invention employs a discrete action space. Includes a set of six-connected discrete actions and dwell actions: During execution, boundary crossing detection and obstacle feasibility assessment are performed to ensure the safety of action decisions.

[0043] Using a discrete action space has two advantages: First, it is consistent with the connectivity of the 3D grid environment and the offline A* reference path, which can achieve a natural mapping between the global reference trajectory and the action sequence, avoiding the problem of inconsistency between the prior path and the actual execution; Second, discretization reduces the complexity of the policy network output dimension and the joint action space, thereby improving training stability and sample efficiency.

[0044] Furthermore, to improve the continuity and smoothness of the trajectory, this invention introduces a B-spline trajectory smoothing method in the post-processing stage, converting the discrete path node sequence into a continuous trajectory. Specifically, a cubic B-spline is used, represented as follows: (3) in, These are control points obtained by fitting discrete path nodes. The basis functions are cubic B-spline functions. It should be noted that this smoothing process is only used for trajectory display and execution reference; it does not participate in reinforcement learning training, nor does it change the definition of the discrete action space.

[0045] 2-3) Reward Function Modeling To achieve a unified optimization that minimizes path length and balances safety constraints, this invention designs a reward function within a reinforcement learning framework that combines potential-based shaping with global prior guidance. This design comprehensively considers factors such as task completion, path advancement, collision penalties, and prior path alignment, ensuring long-term policy convergence while enabling the UAV to achieve a balance between autonomous exploration and global guidance.

[0046] Define the team's average distance to the target as: (4) in Indicates the first A drone at all times The Euclidean distance to the target point. Simultaneously, the safety constraints between obstacles and the drone are expressed as soft constraints. (5) in , These represent the minimum distance between the obstacle and the drone. This indicates taking the non-negative part.

[0047] To enhance the global guidance trend of UAVs, a reference path based on the single-agent A* algorithm is introduced. And define the deviation of the current UAV from the reference path as: (6) Its change Reflects the degree of fit between the drone and the reference trajectory: when At that time, the drone is approaching the prior path. Consider the potential function formed by the target distance and the reference alignment: (7) in At that moment The instant reward is defined as: (8) in, This represents a small penalty at each step, used to suppress redundant actions and promote convergence; and These are the weights for obstacles and inter-machine safety constraints, respectively. The discount factor. Potential function difference. The densely packed shaping elements incur a small penalty when deviating from the global path, while the drone receives a cumulative positive reward when it re-fits the reference trajectory. This mechanism maintains both exploration diversity and the long-term guiding trend towards the global path. and The rewards and penalties are respectively for reaching the target and for collision termination events; among them, and These are the indicator functions for reaching the target and colliding with it, respectively. The value is 10 when the corresponding event occurs, and 0 otherwise.

[0048] Based on the aforementioned reward function, each drone can autonomously plan and collaboratively complete tasks while avoiding collisions. Each drone, according to its state... With action Receive the corresponding instant rewards The system's cumulative discount reward is defined as (9) This objective is consistent with the optimization objective of minimizing the overall trajectory length as expressed in Equation 1, guiding the UAV swarm to converge to the global objective quickly and stably while satisfying the constraints.

[0049] 3) Prior-guided temporal fusion value decomposition algorithm PGL-QMIX To improve the efficiency and stability of collaborative route planning among multiple UAVs in complex environments, this invention proposes a priori-guided temporal fusion value decomposition algorithm, PGL-QMIX. This method combines heuristic prior information with a multi-agent value decomposition structure within a centralized training and distributed execution framework: In a static environment, the system first generates an A* reference path offline for each UAV; during the training phase, it utilizes the global state and the complete reference path, while during the execution phase, each agent only accesses local path segments and local environmental information within its perception range, thus maintaining some observable settings while introducing global trend priors.

[0050] like Figure 2 As shown, PGL-QMIX adopts a two-level temporal modeling structure of "individual layer - system layer". The individual layer inputs local A* path segments and UAV local observations into the LSTM to extract temporal features that fuse prior guidance and local dynamic information; the system layer further fuses the temporal representations of all agents to generate mixed weights and biases that satisfy monotonicity constraints, and constructs a joint action value function.

[0051] The A* algorithm is a typical global route planning method based on heuristic search, and its evaluation function is defined as: (10) in This represents the cumulative cost from the starting point to the current node. This provides a heuristic estimate from the node to the target. A* can efficiently generate reference paths with clear global trends in static environments, but it is only suitable for static obstacles and single-UAV scenarios, and struggles to handle multi-UAV interactions. Therefore, this invention does not use A* as an online decision-making strategy, but instead transforms its output path into a heuristic prior, providing global directional guidance for reinforcement learning.

[0052] like Figure 2 As shown, to accommodate partially observable settings, this invention only captures local A* path segments within the UAV's perception range. For the first... A drone is deployed to calculate its shortest path offline on a static map. And at runtime, based on the current position Extracting locally visible segments using the perception radius: (11) For local path segments This invention comprehensively evaluates the importance of each reference node, and for each node... Construct a comprehensive scoring function. The specific definition is as follows: (12) in, This represents the distance between the node and the nearest obstacle. This represents the minimum distance between the node and other drones. and These represent the distances from the node to the current drone position and the target point, respectively. It comprehensively measures the importance of each node in the current segment; the higher the value, the greater the reference value of the node at the current moment.

[0053] At the individual level, local observation After being concatenated with the reference bias vector, the data is input into a prior LSTM network. Through input gates, forget gates, and output gates, the temporal information of local observations and the reference trend is dynamically integrated to extract individual temporal features. The calculation process is as follows: (13) in, For current drones exist Input at any time and for The hidden state and unit state output at any time. This is the weight matrix. For bias. The forget gate determines whether to retain the cell state from the previous time step. and hidden state Input gate Update the cell state to the current time. And generate candidate cell states. Output gate Output the hidden state at the current moment. In the middle. Its network structure is as follows: Figure 3 As shown.

[0054] In summary, by fusing locally visible reference path segments with local observations, the individual layer achieves unified modeling of global trend guidance and local dynamic information, providing a stable temporal representation for subsequent multi-agent collaborative decision-making.

[0055] 4) Q-value structure optimization under state-action timing fusion In multi-UAV cooperative navigation, the environment and individual strategies evolve over time, and inter-entity interactions exhibit significant non-stationarity. Relying solely on individual-level temporal modeling is insufficient to reflect the dynamic dependencies of group cooperative behavior, easily leading to unstable joint value assessments. Therefore, this invention introduces a system-level LSTM into the PGL-QMIX framework to generate time-dependent nonlinear weights and biases, adaptively adjusting the contribution of each individual's Q-value to the joint Q, thereby enhancing the expressiveness and stability of the joint strategy for non-stationary interactions.

[0056] This invention uses prior embedding LSTM to obtain individual temporal features. Regarding the first For each drone, its individual action value function is defined as: (14) in For immediate team rewards, and The parameters of the evaluation network and the target network are respectively used. The temporal value of the individual Q-function under local observation and prior guidance is used to evaluate the value of the individual's actions. It should be noted that the A* prior only provides global geometric trend information and does not directly guarantee conflict-free cooperation among multiple UAVs; the relevant dynamic conflict mitigation and joint obstacle avoidance are jointly implemented by the time-varying weights of the system-level LSTM and its monotonic hybrid network. First, the temporal features of all agents are collected at time step t, and an input vector reflecting the global cooperative behavior is constructed. The temporal representations of all individuals are concatenated into... System-level LSTM integration in the time dimension Inter-entity dependencies and output hidden states: (15) in These are the parameters for the system-level LSTM. Subsequently, the output layer of this LSTM architecture directly generates the mixed weights and biases: (16) (17) because make sure Furthermore, it can adaptively map the Q-values ​​of each drone to a weighted coefficient and a global bias based on the current state of each drone. Finally, a joint Q-value function satisfying monotonic value decomposition is constructed. This is to achieve dynamic integration and value assessment of the overall strategy. (18) in Meanwhile, to more effectively optimize joint Q-value estimation, the system layer optimizes the joint action value function by minimizing the temporal difference (TD) error. This allows it to gradually approach the true optimal action value. The specific loss function is defined as follows: (19) in, express Target Q value at any given time: (20) The network parameters are synchronized at fixed steps. Through this optimization process, the system layer can dynamically correct the estimation error of the joint Q in non-stationary environments, maintain the continuous convergence of the global value function, and thus enable the multi-UAV swarm to achieve stable collaborative decision-making over long time scales.

[0057] The specific algorithm flow is shown in the table below: To evaluate the scalability of PGL-QMIX in large-scale drone swarm scenarios, let the number of drones be... The number of nodes in the 3D raster is The number of actions is Individual network input dimension is The hidden state dimension of the individual layer LSTM is The training batch size is The sequence expansion length is .

[0058] Offline preprocessing stage: For each drone, the A* algorithm is run offline to generate a reference path. The time complexity of A* can be written as... Therefore, the total complexity of N drones is This process is completed before the task begins, so it does not affect the real-time nature of online decision-making.

[0059] Online execution phase: The main computations at each time step include forward inference of the individual policy network and extraction and scoring of local prior fragments. At the individual level, the complexity of a single agent's forward inference using an LSTM is O(log n). The additional overhead of outputting the action value and making a greedy selection is If the number of local reference nodes that need to be processed within the sensing range is... The complexity of this step is . With the observation radius and grid resolution fixed, The upper bound can be considered a constant. In summary, the complexity of each time step in the online execution phase is: (twenty one) When the network size and sensing radius are fixed, the above formula increases linearly with the number of drones N, meaning the online execution complexity is... This property avoids the exponential overhead caused by explicitly enumerating the joint action space. This is beneficial for real-time decision-making in large-scale cluster scenarios.

[0060] (2) Space complexity: The space overhead mainly consists of network parameters, runtime state, and reference path / replay data. Due to the parameter sharing mechanism, the parameter storage of individual policy networks and hybrid networks is of constant size. With the number of drones It is irrelevant; however, at runtime, it is necessary to maintain the LSTM hidden state and cache information of each agent, with an overhead of In addition, storage is required. There are 10 offline reference paths, with a space complexity of approximately 1000. (in (This is the average path length); the storage overhead of the experience replay pool is linearly related to its capacity. Overall, the online runtime memory of this algorithm varies with... It grows linearly and has good scalability.

[0061] The simulation experiments and performance verification of this invention include the following: To verify the effectiveness of the proposed PGL-QMIX algorithm, this invention built a three-dimensional UAV swarm cooperative navigation simulation environment based on Python 3.12 and PyTorch 2.7.0, and completed the experiment on an Intel Core i9-14900KF CPU and NVIDIA RTX 5080 GPU platform.

[0062] The simulation environment of this invention adopts a voxelized three-dimensional grid space, with the voxel side length set to 1, and three scene scales are set: Scene 1 is 20. 3 Scene 2 is 30 3 Scene 3 is 40 3 Four drones are deployed in the environment, employing a spherical perception model with a detection radius of 3 (voxel units) and a minimum safe distance of 1; the initial position and target position are as follows: Figure 4 As shown.

[0063] To comprehensively evaluate the performance of the proposed method, comparative and ablation experiments were designed. The comparative algorithms included the multi-agent value decomposition methods VDN and QMIX, the policy gradient-based CTDE algorithm MAPPO, and the traditional heuristic path planning algorithm A*. Furthermore, three ablation variants were constructed on the PGL-QMIX framework: 1) No-Prior (removing the A* heuristic prior), 2) No-iLSTM (removing the individual-level LSTM), and 3) No-sLSTM (removing the system-level LSTM), respectively used to analyze the impact of prior guidance, individual temporal modeling, and system-level temporal fusion on the algorithm performance. The relevant hyperparameter settings are shown in Table 1.

[0064] Table 1 Simulation Hyperparameter Settings The learning rate decreases linearly with the number of training steps, when The time decays to 0 to ensure smooth convergence.

[0065] (twenty two) To ensure a consistent comparison of the training performance of different algorithms, this invention conducts experiments under the same 3D grid scene and uniform hyperparameter settings. Each method is trained for 150,000 steps, and offline evaluation is performed every 200 (Δ) steps on 128 random maps to obtain reward sequences and task success rate sequences. A greedy decision-making approach is used during the testing phase; the task is considered successful when all drones reach their respective targets within the maximum step limit.

[0066] Based on the above evaluation results, this invention uses the following indicators for quantitative comparison: the overall learning efficiency is measured by the area under the reward curve (AUC), and the reward-step curve is numerically approximated by a trapezoidal integral: (twenty three) Steady-state performance is measured by the average reward and average success rate of the last 50 evaluations; convergence speed is measured by the number of training steps required for the success rate to first reach 0.8. In this paper, "success rate improvement" is expressed as a percentage point (pp), representing the difference in steady-state success rates between different algorithms.

[0067] like Figure 5 As shown, the present invention applies to three voxelized 3D scenes (20 3 30 3 40 3 Under these conditions, the reward curves of all five algorithms gradually rise and tend to converge as training progresses. Among them, PGL-QMIX exhibits faster convergence speed and higher steady-state reward in all three scenarios, with convergence steps of 32400, 33600, and 37400 respectively, all earlier than other algorithms. As the task size increases, the fluctuation range of the PGL-QMIX curve remains small, demonstrating good training stability. Simulation results show that, measured by the area under the curve (AUC), PGL-QMIX improves overall learning efficiency by 3.61%, 4.43%, and 3.15% compared to NoPrior, respectively. In the steady-state phase (last 50 evaluations), PGL-QMIX has the highest average reward in all three scenarios, improving by 1.32, 7.99, and 4.58 compared to the second-best algorithms, respectively. In contrast, NoPrior typically requires an additional 4000-6000 steps to converge, while QMIX and VDN exhibit more obvious lag in convergence. The results show that the time-varying weighting and potential function shaping reward of the system-level LSTM can jointly improve the stability of joint value estimation and alleviate the training fluctuations caused by early sparse rewards, thereby improving the convergence speed and stability of the algorithm.

[0068] Table 2 Comparison of convergence performance in different scenarios like Figure 6 As shown, the present invention applies to three voxelized 3D scenes (20 3 30 3 40 3 Under the same training budget, the success rate of each algorithm changed throughout the entire process. PGL-QMIX's test success rate curves crossed the 80% threshold first, reaching this level at approximately 41,000, 51,000, and 56,000 steps respectively; No-Prior followed, while No-sLSTM's success rate improvement lagged significantly. Meanwhile, QMIX and VDN, under the same training budget, had not yet reached a clear steady state, and their curves continued to rise.

[0069] Simulation results show that in the steady-state phase (last 50 evaluations), PGL-QMIX achieves the highest average success rates in the three scenarios: 96.7%, 94.3%, and 92.9%, respectively. Furthermore, in all three scenarios, its highest single-round success rate reaches 100%. This indicates that the temporal fusion of local observations and prior scores by the individual-layer LSTM effectively improves the generalization and success rate of the policy. Unlike directly relying on static A* paths, PGL-QMIX transforms prior paths into learnable local evaluation signals, enabling the policy to dynamically adjust path selection based on the real-time environment, thus maintaining a high success rate in stochastic scenarios.

[0070] surface Comparison of task success rates in different scenarios like Figure 7-8 As shown, this invention demonstrates the trajectories planned by various algorithms at different environmental scales in the same test environment (seed42) after training. To improve the geometric executability and graphical readability of the trajectories on the continuous dynamics platform, this invention employs three B-splines to post-process and smooth the discrete paths. It should be noted that policy learning and performance evaluation are still based on discrete action space and discrete path statistics, and the smoothing process does not change the reinforcement learning modeling method and metric definition. The results show that PGL-QMIX can generate collision-free arrival trajectories while maintaining safe distances in all three scenarios, and maintains good trajectory continuity and stability as the scenario scale increases. In contrast, No-sLSTM has relatively insufficient global coordination, No-Prior is more prone to local backtracking and path oscillations, while VDN and QMIX exhibit near-obstacle path segments more frequently.

[0071] This invention statistically analyzed the average path length at different environmental scales on 128 random maps, using the shortest path generated by A* as a reference baseline. The relevant data is shown in Table 4. It can be seen that the average path length of each algorithm increases with the increase in scene scale, but the result of PGL-QMIX is consistently closest to the A* baseline, demonstrating better stability. Compared to traditional QMIX, PGL-QMIX shortens the average path length by approximately 8.8%, 12.3%, and 16.1% in the three scenarios, respectively, and the path length increases approximately linearly with the increase in scene scale, indicating that this method has higher path planning efficiency in complex environments.

[0072] Table 4 Comparison of Algorithm Paths in Different Scenarios The above results indicate that the performance improvement of PGL-QMIX is mainly due to the synergistic effect of the two-layer temporal structure: the individual layer uses A* prior to provide global direction guidance and reduce redundant search; the system layer dynamically coordinates the value contribution and motion trend of each UAV at the team scale, thereby suppressing mutual interference and further shortening the overall path length.

[0073] The present invention has been described in detail above, but the present invention is not limited to the above-described embodiments. Those skilled in the art can make various changes to the present invention based on their knowledge to achieve better results.

Claims

1. A priori-guided temporal fusion multi-UAV cooperative obstacle avoidance path planning method, characterized in that, The method includes the following steps: Step 1: Multi-UAV route planning; A group of drones is set up obstacle collection in Indicates the number of drones. For the number of obstacles in the environment, drones From the starting point Starting from the action space Select an action, and its position changes over time. Finally reached the target point And generate a trajectory Its trajectory length is Where t represents the discrete decision time step, T represents the maximum number of decision steps per round in the mission, and the set of trajectories for all drones is: ; Step 2: Establish the POMDP model; Step 2-1: Observational space modeling; Within the POMDP framework, the global state of the environment Visible only during the intensive training phase; during the execution phase, each agent relies on local observations. Independent decision-making for drones Its local observation Composed of its own state and environmental information within the perceptual domain, local observation By radius The sensor in Obtained within a three-dimensional local grid, it only reflects obstacle and neighboring machine information within the perceptible range; (2) in, for The current position at any given moment; For the target location; For time t, the drone Distance to target; For time t, the drone and drones Euclidean distance ( ); For time t, the drone The distance to the nearest obstacle is used to generate A* paths offline on the grid during the training phase; during the execution phase, each agent can only access local path segments within the observation range and does not access the global grid. Step 2-2: Action space modeling; In multi-UAV cooperative obstacle avoidance planning, the action space determines the set of UAV behaviors and directly affects the efficiency and feasibility of policy learning. A discrete action space is adopted. Includes a set of six-connected discrete actions and dwell actions: During execution, boundary crossing detection and obstacle feasibility assessment are performed to ensure the safety of action decisions. Using a discrete action space has two advantages: First, it is consistent with the connectivity of the 3D grid environment and the offline A* reference path, which can realize the natural mapping between the global reference trajectory and the action sequence, avoiding the problem of inconsistency between the prior path and the actual execution; Second, discretization reduces the complexity of the policy network output dimension and the joint action space, thereby improving training stability and sample efficiency. In the post-processing stage, a B-spline trajectory smoothing method is introduced to convert the discrete path node sequence into a continuous trajectory. Specifically, a cubic B-spline is used, represented as follows: (3) in, These are control points obtained by fitting discrete path nodes. The basis functions are cubic B-spline functions. It should be noted that this smoothing process is only used for trajectory display and execution reference, and does not participate in reinforcement learning training, nor does it change the definition of the discrete action space. Steps 2-3: Modeling the reward function; To achieve a unified optimization of path length minimization and safety constraints, a reward function based on potential-based shaping and global prior guidance is designed in the reinforcement learning framework. This design comprehensively considers factors such as task completion, path advancement, collision penalty and prior path fit, ensuring the long-term convergence of the policy while enabling the UAV to have a balance between autonomous exploration and global guidance. Define the team's average distance to the target as: (4) in Indicates the first A drone at all times The Euclidean distance to the target point, and the safety constraints between the obstacle and the drone are expressed in soft constraint form as follows: (5) in , These represent the minimum distance between the obstacle and the drone. This indicates taking the non-negative part; Step 3: Prior-guided temporal fusion value decomposition algorithm PGL-QMIX; For the A drone is deployed to calculate its shortest path offline on a static map. And at runtime, based on the current position With perception radius Extract locally visible segments: (11) For local path segments Internal comprehensive assessment of the importance of each reference node, for each node The comprehensive scoring function is constructed and defined as follows: (12)， in, This represents the distance between the node and the nearest obstacle. This represents the minimum distance between the node and other drones. and These represent the distances from the node to the current drone position and the target point, respectively. It comprehensively measures the importance of each node in the current segment; the higher the value, the greater the reference value of the node at the current moment. At the individual level, local observation After being concatenated with the reference bias vector, the data is input into a prior LSTM network. Through input gates, forget gates, and output gates, the temporal information of local observations and the reference trend is dynamically integrated to extract individual temporal features. The calculation process is as follows: (13) in, For current drones exist Input at any time and for The hidden state and cell state output at all times. This is the weight matrix. For bias, forget gate Determine whether to retain the cell state from the previous time step. and hidden state Input gate Update the cell state to the current time. And generate candidate cell states. Output gate Output the hidden state at the current moment. middle; Step 4: Q-value structure optimization under state-action timing fusion; In multi-UAV cooperative navigation, the environment and individual strategies evolve over time, and the interactions between the agents exhibit significant non-stationarity. Relying solely on individual-level temporal modeling is insufficient to reflect the dynamic dependence of group cooperative behavior, which can easily lead to instability in joint value assessment. To address this, a system-level LSTM is introduced into the PGL-QMIX framework to generate time-dependent nonlinear weights and biases, which adaptively adjust the contribution of each agent's Q value to the joint Q, thereby enhancing the expressiveness and stability of the joint strategy for non-stationary interactions. Prior embedding LSTM to obtain individual temporal features Regarding the first For each drone, its individual action value function is defined as: (14) in For immediate team rewards, and The evaluation network and target network parameters are evaluated, the temporal value of the individual Q-function under local observation and prior guidance is assessed, and the value of individual actions is evaluated. It should be noted that the A* prior only provides global geometric trend information and does not directly guarantee conflict-free cooperation among multiple UAVs. The related dynamic conflict mitigation and joint obstacle avoidance are jointly implemented by the time-varying weights of the system-level LSTM and its monotonic hybrid network. First, the temporal features of all agents are collected at time step t, and an input vector reflecting the global cooperative behavior is constructed. The temporal representations of all individuals are then concatenated to form a... System-level LSTM integration in the time dimension Inter-entity dependencies and output hidden states: (15) in The system-level LSTM parameters are then used, and subsequently, the output layer of this LSTM architecture directly generates the hybrid weights and biases: (16) (17)。 2. The prior-guided temporal fusion multi-UAV cooperative obstacle avoidance route planning method according to claim 1, characterized in that, Step 1 includes: In multi-UAV cooperative trajectory planning, UAVs typically use path length as the main performance indicator when completing a task. Therefore, the objective function is to minimize the total path length of the entire UAV swarm in a given environment and avoid collisions, including: (1) To ensure the feasibility of the route and the security of the system, each drone The following constraints must be met: A path length constraint, limiting the path length of each drone to no more than the maximum flight distance. For safety constraints, ensure that the Euclidean distance between the drone and other drones and obstacles is never less than the minimum safe distance at any given time. To avoid collisions; initial and final position constraints are in place to ensure that each drone departs from a designated starting point and accurately reaches its target position. If any of the above constraints are violated, the mission is considered a failure.

3. The prior-guided temporal fusion multi-UAV cooperative obstacle avoidance route planning method according to claim 1, characterized in that, Step 2 includes: to enhance the global guidance trend of the UAV, introducing a reference path generated based on the single-agent A* algorithm. And define the deviation of the current UAV from the reference path as: (6) Its change Reflects the degree of fit between the drone and the reference trajectory: when At that time, the UAV is approaching the prior path, and the potential function is formed by the target distance and the reference alignment: (7) in At any moment The instant reward is defined as: (8) in, This represents a small penalty at each step, used to suppress redundant actions and promote convergence; and These are the weights for obstacles and inter-machine safety constraints, respectively. The difference in potential functions is the discount factor. The densely packed shaping elements incur a small penalty when deviating from the global path, while the drone receives a cumulative positive reward when it re-fits the reference trajectory. This mechanism maintains both exploration diversity and the long-term guiding trend towards the global path. and The rewards and penalties are respectively for reaching the target and for collision termination events; among them, and These are the indicator functions for reaching the target and colliding with it, respectively. The value is 10 when the corresponding event occurs, and 0 otherwise. Based on the above reward function, each drone can autonomously plan and collaboratively complete tasks while avoiding collisions. Each drone, according to its state... With action Receive the corresponding instant rewards The system's cumulative discount reward is defined as: (9) This objective is consistent with the optimization objective of minimizing the overall trajectory length as expressed in Equation 1, guiding the UAV swarm to converge to the global objective quickly and stably while satisfying the constraints.

4. The prior-guided temporal fusion multi-UAV cooperative obstacle avoidance route planning method according to claim 1, characterized in that, Step 4 includes: because make sure Furthermore, it can adaptively map the Q-values ​​of each drone to a weighted coefficient and a global bias based on the current state of each drone, ultimately constructing a joint Q-value function that satisfies monotonic value decomposition. This is to achieve dynamic integration and value assessment of the overall strategy. (18) in Meanwhile, to more effectively optimize the joint Q-value estimation, the system layer optimizes the joint action value function by minimizing the temporal difference (TD) error. This allows it to gradually approach the true optimal action value. The specific loss function is defined as follows: (19) in, express Target Q value at any given time: (20) By synchronizing network parameters at fixed steps, the system layer can dynamically correct the estimation error of joint Q in non-stationary environments through this optimization process, maintaining the continuous convergence of the global value function, thereby enabling multi-UAV swarms to achieve stable collaborative decision-making over long time scales.

Images