Unmanned aerial vehicle group path optimization method based on heuristic random integration reinforcement learning

Abstract

The invention discloses an unmanned aerial vehicle group path optimization method based on heuristic random integration reinforcement learning, and relates to the technical field of path optimization. The method comprises the following steps: acquiring current state information of an unmanned aerial vehicle, and judging whether communication is normal; when the communication is normal, a heuristic control action is generated based on the current state information. The heuristic control action includes at least an angular velocity control amount and a linear velocity control amount. And optimizing the heuristic control action by combining the security constraint and multi-Critic network random integration through a random integration double-Q learning strategy with the security constraint, and outputting a candidate control action. And when the communication is abnormal or lost, outputting candidate control actions through an improved dynamic unmanned aerial vehicle sparrow algorithm based on the current state information. And controlling the flight of the unmanned aerial vehicle according to the candidate control action. The key problems of path planning, obstacle avoidance, group collaboration, network interference resistance and the like in multi-unmanned aerial vehicle collaborative navigation in a dynamic environment are solved, and the adaptive capacity and decision-making efficiency of the unmanned aerial vehicle group are improved.

Description

Technical Field

[0001] This invention relates to the field of path optimization technology, and more specifically, to a method for path optimization of unmanned aerial vehicle (UAV) swarms based on heuristic stochastic ensemble reinforcement learning. Background Technology

[0002] With the widespread application of unmanned aerial vehicles (UAVs), the demand for multi-UAV cooperative navigation and autonomous decision-making is becoming increasingly urgent. UAV swarms face dynamic and complex environments during missions, requiring capabilities such as real-time path planning, dynamic obstacle avoidance, swarm collaboration, and resistance to network interference. However, traditional methods, under conditions of limited computing resources, restricted communication bandwidth, and dynamically changing environments, struggle to achieve efficient and stable collaboration among large-scale UAV swarms, posing significant challenges to their adaptability, real-time performance, and security.

[0003] Currently, several technical solutions have been proposed to address these challenges. Centralized path planning methods, which unify path planning through a central control system, can achieve global optimization and collision avoidance. However, as the number of drones increases, they face computational and communication bottlenecks, are prone to single-point-of-failure risks, and struggle to respond quickly to dynamic changes. Virtual force field methods achieve local obstacle avoidance by simulating gravity and repulsion, but are prone to trapping drones in local optima and exhibiting lag in response to dynamic obstacles. Cooperative game theory algorithms are used to optimize task allocation, but their high computational complexity in large-scale groups can lead to uneven task distribution. Redundant communication links enhance anti-interference capabilities by establishing backup channels, but increase system complexity and hardware burden, and may still fail under extreme interference. Adaptive control algorithms can adjust strategies based on environmental changes, but in large-scale groups, delays in local information updates can affect overall coordination.

[0004] In summary, existing technologies generally suffer from insufficient real-time performance, poor adaptability, heavy computational burden, high communication overhead, and high security risks when dealing with large-scale, highly dynamic, and heavily interfered UAV cooperative navigation scenarios. Centralized architectures have limited scalability, while distributed methods struggle to achieve a good balance between obstacle avoidance, cooperation, and anti-interference. A lightweight, adaptive, and integrated solution that simultaneously ensures safety and efficiency is lacking. Therefore, there is an urgent need to propose a novel technical solution that effectively integrates path planning, obstacle avoidance, cooperation, and anti-interference capabilities. Summary of the Invention

[0005] This invention provides a path optimization method for unmanned aerial vehicle (UAV) swarms based on heuristic stochastic ensemble reinforcement learning to improve at least one of the aforementioned technical problems.

[0006] This invention provides a method for optimizing the path of a drone swarm based on heuristic stochastic ensemble reinforcement learning, which includes steps S1 to S5.

[0007] S1. Obtain the current status information of the drone and determine whether the communication is normal.

[0008] S2. When communication is normal, generate heuristic control actions based on the current state information. The heuristic control actions include at least angular velocity control quantities and linear velocity control quantities.

[0009] S3. By using a random ensemble double-Q learning strategy with safety constraints, combined with safety constraints and random ensemble of multiple Critic networks, the heuristic control action is optimized, and candidate control actions are output.

[0010] S4. When communication is abnormal or lost, candidate control actions are output based on the current status information and through the improved dynamic UAV sparrow algorithm.

[0011] S5. Control the drone's flight based on the candidate control actions.

[0012] By adopting the above technical solution, the present invention can achieve the following technical effects: This invention effectively addresses key challenges in multi-UAV cooperative navigation under dynamic environments, including path planning, obstacle avoidance, swarm coordination, and resistance to network interference. It integrates heuristic guidance, a safety-constrained stochastic ensemble double-Q learning strategy, and an improved dynamic UAV sparrow algorithm. This method significantly enhances the adaptive capabilities and decision-making efficiency of UAV swarms. Heuristic rules provide rapid initial path guidance, while safety-constrained reinforcement learning optimizes action decisions, ensuring flight safety and obstacle avoidance reliability. Furthermore, the improved sparrow algorithm enables local path recovery in the event of communication anomalies, thereby enhancing system robustness and mission execution stability. In addition, the distributed adaptive mechanism reduces computational and communication burdens, improves real-time performance and flexibility, and is suitable for large-scale UAV cooperative scenarios, demonstrating broad engineering application value. Attached Figure Description

[0013] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the specific embodiments of the present invention will be briefly introduced below. It should be understood that the following drawings only show some specific embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0014] Figure 1 This is a multi-UAV cooperative navigation framework diagram for UAV swarm path optimization methods.

[0015] Figure 2 This is a flowchart illustrating the drone swarm path optimization method. Detailed Implementation

[0016] The technical solutions of the present invention will now be clearly and completely described with reference to the accompanying drawings in the embodiments of the present invention.

[0017] Please see Figures 1 to 2 The first embodiment of the present invention provides a path optimization method for UAV swarms based on heuristic stochastic ensemble reinforcement learning for cooperative navigation of multiple UAVs in dynamic environments, which is used to achieve efficient path planning, obstacle avoidance and network interference handling of UAV swarms.

[0018] The system architecture of this UAV swarm path optimization method improves the adaptability of UAVs in dynamic environments and the efficiency of collaborative task execution by separating the training and inference ends and combining three core modules: Heuristic Guidance, Safety Deep Reinforcement Learning with REDQ (S-REDQ), and Dynamic Sparrow Search Algorithm (D-SSA).

[0019] The system architecture comprises a server (training end) and multiple clients (inference ends). The server is responsible for offline training, policy optimization, and version updates of the deep reinforcement learning model, and adjusts the policy based on real-time feedback. Clients are deployed on each UAV and are responsible for real-time environmental perception, path decision-making, safety verification, and task execution. The overall system operation is divided into three phases: heuristic guidance, a stochastic ensemble dual-Q learning strategy with safety constraints, and a sparrow algorithm recovery phase. Heuristic methods provide initial path planning and obstacle avoidance strategies, the S-REDQ module optimizes path decisions and safety constraints, and the dynamic UAV sparrow algorithm module handles network interference or connection loss recovery.

[0020] Through the above process, this invention achieves closed-loop control of path planning, obstacle avoidance, task execution, and network recovery, ensuring that the system efficiently completes multi-UAV cooperative navigation tasks in dynamic and unstable environments. The system not only optimizes the efficiency of path planning, obstacle avoidance, and task execution, but also effectively improves the stability and safety of swarm flight, especially demonstrating excellent performance in resisting network interference and recovering lost UAVs.

[0021] The UAV swarm path optimization method based on heuristic stochastic ensemble reinforcement learning in this embodiment can be executed by the UAVs. Specifically, it is executed by one or more processors in the UAVs to implement steps S1 to S5.

[0022] S1. Obtain the current status information of the drone and determine whether the communication is normal.

[0023] S2. When communication is normal, generate heuristic control actions based on the current state information. The heuristic control actions include at least angular velocity control quantities and linear velocity control quantities.

[0024] The Heuristic Guidance Module provides UAVs with initial path planning, obstacle avoidance, and swarm coordination strategies. Its core idea is to guide UAVs toward targets and avoid obstacles using rules customized based on UAV kinematic models and flight safety constraints, while ensuring coordinated flight with other UAVs. The module mainly comprises three strategies: target orientation, obstacle avoidance behavior, and swarm coordination, achieving efficient navigation in multi-UAV collaborative missions.

[0025] S21. Calculate the target guidance component based on the deviation between the current position of the UAV and the position of the mission target.

[0026] The calculation formula for the target guidance component is as follows.

[0027] .

[0028] .

[0029] In the formula The desired angle between the target location and the current location of the drone. ( () represents a function that calculates spatial orientation angles from two three-dimensional coordinates. ( ) represents the three-dimensional coordinates of the target point. () represents the current three-dimensional coordinates of the drone. Target-oriented component. This represents the current angular velocity of the drone. This is the normalization coefficient.

[0030] S21 implements a target-oriented strategy. This strategy guides the UAV towards the target by calculating the angular deviation between the UAV's current position and the target position. Specifically, the strategy generates a target-oriented force based on the difference between the desired angle and the UAV's current angular velocity.

[0031] S22. Calculate the obstacle avoidance steering amount based on the distance and relative direction between the UAV and the obstacle.

[0032] First, calculate the distance from the drone to each obstacle.

[0033] .

[0034] Then, the directional angle between the drone and the obstacle is calculated.

[0035] .

[0036] Finally, based on the relationship between the distance and direction of the obstacle, the obstacle avoidance steering amount is generated.

[0037] .

[0038] In the formula The distance between the drone and the obstacle is the three-dimensional Euclidean distance. The coordinates of the obstacle are 3D. This represents the Euclidean norm, which is the Euclidean distance obtained by taking the square root of the sum of the squares of a three-dimensional vector. The obstacle orientation angle is used to characterize the spatial orientation of the obstacle relative to the drone. This refers to the obstacle avoidance steering input (the angular velocity adjustment component generated by the obstacle avoidance strategy). This represents the current angular velocity of the drone.

[0039] The S22 implements an obstacle avoidance strategy. The obstacle avoidance behavior determines whether to avoid obstacles by detecting the distance between the drone and surrounding obstacles.

[0040] S23. Based on the average angular velocity and average linear velocity of the UAV swarm, calculate the cooperative components of angular velocity and linear velocity.

[0041] First, calculate the average angular velocity and average linear velocity of the drone swarm.

[0042] .

[0043] .

[0044] Then, based on the difference between the current angular velocity of the UAV and the average angular velocity of the group, a cooperative angular velocity component is generated.

[0045] .

[0046] Simultaneously, a linear velocity cooperative component is generated based on the difference between the current linear velocity of the drone and the average linear velocity of the group.

[0047] .

[0048] In the formula The mean angular velocity. This represents the average linear velocity. The number of drones participating in the collaboration. and The first The angular velocity and linear velocity of the drone. Co-component of angular velocity. This is the linear velocity co-component. This represents the maximum linear velocity of the drone.

[0049] S23 implements a swarm coordination strategy. This strategy calculates the average angular velocity and average linear velocity of the drone and other members of the swarm, enabling the drone to adjust its motion to maintain formation.

[0050] S24. The target guidance component, obstacle avoidance steering amount and angular velocity cooperative component are superimposed and the interval is clipped to obtain the final angular velocity control amount.

[0051] .

[0052] .

[0053] S25. Perform interval trimming on the linear velocity cooperative component to obtain the final linear velocity control quantity.

[0054] .

[0055] .

[0056] In the formula This represents the angular velocity before it was restricted. This is the final angular velocity control value for the drone. This represents the linear velocity before it is restricted. This is the final linear velocity control value for the drone. This represents a range clipping function, used to restrict input values ​​to a given range. Inside.

[0057] In this embodiment, the final UAV control quantity is obtained by combining the steering component generated by the target-oriented strategy, the steering component generated by the obstacle avoidance strategy, and the cooperative component generated by the swarm cooperation strategy. First, the angular velocity and linear velocity control quantities before limitations are calculated. Then, to ensure the control quantities are within the allowable range, the angular velocity and linear velocity are pruned to obtain the final control output: The heuristic guidance module of this invention provides basic path planning and obstacle avoidance strategies for UAVs through a rule system customized based on UAV kinematic models and flight safety constraints. Combining three major strategies—target orientation, obstacle avoidance behavior, and group coordination—it ensures that UAVs can achieve autonomous flight in various environments. This module's strategies are simple yet efficient, suitable for initial navigation tasks of UAVs, and can be combined with other advanced algorithms to improve overall flight performance.

[0058] This embodiment does not use readily available heuristics, but instead designs a set of rules specifically for the continuous control problem of unmanned swarms. Unlike existing technologies that treat heuristics as general heuristic costs or offline rule bases, the heuristic guidance module in this invention does not simply call existing heuristic algorithms, but rather combines multi-UAV cooperative path planning and obstacle avoidance tasks, and is a set of continuous control rules specifically customized for UAV dynamic constraints and swarm cooperation requirements.

[0059] Specifically, this invention directly defines the action space of the UAV as a binary continuous control quantity of "angular velocity-linear velocity", and constructs heuristic rules around three types of task requirements (rules): target orientation, local obstacle avoidance, and group coordination.

[0060] The target-oriented rule calculates the desired yaw angle and generates angular velocity control based on the relative geometric relationship between the UAV's current position and the target point, guiding the UAV to converge to the mission target along the optimal direction.

[0061] The obstacle avoidance rule, based on the spatial distribution and safe distance constraints of static obstacles, dynamic obstacles and interference sources, directly provides the angular velocity correction amount that meets the safety constraints, thereby achieving local obstacle avoidance and anti-interference.

[0062] The group coordination rule constructs a formation maintenance term based on the group's average angular velocity and average linear velocity, so that the angular velocity and linear velocity of each UAV converge toward group consistency within a safe range.

[0063] By weighting and combining the above three types of rules, the heuristic module of this invention directly outputs the final control values ​​of angular velocity and linear velocity that conform to the UAV kinematic model and flight safety constraints. Compared with traditional heuristic methods that only work on discrete path grids or scalar heuristic costs, this invention realizes a task-customized heuristic for continuous control space, providing structured priors and high-quality initial strategies for subsequent stochastic ensemble reinforcement learning.

[0064] S3. By using a random ensemble double-Q learning strategy with safety constraints, combined with safety constraints and random ensemble of multiple Critic networks, the heuristic control action is optimized, and candidate control actions are output.

[0065] The core idea of ​​the Stochastic Ensemble Double Q Learning Strategy Module with Safety Constraints (S-REDQ) is to combine safe action decision-making with Stochastic Ensemble Double Q Learning (REDQ). By integrating multiple Critic networks, it optimizes the path planning, obstacle avoidance, and collaborative task execution capabilities of UAVs in complex environments. This module calculates the target Q-value by randomly sampling multiple Critic networks, thereby reducing training instability caused by overfitting of a single Critic network. Safe action decision-making is a crucial part of the module, ensuring that the UAV maintains a safe distance from other UAVs, obstacles, and network interference sources during flight, avoiding collisions and communication interruptions, thus guaranteeing the smooth execution of flight missions.

[0066] The formula for updating the Q value is as follows: .

[0067] In the formula For the first A Critic network for state-action pairs The Q value. The current state is the environmental observation information received by the S-REDQ module at the current time step, including: the UAV's three-dimensional position, current linear velocity and angular velocity, relative distance and relative velocity of neighboring UAVs, obstacle positions, interference source positions and mission target positions, used to characterize the current multi-UAV cooperative navigation environment. The current action is the state. The next step is for the policy network (Actor) to determine the policy based on the current policy. The given "pair of continuous control variables" represents the turning angular velocity and flight linear velocity of the UAV in the next control cycle, which serve as the actual control commands executed. The reward signals returned by the environment after an action is performed include target approach reward, formation consistency reward, obstacle avoidance safety reward, and interference avoidance penalty, which are used to characterize the quality of the current action. This is the decay factor for long-term cumulative returns, used to balance immediate rewards with long-term decision benefits. The next state is for the drone to perform the current action. The next moment's environmental state is then transferred, including updated 3D position, velocity, distance to neighboring machines, and distance to obstacles and interference sources. The next action is in the state. Next, based on the current strategy The selected "next moment UAV continuous control quantity pair" is used to calculate the target Q value. Indicates the state According to the strategy Generate Actions And calculate the mathematical expectation of the Critic network output value.

[0068] Based on the updated Q value, random sampling and Critic integration are performed.

[0069] .

[0070] In the formula This is the integrated result used to calculate the target Q value. This represents a randomly selected set of Critic networks. This represents the number of Critics sampled.

[0071] Specifically, constructing includes A set of Critic networks, from which random samples are taken when calculating the target Q-value. A Critic network. Calculate the next state. The following action In the The average Q-value in each Critic network is used as the target Q-value. .

[0072] The update process of the Actor network is as follows.

[0073] .

[0074] In the formula It is the loss function of the Actor network. This represents the number of Critic networks. This is the index for the Critic network. It is the first The Q-value of the action for each Critic network. This represents the action generated by the Actor network.

[0075] Specifically, the S-REDQ Actor network optimizes the policy by maximizing the average Q-value. The Actor's goal is to select actions that maximize the Q-value, thereby guiding the drone to make efficient decisions in complex environments.

[0076] The target network update process is as follows.

[0077] .

[0078] In the formula These are the parameters of the target network. It is a soft update coefficient. These are the parameters of the current network.

[0079] In this embodiment, S-REDQ employs a soft update mechanism to update the parameters of the target network to ensure the stability of the learning process.

[0080] The safety constraints include rewards for avoiding nearby drones, rewards for avoiding obstacles, and rewards for avoiding interference sources.

[0081] The reward for avoiding nearby drones is as follows.

[0082] Specifically, when the three-dimensional distance between a drone and other drones is less than a preset safety threshold... When this happens, an avoidance behavior will be triggered to prevent a collision.

[0083] Specifically, if drones With neighboring drones The three-dimensional Euclidean distance between them satisfies the following expression.

[0084] .

[0085] To avoid collisions, the following avoidance bonuses are provided.

[0086] .

[0087] In the formula Rewards for avoiding nearby drones. The reward weight for proximity avoidance. For drones Current 3D position coordinates. For nearby drones Current 3D position coordinates. This is the safe distance threshold between drones.

[0088] The obstacle avoidance reward is defined as follows.

[0089] Specifically, to avoid collisions with obstacles, when the three-dimensional distance between the drone and the obstacle is less than a preset safety threshold... When this occurs, obstacle avoidance behavior will be triggered. The specific calculation formula is as follows.

[0090] .

[0091] In the formula Rewards for obstacle avoidance. The weighting of obstacle avoidance rewards. This is the safe distance threshold between the drone and the obstacle. The coordinates are the three-dimensional position coordinates of the obstacle.

[0092] The interference source avoidance reward is defined as follows.

[0093] Specifically, when facing network interference sources, drones need to adjust their flight paths to maintain a sufficient distance to ensure flight stability and communication reliability. When the three-dimensional distance between the drone and the interference source is less than a preset threshold... or At this time, a decision is made to avoid the source of interference. The specific formula is as follows.

[0094] .

[0095] In the formula Rewards for avoiding interference sources. These are the three-dimensional position coordinates of the interference source. This is the first safe distance threshold. This is the second safe distance threshold. for The weight. for The weight. .

[0096] The standard REDQ module optimizes the path planning and obstacle avoidance capabilities of UAVs in complex and dynamic environments by randomly ensembled Critic networks. The stability and efficiency of the training process are ensured through multi-Critic ensemble and random sampling strategies.

[0097] Compared to the standard REDQ algorithm, this invention makes the following adaptive improvements to REDQ for scenarios involving continuous control of multiple UAVs and safety constraints.

[0098] First, the design unifies the action space with heuristic control. In standard REDQ, the action space exists only as an abstract, continuous control variable. This invention explicitly defines the S-REDQ action vector as the UAV's "angular velocity-linear velocity" control pair, i.e. The final control value output by the heuristic guidance module maintains the same action space and the same physical meaning, enabling the strategy learned by REDQ to seamlessly replace or modify heuristic control at the continuous control level.

[0099] Secondly, this invention introduces Safety-REDQ (Stochastic Ensemble Q-learning with Safety Constraints). When calculating the Q-value objective and policy gradient, this invention explicitly embeds safety constraints such as neighbor distance, obstacle distance, and interference source distance into the reward function and action feasible region. Actions that violate these safety constraints are penalized or eliminated during Q-estimation and policy update. Unlike standard REDQ, which only optimizes the expected cumulative reward, this invention's S-REDQ achieves stochastic ensemble reinforcement learning under safety constraints through a joint design of "multi-criteria ensemble + safety constraints," significantly reducing the risk of collisions and disconnection.

[0100] Finally, heuristic guidance is combined with experience sampling and policy improvement. This invention introduces a heuristic guidance mechanism into the experience sampling and policy update process of REDQ. On the one hand, in the early stages of training, angular velocity and linear velocity controls generated by the heuristic module are adopted as actual actions with a certain probability, and the resulting interaction samples are added to the experience replay pool to improve sample quality and initial exploration efficiency. On the other hand, in the policy update of the Actor network, heuristic actions are used as a reference, and regularization terms or action constraints oriented towards heuristic control are introduced, so that the learned stochastic policy maintains compliance with task priors while fully exploring.

[0101] Through the above improvements, the S-REDQ module of this invention is no longer a general REDQ independent of the heuristic module, but a stochastic ensemble reinforcement learning algorithm with safety constraints and heuristic guidance for UAV swarm path optimization tasks.

[0102] In this invention, heuristic algorithms and S-REDQ are coordinated through a "hybrid strategy + unified learning" approach. During the action selection phase of the UAV swarm, a hybrid strategy is employed in the first few training rounds (e.g., in a total of 40,000 training rounds, the first 3,000 rounds). The action selection function invokes heuristic rules with approximately 30% probability, generating the angular velocity and velocity control action for the current step based on the superposition of three types of physical rules: target orientation, obstacle avoidance, and swarm coordination. In other cases, the action is output by the S-REDQ-based policy network, with a small amount of exploration noise superimposed on it.

[0103] Regardless of whether the action originates from heuristic rules or the S-REDQ policy network, it will be uniformly stored in the experience replay buffer in the form of experience tuples. Subsequently, the S-REDQ multi-Q network set and the corresponding Actor will perform value evaluation and policy update from this batch of mixed data distribution of "heuristic guidance + policy exploration". Through mechanisms such as random subset sampling and minimizing the target Q, the overestimation of value is suppressed and the learning stability is improved.

[0104] The heuristic component does not participate in parameter optimization, but only provides physically reasonable and task-related high-quality trajectories at the behavioral level. This helps drone swarms avoid meaningless blind exploration in the early stages, ensuring that the experience pool is rich in positive samples from the beginning that demonstrate "ability to reach the target, avoid obstacles, and maintain group coordination."

[0105] S-REDQ builds upon this by abstracting and re-balancing these rule-based behaviors through long-term reward discounting and value gradient-based policy optimization, gradually learning a continuous control policy that is superior to the original heuristic.

[0106] As training progresses, the probability of heuristic use and exploration noise gradually decrease, and the control smoothly transitions from "rule-driven, reinforcement learning-assisted" to "S-REDQ-driven, heuristics relegated to prior constraints". This accelerates convergence and improves the performance and robustness of the final policy while ensuring training stability, making it suitable for the multi-agent cooperative control scenario described in the patent.

[0107] S4. When communication is abnormal or lost, candidate control actions are output based on the current status information and through the improved dynamic UAV sparrow algorithm.

[0108] To address the issue of "lost drones being unable to continue executing the main strategy (S-REDQ)" due to communication interruptions and link interference in multi-UAV collaborative missions, this invention deeply modifies the traditional Sparrow Search Algorithm (SSA) to form a Dynamic UAV Sparrow Algorithm (D-SSA) suitable for continuous UAV control, dynamic obstacle avoidance, and formation collaboration. The improved D-SSA acts as an independent local path planning module for lost drones, generating angular velocity-linear velocity control variables in real time when communication is unavailable, enabling the drone to maintain safe and consistent flight behavior, and automatically switching back to the main strategy once communication is restored.

[0109] The improved Dynamic Unmanned Aerial Vehicle Sparrow Algorithm Module (D-SSA) process is as follows.

[0110] S41. Encode individual sparrows as pairs of angular velocity and linear velocity control quantities for the UAV.

[0111] .

[0112] In the formula, For the first The control vector for each individual sparrow. For the first The angular velocity control quantity of the drone at the next moment corresponding to each individual sparrow. For the first The linear velocity control quantity of the drone at the next moment corresponding to each individual sparrow.

[0113] Specifically, sparrow individuals are re-encoded into executable control quantities to achieve motion space adaptation for D-SSA. Traditional SSA represents individuals as position information or scalar parameters, which cannot be directly used for UAV control. This invention reconstructs the sparrow individual encoding into pairs of angular and linear velocity control quantities for the UAV.

[0114] This coding method is strictly consistent with the dynamic constraints of the UAV and shares the same action space with the heuristic module and the S-REDQ module, enabling the three to work together seamlessly.

[0115] S42. Calculate the comprehensive fitness function To evaluate individuals.

[0116] .

[0117] In the formula This is the term that approaches the target. for The weight. This is a formation consistency item. for The weight. For obstacle avoidance safety items. for The weight. This is a penalty item for interference. for The weight. This refers to the smoothness of motion. for The weight.

[0118] Specifically, the multi-objective fitness function designed in this embodiment is applicable to UAV navigation, formation following, obstacle avoidance, and anti-interference. To enable a lost UAV to maintain target approach, formation following, obstacle avoidance, and interference source avoidance even without communication, while maintaining stable operation, this invention constructs the fitness as a comprehensive evaluation of five parts.

[0119] Target approximation term Used to measure whether control actions cause the drone to move toward the target.

[0120] .

[0121] In the formula This refers to the three-dimensional position of the UAV at the next moment after executing the current control parameters. This represents the three-dimensional position of the target point.

[0122] Formation Consistency Item Used to maintain formation stability and reduce deviations from the group center position and average group speed.

[0123] .

[0124] In the formula This represents the geometric center of the current group. This represents the speed of the drone in the next moment. The average velocity of the group. This represents the Euclidean norm.

[0125] Obstacle avoidance safety items Used to punish drones for approaching obstacles.

[0126] .

[0127] In the formula Indicates all obstacles Perform summation. This indicates taking the larger value. This is the safe distance threshold between the drone and the obstacle. A three-dimensional position vector representing an obstacle, used to calculate the spatial distance between the drone and the obstacle.

[0128] Interference penalty item This enables D-SSA to automatically avoid sources of radio interference, improving system robustness.

[0129] .

[0130] .

[0131] In the formula This indicates all sources of interference in the environment. Perform summation. This is the first safe distance threshold. This is the second safe distance threshold. for The weight. for The weight. This is an indicator function. The distance is Euclidean. This indicates the location of the interference source.

[0132] motion stability item Limit the range of change in control parameters to ensure continuous and stable flight maneuvers.

[0133] .

[0134] In the formula For the first The angular velocity control quantity of the drone at the previous moment corresponding to each individual sparrow. For the first The linear velocity control quantity of the drone corresponding to each individual sparrow at the previous time step. For the first The angular velocity control quantity of the drone at the next moment corresponding to each individual sparrow. For the first The linear velocity control quantity of the drone at the next moment corresponding to each individual sparrow.

[0135] S43. After each iteration, the control variables generated by individual sparrows are projected into the safe and feasible region to project the control vectors generated by individual sparrows into the safe and feasible region of UAV dynamics.

[0136] .

[0137] In the formula This indicates the operation of projecting control variables onto the feasible region. The safe and feasible domain for drone control variables. This represents the maximum permissible angular velocity. This represents the minimum linear velocity. This represents the maximum linear velocity. This represents the interval clipping function.

[0138] .

[0139] In the formula The safe and feasible domain for drone control variables.

[0140] To ensure that the control output of D-SSA meets the dynamic boundaries and safety constraints of the UAV, the feasible control region is first defined. Then, after each iteration, the control variables generated by the individual sparrows are projected onto the feasible region. Internally, projecting the safe and feasible domain ensures that actions are executable and safe, guaranteeing that the control variables generated by D-SSA can be safely executed on the actual drone controller.

[0141] S44, the improved dynamic drone sparrow algorithm also includes a dynamic environment update mechanism.

[0142] The dynamic environment update mechanism is the core modification of the dynamic UAV sparrow algorithm in this invention. The UAV flight environment is highly dynamic; therefore, this invention incorporates three types of dynamic updates within the standard SSA framework.

[0143] S441, Partial population dynamic reinitialization, used to quickly adapt to environmental mutations.

[0144] Let the first The optimal control vector for the sparrow individual in the next iteration is For the set of individuals that need to be reinitialized Each sparrow in Perform dynamic perturbation updates.

[0145] .

[0146] In the formula For the first Individual sparrows are iterating. The control vector. For iteration The globally optimal control vector. In the interval The perturbation vector of random sampling. It indicates a uniform distribution. This represents the magnitude of the disturbance.

[0147] S442, Obstacle orientation sensitive update, which enables the Sparrow Algorithm to deflect the control amount in a safe direction when approaching an obstacle, thereby reducing the risk of collision.

[0148] In the formula This is the obstacle orientation adjustment coefficient. According to The calculated obstacle avoidance direction guidance vector. This represents the current three-dimensional position of the drone.

[0149] Specifically, to avoid potential collisions when a drone is about to approach an obstacle, this invention utilizes a unit vector of the safe direction. Direction-sensitive adjustments are made to the control vector of individual sparrows.

[0150] S443, Dynamics Constraint Update (Limiting the Rate of Change of Motion), is used to ensure that the rate of change of the output control quantity does not exceed the dynamic tolerance range of the UAV.

[0151] .

[0152] .

[0153] In the formula For the first Individual sparrows are iterating. The increment of angular velocity. For the first Individual sparrows are iterating. The linear velocity increment. This is the maximum permissible angular acceleration. This is the maximum permissible linear acceleration. This represents the interval clipping function.

[0154] This step ensures that the changes in the UAV's control variables satisfy the constraints of maximum angular acceleration and maximum linear acceleration, and limits the increment of the control variables obtained by the sparrow individual in the current iteration: The Dynamic Unmanned Aerial Vehicle Sparrow Search Algorithm (D-SSA) of this invention, based on fitness evaluation and action space modification, retains the three roles of "explorer," "joiner," and "scouter" from the traditional Sparrow Search Algorithm (SSA) to achieve a balance between global search and local correction. The explorer (leading the search): selects high-quality actions based on the comprehensive fitness function and performs a global evaluation of the surrounding environment (group information, obstacles, interference sources). The joiner (locally following): performs a local search under the guidance of the explorer, achieving fine-tuning of actions. The scouter (risk monitoring): monitors whether the current control action will enter an obstacle or interference zone and triggers safety correction or reinitialization. Through the combined action of these three mechanisms, D-SSA can search for UAV control actions that combine safety and efficiency in real time in complex environments.

[0155] Specifically, to ensure the overall consistency and stability of the system, this invention designs the following collaborative strategy: When UAV communication is normal: execute the S-REDQ main strategy. When the UAV briefly loses contact: automatically switch to actions generated by D-SSA. Individuals in D-SSA can use heuristic actions or the most recent REDQ action as the initial population to improve convergence speed. The trajectories generated by D-SSA can be replayed into the S-REDQ experience pool to improve the generalization of the strategy in extreme scenarios. All three modules operate in a unified action space (angular velocity – linear velocity).

[0156] To enable the sparrow search algorithm to adapt to continuous control, dynamic obstacle avoidance, and formation keeping tasks of UAV swarms, this invention makes the following improvements: **Action Space Adaptation:** Individual sparrows are re-encoded into continuous "angular velocity-linear velocity" control quantities executable by the UAV, allowing the algorithm to search within the real flight action space. **Multi-Target Fitness Design:** A comprehensive fitness function is constructed that integrates target approach, formation consistency, obstacle avoidance distance, interference penalty, and action stability, ensuring that the search results simultaneously meet navigation, coordination, and safety requirements. **Dynamic Environment Update Mechanism:** Mechanisms such as partial population dynamic reinitialization, obstacle direction-sensitive updates, and dynamic constraint updates are designed to address the dynamic changes in the UAV environment, enabling the algorithm to respond to environmental changes in real time. **Safe and Feasible Domain Projection:** Constraints such as the aircraft's dynamic limitations, maximum turning speed, and minimum safe distance are embedded into the sparrow search. Feasible domain projection is performed on the population after each iteration to ensure that the output actions are always safe and executable.

[0157] Collaboration mechanism with heuristics and S-REDQ: Lost drones are independently decided by the sparrow algorithm, but their population can be initialized using heuristic actions or S-REDQ actions, and automatically switch back to the S-REDQ strategy after communication is restored. The trajectory data generated by the sparrow algorithm can also be added to the experience replay pool to improve the training stability of S-REDQ in extreme scenarios.

[0158] In summary, the Dynamic Unmanned Aerial Vehicle Sparrow Algorithm (D-SSA) proposed in this invention completely transforms the traditional SSA from "static parameter optimization" into a "dynamic local path planner that can be directly used for UAV control" through action space modification, multi-target fitness, dynamic environment update, and safe feasible domain projection. It can effectively take over the UAV in the event of communication failure and achieve stable and safe flight control.

[0159] S5. Control the drone's flight based on the candidate control actions.

[0160] This invention proposes an innovative solution for multi-UAV cooperative navigation and safety constraints in dynamic environments through the synergistic effect of "heuristic guidance + a randomized ensemble double-Q learning strategy with safety constraints + a dynamic UAV sparrow algorithm". This method combines path planning, obstacle avoidance, network interference recovery, and swarm coordination, and can effectively improve the stability of swarm flight and mission execution efficiency, especially in the face of dynamic environmental changes, network interference, and UAV disconnection.

[0161] This invention optimizes UAV swarm behavior using a distributed adaptive mechanism without relying on traditional centralized control, significantly reducing computational and communication burdens and improving real-time performance and flexibility. In particular, the heuristic guidance strategy provides UAVs with a rapid initial navigation scheme, the S-REDQ module optimizes path decisions through deep reinforcement learning, and safety constraints ensure flight safety and efficiency. The dynamic UAV sparrow algorithm provides an effective solution to network interference and disconnection recovery problems, ensuring coordinated flight of the swarm in complex environments.

[0162] In summary, the technical solution of this invention not only improves the adaptability and decision-making efficiency of UAVs in dynamic environments, but also effectively solves the bottleneck problems in path planning, obstacle avoidance, and network interference recovery in existing technologies. Through an innovative integration method, it achieves a highly efficient and stable multi-UAV cooperative navigation system, possessing strong engineering application value and broad prospects for promotion.

[0163] Obviously, the embodiments described above are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0164] In the several embodiments provided in this invention, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus and method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0165] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0166] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks. It should be noted that in the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0167] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention are also intended to include the plural forms unless the context clearly indicates otherwise.

[0168] It should be understood that the term "and / or" used in this invention is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Furthermore, the character " / " in this invention generally indicates that the preceding and following related objects have an "or" relationship.

[0169] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0170] The terms "first" and "second" used in the embodiments are merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" can be interchanged in a specific order or sequence where permitted. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0171] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for optimizing paths of a UAV swarm based on heuristic random ensemble reinforcement learning, characterized in that, Include: S1. Obtain the current status information of the drone and determine whether the communication is normal; S2. When communication is normal, generate a heuristic control action based on the current state information; the heuristic control action includes at least angular velocity control quantity and linear velocity control quantity; S3. By using a random ensemble double-Q learning strategy with safety constraints, combined with safety constraints and random ensemble of multi-Critic networks, the heuristic control action is optimized and candidate control actions are output. S4. When communication is abnormal or lost, candidate control actions are output based on the current status information and through the improved dynamic UAV sparrow algorithm. S5. Control the drone's flight based on the candidate control actions. 2.The method of claim 1, wherein, The specific operation of S2 is as follows; Calculate the target guidance component based on the deviation between the current position of the UAV and the position of the mission target; Calculate the obstacle avoidance steering amount based on the distance and relative direction between the drone and the obstacle; Based on the average angular velocity and average linear velocity of the UAV swarm, calculate the cooperative components of angular velocity and linear velocity. The target guidance component, obstacle avoidance steering amount and angular velocity cooperative component are superimposed and the interval is clipped to obtain the final angular velocity control amount; The linear velocity cooperative component is truncated to obtain the final linear velocity control quantity. 3.The method of claim 2, wherein, The calculation formula for the target guidance component is as follows; ； ； wherein is the desired angle between the target position and the current position of the UAV; ( ) is a function that calculates the spatial direction angle from two three-dimensional coordinates; is the three-dimensional coordinate of the target point; is the current three-dimensional coordinate of the UAV; is the target steering component; is the current angular velocity of the UAV; is a normalization coefficient; The formula for calculating the obstacle avoidance steering amount is as follows; ； ； ； In the formula is the three-dimensional Euclidean distance between the UAV and the obstacle; is the three-dimensional coordinates of the obstacle; represents the Euclidean norm; is the direction angle of the obstacle; is the obstacle avoidance steering vector; is the current angular velocity of the UAV; The average angular velocity is: ; The average linear velocity is: ; The angular velocity cooperative component is: ; The linear velocity cooperative component is: ; wherein is the average angular velocity; is the average linear velocity; is the number of drones participating in the coordination; and are the angular and linear velocities of the i-th drone, respectively; are the angular and linear velocities of the i-th drone, respectively; is the angular coordination component; is the linear coordination component; is the maximum linear velocity of the drone; The formula for calculating the final angular velocity control quantity is as follows; ； ； The formula for calculating the final linear velocity control quantity is as follows; ； ； wherein is the angular velocity before limitation; is the final angular velocity control amount of the UAV; is the linear velocity before limitation; is the final linear velocity control amount of the UAV; denotes an interval clipping function. 4.The method of claim 1, wherein, The Q-value update formula for training a randomized ensemble double-Q learning policy with safety constraints is as follows; ； where is the th Critic network output value for the state-action pair ; is the current state; is the current action; is the reward signal returned by the environment after executing the action; is the decay factor for long-term cumulative rewards; is the next state; is the next action; denotes the mathematical expectation of the Critic network output value for the action generated according to the policy in the state Based on the updated Q value, random sampling and Critic integration are performed; ； wherein is the integrated result for computing the target Q value; represents a randomly selected set of Critic networks; is the number of sampled Critics; The update process of the Actor network is as follows; ； wherein is the loss function of the Actor network; is the number of Critic networks; is the index of the Critic network; is the Q value of the action by the th Critic network; denotes the action generated by the Actor network; The target network parameters are updated using a soft update mechanism; ； wherein are parameters of the target network; is a soft update coefficient; are parameters of the current network.

5. The method of claim 1, wherein, The safety constraints include rewards for avoiding nearby drones, rewards for avoiding obstacles, and rewards for avoiding interference sources. The adjacent drone avoidance reward As follows; ； wherein is a reward weight for proximity avoidance; is a drone is a current three-dimensional position coordinate; is a proximity drone is a current three-dimensional position coordinate; is a safety distance threshold between drones; The obstacle avoidance reward Is defined as follows; ； In the formula is an obstacle avoidance reward weight; is a safety distance threshold between the UAV and the obstacle; is a three-dimensional position coordinate of the obstacle; The interference source avoidance reward Is defined as follows; ； wherein is a three-dimensional position coordinate of the source of interference; is a first safety distance threshold value; is a second safety distance threshold value; is a weight of is a weight of .​​ 6. The heuristic random integrated reinforcement learning based path optimization method for UAV swarm according to any one of claims 1 to 5, characterized in that, The improved dynamic drone sparrow algorithm is as follows; Individual sparrows are encoded as pairs of angular and linear velocity control variables for the UAV; ； In the formula, For the first The control vector for each individual sparrow; For the first The angular velocity control quantity of the drone at the next moment corresponding to each individual sparrow; For the first The linear velocity control quantity of the drone at the next moment corresponding to each individual sparrow; by calculating a combined fitness function evaluating the individual; ； In the formula For the term approaching the target; for The weights; For formation consistency; for The weights; For obstacle avoidance safety items; for The weights; This is a penalty item for interference; for The weights; For the smoothness of motion; for The weights; After each iteration, the control variables generated by individual sparrows are projected into the safe and feasible region to project the control vectors generated by individual sparrows into the safe and feasible region of UAV dynamics. ； In the formula This represents the operation of projecting control variables onto the feasible region; The safe and feasible domain for drone control variables; The maximum permissible angular velocity; This is the minimum linear velocity; This represents the maximum linear velocity. This represents the interval clipping function.

7. The method for unmanned aerial vehicle swarm path optimization based on heuristic stochastic ensemble reinforcement learning according to claim 6, characterized in that, Target approximation term as follows; ； In the formula The three-dimensional position of the UAV at the next moment after executing the current control input; The three-dimensional position of the target point; Formation Consistency Item as follows; ； In the formula This represents the geometric center of the current group. The speed of the drone in the next moment; The group average speed; Represents the Euclidean norm; Obstacle avoidance safety items as follows; ； In the formula Indicates all obstacles Perform summation; This indicates taking the larger value; The safe distance threshold between the drone and the obstacle; Represents the three-dimensional position vector of the obstacle; Interference penalty item as follows; ； In the formula This indicates all sources of interference in the environment. Perform summation; This is the first safe distance threshold; This is the second safe distance threshold; for The weights; for The weights; For indicator functions; The distance is Euclidean. motion stability item as follows; ； In the formula For the first The angular velocity control quantity of the drone at the previous moment corresponding to each individual sparrow; For the first The linear velocity control quantity of the drone corresponding to each individual sparrow at the previous time step.

8. A method for optimizing the path of an unmanned aerial vehicle swarm based on heuristic stochastic ensemble reinforcement learning according to any one of claims 1 to 5, characterized in that, The improved dynamic drone sparrow algorithm also includes a dynamic environment update mechanism; Partial population dynamics reinitialization; ； In the formula For the first Individual sparrows are iterating. The control vector; For iteration The globally optimal control vector; In the interval The perturbation vector sampled from the upper random sample; Indicates a uniform distribution; The magnitude of the disturbance; Obstacle orientation-sensitive updates; In the formula This is the obstacle orientation adjustment coefficient; According to The calculated obstacle avoidance direction guidance vector; This represents the current three-dimensional position of the drone; Dynamic constraint update; ； ； In the formula For the first Individual sparrows are iterating. The increment of angular velocity; For the first Individual sparrows are iterating. The linear velocity increment; This is the maximum permissible angular acceleration; This is the maximum permissible linear acceleration; This represents the interval clipping function.

9. The method for unmanned aerial vehicle swarm path optimization based on heuristic stochastic ensemble reinforcement learning according to claim 1, characterized in that, The current status information includes the position and speed of each drone in the drone swarm, information on nearby drones, obstacle information, interference source information, and mission target information.

Images