An unmanned aerial vehicle formation path planning method based on RND-SAC algorithm
By introducing the RND-SAC algorithm and combining intrinsic and extrinsic rewards, and designing state space and action space functions, the problem of low exploration efficiency in UAV formation path planning is solved, and efficient path planning and improved collaborative performance of UAVs in complex environments are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ANHUI POLYTECHNIC UNIV MECHANICAL & ELECTRICAL COLLEGE
- Filing Date
- 2026-04-29
- Publication Date
- 2026-07-21
Smart Images
Figure CN122429818A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of deep learning and UAV formation technology, specifically to a UAV formation path planning method based on the RND-SAC algorithm. Background Technology
[0002] In recent years, with the continuous development and progress of sensor technology, wireless communication technology, and intelligent control technology, unmanned aerial vehicle (UAV) technology has also developed rapidly. Due to their low cost, flexible operation, and lack of scene restrictions, UAVs have been widely used in various fields. Compared with single UAVs, multiple UAVs can form a formation to perform missions, effectively overcoming the performance deficiencies of single UAVs. When performing missions such as battlefield support, reconnaissance, and coordinated attacks, the formation needs to maintain stability and coordination in complex environments.
[0003] Reinforcement learning is an experience-based machine learning method that continuously improves system performance. Essentially, it learns to make optimal decisions in specific situations through continuous interaction and trial and error. In multi-UAV decision-making scenarios, reinforcement learning helps UAVs achieve optimal decisions in complex environments, thereby achieving predetermined mission objectives. Combining reinforcement learning with deep neural networks has led to deep reinforcement learning methods. This innovative fusion endows it with exceptional capabilities, efficiently handling high-dimensional state spaces composed of complex factors such as the positions of multiple UAVs.
[0004] Therefore, this invention proposes a UAV formation path planning method based on the RND-SAC algorithm, aiming to realize path reliability planning for multi-UAV formations and provide theoretical support for the path planning problem of swarm UAVs. Summary of the Invention
[0005] The purpose of this invention is to provide a UAV formation path planning method based on the RND-SAC algorithm. This method introduces a Random Network Distillation (RND) module, combining intrinsic and extrinsic rewards to address the problem of low exploration efficiency of reinforcement learning algorithms in unknown environments. This significantly improves the algorithm's exploration efficiency and reduces its convergence difficulty.
[0006] To achieve the above functions, this invention designs a UAV formation path planning method based on the RND-SAC algorithm, which executes the following steps S1-S4 to complete the UAV formation path planning:
[0007] Step S1: Establish a leader path goal optimization model, describe the path planning problem of the formation drones as a Markov decision process, and design the state space function, action space function and non-sparse reward function;
[0008] Step S2: Construct the RND-SAC algorithm model, set the target parameters in the RND-SAC algorithm model, initialize the state parameters in the RND-SAC algorithm model, obtain the initial state information and select actions according to the current policy, use the RND algorithm to calculate the intrinsic reward and obtain the total reward, and store each state information into the experience pool.
[0009] Step S3: Select the sampling queue size, select N experiences from the experience pool. If the experience pool size is smaller than the sampling queue size, continue the iteration and increase the experience pool data.
[0010] Step S4: Based on the obtained experience pool data, update the RND prediction network to minimize the prediction error, update the value network in the SAC algorithm and calculate the Q value, update the policy network in the SAN algorithm and maximize the entropy regularization policy.
[0011] Step S5: When the UAV training iterations reach the maximum number and the model reward function curve tends to converge, the model training is complete; the trained policy network is used to output UAV action commands, and the value network is used to output state evaluation values; the initial positions of multiple UAVs are used as inputs to run the trained model to achieve multi-UAV path planning.
[0012] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method.
[0013] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method.
[0014] Beneficial effects: Compared with the prior art, the advantages of the present invention include:
[0015] (1) This invention performs in-depth modeling of the UAV formation system and path targets, and designs mathematical models of the state space and action space of the navigator UAV. In addition, to accelerate the UAV learning process and improve the overall learning efficiency, this invention carefully designs a non-sparse reward function, aiming to build a more continuous and smooth reward distribution. This provides an important mathematical model foundation for the development of the RND-SAC control algorithm.
[0016] (2) Based on the deep mathematical modeling and related mathematical models of the UAV formation system and path targets, this invention develops a novel RND-SAC control algorithm. Compared with existing formation algorithms, it effectively solves the reward sparsity problem in UAV formation path planning, significantly improves policy search efficiency and formation coordination performance, and ensures that UAVs can complete tasks efficiently and safely in complex environments. Attached Figure Description
[0017] Figure 1 This is a flowchart of a UAV formation path planning method based on the RND-SAC algorithm provided in an embodiment of the present invention;
[0018] Figure 2 This is a schematic diagram of multiple obstacles provided according to an embodiment of the present invention;
[0019] Figure 3 This is a schematic diagram of the SAC algorithm network framework provided in an embodiment of the present invention;
[0020] Figure 4 This is a schematic diagram of the RND algorithm network framework provided in an embodiment of the present invention;
[0021] Figure 5 This is a graph showing the change in reward value according to an embodiment of the present invention;
[0022] Figure 6 This is a schematic diagram of the RND-SAC algorithm path provided in an embodiment of the present invention;
[0023] Figure 7 This is a schematic diagram of the SAC algorithm path provided according to an embodiment of the present invention;
[0024] Figure 8 This is a schematic diagram of the DQN algorithm path provided in an embodiment of the present invention;
[0025] Figure 9 This is a schematic diagram of the trajectory of the RND-SAC algorithm under various obstacles according to an embodiment of the present invention. Detailed Implementation
[0026] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.
[0027] This invention provides a method for UAV formation path planning based on the RND-SAC algorithm, referring to... Figure 1 Perform the following steps S1-S5 to complete the drone formation path planning:
[0028] Step S1: Establish a leader path goal optimization model, describe the path planning problem of the formation drones as a Markov decision process, and design the state space function, action space function and non-sparse reward function;
[0029] The specific steps of step S1 are as follows:
[0030] Step S1.1: Establish the leader path goal optimization model as follows:
[0031] ;
[0032] in, For task duration, This is the total length of the entire path. This is the initial state of the Navigator drone. Represents the loss function. The optimal strategy for Navigator drones;
[0033] Step S1.2: Refer to Figure 2 Design the state space of the Navigator drone As shown below:
[0034] ;
[0035] in, The relative distance between the current position of the Navigator drone and the position of the mission target. The distance to the nearest obstacle to the current position of the navigator drone; The speed information for the Navigator drone includes velocity components along the x-axis and y-axis. For the direction of the resultant velocity, This refers to the current location information of the Navigator drone.
[0036] Step S1.3: Design the motion space of the Navigator UAV As shown below:
[0037] ;
[0038] In the formula, For the Navigator drone, the velocity components on the x and y axes;
[0039] Step S1.4: Design a non-sparse reward function, including a reward function related to distance. Reward function for angle Regarding the reward function for obstacle avoidance Furthermore, if the drone ultimately reaches the mission objective, a positive deadline reward will be given. And penalties will be imposed based on the time cost. This is intended to incentivize drones to reach the finish line at a faster speed. Its form is as follows:
[0040] ;
[0041] In the formula, Indicates the state transition probability;
[0042] Design an external composite reward function based on a non-sparse reward function. As shown in the following formula:
[0043] ;
[0044] in, , They are respectively , Reward weighting , These are the current location information of the Navigator drone and the current location information of the mission target, respectively. The maximum permissible distance between the position of the navigator drone and the position of the mission target.
[0045] Step S2: Construct the RND-SAC algorithm model, set the target parameters in the RND-SAC algorithm, initialize the state parameters in the algorithm, obtain the initial state information, and select action S according to the current policy. t The intrinsic reward is calculated using RND and the total reward is obtained. The information of each state is stored in the experience pool D.
[0046] Reference Figure 3 , Figure 4 The specific steps of step S2 are as follows:
[0047] Step S2.1: Establish the optimization objective function of the SAC algorithm module as follows:
[0048] ;
[0049] in, Representation Strategy In state The entropy below, For temperature coefficient, Represents the motion vector of the drone. This represents a bounded reward tuple. This indicates the state of the trajectory distribution caused by the strategy. This represents the total amount of rewards;
[0050] The value network and the double Q network together form the evaluation network. The update objective of the Q network is to minimize the mean square error of the temporal difference error, and its loss function is:
[0051] ;
[0052] in, These correspond to two Q networks respectively. Let represent the iteration value of the Q-network; the temporal difference objective value y is given by the following formula:
[0053] ;
[0054] In the formula, This indicates the limited rewards released by the environment. The next state is... It is an action generated by the policy network. It is the target Q network. It's a temperature parameter. It is a discount factor;
[0055] The goal of updating the value network is to approximate the following objective value:
[0056] ;
[0057] Its loss function is:
[0058] ;
[0059] in, It is the value network's estimate of the value of state s. This represents the target approximation value for updating the target;
[0060] Step S2.2: First, input the status. The target network is reached to obtain the output. Then input the same status. The prediction network is used to obtain the output. Intrinsic rewards The mean square error between the two:
[0061] ;
[0062] Step S2.3: Based on the external composite reward function and the intrinsic reward, establish the total reward function as follows:
[0063] ;
[0064] In the formula, R represents the external composite reward function. Indicates intrinsic reward. As intrinsic reward weight, The total reward function is used; based on the total reward function, the SAC algorithm module and the curiosity module are trained.
[0065] Step S3: Select the sampling queue size, select N experiences from the experience pool. If the experience pool size is smaller than the sampling queue size, continue the iteration and increase the experience pool data.
[0066] Step S4: Based on the obtained empirical data, update the RND prediction network to minimize the prediction error, update the value network in the SAC algorithm and calculate the Q value, update the policy network in the SAN algorithm and maximize the entropy regularization policy.
[0067] Step S5: When the drone training iterations reach the maximum number and the model reward function curve tends to converge, the model training is complete. The trained policy network is used to output drone action commands, and the value network is used to output state evaluation values. By taking the initial positions of multiple drones as input and running the trained model, multi-drone path planning can be achieved.
[0068] In one embodiment, the specific parameter settings and reward function parameter settings are shown in Tables 1 and 2 below:
[0069] Table 1 Algorithm Parameters
[0070]
[0071] Table 2. Parameters of the reward function
[0072]
[0073] The training results are analyzed as follows:
[0074] The training phase consists of 10,000 rounds. There is no time limit for each round, but exceeding the maximum number of steps results in task failure. The average reward over 100 rounds is used for plotting. The RND-SAC algorithm is used as a benchmark, compared to the traditional SAC and DQN algorithms. The reward curve has been smoothed, as detailed below. Figure 5 As shown:
[0075] The figure shows the reward value changes of three algorithms—RND-SAC, SAC, and DQN—during the training process. The reward of the RND-SAC algorithm rises rapidly from around -150, reaching convergence after about 2000 rounds, after which the reward value stabilizes with minimal fluctuations, demonstrating an excellent exploration strategy. The reward of the SAC algorithm starts at around -180, showing no upward trend in the first 2000 rounds. Then, between 2000 and 5000 rounds, the reward begins to rise, but at a slow rate, stabilizing and converging after 5000 rounds. The reward of the DQN algorithm fluctuates at a low value throughout the training process, with no significant overall improvement.
[0076] The simulation results are analyzed as follows:
[0077] First, the feasibility of the RND-SAC, SAC, and DQN algorithms was verified on the same map containing elliptical obstacles. The map size was 140×250 meters, with the starting point set at coordinates (5, 35) and the target point at (225, 80). For algorithms with continuous action space, the velocity component... Range set meters per second Range set meters per second. Since DQN is an algorithm with a discrete action space, its action space is set to... meters per second The speed is meters per second, and the sampling time is set to 0.2 seconds.
[0078] Figure 6 , Figure 7 , Figure 8 The following are path visualizations for the RND-SAC algorithm, SAC algorithm, and DQN algorithm. The time taken for each algorithm to reach the target point is shown in Table 3 below:
[0079] Table 3. Algorithm execution time
[0080]
[0081] The visualized paths show that the improved RND-SAC algorithm generates the smoothest path. This demonstrates that the RND-SAC algorithm effectively balances exploration and utilization during path planning, generating paths that not only avoid obstacles but also maintain high smoothness, which is of great significance in practical applications.
[0082] In terms of time consumption, the RND-SAC algorithm takes the shortest time, the SAC algorithm's time falls between that of RND-SAC and DQN, while the DQN algorithm takes the longest. This indicates that the RND-SAC algorithm is highly efficient in path planning, able to find a smooth and effective path in a short time, which is a significant advantage for applications with high real-time requirements.
[0083] Finally, an experimental analysis was conducted to assess the adaptability of the RND-SAC algorithm. The randomly generated obstacles in the map were replaced with triangles, rectangles, hexagons, and mixed obstacles. This will verify the algorithm's path planning and obstacle avoidance capabilities in different scenarios. Figure 9 In the diagram, (a) represents the path planning of the UAV in a rectangular obstacle environment. Figure 9 (b) in the diagram represents the path planning of the UAV in a triangular obstacle environment. Figure 9 In the diagram, (c) represents the path planning of the UAV in a hexagonal obstacle environment. Figure 9 In the diagram, (d) represents the path planning of the UAV in a mixed obstacle environment; Figure 9 The overall demonstration shows that the RND-SAC algorithm can successfully avoid obstacles and complete navigation tasks in different obstacle environments, which fully demonstrates its strong adaptability.
[0084] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method.
[0085] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method.
[0086] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.
Claims
1. A method for UAV formation path planning based on the RND-SAC algorithm, characterized in that, Perform the following steps S1-S5 to complete the drone formation path planning: Step S1: Establish a leader path goal optimization model, describe the path planning problem of the formation drones as a Markov decision process, and design the state space function, action space function and non-sparse reward function; Step S2: Construct the RND-SAC algorithm model, set the target parameters in the RND-SAC algorithm model, initialize the state parameters in the RND-SAC algorithm model, obtain the initial state information and select actions according to the current policy, use the RND algorithm to calculate the intrinsic reward and obtain the total reward, and store each state information into the experience pool. Step S3: Select the sampling queue size, select N experiences from the experience pool. If the experience pool size is smaller than the sampling queue size, continue the iteration and increase the experience pool data. Step S4: Based on the obtained experience pool data, update the RND prediction network to minimize the prediction error, update the value network in the SAC algorithm and calculate the Q value, update the policy network in the SAN algorithm and maximize the entropy regularization policy. Step S5: When the UAV training iterations reach the maximum number and the model reward function curve tends to converge, the model training is complete; the trained policy network is used to output UAV action commands, and the value network is used to output state evaluation values; the initial positions of multiple UAVs are used as inputs to run the trained model to achieve multi-UAV path planning.
2. The UAV formation path planning method based on the RND-SAC algorithm according to claim 1, characterized in that, The specific steps of step S1 are as follows: Step S1.1: Establish the leader path goal optimization model as follows: ； in, For task duration, This is the total length of the entire path. This is the initial state of the Navigator drone. Represents the loss function. The optimal strategy for Navigator drones; Step S1.2: Design the state space of the Navigator UAV As shown below: ； in, The relative distance between the current position of the Navigator drone and the position of the mission target. The distance to the nearest obstacle to the current position of the navigator drone; The speed information for the Navigator drone includes velocity components along the x-axis and y-axis. For the direction of the resultant velocity, This refers to the current location information of the Navigator drone. Step S1.3: Design the motion space of the Navigator UAV As shown below: ； In the formula, For the Navigator drone, the velocity components on the x and y axes; Step S1.4: Design a non-sparse reward function, including a reward function related to distance. Reward function for angle Regarding the reward function for obstacle avoidance Furthermore, if the drone ultimately reaches the mission objective, a positive deadline reward will be given. And penalties will be imposed based on the time cost. This is to incentivize drones to reach their destination at a faster speed; it takes the following form: ； In the formula, Indicates the state transition probability; Design an external composite reward function based on a non-sparse reward function. As shown in the following formula: ； in, , They are respectively , Reward weighting , These are the current location information of the Navigator drone and the current location information of the mission target, respectively. The maximum permissible distance between the position of the navigator drone and the position of the mission target.
3. The UAV formation path planning method based on the RND-SAC algorithm according to claim 1, characterized in that, In step S1.4, the reward function for distance is designed as follows: ； In the formula, Indicates the distance between the drone and the mission target location. , Let be the position of the drone at time t. Let t be the position of the drone at time t+1.
4. The UAV formation path planning method based on the RND-SAC algorithm according to claim 1, characterized in that, In step S1.4, the reward function for the angle is designed as follows: ； in, and These are the heading angle and the target heading angle, respectively, and their definitions are as follows: ； in, , These are the components of the UAV's velocity vector on the x-axis and y-axis at time t, respectively. , These represent the current displacement of the task target around the x and y axes, respectively.
5. The UAV formation path planning method based on the RND-SAC algorithm according to claim 1, characterized in that, In step S1.4, the reward function for obstacle avoidance is designed as follows: ； In the formula, The distance to the nearest obstacle to the current position of the Navigator drone. Indicates the set danger radius. , This is the preset reward value.
6. The UAV formation path planning method based on the RND-SAC algorithm according to claim 1, characterized in that, The specific steps of step S2 are as follows: Step S2.1: Establish the optimization objective function of the SAC algorithm module as follows: ； in, Representation Strategy In state The entropy below, For temperature coefficient, Represents the motion vector of the drone. This represents a bounded reward tuple. This indicates the state of the trajectory distribution caused by the strategy. This represents the total amount of rewards; The value network and the double Q network together form the evaluation network. The update objective of the Q network is to minimize the mean square error of the temporal difference error, and its loss function is: ； in, These correspond to two Q networks respectively. Let represent the iteration value of the Q-network; the temporal difference objective value y is given by the following formula: ； In the formula, This indicates the limited rewards released by the environment. The next state is... It is an action generated by the policy network. It is a target Q-network. It's a temperature parameter. It is a discount factor; The goal of updating the value network is to approximate the following objective value: ； Its loss function is: ； in, It is the value network's estimate of the value of state s. This represents the target approximation value for updating the target; Step S2.2: First, input the status. Reach the target network and obtain the output. Then input the same status. The prediction network is used to obtain the output. Intrinsic rewards The mean square error between the two: ； Step S2.3: Based on the external composite reward function and the intrinsic reward, establish the total reward function as follows: ； In the formula, R represents the external composite reward function. Indicates intrinsic reward. As intrinsic reward weight, The total reward function is used; based on the total reward function, the SAC algorithm module and the curiosity module are trained.
7. The UAV formation path planning method based on the RND-SAC algorithm according to claim 1, characterized in that, In step S5, a simulation environment is built using Python and the RND-SAC model is trained. Then, MATLAB is used to visualize the results.
8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements each step of the method according to any one of claims 1 to 8.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.