Unmanned aerial vehicle path planning method based on TD3-PSO algorithm
Through the TD3-PSO algorithm combined with deep reinforcement learning and improved particle swarm algorithm, the problems of high-dimensional path planning and local optimal solutions in drone path planning are solved, and efficient path planning is realized in complex environments, reducing energy consumption and Improved security.
Patent Information
- Application Number
- CN202411066371.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-11-29
AI Technical Summary
The existing drone path planning algorithms are difficult to deal with high-dimensional path planning problems in complex environments, and are easily trapped in local optimal solutions and difficult to find the optimal path.
Using the UAV path planning method based on TD3-PSO algorithm, by establishing a path model with flight distance, flight altitude, flight deflection angle and obstacle avoidance ability as indicators, combined with improved particle swarm algorithm and deep reinforcement learning training algorithm, we find the best Optimize the path and perform mutating operations when trapped in the local optimal solution to improve the algorithm accuracy.
It effectively shortens the flight distance, reduces the energy consumption of drones, improves obstacle avoidance capabilities, can find the global optimal path in high-dimensional problems, and improves the accuracy and iteration speed of the algorithm.
Smart Images

Figure BDA0004979457450000021 
Figure BDA0004979457450000022 
Figure BDA0004979457450000031
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of unmanned aerial vehicle path planning, and in particular relates to a unmanned aerial vehicle path planning method based on a TD3-PSO algorithm. Background Art
[0002] With the rapid development of science and technology, drones are increasingly used in military reconnaissance, logistics distribution, environmental monitoring and other fields. In these applications, path planning is a key technology to achieve efficient and safe operation of drones. The main purpose of drone path planning is to find the optimal or approximately optimal flight path to reduce energy consumption and flight time, avoid obstacles and ensure safety while ensuring the completion of the mission. Therefore, studying and improving drone path planning algorithms has important practical significance and application value.
[0003] Aiming at the problem of UAV path planning, Luo Wenguan et al. proposed a path planning method based on Q-learning algorithm to select cuckoo flight action in "Emergency UAV path planning based on reinforcement learning cuckoo search algorithm". Catastrophology, 2023, 38(2): 206-212. This method approximates the optimal path through multiple search strategies, but due to the discreteness of the action space, it is difficult to handle high-dimensional path planning problems in complex environments. Xu, C et al. proposed an improved artificial bee colony optimization algorithm (Artificial Bee Colony, ABC) based on chaos theory in "Chaotic artificial bee colony approach to Uninhabited Combat Air Vehicle (UCAV) path planning". Aerospace science and technology, 2010, 14(8), 535-541. The algorithm uses the ergodicity and irregularity of chaotic variables to help the basic ABC algorithm jump out of the local optimal solution, which is used to solve the path planning problem of unmanned fighter jets in battlefield environments. However, this method depends on parameter settings and has a slow iteration rate. Zhang Shu et al. proposed an adaptive parameter optimization method to solve the parameter setting problem in "Improved PSO algorithm and its application in UAV path planning". Computer Systems Applications, 2023, 32(3): 330–337. However, this method determines the parameters at the beginning of the algorithm, lacks flexibility, is prone to fall into local optimal solutions, and is difficult to find the optimal path. Summary of the invention
[0004] The purpose of this invention is to provide a UAV path planning method based on the TD3-PSO algorithm. It aims to plan the UAV path by using the improved particle swarm algorithm, combined with evaluation indicators such as energy consumption and obstacle avoidance ability, and at the same time, through the key parameters in the deep reinforcement learning training algorithm, help the system find the optimal path, and provide a new way for UAV path planning.
[0005] The technical solution to achieve the purpose of the present invention is: a UAV path planning method based on TD3-PSO algorithm, comprising the following steps:
[0006] Step (1): Establish a path model with flight distance, flight altitude, flight deflection angle and obstacle avoidance capability as indicators;
[0007] Step (2): The evaluation functions of the four indicators are weighted and summed to establish a fitness function. Based on the fitness function, an improved particle swarm algorithm is used to find the optimal path. When the optimization process falls into a local optimal solution, the optimal particle position is mutated.
[0008] Step (3): Take the overall population distribution of the particle swarm algorithm as the input state, the three parameter values of speed inertia weight, individual acceleration factor and social acceleration factor as the output action, the rate of change of the fitness function value as the reward function, take the optimal path search as the environment, adopt the dual critic network module to build a deep reinforcement learning network model, and perform offline training;
[0009] Step (4): Use the improved particle swarm algorithm to find the optimal path based on the fitness function. After each round of iteration, the algorithm updates the speed inertia weight, individual acceleration factor and social acceleration factor according to the pre-trained network model; updates the speed and position of the particles based on the parameters, and calculates the new fitness function value to enter the next round of iteration, and finally finds the minimum value of the fitness function, that is, the global optimal path.
[0010] Furthermore, step (1) is specifically as follows:
[0011] A path model is established in three-dimensional space with flight distance, flight altitude, flight deflection angle and obstacle avoidance capability as indicators. The obstacle is modeled as a cylinder. The path model is as follows:
[0012]
[0013] N i =(x i ,y i ,z i ) is the coordinate of the path node that the UAV passes through, N i N i+1 is the vector of the i-th path, n is the total number of path nodes, α i is the angle between adjacent paths, Rj is the radius of the jth obstacle, m is the total number of obstacles, u is the safety distance, d i is the shortest distance from the i-th path to the center of the obstacle circle, Risk j (N i N i+1 ) is the risk cost of the i-th path to the j-th obstacle, F 1 is the evaluation function of the UAV path length, F 2 is the evaluation function of the UAV flight altitude, F 3 is the evaluation function of the UAV flight deflection angle, F 4 is the evaluation function of the obstacle avoidance capability of the UAV.
[0014] Furthermore, step (2) specifically includes the following steps:
[0015] Step (21): Establish a fitness function: The weighted sum of the evaluation functions of the four indicators in step (1) is defined as the fitness function:
[0016]
[0017] In the formula, F(x i ) is the total evaluation function of the ith path, f k is the weight corresponding to the kth evaluation index, F k (x i ) is the evaluation value of the kth indicator of the ith path;
[0018] Step (22): Initialize the population: Randomly initialize the particle population, calculate the fitness function value of each particle, find the individual extreme value of the particle and the global extreme value of the entire population;
[0019] Step (23): Update particle position and velocity: In each iteration of the algorithm, the velocity and position of each particle are updated based on the individual extreme value of the particle and the global extreme value of the entire population. The update formula is:
[0020]
[0021] Where w is the speed inertia weight, c 1 is the individual acceleration factor, c 2 is the social acceleration factor, pbest l (t) is the historical optimal position of the lth particle at the tth iteration, gbest(t) is the global optimal position at the tth iteration; l (t) is the position of the lth particle at the tth iteration, and vl(t) is the velocity of the lth particle at the tth iteration;
[0022] Step (24): Mutation of the best particle position: When the variance σ 2 When it is less than the set threshold, the global optimal position gbest(t) is mutated, and the expression is:
[0023] gbest(t)=gbest(t)(1+0.8μ)
[0024] Where μ is Gaussian white noise;
[0025] Step (25): Determine whether the termination condition is met. If not, return to step (23); otherwise, terminate.
[0026] Furthermore, step (3) is specifically as follows:
[0027] Step (31): Establish a neural network model: Establish a reinforcement learning architecture including an Actor network and a Target_Actor network, as well as two Critic networks and two Target_Critic networks;
[0028] Step (32): Establish an experience replay pool: Establish a replay buffer, the data structure is (s t ,a t ,r t ,s t+1 ), the data generated during the training process are stored in this buffer;
[0029] Step (33): Select an action based on the Actor network and get a reward after executing the action;
[0030] Step (34): Update the Actor and Critic networks;
[0031] Step (35): Determine whether the termination condition is met. If not, return to step (33); otherwise, terminate and complete the offline training.
[0032] Furthermore, step (33) is specifically as follows:
[0033] The input state is an 8-dimensional state space. The first four dimensions represent the change in the average fitness function of the particle swarm in the past, and the last four dimensions represent the minimum, maximum, average, and standard deviation of the fitness function of the latest iteration. The formula is:
[0034]
[0035] Where F mean represents the mean of the fitness function, F min Represents the minimum value of the fitness function, F max Represents the maximum value of the fitness function, F stdrepresents the standard deviation of the fitness function;
[0036] The eight states are sent as input to the Actor network, and the Actor network outputs a 3D vector, which respectively represents the velocity inertia weight, individual acceleration factor, and social acceleration factor of the particle swarm algorithm; Gaussian noise is added to the action vector output by the Actor network, and the adaptive noise standard deviation calculation formula is used:
[0037]
[0038] Where N is the added noise, N min is the minimum noise value, N max is the maximum value of noise, T is the current iteration number, T max is the maximum number of iterations;
[0039] The action vector with added noise is given as a parameter to the particle swarm algorithm to start calculation, and a new state vector is obtained. Based on the new state vector, the reward value is obtained, and its expression is:
[0040]
[0041] In the formula, gvalue is the minimum value of the fitness function in each round of iteration;
[0042] The state, action, reward, and new state of this round are (s t ,a t ,r t ,s t+1 ) into the playback buffer and wait for training.
[0043] Furthermore, step (34) is specifically as follows:
[0044] Take a batch of data from the experience replay pool and convert s t+1 As input, it is sent to the Target_Actor network to obtain the action amount, and a noise is added to form the final action amount a t+1 , will s t+1 and a t+1 As input, it is sent to two Target_Critic networks to obtain two corresponding Q values, then s t ,a t The corresponding Q value expression is:
[0045]
[0046] Where TargetQ1 and TargetQ2 are the output values of the two Target_Critic networks respectively;
[0047] Will s t ,at As input, it is sent to two Critic networks to obtain the corresponding two Q values, namely Q1 and Q2. The mean square error between this batch of data Q1 and Q2 and TargetQ is solved, and it is used as the loss function to update the two Critic networks using batch gradient descent. The loss function formula is as follows:
[0048]
[0049] After two updates of the Critic network, s t As input, it is sent to the Actor network to get a new action a t , and then s t and a t It is sent as input to the Critic1 network to obtain the corresponding Q value, and the mean Q value of this batch of data is solved. It is used as the loss function to update the Actor network using the batch gradient descent method. The formula of the loss function is as follows:
[0050]
[0051] Update the parameters of the Actor network to Target_Actor, and update the parameters of the two Critic networks to the corresponding Target_Critic networks. Use delayed soft update. The update method is as follows:
[0052] θ′=τθ+(1-τ)θ′
[0053]
[0054] Furthermore, step (4) is as follows: after loading the trained Actor network model, the population state vector s after each iteration is t Send it to the Actor network to get the corresponding action vector a t , a t It is sent as a parameter to the particle swarm algorithm for a new round of calculation until the iteration ends and the optimal path is found.
[0055] Compared with the prior art, the present invention has the following significant advantages:
[0056] (1) The method of the present invention proposes a path model with flight distance, flight altitude, flight deflection angle and obstacle avoidance capability as indicators. By shortening the flight distance and lowering the flight altitude, the energy consumption of the UAV can be reduced, and improving the obstacle avoidance capability can ensure flight safety.
[0057] (2) The method of the present invention proposes an algorithm for finding the optimal path using a particle swarm optimization algorithm based on a multi-index fitness function, and improves the algorithm by mutating the position of the optimal particle when it falls into a local optimal solution, thereby improving the accuracy of the algorithm.
[0058] (3) The method of the present invention proposes a framework for training the key parameters of the particle swarm algorithm based on double-delay deep deterministic policy gradient, which can output the values of three parameters in the continuous action space, namely, the speed inertia weight, the individual acceleration factor and the social acceleration factor, effectively solving the problem of algorithm parameter setting and helping to find the optimal path in high-dimensional problems.
[0059] (4) The method of the present invention proposes a reward function based on the rate of change of the fitness function and a method of adaptively adding noise, which can expand the search space during training and help to find the global optimal solution, that is, the optimal path, more quickly. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] Figure 1 It is the overall algorithm flow chart of the present invention;
[0061] Figure 2 This is the TD3 algorithm training flow chart;
[0062] Figure 3 It is the flow chart of particle swarm optimization algorithm;
[0063] Figure 4 It is the result graph of the fitness function value changing with the number of iterations;
[0064] Figure 5 is the optimal path three-dimensional graph;
[0065] Figure 6 It is the top view of the optimal path. DETAILED DESCRIPTION
[0066] The present invention is further described in detail below in conjunction with the accompanying drawings.
[0067] A UAV path planning method based on TD3-PSO algorithm includes the following steps:
[0068] Step 1: Establish a path model with flight distance, flight altitude, flight deflection angle and obstacle avoidance capability as indicators;
[0069] The details are as follows:
[0070] A path model is established in three-dimensional space with flight distance, flight altitude, flight deflection angle and obstacle avoidance capability as indicators. To simplify the model, obstacles are modeled as cylinders. i =(x i ,y i ,z i ) is the coordinate of the path node that the UAV passes through, N i N i+1 is the vector of the i-th path, n is the total number of path nodes, α iis the angle between adjacent paths, R j is the radius of the jth obstacle, m is the total number of obstacles, u is the safety distance, d i is the shortest distance from the i-th path to the center of the obstacle circle, Risk j (N i N i+1 ) is the risk cost of the i-th path to the j-th obstacle, F 1 is the evaluation function of the UAV path length, F 2 is the evaluation function of the UAV flight altitude, F 3 is the evaluation function of the UAV flight deflection angle, F 4 is the evaluation function of the obstacle avoidance capability of the UAV.
[0071]
[0072] Step 2: The evaluation functions of the four indicators are weighted and summed to form a fitness function. Based on this function, an improved particle swarm algorithm is used to find the optimal path. When the optimization process falls into a local optimal solution, the optimal particle position is mutated;
[0073] The details are as follows:
[0074] Step (21): Establish fitness function
[0075] The weighted sum of the evaluation functions of the four indicators in step (1) is defined as the fitness function:
[0076]
[0077] In the formula, F(x i ) is the total evaluation function of the ith path, f k is the weight corresponding to the kth evaluation index, F k (x i ) is the evaluation value of the kth indicator of the ith path.
[0078] Step (22): Initialize the population
[0079] Randomly initialize the particle population, calculate the fitness function value of each particle, and find the individual extreme value of the particle and the global extreme value of the entire population.
[0080] Step (23): Update particle position and velocity
[0081] In each iteration of the algorithm, the speed and position of each particle are updated based on the individual extreme value of the particle and the global extreme value of the entire population. The update formula is:
[0082]
[0083] Where w is the speed inertia weight, c 1 is the individual acceleration factor, c 2 is the social acceleration factor, pbest l (t) is the historical optimal position of the lth particle at the tth iteration, gbest(t) is the global optimal position at the tth iteration; l (t) is the position of the lth particle at the tth iteration, and vl(t) is the velocity of the lth particle at the tth iteration;
[0084] Step (24): Mutate the best particle position
[0085] When the variance σ 2 When it is less than the set threshold, the global optimal position gbest(t) is mutated, and the expression is:
[0086] gbest(t)=gbest(t)(1+0.8μ) (4)
[0087] Where μ is Gaussian white noise;
[0088] Step (25): Determine whether the termination condition is met. If not, return to step (23); otherwise, terminate.
[0089] Step 3: Take the overall population distribution of the particle swarm algorithm as the input state, the three parameter values of speed inertia weight, individual acceleration factor and social acceleration factor as the output action, the change rate of the fitness function value as the reward function, and the optimal path search as the environment. Use the double critic network module to build a deep reinforcement learning network model and perform offline training;
[0090] The details are as follows:
[0091] Step (31): Establish a neural network model
[0092] Build a reinforcement learning architecture consisting of an Actor network and a Target_Actor network, as well as two Critic networks and two Target_Critic networks.
[0093] Step (32): Create an experience replay pool
[0094] Create a playback buffer with the data structure (s t ,a t ,r t ,s t+1 ), all data generated during the training process will be stored in this buffer.
[0095] Step (33): Select actions based on the Actor network and get rewards after executing the actions
[0096] The input state is an 8-dimensional state space. The first four dimensions represent the change in the average fitness function of the particle swarm in the past, and the last four dimensions represent the minimum, maximum, average, and standard deviation of the fitness function of the latest iteration. The formula is:
[0097]
[0098] Where F mean represents the mean of the fitness function, F min Represents the minimum value of the fitness function, F max Represents the maximum value of the fitness function, F std Represents the standard deviation of the fitness function.
[0099] These 8 states are fed into the Actor network as input, and the network outputs a 3D vector, which represents the velocity inertia weight, individual acceleration factor, and social acceleration factor of the particle swarm algorithm. In order to enhance the exploration ability during training, Gaussian noise is added to the action vector output by the Actor network. At the same time, in order to balance exploration and utilization, an adaptive noise standard deviation calculation formula is used:
[0100]
[0101] Where N is the added noise, N min is the minimum noise value, N max is the maximum value of noise, T is the current iteration number, T max is the maximum number of iterations.
[0102] The action vector with added noise is given as a parameter to the particle swarm algorithm to start calculation, and a new state vector is obtained. Based on the new state vector, the reward value is obtained, and its expression is:
[0103]
[0104] Where gvalue is the minimum value of the fitness function in each round of iteration.
[0105] The state, action, reward, and new state of this round are (s t ,a t ,r t ,s t+1 ) into the playback buffer and wait for training.
[0106] Step (34): Update the Actor and Critic networks
[0107] Take a batch of data from the experience replay pool, with a data (s t ,a t ,r t ,st+1 ) as an example, t+1 As input, it is sent to the Target_Actor network to obtain the action amount, and a noise is added to form the final action amount a t+1 . t+1 and a t+1 As input, it is sent to two Target_Critic networks to obtain two corresponding Q values, then s t ,a t The corresponding Q value expression is:
[0108]
[0109] Where TargetQ1 and TargetQ2 are the output values of the two Target_Critic networks respectively.
[0110] Will s t ,a t As input, it is sent to two Critic networks to obtain the corresponding two Q values, namely Q1 and Q2. The mean square error between this batch of data Q1 and Q2 and TargetQ is solved, and it is used as the loss function to update the two Critic networks using batch gradient descent. The loss function formula is as follows:
[0111]
[0112] After two updates of the Critic network, s t As input, it is sent to the Actor network to get a new action a t , and then s t and a t It is sent as input to the Critic1 network to obtain the corresponding Q value, and the mean Q value of this batch of data is solved. It is used as the loss function to update the Actor network using the batch gradient descent method. The formula of the loss function is as follows:
[0113]
[0114] Update the parameters of the Actor network to Target_Actor, and update the parameters of the two Critic networks to the corresponding Target_Critic networks. Use delayed soft update. The update method is as follows:
[0115]
[0116] Step (35): Determine whether the termination condition is met. If not, return to step (33); otherwise, terminate and complete the offline training.
[0117] Step 4: Use the improved particle swarm algorithm to find the optimal path based on the fitness function. After each round of iteration, the algorithm updates the three parameters of speed inertia weight, individual acceleration factor and social acceleration factor according to the pre-trained network model. Based on these parameters, the speed and position of the particles are updated, and the new fitness function value is calculated to enter the next round of iteration, and finally the minimum value of the fitness function, that is, the global optimal path, is found.
[0118] The details are as follows:
[0119] After loading the trained Actor network model, the population state vector s after each iteration is t Send it to the Actor network to get the corresponding action vector a t , a t It is sent as a parameter to the particle swarm algorithm for a new round of calculation until the iteration ends and the optimal path is found.
[0120] Example 1
[0121] A more detailed description is given below in conjunction with embodiments.
[0122] The simulation and algorithm parameter settings are shown in the following table:
[0123] Table 1 Simulation environment parameters
[0124]
[0125] Table 2 Particle swarm algorithm parameters
[0126]
[0127] Table 3 TD3 algorithm parameters
[0128]
[0129]
[0130] After completing the simulation and setting the algorithm parameters, we use the TD3 algorithm to train the three parameters of the particle swarm algorithm. After the training, the particle swarm algorithm will load the Actor network to find the best path and compare it with the basic PSO algorithm. The starting point coordinates are (0,0,0) and the end point coordinates are (200,200,50). The fitness function value calculation results are shown in Table 4.
[0131] Table 4 Calculation results of fitness function values
[0132]
[0133] Figure 4 The results of the fitness function values of the two algorithms changing with the number of iterations are shown. Figure 5 and Figure 6 The three-dimensional and top-view images of the optimal paths found by the two algorithms are shown in Figure 4. From these three figures and the data in Table 4, we can see that the TD3-PSO algorithm can effectively find the global optimal solution, that is, the optimal path, when solving high-dimensional problems. This algorithm not only performs well in accuracy, but also significantly outperforms the traditional PSO algorithm in iteration speed.
Claims
1. A UAV path planning method based on TD3-PSO algorithm, characterized in that: The steps include: Step (1): Establish a path model with flight distance, flight altitude, flight deflection angle and obstacle avoidance capability as indicators; Step (2): The evaluation functions of the four indicators are weighted and summed to establish a fitness function. Based on the fitness function, an improved particle swarm algorithm is used to find the optimal path. When the optimization process falls into a local optimal solution, the optimal particle position is mutated. Step (3): Take the overall population distribution of the particle swarm algorithm as the input state, the three parameter values of speed inertia weight, individual acceleration factor and social acceleration factor as the output action, the rate of change of the fitness function value as the reward function, take the optimal path search as the environment, adopt the dual critic network module to build a deep reinforcement learning network model, and perform offline training; Step (4): Use the improved particle swarm algorithm to find the optimal path based on the fitness function. After each round of iteration, the algorithm updates the speed inertia weight, individual acceleration factor and social acceleration factor according to the pre-trained network model; The particle speed and position are updated based on the parameters, and the new fitness function value is calculated to enter the next round of iteration, and finally the minimum value of the fitness function, that is, the global optimal path, is found.
2. The method according to claim 1, characterized in that: Step (1) is specifically as follows: A path model is established in three-dimensional space with flight distance, flight altitude, flight deflection angle and obstacle avoidance capability as indicators. The obstacle is modeled as a cylinder. The path model is as follows: N i =(x i ,y i ,z i ) is the coordinate of the path node that the UAV passes through, N i N i+1 is the vector of the i-th path, n is the total number of path nodes, α i is the angle between adjacent paths, R j is the radius of the jth obstacle, m is the total number of obstacles, u is the safety distance, d i is the shortest distance from the i-th path to the center of the obstacle circle, Risk j (N i N i+1 ) is the risk cost of the i-th path to the j-th obstacle, F1 is the evaluation function of the UAV path length, F2 is the evaluation function of the UAV flight altitude, F3 is the evaluation function of the UAV flight deflection angle, and F4 is the evaluation function of the UAV obstacle avoidance capability.
3. The method according to claim 2, characterized in that Step (2) specifically includes the following steps: Step (21): Establish a fitness function: The weighted sum of the evaluation functions of the four indicators in step (1) is defined as the fitness function: In the formula, F(x i ) is the total evaluation function of the ith path, f k is the weight corresponding to the kth evaluation index, F k (x i ) is the evaluation value of the kth indicator of the ith path; Step (22): Initialize the population: Randomly initialize the particle population, calculate the fitness function value of each particle, find the individual extreme value of the particle and the global extreme value of the entire population; Step (23): Update particle position and velocity: In each iteration of the algorithm, the velocity and position of each particle are updated based on the individual extreme value of the particle and the global extreme value of the entire population. The update formula is: Where w is the speed inertia weight, c1 is the individual acceleration factor, c2 is the social acceleration factor, and pbest l (t) is the historical optimal position of the lth particle at the tth iteration, gbest(t) is the global optimal position at the tth iteration; l (t) is the position of the lth particle at the tth iteration, vl(t) is the velocity of the lth particle at the tth iteration; Step (24): Mutation of the best particle position: When the variance σ 2 When it is less than the set threshold, the global optimal position gbest(t) is mutated, and the expression is: gbest(t)=gbest(t)(1+0.8μ) Where μ is Gaussian white noise; Step (25): Determine whether the termination condition is met. If not, return to step (23); otherwise, terminate.
4. The method according to claim 3, characterized in that Step (3) is specifically as follows: Step (31): Establish a neural network model: Establish a reinforcement learning architecture including an Actor network and a Target_Actor network, as well as two Critic networks and two Target_Critic networks; Step (32): Establish an experience replay pool: Establish a replay buffer, the data structure is (s t ,a t ,r t ,s t+1 ), the data generated during the training process are stored in this buffer; Step (33): Select an action based on the Actor network and get a reward after executing the action; Step (34): Update the Actor and Critic networks; Step (35): Determine whether the termination condition is met. If not, return to step (33); otherwise, terminate and complete the offline training.
5. The method according to claim 4, characterized in that Step (33) is specifically: The input state is an 8-dimensional state space. The first four dimensions represent the change in the average fitness function of the particle swarm in the past, and the last four dimensions represent the minimum, maximum, average, and standard deviation of the fitness function of the latest iteration. The formula is: Where F mean represents the mean of the fitness function, F min Represents the minimum value of the fitness function, F max Represents the maximum value of the fitness function, F std represents the standard deviation of the fitness function; The eight states are sent as input to the Actor network, and the Actor network outputs a 3D vector, which respectively represents the velocity inertia weight, individual acceleration factor, and social acceleration factor of the particle swarm algorithm; Gaussian noise is added to the action vector output by the Actor network, and the adaptive noise standard deviation calculation formula is used: Where N is the added noise, N min is the minimum noise value, N max is the maximum value of noise, T is the current iteration number, T max is the maximum number of iterations; The action vector with added noise is given as a parameter to the particle swarm algorithm to start calculation, and a new state vector is obtained. Based on the new state vector, the reward value is obtained, and its expression is: In the formula, gvalue is the minimum value of the fitness function in each round of iteration; The state, action, reward, and new state of this round are (s t ,a t ,r t ,s t+1 ) into the playback buffer and wait for training.
6. The method according to claim 5, characterized in that Step (34) is specifically: Take a batch of data from the experience replay pool and convert s t+1 As input, it is sent to the Target_Actor network to obtain the action amount, and a noise is added to form the final action amount a t+1 , will s t+1 and a t+1 As input, it is sent to two Target_Critic networks to obtain two corresponding Q values, then s t ,a t The corresponding Q value expression is: Where TargetQ1 and TargetQ2 are the output values of the two Target_Critic networks respectively; Will s t ,a t As input, it is sent to two Critic networks to obtain the corresponding two Q values, namely Q1 and Q2. The mean square error between this batch of data Q1 and Q2 and TargetQ is solved, and it is used as the loss function to update the two Critic networks using batch gradient descent. The loss function formula is as follows: After two updates of the Critic network, s t As input, it is sent to the Actor network to get a new action a t , and then s t and a t It is sent as input to the Critic1 network to obtain the corresponding Q value, and the mean Q value of this batch of data is solved. It is used as the loss function to update the Actor network using the batch gradient descent method. The formula of the loss function is as follows: Update the parameters of the Actor network to Target_Actor, and update the parameters of the two Critic networks to the corresponding Target_Critic networks. Use delayed soft update. The update method is as follows: θ′=τθ+(1-τ)θ′ 7. The method according to claim 6, characterized in that Step (4) is as follows: After loading the trained Actor network model, the population state vector s after each iteration is t Send it to the Actor network to get the corresponding action vector a t , a t It is sent as a parameter to the particle swarm algorithm for a new round of calculation until the iteration ends and the optimal path is found.
Citation Information
Cited By
Continuous trajectory planning method for excavator working device
CN119692582A
Multi-unmanned aerial vehicle cooperative target tracking path planning method based on spider wasp optimization algorithm
CN119717845A
Unmanned aerial vehicle flight path planning method and device for radio map construction and distribution, computer equipment and storage medium
CN121498700A
