An unmanned aerial vehicle trajectory planning method based on continuous action advantage function learning

Abstract

The application discloses a kind of unmanned vehicle trajectory planning methods based on continuous action advantage function learning, belong to robot intelligence decision and control field;First, the Markov decision process model of unmanned vehicle trajectory planning is constructed, respectively the expression of state variable, control variable, transition model, loss function of unmanned vehicle trajectory planning is obtained;Then strategy network and evaluation network are established;Again, after every step of unmanned vehicle, strategy network and evaluation network are trained and updated by continuous action advantage function learning, until convergence;Finally, the strategy network for unmanned vehicle trajectory planning is obtained.The application realizes the trajectory planning of unmanned vehicle under the condition that the dynamics model of unmanned vehicle and the environment it is in are completely unknown, so that it reaches the predetermined target in the shortest time, and has very high practical value.

Description

TECHNICAL FIELD

[0001] The present application relates to a method for unmanned aerial vehicle trajectory planning based on continuous advantage function learning, and belongs to the technical field of robot intelligent decision and control. BACKGROUND

[0002] With the development of society and the progress of technology, unmanned aerial vehicles (UAVs) are increasingly used in military and civilian scenarios. However, most existing UAVs still work in pre-programmed or remote control mode, requiring human-in-the-loop control, which limits the application range of UAVs. The task steps of a UAV include three items: environment perception, trajectory planning, and low-level control, among which the intelligence of trajectory planning is a key step to reflect the intelligence of the UAV.

[0003] The commonly used trajectory planning methods for UAVs can be roughly divided into three categories: graph search-based, random sampling-based, and optimization problem-based. Typical methods based on graph search include Dijkstra algorithm and A* algorithm, etc. The basic steps of these methods are to construct a graph including nodes and edges from the map of the UAV workspace, and then perform path search, i.e., update a sequence of visited nodes, and loop through the three steps of "remove node, expand node, and insert node" until the end point is reached. Typical methods based on random sampling include single-query probabilistic road map (PRM) and rapidly-exploring random tree (RRT), etc. These methods do not need to traverse the entire map of the UAV workspace, but uniformly sample in the state space, thereby constructing a connected graph and detecting collisions, etc., until the initial point and the termination point are both in the graph. Compared with the graph search-based methods, the random sampling-based methods have higher planning efficiency.

[0004] In practical applications, the trajectory planning of a UAV needs to consider multiple constraints such as dynamics, energy consumption, external threats, etc., and different objectives. However, the above two methods almost do not consider the dynamics constraints of the UAV, but only focus on the path planning of the UAV. To address this issue, existing research often uses optimal control in optimization-based methods for analytical or numerical solutions. However, for UAVs with nonlinear, coupled, and uncertain dynamics, optimization-based methods also face many challenges. In response to this challenge, model-free reinforcement learning (RL) has been widely developed in recent years, which is a general decision-making framework that does not require prior knowledge of the dynamics of the UAV. Through the interaction between the UAV and the environment, the optimal strategy can be learned, and the RL algorithm can be divided into three categories: value function method, policy method, and actor-critic method.

[0005] Q-learning algorithms and their extensions, particularly deep Q-learning, are typical examples of value-function reinforcement learning (RL) algorithms, deriving policies from value functions. Furthermore, value-function reinforcement learning algorithms are primarily applied to Markov Decision Processes (MDPs) with small and discrete action spaces. However, in real-world scenarios, discretization techniques are required, making it difficult to learn optimal policies. In contrast, policy-based reinforcement learning algorithms do not require value functions and can directly learn continuous policies. However, these algorithms can be inefficient and difficult to implement, especially when dealing with tractable policy gradients. Combining these two types of algorithms, actor-commentator reinforcement learning algorithms can overcome their shortcomings, where the actor estimates the policy to be taken in a given state, and the commentator approximates the value function to evaluate the current policy.

[0006] Actor-Critic reinforcement learning algorithms for continuous action spaces can be broadly categorized into two types: methods based on TD error and methods based on policy gradients. Methods based on TD error, such as DDPG, TD3, and SAC, primarily obtain the optimal action-value function estimate through Bellman equation updates. These methods offer high sample efficiency but introduce bias. Methods based on policy gradients, such as TRPO and PPO, often employ Monte Carlo methods to estimate the value function to reduce bias, thus providing better convergence guarantees, but at the cost of some sample efficiency.

[0007] The time-minimum trajectory planning problem for unmanned aerial vehicles (UAVs) presents significant challenges. In this task, the UAV's dynamic model exhibits nonlinear, coupled, and uncertain dynamic properties. Numerical solutions using optimization methods require substantial computation, while reinforcement learning frameworks present the challenge of reward sparsity. Therefore, further research is needed to address these issues. Summary of the Invention

[0008] In view of this, the purpose of this invention is to provide a UAV trajectory planning method based on continuous action advantage function learning. This method does not rely on the UAV dynamics model, but only uses the UAV position and the target position as neural network inputs and the UAV angular velocity control command as output to train a UAV trajectory planning strategy network, enabling the UAV to reach the predetermined target position in the shortest possible time.

[0009] A method for UAV trajectory planning based on continuous action advantage function learning includes:

[0010] Step 1) Construct the shortest time trajectory optimization problem for the UAV, including determining the objective function and constraints:

[0011] 1-1) Determine the objective function;

[0012] The objective function J of the shortest time trajectory optimization problem for UAVs is expressed as:

[0013]

[0014] in, This represents the mean operator, r k Let k represent the reward function of the drone, and k represent the k-th step of the state transition. f u is the number of steps to the finish line. k≥1 Let represent the UAV control input sequence with step number k≥1, and γ represent the decay factor of the weight of future loss, 0<γ<1;

[0015] 1-2) The expression for the constraints of the UAV trajectory optimization problem is:

[0016] s k+1 =g(s k ,a k ), k = 1, ..., k f k f The number of steps to the finish line;

[0017] Among them, s k and s k+1 Let a be the state vector of the UAV at step k and step (k+1), respectively. k Let g(·,·) represent the control variables at step k, and let g(·,·) represent the dynamic model of the UAV, i.e., the transfer model.

[0018] Step 2) Based on the shortest time trajectory optimization problem of the UAV constructed in Step 1), establish a Markov decision process model for UAV trajectory planning.

[0019] A Markov decision process model is represented by a quadruple, namely, state variables, control variables, transition model, and loss function. The specific steps for its construction are as follows:

[0020] 2-1) Determine the state variables;

[0021] Establish a Cartesian coordinate system at the altitude of the drone, with its initial position as the origin, and the state variable s. k The expression is:

[0022] s k =[ρ k θ k , ψ k f(p) k ,k)] T

[0023] Where, p k =(ρ k θ k , ψ kf(P) represents the spherical coordinate position of the UAV at the current k-th step. k ,k) represents the obstacle information obtained by the UAV at the current k-th step;

[0024] 2-2) Determine the control variables;

[0025] The control variable is expressed as the angular velocity α of the UAV. k =[ω k φ k ] T That is, (θ) k , ψ k Rate of change over time;

[0026] 2-3) The expression for the transfer model is:

[0027] s k+1 =g(s k a k );

[0028] This transition model represents returning the state variables at the next time step, given the current k-step state variables and control variables.

[0029] 2-4) Determine the loss function;

[0030] The loss function is used to evaluate the drone state s k The control variable a is executed below. k The effect, expressed as:

[0031]

[0032] Where ρ and λ represent the weight coefficients of each item, respectively;

[0033] Δ d (k)=||(x k -x r y k -y r , z k -z r )′||-||(x k-1 -x r y k-1 -y r , z k-1 -z r )′||,

[0034] r s (k)=1 / ||(x k -x o y k -y o , z k -z o )′||

[0035] Where “′” indicates transpose; (x k y k , z k ) represents the current spherical coordinate position p of the UAV. k In a Cartesian coordinate system, (x) r y r , z r (x) represents the target position of the UAV in a Cartesian coordinate system; o y o , z o ( ) represents the obstacle coordinates in a Cartesian coordinate system. When the drone approaches the obstacle to a certain range, the indicator function... The value is 1; let the reward amount rk be a negative loss function, i.e., r k =-c(s) k a k );

[0036] Step 3) Design reinforcement learning algorithm and train planning strategy;

[0037] 3-1) Construct a reinforcement learning neural network that includes an evaluation network and a policy network;

[0038] The evaluation network and policy network share a two-layer fully connected network with the ReLU activation function. After the two fully connected layers, two branches are formed. One branch connects to a single-layer fully connected network, which together with the two fully connected layers forms the evaluation network. This single-layer fully connected network uses the Identity activation function. The other branch also connects to a single-layer fully connected network, which together with the two fully connected networks forms the policy network. This single-layer fully connected network uses the Tanh activation function.

[0039] Evaluating the network: using ω s ω c Parameterized evaluation network V(s) k |ω s ω c ), ω s ω represents the weight parameters shared with the policy network. c The evaluation network's unique weight parameters are represented by V(s). k |ω s ω c The input to ) is the state variable s k The output is the cumulative reward amount, and the function expression for the cumulative reward amount is as follows:

[0040]

[0041] Policy network: using ω s ω aParameterize a policy network ω s The weight parameters shared by the evaluation network; ω a This represents the weight parameters unique to the policy network. The input to the policy network is the state variable s. k The direct output is the action mean μ(s) k |ω s ω a ) and covariance matrix L(s k |ω s ω a Thus, the control variable a is constructed. k , represented as:

[0042]

[0043] Among them, a k Given a mean of μ(s) k |ω s ω a ), covariance is ∑(s k |ω s ω c The multidimensional Gaussian distribution sampling is obtained; ∑(s) k |ω s ω c )=L(s k |ω s ω a )L(s k |ω s ω a ) T ;

[0044] 3-2) The method for updating the evaluation network is to minimize the following Bellman residuals:

[0045]

[0046] Where, ω s ω a ω c These are the parameters for the shared layer, the policy network, and the evaluation network, respectively. D represents the storage of historical data (s). k a k ,c(s k a k ), s k+1 (s) replay memory; k a k )~D represents s k a k The data extracted from D; C(ω) s ω a ω cThe expression for the TD error with respect to the target neural network is as follows:

[0047] C(ω s ω a ω c ) = r k+1 +γV′(s k+1 |ω s ω c )-V(s k |ω s ω c )-A(s k a k |ω s ω c )

[0048]

[0049] Y(s k a k ) = L -1 (s k |ω s ω c (a) k -μ(s k |ω s ω a ))

[0050] The target neural network has the same structure as the current reinforcement learning neural network, but its parameter updates are delayed compared to the current network. The evaluation network and policy network also have their own corresponding target networks; V′(s k+1 |ω s ω c A(s) represents the cumulative reward output of the target evaluation network; k a k |ω s ω c Y(s) represents the advantage function estimate of the UAV taking the current action relative to other actions in the current state, approximated by a quadratic form. k a k ) is the advantage function A(s) k a k |ω s ω c Intermediate variables in the calculation process, V(s) k |ω s ω c (This is obtained from the output of the evaluation network;)

[0051] Step 4) The specific steps to obtain the shortest time trajectory plan for the UAV are as follows:

[0052] 4-1) Parameter settings;

[0053] Set the maximum number of training iterations M for the evaluation network and policy network, the maximum number of update steps T for each training iteration, the size N of the training set extracted for experience replay, the tracking ratio β of the target neural network, the synchronous update learning rate α for the evaluation network and policy network, and the decay factor γ.

[0054] 4-2) Initialization;

[0055] The weight parameters of the evaluation network and policy network, which share a portion of the network layers, are initialized to 0; the initialized policy network is denoted as the current policy network. The initialized evaluation network is denoted as the current evaluation network V(s). k |ω s ω c );

[0056] Let D be an empty set of experience queues;

[0057] 4-3) To begin the iteration, let the initial number of iterations be m = 1;

[0058] 4-4) Initialize the time k=0, initialize the current position of the UAV, the position of the obstacle, and the position of the target;

[0059] 4-5) Use the current policy network Generate the control variable at time k; input the control variable a k With state s k The drone returns the state variable s for the next time step through the transition model. k+1 The reward r for this transition step is calculated using the reward function. k+1 ;

[0060] 4-6) Record the data obtained in steps 4-4) to 4-5) as an empirical sample d. k =(s k a k r k+1 s k+1 ), and store in the experience queue D;

[0061] 4-7) Let N be the number of experience samples in the statistical experience queue D. D And determine: if N D If ≤ N, then proceed to step 4-5); if N R If the value is greater than N, proceed to steps 4-8.

[0062] 4-8) Randomly select N experience samples d1, d2, ..., d3 from the experience queue D. N ; Calculate a C(ω) using each empirical sample s ω a ω cThe value of ), the i-th value is defined as C. i (ω s ω a ω c );

[0063] 4-9) Use

[0064]

[0065]

[0066]

[0067] Update the weight parameters ω of the evaluation network and the policy network. s ω a ω c ;

[0068] use Update the weight parameters of the target neural network Update the training count m using m+1;

[0069] The updated evaluation network and policy network are then denoted as the new evaluation network and the new policy network.

[0070] 4-10) Determination: If k > T or the current state of the drone meets the set termination condition, proceed to step 4-11); otherwise, return to step 4-5) and update the step number k using k+1.

[0071] 4-11) Make a judgment: If m < M, then return to step 4-4), the UAV initializes again and starts a new trajectory planning task; otherwise, the iteration ends, and the current policy network at the time of iteration termination is used. The final policy network is denoted as... Input the state variables into the final policy The shortest time trajectory of the UAV is obtained by outputting the control variables.

[0072] Preferably, in steps 4-10), the termination condition is set when the distance from the drone to the desired location is less than a set threshold ξ. p .

[0073] The present invention has the following beneficial effects:

[0074] 1) This invention proposes a UAV trajectory planning method based on continuous action advantage function learning, belonging to the field of robot intelligent decision-making and control. First, a Markov decision process model for UAV trajectory planning is constructed, obtaining the expressions for the state variables, control variables, transition model, and loss function. Then, a policy network and an evaluation network are established. Next, through continuous action advantage function learning, the policy network and evaluation network are trained and updated after each step the UAV takes forward until they converge. Finally, a policy network for UAV trajectory planning is obtained. This invention achieves UAV trajectory planning under conditions where the UAV's dynamics model and environment are completely unknown, enabling it to reach the predetermined target in the shortest time, and has high practical value.

[0075] 2) This invention achieves trajectory planning under the condition that the UAV model is unknown. The policy gradient update method learns the optimal evaluation network and policy network autonomously through sampled data of UAV trajectory and sensor information. This process does not require any assumptions or parameters of the UAV model, and only designs the reward function according to the required objective when modeling the Markov decision process.

[0076] 3) The method of this invention combines deterministic policy gradient theory with artificial neural networks, designing an evaluation network and a policy network with shared layers. The evaluation network is used to fit the value function (V function) in reinforcement learning, while the policy network generates the corresponding control policy given a state. Utilizing the nonlinear fitting capability of neural networks, an ideal trajectory planning policy can still be learned even when the UAV model and complex obstacle environment are unknown. This invention effectively solves the problem of finding the extrema in the continuous action space through quadratic dominance function estimation and a Dueling architecture.

[0077] 4) The method of this invention has strong scalability. The method of this invention designs trajectory planning methods for a wide range of UAV target tracking and trajectory tracking tasks. There are no prior assumptions about the implementation environment. It can be extended to UAV trajectory planning tasks in different environments simply by resetting the state variables, control inputs and loss functions. Attached Figure Description

[0078] Figure 1 This is a schematic diagram of a three-dimensional planar model of the UAV in an embodiment of the present invention.

[0079] Figure 2 This is a schematic diagram of the shared evaluation strategy network of the present invention in an embodiment of the present invention.

[0080] Figure 3 This is a schematic diagram of the continuous action advantage function learning architecture in an embodiment of the present invention.

[0081] Figure 4(a) , 4(b)Figures 4(c) and 4(d) are schematic diagrams illustrating the learning status of the continuous action advantage function learning reinforcement learning algorithm in four general test environments, respectively, in the embodiments of the present invention.

[0082] Figure 5(a) , 5(b) Figures 5(c) are schematic diagrams showing the changes in average cumulative reward, average completion time, and Bellman loss of the deep continuous action advantage function learning reinforcement learning algorithm in the trajectory planning environment in the embodiments of the present invention.

[0083] Figure 6(a) , 6(b) 6(c) and 6(d) are schematic diagrams of the planned trajectories of the UAV at four different target points in the embodiments of the present invention. Detailed Implementation

[0084] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0085] This invention proposes a UAV trajectory planning method based on continuous action advantage function learning. This method does not require a UAV dynamics model; it only requires continuous interaction between the UAV and the environment to train the trajectory planning strategy. The method includes the following steps:

[0086] Step 1) Construct the shortest time trajectory optimization problem for the drone;

[0087] The shortest time trajectory optimization problem for UAVs involves determining the objective function and constraints.

[0088] 1-1) Determine the objective function;

[0089] The objective function J of the shortest time trajectory optimization problem for UAVs is expressed as:

[0090]

[0091] in, This represents the mean operator, r k Let k represent the reward function of the drone, and k represent the k-th step of the state transition. f u is the number of steps to the finish line. k≥1 Let represent the UAV control input sequence with step number k≥1, and γ represent the decay factor of the weight of future loss, 0<γ<1;

[0092] 1-2) Determine the constraints;

[0093] The constraints for the trajectory optimization problem of a drone are expressed as follows:

[0094] s k+1 =g(s k a k ), k = 1, ..., kf k f The number of steps to the finish line;

[0095] Among them, s k and s k+1 Let a be the state vector of the UAV at step k and step (k+1), respectively. k Let g(·,·) represent the control variables at step k, and let g(·,·) represent the dynamic model of the UAV, i.e., the transfer model.

[0096] Step 2) Based on the shortest time trajectory optimization problem of the UAV constructed in Step 1), establish a Markov decision process model for UAV trajectory planning.

[0097] A Markov decision process model is represented by a quadruple, namely, state variables, control variables, transition model, and loss function; the specific steps for its construction are as follows:

[0098] 2-1) Determine the state variables;

[0099] This invention addresses the task of planning the shortest trajectory for unmanned aerial vehicles (UAVs). Therefore, a Cartesian coordinate system is established at the altitude of the UAV, with its initial position as the origin, and the state variable s... k The expression is:

[0100] s k =[ρ k θ k , ψ k f(p) k ,k)] T

[0101] Where, p k =(ρ k θ k , ψ k f(p) represents the spherical coordinate position of the UAV at the current k-th step. k ,k) represents the obstacle information obtained by the UAV at the current k-th step;

[0102] 2-2) Determine the control variables;

[0103] The control variable is expressed as the angular velocity α of the UAV. k =[ω k φ k ] T That is, (θ) k , ψ k Rate of change over time;

[0104] 2-3) Determine the transfer model;

[0105] The expression for the transfer model is:

[0106] sk+1 =g(s k a k ),

[0107] This transition model represents returning the state variables at the next time step, given the current k-step state variables and control variables.

[0108] 2-4) Determine the loss function;

[0109] The loss function is used to evaluate the drone state s k The control variable a is executed below. k The effect, expressed as:

[0110]

[0111] Where ρ and λ represent the weight coefficients of each item, respectively;

[0112] Δ d (k)=||(x k -x r y k -y r , z k -z r )′||-||(x k-1 -x r y k-1 -y r , z k-1 -z r )′||,

[0113] r s (k)=1 / ||(x k -x o y k -y o , z k -z o )′||

[0114] Where “′” indicates transpose; (x k y k , z k ) represents the current spherical coordinate position p of the UAV. k In the above Cartesian coordinate system, (x) r y r , z r (x) represents the target position of the UAV in a Cartesian coordinate system; o y o , z o ( ) represents the obstacle coordinates in a Cartesian coordinate system. When the drone approaches the obstacle to a certain range, the indicator function... The value is 1. Let the reward amount rk be a negative loss function, i.e., r k=--c(s) k a k );

[0115] Step 3) Design reinforcement learning algorithm and train planning strategy;

[0116] 3-1) Construct a reinforcement learning neural network that includes an evaluation network and a policy network;

[0117] The evaluation network and policy network share a two-layer fully connected network with the ReLU activation function. After the two fully connected layers, two branches are formed. One branch connects to a single-layer fully connected network, which together with the two fully connected layers forms the evaluation network. This single-layer fully connected network uses the Identity activation function. The other branch also connects to a single-layer fully connected network, which together with the two fully connected networks forms the policy network. This single-layer fully connected network uses the Tanh activation function.

[0118] Evaluating the network: using ω s ω c Parameterized evaluation network V(s) k |ω s ω c ), ω s ω represents the weight parameters shared with the policy network. c The evaluation network's unique weight parameters are represented by V(s). k |ω s ω c The input to ) is the state variable s k The output is the cumulative reward amount, and the function expression for the cumulative reward amount is as follows:

[0119]

[0120] Policy network: using ω s ω a Parameterize a policy network ω s The weight parameters shared by the evaluation network; ω a This represents the weight parameters unique to the policy network. The input to the policy network is the state variable s. k The direct output is the action mean μ(s) k |ω s ω a ) and covariance matrix L(s k |ω s ω a Thus, the control variable a is constructed. k , represented as:

[0121]

[0122] Among them, a k Given a mean of μ(s) k |ω s ω a ), covariance is ∑(s k |ω s ω c The multidimensional Gaussian distribution sampling is obtained; ∑(s) k |ω s ω c )=L(s k |ω s ω a )L(s k |ω s ω a ) T Its positive definiteness is guaranteed by Kolesky decomposition.

[0123] 3-2) The method for updating the evaluation network is to minimize the following Bellman residuals:

[0124]

[0125] Where, ω s ω a ω c These are the parameters for the shared layer, the policy network, and the evaluation network, respectively. D represents the storage of historical data (s). k a k ,c(s k a k ), s k+1 (s) replay memory; k a k )~D represents s k a k The data extracted from D; C(ω) s ω a ω c The expression for the TD error with respect to the target neural network is as follows:

[0126] C(ω s ω a ω c ) = r k+1 +γV′(s k+1 |ω s ω c )-V(s k |ω s ω c )-A(s k a k |ω s ω c )

[0127]

[0128] Y(s k a k ) = L -1 (s k |ω s ω c (a) k -μ(s k |ω s ω a ))

[0129] The target neural network has the same structure as the current reinforcement learning neural network, but its parameter updates are delayed compared to the current network. The evaluation network and policy network also have their own corresponding target networks; V′(s k+1 |ω s ω c A(s) represents the cumulative reward output of the evaluation network. k a k |ω s ω c Y(s) represents the advantage function estimate of the UAV taking the current action relative to other actions in the current state, approximated by a quadratic form. k a k ) is the advantage function A(s) k a k |ω s ω c Intermediate variables in the calculation process, V(s) k |ω s ω c (This is obtained from the output of the evaluation network;)

[0130] Step 4) The specific steps to obtain the shortest time trajectory plan for the UAV are as follows:

[0131] 4-1) Parameter settings;

[0132] Let the maximum number of training iterations for the evaluation network and policy network be M = 100000, the maximum number of update steps per training iteration be T = 1000, the size of the training set extracted from the experience replay be N = 80, the tracking ratio of the target neural network be β = 0.995, the synchronous update learning rate of the evaluation network and policy network be α = 0.001, and the decay factor be γ = 0.99.

[0133] 4-2) Initialization;

[0134] The weight parameters of the evaluation network and policy network, which share a portion of the network layers, are initialized to 0; the initialized policy network is denoted as the current policy network. The initialized evaluation network is denoted as the current evaluation network V(s). k|ω s ω c );

[0135] Let there be an empty experience queue set D, and set the size of the set to 1 million;

[0136] 4-3) To begin the iteration, let the initial number of iterations be m = 1;

[0137] 4-4) Initialize the time k = 0, and initialize the current position p of the UAV. k , obstacle position p o and target position p r ;

[0138] 4-5) Use the current policy network Generate the control variable at time k; input the control variable a k With state s k The drone returns the state variable s for the next time step through the transition model. k+1 The reward r for this transition step is calculated using the reward function. k+1 ;

[0139] 4-6) Record the data obtained in steps 4-4) to 4-5) as an empirical sample d. k =(s k a k rk +1 s k+1 ), and store in the experience queue D;

[0140] 4-7) Let N be the number of experience samples in the statistical experience queue D. D And determine: if N D If ≤ N, then proceed to step 4-5); if N R If the value is greater than N, proceed to steps 4-8.

[0141] 4-8) Randomly select N experience samples d1, d2, ..., d3 from the experience queue D. N ; Calculate a C(ω) using each empirical sample s ω a ω c The value of ), the i-th value is defined as C. i (ω s ω a ω c );

[0142] 4-9) Use

[0143]

[0144]

[0145]

[0146] Update the weight parameters ω of the evaluation network and the policy network. s ω a ω c ;

[0147] use Update the weight parameters of the target neural network Update the training count m using m+1;

[0148] The updated evaluation network and policy network are then denoted as the new evaluation network and the new policy network.

[0149] 4-10) Decision: If k > T or the current state of the UAV satisfies the termination condition ||p(k) - p d ||≤ξ p That is, the distance the drone travels to the desired location is less than the threshold ξ. p If the result is positive, proceed to step 4-11); otherwise, return to step 4-5) and update the step number k using k+1.

[0150] 4-11) Make a judgment: If m < M, then return to step 4-4), the UAV initializes again and starts a new trajectory planning task; otherwise, the iteration ends, and the current policy network at the time of iteration termination is used. The final policy network is denoted as... Input the state variables into the final policy The shortest time trajectory of the UAV is obtained by outputting the control variables.

[0151] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1.A method for unmanned aerial vehicle (UAV) trajectory planning based on continuous action advantage function learning, characterized in that, Comprise: Step 1), the shortest time trajectory optimization problem of unmanned aerial vehicle is constructed, including determining the objective function and constraint condition: 1-1) determining the objective function; The expression of the objective function J of the shortest time trajectory optimization problem of unmanned aerial vehicle is: wherein, denotes the mean operator, r k denotes a reward function for the drone, k denotes the kth step of state transition, k f is the terminal step number, u k≥1 denotes a sequence of drone control inputs for steps k≥1, γ denotes a decay factor for the weight of future losses, 0<γ<1 ; 1-2) the expression of the constraint condition of the trajectory optimization problem of unmanned aerial vehicle is: s k+1 = g(s k , a k ), k = 1,..., k f , k f is the number of steps to the end point; where s k and s k+1 are the state vectors of the UAV at the kth step and the k+1th step, respectively, a k denotes the control variable at the kth step, and the function g(·, ·) denotes the dynamics model of the UAV, i.e., the transition model; Step 2), according to the shortest time trajectory optimization problem of unmanned aerial vehicle constructed in step 1), a Markov decision process model of unmanned aerial vehicle trajectory planning is established; The Markov decision process model is represented by a four-tuple, namely state variable, control variable, transition model and loss function, and the specific steps of construction are as follows: 2-1) determining the state variable; A plane rectangular coordinate system is established at the height of the UAV, with its initial position as the origin, and the state variable s k The expression is: s k = [p k , θ k , ψ k , f(p k , k)] T where p k = (ρ k , θ k , ψ k ) represents the spherical coordinate position of the UAV at the current kth step, and f(p k , k) represents the obstacle information learned by the UAV at the current kth step. 2-2) determining the control variable; The control variable expression is the heading angular velocity a of the drone k = [ω k , φ k ] T , i.e. the rate of change of (θ k , ψ k ) over time; 2-3) the expression of the transition model is: s k+1 = g(s k , a k ); The transition model represents the state variable at the next time under the given current k step state variable and control variable; 2-4) determining the loss function; The loss function is used to evaluate the state s of the UAV k The effect of the control variable a is performed k The expression is: Where, ρ, λ respectively represent the weight coefficient of each term; Δ d (k) = ‖(x k -x r ,y k -y r ,z k -z r )′‖ - ‖(x k-1 -x r ,y k-1 -y r ,z k-1 -z r )′‖, r s (k) = 1 / ‖(x k -x o ,y k -y o ,z k -z o )′‖ wherein "′" represents transpose; (x k ,y k ,z k ) is the current spherical coordinate position p k of the UAV in the plane rectangular coordinate system; (x r ,y r ,z r ) is the target position of the UAV in the rectangular coordinate system; (x o ,y o ,z o ) is the obstacle coordinate in the rectangular coordinate system, and when the UAV approaches to within a certain range of the obstacle, the indicator function takes the value 1; and the reward amount r k is a negative loss function, i.e., r k =-c(s k ,a k ). Step 3), designing reinforcement learning algorithm and training planning strategy; 3-1) constructing reinforcement learning neural network including evaluation network and policy network; The evaluation network and the policy network share a two-layer fully connected network, which adopts ReLU activation function; After the two-layer fully connected network, two branches are separated, one of which connects a single-layer fully connected network, which, together with the two-layer fully connected network, constitutes the evaluation network, and the single-layer fully connected network adopts Identity activation function; The other branch also connects a single-layer fully connected network, which, together with the two-layer fully connected network, constitutes the policy network, and the single-layer fully connected network adopts Tanh activation function; Evaluation network: use ω s ,ω c Parameterize evaluation network V(s k |ω s ,ω c ),ω s Indicate weight parameters shared with policy network,ω c Indicate weight parameters unique to evaluation network; input of evaluation network V(s k |ω s ,ω c ) is state variable s k , and output is cumulative reward amount, and function expression of cumulative reward amount is as follows: Policy network: using ω s ,ω a Parameterize a policy network ω s The weight parameters shared by the evaluation network; ω a The weights represent the unique weight parameters of the policy network; the input to the policy network is the state variable s. k The direct output is the action mean μ(s) k |ω s ,ω a ) and covariance matrix L(s k |ω s ,ω a Thus, the control variable a is constructed. k , represented as: Among them, a k Given a mean of μ(s) k |ω s ,ω a ), covariance is Σ(s k |ω s ,ω c The multidimensional Gaussian distribution sampling is obtained; Σ(s) k |ω s ω c )=L(s k |ω s ω a )L(s k |ω s ω a ) T ; 3-2) the method of updating the evaluation network is to minimize the Bellman residual error as follows: where ω s ,ω a ,ω c are shared layer parameters, policy network parameters and evaluation network parameters, respectively; D is a replay memory storing historical data (s k ,a k ,c(s k ,a k ),s k+1 ); (s k ,a k ) ~ D indicates that s k ,a k is extracted from D; C(ω s ,ω a ,ω c ) represents a TD error expression of the target neural network: C(ω s ,ω a ,ω c )=r k+1 +γV′(s k+1 |ω s ,ω c )-V(s k |ω s ,ω c )-A(s k ,a k |ω s ,ω c ) Y(s k , a k ) = L -1 (s k |ω s ,ω c )(a k - μ(s k |ω s ,ω a )) Wherein, the target neural network is the same as the current reinforcement learning neural network structure, the parameter update of which is delayed compared with the current network, the evaluation network and the policy network also have their own target networks; V'(s k+1 |ω s ,ω c ) is the cumulative reward output by the target evaluation network; A(s k ,a k |ω s ,ω c ) represents the advantage function estimation of the UAV in the current state taking the current action compared with other actions, which is approximated in the form of a quadratic function, Y(s k ,a k ) is an intermediate variable in the calculation process of the advantage function A(s k ,a k |ω s ,ω c ), and V(s k |ω s ,ω c ) is obtained from the output of the evaluation network; Step 4), the specific steps of getting the shortest time trajectory planning of unmanned aerial vehicle are as follows: 4-1) parameter setting; Set the maximum training times M of the evaluation network and the policy network, the maximum update step T of each training, the training set size N extracted by experience replay, the tracking ratio β of the target neural network, the synchronous update learning rate α of the evaluation network and the policy network, and the decay factor γ; 4-2) initialization; The weight parameters of the evaluation network and the policy network sharing the number of layers are initialized as 0; the initialized policy network is denoted as a current policy network The initialized evaluation network is denoted as a current evaluation network V(s k |ω s ,ω c ) Set empty experience queue set D; 4-3) iteration starts, let the initialization iteration number be m = 1; 4-4) initialize time k = 0, initialize the current position of unmanned aerial vehicle, obstacle position and target position; 4-5) Using the current policy network Generate k-th control variable; input control variable a k With state s k The UAV returns the next time state variable s through the transition model k+1 Calculate the reward r of this step transition using the reward function k+1 ; 4-6) The data obtained in steps 4-4) to 4-5) are noted as an empirical sample d k = (s k , a k , r k+1 , s k+1 ), are stored in the empirical queue D; 4-7) Set the number of experience samples in the experience queue D to N D and determine: if N D ≤ N, go to step 4-5); if N R > N, go to step 4-8); 4-8) randomly draw N experience samples di, d2,..., dN from the experience queue D n ; compute a value of C(ω s ,ω a ,ω c ) using each experience sample, the i-th value defined as C i (ω s ,ω a ,ω c ); 4-9) use updating the weight parameters ω of the evaluation network and the policy network s ,ω a ,ω c ; Using updating the weight parameters of the target neural network using m+1 to update the training number m; And the updated evaluation network and policy network are recorded as new evaluation network and new policy network; 4-10) judgment: if k > T or the current state of unmanned aerial vehicle meets the set termination condition, step 4-11) is entered, otherwise step 4-5) is returned, and the k+1 update step k is used; 4-11) make a decision: if m < M, return to step 4-4) again, the UAV is initialized once more and starts a new trajectory planning task; otherwise, the iteration ends, and the current policy network at the end of the iteration is recorded as As the final policy network, it is denoted as Input the state variables into the final policy Output the control variables to obtain the shortest time trajectory of the UAV. 2.The UAV trajectory planning method based on continuous action advantage function learning of claim 1, wherein, In the 4-10), the termination condition set is that the distance of the UAV to the desired position is less than a set threshold value ξ p .

Images