A deep deterministic policy gradient control method for UAV trajectory tracking and state compensation

Abstract

The present invention discloses a state-compensated deep deterministic policy gradient control method for unmanned aerial vehicle trajectory tracking. The method adopts a deep deterministic policy gradient algorithm. The network structure includes a control network, an evaluation network, and a compensation network. The control quantity output of the control network is combined with the compensation output of the compensation network as the output control quantity to interact with the environment. The evaluation network evaluates the value of the output control quantity, so that the unmanned aerial vehicle can quickly track dynamic targets in the most accurate, continuous, and smooth manner. Random noise is added on the basis of generating the control quantity to achieve a certain range of exploration, so that the control quantity value estimation is more accurate. Aiming at the low training efficiency and unstable convergence of the unmanned aerial vehicle trajectory tracking control algorithm based on deep reinforcement learning in an unknown environment, the present invention proposes a state-compensated deep deterministic policy gradient algorithm combined with a state compensation network, which can significantly improve the training efficiency and convergence stability of the unmanned aerial vehicle in trajectory tracking tasks.

Description

Technical Field

[0001] The present invention relates to a robot trajectory tracking technology, and in particular to a drone trajectory tracking state compensation deep deterministic strategy gradient control method. Background Art

[0002] For mobile robots, trajectory following is related to time series. The robot must reach the originally set position through the control of the trajectory following system within a specified time. Therefore, a controller with high-performance trajectory following capability is needed by mobile robots. Designing a controller for a mobile robot is challenging. The uncertainty of the dynamic model, strong environmental disturbances, under-actuation of the model, and incomplete kinematic constraints are the problems that designers must deal with when designing a mobile robot trajectory following controller. Its complex sensor suite and multi-layer software will bring noise and delay to information transmission, and traditional control theory is often not enough to effectively solve these problems. The development of specialized control methods to solve this complex problem usually requires a long design process and arduous parameter adjustments. Learning-based robot control methods are the latest research hotspot in the field of control. They ignore the robot's dynamic model and learn control methods through a large amount of motion data. At present, they have received widespread attention in the field of automatic control.

[0003] Without knowing the dynamic model of the environment, model-free reinforcement learning algorithms can directly evaluate the quality of the strategy, or find the optimal value function and the optimal strategy through the actual interaction between the individual and the environment. A lot of research has been done on this issue from theory to experiment, and fruitful results have been achieved in theoretical analysis, numerical calculation and experimental verification. Among them, value-based deep reinforcement learning algorithms such as Q-learning, Sarsa and deep Q-learning algorithm (DQN) can only achieve control of discrete control quantity space, so they can only achieve discrete direction control of the robot. However, in the case of large-scale control quantity space or continuous control quantity, it is difficult for value-based reinforcement learning to obtain good results. For trajectory tracking, value-based control methods only use discrete control quantity space. It is difficult to achieve accurate tracking. In this case, strategy learning can be performed directly. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a state-compensated deep deterministic policy gradient control method for UAV trajectory tracking. This method adopts deep reinforcement learning and integrates the state-compensated deep deterministic policy gradient algorithm of different state-space networks, which can significantly improve the training efficiency and convergence stability.

[0005] Technical solution: A drone trajectory tracking state compensation deep deterministic policy gradient control method of the present invention comprises the following steps:

[0006] S1. Establishing the Markov decision process tuple for UAV dynamic tracking<S,A,P,R,γ> , S is a finite state set, A is a finite control quantity set, P is a state transition probability matrix based on the control quantity, R is a reward function based on the state and control quantity, and γ is a decay factor;

[0007] S2, establish the motion state equation of UAV dynamic tracking;

[0008] S3, respectively constructing a first control network, a first evaluation network, a first compensation network, and corresponding second control network, second evaluation network, and second compensation network having the same network structure;

[0009] S4. Execute each network to obtain the state transfer process, and store all state transfer processes into the experience replay pool;

[0010] Set the current state of the drone t The output of the first control network is input into the first compensation network, and the output of the first control network is added to the output of the first compensation network and random noise is added to the output of the first control network. The output is used as the first control amount a to interact with the environment to obtain the reward R. t+1 , and make the drone enter the next state s t+1 , guides the UAV to track the dynamic target, and takes the first control quantity as the input of the first evaluation network. The first evaluation network evaluates the current state of the UAV according to s t and the first control quantity to evaluate the value, and output the first evaluation value; the next state of the UAV s t+1 The output of the second control network is input into the second compensation network, and the output of the second control network is added to the output of the second compensation network as the second control quantity and input into the second evaluation network. The second evaluation network is based on the next state s of the UAV. t+1 The second control amount is evaluated and the second evaluation value is output. The second evaluation value is calculated with the reward to obtain the target value Q Target ; Then the system repeats the above process and sets the next state of the drone to s t+1 As the current state s in the next process t Carry out the subsequent process and convert the s t ,a,R t+1 ,s t+1 Described as a transfer process, multiple transfer processes generated after multiple environmental interactions are all represented by (s t ,a,R t+1 ,s t+1 ) indicates that all state transfer processes (s t ,a,Rt+1 ,s t+1 ) is stored in the experience replay pool, s t+1 is the current state t The next state obtained after executing the first control quantity, R t+1 It is the reward obtained in the process;

[0011] S5, randomly batch sample X transfer processes from the experience replay pool (s i ,a i ,R i+1 ,s i+1 ), i = 1, 2, ..., X is the sampling order, the time difference error between each first evaluation value and the target value is calculated, each network parameter is updated through the target gradient function until it meets the training target, and each network model is saved;

[0012] S6. When the UAV performs a trajectory tracking task, the target tracking position is set, the current state of the UAV is input into the trained first control network model, and then the output control quantity of the network model is executed to update the UAV state, and this process is repeated to complete the target tracking task.

[0013] Furthermore, the Markov decision process tuple of the UAV dynamic tracking in step S1 is<S,A,P,R,γ> Specifically:

[0014]

[0015] Where S is a finite state set, s t is the current state, is a set of states belonging to the state set S, p x 、p y 、p z They are the position coordinates of the drone in the three coordinate axis directions in the Cartesian coordinate system. They are the speeds of the drone in the three coordinate axes in the Cartesian coordinate system, t x ,t y ,t z are the target position coordinates of the UAV in the three coordinate axis directions in the Cartesian coordinate system, A is a finite control quantity set, a is a set of control quantity representations in the control quantity set A, They are the acceleration of the UAV in the three coordinate axes in the Cartesian coordinate system, P is the state transition probability matrix based on the control quantity, represents a set of transition probabilities in the state transition probability matrix P, Indicates that from the current state s t After the control amount a, transfer to the next state s t+1 The transfer probability, the reward obtained in this process is R t+1, the subscripts t and t+1 in the above variables represent the current time step and the next time step respectively, the tuple<S,A,P,R,γ> R in the example is a reward function based on the state and control volume, Represents a set of reward values ​​in the reward function R, that is, from the current state s t The reward obtained after the control amount a, γ is the decay factor, S t is the finite state set at the current time step t, A t is a finite set of control quantities at the current time step t.

[0016] Furthermore, the motion state equation of the UAV dynamic tracking in step S2 is:

[0017]

[0018] Among them, π θ (s t ,a) is the UAV trajectory tracking control strategy, θ is the strategy parameter, P[A t =a|S t =s t ,θ] is in a given state s t ∈S and the probability distribution matrix of executing the control amount a under the policy parameter θ, f(S t =s t ,A t =a|π θ (s t ,a)) is to continuously update s through control methods t ,a,s t+1 The dynamic equation obtained is, t A random noise function is used to explore a small range around the control quantity.

[0019] Furthermore, the UAV adopts trajectory tracking control strategy π θ (s t ,a)Execution control quantity Post Updates t ,a,s t+1 , Where (p′ x ,p′ y ,p′ z ) is the drone in state s t+1 The position at For state s t+1 The speed at time (t x ,t y ,t z ) is used to track the target position, and the update formula is:

[0020]

[0021] Furthermore, in step S3, the first control network and the second control network have the same structure, both including an input layer, three hidden layers and an output layer, the input layer is used to receive the drone status data, and then the output passes through its first hidden layer, the second hidden layer, the third hidden layer in sequence, and then outputs through the output layer, and the above layers are all fully connected;

[0022] The first evaluation network and the second evaluation network have the same structure, both of which include two input layers, four hidden layers and one output layer. One input layer receives the drone status data, and then the output passes through its first hidden layer and second hidden layer in sequence and then inputs into the fourth hidden layer. Another input layer receives the control amount, and then the output passes through the third hidden layer and then inputs into the fourth hidden layer. The output of the fourth hidden layer outputs the evaluation value through the output layer. All the above layers are fully connected.

[0023] The first compensation network and the second compensation network have the same structure, both of which include an input layer, three hidden layers and an output layer. The input layer receives the drone status data on the one hand, and receives the output of the first control network or the second control network on the other hand. The output then passes through its first hidden layer, second hidden layer, and third hidden layer in sequence, and then is output through the output layer. All of the above layers are fully connected.

[0024] Furthermore, the calculation formula of the timing difference error value Loss in step S5 is:

[0025]

[0026] Among them, Q Target is the target value, Q Target =R i+1 +γQ′(s i ,A′(s i |θ A′ ,ω C′ )), γ is the attenuation factor, Q′(s i ,A′(s i |θ A′ ,ω C′ )) is the value of the second evaluation value after sampling, Q(s i ,a i |θ Q ,ω C ) is the value of the first evaluation value after sampling, θ Q ,θ A and ω C They are respectively the value function Q(s) of the first evaluation network t ,a|θ Q ), the control quantity function A(s) of the first control network t |θ A ) and the function C(s) of the first compensation networkt |ω C ) parameters, θ Q′ ,θ A′ and ω C′ are the value function Q′(s t ,a|θ Q′ ), the control quantity function A′(s t |θ A′ ) and the function C′(s t |ω C′ ) parameters.

[0027] Furthermore, in step S5, the target gradient function The formula is:

[0028]

[0029] in, is the symbol of the gradient function with a as the independent variable, is θ A is the symbol of the gradient function of the independent variable, Is ω C is the symbol of the gradient function of the independent variable;

[0030] The network parameters are updated as follows:

[0031] Using the time difference error value Loss and the target gradient function The gradient ascent method is used to update the first evaluation network parameter θ using the Adam optimizer in the Python machine learning library Q and the first control network parameter θ A ; Use the first evaluation network parameter θ Q and the first control network parameter θ A The second evaluation network parameters and the second control network parameters are slightly updated, and the formula is:

[0032]

[0033] Among them, η∈[0,1] is the update parameter;

[0034] The gradient descent method is used to update the compensation network parameters. The parameter update formula is:

[0035]

[0036] Among them, the parameter update amount is △ω, and ε∈[0,1] is the learning rate of the compensation network.

[0037] A drone trajectory tracking state compensation deep deterministic policy gradient control system of the present invention comprises:

[0038] Tuple building module for building Markov decision process tuples for UAV dynamic tracking<S,A,P,R,γ> , S is a finite state set, A is a finite control quantity set, P is a state transition probability matrix based on the control quantity, R is a reward function based on the state and control quantity, and γ is a decay factor;

[0039] The motion state equation building module is used to build the motion state equation for the UAV dynamic tracking, including the state of the next time step and the UAV trajectory tracking control strategy;

[0040] A network construction module, used to construct a first control network, a first evaluation network, a first compensation network, and corresponding second control network, second evaluation network, and second compensation network having the same network structure;

[0041] An execution control module is used to execute each network and obtain multiple state transfer processes;

[0042] The network parameter update module is used to update each network parameter through the target gradient function until it meets the training objectives.

[0043] A device of the present invention includes a memory and a processor, wherein:

[0044] A memory for storing computer programs that can be run on the processor;

[0045] A processor is used to execute the steps of the above-mentioned drone trajectory tracking state compensation deep deterministic policy gradient control method when running the computer program.

[0046] A storage medium of the present invention stores a computer program, and when the computer program is executed by at least one processor, the steps of the above-mentioned drone trajectory tracking state compensation deep deterministic policy gradient control method are implemented.

[0047] Beneficial effects: Compared with the prior art, the present invention has the following technical effects:

[0048] (1) The method of the present invention combines the action output of the control network with the compensation output of the compensation network as the output action for interacting with the environment by adding a compensation network, so that the UAV can quickly track the dynamic target in the most accurate, continuous and smooth manner; in addition, adding random noise on the basis of generating the control amount realizes a certain range of exploration, making the control amount estimation more accurate;

[0049] (2) The proposed method was verified using the OpenAI Gym tool. Simulation results showed that the method can significantly improve training efficiency, effectively improve accuracy and convergence stability; under the same computer configuration, the computational cost of the proposed method is basically the same as other reinforcement learning algorithms, and better tracking effect is achieved without increasing the computational cost;

[0050] (3) During the training process, the algorithm designed by the method of the present invention sets the target point to randomly change its spatial position, so that any target position within the specified spatial range can be achieved. The UAV will quickly and smoothly approach the target point to achieve trajectory tracking training for the UAV. During the training process, under the same training time, the tracking error after stabilization is reduced by half compared with the ordinary reinforcement learning algorithm based on policy learning. Under the same tracking accuracy, the algorithm designed by the present invention is significantly higher in learning efficiency than other similar reinforcement learning algorithms and has higher convergence stability, indicating that the algorithm has a certain generalization ability and can be trained according to different environments and applied to trajectory tracking tasks in multiple fields such as unmanned vehicles, UAVs, mobile robots, and multi-legged robots.

[0051] (4) For the UAV trajectory tracking control problem of the model unknown system, reinforcement learning requires a series of iterative processes in the learning process. The initial strategy training cycle is long, resulting in a large amount of training time required for controller development. The algorithm proposed in the present invention overcomes the above shortcomings of similar reinforcement learning algorithms, can significantly speed up the training time, improve learning efficiency, and stabilize the balance between exploration and utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flow chart of the method of the present invention;

[0053] Figure 2 A schematic diagram of the network structure designed for the present invention;

[0054] Figure 3 This is a schematic diagram showing the principle of the method of the present invention;

[0055] Figure 4 Schematic diagram of the execution flow of step 4 and step 5 in the method of the present invention;

[0056] Figure 5 A comparison chart of rewards between the method of the present invention and a common strategy learning algorithm during the training process;

[0057] Figure 6 It is a comparison chart of the iteration time of the method of the present invention and the common strategy learning algorithm during the training process;

[0058] Figure 7 A comparison chart of the operation time of the method of the present invention and the common strategy learning algorithm during the training process;

[0059] Figure 8 A comparison diagram of tracking trajectories of different algorithms for the UAV in three-dimensional space in the embodiment;

[0060] Fig. 9 This is a comparison chart of tracking errors generated by different algorithms during the three-dimensional space trajectory tracking of the UAV in the embodiment. DETAILED DESCRIPTION

[0061] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the scope of protection of the present invention.

[0062] In view of the shortcomings of the UAV trajectory tracking control algorithm based on deep reinforcement learning, such as low training efficiency and unstable convergence in unknown environments, the present invention proposes a UAV trajectory tracking control method with the breakthrough point of improving learning efficiency without reducing training accuracy, namely: using deep reinforcement learning and integrating the state compensation deep deterministic policy gradient algorithm of different state space networks, which can significantly improve the training efficiency and convergence stability.

[0063] like Figure 1 As shown, a drone trajectory tracking state compensation deep deterministic policy gradient control method of the present invention comprises the following steps:

[0064] Step 1: Establish the Markov decision process tuple for UAV dynamic tracking<S,A,P,R,γ> :

[0065]

[0066] Among them, the tuple<S,A,P,R,γ> S in the formula (a) is a finite state set. t is a set of state representations in the state set S, where p x 、p y 、p z They are the position coordinates of the drone in the three coordinate axis directions in the Cartesian coordinate system. They are the speeds of the drone in the three coordinate axes in the Cartesian coordinate system, t x ,t y ,t z They are the target position coordinates of the drone in the three coordinate axis directions in the Cartesian coordinate system. Describes the current position of the drone (p x ,p y ,p z ),speed and target position (t x ,t y,t z ) composed of drone status, tuple<S,A,P,R,γ> A in the formula (a) is a finite set of control quantities, and a is a set of control quantities in the control quantity set A. They represent the acceleration of the drone in the three coordinate axes in the Cartesian coordinate system. Describes the control components executed in the three Cartesian coordinate axes under the control of the system, the tuple<S,A,P,R,γ> The P in the formula (a) is the state transition probability matrix based on the control quantity. represents a set of transition probabilities in the state transition probability matrix P, Indicates that from the current state s t After the control amount a, transfer to the next state s t+1 The transfer probability, the reward obtained in this process is R t+1 , the subscripts t and t+1 in the above variables represent the current time step and the next time step respectively, the tuple<S,A,P,R,γ> The R in the formula is a reward function based on the state and control amount. Represents a set of reward values ​​in the reward function R, that is, from the current state s t The reward obtained after the control amount a, tuple<S,A,P,R,γ> γ in the equation is the attenuation factor, S t is the finite state set at the current time step t, A t is a finite set of control quantities at the current time step t.

[0067] Step 2: Establish the motion state equation for UAV dynamic tracking:

[0068]

[0069] In formula (b), π θ (s t ,a) is the UAV trajectory tracking control strategy, θ is the strategy parameter, P[A t =a|S t =s t ,θ] is in a given state s t ∈S and the probability distribution matrix of executing the control amount a under the policy parameter θ, f(S t =s t ,A t =a|π θ (s t ,a)) is to continuously update s through the control method of the present invention t ,a,s t+1 The dynamic equation obtained is, t A random noise function is used to explore a small range around the control quantity.

[0070] The UAV adopts trajectory tracking control strategy π θ (s t ,a)Execution control quantity Post Updates t ,a,s t+1 , Where (p′ x ,p′ y ,p′ z ) is the drone in state s t+1 The position and For state s t+1 The speed at time (t x ,t y ,t z ) is the tracking target position, and the updating method is formula (c).

[0071]

[0072] Step 3, respectively constructing a first control network, a first evaluation network, a first compensation network, and corresponding second control network, second evaluation network, and second compensation network having the same network structure;

[0073] like Figure 2 As shown, the first control network and the second control network have the same structure, both including an input layer, three hidden layers and an output layer. The input layer is used to receive the drone status data, and then the output passes through its first hidden layer, second hidden layer, third hidden layer in sequence, and then outputs through the output layer. All of the above layers are fully connected.

[0074] The first evaluation network and the second evaluation network have the same structure, both of which include two input layers, four hidden layers and one output layer. One input layer receives the drone status data, and then the output passes through its first hidden layer and second hidden layer in sequence and then inputs the fourth hidden layer. The other input layer receives the control amount, and then the output passes through the third hidden layer and then inputs the fourth hidden layer. The output of the fourth hidden layer outputs the evaluation value through the output layer. All of the above layers are fully connected.

[0075] The first compensation network and the second compensation network have the same structure, both of which include an input layer, three hidden layers and an output layer. The input layer receives the drone status data on the one hand, and receives the output of the first control network or the second control network on the other hand. The output then passes through its first hidden layer, second hidden layer, and third hidden layer in sequence, and then is output through the output layer. All of the above layers are fully connected.

[0076] The input received by the evaluation network is the state s and control amount a detected by the drone. The hidden layers that process the state and control amount are operated separately first. The input state passes through two hidden layers with 256 and 128 neurons respectively, and the input control amount passes through a hidden layer with 128 neurons. The two are fully connected through the last hidden layer with 128 neurons, and then the evaluation value is output together; the input received by the control network is the state s detected by the drone, and the specific value of each control amount is output; the input of the compensation network is the output of the control network and the state detected by the drone. The output of the compensation network is added to the noise function and the output of the control network as the control amount a executed by the drone. The control network and the compensation network are designed as 3 hidden layers (the number of neurons is 256, 128, and 64 respectively); the above layers are all fully connected, and the activation function used is relu or tanh (such as Figure 2 shown).

[0077] Step 4: Execute each network to obtain the state transfer process, and store all state transfer processes in the experience replay pool;

[0078] like Figure 3 As shown, the current state of the drone is s t The outputs of the first control network and the first compensation network are respectively inputted, and the outputs of the first control network and the first compensation network are added and random noise is added to them as the first control quantity a to interact with the environment to obtain the reward R t+1 , and make the drone enter the next state s t+1 , guides the UAV to track the dynamic target, and takes the first control quantity as the input of the first evaluation network. The first evaluation network evaluates the current state of the UAV according to s t And the first control quantity is evaluated, and the first evaluation value is output; in the same way, the next state of the drone s t+1 The output of the second control network is input into the second compensation network, and the output of the second control network is added to the output of the second compensation network as the second control quantity and input into the second evaluation network. The second evaluation network is based on the next state s of the UAV. t+1 The second control amount is evaluated and the second evaluation value is output. The second evaluation value is calculated with the above reward to obtain the target value Q Target , then the system will repeat the above process and set the next state of the drone to s t+1 As the current state s in the next process t Carry out the subsequent process; t ,a,R t+1 ,s t+1 Described as a transfer process, multiple transfer processes generated after multiple environmental interactions are all represented by (s t,a,R t+1 ,s t+1 ) indicates that all state transfer processes (s t ,a,R t+1 ,s t+1 ) is stored in the experience replay pool, s t+1 is the state t The next state obtained after executing the control amount a, R t+1 It is the reward obtained in the process.

[0079] Step 5: Randomly batch sample X transfer processes (s i ,a i ,R i+1 ,s i+1 ), i = 1, 2, ..., X is the sampling order, calculate the temporal difference (TD) error of each first evaluation value and target value, update each network parameter through the target gradient function, update the second evaluation network parameter and the second control network parameter in a small amount until the training target is met, and save each network model;

[0080] The compensation network of the auxiliary network exploration training is a neural network whose input is the drone state and output is the compensation control amount; let θ Q ,θ A and ω C They are respectively the value function Q(s) of the first evaluation network t ,a|θ Q ), the control quantity function A(s) of the first control network t |θ A ) and the function C(s) of the first compensation network t |ω C ) parameters, θ Q′ ,θ A′ and ω C′ is the value function Q′(s t ,a|θ Q′ ), the control quantity function A′(s t |θ A′ ) and the function C′(s t |ω C′ ) parameter, and the value of the first evaluation value after sampling is Q(s i ,a i |θ Q ,ω C ), the value of the second evaluation value is Q′(s i ,A′(s i |θ A′ ,ω C′ )), then the timing difference error value Loss is:

[0081]

[0082] Where, the target value Q in formula (d) Target =R i+1 +γQ′(s i ,A′(s i |θ A′ ,ω C′ )), γ is the attenuation factor. The target gradient function Formula (e):

[0083]

[0084] in, is the symbol of the gradient function with a as the independent variable, is θ A is the symbol of the gradient function of the independent variable, So ω C is the symbol of the gradient function of the independent variable.

[0085] The network parameters are updated as follows:

[0086] Using the time difference error value Loss and the target gradient function , using the gradient ascent method and the Adam optimizer in the Python machine learning library to update the first evaluation network parameters θ Q and the first control network parameter θ A ;

[0087] Use the first evaluation network parameter θ Q and the first control network parameter θ A The second evaluation network parameters and the second control network parameters are slightly updated, and the formula is:

[0088]

[0089] Among them, η∈[0,1] is the update parameter;

[0090] The gradient descent method of formula (g) is used to update the parameters of the first compensation network:

[0091]

[0092] Among them, the parameter update amount is △ω, is the above objective gradient function, and ε∈[0,1] is the learning rate of the compensation network.

[0093] The execution flow diagram of step 4 and step 5 is as follows: Figure 4 shown.

[0094] (1) Initialization; including:

[0095] Initializing a first control network, a first evaluation network, a first compensation network, a second control network, a second evaluation network, and a second compensation network;

[0096] Initialize the experience replay pool;

[0097] Randomly initialize the noise function and drone state s t ;

[0098] (2) Network training: network model training is performed according to the above steps 4 and 5;

[0099] First, the drone executes the control quantity a;

[0100] Get the next state of the drone t+1 and reward R t+1 ;

[0101] Storage of drone state transfer process (s t ,a,R t+1 ,s t+1 );

[0102] Calculate the target value and update the parameters of the control network, evaluation network, and compensation network through the time difference error value and the target gradient function;

[0103] Determine whether the training goal is met, that is, determine whether the set number of training times or the set cumulative reward value is reached. If so, save the training model and conduct a simulation experiment; otherwise, return to re-execute step (2).

[0104] Step 6: When the UAV performs a trajectory tracking task, the target tracking position is set, the current state of the UAV is input into the trained first control network model, and then the output control quantity of the network model is executed to update the UAV state, and this process is repeated to complete the target tracking task.

[0105] A drone trajectory tracking state compensation deep deterministic policy gradient control system of the present invention comprises:

[0106] Tuple building module for building Markov decision process tuples for UAV dynamic tracking<S,A,P,R,γ> ;

[0107] The motion state equation building module is used to build the motion state equation for the UAV dynamic tracking, including the state of the next time step and the UAV trajectory tracking control strategy;

[0108] A network construction module, used to construct a first control network, a first evaluation network, a first compensation network, and corresponding second control network, second evaluation network, and second compensation network having the same network structure;

[0109] An execution control module is used to execute each network and obtain multiple state transfer processes;

[0110] The network parameter update module is used to update each network parameter through the target gradient function until it meets the training objectives.

[0111] A device of the present invention includes a memory and a processor, wherein:

[0112] A memory for storing computer programs that can be run on the processor;

[0113] The processor is used to execute the steps of the above-mentioned drone trajectory tracking state compensation deep deterministic policy gradient control method when running the computer program, and can achieve the same technical effect as the above-mentioned method.

[0114] A storage medium of the present invention stores a computer program, which, when executed by at least one processor, implements the steps of the above-mentioned drone trajectory tracking state compensation deep deterministic policy gradient control method and can achieve the same technical effect as the above-mentioned method.

[0115] Embodiment: To verify that the method disclosed in the present invention can significantly improve the training efficiency and convergence stability by adopting the state-compensated deep deterministic policy gradient method integrating different state-space networks, the method of the present invention is simulated through simulation experiments. The experimental operation platform is configured as: Intel (R) Core (TM) i5-7300HQ, simulation toolkit: OpenAI Gym. Two groups of simulation comparison experiments are carried out in this implementation case: 1) The deep deterministic policy gradient (DDPG) algorithm is used as the comparison benchmark. Since the algorithm of the present invention is a state-compensated deep deterministic policy gradient control method, it is referred to as the CDDPG algorithm for short; 2) Since DDPG is developed from the value-based Actor-critic (QAC) algorithm, the QAC algorithm is also used as a comparison benchmark in this implementation case. After being improved according to the algorithm of the present invention, it becomes a state-compensated QAC algorithm, referred to as CQAC. The above four algorithms use the absolute root mean square error of the absolute trajectory error as a quantitative evaluation indicator.

[0116] The simulation environment sets the flight range of the drone to a 10×10×10(m) three-dimensional space, and sets the tracking training accuracy r to 0.3m. During the training process, the drone is considered to be successfully followed when it is within 0.3m of the target point. Each successful follow-up is considered to be a complete experience. During this process, the target point will randomly change its spatial position to achieve trajectory tracking training for the drone. Four algorithms, QAC, DDPG, CQAC and CDDPG, are used to track the target point. A complete experience contains multiple iterations. The reward after each iteration is the negative value of the linear distance between the tracking point and the target point. Each reward is defined as the total reward of all iterations in a complete experience, so the maximum reward is 0. Set the learning weights γ=0.999, η=ε=0.001 and simulate.

[0117] After completing 1000 training steps, save all training models to prepare for subsequent effect verification. Figure 5 The rewards for 1000 experiences are shown. It can be seen that starting from the 11th experience, the rewards are close to zero, but from the previous rewards, the CQAC algorithm and CDDPG algorithm have significantly improved compared with QAC and DDPG during the training process. Figure 6 From the total number of iterations of each algorithm, we can see that in the early stage of training, QAC and DDPG algorithms need tens of thousands of iterations to complete each successful follow-up, while CQAC and CDDPG require only 30% of the iteration steps of the original algorithm. Figure 7 It can be seen that under the same computer configuration, the computational costs of the four algorithms are basically the same (about 0.998 milliseconds), which fully demonstrates that the training time is improved by about 70% with the same training accuracy.

[0118] In order to verify the dynamic tracking effect of the above training model, this embodiment designs a simulation experiment to track the target point as accurately as possible in the Cartesian coordinate system to complete the spiral trajectory motion. In this simulation, the target point completes the predetermined trajectory tracking task in a uniform motion. Since its angular velocity on the XY plane is π / 15rad / s, and its rising speed on the z-axis is π / 30m / s, it completes two spiral motions with a radius of 3m and a pitch of πm in space, and it takes 60s to complete the trajectory tracking. By adopting four algorithm models, Figure 8 The trajectory tracking effect is shown. Visually, the CQAC and CDDPG algorithms are closer to the reference trajectory than the QAC and DDPG algorithms. To further illustrate the superiority of the proposed algorithm, Fig. 9The tracking errors of the four algorithms are shown. Obviously, due to the addition of the compensation network, the tracking accuracy and convergence stability of the method of the present invention have been effectively improved, and the stable tracking error is reduced by about 50% compared with the original algorithm. The results show that the compensation network has stronger active control capabilities during the flight of the UAV, reduces the position error, and improves the tracking effect.

[0119] The embodiments show that the state-compensated deep deterministic policy gradient control method for UAV trajectory tracking proposed in the present invention can significantly improve the training efficiency and convergence stability; under the same computer configuration, the computational cost is basically the same as other reinforcement learning algorithms, and better tracking effect is achieved without increasing the computational cost; 3) During the training process, under the condition of the same tracking accuracy, the learning efficiency is higher; under the same training time, the stabilization tracking error is reduced by half compared with the ordinary reinforcement learning algorithm based on policy learning.

Claims

1. A deep deterministic policy gradient control method for UAV trajectory tracking state compensation, characterized in that: The following steps are involved: S1. Establishing the Markov decision process tuple for UAV dynamic tracking<S,A,P,R,γ> , S is a finite state set, A is a finite control quantity set, P is a state transition probability matrix based on the control quantity, R is a reward function based on the state and control quantity, and γ is a decay factor; S2, establish the motion state equation of UAV dynamic tracking; S3, respectively constructing a first control network, a first evaluation network, a first compensation network, and corresponding second control network, second evaluation network, and second compensation network having the same network structure; S4. Execute each network to obtain the state transfer process, and store all state transfer processes into the experience replay pool; Set the current state of the drone t The output of the first control network is input into the first compensation network, and the output of the first control network is added to the output of the first compensation network and random noise is added to the output of the first control network. The output is used as the first control amount a to interact with the environment to obtain the reward R. t+1 , and make the drone enter the next state s t+1 , guides the UAV to track the dynamic target, and takes the first control quantity as the input of the first evaluation network. The first evaluation network evaluates the current state of the UAV according to s t Performing value evaluation with the first control quantity and outputting a first evaluation value; Set the drone to the next state s t+1 The output of the second control network is input into the second compensation network, and the output of the second control network is added to the output of the second compensation network as the second control quantity and input into the second evaluation network. The second evaluation network is based on the next state s of the UAV. t+1 The second control amount is evaluated and the second evaluation value is output. The second evaluation value is calculated with the reward to obtain the target value Q Target ; Then the system repeats the above process and sets the next state of the drone to s t+1 As the current state s in the next process t Carry out the subsequent process and convert the s t ,a,R t+1 ,s t+1 Described as a transfer process, multiple transfer processes generated after multiple environmental interactions are all represented by (s t ,a,R t+1 ,s t+1 ) indicates that all state transfer processes (s t ,a,R t+1 ,s t+1 ) is stored in the experience replay pool; S5, randomly batch sample X transfer processes from the experience replay pool (s i ,a i ,R i+1 ,s i+1 ), i = 1, 2, ..., X is the sampling order, the time difference error between each first evaluation value and the target value is calculated, each network parameter is updated through the target gradient function until it meets the training target, and each network model is saved; S6. When the UAV performs a trajectory tracking task, the target tracking position is set, the current state of the UAV is input into the trained first control network model, and then the output control quantity of the network model is executed to update the UAV state, and this process is repeated to complete the target tracking task.

2. The method for controlling the state compensation of the trajectory tracking of unmanned aerial vehicles according to claim 1 is characterized in that: The Markov decision process tuple for the UAV dynamic tracking in step S1<S,A,P,R,γ> Specifically: Among them, s t is the current state, is a set of states belonging to the state set S, p x 、p y 、p z They are the position coordinates of the drone in the three coordinate axis directions in the Cartesian coordinate system. They are the speeds of the drone in the three coordinate axes in the Cartesian coordinate system, t x ,t y ,t z are the target position coordinates of the UAV in the three coordinate axis directions in the Cartesian coordinate system, a is a set of control quantity representations in the control quantity set A, They are the accelerations of the drone in the three coordinate axes in the Cartesian coordinate system. represents a set of transition probabilities in the state transition probability matrix P, Indicates that from the current state s t After the control amount a, transfer to the next state s t+1 The transfer probability, the reward obtained in this process is R t+1 , the subscripts t and t+1 in the above variables represent the current time step and the next time step respectively. Represents a set of reward values ​​in the reward function R, that is, from the current state s t The reward obtained after controlling the amount a, S t is the finite state set at the current time step t, A t is a finite set of control quantities at the current time step t.

3. The method for controlling the state compensation of the trajectory tracking of unmanned aerial vehicles according to claim 1 is characterized in that: The motion state equation of the UAV dynamic tracking in step S2 is: Among them, π θ (s t ,a) is the UAV trajectory tracking control strategy, θ is the strategy parameter, P[A t =a|S t =s t ,θ] is in a given state s t ∈S and the probability distribution matrix of executing the control amount a under the policy parameter θ, f(S t =s t ,A t =a|π θ (s t ,a)) is to continuously update s through control methods t ,a,s t+1 The dynamic equation obtained is, t A random noise function is used to explore a small range around the control quantity.

4. The method for controlling the state compensation of the trajectory tracking of unmanned aerial vehicles according to claim 3 is characterized in that: The UAV adopts trajectory tracking control strategy π θ (s t ,a)Execution control quantity Post Updates t ,a,s t+1 , Among them (p x ′,p y ′,p z ′) is the state of the drone in s t+1 The position at For state s t+1 The speed at time (t x ,t y ,t z ) is used to track the target position, and the update formula is: and 5. The method for controlling the state compensation of the trajectory tracking of unmanned aerial vehicles according to claim 1 is characterized in that: In step S3, the first control network and the second control network have the same structure, both including an input layer, three hidden layers and an output layer. The input layer is used to receive the drone status data, and then the output passes through its first hidden layer, second hidden layer, third hidden layer in sequence, and then outputs through the output layer. All the above layers are fully connected. The first evaluation network and the second evaluation network have the same structure, both of which include two input layers, four hidden layers and one output layer. One input layer receives the drone status data, and then the output passes through its first hidden layer and second hidden layer in sequence and then inputs into the fourth hidden layer. Another input layer receives the control amount, and then the output passes through the third hidden layer and then inputs into the fourth hidden layer. The output of the fourth hidden layer outputs the evaluation value through the output layer. All the above layers are fully connected. The first compensation network and the second compensation network have the same structure, both of which include an input layer, three hidden layers and an output layer. The input layer receives the drone status data on the one hand, and receives the output of the first control network or the second control network on the other hand. The output then passes through its first hidden layer, second hidden layer, and third hidden layer in sequence, and then is output through the output layer. All of the above layers are fully connected.

6. The method for controlling the state compensation of the trajectory tracking of unmanned aerial vehicles according to claim 1 is characterized in that: The calculation formula of the timing difference error value Loss in step S5 is: Among them, Q Target is the target value, Q Target =R i+1 +γQ′(s i ,A′(s i |θ A′ ,ω C′ )), γ is the attenuation factor, Q′(s i ,A′(s i |θ A′ ,ω C′ )) is the value of the second evaluation value after sampling, Q(s i ,a i |θ Q ,ω C ) is the value of the first evaluation value after sampling, θ Q ,θ A and ω C They are respectively the value function Q(s) of the first evaluation network t ,a|θ Q ), the control quantity function A(s) of the first control network t |θ A ) and the function C(s) of the first compensation network t |ω C ) parameters, θ Q′ ,θ A′ and ω C′ are the value function Q′(s t ,a|θ Q′ ), the control quantity function A′(s t |θ A′ ) and the function C′(s t |ω C′ ) parameters.

7. The method for controlling the state compensation of the trajectory tracking of unmanned aerial vehicles according to claim 1 is characterized in that: The target gradient function in step S5 The formula is: in, is the symbol of the gradient function with a as the independent variable, is θ A is the symbol of the gradient function of the independent variable, So ω C is the symbol of the gradient function of the independent variable; The network parameters are updated as follows: Using the time difference error value Loss and the target gradient function The gradient ascent method is used to update the first evaluation network parameter θ using the Adam optimizer in the Python machine learning library Q and the first control network parameter θ A ; Use the first evaluation network parameter θ Q and the first control network parameter θ A The second evaluation network parameters and the second control network parameters are slightly updated, and the formula is: Among them, η∈[0,1] is the update parameter; The gradient descent method is used to update the compensation network parameters. The parameter update formula is: Among them, the parameter update amount is △ω, and ε∈[0,1] is the learning rate of the compensation network.

8. A system for the drone trajectory tracking state compensation deep deterministic policy gradient control method of claim 1, characterized in that: include: Tuple building module for building Markov decision process tuples for UAV dynamic tracking<S,A,P,R,γ> , S is a finite state set, A is a finite control quantity set, P is a state transition probability matrix based on the control quantity, R is a reward function based on the state and control quantity, and γ is a decay factor; The motion state equation building module is used to build the motion state equation for the UAV dynamic tracking, including the state of the next time step and the UAV trajectory tracking control strategy; A network construction module, used to construct a first control network, a first evaluation network, a first compensation network, and corresponding second control network, second evaluation network, and second compensation network having the same network structure; An execution control module is used to execute each network and obtain multiple state transfer processes; The network parameter update module is used to update each network parameter through the target gradient function until it meets the training objectives.

9. A device, characterized in that: comprising a memory and a processor, wherein: A memory for storing computer programs that can be run on the processor; A processor is used to execute the steps of a drone trajectory tracking state compensation deep deterministic policy gradient control method as described in any one of claims 1 to 7 when running the computer program.

10. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by at least one processor, implements the steps of a drone trajectory tracking state compensation deep deterministic policy gradient control method as described in any one of claims 1 to 7.

Images