A method for unmanned aerial vehicle trajectory tracking control based on adaptive exploration deep reinforcement learning

Abstract

The application discloses a kind of unmanned vehicle trajectory tracking control methods based on adaptive exploration depth reinforcement learning.In training phase,randomly initialize network parameters and reference trajectory, action is obtained by using action network through current state and noise is added in initial interval, the next state and corresponding reward are obtained by using action, state-action-reward-next state is stored as a tuple, and the parameters of action network, penalty network and value network are updated using the stored tuple.The application uses time decay and periodic reward to adaptively explore, trains the depth reinforcement learner through environment interaction, enables the depth reinforcement learner to master the learning ability in task faster, greatly improves the learning efficiency of depth reinforcement learning and the accuracy of trajectory tracking, and has very important significance for improving the autonomy and intelligence of unmanned system and the model-free intelligent control performance of complex system.

Description

Technical Field

[0001] This invention belongs to the field of intelligent control and unmanned systems, and specifically addresses the poor trajectory tracking performance of unmanned aerial vehicles (UAVs) with inaccurate motion models. It relates to a UAV trajectory tracking control method based on adaptive exploratory deep reinforcement learning. Background Technology

[0002] Due to their flexibility and operability, unmanned aerial vehicles (UAVs) have shown great potential in a wide range of applications. In trajectory tracking tasks, control laws need to be designed for the UAV to track a reference trajectory assigned by the navigation system. Existing control methods include sliding mode control, backstepping control, model predictive control, distributed and linear time-varying control, and neural networked control. Among traditional control algorithms, model-based methods are widely used due to their good performance. However, these methods heavily rely on accurate dynamic modeling of the UAV; most work only considers the UAV's dynamic model, ignoring the complex dynamic environment in reality. Furthermore, due to the high-speed movement of the UAV, any slight disturbance, model mismatch, and long control intervals can lead to catastrophic collisions. Therefore, trajectory tracking control of UAVs in complex environments is extremely challenging. Moreover, the control law needs to be robust to disturbances and adapt to the time-varying reference trajectory in real time.

[0003] To address modeling inaccuracies, researchers have proposed various control methods. For example, Manuel J. Reinoso et al., in their paper (Reinoso, Manuel J., et al. "Trajectory tracking of a quadrotor using sliding mode control." IEEE Latin America Transactions 14.5 (2016): 2157-2166.), simplified the quadrotor dynamics model through small-angle approximation, thus reducing the complexity of the SMC design equations. However, chattering occurs at the trajectory reference point. In practical applications, so-called equilibrium point chattering always exists, which is fatal for UAVs. To solve this problem, Michael Neuert et al., in their paper (Neunert, Michael, et al. "Fast nonlinear model predictive control for unified trajectory optimization and tracking." 2016 IEEE international conference on robotics and automation (ICRA). IEEE, 2016.), designed a millisecond-level nonlinear MPC framework. This method still relies on the model and cannot completely solve the problem.

[0004] Furthermore, model-free control methods, such as deep reinforcement learning, can address the problem of over-reliance on models. Deep reinforcement learning provides a learning framework that optimizes control policies through interaction with the environment. Therefore, it can handle complex dynamics without requiring modeling, as environmental noise is always present during tracking. To overcome this issue, Jiying Wu et al. proposed a compensation network for deep reinforcement learning action networks in the paper (Wu, Jiying, et al. "A State-Compensated Deep Deterministic Policy Gradient Algorithm for UAV Trajectory Tracking." Machines 10.7(2022):496.), which can better handle noise problems. However, the compensation method needs further improvement to cope with complex environmental changes. Typically, model-free deep reinforcement learning algorithms encourage exploration by injecting action noise during training, optionally following a linear annealing scheme. Different exploration noises will lead to different results, even local optima.

[0005] Therefore, to solve the above problems, it is necessary to design a learning-based, accurate, and efficient UAV trajectory tracking method that can avoid local optima during training and enable the algorithm to converge quickly, thereby achieving accurate trajectory tracking. To date, no method can guarantee high-precision tracking capability within an extremely short control time. Summary of the Invention

[0006] To address the aforementioned issues, this invention provides a UAV trajectory tracking control method based on adaptive exploratory deep reinforcement learning. This method introduces a time variable into existing deep reinforcement learning algorithms, using exponential decay over time to adaptively adjust the exploration amplitude, thereby accelerating training, shortening the algorithm's convergence time, and considering the reward within a cycle as an evaluation to avoid getting trapped in local optima. Furthermore, in the early stages of training, multiple paths are randomly initialized to enhance the controller's trajectory generalization ability in tracking tasks.

[0007] The technical solution of this invention is:

[0008] A UAV trajectory tracking control method based on adaptive exploratory deep reinforcement learning includes the following steps:

[0009] S1, Construct the network model required for deep reinforcement learning. The constructed network model includes an action network and a value network.

[0010] S2, based on the current state vector s, use the action network from step S1 to interact with the environment to obtain the next state vector s′. Store all state vectors and action vectors output by the action network as tuples. The batch size of the tuples needs to be greater than the set threshold.

[0011] The tuple is (s,a,r,s′,a′), where r is the reward obtained by interacting with the environment using action vector a under the current state vector s, the initial state vector s0 is the initial trajectory data of the given drone, the action vector output by the action network corresponding to the current state vector s is a, the action vector output by the action network corresponding to the next state vector s′ is a′, and the action vector output by the action network corresponding to the initial state vector s0 is a0.

[0012] S3, randomly sample from the data stored in step S2, according to the formula The Q-value of the state vector s, i.e., the value of the state vector s, is obtained according to the formula. Update the parameters of the value network;

[0013] S4, according to formula To obtain a reward within a cycle, based on δ and the formula σ←βinput max e -λt +(1-β)input max e-δ Obtain the variance of the Gaussian noise, and use the obtained variance of the Gaussian noise to update the Gaussian noise;

[0014] S5, repeat steps S2-S4, and calculate the gradient when the number of iterations meets the set threshold. Based on the obtained gradient The action network and value network are updated using gradient descent to obtain new action network parameters, which are then determined based on θ′. k ←τθ k +(1-τ)θ′ k φ′←τφ+(1-τ)φ′ updates the value network parameters and action network parameters;

[0015] S6 repeats steps S2-S5 until the number of iterations reaches the set maximum training period n, at which point the loop ends, resulting in a drone trajectory tracking controller based on deep reinforcement learning. The obtained drone trajectory tracking controller is then used to perform trajectory tracking control on the drone.

[0016] In step S3, the formula In this context, i represents one of the two value networks, Q represents the value network, and θ is the value network parameter.

[0017] In step S4, the formula In this context, `batch` represents a randomly sampled data set, `j` represents a single data point within that set, and `r` represents a random sample of data. min r max These represent the minimum and maximum rewards in the historical reward history, respectively. The formula is σ←βinput. max e -λt +(1-β)input max e -δ In this context, t represents the number of iterations from S2 to S5.

[0018] In step S5, the gradient In this equation, φ represents the action network parameters, and the formula θ′ k ←τθ k +(1-τ)θ′ k ,φ′←τφ+(1-τ)φ′ where τ is a constant, θ k φ is the parameter of the k-th value network, and φ is the parameter of the action network.

[0019] Beneficial effects

[0020] (1) The UAV trajectory tracking control method designed in this invention based on adaptive exploratory deep reinforcement learning uses deep reinforcement learning to solve high-dimensional continuous control problems and generates a controller with strong interference suppression, high agility and strong robustness through random trajectory pre-training.

[0021] (2) The neural network controller trained by the method of the present invention can adapt online and achieve better tracking performance for newly given trajectories. This method eliminates the control accuracy problem caused by model inaccuracy and greatly reduces the operation time by 10 times compared with the solution optimization method.

[0022] (3) This invention discloses a UAV trajectory tracking control method based on adaptive exploratory deep reinforcement learning. During the training phase, network parameters and a reference trajectory are first randomly initialized. Actions are obtained from the current state using the action network, and noise is added to the initial interval. The next state and corresponding reward are then obtained using the action. The state-action-reward-next state sequence is stored as a tuple. The parameters of the action network, penalty network, and value network are updated using the stored tuple. The noise interval is updated based on time and the reward within a certain period. This process is repeated until training is complete, resulting in a high-performance deep reinforcement controller. This invention utilizes time decay and periodic rewards for adaptive exploration, training the deep reinforcement learner through environmental interaction. This enables the deep reinforcement learner to master learning capabilities more quickly in tasks, significantly improving the learning efficiency and trajectory tracking accuracy of deep reinforcement learning. This is of great significance for enhancing the autonomy and intelligence of unmanned systems and the model-free intelligent control performance of complex systems. Attached Figure Description

[0023] Figure 1 This is a comparison of the performance of the UAV trajectory tracking control method based on adaptive exploratory deep reinforcement learning provided by this invention with other algorithms on a deep reinforcement learning algorithm verification platform.

[0024] Figure 2 This is a diagram illustrating the operational effect of an embodiment of an UAV trajectory tracking control method based on adaptive exploratory deep reinforcement learning provided by the present invention. Figure 2 a is the LOSRT trajectory. Figure 2 b represents the COS trajectory. Detailed Implementation

[0025] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. The following embodiments and drawings are for illustrative purposes only and are not intended to limit the scope of the invention.

[0026] This invention provides a UAV trajectory tracking control method based on adaptive exploratory deep reinforcement learning. Through continuous interaction with the environment, a deep reinforcement learning-based UAV trajectory tracking controller with small tracking errors and short control cycles is ultimately obtained. The method includes the following steps:

[0027] S1. Prepare the network model and initial trajectory data required for deep reinforcement learning. Set the given trajectory length T = 3000 for both training and testing tasks, with a time interval of 0.001s between each trajectory point, resulting in a total time length of 3s. In the simulation, set the UAV's mass to 1.5kg, the maximum propeller lift to 20N / kg, the minimum lift to 0N / kg, the maximum angular velocity to 6rad / s, and the minimum angular velocity to -6rad / s. Set two test trajectories. Randomly initialize the network parameters θ1, θ2, and φ, and set the algorithm hyperparameters, including a 13-dimensional state space (UAV's 3D position, 3D velocity, 4D quaternion attitude, and 3D vector pointing from the next reference point to the current position), a 4D motion space lift, and a 3D angular velocity u = (f, w) along the three axes. x w y w z Learning rate l r =0.99, batch size = 255, maximum storage B, reward discount γ = 0.99, exponential decay factor β = 0.08, reward influence factor λ = 0.01, maximum training period of outer loop n = 100000, and fast parameter update frequency of inner loop d = 2.

[0028] S2 obtains batch data through interaction with the environment. At the beginning of each training round, based on the current state vector s, the corresponding action vector a1 is obtained. Gaussian noise is added to the action vector, where the mean of the Gaussian noise is 0 and the initial variance is 1. The noisy action vector a is then used to interact with the environment for the next step, and the next state vector s′ is obtained based on the given action vector a1. At the same time, the environment provides the action score reward r. Finally, (s,a,r,s′) is stored as a batch data tuple. If the batch data volume exceeds the set maximum storage B, the first stored data is deleted and the new data is stored.

[0029] S3: Before the storage volume reaches the batch processing size, repeat step S2 until the minimum batch processing size is reached. If the batch processing condition is met, randomly sample the stored data, use the network model to obtain the action vector a′ corresponding to the state vector s′, and obtain the tuple (s,a,r,s′,a′). Select the smaller Q value in the double Q network, according to the formula... Obtain the discounted Q-value of the current state vector s, and use the Q-value according to the formula. Update the value network once.

[0030] S4, according to formula The reward within a given period is obtained, and its magnitude is constrained by δ to decay exponentially over time. If the reward value within a period is considered too small throughout the training process, the value of δ will decrease accordingly, and the periodic reward influence term in the variance will increase; conversely, it will decrease. According to the formula σ←βinput... maxe -λt +(1-β)input max e -δ The variance of the Gaussian noise is used to update the exploration amplitude, so that σ can adaptively decay with each training cycle iteration.

[0031] S5, when the inner loop update count is satisfied, it is necessary to update the Q-network model parameters θ1, θ2 and the action network model parameters φ again, according to the formula. Calculate the gradient New network parameters are obtained using gradient descent. To prevent overfitting, θ′ is used. k ←τθ k +(1-τ)θ′ k φ′←τφ+(-τ)φ′ is used to update the value network and action parameters.

[0032] S6, repeat S2-S5 until the number of iterations reaches the maximum training period n of the outer loop deep reinforcement learner, and finally obtain the deep reinforcement learning UAV trajectory tracking controller based on adaptive exploration.

[0033] S7 uses an adaptive exploration-based deep reinforcement learning UAV trajectory tracking controller for tracking control in a test task. The test task uses LOSRT trajectory and COS trajectory for tracking, and finally compares it with other algorithms in terms of tracking accuracy, input loss and total control time.

[0034] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0035] Example

[0036] This invention provides a UAV trajectory tracking control method based on adaptive exploratory deep reinforcement learning, which integrates the UAV's current position, velocity, attitude, next reference trajectory point, and the distance from the next reference position to the current position.

[0037] The direction vector serves as the state vector during interaction with the environment, and the output is the UAV's four-dimensional control vector, enabling end-to-end UAV trajectory tracking control. During the training phase, network parameters and algorithm hyperparameters are randomly initialized using random reference trajectories. Collected batch data is used to update network parameters by adding Gaussian noise, making the controller more robust. Simultaneously, attention is paid to the time variable, employing exponential time decay and periodic rewards to update the noise amplitude, accelerating training and significantly avoiding local optima.

[0038] This invention provides a UAV trajectory tracking control method based on adaptive exploratory deep reinforcement learning. Through continuous interaction with the environment, a deep reinforcement learning-based UAV trajectory tracking controller with small tracking errors and short control cycles is ultimately obtained. The method includes the following steps:

[0039] S1. Prepare the network model and initial trajectory data required for deep reinforcement learning. The entire task is divided into two categories: training and testing tasks, used for training and testing the learning effect, respectively. The given trajectory length for both training and testing tasks is set to T = 3000, and the time interval between each trajectory point is 0.001s, i.e., the total time length is 3s. In the simulation, the drone's mass is set to 1.5kg, the maximum propeller lift is 20N / kg, the minimum lift is 0N / kg, the maximum angular velocity is 6rad / s, and the minimum angular velocity is -6rad / s. Two test trajectories are set. Randomly initialize the network parameters θ1, θ2, φ, and set the algorithm hyperparameters, including a 13-dimensional state space (3D position, 3D velocity, 4D quaternion attitude, and 3D vector from the next reference point to the current position), a 4D motion space lift, and a 3D angular velocity u = (f, w)... x w y w z Learning rate l r =0.99, batch size = 255, maximum storage B, reward discount γ = 0.99, exponential decay factor β = 0.08, reward influence factor λ = 0.01, maximum training period of outer loop n = 100000, and fast parameter update frequency of inner loop d = 2.

[0040] S2 obtains batch data through interaction with the environment. At the beginning of each training round, based on the current state vector s, the corresponding action vector a1 is obtained. Gaussian noise is added to the action vector, with a mean of 0 and an initial variance of 1. The noisy action vector a is then used to interact with the environment for the next step, and the next state vector s′ is obtained based on the given action vector a1. At the same time, the environment provides an action score reward r. Finally, (s,a,r,s′) is stored as a batch data tuple. If the batch data volume exceeds the set maximum storage B, the earliest stored data is deleted and new data is stored.

[0041] S3: Before the storage volume reaches the batch processing size, repeat step S2 until the minimum batch processing size is reached. If the batch processing condition is met, randomly sample the stored data, use the network model to obtain the action vector a′ corresponding to the state vector s′, and obtain the tuple (s,a,r,s′,a′). Select the smaller Q value in the double Q network, according to the formula... Obtain the discounted Q-value of the current state vector s, and use the Q-value according to the formula. Update the value network once.

[0042] S4, according to formula The reward within a given period is obtained, and its magnitude is constrained by δ to decay exponentially over time. If the reward value within a period is considered too small throughout the training process, the value of δ will decrease accordingly, and the periodic reward influence term in the variance will increase; conversely, it will decrease. According to the formula σ←βinput... max e -λt +(1-β)input max e -δ The variance of the Gaussian noise is used to update the exploration amplitude, so that σ can adaptively decay with each training cycle iteration.

[0043] S5, when the inner loop update count is satisfied, it is necessary to update the Q network (i.e., the value network model parameters θ1, θ2) and the action network model parameters φ again, according to the formula. Calculate the gradient New network parameters are obtained using gradient descent. To prevent overfitting, θ′ is used. k ←τθ k +(1-τ)θ′ k φ′←τφ+(-τ)φ′ is used to update network parameters.

[0044] S6, repeat S2-S5 until the number of iterations reaches the maximum training period n of the outer loop deep reinforcement learner, and finally obtain the deep reinforcement learning UAV trajectory tracking controller based on adaptive exploration.

[0045] S7 uses an adaptive exploration-based deep reinforcement learning UAV trajectory tracking controller for tracking control in a test task. The test task uses LOSRT trajectory and COS trajectory for tracking, and finally compares it with other algorithms in terms of tracking accuracy, input loss and total control time.

[0046] like Figure 2 a and Figure 2 As shown in Figure b, this invention provides an example of the tracking performance of a UAV trajectory tracking control method based on adaptive exploratory deep reinforcement learning on an untrained trajectory. In the figure, Reference represents a given trajectory. Figure 2 a is the LOSRT trajectory. Figure 2b represents the COS trajectory. The trajectory tracking error is 7467.54 for the traditional method and 3283.77 for the present invention on the LOSRT trajectory. On the COS trajectory, the error is 2762.37 for the traditional method and 2356.75 for the present invention. Regarding control time, we averaged the computation time of the traditional method and our method three times respectively. The average running time of the present invention on the LOSRT trajectory is 4.754059 seconds, while the traditional method is 16.305317 seconds. The average running time of the present invention on the COS trajectory is 4.879404 seconds, while the traditional method is 19.360494 seconds. This demonstrates the effectiveness of the present invention in UAV tracking and control.

[0047] like Figure 1 As shown, Figure 1 To compare the UAV trajectory tracking control method based on adaptive exploratory deep reinforcement learning provided by this invention with the basic deep reinforcement learning algorithm in other environments, three random seed experiments were conducted on three environments running on the MUJOCO general algorithm verification platform for deep reinforcement learning. To avoid randomness, the hyperparameters of other algorithms were kept consistent with this algorithm. By comparing the convergence time of the algorithms, where the light-colored area represents the reward obtained by this invention with the training time, it was verified that this invention improves the training speed compared with other algorithms.

[0048] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for trajectory tracking control of a UAV based on adaptive exploration depth reinforcement learning, characterized in that Comprising the following steps: S1, constructing a network model required by deep reinforcement learning, the constructed network model including an action network and a value network; S2, interacting with the environment using the action network in step S1 according to the current state vector, obtaining a next state vector, and storing all state vectors and action vectors output by the action network as tuples; S3, randomly sampling in the data stored in step S2, and obtaining the value of the state vector, i.e. the value of the state vector, and updating the value network parameters; S3, randomly sampling in the data stored in step S2, and obtaining the value of the state vector, i.e. the value of the state vector, and updating the value network parameters; S4, obtaining a reward in a cycle, calculating the variance of the Gaussian noise according to the obtained reward and the number of cycles, and updating the Gaussian noise using the obtained variance of the Gaussian noise; S5, repeating steps S2-S4, calculating the gradient, updating the action network and the value network according to the obtained gradient, and obtaining new action network and value network parameters; S6, repeating steps S2-S5 until the number of cycles reaches the set maximum training period, obtaining a drone trajectory tracking controller based on deep reinforcement learning, and using the obtained drone trajectory tracking controller to control the trajectory tracking of the drone. The tuple in the step S2 is , The current state vector is The action vector used below is The reward obtained by interacting with the environment is The initial trajectory data of a given unmanned aerial vehicle is The action vector output by the action network corresponding to the current state vector is The next state vector is The action vector output by the action network corresponding to the next state vector is The initial state vector is The action vector output by the action network corresponding to the initial state vector is ; In the step S3, the value of the state vector is obtained, i.e. the value of the state vector is obtained. is obtained.​ In the step S3, the formula updating the value function network parameters.

2. The unmanned aerial vehicle trajectory tracking control method based on adaptive exploration deep reinforcement learning according to claim 1, characterized in that: In step S2, the tuple batch size is greater than a set threshold.

3. The unmanned aerial vehicle trajectory tracking control method based on adaptive exploration deep reinforcement learning according to claim 2, characterized in that: In step S2, the set threshold of the tuple batch size is 255.

4. The unmanned aerial vehicle trajectory tracking control method based on adaptive exploration deep reinforcement learning according to claim 1, characterized in that: In the step S4, the formula A reward is obtained for a period.

5. The unmanned aerial vehicle trajectory tracking control method based on adaptive exploration deep reinforcement learning according to claim 4, characterized in that: In the step S4, the variance of the Gaussian noise is obtained according to and the formula The variance of the Gaussian noise is obtained.

6. The unmanned aerial vehicle trajectory tracking control method based on adaptive exploration deep reinforcement learning according to claim 1, characterized in that: In the step S5, the number of times of the steps S2-S4 needs to meet a set threshold value, and the gradient obtained is The action network and the value network are updated using the gradient descent method to obtain new action network and value network parameters.

7. The unmanned aerial vehicle trajectory tracking control method based on adaptive exploration deep reinforcement learning according to claim 1, characterized in that: In the step S5, the value network and the action network parameters are updated according to , updating the value network and the action network parameters.

Images