A model-enhanced reinforcement learning optimization method for UAV flight trajectory
By combining EGO-Planner and SAC algorithms, the drone flight simulation model is established, the flight trajectory of the drone is optimized, the problems of local optimal solution and training instability in the existing technology are solved, and efficient and robust Drone flight obstacle avoidance capability.
Patent Information
- Application Number
- CN202210604906.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-30
- Publication Date
- 2024-12-17
- Estimated Expiration
- 2042-05-30
AI Technical Summary
Existing drone flight obstacle avoidance algorithms, such as EGO-Planner, are prone to falling into local optimal solutions, and the training process of reinforcement learning algorithms is unstable, poor robustness, and it is difficult to achieve efficient drone flight.
Combining the EGO-Planner algorithm and SAC algorithm, we establish a drone flight simulation model, optimize the flight trajectory through Markov decision-making process and neural network encoding, and use reward functions and state transition probability to perform action selection and training to achieve real-time obstacle avoidance and flight efficiency Improvement.
It realizes efficient obstacle avoidance flight in complex environments, reduces the demand for global Euclidean vector field construction, and improves the robustness and efficiency of flight decisions.
Smart Images
Figure CN114879738B_ABST
Abstract
Description
Technical Field
[0001] The invention provides a model-enhanced unmanned aerial vehicle flight trajectory reinforcement learning optimization method, which belongs to the technical field of machine learning. Background Art
[0002] Drones are widely used in agriculture, exploration, disaster relief, public safety, entertainment, future warfare and other fields. In many complex scenarios, drones are required to complete flight missions autonomously and efficiently. Therefore, finding a method that can ensure obstacle-free flight of drones while reducing the flight overhead of drones as much as possible can greatly improve flight efficiency. At present, traditional obstacle avoidance algorithms based on B-spline curves and gradients are very mature, but such obstacle avoidance algorithms need to construct a global Euclidean vector field before flight, which often takes up a lot of flight time. However, in actual drone flight missions, drones are often required to avoid obstacles in real time based on the local environment and achieve high-efficiency flight as much as possible.
[0003] EGO-Planner is a drone flight obstacle avoidance algorithm based on local Euclidean vector fields. It can perceive the flight environment in real time and make flight decisions quickly. Reinforcement learning algorithm is an algorithm that continuously interacts with the environment to learn strategies. In drone flight planning, the flight state and actions of the drone are often encoded into the neural network, and the calculated reward value is used to train and optimize the flight time.
[0004] The main problems of these algorithms are as follows: the EGO-Planner algorithm often falls into the local optimal solution due to too many control algorithm restrictions, and cannot guarantee the obstacle avoidance flight of the UAV. The single reinforcement learning algorithm needs to rely on the neural network for learning, and the training process is often unstable and has poor robustness. Summary of the invention
[0005] In order to overcome the deficiencies in the prior art, the present invention aims to solve the technical problem of providing an improved method for optimizing the flight trajectory of a UAV using a model-enhanced reinforcement learning method.
[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: a model-enhanced UAV flight trajectory reinforcement learning optimization method, comprising the following steps:
[0007] Step 1: construct a UAV flight simulation scene in a three-dimensional space vector field, including: flight obstacles; obstacle locations; the starting point and end point of the UAV;
[0008] Step 2: Establish an environmental model based on the Markov decision process, including the flight environment E, state space S, action space A, reward function R, and state transition probability P;
[0009] Step 3, based on the EGO-Planner algorithm, an EGO-Planner flight obstacle avoidance model is established, including the flight environment E, the obstacle avoidance flight trajectory Φ, the obstacle avoidance flight distance d, and the obstacle avoidance time T;
[0010] Step 4: Under the EGO-Planner flight obstacle avoidance model, the drone selects actions based on the state and strategy, interacts with the environment while flying, obtains the state and current reward value at the next moment, and stores them in the experience cache pool;
[0011] Step 5: Use a neural network to encode the state-action of the sample obtained from the environment, and then use the SAC algorithm to train and learn the sample to update the neural network parameters;
[0012] Step 6: Select the action with the highest reward value in the sample state as the optimal action, and then get the optimal strategy.
[0013] In step 2, an environment model is established based on the Markov decision process, including the flight environment E, the state space S, the action space A, the reward function R, and the state transition probability P, which is specifically:
[0014] The established flight environment E includes the three-dimensional coordinate set {M0, M1, ..., M n}, the three-dimensional coordinate set of obstacle points {O0, O1, ..., O k};
[0015] M0, M1 and M n are the three-dimensional coordinate vectors of the flight mission points 0, 1, and n, O0, O1, and O k are the three-dimensional coordinate vectors of obstacle points 0, 1, and k respectively;
[0016] The state space S includes the coordinate information l of the current position of the drone, the flight distance d from the current flight mission point to the next flight mission point when the drone uses the EGO-Planner flight obstacle avoidance model to avoid obstacles, and the flight speed v of the drone, recorded as S = (l, d, v);
[0017] Action space A includes the changes in flight trajectory and flight speed after the drone uses the EGO-Planner flight obstacle avoidance model to avoid obstacles;
[0018] The reward function R is:
[0019]
[0020]
[0021] Where, d i-1,iis the flight distance between two flight mission points, T i is the time taken from the last flight mission point to the i-th flight mission point, R is the reward value of the drone reaching the end point after traversing each mission point from the starting point, which is described by the negative logarithm of the total flight time of the drone, st T i It is the flight time constraint;
[0022] The state transition probability is P, which represents the probability that the drone will transition from state s to state s' after performing an action in state s.
[0023] In step 3, an EGO-Planner flight obstacle avoidance model is established based on the EGO-Planner algorithm, including the flight environment E, the obstacle avoidance flight trajectory Φ, the obstacle avoidance flight distance d, and the obstacle avoidance time T, specifically:
[0024] The established flight environment E includes the three-dimensional coordinate set {M0, M1, ..., M n}, the three-dimensional coordinate set of obstacle points {O0, O1, ..., O k};
[0025] The obstacle avoidance flight trajectory Φ includes a set of collision-free flight trajectories {Φ1, Φ2, L} from the current flight mission point to the next flight mission point generated by the EGO-Planner obstacle avoidance algorithm, where Φ1 is the first flight trajectory point and Φ2 is the second flight trajectory point;
[0026] The obstacle avoidance flight distance d represents the flight distance from the current flight mission point to the next flight mission point after the UAV uses the EGO-Planner flight obstacle avoidance model to avoid obstacles;
[0027] The obstacle avoidance time T represents the time taken from the last flight mission point to the i-th flight mission point.
[0028] In step 4, the drone transfers from the current state to the new state and calculates the reward value, and stores the experience quadruple (state, action, reward, next moment state) in the experience buffer pool of the SAC algorithm for algorithm sampling training. Further, the storage method is to use the reward value to calculate the sampling probability of each group of samples in the experience pool of the SAC algorithm, and store the samples and the sampling probabilities of the samples in the experience buffer pool.
[0029] The sampling process in step 4 is performed according to priority, and samples with high sampling probability in the experience buffer pool are preferentially used, and random sampling disturbance is applied at the same time. The samples obtained after sampling by the improved SAC algorithm are represented as X.
[0030] In step 5, the SAC algorithm preferentially samples from the cache pool according to probability, and uses two two-layer neural networks MLP to encode the actions and states of the samples to obtain the value network and the judgment network respectively. The algorithm uses the TD error of the two networks to perform gradient descent and then update the network parameters, specifically:
[0031] The MLP neural network is used to encode the state-action as follows:
[0032] e a ,e s =g(s,a,θ);
[0033] In the formula, s represents the state, a represents the action, g represents an MLP neural network, θ is the network parameter, and e a and e s Represents the encoded state and action vector;
[0034] When updating network parameters, the SAC algorithm uses a judgment network and a value network to calculate the Q value of the state action respectively, and uses the TD error calculated by the judgment network to update the network, which is expressed as:
[0035]
[0036]
[0037] Where, L Q (ψ) represents the TD error calculated by the two networks, y is the optimization target, r t is the reward value at time t, γ is the discount factor, is in state s t+1 The expected Q value of each action, s t is the state of the drone at time t, a t is the drone action at time t, s t+1 is the state of the drone at time t+1, a t+1 is the drone action at time t+1, Q ψ is the value network, Q ψ′ To judge the network, α is the hyperparameter after adding the maximum entropy to the reward value part, following the SAC algorithm model.
[0038] The beneficial effects of the present invention compared with the prior art are as follows: the present invention establishes a UAV flight simulation model based on the EGO-Planner and SAC algorithms. During the flight, the SAC algorithm is used to optimize the global flight time of the UAV, and the EGO-Planner algorithm is introduced to make obstacle avoidance strategies in real time. The fusion method does not require the UAV to fully understand the flight environment, and there is no need to establish the Euclidean vector field of the entire flight environment. At the same time, the UAV flight strategy is continuously trained and learned through the reinforcement learning algorithm, which can achieve more efficient flight. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] The present invention will be further described below in conjunction with the accompanying drawings:
[0040] Figure 1 A simulated environment diagram of the UAV of the present invention;
[0041] Figure 2 The figure is a flow chart of the algorithm of the present invention. DETAILED DESCRIPTION
[0042] like Figure 1-2 As shown, the present invention is a method for optimizing the flight trajectory of a UAV that integrates the EGO-Planner obstacle avoidance algorithm and deep reinforcement learning. The simulation environment is as follows: Figure 1 As shown in Figure 1, the drone flies in an environment with unknown obstacles, taking off from the starting point and ending at the end point. The flow chart of the flight trajectory optimization method is as follows: Figure 2 As shown in the figure: The UAV flight action and strategy will first be generated by the EGO-Planner algorithm, and then interact with the environment to obtain the state and current reward value at the next moment, and store them in the experience buffer pool. The SAC algorithm preferentially samples from the buffer pool according to probability, and uses two two-layer neural networks MLP to encode the action and state of the samples to obtain the value network and the judgment network respectively. The algorithm uses the TD error of the two networks to perform gradient descent, and then updates the network parameters, and finally takes the action that maximizes the total reward value as the optimal flight strategy.
[0043] A method for optimizing the flight trajectory of a UAV by integrating the EGO-Planner obstacle avoidance algorithm and deep reinforcement learning of the present invention comprises the following steps:
[0044] Step 1: Construct a UAV flight simulation in a three-dimensional space vector field: (1) flight obstacles; (2) obstacle locations; (3) the starting and ending points of the UAV. Figure 1 As shown;
[0045] Step 2: Establish an environmental model based on the Markov decision process, including the flight environment E, state space S, action space A, reward function R, and state transition probability P;
[0046] Taking the obstacle avoidance flight optimization process of a certain UAV as an example, the established flight environment E includes the three-dimensional coordinate set {M0, M1, ..., M n}, the three-dimensional coordinate set of obstacle points {O0, O1, ..., O k};
[0047] M0, M1 and M n are the three-dimensional coordinate vectors of the flight mission points 0, 1, and n, O0, O1, and O k are the three-dimensional coordinate vectors of obstacle points 0, 1, and k respectively
[0048] The state space S includes the coordinate information l of the current position of the drone, the flight distance d from the current flight mission point to the next flight mission point when the drone uses the EGO-Planner flight obstacle avoidance model to avoid obstacles, and the flight speed v of the drone, recorded as S = (l, d, v);
[0049] Action space A includes the changes in flight trajectory and flight speed after the drone uses the EGO-Planner flight obstacle avoidance model to avoid obstacles;
[0050] The reward function R is designed as:
[0051]
[0052]
[0053] Where, d i-1,i is the flight distance between two flight mission points, T i is the time taken from the last flight mission point to the i-th flight mission point, R is the reward value of the drone reaching the end point after traversing each mission point from the starting point, which is described by the negative logarithm of the total flight time of the drone, st T i It is the flight time limitation.
[0054] The state transition probability is P, which represents the probability that the drone will transition from state s to state s' after performing an action in state s.
[0055] Step 3, based on the EGO-Planner algorithm, an EGO-Planner flight obstacle avoidance model is established, including the flight environment E, the obstacle avoidance flight trajectory Φ, the obstacle avoidance flight distance d, and the obstacle avoidance time T;
[0056] The established flight environment E includes the three-dimensional coordinate set {M0, M1, ..., M n}, the three-dimensional coordinate set of obstacle points {O0, O1, ..., O k};
[0057] The obstacle avoidance flight trajectory Φ includes the collision-free flight trajectory set {Φ1, Φ2, L} from the current flight mission point to the next flight mission point generated by the EGO-Planner obstacle avoidance algorithm, where Φ1 is the first flight trajectory point and Φ2 is the second flight trajectory point
[0058] The obstacle avoidance flight distance d represents the flight distance from the current flight mission point to the next flight mission point after the UAV uses the EGO-Planner flight obstacle avoidance model to avoid obstacles;
[0059] Step 4: The drone selects actions based on the state and strategy under the EGO-Planner flight obstacle avoidance model, and interacts with the environment while flying. The drone transfers from the current state to the new state and calculates the reward value, and stores the experience quadruple <state, action, reward, next moment state> in the experience cache pool of the SAC algorithm for algorithm sampling training;
[0060] Its storage method is to use the reward value to calculate the sampling probability of each group of samples in the experience pool of the SAC algorithm, and store the samples and their sampling probabilities in the experience buffer pool. In addition, the sampling process is carried out according to priority, and samples with high sampling probabilities in the experience buffer pool are given priority. At the same time, random sampling disturbances are applied. The samples obtained after sampling by the improved SAC algorithm are represented as X.
[0061] Step 5: Use a neural network to encode the state-action of the sample obtained from the environment, and then use the SAC algorithm to train and learn the sample to update the neural network parameters;
[0062] The MLP neural network is used to encode the state-action as follows:
[0063] e a ,e s =g(s,a,θ);
[0064] In the formula, s represents the state, a represents the action, g represents an MLP neural network, θ is the network parameter, and e a and e s Represents the encoded state and action vectors.
[0065] When updating network parameters, the SAC algorithm uses a judgment network and a value network to calculate the Q value of the state action respectively, and uses the TD error calculated by the judgment network to update the network, which is expressed as:
[0066]
[0067]
[0068] Where, LQ (ψ) represents the TD error calculated by the two networks, y is the optimization target, r t is the reward value at time t, γ is the discount factor, is in state s t+1 The expected Q value of each action, s t is the state of the drone at time t, a t is the drone action at time t, s t+1 is the state of the drone at time t+1, a t+1 is the drone action at time t+1, Q ψ is the value network, Q ψ′ To judge the network, α is the hyperparameter after adding the maximum entropy to the reward value part, following the SAC algorithm model.
[0069] Step 6: Select the action with the highest reward value in the sample state as the optimal action, and then get the optimal strategy.
[0070] With the continuous simulation, iteration and optimization of the flight process, the drone will adopt a flight strategy that can maximize the global benefits when executing flight decisions, rather than just seeing the flight environment in front of it. Ultimately, while achieving real-time obstacle avoidance flight, it will obtain the optimal flight strategy in a global sense.
[0071] The main innovation and key points of the present invention are: embedding the control algorithm in the deep reinforcement learning algorithm, realizing the fusion of cross-domain models, and combining the control obstacle avoidance algorithm EGO-Planner with the deep reinforcement learning SAC. EGO-Planner is used to ensure that the aircraft can perform reliable, stable and obstacle-free flight, while the SAC reinforcement learning algorithm uses flight time as a reward value to provide a more efficient flight strategy. The fusion of the control algorithm and the deep reinforcement learning algorithm improves the robustness of flight decisions and can improve flight efficiency.
[0072] What needs to be explained about the specific structure of the present invention is that the connection relationship between the various component modules adopted in the present invention is definite and feasible. Except for the special instructions in the embodiments, the specific connection relationship can bring about the corresponding technical effects, and solve the technical problems raised by the present invention without relying on the execution of the corresponding software program. Except for the specific instructions, the models and connection methods of the components, modules and specific components appearing in the present invention belong to the existing technologies such as published patents, published journal articles, or common knowledge that can be obtained by technical personnel in this field before the application date, and there is no need to elaborate, so that the technical solution provided in this case is clear, complete and feasible, and the corresponding physical products can be reproduced or obtained according to the technical means.
[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A model-enhanced UAV flight trajectory reinforcement learning optimization method, characterized by: The steps include: Step 1: construct a UAV flight simulation scene in a three-dimensional space vector field, including: flight obstacles; obstacle locations; the starting point and the end point of the UAV; Step 2: Establish an environmental model based on the Markov decision process, including the flight environment E, state space S, action space A, reward function R, and state transition probability P; Step 3, based on the EGO-Planner algorithm, an EGO-Planner flight obstacle avoidance model is established, including the flight environment E, the obstacle avoidance flight trajectory Φ, the obstacle avoidance flight distance d, and the obstacle avoidance time T; Step 4: Under the EGO-Planner flight obstacle avoidance model, the drone selects actions based on the state and strategy, interacts with the environment while flying, obtains the state and current reward value at the next moment, and stores them in the experience cache pool; Step 5: Use a neural network to encode the state-action of the sample obtained from the environment, and then use the SAC algorithm to train and learn the sample to update the neural network parameters; The sampling process in step 4 is performed according to priority, and samples with high sampling probability in the experience buffer pool are preferentially used, and random sampling disturbance is applied at the same time. The samples obtained after sampling by the improved SAC algorithm are represented as X; Step 6: Select the action with the highest reward value in the state in the sample as the optimal action, and then get the optimal strategy; In step 5, the SAC algorithm preferentially samples from the cache pool according to probability, and uses two two-layer neural networks MLP to encode the actions and states of the samples to obtain the value network and the judgment network respectively. The algorithm uses the TD error of the two networks to perform gradient descent and then update the network parameters, specifically: The MLP neural network is used to encode the state-action as follows: e a ,e s =g(s,a,θ); In the formula, s represents the state, a represents the action, g represents an MLP neural network, θ is the network parameter, and e a and e s Represents the encoded state and action vector; When updating network parameters, the SAC algorithm uses a judgment network and a value network to calculate the Q value of the state action respectively, and uses the TD error calculated by the judgment network to update the network, which is expressed as: Where, L Q (ψ) represents the TD error calculated by the two networks, y is the optimization target, r t is the reward value at time t, γ is the discount factor, is in state s t+1 The expected Q value of each action, s t is the state of the drone at time t, a t is the drone action at time t, s t+1 is the state of the drone at time t+1, a t+1 is the drone action at time t+1, Q ψ is the value network, Q ψ′ To judge the network, α is the hyperparameter after adding the maximum entropy to the reward value part, following the SAC algorithm model.
2. The model-enhanced UAV flight trajectory reinforcement learning optimization method according to claim 1 is characterized in that: In step 2, an environment model is established based on the Markov decision process, including the flight environment E, the state space S, the action space A, the reward function R, and the state transition probability P, which is specifically: The established flight environment E includes the three-dimensional coordinate set {M0, M1, ..., M n }, the three-dimensional coordinate set of obstacle points {O0, O1, ..., O k }; M0, M1 and M n are the three-dimensional coordinate vectors of the flight mission points 0, 1, and n, O0, O1, and O k are the three-dimensional coordinate vectors of obstacle points 0, 1, and k respectively; The state space S includes the coordinate information l of the current position of the drone, the flight distance d from the current flight mission point to the next flight mission point when the drone uses the EGO-Planner flight obstacle avoidance model to avoid obstacles, and the flight speed v of the drone, recorded as S = (l, d, v); Action space A includes the changes in flight trajectory and flight speed after the drone uses the EGO-Planner flight obstacle avoidance model to avoid obstacles; The reward function R is: Where, d i-1,i is the flight distance between two flight mission points, T i is the time taken from the last flight mission point to the i-th flight mission point, R is the reward value of the drone reaching the end point after traversing each mission point from the starting point, which is described by the negative logarithm of the total flight time of the drone, s.tT i It is the flight time constraint; The state transition probability is P, which represents the probability that the drone will transition from state s to state s' after performing an action in state s.
3. The model-enhanced UAV flight trajectory reinforcement learning optimization method according to claim 1, characterized in that: In step 3, an EGO-Planner flight obstacle avoidance model is established based on the EGO-Planner algorithm, including the flight environment E, the obstacle avoidance flight trajectory Φ, the obstacle avoidance flight distance d, and the obstacle avoidance time T, specifically: The established flight environment E includes the three-dimensional coordinate set {M0, M1, ..., M n }, the three-dimensional coordinate set of obstacle points {O0, O1, ..., O k }; The obstacle avoidance flight trajectory Φ includes a set of collision-free flight trajectories {Φ1, Φ2,} from the current flight mission point to the next flight mission point generated by the EGO-Planner obstacle avoidance algorithm, where Φ1 is the first flight trajectory point and Φ2 is the second flight trajectory point; The obstacle avoidance flight distance d represents the flight distance from the current flight mission point to the next flight mission point after the UAV uses the EGO-Planner flight obstacle avoidance model to avoid obstacles; The obstacle avoidance time T represents the time taken from the last flight mission point to the i-th flight mission point.
4. The model-enhanced UAV flight trajectory reinforcement learning optimization method according to claim 1, characterized in that: In step 4, the drone transfers from the current state to the new state and calculates the reward value, and stores the experience quadruple (state, action, reward, next moment state) in the experience buffer pool of the SAC algorithm for algorithm sampling training. The storage method is to use the reward value to calculate the sampling probability of each group of samples in the experience pool of the SAC algorithm, and store the samples and the sampling probabilities of the samples in the experience buffer pool.