A UAV route planning method based on improved q-learning algorithm

Abstract

The invention discloses an unmanned aerial vehicle route planning method based on an improved Q-learning algorithm. The method comprises the steps of: training an unmanned aerial vehicle in differentsimulation environments to obtain a priori knowledge list, and guiding the unmanned aerial vehicle to explore by utilizing the priori knowledge obtained by training in an unknown environment, so thatthe exploring step number of the unmanned aerial vehicle in the unknown environment is reduced; and introducing a single-position action value function convergence criterion, so that the traditional Q-learning based on a Markov process chain convergence principle is changed, and the convergence speed of an action value function is increased.

Description

technical field [0001] The invention relates to the field of UAV route planning, in particular to a UAV route planning method based on an improved Q-learning algorithm in an unknown environment. Background technique [0002] Unmanned aircraft, referred to as UAV, refers to an aircraft that is not controlled by a pilot, and can be navigated and controlled by its onboard equipment during flight, and can also be remotely operated by the ground. Since the UAV does not require a pilot to drive, it can avoid the physiological limitations of the pilot and ensure the safety of the staff. Compared with manned aircraft, drones are small in size, low in cost, high in safety, and good in concealment. Compared with traditional work carriers such as satellites, UAVs have low overall cost, high cost-effectiveness, and are flexible and maneuverable in use. Therefore, all countries are actively expanding the application range of drones. The technical and economic effects of using drones in...

AI Technical Summary

Problems solved by technology

However, in the final analysis, reinforcement learning is a data-driven optimization algorithm. Its unavoidable shortcomings are greater computing pressure and the need for more interactive data. The following three problems make it difficult for UAV route planning algorithms based on reinforcement learning to meet practical applications. Requirements:

[0006] 1) In a large-scale state space and action space, the algorithm performs infinite repeated training on each state and action, which will lead to the disaster of dimensionality

[0007] 2) After the UAV performs an action, the reward function value obtained is often not instant, which increases the time complexity

[0009] UAVs must face the above three problems for route planning in an unknown environment. In order to speed up the convergence speed of UAVs using reinforcement learning for route planning, some scholars have added the Dyna learning framework to the Q-learning algorithm. Establish an environment model with a little real data, update the value function by using virtual samples, and reduce the time complexity by increasing the computational complexity, but this violates the requirement of complete autonomy for UAV route planning

The layered reinforcement learning algorithm introduces layered thinking into the Q-learning algorithm, decomposes the original entire task into some independent sub-tasks for learning, and speeds up the learning rate. Some scholars have proposed a layered learning method based on selection. The first layer uses the Q-learning algorithm to train the basic behaviors of the movement separately, and coordinates these basic behaviors in the second layer to solve the planning task. The simulation results show that the algorithm can be well applied to path planning in unknown environments and improve the convergence speed. However, it should be pointed out that the algorithm still does not solve the problem that drones need a lot of exploration in unknown environments.

Images