A UAV intelligent navigation method based on deep reinforcement learning

Abstract

A method for intelligent navigation of unmanned aerial vehicles based on deep reinforcement learning, the scheme of which is as follows: (1) establishing a quadcopter unmanned aerial vehicle model in a three-dimensional simulation environment and generating an environment of obstacles and target points; (2) designing a state space and a discrete action space for multi-dimensional feature fusion; (3) designing a reward function based on Euclidean distance; (4) designing a strategy network and a state value network; (5) designing a temporary experience pool to store information feedback when the unmanned aerial vehicle interacts with the environment; (6) training a reinforcement learning network based on a discrete PPO algorithm; (7) realizing intelligent navigation of the unmanned aerial vehicle. The present invention can effectively accelerate the learning and training efficiency of unmanned aerial vehicles and improve the quality of decision-making of unmanned aerial vehicles in different environments.

Description

Technical Field

[0001] The present invention belongs to the field of navigation and unmanned driving technology, and in particular relates to an unmanned aerial vehicle intelligent navigation method based on deep reinforcement learning. Background Art

[0002] In most current scenarios, drones are still controlled manually. This method can easily cause the drone to fail to avoid obstacles and crash due to improper operation during flight. Traditional drone obstacle avoidance methods such as visualization graphs and particle swarm optimization algorithms are complex modeling processes for different tasks and require large amounts of computation in three-dimensional space, which is very time-consuming. In actual flight, it is often necessary to avoid obstacles in real time in unknown environments.

[0003] Reinforcement learning is a method that allows robots to continuously interact with the environment and improve the actions of the robot in a specific state based on the rewards obtained. After continuous exploration of the environment, the optimal action in each state can be obtained to achieve a specific task. Reinforcement learning has been studied in the field of drone obstacle avoidance, but the current research on drone simulation environments is simple and can hardly be applied to complex real-world scenarios. In addition, the observation information obtained is often very small, making it impossible to detect the surrounding environment well, and thus unable to make the optimal action in the appropriate state. Summary of the invention

[0004] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a UAV intelligent navigation method based on deep reinforcement learning, so as to solve the problems that UAV navigation relies on a known simulation environment and cannot make timely and accurate responses in the specific state of an unknown environment, as well as the problem that UAVs cannot effectively use environmental information of different dimensions to guide training when exploring the environment using reinforcement learning. The convergence speed of the algorithm of the UAV using deep reinforcement learning when exploring three-dimensional space is improved by designing discrete action space, sampling probabilities of different experience pools, adding strategy entropy and other methods.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A method for intelligent navigation of a drone based on deep reinforcement learning comprises the following steps:

[0007] Step (1), establishing a drone model in a three-dimensional simulation environment, and generating obstacles and target points, the drone is equipped with an onboard front camera and a laser radar, the front camera senses the two-dimensional outline of the obstacle, the laser radar senses the one-dimensional distance L between the drone and the obstacle, and the initial position of the drone is randomly generated in the environment, and the initial state of the drone is a landing state;

[0008] Step (2), designing a state space S and a discrete action space A for multi-dimensional feature fusion; the state space S includes the direction position of the drone, the quaternion used to calculate the pitch angle pitch and the roll angle roll of the drone's current flight state, the one-dimensional distance L between the drone and the obstacle, and the state vector Cam of the two-dimensional contour; the drone flies in a limited space; the actions of the discrete action space A include moving forward, moving backward, turning left, turning right, ascending, and landing;

[0009] Step (3), design a reward function based on Euclidean distance, expressed as:

[0010] R=r p +r s +r f

[0011] In the formula, R represents the total reward, r p is the continuous reward for performing an action when the drone is exploring the environment, r s is the sparse reward for successfully flying to the target point after a round of exploration, r f A sparse reward for a round of exploration that ends with a failure and fails to reach the target point;

[0012] Step (4), design the strategy network and state value network;

[0013] The input of the strategy network is the state space S of multi-dimensional feature fusion, and the output is the probability vector a_prob of the action to be performed by the drone, whose length is the number of actions in the action space. Action a is selected according to probability sampling and converted into the control amount of the drone to make the drone perform the action;

[0014] The input of the state value network is the state space S of multi-dimensional feature fusion, and the output is the evaluation V(s) of the current state;

[0015] Step (5), design a temporary experience pool to store the information feedback when the drone interacts with the environment;

[0016] Step (6), deep reinforcement learning network training based on discrete PPO algorithm;

[0017] Step (7), the policy network trained to convergence is used as the action decision controller of the UAV to achieve intelligent navigation.

[0018] In one embodiment, in step (1), the obstacle is a cuboid of the same color, the environment has a boundary, and the minimum boundary ws that the drone can reach is min =(x min ,y min ,z min ), the maximum boundary ws that the drone can reach max=(x max ,y max ,z max ), (x min ,y min ,z min ) and (x max ,y max ,z max ) are coordinate points in the environment; boundaries and obstacles are distinguished by walls of different colors.

[0019] In one embodiment, the state space established in step (2) is as follows:

[0020] S=(P x ,P y ,P z ,O w ,O x ,O y ,O z ,L,Cam)

[0021] P x is the x-axis position of the drone, P y is the y-axis position of the drone, P z is the z-axis position of the drone, satisfying:

[0022] ws min ≤(P x ,P y ,P z )≤ws max

[0023] O w , O x , O y , O z That is, the quaternion, roll angle roll, and pitch angle pitch satisfy:

[0024]

[0025] pitch = a sin [2 (O w O y -O z O x )]

[0026] The minimum value of the one-dimensional distance L between the drone and the obstacle is 0, and it is assigned a value of 5 when no obstacle is detected in front;

[0027] The state vector Cam of the two-dimensional profile is obtained in the following way:

[0028] After the front camera receives the RGB three-channel image data, it is input into the convolutional neural network for feature extraction to obtain the state vector Cam;

[0029] The action in the discrete action space A is completed by sending instructions for changing the linear velocity and angular velocity to the topic for controlling the flight of the drone.

[0030] In one embodiment, the convolutional neural network has three layers. The first layer has 3 input channels, 32 output channels, a convolution kernel size of 3x3, and a sampling step of 2. After the convolution, the network is activated by the ReLU function, and then a maximum pooling sampling operation is performed with a pooling size of 2x2. After processing, the network is passed to the second layer. The second layer has 32 input channels, 16 output channels, a convolution kernel size of 3x3, and a sampling step of 2. After the convolution, the network is activated by the ReLU function. The third layer is a fully connected neural network, which expands the result obtained by the convolutional neural network into a one-dimensional vector, and then converts the result into a vector with one row and three columns through the fully connected neural network, and then activates the state vector Cam through the ReLU function.

[0031] In one embodiment, in step (3), when the round end flag bit done is 0, r is calculated. p As current rewards:

[0032] r p =(500-step)*dif*ηdone=0

[0033] step is the current number of steps taken in one round of training. The more steps there are, the more rewards r will be. p The smaller the target point is, the position of the target point is G = (g x ,g y ,g z ), the current position is cur=(cur x ,cur y ,cur z ), the position of the previous state is pred=(pred x ,pred y ,pred z ), dif is the offset between the current position and the position of the previous action relative to the target point, which is equal to the current distance to the target point minus the Euclidean distance from the previous action to the target point;

[0034] η is a hyperparameter used to adjust r p The size of , makes the magnitude of all rewards balanced;

[0035] When done is 1 and des is True, calculate r s As current rewards:

[0036] rs =100done=1 and des=True

[0037] des is the flag for reaching the target point. If the current distance between the drone and the target point is less than ε, the drone is considered to have reached the target point and des is True. If the condition is not met, it is False.

[0038] When done is 1 and des is False, calculate r f As current rewards:

[0039] r f =-100done=1 and des=False.

[0040] In one embodiment, the following factors are used to determine whether the round is finished:

[0041] insi represents whether the drone is currently in the observation space. If the current position cur is greater than the minimum boundary ws min and is less than the maximum boundary ws max , then insi is true;

[0042] clp indicates whether the drone is very close to the ground. If the z direction of the current position is cur z Less than the minimum height threshold h min , then clp is true;

[0043] flip represents whether the drone crashes. If the current Euler angle eul = (pitch, roll) is less than the minimum Euler angle threshold eul min Or greater than the maximum Euler angle threshold eul max , then flip is true;

[0044] clo represents whether it is very close to an obstacle. If the radar reading L is less than the minimum threshold L min , then clo is true;

[0045] When insi is false or one of the other four flags is true, done is 1, indicating the end of an exploration round.

[0046] In one embodiment, the policy network comprises four layers, the first layer is an input layer, the number of inputs is 11, the second layer is a hidden layer, the number of nodes is 128, the third layer is a hidden layer, the number of nodes is 64, the fourth layer is an output layer, the number of nodes is the number of actions in the action space, and the activation function between all hidden layers of the policy network is a Tanh activation function;

[0047] The state value network includes four layers. The first layer is the input layer with 11 inputs, the second layer is the hidden layer with 128 nodes, the third layer is the hidden layer with 64 nodes, and the fourth layer is the output layer with 1 node, which represents the score of the state value network for the current input state. The activation function between all hidden layers of the state value network is the Tanh activation function.

[0048] In one embodiment, the temporary experience pool is also used to calculate the advantage function. When the current step number step exceeds the maximum training step number step specified in one round max , and when the drone is in good condition and still exploring, done = 0, but the state quantity at this time is not the terminal state. A variable flag is defined. If flag is True, it means normal termination, otherwise it means reaching the maximum number of steps in a single round to distinguish it from normal termination;

[0049] The information feedback when the drone interacts with the environment is the current state s generated by the drone's interaction with the environment each round. t , the current action a, the probability p of the action θ (a t |s t ), the reward r, the state s at the next moment t ′, flag, and done.

[0050] In one embodiment, the step (6) uses the PPO algorithm to perform obstacle avoidance and navigation training for the UAV, sets the total number of training cycles N, and in each cycle the UAV interacts with the environment, and based on the obtained state information s t The probability of the action is passed into the strategy network, the output action is sampled, and the new state information s is obtained after the action is executed. t ′, and judge the values ​​of flag and done, and store the obtained values ​​in the temporary experience pool in order; whenever the temporary experience pool is full, the data is taken out in batches to update the strategy network and the state value network. After the data is used up, the temporary experience pool is emptied, and the drone is allowed to explore the environment again and update the network until the network structure converges to a stable state.

[0051] In one embodiment, the action decision controller inputs the current state information s t , output the action probability and then sample it to get action a, and control the drone to perform action a until the drone reaches the target point.

[0052] Compared with the prior art, the present invention has the following advantages:

[0053] First, the present invention uses a multi-dimensional feature fusion method to obtain state information and pass it into the reinforcement learning neural network, so that the drone can make comprehensive decisions by making full use of environmental information of different dimensions, thereby becoming more intelligent in the process of obstacle avoidance and navigation, and making better actions.

[0054] Second, the simulation environment modeled by the present invention is realistic, and fully considers and handles various problems encountered by drones during flight, which has a strong reference significance for conducting drone reinforcement learning simulation in real scenes.

[0055] Third, the present invention adopts the discrete form of the PPO algorithm to design the action space. The appropriate action space is designed so that when the drone explores the environment, the dimension of the solution space is compressed and the exploration efficiency is improved without affecting the reduction of the control function accuracy, and the network is easier to converge.

[0056] Fourth, the present invention splits the experience pool and designs a sampling probability function to sample different sampling pools at different training stages, effectively solving the problem of slow training due to sparse rewards in the early stage of training and easy overfitting due to insufficient exploration efficiency in the later stage of training. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a schematic diagram of the overall principle of the present invention.

[0058] Figure 2 It is a schematic diagram of the multi-dimensional state quantity fusion of the present invention.

[0059] Figure 3 This is a strategic network structure diagram of the present invention.

[0060] Figure 4 This is the state value network structure diagram of the present invention.

[0061] Figure 5 , Figure 6 , Figure 7 Schematic diagram of the results of three experiments.

[0062] Figure 8 The following are the flight trajectory diagrams of the UAV in the three experiments. DETAILED DESCRIPTION

[0063] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.

[0064] As mentioned above, existing deep reinforcement learning obstacle avoidance navigation methods for drones often rely on known simulation environments, while reality is often unknown, so they do not have much practical significance. In addition, in simulations based on unknown environments, drones cannot fully perceive environmental information in different states, so they cannot make timely and accurate responses in specific states, and the performance effect is poor. Reinforcement learning training in three-dimensional space will result in slow network training efficiency or even non-convergence.

[0065] To solve these problems, the present invention proposes a UAV obstacle avoidance navigation method based on reinforcement learning. A simulation experiment is carried out in a three-dimensional environment. The two-dimensional image information and the one-dimensional state information are integrated and input into the reinforcement learning network for training, so that the UAV can fully detect the surrounding environment. The network convergence speed is optimized by designing discrete action space, splitting the experience pool for probability sampling, adding strategy entropy, etc., so as to make better actions under specific conditions, thereby achieving good navigation effect.

[0066] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods.

[0067] The present invention is a UAV intelligent navigation method based on deep reinforcement learning. Figure 1 As shown in the figure, a simulation environment for drones is established. The intelligent agent is composed of a strategy network and a state value network. After receiving the state quantity of the environment, the intelligent agent first fuses the state information of different dimensions and then sends the fused state information s t Passed to the policy network, the policy network calculates the probability p of each action in the action space based on the input state θ (a t |s t ), sample and select the action a, send the action to the agent control layer to make the agent take the corresponding action, and perceive the latest state s after the action is completed t+1 And calculate the reward r t and flag bit done,flag, the data sequence obtained above [s t ,a,p θ (a t |s t ),r,s t+1 ,done,flag] into the experience pool. When the experience pool is full, data is selected in batches. t ,s t+1 The input value network calculates the value estimates V(s) of the two states. t ),V(s t+1 ), combined with the r in the experience pool t ,flag calculates the advantage function A, and adds the calculated advantage function and pθ (a t |s t ) is passed to the policy network for parameter update, and the calculated advantage function and V(s t ) is passed to the value network for parameter update until the network converges. The converged policy network can guide the agent to make the best strategy in each state, thereby achieving obstacle avoidance navigation.

[0068] Specifically, the present invention comprises the following steps:

[0069] Step 1: Build a drone model in a 3D simulation environment and generate obstacles and target points.

[0070] In the present invention, a four-rotor drone is used, on which an onboard front camera and a laser radar are installed. The front camera is mainly used to perceive the two-dimensional outline of obstacles, and the laser radar is mainly used to perceive the one-dimensional distance L between the drone and the obstacles.

[0071] For example, the obstacle is a cuboid of the same color, and the environment has a boundary, which can be distinguished by a wall of a different color from the obstacle. The minimum boundary ws that the drone can reach min =(x min ,y min ,z min ), the maximum boundary ws that the drone can reach max =(x max ,y max ,z max ), (x min ,y min ,z min ) and (x max ,y max ,z max ) are coordinate points in the environment.

[0072] In the embodiment of the present invention, the size of the environment is a cube with a length and width of 10 meters and a height of 6 meters.

[0073] Before training begins, you need to initialize the drone's front camera and lidar to ensure that they can receive data correctly. Randomly generate the drone's initial position in the environment. The drone's initial state is landing. Send a command to the / takeoff topic to take off.

[0074] Step 2: Design the state space S and discrete action space A for multi-dimensional feature fusion.

[0075] refer to Figure 2 , the established state space S is as follows:

[0076] S=(P x ,P y ,P z,O w ,O x ,O y ,O z ,L,Cam)

[0077] P x is the x-axis position of the drone, P y is the y-axis position of the drone, P z is the z-axis position of the drone, which must satisfy the following requirements:

[0078] ws min ≤(P x ,P y ,P z )≤ws max

[0079] That is, ensure that the drone flies within a limited space.

[0080] QuaternionO w , O x , O y , O z Used to calculate the pitch angle and roll angle of the drone's current flight state:

[0081]

[0082] pitch = a sin [2 (O w O y -O z O x )]

[0083] The one-dimensional distance L between the drone and the obstacle is obtained through the reading of the lidar sensor, with a minimum value of 0 and a value of 5 when no obstacle is detected in front.

[0084] Cam is the state vector of the two-dimensional profile, which can be obtained as follows:

[0085] After the front camera receives the transmitted RGB three-channel image data (in this embodiment, its size is 640x360x3), it is input into the convolutional neural network for feature extraction.

[0086] In this embodiment, the convolutional neural network has three layers. The first layer has 3 input channels, 32 output channels, a convolution kernel size of 3x3, and a sampling step of 2. After the convolution is completed, it is activated by the ReLU function, and then the maximum pooling sampling operation is performed, and the pooling size is 2x2. After processing, it is passed to the second layer of the network. The second layer has 32 input channels, 16 output channels, a convolution kernel size of 3x3, a sampling step of 2, and the convolution is completed. The third layer is a fully connected neural network, which expands the result obtained by the convolutional neural network into a one-dimensional vector, and then converts the result into a vector of one row and three columns through the fully connected neural network, and then activates the state vector Cam through the ReLU function.

[0087] The action space of the present invention is a discrete action space A, which includes forward, backward, left turn, right turn, rise, and land, and is completed by sending instructions to change the linear velocity and angular velocity to the topic controlling the flight of the drone. For example, the execution time T of each action can be set to 0.3s.

[0088] Step 3: Design a reward function based on Euclidean distance.

[0089] R=r p +r s +r f

[0090]

[0091] In the formula, R represents the total reward, r p It is the continuous reward for performing an action when the drone is exploring the environment. When the round end flag bit done is 0, calculate r p As the current reward. step is the current number of steps taken in one round of training. The more steps there are, the more rewards r p The smaller the target point is, the position of the target point is G = (g x ,g y ,g z ), the current position is cur=(cur x ,cur y ,cur z ), the position of the previous state is pred=(pred x ,pred y ,pred z ), dif is the offset between the current position and the position of the previous action relative to the target point, which is equal to the current distance to the target point minus the distance from the previous action to the target point. The distance here refers to the Euclidean distance.

[0092]

[0093] η is a hyperparameter used to adjust rp The size of , makes the magnitude of all rewards balanced.

[0094] r s It is a sparse reward for successfully flying to the target point after a round of exploration. des is the flag of reaching the target point, satisfying:

[0095]

[0096] If the current distance between the drone and the target point is less than ε, the drone is considered to have reached the target point and des is True. If the condition is not met, it is False. When done is 1 and des is True, calculate r s As a current reward.

[0097] r f It is a sparse reward for a round of exploration that ends but fails to reach the target point. In this case, done is 1 and des is False.

[0098] The following factors determine whether the round is over:

[0099] insi represents whether the drone is currently in the observation space. If the current position cur is greater than the minimum value of the observation space (i.e. the minimum boundary), ws min And it is smaller than the maximum value (i.e. the maximum boundary) of the observation space ws max , then insi is true;

[0100]

[0101] clp indicates whether the drone is very close to the ground. If the z direction of the current position is cur z Less than the minimum height threshold h min , then clp is true;

[0102]

[0103] flip represents whether the drone crashes. If the current Euler angle eul = (pitch, roll) is less than the minimum Euler angle threshold eul min Or greater than the maximum Euler angle threshold eul max , then flip is true;

[0104]

[0105] clo represents whether it is very close to an obstacle. If the radar reading L is less than the minimum threshold L min , then clo is true.

[0106]

[0107] When insi is false or one of the other four flags is true, done is 1, indicating the end of an exploration round.

[0108]

[0109] Step 4: Design the strategy network and state value network.

[0110] The network of the reinforcement learning PPO algorithm includes a policy network and a state value network.

[0111] The input of the policy network is a multi-dimensional state quantity S = (P x ,P y ,P z ,O w ,O x ,O y ,O z ,L,Cam), the output is the probability vector a_prob of the action to be performed by the drone, and the length is the number of actions in the action space, such as Figure 3 As shown, action a is selected according to probability sampling and converted into the control amount of the UAV to make the UAV perform the action.

[0112] For example, the policy network includes four layers, the first layer is the input layer, the number of inputs is 11, the second layer is the hidden layer, the number of nodes is 128, the third layer is the hidden layer, the number of nodes is 64, and the fourth layer is the output layer, the number of nodes is the number of actions in the action space.

[0113] The activation function between all hidden layers of the policy network is the Tanh activation function. When generating the network, all network layers are orthogonally initialized. The weight matrix is ​​initialized with a Gaussian distribution with a mean of 0 and a standard deviation of 1. The weight matrix is ​​subjected to singular value decomposition to obtain two orthogonal matrices, one of which is taken as the weight matrix of the neural network layer.

[0114] The objective function that the policy network ultimately maximizes is:

[0115]

[0116] Where p θ (a t |s t ) represents the new strategy, represents the old policy, Represents the old advantage function. If the difference between the new and old strategies is too large, it will affect the results. The clip function is used to limit the ratio of the new and old strategies to between 1-ε and 1+ε, thereby improving the effectiveness of training. The process of updating each parameter is:

[0117]

[0118] The input of the state value function estimation network is also the state quantity S, and the output is the evaluation of the current state V(s). Figure 4 As shown in the figure, the state value network also contains four layers. The first layer is the input layer with 11 inputs, the second layer is the hidden layer with 128 nodes, the third layer is the hidden layer with 64 nodes, and the fourth layer is the output layer with 1 node, which represents the score of the state value network for the current input state. The activation function between all hidden layers of the state value network is the Tanh activation function, and all network layers are orthogonally initialized when generating the network.

[0119] Update the state value function using the squared error and gradient descent method:

[0120]

[0121] Step 5: Design a temporary experience pool to store interaction information.

[0122] The temporary experience pool of the present invention is also used to calculate the advantage function. In this training framework, the current step number step exceeds the maximum training step number step specified in one round. max When the drone is in good condition and still exploring, done = 0. The state quantity S at this time is not the termination state. To distinguish it from the normal termination, a variable flag is defined. If flag is True, it indicates normal termination, otherwise it indicates that the maximum number of steps in a single round has been reached and the termination has been completed.

[0123] In the present invention, the information feedback when the drone interacts with the environment is the current state s generated by the drone's interaction with the environment each round. t , the current action a, the probability p of selecting the corresponding action in a certain state θ (a t |s t ), the reward r, the state s at the next moment t ′, flag, done, the specific form is [s t ,a,p θ (a t |s t ),r,s t+1 ,flag,done].

[0124] The experience pool is divided into a positive experience pool and a normal experience pool. The positive experience pool stores training samples with r greater than zero, and the normal experience pool stores all samples. The algorithm is based on the sampling probability. Sampling from the positive experience pool, according to the sampling probability 1-P sam Sample from the normal experience pool. epi is the training period. In the early stage of training, there is a high probability of sampling from the positive experience pool to train the network. As the training period increases, the sampling probability Psam Gradually tends to zero, and all samples come from the common experience pool, which effectively overcomes the problem of sparse rewards in the early stage of training, slow training efficiency, and insufficient exploration efficiency in the later stage of training, resulting in overfitting.

[0125] Step 6: Perform reinforcement learning network training based on the PPO algorithm.

[0126] Advantage function A t (s t ,a t ) is calculated in GAE form:

[0127]

[0128] In the formula, γ is the discount rate, and λ is the adjustment factor. The closer it is to 1, the larger the variance and the smaller the deviation, and vice versa if it is close to 0. t ) is the current output value of the value evaluation network.

[0129] In order to distinguish whether the flag is True or False, that is, the drone training ends normally or is abnormally truncated, the advantage function Calculation, add conditional flag:

[0130]

[0131] When flag is True, it means that the agent ends the exploration normally and there is no next state V(s t+1 ).

[0132] In order to increase the exploration ability of the algorithm, a policy entropy is added to the objective function of the policy network:

[0133] J(θ)=J(π θ )+entr*coef

[0134] Where entr is the policy entropy and coef is the coefficient.

[0135] The PPO algorithm is used to train the obstacle avoidance and navigation of the UAV. The total number of training cycles N is set. In each cycle, the UAV interacts with the environment and obtains the state information s. t The probability of the action is passed into the strategy network, the output action is sampled, and the new state information s is obtained after the action is executed. t+1 , and judge the values ​​of variables such as flag and done, and store these values ​​in the experience pool in order; whenever the experience pool is full, the data is taken out in batches to calculate the gradient update strategy network and state value network. After the experience pool data is used up, the experience pool is cleared and the drone is allowed to explore the environment again and update the network until the network structure converges to a stable state.

[0136] Step 7: Implementation of UAV intelligent navigation.

[0137] The policy network trained to convergence is directly used as the action decision controller of the drone, and the current state information s is passed in t , output the action probability and then sample it to get action a, and control the drone to perform action a until the drone reaches the target point. Figure 5 , Figure 6 , Figure 7 There are three simulation experiments. When facing obstacles, the policy network outputs action decisions to guide the drone to avoid obstacles and reach the vicinity of the target point. The white parts around are walls, the black cubes with shadows inside are obstacles, and the four white lines emitted from the front of the drone are the field of view of the front camera. Figure 8 The flight trajectories of the drone in three experiments are shown. It can be seen that in the three experiments, the drone successfully arrived near the target point after avoiding obstacles.

Claims

1. A drone intelligent navigation method based on deep reinforcement learning, characterized in that: The steps include: Step (1), establishing a drone model in a three-dimensional simulation environment, and generating obstacles and target points, the drone is equipped with an onboard front camera and a laser radar, the front camera senses the two-dimensional outline of the obstacle, the laser radar senses the one-dimensional distance L between the drone and the obstacle, and the initial position of the drone is randomly generated in the environment, and the initial state of the drone is a landing state; Step (2) designs a state space S and a discrete action space A for multi-dimensional feature fusion; the state space S includes the direction position of the drone, the quaternion used to calculate the pitch angle pitch and the roll angle roll of the drone's current flight state, the one-dimensional distance L between the drone and the obstacle, and the state vector Cam of the two-dimensional profile; the drone flies in a limited space; the actions of the discrete action space A include forward, backward, left turn, right turn, rise, and land; the state vector Cam of the two-dimensional profile is obtained by the following method: After the front camera receives the RGB three-channel image data, it is input into the convolutional neural network for feature extraction to obtain the state vector Cam; Step (3), design a reward function based on Euclidean distance, expressed as: R＝r p +r s +r f In the formula, R represents the total reward, r p is the continuous reward for performing an action when the drone is exploring the environment, r s is the sparse reward for successfully flying to the target point after a round of exploration, r f A sparse reward for a round of exploration that ends with a failure and fails to reach the target point; Among them, when the round end flag bit done is 0, calculate r p As current rewards: r p =(500-step)*dif*ηdone=0 step is the current number of steps taken in one round of training. The more steps there are, the more rewards r will be. p The smaller the target point is, the position of the target point is G = (g x , g y , g z ), the current position is cur=(cur x ,cur y ,cur z ), the position of the previous state is pred=(pred x , pred y , pred z ), dif is the offset between the current position and the position of the previous action relative to the target point, which is equal to the current distance to the target point minus the Euclidean distance from the previous action to the target point; η is a hyperparameter used to adjust r p The size of , makes the magnitude of all rewards balanced; When done is 1 and des is True, calculate r s As current rewards: r s =100done=1and des=True des is the flag for reaching the target point. If the current distance between the drone and the target point is less than ε, the drone is considered to have reached the target point and des is True. If the condition is not met, it is False. When done is 1 and des is False, calculate r f As current rewards: r f =-100done=1and des=False The following factors determine whether the round is over: insi represents whether the drone is currently in the observation space. If the current position cur is greater than the minimum boundary ws min and is less than the maximum boundary ws max , then insi is true; clp indicates whether the drone is very close to the ground. If the z direction of the current position is cur z Less than the minimum height threshold h min , then clp is true; flip represents whether the drone crashes. If the current Euler angle eul = (pitch, roll) is less than the minimum Euler angle threshold eul min Or greater than the maximum Euler angle threshold eul max , then flip is true; clo represents whether it is very close to an obstacle. If the radar reading L is less than the minimum threshold L min , then clo is true; When insi is false or one of the other four flags is true, done is 1, indicating the end of an exploration round; Step (4), design the strategy network and state value network; The input of the strategy network is the state space S of multi-dimensional feature fusion, and the output is the probability vector a_prob of the action to be performed by the drone, whose length is the number of actions in the action space. Action a is selected according to probability sampling and converted into the control amount of the drone to make the drone perform the action; The input of the state value network is the state space S of multi-dimensional feature fusion, and the output is the evaluation V(s) of the current state; Step (5), design a temporary experience pool to store the information feedback when the drone interacts with the environment; Step (6), deep reinforcement learning network training based on discrete PPO algorithm; Step (7), the policy network trained to convergence is used as the action decision controller of the UAV to achieve intelligent navigation.

2. The UAV intelligent navigation method based on deep reinforcement learning according to claim 1 is characterized in that: In step (1), the obstacle is a cuboid of the same color, the environment has a boundary, and the minimum boundary ws that the drone can reach min =(x min ,y min , z min ), the maximum boundary Ws that the drone can reach max =(x max ,y max , z max ), (x min ,y min , z min ) and (x max ,y max , z max ) are coordinate points in the environment; boundaries and obstacles are distinguished by walls of different colors.

3. The UAV intelligent navigation method based on deep reinforcement learning according to claim 2 is characterized in that: In step (2), the state space established is as follows: S=(P x ,P y ,P z ,O w ,O x ,O y ,O z ,L,Orange) P x is the x-axis position of the drone, P y is the y-axis position of the drone, P z is the z-axis position of the drone, satisfying: ws min ≤(P x ,P y ,P z )≤ws max O w , O x , O y , O z That is, the quaternion, roll angle roll, and pitch angle pitch satisfy: pitch=asin[2(O w EITHER y -EITHER z EITHER x )] The minimum value of the one-dimensional distance L between the drone and the obstacle is 0, and it is assigned a value of 5 when no obstacle is detected in front; The action in the discrete action space A is completed by sending instructions for changing the linear velocity and angular velocity to the topic for controlling the flight of the drone.

4. The UAV intelligent navigation method based on deep reinforcement learning according to claim 3 is characterized in that: The convolutional neural network has three layers. The first layer has 3 input channels, 32 output channels, a convolution kernel size of 3x3, and a sampling step of 2. After the convolution, it is activated by the ReLU function, and then the maximum pooling sampling operation is performed with a pooling size of 2x2. After processing, it is passed to the second layer of the network. The second layer has 32 input channels, 16 output channels, a convolution kernel size of 3x3, and a sampling step of 2. After the convolution, it is activated by the ReLU function. The third layer is a fully connected neural network, which expands the result obtained by the convolutional neural network into a one-dimensional vector, and then converts the result into a vector with one row and three columns through the fully connected neural network, and then activates the state vector Cam through the ReLU function.

5. The UAV intelligent navigation method based on deep reinforcement learning according to claim 1 is characterized in that: The strategy network comprises four layers, the first layer is the input layer, the number of inputs is 11, the second layer is the hidden layer, the number of nodes is 128, the third layer is the hidden layer, the number of nodes is 64, the fourth layer is the output layer, the number of nodes is the number of actions in the action space, and the activation function between all hidden layers of the strategy network is the Tanh activation function; The state value network includes four layers. The first layer is the input layer with 11 inputs, the second layer is the hidden layer with 128 nodes, the third layer is the hidden layer with 64 nodes, and the fourth layer is the output layer with 1 node, which represents the score of the state value network for the current input state. The activation function between all hidden layers of the state value network is the Tanh activation function.

6. The UAV intelligent navigation method based on deep reinforcement learning according to claim 1 is characterized in that: The temporary experience pool is also used to calculate the advantage function when the current step number step exceeds the maximum training step number step specified in one round. max , and when the drone is in good condition and still exploring, done = 0, but the state quantity at this time is not the terminal state. A variable flag is defined. If flag is True, it means normal termination, otherwise it means reaching the maximum number of steps in a single round to distinguish it from normal termination; The information feedback when the drone interacts with the environment is the current state s generated by the drone's interaction with the environment each round. t , the current action a, the probability p of the action θ (a t |s t ), the reward r, the state s at the next moment t ′, flag, and done.

7. The UAV intelligent navigation method based on deep reinforcement learning according to claim 1 is characterized in that: In step (6), the PPO algorithm is used to perform obstacle avoidance and navigation training for the UAV, and the total number of training cycles N is set. In each cycle, the UAV interacts with the environment and obtains the state information s. t The probability of the action is passed into the strategy network, the output action is sampled, and the new state information s is obtained after the action is executed. t ′, and judge the values ​​of flag and done, and store the obtained values ​​in the temporary experience pool in order; Whenever the temporary experience pool is full, the data is taken out in batches to update the strategy network and the state value network. After the data is used up, the temporary experience pool is cleared, and the drone is allowed to explore the environment again and update the network until the network structure converges to a stable state.

8. The UAV intelligent navigation method based on deep reinforcement learning according to claim 1 is characterized in that: The action decision controller passes in the current state information s t , output the action probability and then sample it to get action a, and control the drone to perform action a until the drone reaches the target point.

Images