Path planning method combining double-adversarial technology and recursion mechanism

Abstract

The invention belongs to the technical field of aircraft control, and particularly relates to a path planning method combining a double-confrontation technology and a recursion mechanism. According to the method, an optimal Q value function is calculated through a Bellman equation, and a deep learning Q network model DQN which continuously trains and optimizes parameters and replaces a traditional Q value table with a network output value is established; constructing a deep recursion Q network model DRQN containing a cyclic network; in combination with a double-adversarial technology, a double-adversarial deep recursion Q network model D3RQN which combines a value function and a reward value function and introduces a target network parameter learning rate is established, and path planning of the four-rotor aircraft in a semi-known environment is realized. According to the method, on the premise of a semi-known environment, the burden of the calculation amount is relieved, the performance of the POMDP of the incomplete observable Markov decision process is improved, the training speed is increased, meanwhile, the problem that value estimation is too optimistic is solved, the system stability and accuracy are improved, and the path planning performance of the quadrotor unmanned aerial vehicle for reconnaissance and transportation is improved.

Description

Technical Field The invention belongs to the field of aircraft control technology, and in particular relates to a path planning method combining a dual confrontation technology and a recursive mechanism. Background Art In order to meet the engineering application requirements of quadrotor reconnaissance and transportation, a dual-adversarial deep recursive Q-network was designed based on quadrotor path planning to overcome the structural defects of the existing quadrotor path planning using DQN (deep Q-network) structure, such as poor performance, slow training speed, and overly optimistic value estimation. However, the complex structural characteristics and multiple relationships of the dual-adversarial deep recursive Q-network make it difficult to establish the network, and it is directly related to the stability of the operation of the quadrotor for reconnaissance and transportation and the accuracy of transportation. The paper "Path planning for active SLAM based on deep reinforcement learning under unknown environments." (Wen, S. et al., Intell. Serv. Robot. 2020, 13, pp. 263–272) designs a method based on simultaneous localization and mapping (SLAM) to plan paths in the absence of prior information. The method updates nearby environmental observations based on onboard sensor information, identifies obstacles, and avoids them to continue moving forward. This method has two shortcomings: (1) it is not sufficient to support the effective and safe flight of a quadcopter in a semi-known environment; (2) when encountering unlisted or unset obstacles, the method may no longer be effective and requires a large amount of computing resources from the onboard processor. In the literature "Human-level control through deep reinforcement learning" (Mnih, V. et al., Nature 2015, 518, pp. 529–533), a method based on deep reinforcement learning for global path planning and autonomous obstacle avoidance that no longer relies on real-time feature matching is designed to address the heavy computational pressure. This method has some shortcomings: (1) It is not sufficient to support the effective and safe flight of a quadcopter drone in a semi-known environment. Summary of the invention In order to overcome the shortcomings of the prior art, the present invention establishes a DQN (deep Q-network) model for a quad-rotor UAV for reconnaissance and transportation, and proposes a path planning method that combines dual adversarial technology and recursive mechanism. Under the premise of semi-known environmental information, the path planning method first designs a global path planning method that takes an obstacle map as input to generate possible paths from the starting position to the target position without collision detection during planning, and then designs a perception module based on unsupervised contrastive learning, which can not only autonomously avoid obstacles in a semi-known environment, but also improve the stability and accuracy of global path planning. In order to achieve the above-mentioned invention object, the present invention adopts the following technical solution: a path planning method combining double confrontation technology and recursive mechanism, comprising the following steps: Step 1: Calculate the optimal Q value function through the Bellman equation to establish a deep learning Q network model DQN that continuously trains and optimizes parameters and uses network output values ​​to replace the traditional Q value table; Step 2: Construct a deep recursive Q network model DRQN ​​including a recurrent network; Step 3. Combined with the dual adversarial technology, a dual adversarial deep recursive Q network model D3RQN is established, which combines the value function with the reward value function and introduces the target network parameter learning rate to realize the path planning of the quadrotor aircraft in a semi-known environment. Further, as a preferred technical solution of the present invention, in step 1, path planning and obstacle avoidance are defined as an array (S, A, T, R, O, D); Among them, S is a set of state values, called the system state space; A is a set of action values, referring to the action space of the quadrotor; T is a set of transition values, describing how the state is updated based on the selected operation and the current state; R is a set of reward values ​​obtained by the agent for performing each action; O is a set of observation values, which is the global environment information used for global path planning or sensor information for obstacle avoidance; D is a set of probability distributions; Each time it advances t steps, the agent receives observations o t ∈O, action value a t ∈A and the reward function R(s t , a t ) The reward r given t ; The system is based on the conversion model T(s t+1 |s t , a t ) Generate a new state s t+1 , and the agent updates the observation o t+1 ; In global path planning, the observation value o t Equal to the state value s t ; From the strategy π=P(a t |st ) is sampled to obtain the action value a t ; Through the action state value function Q(s t , a t ) approximates the expected value of the cumulative reward; the optimal strategy π is used as a solution to make the cumulative future return Maximize, where γ is the discount factor; the Bellman equation is used to calculate the optimal Q value function: Further, as a preferred technical solution of the present invention, in step 2, a recursive mechanism is used to extract useful environmental information from continuous observations, and the first convolutional fully connected layer is replaced by an LSTM recurrent layer; The recursive mechanism refers to replacing the third or fourth fully connected layer with an LSTM layer, and then returning to the LSTM layer after the LSTM layer until the recursion ends; for the POMDP type obstacle avoidance of monocular vision, the observation value o t Cannot completely replace the system status value s t , which leads to the estimated value of Q(o t , a t |π)≠Q(s t , a t |π); Therefore, the recursive mechanism is used to extract useful environmental information from continuous observations, which can well estimate the state sequence from the observation value sequence; The convolution layer is a layer that reduces the dimension and extracts features of the input image through convolution operations. The fully connected layer is a matrix multiplication, which is equivalent to a feature space transformation. It is a layer that has the function of extracting and integrating all the previous useful information. The convolution fully connected layer is a layer that extracts local features through convolution operations and reassembles all local features into a complete graph through the weight matrix through full connection operations. LSTM layer refers to the long short-term memory network LSTM (long short-term memory); the core concepts of the LSTM layer are the cell state and the "gate" structure; the cell state is equivalent to the path of information transmission, allowing information to be passed on in the sequence; the "gate" structure will learn what information to save or forget during the training process. Further, as a preferred technical solution of the present invention, in step 3, an adversarial network is introduced on the DRQN ​​network structure, and the features extracted by the LSTM convolutional neural network are divided into two paths, namely: the state value function and the state-dependent action advantage function; The adversarial network consists of two streams representing the value and advantage functions, while sharing a common convolutional feature learning module, and then combining the two streams through a special aggregation layer to produce an estimate of the state-action value function Q; the adversarial network automatically produces independent estimates of the state-value function and the advantage function without any additional supervision; In the D2RQN network structure, a Q′ network is added, and both Q and Q′ networks are used at the same time. One Q network is used to update parameters to select actions, and the other fixed Q′ network is used as the target network to calculate the Q value. Due to errors in the network, overestimated actions will be repeatedly selected, so the Q value is often overestimated. Using both Q and Q' networks can avoid the overestimation of the Q value. Because the state value s is not available in the imperfectly observable Markov decision process (POMDP), the Q-value function and loss function are expressed as: Q π (Ω t ,a t )=A π (Ω t ,a t )+V π (Ω t ) in, Ω t is the observation sequence before time t, θ - is a fixed and independent parameter of the target network. Therefore, the network can learn an estimate of the value function by minimizing the loss function. The derivative of this strategy is: The D3RQN model acts as a planner, selecting the best action based on previous observations and then updating the next position of the quadrotor. This operation is repeated until the quadrotor reaches the destination. The collection of optimal actions constitutes a feasible path for the quadrotor in the environment. The reward function of the global path planning model is defined as: Among them, d curr is the distance between the current position and the target position, d next is the distance between the next position and the target position, d step is the length of the step. K is a constant 3. In this design, the reward is usually negative before reaching the goal, which can avoid ineffective wandering of the agent. To train the encoder, random data augmentation is used to generate positive and negative pairs. Two samples generated from any sample with different augmentation operators are considered positive pairs. Two samples generated from two different samples with arbitrary augmentation operators are considered negative pairs. The encoder should learn to maximize the similarity of the representation features extracted from any positive pair while minimizing the similarity of any negative pair. A contrastive loss is introduced to train the encoder, which is defined as follows: In the training of the decision module, the reward function is defined as: Here is the nst It is the distance to the nearest obstacle at time t, that is, the safety distance is 0.5 meters. The path planning method combining the dual confrontation technology and the recursive mechanism described in the present invention adopts the above technical solution and has the following technical effects compared with the prior art: (1) The present invention reduces the computational burden by adopting a deep learning Q network. (2) The present invention improves the performance of the imperfectly observable Markov decision process (POMDP) ​​and increases the training speed by adopting a recursive mechanism and a double duel technique, while solving problems such as overly optimistic value estimates. (3) The present invention designs a path planning method for a quadrotor aircraft in a semi-known environment based on a dual-adversarial deep recursive Q network to improve system stability and accuracy. Therefore, this method can effectively improve the path planning performance of a quadrotor drone for reconnaissance and transportation in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 It is a schematic diagram of the path planning method flow of the present invention; Figure 2 This is a real picture of a quad-rotor drone used for reconnaissance and transportation; Figure 3 It is a system principle diagram of the deep learning Q network combining the recursive mechanism and the double adversarial technology of the present invention; Figure 4 is a diagram of the process of training the encoder; Figure 5 It is the overall structure diagram of the control system; Figure 6 is the learning curve of the global path planning model; Figure 7 is the learning curve of the obstacle avoidance model; In the attached figure, 1. High-definition camera; 2. Motor and propeller; 3. External equipment; 4. Flight control computer; 5. Bracket. DETAILED DESCRIPTION The present invention is further explained below in detail with reference to the accompanying drawings so that those skilled in the art can more deeply understand the present invention and be able to implement it. However, the following reference examples are only used to explain the present invention and are not intended to limit the present invention. like Figure 1 As shown, the present invention first calculates the optimal Q value function through the Bellman equation to establish a deep learning Q network model (Deep Qnetwork, DQN) that continuously trains and optimizes parameters and uses network output values ​​to replace the traditional Q value table, thereby avoiding falling into the dimensionality disaster; further, a deep recurrent Q network model (Deep Recurrent Q network, DRQN) containing a recurrent network is constructed to improve the convergence of the algorithm; then, a double dueling technology is introduced to establish a double dueling deep recurrent Q network model (Double Dueling Deep Recurrent Q network, D3RQN) that combines the value function with the reward value function and introduces the learning rate of the target network parameters, so as to realize the path planning of the quadcopter in a semi-known environment. Figure 2 The figure shows a real-life picture of a quad-rotor UAV for reconnaissance and transportation, including a high-definition camera 1, a motor and propeller 2, external equipment 3, a flight control computer 4, and a bracket 5. The specific method of the present invention is as follows: Step 1, calculate the optimal Q value function through the Bellman equation to establish a deep learning Q network model (DeepQnetwork, DQN) that continuously trains and optimizes parameters and uses network output values ​​to replace the traditional Q value table, thereby avoiding falling into the dimensionality disaster. The path planning problem and obstacle avoidance problem are defined as an array (S, A, T, R, O, D). Among them, S is a set of state values, called the system state space. A is a set of action values, which refers to the action space of the quadrotor. T is a set of transition values ​​that describes how the state is updated based on the selected action and the current state. R is a set of reward values ​​that the agent receives for performing each action. O is a set of observations, which may be global environment information for global path planning or sensor information for obstacle avoidance. D is a set of probability distributions.

[0053] Each time it advances t steps, the agent receives observations o t ∈O, action value a t ∈A and the reward function R(s t , a t ) The reward r given t The system is based on the conversion model T(s t +1|s t , a t ) Generate a new state s t + 1 , and the agent updates the observation ot+1 In global path planning, the observation value o t Equal to the state value s t From the strategy π=P(a t |s t ) is sampled to obtain the action value a t . Through the action state value function Q(s t , a t ) approximates the expected value of the cumulative reward. The optimal strategy π is used as a solution to make the cumulative future return Maximize, where γ is the discount factor. The Bellman equation is used to calculate the optimal Q value function: Step 2: Construct a deep recurrent Q network model (DRQN) containing a recurrent network to improve the convergence of the algorithm. like Figure 3 As shown in Figure 1, the recursive mechanism refers to replacing the third or fourth fully connected layer with an LSTM layer, and continuing to return to the LSTM layer after the LSTM layer until the recursion ends. For monocular vision POMDP obstacle avoidance, the observation value o t Cannot completely replace the system status value s t , which leads to the estimated value of Q(o t , a t Iπ)≠Q(s t , a t |π). Therefore, the recursive mechanism is used to extract useful environmental information from continuous observations, which can well estimate the state sequence from the observation sequence. The convolution layer is a layer that performs dimensionality reduction and feature extraction on the input image through convolution operations. The fully connected layer is a matrix multiplication, which is equivalent to a feature space transformation. It is a layer that has the function of extracting and integrating all the previous useful information. The convolution fully connected layer is a layer that extracts local features through convolution operations and reassembles all local features into a complete graph through the weight matrix through the fully connected operation. LSTM layer refers to the long short-term memory network LSTM (long short-term memory). The core concept of LSTM layer lies in the cell state and the "gate" structure. The cell state is equivalent to the path of information transmission, allowing information to be passed on in the sequence connection. The "gate" structure will learn what information to save or forget during the training process. Step 3. Introduce double dueling technology and establish a double dueling deep recurrent Q network model (D3RQN) that combines the value function with the reward value function and introduces the target network parameter learning rate to realize the path planning of the quadrotor aircraft in a semi-known environment. In the DRQN ​​network structure, an adversarial network is added to divide the features extracted by the LSTM convolutional neural network into two paths, namely: the state value function and the state-dependent action advantage function. The adversarial network consists of two streams representing the value and advantage functions, while sharing a common convolutional feature learning module, and then combining the two streams through a special aggregation layer to produce an estimate of the state-action value function Q. The adversarial network automatically produces independent estimates of the state-value function and the advantage function without any additional supervision. In the D2RQN network structure, the Q' network is added, and the two networks Q and Q' are used at the same time. One Q network is used to update parameters to select actions, and the other fixed Q' network is used as the target network to calculate the Q value. Due to errors in the network, overestimated actions will be repeatedly selected, so the Q value is often overestimated. Using both Q and Q' networks can avoid overestimation of the Q value. Because the state value s is not available in the imperfectly observable Markov decision process (POMDP), the Q-value function and loss function are expressed as: Q π (Ω t , a t )=A π (Ω t , a t )+V π (Ω t ) in, Ω t is the observation sequence before time t, θ - is a fixed and independent parameter of the target network. Therefore, the network can learn an estimate of the value function by minimizing the loss function. The derivative of this strategy is: The D3RQN model acts as a planner, selecting the best action based on previous observations and then updating the next position of the quadrotor. This operation is repeated until the quadrotor reaches the destination. The collection of optimal actions constitutes a feasible path for the quadrotor in the environment. The reward function of the global path planning model is defined as: Among them, d curr is the distance between the current position and the target position, d next is the distance between the next position and the target position, d step is the length of the step. K is a constant 3. In this design, the reward is usually negative before reaching the goal, which can avoid ineffective wandering of the agent. like Figure 4 , in order to train the encoder, random data augmentation is used to generate positive and negative pairs. Two samples generated from any samples with different augmentation operators are considered positive pairs. Two samples generated from two different samples with arbitrary augmentation operators are considered negative pairs. The encoder should learn to maximize the similarity of the representation features extracted from any positive pair while minimizing the similarity of any negative pair. The contrastive loss is introduced to train the encoder, which is defined as follows: In the training of the decision module, the reward function is defined as: Here is the nst It is the distance to the nearest obstacle at time t, that is, the safety distance is 0.5 meters. like Figure 5 As shown in the figure, in order to realize autonomous flight, the overall design of the quadcopter UAV system is carried out. It can be seen that the entire control structure is divided into a trajectory planning layer and a controller layer. The trajectory planning provides the controller with the expected position and the expected yaw angle; the position controller obtains the expected acceleration and the expected pitch angle and roll angle by the deviation between the expected position and the actual position; the expected attitude angle is used as the input of the attitude controller to obtain the expected angular acceleration; the expected acceleration and angular acceleration are provided to the UAV for control, and finally the UAV and load status are fed back to the trajectory planning and control layer to realize real-time feedback control. In order to verify the practicality and feasibility of the path planning algorithm of the quadrotor aircraft based on the dual adversarial deep recursive Q network in a semi-known environment in a multi-obstacle environment, the present invention simulates the path planning of the traditional DQN algorithm, D3QN algorithm, DRQN ​​algorithm and D3RQN algorithm in the same multi-obstacle environment map, and verifies that the path planning algorithm of the quadrotor aircraft based on the dual adversarial deep recursive Q network in a semi-known environment has significant improvements on the path planning results and training speed of the quadrotor aircraft, while solving the problem of overly optimistic value estimation, and generating an optimal path with good obstacle avoidance effect and high smoothness. The simulation experiment uses an Inter(R)Core(TM) i5-82651.80GHz processor, generated in Windows11, PyCharm Community Edition 2021.3.2, Anaconda3 (64-bit), and Python3.7. The training of the global path planner is based on random obstacle generation. In each training iteration, the positions and sizes of multiple obstacles are randomly changed. The random obstacle information is converted into a 120*160 tensor, and the random obstacle information is used together with the target position information and the current position information to form the observation variable. The trained global path planner is evaluated in randomly generated scenarios. The global path planner proposed in the present invention is compared with planners based on the DQN algorithm, the D3QN algorithm, the DRQN ​​algorithm and the D3RQN algorithm. The global path planner proposed in the present invention has the same action space and the same number of network layers as the planners based on the DQN algorithm, the D3QN algorithm, the DRQN ​​algorithm and the D3RQN algorithm. DQN has three convolutional layers and two fully connected layers; the first four layers of D3QN are the same as DQN, and the fifth layer is divided into two branches, which has computational advantages and value. The D3RQN-based planner network proposed in the present invention can be obtained by replacing the fourth layer of D3QN with an LSTM layer. In addition, DQN and D3QN have the same training hyperparameters as the planner proposed in the present invention. The learning curve of the planner is shown in Figure 2. Figure 6 shown. The autonomous obstacle avoidance planner is trained by extracting low-dimensional representation features from high-dimensional visual observations and selecting the best obstacle avoidance actions. The sensor observation data are RGB images captured by a simulated fixed camera. The training of the autonomous obstacle avoidance planner is performed using a step-by-step strategy. First, features are extracted from the original scene images to train the encoder. Then, the D3RQN algorithm is trained for obstacle avoidance based on the pre-trained encoder. The training images are collected by a monocular camera through a manually controlled quadrotor in a simulated environment. The current q-value of the observer is estimated by training the D3RQN model, and the feature vector is generated by the pre-trained encoder. Finally, the trained autonomous obstacle avoidance planner is evaluated in the simulation scene. If the quadrotor safely flies more than 50 steps, it is considered a success. The safety performance of the autonomous obstacle avoidance planner is evaluated by calculating the success rate after 2000 test flights and compared with the DQN, DRQN ​​and D3QN obstacle avoidance planners with the same action space and the same number of network layers. DQN has four fully connected layers. The first three layers of D3QN are the same as DQN, and the fourth layer is divided into two branches in order to calculate the advantage and value functions. DRQN ​​can be obtained by replacing the third layer of DQN with an LSTM layer. By replacing the third layer of D3QN with the LSTM layer, the obstacle avoidance planner proposed in the present invention can be obtained. In addition, all obstacle avoidance planners have the same training hyperparameters. Figure 7 is the learning curve for these models. When obstacles are out of the field of view of the onboard camera, the quadrotor cannot maintain a safe distance from the obstacles. Expanding the field of view of the sensor or increasing the number of sensors will help solve this problem, and these improvements will be considered in future work. Figure 6 and Figure 7 It shows that in the presence of uncertain factors such as semi-known environment, external interference, air flow resistance and system delay, the path planning method for quadrotor aircraft based on dual adversarial deep recursive Q network in a semi-known environment proposed by the present invention is more efficient than the classic path planner (such as DQN or D3QN). It proves that the proposed model can plan feasible paths with obstacle map information as input, and can be used to perform global path planning and autonomous obstacle avoidance in a semi-known environment. In summary, the present invention is a path planning method for a quadcopter based on a double dueling deep recursive Q network in a semi-known environment. First, the best Q value function is calculated by the Bellman equation to establish a deep learning Q network model (Deep Q network, DQN) that continuously trains and optimizes parameters and uses network output values ​​to replace the traditional Q value table, thereby avoiding falling into the dimensionality disaster; further, a deep recursive Q network model (Deep Recurrent Qnetwork, DRQN) containing a recurrent network is constructed to improve the convergence of the algorithm; then, a double dueling technology is introduced to establish a double dueling deep recurrent Q network model (Double Dueling DeepRecurrent Qnetwork, D3RQN) that combines the value function with the reward value function and introduces the learning rate of the target network parameters, so as to realize the path planning of the quadcopter in a semi-known environment. Under the premise of a semi-known environment, the present invention not only greatly reduces the burden of calculation, but also improves the performance of the incompletely observable Markov decision process (POMDP) ​​and improves the training speed, while solving the problems of overly optimistic value estimation, improving system stability and accuracy, thereby improving the path planning performance of the quadcopter UAV for reconnaissance and transportation. The specific implementation scheme described above further describes in detail the purpose, technical scheme and beneficial effects of the present invention. It should be understood that the above is only a specific implementation scheme of the present invention and is not intended to limit the scope of the present invention. Any equivalent changes and modifications made by any technician in the field without departing from the concept and principle of the present invention should fall within the scope of protection of the present invention.

Claims

1. A path planning method combining dual confrontation technology and recursive mechanism, characterized in that: The following steps are involved: Step 1: Calculate the optimal Q value function through the Bellman equation to establish a deep learning Q network model DQN that continuously trains and optimizes parameters and uses network output values ​​to replace the traditional Q value table; Step 2: Construct a deep recursive Q network model DRQN ​​including a recurrent network; Step 3. Combined with the dual adversarial technology, a dual adversarial deep recursive Q network model D3RQN is established, which combines the value function with the reward value function and introduces the target network parameter learning rate to realize the path planning of the quadrotor aircraft in a semi-known environment.

2. A path planning method combining dual confrontation technology and recursive mechanism according to claim 1, characterized in that: In step 1, path planning and obstacle avoidance are defined as an array (S, A, T, R, O, D); Among them, S is a set of state values, called the system state space; A is a set of action values, referring to the action space of the quadrotor; T is a set of transition values, describing how the state is updated based on the selected operation and the current state; R is a set of reward values ​​obtained by the agent for performing each action; O is a set of observation values, which is the global environment information used for global path planning or sensor information for obstacle avoidance; D is a set of probability distributions; Each time it advances t steps, the agent receives observations o t ∈O, action value a t ∈A and the reward function R(s t , a t ) The reward r given t ; The system is based on the conversion model T(s t+1 |s t , a t ) Generate a new state s t+1 , and the agent updates the observation o t+1 ; In global path planning, the observation value o t Equal to the state value s t ; From the strategy π=P(a t |s t ) is sampled to obtain the action value a t ; Through the action state value function Q(s t , a t ) approximates the expected value of the cumulative reward; the optimal strategy π is used as a solution to make the cumulative future return Maximize, where γ is the discount factor; the Bellman equation is used to calculate the optimal Q value function:

3. A path planning method combining dual confrontation technology and recursive mechanism according to claim 2, characterized in that: In step 2, a recursive mechanism is used to extract useful environmental information from continuous observations, and the first convolutional fully connected layer is replaced by an LSTM recurrent layer.

4. A path planning method combining dual confrontation technology and recursive mechanism according to claim 3, characterized in that: In step 3, an adversarial network is introduced into the DRQN ​​network structure to divide the features extracted by the LSTM convolutional neural network into two paths, namely: state value and action advantage dependent on the state; The adversarial network consists of two streams representing value and advantage functions, while sharing a common convolutional feature learning module, and then combines the two streams through a special aggregation layer to produce an estimate of the state-action value function Q; the adversarial network automatically generates independent estimates of the state value function and the advantage function without any additional supervision; on the D2RQN network structure, a Q′ network is added, and both Q and Q′ networks are used at the same time, one Q network is used to update parameters to select actions, and the other fixed Q′ network is used as the target network to calculate the Q value; due to errors in the network, overestimated actions will be repeatedly selected, so the Q value is often overestimated, and the Q and Q′ networks are used at the same time to avoid overestimation of the Q value.

Images