UAV path planning method based on incremental developmental deep reinforcement learning

Abstract

In order to overcome the problems of long training time and slow convergence speed of deep reinforcement learning, this paper proposes a UAV path planning method based on incremental developmental deep reinforcement learning, introduces an incremental developmental knowledge base, and improves the deep deterministic policy gradient algorithm. First, a threat pattern knowledge base is established based on the idea of ​​fuzzy matching, and the dense dynamic obstacles encountered during the flight are incrementally stored as knowledge to avoid repeated training of similar obstacle environments. Secondly, an obstacle avoidance strategy knowledge base is constructed based on the planning of safe routes by the underlying DDPG algorithm, and the obstacle avoidance strategy is directly output according to the threat pattern to shorten the training time. Finally, a developmental threat-obstacle avoidance mapping relationship is established to achieve "online real-time obstacle avoidance and offline autonomous optimization", and continuously improve the obstacle avoidance performance of UAV.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicle path planning, and specifically relates to a unmanned aerial vehicle path planning method based on incremental developmental deep reinforcement learning. Background Art

[0002] With the further opening of the UAV (Unmanned Aerial Vehicle, referred to as: UAV) field, the dense dynamic obstacles in complex environments such as cities and mountains have greatly threatened the flight safety of UAVs. * , D * Heuristic algorithms such as the graph-based method such as the through-view method and the Voronoi diagram method can only cope with simple environments where obstacle information is known in advance. Due to the complex and changeable terrain of cities and mountains, and the difficulty in obtaining the specific parameters of obstacles, the application scope of traditional obstacle avoidance algorithms is limited.

[0003] Different from the above traditional path planning methods, the navigation method based on reinforcement learning draws on the learning method of biological acquired perception and development, and continuously optimizes the obstacle avoidance strategy through continuous interaction with the environment. It not only avoids the dependence on obstacle modeling and expert supervision, but also has strong generalization ability and robustness. In particular, deep reinforcement learning (DRL) has made rapid progress in recent years. DRL uses the powerful perception and fitting ability of deep learning to effectively solve the problem of "exponential explosion" in high-dimensional environments, thereby obtaining a better obstacle avoidance strategy, which provides a new idea for the path planning problem of UAV in dense dynamic obstacle environments. Sliver, Google DeepMind team, Dr. John Schulman of the University of Berkeley and OpenAI have successively proposed deep reinforcement learning algorithms such as DDPG (Deep Deterministic Policy Gradient, Deep Deterministic Policy Gradient) algorithm, Asynchronous Advantage ActorCritic (A3C) algorithm, Trust Region Policy Optimization (TRPO) algorithm and proximal policy optimization (PPO) algorithm.

[0004] Although these methods have obvious advantages in UAV path planning, they often need to explore a large number of random obstacle environment samples during the training phase, resulting in long offline training time and slow convergence. Therefore, it is necessary to introduce a new learning mechanism. Summary of the invention

[0005] In order to overcome the problems in the prior art, the present invention proposes a UAV path planning method based on incremental developmental deep reinforcement learning.

[0006] The technical solution of the present invention to solve the above technical problems is as follows:

[0007] A method for UAV path planning based on incremental developmental deep reinforcement learning, comprising the following steps:

[0008] Step 1. Build UAV motion model;

[0009] Step 2. Based on the DDPG network model, adopt the Actor-Critic architecture and use the DDPG neural network architecture to continuously output decision actions;

[0010] Step 3. Introduce an incremental developmental knowledge base, train and improve the DDPG network model, update the network parameters of the DDPG network model, and plan a collision-free flight path.

[0011] Furthermore, constructing the UAV motion model in step 1 includes the following steps:

[0012] During the mission, the UAV flight trajectory is adjusted by the heading angle, climb angle and flight speed, and the kinematic equation is:

[0013]

[0014] In the formula, are the flight velocity components of the UAV on the three coordinate axes, v is the flight velocity of the UAV, ψ is the heading angle of the UAV, and θ is the climb angle of the UAV;

[0015] The flight speed, heading angle, heading angular velocity, climb angle, and climb angular velocity of the UAV vary in the following ranges:

[0016]

[0017] In the formula, is the heading angular velocity, is the climbing angular velocity.

[0018] Furthermore, the DDPG neural network architecture in step 2 includes four networks: an online Actor network, a target Actor network, an online Critic network, and a target Critic network.

[0019] Furthermore, in the Actor network, the online Actor network is based on the state information s obtained from the environment. i Select action a i , and the online policy network parameters θ uUpdate; the target Actor network randomly samples batches of data from the experience pool for training, and updates the state information s i+1 Select next action a i+1 , and update the target policy network parameters θ in a soft update manner u′ ;

[0020] In the Critic network, the online Critic network is based on the state s i and action a i , calculate the current Q value to evaluate the quality of the action, and adjust the online Q network parameters θ by minimizing the loss function Q Update; the target Critic network is updated according to the next state s i+1 and the next step a i+1 Calculate the Q value of the next state and use θ Q Update the target Q network parameters θ Q' .

[0021] Furthermore, the incremental development knowledge base in step 3 includes a threat pattern knowledge base B M , obstacle avoidance strategy knowledge base A M And the threat-obstacle avoidance mapping relationship knowledge base BA M .

[0022] Furthermore, the three knowledge bases form the following mapping relationship:

[0023]

[0024] When the UAV detects a threat pattern, if there is a threat pattern knowledge base that matches the detected obstacle environment B i For similar threat patterns, the corresponding obstacle avoidance strategy is directly output according to the threat-obstacle avoidance mapping relationship, the corresponding maneuvers are performed to avoid the obstacle, and the optimal obstacle avoidance strategy is continuously searched offline;

[0025] For threat patterns that do not exist in the threat pattern knowledge base, that is, threat patterns that are trained for the first time, it is necessary to train the network parameters according to the DDPG algorithm, and add the learned threat patterns, obstacle avoidance strategies, and mapping relationships as new knowledge to the knowledge base.

[0026] Furthermore, threat pattern knowledge base B M The idea of ​​fuzzy matching is introduced to determine whether there is a similar threat pattern in the knowledge base based on the incremental judgment function; for threat pattern B not stored in the knowledge base i , it will be added to the knowledge base as new knowledge, and the incremental development of the threat pattern knowledge base will be achieved in the process of continuous accumulation of knowledge.

[0027] Furthermore, the step of judging whether there is a similar threat pattern in the knowledge base according to the incremental judgment function is as follows:

[0028] Assume that the UAV encounters an unexpected threat environment B, in which there are multiple dynamic and static obstacles, any single obstacle can be expressed by formula (4):

[0029] b=[d,ψ d ,θ d ,v,ψ v ,θ v ] T (4)

[0030] Where: d is the Euclidean distance of the obstacle relative to the UAV; ψ d is the relative distance heading angle; θ d is the relative distance climbing angle; v is the speed of the obstacle relative to the UAV; ψ v is the relative motion speed heading angle; θ v is the relative motion speed climbing angle;

[0031] The above parameters are used i ,i∈{1,2,…,6} means, that is,

[0032] b=[A1,A2,A3,A4,A5,A6] T (5)

[0033] In the formula, A1-A6 represent the six attributes of obstacle b respectively;

[0034] The fuzzy control algorithm is introduced to calculate the similarity between two obstacles; the six attributes A1-A6 of obstacles k and j are converted into fuzzy inputs of the fuzzy controller respectively, and then the membership function is used to obtain the membership of the output domain set. The area average method is selected for defuzzification, and finally N fuzzy sets are obtained; among which the input is the specific determined value of the attributes A1-A6, and the output is the threat urgency of the obstacle;

[0035] The comparison between the obstacle modes of the above two obstacles is converted into a comparison between fuzzy sets: if all the attributes of obstacle k and obstacle j are in the same fuzzy set, the two obstacles are considered to be similar, and the similarity value can be quantitatively evaluated by comparing the closeness function; the closeness function is constructed according to the lattice closeness algorithm as follows:

[0036]

[0037] Where: A ik , A ij Respectively represent obstacles k and j in attribute A iWhen the closeness of the two obstacles on the six attributes is greater than the set closeness threshold, Sim(A ik ,A ij )≥th i ,i∈{1,2,…,6}, then the two obstacles belong to the same obstacle mode.

[0038] Furthermore, for threat pattern B not stored in the knowledge base i , it is added to the knowledge base as new knowledge, and the incremental development of the threat pattern knowledge base is achieved in the process of continuous knowledge accumulation. The specific steps are as follows:

[0039] The threat mode is defined as the set of all obstacle modes within the UAV’s perception range in the current obstacle environment, that is,

[0040] B=[b1,b2,…,b m ] T (7)

[0041] Assume there is an obstacle environment B1=[b1,b2,…,b m ] T ,B2=[b1,b2,…,b n ] T , if B1 goes from b1 to b m Any obstacle mode b j The closeness of the corresponding six attributes can be found in B2 and is greater than the closeness threshold th i If the threat pattern B1 is completely replaced by B2, it is considered that threat pattern B1 can be completely replaced by B2 without retraining; otherwise, threat pattern B1 will be added to the threat pattern knowledge base as a newly learned threat pattern to continuously supplement the knowledge base;

[0042] The knowledge stored in the threat pattern knowledge base is the set of all threat patterns encountered during the DDPG algorithm training process:

[0043] B M =[B1,B2,…,B m ] (8).

[0044] Furthermore, the threat model B is trained in the DDPG neural network i In the process, the network continuously iterates and updates the network parameters, and finally obtains the threat mode B. i The optimal network parameters under the obstacle avoidance strategy knowledge base will correspond to the threat mode B i The optimal network parameters are stored as obstacle avoidance strategy knowledge:

[0045] A i ={θ u ,θ Q} (9)

[0046] Among them, θ u are the online actor network parameters, θ Q is the online critic network parameter

[0047] Compared with the prior art, the present invention has the following technical effects:

[0048] In order to overcome the problems of long training time and slow convergence speed of deep reinforcement learning, the present invention introduces an incremental developmental knowledge base for UAV path planning in dense dynamic obstacle environments, and improves the deep deterministic policy gradient algorithm. First, a threat pattern knowledge base is established based on the idea of ​​fuzzy matching, and the dense dynamic obstacles encountered during the flight are incrementally stored as knowledge to avoid repeated training of similar obstacle environments. Secondly, an obstacle avoidance strategy knowledge base is constructed based on the planning of safe routes by the underlying DDPG algorithm, and the obstacle avoidance strategy is directly output according to the threat pattern to shorten the training time. Finally, a developmental threat-obstacle avoidance mapping relationship is established to achieve "online real-time obstacle avoidance and offline autonomous optimization", and continuously improve the obstacle avoidance performance of UAVs. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a schematic diagram of the task environment of the present invention;

[0050] Figure 2 Schematic diagram of the DDPG algorithm training framework of the present invention;

[0051] Figure 3 is the input quantity d of the present invention q Membership function relationship diagram of ;

[0052] Figure 4 is a membership function relationship diagram of the output r1 of the present invention;

[0053] Figure 5 This is a structural diagram of the improved threat-obstacle avoidance mapping relationship knowledge base of the present invention;

[0054] Figure 6 It is a sample diagram of the obstacle environment of the present invention;

[0055] Figure 7 is a relationship diagram between the number of knowledge and the number of obstacle environment samples of the present invention;

[0056] Figure 8 This is a diagram of the obstacle avoidance effect of the UAV tested in a simulation scenario. DETAILED DESCRIPTION

[0057] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0058] In order to overcome the problems of long training time and slow convergence speed of deep reinforcement learning, this paper proposes a new UAV path planning algorithm based on incremental developmental deep reinforcement learning. On the basis of planning safe routes by the underlying DDPG algorithm, a threat pattern knowledge base, an obstacle avoidance strategy knowledge base and a threat-obstacle avoidance mapping relationship knowledge base are built to reduce repeated calculations of similar obstacle environments, thereby shortening offline training time. First, in the DDPG offline training stage, the idea of ​​fuzzy matching is introduced. The incremental judgment function is used to determine whether there is a similar obstacle environment in the threat pattern knowledge base. For existing obstacle scenes, it is directly skipped without retraining; secondly, the UAV directly outputs the corresponding obstacle avoidance strategy based on the similar threat pattern matched in the threat pattern knowledge base, without going through the underlying obstacle avoidance algorithm, so as to realize UAV real-time obstacle avoidance; finally, after each obstacle avoidance behavior, the UAV continues to search for a better obstacle avoidance strategy offline, establishes a developmental threat-obstacle avoidance mapping relationship, and realizes "online real-time obstacle avoidance and offline autonomous optimization". Through a large number of simulated obstacle avoidance training in dense dynamic obstacle environments, the knowledge base continues to develop, and ultimately the UAV can plan a collision-free flight path without going through the underlying, time-consuming DDPG algorithm, thereby shortening the training time and greatly improving the UAV's obstacle avoidance capabilities in dense dynamic obstacle environments.

[0059] The dense dynamic obstacle environment established by the present invention is as follows Figure 1 Assume that the flight mission environment Q is a three-dimensional area with a length of L, a width of W, and a height of H. There are dynamic and static obstacles of different threat levels in the environment, and the position, movement speed, and direction of the obstacles are unknown.

[0060] The path planning task of UAV in a dense dynamic obstacle environment can be divided into two subtasks:

[0061] The first is how to maneuver the UAV to avoid an unknown obstacle within the flight constraints when it suddenly detects an unknown obstacle during flight;

[0062] The second task is to use past obstacle avoidance experience to establish and continuously expand the knowledge base. When encountering a similar obstacle scene again, the UAV directly calls up the obstacle avoidance strategy without going through the underlying obstacle avoidance algorithm that takes too long, thus avoiding repeated calculations of similar obstacle scenes.

[0063] Step 1. Build a UAV motion model.

[0064] During the mission, the UAV flight trajectory can be adjusted by heading angle, climb angle and flight speed. The kinematic equation is:

[0065]

[0066] In the formula, are the flight velocity components of the UAV on the three coordinate axes, v is the flight velocity of the UAV, ψ is the heading angle of the UAV, and θ is the climb angle of the UAV.

[0067] Considering the flight performance limitations, the flight speed, heading angle, heading angular velocity, climb angle, and climb angular velocity of the UAV must all vary within a certain range:

[0068]

[0069] In the formula, is the heading angular velocity, is the climbing angular velocity.

[0070] Step 2. Based on the DDPG network model, adopt the Actor-Critic architecture and use the deep neural network to continuously output decision actions.

[0071] The DDPG network is based on the Deterministic Policy Gradient (DPG) algorithm, adopts the Actor-Critic architecture, and uses deep neural networks to continuously output decision actions. The algorithm training framework is as follows Figure 2 shown.

[0072] The DDPG neural network architecture consists of four networks: online Actor network, target Actor network, online Critic network and target Critic network.

[0073] In the Actor Network, the online Actor Network is based on the state information obtained from the environment. i Select action a i , and the online policy network parameters θ u Update; the target Actor network randomly samples batches of data from the experience pool for training, and updates the state information s i+1 Select next action a i+1 , and update the target policy network parameters θ in a soft update manner u′ .

[0074] In the Critic network, the online Critic network is based on the state s i and action a i , calculate the current Q value to evaluate the quality of the action, and adjust the online Q network parameters θ by minimizing the loss function Q Update; the target Critic network is updated according to the next state s i+1 and the next step a i+1 Calculate the Q value of the next state and use θ Q Update the target Q network parameters θ Q' .

[0075] Step 3. Introduce an incremental developmental knowledge base, train and improve the DDPG network model, and update the network parameters of the DDPG network model.

[0076] The DDPG algorithm obtains state information s from the obstacle environment and selects obstacle avoidance action a, and then updates the network parameters. In other words, after the DDPG network is trained, one obstacle environment corresponds to a set of optimal network parameters. In order to train a network suitable for most obstacle environments, the UAV needs to be trained in tens of thousands of random obstacle environments. However, random obstacle environments have a certain probability of repetition, which easily leads to repeated training of similar obstacle environments. Therefore, an incremental developmental knowledge base is introduced to improve the DDPG algorithm and shorten the offline training time.

[0077] The knowledge learned during the UAV obstacle avoidance flight is stored in the threat pattern knowledge base B M , obstacle avoidance strategy knowledge base A M And the threat-obstacle avoidance mapping relationship knowledge base BA M In , the three knowledge bases form the following mapping relationship:

[0078]

[0079] Among them, the threat pattern knowledge base stores the obstacle environment of this flight mission, including the relative distance and movement speed of obstacles. The obstacle avoidance strategy knowledge base stores the DDPG network parameters obtained after training for the obstacle environment. The threat-obstacle avoidance mapping relationship knowledge base is used to manage the mapping relationship between the two, and provide decision-making strategies for subsequent collision avoidance based on the incremental developmental knowledge base, ensuring that the UAV avoids potential threats in a timely, effective and safe manner.

[0080] Step 3-1. Build threat model knowledge base B M

[0081] Assume that the UAV encounters an unexpected threat environment B, in which there are multiple dynamic and static obstacles, any single obstacle can be expressed by formula (4):

[0082] b=[d,ψ d ,θ d ,v,ψ v ,θ v ] T (4)

[0083] Where: d is the Euclidean distance of the obstacle relative to the UAV; ψ d is the relative distance heading angle; θ d is the relative distance climbing angle; v is the speed of the obstacle relative to the UAV; ψ v is the relative motion speed heading angle; θ v is the relative motion speed climbing angle. For the convenience of description, the above parameters are expressed as Ai ,i∈{1,2,…,6} means, that is,

[0084] b=[A1,A2,A3,A4,A5,A6] T (5)

[0085] Where A1-A6 represent the six attributes of obstacle b respectively.

[0086] The core of the knowledge base is to design the knowledge base incremental judgment function. In order to compare whether a single obstacle k and a single obstacle j belong to the same obstacle mode, a fuzzy control algorithm is introduced to calculate the similarity between the two obstacles. The six attributes A1-A6 of obstacles k and j are converted into fuzzy inputs of the fuzzy controller respectively, and then the membership function is used to obtain the membership of the output domain set. The area average method is selected for defuzzification, and finally N fuzzy sets are obtained. Among them, the input is the specific determined value of the attribute A1-A6, and the output can be understood as the threat urgency of the obstacle. The smaller the distance between the UAV and the obstacle, the greater the relative movement speed, the smaller the relative heading angle and relative climb angle, the higher the urgency. The importance of attributes A1-A6 to the threat urgency of obstacles is: A4>A1>A5=A6>A2=A3.

[0087] Taking attribute A1 as an example, the input of the fuzzy controller is the relative distance d between the UAV and the obstacle. q =P u P oq (q=k orj), where P u is the three-dimensional position coordinate of UAV, P oq is the three-dimensional position coordinate of obstacle k or obstacle j, and the domain is [0,d sensor ], where d sensor is the perceived distance of the UAV. The domain of the fuzzy output value r1 is [-n1,n1], where the value of n1 depends on the importance of attribute A1 among the six attributes of the obstacle. q (q=k or j) can be fuzzified to get D={NS,PS,PM,PB}. Similarly, the output value r1 can be fuzzified to get R1={NS,PS,PM,PB}, where NS, PS, PM, and PB represent negative small, positive small, positive medium, and positive large, respectively. The fuzzy logic rules are designed as follows:

[0088] Rule_o1: IF D is NS THEN R1 is NS;

[0089] Rule_o2: IF D is PS THEN R1 is PS;

[0090] Rule_o3: IF D is PM THEN R1 is PM;

[0091] Rule_o4: IF D is PB THEN R1 is PB.

[0092] The input and output membership functions of the fuzzy system are as follows: Figure 3 , Figure 4 Given.

[0093] If the output r1 of obstacle k and obstacle j falls into the same fuzzy set according to the corresponding membership degree of fuzzy logic rules, then the two obstacles are considered to be similar in attribute A1.

[0094] According to the above principle, the comparison between the obstacle modes of the above two obstacles can be converted into the comparison between fuzzy sets: if all the attributes of obstacle k and obstacle j are in the same fuzzy set, the two obstacles are considered to be similar, and the similarity value can be quantitatively evaluated by comparing the closeness function. The closeness function is constructed according to the grid closeness algorithm as follows:

[0095]

[0096] Where: A ik , A ij Respectively represent obstacles k and j in attribute A i The larger the value of Sim(·), the greater the membership degree of obstacles k and j in attribute A. i When the similarity of the two obstacles in the six attributes is greater than the set similarity threshold, Sim(A ik ,A ij )≥th i ,i∈{1,2,…,6}, the two obstacles can be considered to belong to the same obstacle mode.

[0097] Since there is often more than one obstacle in an obstacle environment, the concept of threat mode is introduced. The threat mode is defined as the set of obstacle modes of all obstacles within the UAV’s perception range in the current obstacle environment, that is,

[0098] B=[b1,b2,…,b m ] T (7)

[0099] Assume there is an obstacle environment B1=[b1,b2,…,b m ] T ,B2=[b1,b2,…,b n ] T , if B1 goes from b1 to b m Any obstacle mode b j The closeness of the corresponding six attributes can be found in B2 and is greater than the closeness threshold thi If there is an obstacle pattern, it is considered that threat pattern B1 can be completely replaced by B2, and there is no need to train again, avoiding repeated training for similar obstacle environments; otherwise, threat pattern B1 will be added to the threat pattern knowledge base as a newly learned threat pattern, and the knowledge base will be continuously supplemented, thereby realizing the incremental development of the threat pattern knowledge base.

[0100] The knowledge stored in the easy-to-obtain threat pattern knowledge base is the set of all threat patterns encountered during the DDPG algorithm training process:

[0101] B M =[B1,B2,…,B m ] (8)

[0102] Step 3-2. Build obstacle avoidance strategy knowledge base A M

[0103] Based on the obstacle environment training by the underlying DDPG algorithm, a knowledge base of obstacle avoidance strategies for UAV is constructed. i In the process, the network continuously iterates and updates the network parameters, and finally obtains the threat mode B. i The optimal network parameters under the obstacle avoidance strategy knowledge base will correspond to the threat mode B i The optimal network parameters are stored as obstacle avoidance strategy knowledge:

[0104] A i ={θ u ,θ Q} (9)

[0105] Among them, θ u are the online actor network parameters, θ Q are the online critic network parameters.

[0106] The obstacle avoidance strategy knowledge base constructed by the present invention is not a simple combination of basic flight actions in the planned optimal route. This is because the threat patterns are similar but not immutable. The safe flight action selection sequence set planned for the current threat pattern is not completely applicable to similar obstacle environments. Storing network parameters is more portable and applicable, and occupies less memory space. When the UAV encounters a B i When similar threat patterns are encountered, the time-consuming neural network training process can be omitted, and the trained DDPG network parameters can be directly called for reasoning, thereby achieving the purpose of real-time planning of safe routes and shortening network training time.

[0107] During the UAV obstacle avoidance flight, the network parameters are continuously adjusted, and the finalized network parameters are added to the obstacle avoidance strategy knowledge base as the obstacle avoidance strategy, thereby realizing the incremental development of the obstacle avoidance strategy knowledge base. The knowledge stored in the easy-to-obtain obstacle avoidance strategy knowledge base is the set of all optimal obstacle avoidance strategies generated during the DDPG network training process:

[0108] A M ={A1,…,A m} (10)

[0109] Step 3-3. Construct threat-obstacle avoidance mapping relationship knowledge base BA M

[0110] The offline training process of the DDPG algorithm is as follows Figure 5 As shown in Figure 1, in the initial stage with a small data scale, the UAV perceives the obstacle environment in real time, iterates the training according to the underlying DDPG algorithm, and plans the obstacle avoidance path. At this time, the threat mode and the obstacle avoidance strategy are mapped one by one, that is, for the newly added threat mode B i After DDPG network training, there is only one matching obstacle avoidance strategy A i As the knowledge base grows, the threat patterns detected by UAVs i There is a certain probability of repetition with the threat patterns stored in the knowledge base. At this time, it will no longer be necessary to undergo iterative neural network training. The UAV directly retrieves the matching avoidance strategy to execute the corresponding avoidance behavior, thereby achieving the purpose of shortening the offline training time of the DDPG algorithm. At the same time, the threat-obstacle avoidance mapping relationship constructed by the present invention is a developmental learning mapping relationship, which can not only directly output the obstacle avoidance strategy to execute the obstacle avoidance action, but also continue to search for the optimal obstacle avoidance strategy through the DDPG algorithm offline, dynamically improve the threat-obstacle avoidance mapping matching structure, increase the calculation speed, and enhance the safety of obstacle avoidance.

[0111] Incremental development means that during the offline obstacle avoidance training of the DDPG algorithm, the UAV continuously avoids obstacles and learns new threat patterns and obstacle avoidance strategies, and adds the learned threat patterns to the threat pattern knowledge base, adds the trained network parameters as obstacle avoidance strategies to the obstacle avoidance strategy knowledge base, and adds the mapping relationship between the two to the threat-obstacle avoidance mapping relationship knowledge base, continuously expanding the scale of the knowledge base.

[0112] Specifically, the threat pattern knowledge base introduces the idea of ​​fuzzy matching, and determines whether there are similar threat patterns in the knowledge base based on the incremental judgment function. For threat patterns Bi that are not stored in the knowledge base, they are added to the knowledge base as new knowledge, and the incremental development of the threat pattern knowledge base is achieved in the process of continuous knowledge accumulation.

[0113] In the process of avoiding obstacles, the UAV continuously iterates and trains the network parameters according to the DDPG algorithm, and updates the online actor network parameters θ u , online critic network parameters θ Q As an obstacle avoidance strategy that adapts to the current threat mode, it is added to the obstacle avoidance strategy knowledge base to realize the update and development of the obstacle avoidance strategy knowledge base. The incremental development of the obstacle avoidance strategy knowledge base is the process of the obstacle avoidance strategy knowledge base continuously accumulating new network parameters.

[0114] The threat-obstacle avoidance mapping relationship knowledge base is mainly used to manage the mapping relationship between threat patterns and obstacle avoidance strategies. It adopts a developmental growth learning mechanism, which can not only directly output the corresponding obstacle avoidance strategy according to the threat pattern, but also dynamically improve the threat-obstacle avoidance mapping matching structure by searching for the optimal obstacle avoidance strategy offline.

[0115] In the subsequent obstacle avoidance process, when the UAV detects a threat pattern, if there is a threat pattern knowledge base that matches the detected obstacle environment B i For similar threat patterns, the corresponding obstacle avoidance strategy is directly output according to the threat-obstacle avoidance mapping relationship, the corresponding maneuver is performed to avoid the obstacle, and the optimal obstacle avoidance strategy is continued to be searched offline, avoiding repeated training of the same obstacle environment, thus shortening the training time of the DDPG algorithm. For threat patterns that do not exist in the threat pattern knowledge base, that is, threat patterns that are trained for the first time, it is necessary to train the network parameters according to the DDPG algorithm, and add the learned threat patterns, obstacle avoidance strategies, and mapping relationships as new knowledge to the knowledge base to achieve incremental development of the knowledge base.

[0116] In order to test the performance of the improved incremental developmental deep reinforcement learning algorithm of the present invention, a comparative experiment was conducted on the DDPG algorithm before and after the improvement in a dense dynamic obstacle environment to quantify the obstacle avoidance performance.

[0117] The simulation experiment environment is: Pyhton 3.7, 2.42GHz Intel Core i5 processor, and Win10 operating system. First, it is necessary to randomly generate multiple dense dynamic obstacle environments in the simulation environment and perform offline training of the DDPG algorithm to accumulate initial obstacle avoidance experience in the knowledge base. Assume that the proximity thresholds of obstacle attributes A1-A6 are shown in Table 1.

[0118] Table 1 Closeness threshold

[0119]

[0120] Some examples of obstacle environments are as follows: Figure 6 As shown in the figure, 9 different dense dynamic obstacle environments were selected for training to expand the threat pattern knowledge base.

[0121] After the training is completed, the relationship between the amount of knowledge in the three knowledge bases and the number of obstacle environment samples is as follows: Figure 7 As shown in Figure 2. With the introduction of the concept of fuzzy matching, the knowledge base summarizes and integrates the three types of knowledge according to the similarity of threat patterns, and reduces the dimension of duplicate data while continuously expanding the knowledge base, thereby effectively avoiding repeated training of similar obstacle environments. After the training is completed, the amount of knowledge in the knowledge base is less than the number of obstacle environment samples. Figure 6 (a) and Figure 6 The threat pattern similarity of (b) is 0.85. Figure 6 (d) and Figure 6 The similarity of (e) is 0.92, which is higher than the similarity threshold. Therefore, the above two groups of threat patterns can be merged separately to share the obstacle avoidance strategy and reduce the amount of calculation. Figure 6 (a) and Figure 6 The threat pattern similarity of (h) is only 0.13, which is lower than the similarity threshold. At this time, it is necessary to construct two sets of threat pattern information in the threat pattern knowledge base respectively.

[0122] After the development is completed, the obstacle avoidance performance of the UAV is tested in a simulation environment. The simulation environment is set as a three-dimensional area of ​​500m×500m×500m. There are 10 dynamic obstacles in the area that move at different speeds along different heading angles and climbing angles. The UAV takes off from the starting point (0,0,0) and finally reaches the target point (500,500,500).

[0123] In the simulation scenario with the same obstacles, the present invention selects indicators such as collision-free flight distance, collision-free flight time, and offline training duration to quantify the obstacle avoidance performance of the DDPG algorithm before and after improvement. Figure 8 The quantitative test results are shown in Table 2. Although the DDPG before and after the improvement can guide the UAV to avoid obstacles effectively, in the dense dynamic obstacle environment, because the knowledge base provides a fast and accurate obstacle avoidance strategy reference, the average flight speed of the UAV controlled by the improved DDPG algorithm is increased by 20.7%, and the network training time is shortened by 62 times, which greatly improves the network training efficiency.

[0124] Table 2 Quantitative obstacle avoidance performance comparison

[0125]

[0126] In order to solve the problem of long training time and slow convergence speed of DDPG algorithm, this paper proposes a UAV path planning algorithm based on incremental developmental deep reinforcement learning. Through simulation training in dense dynamic obstacle environment, the knowledge base is continuously enriched, similar threat patterns are merged and summarized, and repeated training of DDPG in similar obstacle environment is avoided. Simulation comparison experiments show that the improved DDPG algorithm of this invention can effectively shorten the training time and improve the convergence efficiency of the algorithm.

[0127] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for UAV path planning based on incremental developmental deep reinforcement learning, characterized in that: The following steps are involved: Step 1. Build UAV motion model; Step 2. Based on the DDPG network model, adopt the Actor-Critic architecture and use the DDPG neural network architecture to continuously output decision actions; Step 3. Introduced the incremental developmental knowledge base, trained and improved the DDPG network model, updated the network parameters of the DDPG network model, and planned a collision-free flight path; The incremental development knowledge base in step 3 includes the threat pattern knowledge base B M , obstacle avoidance strategy knowledge base A M And the threat-obstacle avoidance mapping relationship knowledge base BA M ; The three knowledge bases form the following mapping relationship: When the UAV detects a threat pattern, if there is a threat pattern knowledge base that matches the detected obstacle environment B i For similar threat patterns, the corresponding obstacle avoidance strategy is directly output according to the threat-obstacle avoidance mapping relationship, the corresponding maneuvers are performed to avoid the obstacle, and the optimal obstacle avoidance strategy is continuously searched offline; For threat patterns that do not exist in the threat pattern knowledge base, that is, threat patterns that are trained for the first time, it is necessary to train network parameters according to the DDPG algorithm, and add the learned threat patterns, obstacle avoidance strategies, and mapping relationships as new knowledge to the knowledge base; Threat Pattern Knowledge Base B M The idea of ​​fuzzy matching is introduced to determine whether there is a similar threat pattern in the knowledge base based on the incremental judgment function; for threat pattern B not stored in the knowledge base i , then add it into the knowledge base as new knowledge, and realize the incremental development of threat pattern knowledge base in the process of continuous knowledge accumulation; The specific steps of judging whether there is a similar threat pattern in the knowledge base according to the incremental judgment function are as follows: Assume that the UAV encounters an unexpected threat environment B, in which there are multiple dynamic and static obstacles, any single obstacle can be expressed by formula (4): b=[d,ψ d ,i d ,v,ψ v ,i v ] T (4) Where: d is the Euclidean distance of the obstacle relative to the UAV; ψ d is the relative distance heading angle; θ d is the relative distance climbing angle; v is the speed of the obstacle relative to the UAV; ψ v is the relative motion speed heading angle; θ v is the relative motion speed climbing angle; The above parameters are used i ,i∈{1,2,…,6} means, that is, <h2 style=";text-align:left;direction:ltr">b=[A1,A2,A3,A4,A5,A6]<h2 style=";text-align:left;direction:ltr"> T <h2 style=";text-align:left;direction:ltr"> (5) In the formula, A1-A6 represent the six attributes of obstacle b respectively; The fuzzy control algorithm is introduced to calculate the similarity between two obstacles; the six attributes A1-A6 of obstacles k and j are converted into fuzzy inputs of the fuzzy controller respectively, and then the membership function is used to obtain the membership of the output domain set. The area average method is selected for defuzzification, and finally N fuzzy sets are obtained; among which the input is the specific determined value of the attributes A1-A6, and the output is the threat urgency of the obstacle; The comparison between the obstacle modes of the above two obstacles is converted into a comparison between fuzzy sets: if all the attributes of obstacle k and obstacle j are in the same fuzzy set, the two obstacles are considered to be similar, and the similarity value can be quantitatively evaluated by comparing the closeness function; the closeness function is constructed according to the lattice closeness algorithm as follows: Where: A ik , A ij Respectively represent obstacles k and j in attribute A i When the closeness of the two obstacles on the six attributes is greater than the set closeness threshold, Sim(A ik ,A ij )≥th i ,i∈{1,2,…,6}, then the two obstacles belong to the same obstacle mode; For threat pattern B not stored in the knowledge base i , it is added to the knowledge base as new knowledge, and the incremental development of the threat pattern knowledge base is achieved in the process of continuous knowledge accumulation. The specific steps are as follows: The threat mode is defined as the set of all obstacle modes within the UAV’s perception range in the current obstacle environment, that is, B=[b1,b2,…,b m ] T (7) Assume there is an obstacle environment B1=[b1,b2,…,b m ] T ,B2=[b1,b2,…,b n ] T , if B1 goes from b1 to b m Any obstacle mode b j The closeness of the corresponding six attributes can be found in B2 and is greater than the closeness threshold th i If the threat pattern B1 is completely replaced by B2, it is considered that threat pattern B1 can be completely replaced by B2 without retraining; otherwise, threat pattern B1 will be added to the threat pattern knowledge base as a newly learned threat pattern to continuously supplement the knowledge base; The knowledge stored in the threat pattern knowledge base is the set of all threat patterns encountered during the DDPG algorithm training process: B M =[B1,B2,…,B m ] (8) Threat Model B in DDPG Neural Network Training i In the process, the network continuously iterates and updates the network parameters, and finally obtains the threat mode B. i The optimal network parameters under the obstacle avoidance strategy knowledge base will correspond to the threat mode B i The optimal network parameters are stored as obstacle avoidance strategy knowledge: A i ={θ u ,i Q } (9) Among them, θ u are the online actor network parameters, θ Q are the online critic network parameters.

2. The method for UAV path planning based on incremental developmental deep reinforcement learning according to claim 1, characterized in that: Constructing the UAV motion model in step 1 includes the following steps: During the mission, the UAV flight trajectory is adjusted by the heading angle, climb angle and flight speed, and the kinematic equation is: In the formula, are the flight velocity components of the UAV on the three coordinate axes, v is the flight velocity of the UAV, ψ is the heading angle of the UAV, and θ is the climb angle of the UAV; The flight speed, heading angle, heading angular velocity, climb angle, and climb angular velocity of the UAV vary in the following ranges: In the formula, is the heading angular velocity, is the climbing angular velocity.

3. The method for UAV path planning based on incremental developmental deep reinforcement learning according to claim 1, characterized in that: The DDPG neural network architecture in step 2 includes four networks: online Actor network, target Actor network, online Critic network and target Critic network.

4. A method for UAV path planning based on incremental developmental deep reinforcement learning according to claim 3, characterized in that: In the Actor Network, the online Actor Network is based on the state information obtained from the environment. i Select action a i , and the online policy network parameters θ u Update; the target Actor network randomly samples batches of data from the experience pool for training, and updates the state information s i+1 Select next action a i+1 , and update the target policy network parameters θ in a soft update manner u′ ; In the Critic network, the online Critic network is based on the state s i and action a i , calculate the current Q value to evaluate the quality of the action, and adjust the online Q network parameters θ by minimizing the loss function Q Make updates; The target Critic network is based on the next state s i+1 and the next step a i+1 Calculate the Q value of the next state and use θ Q Update the target Q network parameters θ Q′ .

Images