An unmanned aerial vehicle autonomous path planning algorithm based on state decomposition in complex environment

Abstract

The application discloses an unmanned aerial vehicle (UAV) autonomous path planning algorithm based on state decomposition in a complex environment. The application belongs to the field of UAV control and mainly solves the problem of real-time obstacle avoidance and autonomous path planning of the UAV in a large-scale complex and obstacle-dense dynamic environment. The algorithm models the UAV path planning problem as a Markov decision process and solves the problem by using deep reinforcement learning. By decomposing the state space of the UAV, two sub-networks are introduced to redesign the "actor-critic" architecture. The smooth L1 loss function is used to process abnormal training data. At the same time, a dynamic epsilon decay strategy is designed to balance the exploration-exploitation balance. The algorithm ensures that the UAV can autonomously complete path planning while avoiding obstacles. A large number of simulation experiments prove the effectiveness and robustness of the algorithm.

Description

Technical Field

[0001] This invention belongs to the field of unmanned aerial vehicle (UAV) control, and specifically relates to the design of an autonomous trajectory planning algorithm for UAVs based on deep reinforcement learning. Background Technology

[0002] Compared to manned aircraft, unmanned aerial vehicles (UAVs) are widely used in both military and civilian fields due to their small size, low cost, high flexibility, and strong survivability. In the military field, UAVs are widely used in target reconnaissance, terrain mapping, precision strikes, communications jamming, communications relay, and other battlefield situational awareness scenarios. In the civilian field, UAVs play an important role in disaster relief, aerial photography, news reporting, agricultural irrigation, power line inspection, and express delivery.

[0003] The widespread application of drones relies heavily on trajectory planning technology. The drone trajectory planning problem involves planning a globally optimal flight path for a drone, taking into account factors such as flight time, threat sources, flight area, and surrounding environment, to ensure the drone can successfully complete various tasks. Currently, domestic and international research on drone trajectory planning mainly categorizes it into non-learning traditional trajectory planning algorithms and artificial intelligence-based learning trajectory planning algorithms. Traditional trajectory planning algorithms primarily include the artificial potential field method, A... * Algorithms such as particle swarm optimization and genetic algorithms are used. These algorithms are highly dependent on environmental information and require the UAV to have strong perception capabilities. In mission scenarios where environmental information is insufficient and the UAV's situational awareness is interfered with, the UAV struggles to demonstrate good trajectory planning capabilities. Therefore, traditional trajectory planning algorithms are mainly suitable for simple mission scenarios with sparse obstacles.

[0004] In large-scale, complex, and dynamic environments, AI-based UAV trajectory planning algorithms have demonstrated strong environmental adaptability. Deep reinforcement learning, in particular, is widely used in UAV trajectory planning due to its high adaptability and strong generalization capabilities. The concept of reinforcement learning (RL) originates from the trial-and-error learning process in psychology and was an important early research direction in artificial intelligence. The basic process of reinforcement learning involves an intelligent agent interacting with the environment, receiving feedback (called reward), making sequential decisions, and maximizing the cumulative reward. Deep reinforcement learning combines the decision-making capabilities of reinforcement learning with the perceptual capabilities of deep learning, utilizing the powerful representational capabilities of neural networks to solve the perceptual decision-making problems of complex systems.

[0005] Autonomous UAV trajectory planning algorithms based on deep reinforcement learning mainly include Deep Q-Network (DQN), Double DQN (DDQN), and Deep Deterministic Policy Gradient (DDPG). DQN combines deep learning and Q-learning, using neural networks to fit Q-values, solving the representation problem of traditional Q-learning in high-dimensional state and action spaces. However, DQN suffers from overestimating Q-values, resulting in a certain deviation from the true Q-value. To address this, DDQN improves upon DQN's objective function, thus reducing the overestimation problem to some extent and making the estimated Q-value closer to the true value. However, regardless of whether it's DQN or DDQN, in UAV trajectory planning applications, the UAV's action space is defined as discrete, while in reality, the UAV's action space is usually continuous. Therefore, DQN and DDQN have certain limitations in practical applications.

[0006] The DDPG algorithm addresses the problem of discrete action space. DDPG primarily extends the discrete action space of DQN to a continuous action space based on the "actor-critic" framework. In this framework, the "actor" outputs a specific execution strategy using a neural network, while the "critic" evaluates the strategy by learning the action value function Q. The "actor" improves its strategy based on the "critic's" evaluation, and the "critic" continuously learns to improve the accuracy of its evaluation. Through this process, DDPG can ultimately learn an optimal strategy, thus planning an optimal path for the UAV. However, the original DDPG algorithm is particularly sensitive to the hyperparameter settings of the neural network; these settings directly affect the model's convergence speed. Furthermore, in large-scale, complex dynamic environments, the DDPG algorithm exhibits significant instability, posing a certain risk to its application in UAV trajectory planning tasks.

[0007] Considering the broad application prospects of UAVs and the various problems existing in current UAV trajectory planning, this invention proposes the ImDDPG algorithm based on deep reinforcement learning to address the UAV trajectory planning problem in large-scale complex dynamic environments. Compared with the original DDPG algorithm, the algorithm proposed in this invention can provide stronger trajectory planning capabilities for UAVs in large-scale complex environments. Furthermore, this algorithm also demonstrates strong environmental adaptability and generalization ability. Summary of the Invention

[0008] The purpose of this invention is to provide an efficient and highly generalizable trajectory planning algorithm for unmanned aerial vehicles (UAVs) to perform various tasks in large-scale, complex, and obstacle-dense dynamic environments. This algorithm ensures that the UAV avoids obstacles and successfully reaches its destination while enabling it to autonomously plan a globally optimal flight path. To achieve this objective, the steps employed in this invention are as follows:

[0009] Step 1: Based on the relevant theoretical knowledge of reinforcement learning, the UAV trajectory planning problem is modeled as a Markov decision process. A Markov decision process can usually be represented by a tuple: (S, A, P, γ, R), where S represents the state space, A represents the action space, P∈[0,1] represents the state transition probability of the agent from the current state s to a state s', γ∈[0,1] is the discount factor, and R represents the reward function;

[0010] Step 2: Establish the kinematic model of the UAV during flight. In the Cartesian coordinate system, the initial position of the UAV can be represented by (x0, y0, z0); for the convenience of solving the problem, this invention fixes the flight altitude of the UAV to a constant C. H Therefore, the position of the drone only changes in the xy plane; considering the continuously changing motion space of the drone, this invention updates the position of the drone by defining the drone's velocity v and heading angle θ.

[0011] Step 3: Based on the mission scenario of UAV autonomous trajectory planning and the inventor's domain knowledge of UAV flight, a non-sparse reward function is designed to accelerate the model's convergence speed. The reward function mainly consists of three parts: the first part is the reward r1 related to obstacle collision; the second part is the reward r2 related to the target; and the third part is the reward r3 related to training speed.

[0012] Step 4: To address the various problems existing in the original DDPG algorithm, an improved version of the DDPG algorithm (ImDDPG) is proposed. The proposed algorithm improves the "actor" neural network used in the original algorithm by dividing it into two "actor" sub-networks, each training independently on a portion of the input state information. The outputs of the two sub-networks are then concatenated with the global state information and input into the global "actor" network for training, resulting in the final execution strategy. To solve the "exploration-exploitation" problem in reinforcement learning, this invention incorporates a dynamic ε-decay strategy into the UAV's execution strategy to balance the relationship between "exploration" and "exploitation." Furthermore, to address the issue of outliers in the data collected by the neural network in large-scale, complex, and dynamic environments, this invention replaces the L2 loss function in the original paper with a smooth L1 loss function to stabilize the neural network training process.

[0013] The effectiveness of the UAV autonomous trajectory planning algorithm based on state decomposition in complex environments proposed in this invention has been verified through simulation experiments. Figure 4 Performance test results of the proposed algorithm and two other baseline algorithms, DDPG and TD3, under different obstacle densities are presented in the figure; (Appendix) Figure 5 The simulation environment area is 1800×1800m², as shown in the model trained by the proposed algorithm. 2 The actual flight trajectory of the UAV when the obstacle density is 0.6. (Attached) Figure 6 , attached Figure 7 and appendix Figure 8 It is a flight trajectory diagram of a drone in a dynamic environment. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the state space of the unmanned aerial vehicle (UAV) according to the present invention;

[0015] Figure 2 This is a schematic diagram of the neural network structure of the present invention;

[0016] Figure 3 This is a schematic diagram comparing the convergence speed of the algorithm of this invention with two other baseline algorithms;

[0017] Figure 4 This is a graph showing the performance test results of the algorithm of this invention under different obstacle densities;

[0018] Figure 5 The UAV of this invention operates in a simulated environment with an area of ​​1800×1800m. 2 The actual flight trajectory diagram when the obstacle density is 0.6;

[0019] Figure 6 This is a schematic diagram a of the UAV of the present invention autonomously avoiding obstacles in a dynamic environment;

[0020] Figure 7 This is a schematic diagram (b) illustrating the autonomous obstacle avoidance of the UAV in a dynamic environment according to the present invention;

[0021] Figure 8 This is a schematic diagram (c) of the UAV autonomously avoiding obstacles in a dynamic environment according to the present invention; Detailed Implementation

[0022] The present invention will now be described in further detail with reference to the accompanying drawings.

[0023] Step 1: Model the UAV autonomous trajectory planning problem as a Markov decision process.

[0024] 1. Definition of state space

[0025] The state space is the foundation for UAV decision-making and long-term cumulative reward evaluation, and its design directly affects the model's convergence speed and the performance of UAV trajectory planning. In this invention, the UAV's state space is divided into three parts: the UAV's internal state space, the obstacle-related state space, and the target-related state space. A schematic diagram of the UAV's state space is attached. Figure 1 As shown.

[0026] The state space inside the drone is defined as s i It is determined by the heading angle θ of the UAV at time t. t and the speed v of the drone t Composition. Please note that this invention does not incorporate the absolute position of the drone into the drone's internal state space as in existing research; therefore, doing so would weaken the drone's generalization ability in different environments.

[0027] The state space related to obstacles is defined as s o To enable the drone to avoid obstacles and other flying objects during flight, this invention uses a distance measuring instrument to detect environmental information around the drone and uses the data returned by the distance measuring instrument as the basis for determining whether a collision has occurred. o = [d1, d2, ..., d9].

[0028] The state space related to the target is defined as follows: in, This represents the distance between the drone and the target at time t. The distance data can be obtained from the drone's onboard GPS data. This represents the deviation between the UAV's heading and the target direction at time t. Therefore, the overall state space of the UAV can be represented as:

[0029]

[0030] 2. Definition of Action Space

[0031] Action space refers to the set of all actions that a drone can take. To achieve a continuous action space for the drone, this invention defines the drone's action space as A = [a...]. t , φ t ]. Among them, a t ∈[-1, 1] is used to control the speed change of the drone, φ t ∈[-1, 1] is used to control the heading of the UAV. In this invention, a t and φ t All of these are obtained through the output of a neural network.

[0032] 3. State transition probability

[0033] The state transition probability refers to the probability given the current state s. t And Action A t transition to state s t+1 The probability distribution, i.e.

[0034]

[0035] The state transition probability P here can also be understood as the system model. Reinforcement learning algorithms are often divided into model-free reinforcement learning algorithms and model-based reinforcement learning algorithms. The algorithm proposed in this invention belongs to the model-free reinforcement learning algorithm.

[0036] 4. Discount Factor

[0037] The discount factor γ diminishes the contribution of future rewards to the current state value, and its value is typically between [0, 1]. The more steps required to complete a task, the closer the discount factor γ is to 1. In this invention, the value of the discount factor γ is set to 0.99.

[0038] 5. Reward Function

[0039] The reward function R refers to the reward function given state s. t Take action A t Afterwards, the reward received by the drone is r = R(s) t A t Reinforcement learning algorithms are often highly sensitive to the reward function R, with different reward functions significantly impacting convergence and learning speed. The simplest approach is to set the reward to sparse, meaning the reward is only given after the drone completes the designated task. While simple, this approach results in low learning efficiency and makes convergence difficult. Therefore, in practical applications, reward reshaping techniques are often used to transform sparse rewards into dense rewards. This will be explained in detail in step 3.

[0040] Step 2: Establish the dynamic model of the UAV

[0041] Considering the high complexity of autonomous flight path planning for UAVs in real-world environments, this invention constructs a simulation environment that highly matches the real-world environment. In this simulation environment, the position of the UAV at time t is represented as (x... t y t , z t To facilitate problem-solving, this invention fixes the UAV's flight altitude at a constant value, namely z. t =C H At this point, the drone's motion is fixed in the xy plane, and its dynamic equation can be expressed as:

[0042]

[0043] Where, p t =[x t y t ] represents the position of the drone at time t, θ t The heading of the UAV at time t, φ t ∈[-1, 1] is used to control the drone's heading; v t The speed of the drone at time t, a t ∈[-1, 1] is used to control the speed variation of the drone.

[0044] Step 3: Reshape the reward function

[0045] The reward function acts as a signal, evaluating the merits of taking a certain action in a specific state. As mentioned earlier, a simple approach is to define the reward function as a sparse reward. However, using a sparse reward would cause the drone to lose its ability to avoid obstacles in real time when dense obstacles are scattered in the environment. Therefore, this invention transforms the sparse reward into a non-sparse reward through a reward reshaping technique. Reward reshaping provides the drone with a specific form of non-sparse reward while maintaining policy invariance. Specifically, the non-sparse reward in this invention consists of three parts, namely, a reward r1 related to obstacle collision:

[0046]

[0047] Where σ is a constant greater than 0, used to characterize the degree of penalty; in this invention, the value of σ is set to 50. min(·) represents finding the minimum value in the distance data returned by the distance measuring instrument, D r This is the actual measurement range of the distance measuring instrument, set to 100 meters. It can be seen that the obstacle penalty is modeled as a continuous function of distance rather than a discrete penalty value; when there are no obstacles within the measurement range of the drone's distance measuring instrument, i.e., d = D. r When r1 = 0, the closer the drone is to the obstacle within the range of the distance measuring instrument, the greater the penalty. This helps the drone better avoid various threats in complex environments and more effectively aids model convergence.

[0048] The second part is the target-related reward r2. This reward is mainly intended to encourage the drone to move towards the target. Any deviation from the target will be penalized. r2 is defined as follows:

[0049]

[0050] in, D0 represents the distance between the UAV and the target at time t. D0 is a constant greater than 0, used to prevent the UAV from deviating too far from the target. In this invention, the value of D0 is set to 1000 meters.

[0051] The third part is a fixed time step penalty, mainly to ensure the drone can fly to the target as quickly as possible to complete the designated task. r3 is defined as: r3 = -1. Therefore, the global reward function is defined as:

[0052] r = r1 + r2 + r3 (6)

[0053] Step 4: Neural Network Design and Algorithm Optimization

[0054] The algorithm proposed in this invention is mainly an improvement on the DDPG algorithm. The DDPG algorithm employs an "actor-critic" neural network structure, where the "critic" network is responsible for evaluating the quality of the behavior generated by the "actor" network. The "actor" can adjust based on the "critic's" evaluation value, thereby producing higher quality behavior. In reinforcement learning, the action-value function represents the expected cumulative reward obtained by taking action a at state s, transitioning to state s′, and executing policy π from state s′. Its mathematical definition is:

[0055]

[0056] However, for practical applications, the state space and motion space of drones are enormous. The motion value function cannot be accurately obtained; therefore, it needs to be parameterized. To make the parameterized motion-value function closer to the real motion-value function, it is necessary to utilize the time difference error. The update is performed using the following formula:

[0057]

[0058] Here, α represents the learning rate, which is used to control the magnitude of updates. This is called a time-difference objective. This is the time difference error. Once formula (8) converges, the optimal strategy can be obtained:

[0059]

[0060] In the "Critics" network, the network input is the state S and the sampled action A, and the network output is the action value. The action value function is learned using Q-learning. The loss function of the "Critics" network is the square of the time difference error, i.e.

[0061] L(s, a|θ) Q )=[r(s,a)+γQ′(s′,a′|θQ′ )-Q(s,a|θ Q )] 2 (10)

[0062] In the "actor" network, the action-value function is parameterized into a deterministic policy a = μ(s|θ) μ Using the chain rule of differentiation, the parameters of the "actor" network are updated according to the following formula:

[0063]

[0064] The algorithm proposed in this invention uses the technique of experience replay, that is, after the UAV completes each action, the algorithm will record the data (s) obtained from the interaction with the environment. t a t r t s t+1 The data is stored in the experience pool. When the experience pool reaches a certain amount, N data points are randomly selected from the experience pool to update the neural network parameters. For the "critic" network, this is done by minimizing the loss function, i.e.

[0065]

[0066] in,

[0067] y k =r k +γQ′(s k+1 ,μ′(s k+1 |θ μ′ )|θ Q′ (13)

[0068] Formula (12) is also known as L2 loss. In practical applications, to prevent gradient explosion, the learning rate of L2 loss must be adjusted very carefully, and outliers in the training data will also affect L2 loss. Therefore, this invention uses smooth L1 loss instead of L2 loss. The definition of smooth L1 loss is:

[0069]

[0070] Smooth L1 loss combines the advantages of L1 loss and L2 loss, making the network more robust to abnormal data.

[0071] To enable UAVs to successfully complete missions in large-scale, complex, and obstacle-dense environments, this invention redesigns the "actor" network in the original DDPG algorithm, as shown in the attached diagram. Figure 2 As shown.

[0072] First, this invention adds two "actor" sub-networks to the original "actor" network to generate two sub-actions a. g and a o For subnetwork 1, the input to the network is the state s of the UAV's state space with respect to the target portion. g After passing through two fully connected layers, sub-action a is obtained. g For subnetwork 2, the input to the network is the state s in the UAV's state space regarding the obstacle portion. o After passing through two fully connected layers, sub-action a is obtained. o Next, the present invention concatenates the outputs of the two sub-networks with the state space of the UAV, and uses this as the input to the global Actor network. After passing through two fully connected layers, the final output action A is obtained.

[0073] The motivation for using two sub-networks for pre-training lies in the fact that the drone's state space consists of three parts, each containing distinct information. Training with a single network would prevent the network from correctly identifying the weights of each part, potentially leading to a poor policy output. Conversely, by pre-decomposing the drone's state space and training each part separately, the drone can better extract useful information from each part, thereby outputting a more effective behavioral policy.

[0074] Since the DDPG algorithm outputs a deterministic behavioral policy, it achieves limited environmental exploration by adding an Ornstein-Uhlenbeck (OU) process. Therefore, the final action executed by the UAV can be represented as:

[0075] a=μ(s|θ μ )+N (15)

[0076] Where N□OU(μ, θ, σ). In actual training, it is usually necessary to perform a clip operation on the action values ​​to ensure that they are within a suitable range, i.e.

[0077] a = clip(a, a) low a high (16)

[0078] The choice of σ value during the OU process may cause the action value to remain at the boundary value a. low or a high This can reduce the stability and effectiveness of the algorithm. Therefore, this invention adds a dynamic ε-decay strategy to the OU process to balance the "exploitation-exploitation" relationship in reinforcement learning. Specifically:

[0079] a=μ(s|θ μ)+εN (17)

[0080] In the early stages of training, exploration noise accounts for a large proportion to achieve full exploration of the environment; as training progresses, ε will continuously decay, causing the proportion of noise to gradually decrease; in the "exploitation" stage, the action values ​​output by the "actor" network dominate, and the drone takes corresponding actions based on the strategies it learns from the environment.

[0081] In this way, drones will gradually learn to make autonomous decisions about what behavior to take next, thus achieving the goal of autonomous flight path planning for drones in large-scale, complex, and dense obstacle environments.

[0082] The contents not described in detail in this application are existing technologies known to those skilled in the art.

Claims

1. An unmanned aerial vehicle autonomous path planning algorithm based on state decomposition in a complex environment, the steps adopted are: Step 1: modeling the unmanned aerial vehicle path planning problem as a Markov decision process; a Markov decision process can usually be represented by a tuple: (S, A, P, gamma, R), wherein S represents the state space, A represents the action space, P represents the state transition probability of an intelligent agent from the current state s to a state s', gamma is a discount factor, and R represents a reward function; Step 2: Establish the kinematic model of the flight of the unmanned aerial vehicle. Considering the high complexity of autonomous path planning of the unmanned aerial vehicle in the real environment, the application builds a simulation environment highly matched with the real environment. In the simulation environment, the position of the unmanned aerial vehicle at time t is represented as (x t , y t , z t ). In order to facilitate the solution of the problem, the application fixes the flight height of the unmanned aerial vehicle at a constant value, i.e. z t =C H . At this time, the motion of the unmanned aerial vehicle is fixed in the x-y plane, and the dynamics equation can be represented as: wherein P t = [x t , y t ] represents the position of the UAV at time t, θ t is the heading of the UAV at time t, φ t ∈ [-1, 1] is used to control the heading of the UAV; v t is the speed of the UAV at time t, a t ∈ [-1, 1] is used to control the speed variation of the UAV. Step 3: remodeling the reward function. A non-sparse reward function is designed to accelerate the convergence speed of the model; the reward function mainly includes three parts: the first part is the reward related to the collision with obstacles r1; the second part is the reward related to the target r2; and the third part is the reward related to the training speed r3; Step 4: neural network design and algorithm optimization. An unmanned aerial vehicle autonomous path planning algorithm based on state decomposition is proposed; the algorithm improves the "actor" neural network in the original algorithm, that is, two "actor" sub-networks are divided to train the partial state information, then the outputs of the two sub-networks and the global state information are spliced and input into the global "actor" network for training to obtain the final execution strategy; the invention adds a dynamic epsilon decay strategy to the execution strategy of the unmanned aerial vehicle to balance the relationship between "exploration-exploitation"; at the same time, the invention designs the loss function as smoothL1, that is 2. The algorithm according to claim 1, wherein The specific method for modeling the unmanned aerial vehicle path planning problem as a Markov decision process is: (1) Definition of state space The state space is the basis for the unmanned aerial vehicle to make decisions and evaluate long-term cumulative rewards, and the design of the state space directly affects the convergence speed of the model and the performance of the unmanned aerial vehicle path planning; in the invention, the state space of the unmanned aerial vehicle is divided into three parts, namely the state space of the unmanned aerial vehicle itself, the state space related to obstacles and the state space related to the target; The state space inside the UAV is defined as s i ; it consists of the heading angle θ t of the UAV at time t and the velocity v t of the UAV; please note that the present invention does not add the absolute position of the UAV to the state space inside the UAV as the existing research does, so it weakens the generalization ability of the UAV in different environments; The state space related to the obstacle is defined as s o In order to enable the unmanned aerial vehicle to avoid the threat of obstacles and other flying objects in the flight process, the distance measuring instrument is used to detect the environmental information around the unmanned aerial vehicle, and the data returned by the distance measuring instrument is used as the basis for judging whether the unmanned aerial vehicle collides, so that s o =[d1, d2,..., d9]; The state space related to the target is defined as wherein, represents the distance between the UAV and the target at time t, which can be obtained by the on-board GPS data of the UAV; represents the deviation value of the heading of the UAV at time t from the direction of the target; thus, the state space of the UAV as a whole can be represented as: (2) Definition of action space The action space refers to a set of all actions that the unmanned aerial vehicle can take; in order to realize the continuous action space of the unmanned aerial vehicle, the action space of the unmanned aerial vehicle is defined as A=[a t , φ t ] in the application; wherein a t belongs to [-1, 1] and is used to control the speed change of the unmanned aerial vehicle, and φ t ∈[-1, 1] is used to control the heading of the unmanned aerial vehicle; in the application, a t and φ t are obtained through the output of a neural network; (3) State transition probability The state transition probability refers to the probability distribution of transitioning to state s t and action A t , given the current state s t+1 , i.e. The state transition probability P here can also be understood as a system model, and reinforcement learning algorithms can be divided into model-free reinforcement learning algorithms and model-based reinforcement learning algorithms; the algorithm proposed in the invention belongs to a model-free reinforcement learning algorithm; (4) Discount factor The discount factor gamma is the contribution of future rewards to the current state value, and its value is usually between 0 and 1; the more steps are needed to complete a task, the closer the value of the discount factor gamma is to 1; in the invention, the value of the discount factor gamma is set to 0.99; (5) Reward function The reward function R refers to the reward function given state s. t Take action A t Afterwards, the reward received by the drone is r = R(s) t A t Reinforcement learning algorithms are often very sensitive to the reward function R, and different reward functions have a great impact on convergence and learning speed. The simplest approach is to set the reward as a sparse reward, that is, the drone can only get the reward after completing the specified task. Although this approach is simple, it will make the learning efficiency of the algorithm low and the algorithm will be difficult to converge. Therefore, in practical applications, reward reshaping techniques are often used to transform sparse rewards into dense rewards.

3. The algorithm according to claim 1, wherein The specific method for remodeling the reward function is: The reward function acts as a signal to evaluate the good or bad of taking a certain behavior in a certain state; as mentioned before, a simple way is to define the reward function as sparse reward, but using sparse reward will lose the ability to avoid obstacles in real time when the UAV is used to scatter dense obstacles in the environment; for this purpose, the present application changes the sparse reward to the non-sparse reward through the reward reshaping technology; the reward reshaping provides a specific form of non-sparse reward for the UAV under the premise of strategy invariance; specifically, the non-sparse reward in the present application is divided into three parts, namely the reward r1 related to the collision with the obstacle: where σ is a constant greater than 0, used to characterize the degree of penalty, in the present application the value of σ is set to 50; min(·) represents the minimum value in the distance data returned by the distance measuring instrument, D r is the actual measurement range of the distance measuring instrument, the value of which is set to 100 meters; it can be seen that the obstacle penalty is modeled as a continuous function of distance rather than a discrete penalty value, when there is no obstacle within the measurement range of the distance measuring instrument of the UAV, i.e. d = D r , then r1 = 0; when an obstacle appears within the measurement range of the distance measuring instrument, the closer the UAV is to the obstacle, the greater the penalty it receives; The second part is the reward r2 related to the target; this part of the reward is mainly to encourage the UAV to move towards the target, and any deviation from the target will be punished; the definition of r2 is as follows: wherein, D0 is a constant greater than 0, used to prevent the UAV from deviating too far from the target; in the present application, the value of D0 is set to 1000 meters; The third part is the fixed time step penalty, mainly to enable the UAV to fly to the target as soon as possible to complete the specified task; r3 is defined as: r3=-1; then the global reward function is defined as: r＝r1+r2+r3 (6) The reward function is set according to formula (6).

4. The autonomous path planning algorithm for UAV based on state decomposition in complex environment according to claim 1, characterized in that The specific method for building and optimizing the neural network structure is: The algorithm of the present application is mainly improved based on the DDPG algorithm, the DDPG algorithm adopts the "actor-critic" neural network structure, wherein the "critic" network is responsible for judging the quality of the behavior generated by the "actor" network, and the "actor" can adjust according to the evaluation value of the "critic" to generate a higher quality behavior; in reinforcement learning, the action value function represents the expectation of the discounted cumulative return obtained by starting from state s' after moving to state s' by taking action a at state s, and executing the strategy pi, and its mathematical definition is: However, for practical applications, the state space and action space dimensions of the unmanned aerial vehicle are huge; the action value function cannot be accurately obtained, so the action value function needs to be parameterized, that is In order to make the parameterized action value function closer to the real action value function, the present application uses the time difference error to update The update formula is: wherein a represents a learning rate for controlling the magnitude of the update; called time-difference target, is the time-difference error; when formula (8) converges, the optimal strategy can be obtained: In the "critic" network, the input of the network is the state S and the sampled action A, and the output of the network is the action value; the method of Q learning is used to learn the action value function, and the loss function of the "critic" network is the square of the time difference error, that is L(s, a | 0 Q ) = [r(s, a) + γQ'(s', a' | 0 Q′ ) - Q(s, a | 0 Q )] 2 (10) In the "actor" network, the action value function is parameterized as a deterministic policy a = μ(s|θ μ ), and the "actor" network parameters are updated following the chain rule of differentiation: The algorithm uses the experience replay technique, that is, after each action of the unmanned aerial vehicle is performed, the algorithm stores the data (s t , a t , r t , s t+1 ) obtained by interacting with the environment into an experience pool; when the data in the experience pool reaches a certain amount, N data are randomly extracted from the experience pool to update the neural network parameters; for the "critic" network, the updating is performed by minimizing the loss function, that is, Formula (12) is also called L2 loss. In practical application, in order to prevent the problem of gradient explosion, it is necessary to carefully adjust the learning rate of L2 loss, and the outliers in the training data will also affect the L2 loss. For this purpose, the present application designs the loss function smooth L1 to replace the L2 loss. y k = r k + γQ'(s k+1 , μ'(s k+1 |θ μ′ )|θ Q′ ) (13) The motivation of using two sub-networks for pre-training lies in that the state space of the UAV is composed of three parts, and the information contained in each part is different; using a single network for training will make the network unable to correctly distinguish the weight of each part, and finally may lead to a relatively poor strategy output by the network; on the contrary, by decomposing the state space of the UAV in advance and training respectively, the UAV can better extract useful information from each part of the state space, so as to output a more effective behavior strategy; In order to enable the unmanned aerial vehicle to successfully complete the task in a large-scale complex and obstacle-dense environment, two "actor" sub-networks are added on the basis of the original "actor" network for generating two sub-actions a g and a o ; for the sub-network 1, the input of the network is the state s g of the target part in the unmanned aerial vehicle state space, after passing through two layers of full connection network, the sub-action a g is obtained; for the sub-network 2, the input of the network is the state s o of the obstacle part in the unmanned aerial vehicle state space, after passing through two layers of full connection network, the sub-action a o is obtained; then, the outputs of the two sub-networks and the state space of the unmanned aerial vehicle are spliced as the input of the global "actor" network, after passing through two layers of full connection network, the final output action A is obtained; Since the DDPG algorithm outputs a deterministic behavior strategy, the algorithm realizes a small range of exploration of the environment by adding the Ornstein-Uhlenbeck (OU) process; at this time, the action used for the final execution of the UAV can be represented as: ​ wherein, In actual training process, it is usually required to perform clip operation on action value to ensure that it is within a proper size range, i.e. a = clip(a, a low , a high ) (15) The selection of the value of sigma in the OU process can cause the action value to always stay at the boundary value a low or a high Therefore, the stability and effectiveness of the algorithm are reduced; for this purpose, the dynamic epsilon decay strategy is added in the OU process to balance the relationship between "exploration-exploitation" in reinforcement learning; the specific method is as follows: In the early stage of training, the exploration noise occupies a large proportion to achieve sufficient exploration of the environment; as the training proceeds, epsilon is continuously attenuated, so that the proportion of noise gradually decreases; in the "utilization" stage, the action value output by the "actor" network dominates, and the UAV takes corresponding behavior according to the strategy learned from the environment.

Images