UAV Autonomous Path Planning Method Based on Lightweight Continuous SAC Algorithm

Abstract

The present invention discloses a method for autonomous path planning of an unmanned aerial vehicle based on a lightweight continuous SAC algorithm. The steps are as follows: constructing a mathematical model for flight control of the unmanned aerial vehicle; designing a state space, an action space, and a reward function; building a deep reinforcement learning neural network model; generating an empirical data set; training the deep neural network using the SAC algorithm; and performing model distillation with the trained network as a teacher network. Based on deep reinforcement learning, with the SAC algorithm as the basic framework of the model, the present invention autonomously designs a reward function to improve the training efficiency, and uses model distillation to reduce the network scale, realizing a high-exploration and lightweight path planning method for unmanned aerial vehicles, and solving the problems that the unmanned aerial vehicle sometimes spins in place with poor smoothness, makes misjudgments in the case of more noise, has poor training efficiency and stability, and has a slow model response speed.

Description

Technical Field

[0001] The present invention belongs to the field of communication technologies, and further relates to a method for autonomous path planning of an unmanned aerial vehicle (UAV) based on a lightweight continuous Soft Actor-Critic (SAC) algorithm in the field of UAV technologies. The present invention can be applied to UAVs in different environments to achieve autonomous decision-making during movement, so as to meet the requirement of efficiently and safely reaching the target range with a relatively smooth trajectory in the unmanned state, and realize the autonomous path planning of the UAV. Background Art

[0002] Autonomous path planning of an unmanned aerial vehicle (UAV) is a technology that enables the UAV to autonomously navigate to the destination without human intervention. During the autonomous flight of the UAV, due to the relatively high flight speed, there are high requirements for the decision-making response speed and accuracy, and it is highly dependent on the flight environment. Traditional solutions usually include some methods based on technologies such as genetic algorithms, dynamic Bayesian networks, and approximate dynamic programming. Most of these methods have problems such as the need for complex modeling, low real-time decision-making efficiency, the need for a large amount of data sets for support, and a large amount of computation, which are prone to cause the curse of dimensionality and other problems. Due to the particularity of UAV autonomous path planning, these problems will result in slow decision-making of the high-speed flying UAV, unstable effects in different application scenarios, and too high model training costs, causing great difficulties for the practical application of UAV autonomous path planning. Some existing solutions based on deep reinforcement learning mostly use algorithms such as Deep Q-network (DQN), Deep Deterministic Policy Gradient (DDPG), and Twin Delayed Deep Deterministic policy gradient algorithm (TD3) as Markov decision models, and use multiple basic actions to construct a discrete action space. When calculating the target value, both DQN and DDPG use the same Q-network for action selection and evaluation, which will generate overestimated value estimates in the presence of noise and errors, usually referred to as the overestimation problem. This has a great impact on the flight decision-making of the UAV. Although the TD3 algorithm improves the overestimation problems of the two, in the task scenario of UAV real-time decision-making, the training efficiency and stability are still poor. In addition, using a discrete action space also makes the autonomous flight path of the UAV not smooth enough and is prone to situations of spinning in place.

[0003] A method of autonomous path planning is adopted in the autonomous decision-making of UAV air combat based on deep reinforcement learning, which was disclosed by Jinwen Hu et al. in their published paper "Autonomous Maneuver Decision Making of Dual-UAV Cooperative Air Combat Based on Deep Reinforcement Learning" (Hu.J, Wang.L, Hu.T, Guo.C, Wang.Y. Autonomous Maneuver Decision Making of Dual-UAV Cooperative Air Combat Based on Deep Reinforcement Learning. Electronics 2022, 11, 467.). This method designs fifteen typical action instructions for UAVs, models the aircraft actions using a discrete action space, and uses the DDPG algorithm as a Markov decision model to generate the path planning strategy for UAVs. The deficiencies of this method are as follows. First, due to its characteristics, the DDPG algorithm has an overestimation problem, which will cause misjudgment in the case of more noise, and the training efficiency and stability are poor. Second, a discrete action space is used. Since the discrete actions are not flexible enough during the control process, the UAV can only fly in multiple fixed postures, resulting in an uneven flight path of the UAV, which in turn causes the UAV to sometimes spin in place.

[0004] A method of autonomous path planning is adopted in the autonomous decision-making of UAV combat based on the deep reinforcement learning TD3 algorithm, which was disclosed by the National University of Defense Technology of the Chinese People's Liberation Army in its patent document "Method for Autonomous Decision-making of UAV Combat Based on Deep Reinforcement Learning TD3 Algorithm" (Application No.: 202210264539.2, Application Date: March 17, 2022, Publication No.: CN114706418 A). The specific steps of this method are as follows. First, establish a UAV motion model. Second, based on the UAV motion model, establish a UAV air combat model based on the Markov decision process, which is represented by a quadruple including a state space, an action space, a reward function, and a discount factor, where the UAV motion model represents the state transition function in the UAV air combat model. Third, based on the UAV air combat model, train the UAV to learn the maneuver strategy based on the TD3 algorithm. The deficiencies of this method are as follows. The algorithm training process is not stable enough, and the training efficiency is poor. Since the TD3 algorithm adopts a more complex network model structure, the slow model inference speed leads to a slower response speed during the UAV decision-making process. Summary of the Invention

[0005] The object of the present invention is to propose an autonomous path planning method for unmanned aerial vehicles (UAVs) based on a lightweight continuous SAC algorithm in view of the deficiencies of the above-mentioned prior art, so as to solve the problems existing in the prior art, such as the UAV sometimes spinning in place with poor smoothness, misjudgment in the case of more noise, poor training efficiency and stability, and slow model response speed.

[0006] The specific idea for achieving the object of the present invention is as follows: when dealing with the real-time path planning problem of UAVs, the present invention models the motion mode of UAVs with a three-degree-of-freedom flight model, and the model defines how the UAV will displace in three-dimensional space in real time under different actions. Then, the environment of the UAV is modeled, including: the state space, action space and reward function of the UAV. The state space consists of the position information, speed, end position coordinate information, safety distance, etc. of the UAV in three-dimensional coordinates; an action space based on continuous actions is established to drive the flight of the UAV, and the action space consists of three elements: tangential overload, normal overload and roll angle, making the control of the UAV more flexible and overcoming the problem of poor smoothness of sometimes spinning in place in the prior art; a reward function mainly composed of three factors: distance, angle and height is designed, taking the distance as the main line reward, and the angle and height as auxiliary rewards, and a potential energy-based reward mechanism is adopted, making the rewards of the UAV denser during flight and being able to guide the convergence of the UAV policy network more efficiently. A deep reinforcement learning neural network based on the SAC algorithm is constructed. Compared with the DQN and DDPG algorithms, the SAC algorithm uses two Q networks and takes the smaller value of them, overcoming the problem of misjudgment in the case of more noise existing in the prior art. Compared with the TD3 algorithm, the maximum entropy is introduced into the loss function of the SAC algorithm, greatly increasing the exploration ability of the model and effectively improving the model training efficiency. On the basis of the SAC algorithm, a prioritized experience replay mechanism is also added, and weights are assigned to each piece of data added to the experience pool to improve the training efficiency. The present invention overcomes the problems of poor training efficiency and stability in the prior art by designing a potential energy-based reward function, constructing a SAC deep reinforcement learning neural network and introducing a prioritized experience replay mechanism. Finally, the policy network performs model distillation operation to train a smaller student network to lightweight the model, overcoming the problem of slow model response speed existing in the prior art.

[0007] To achieve the above object, the technical solution adopted by the present invention includes the following steps:

[0008] Step 1, construct a three-degree-of-freedom flight model for the motion mode of the UAV;

[0009] Step 2, autonomously design a state space set, a continuous action space set and a reward function based on the UAV flight control mathematical model and the Markov decision process;

[0010] Step 3, construct a deep reinforcement learning neural network based on the SAC algorithm:

[0011] Step 3.1, build a policy sub-network, the structure of which is composed of six fully connected layers connected in series in sequence. Set the number of nodes in the six fully connected layers to 14, 512, 512, 512, 512, and 3 respectively;

[0012] Step 3.2, build two action-value evaluation sub-networks with the same structure. Each sub-network is composed of six fully connected layers connected in series in sequence. Set the number of nodes in the six fully connected layers to 17, 512, 512, 512, 512, and 1 respectively;

[0013] Step 3.3, build a state-value evaluation sub-network and a state-value evaluation target sub-network with the same structure. Each sub-network is composed of six fully connected layers connected in series in sequence. Set the number of nodes in the six fully connected layers to 14, 512, 512, 512, 512, and 1 respectively;

[0014] Step 3.4, connect the two action-value evaluation sub-networks in parallel and then connect them to the policy sub-network and the state-value sub-network respectively to form a deep reinforcement learning neural network;

[0015] Step 4, generate an empirical data set:

[0016] Form the quadruple empirical data corresponding to the action by the state space set, action space set, reward information obtained, and state space set after the action of the drone each time, store it in the experience pool, and form an empirical data set from the quadruple empirical data of at least 10,000 actions stored in the experience pool;

[0017] Step 5, train the deep neural network using the SAC algorithm:

[0018] Step 5.1, randomly initialize a state space set and input it into the policy sub-network. The policy sub-network outputs an action space set, making the drone act according to the action space set given by the policy sub-network, and store the quadruple composed of the state set of the drone before the action, the action space set given by the policy sub-network, the reward value generated by the reward function, and the state set of the drone after the action in the experience pool. Assign the highest weight to the data currently stored in the experience pool, and decay the weights of the remaining data in the order of storage;

[0019] Step 5.2, extract a subset of 128 experience data from the updated experience pool according to the prioritized experience replay and input it into the deep neural network, outputting two action values, a state value 1, a state value 2, and a reward value. Substitute the two current output action values and state value 1 into the loss function L1; substitute the smaller value of the two current output action values and state value 2 into the loss function L2; substitute the smaller value of the two current output action values into the loss function L3. Use the gradient backpropagation method to update the weight parameters of the action value evaluation sub-network, state value evaluation sub-network, and policy network respectively, and use the exponential decay averaging method to update the weight parameters of the state value evaluation target sub-network, obtaining the updated weight parameters of the deep neural network;

[0020] Step 5.3, determine whether the currently output reward value converges. If so, after obtaining the trained deep neural network, execute Step 6; otherwise, execute Step 5.1;

[0021] Step 6, perform model distillation on the policy sub-network:

[0022] Step 6.1, randomly extract the action space set from 10,000 experience data in the experience pool as the training set for the policy student network;

[0023] Step 6.2, input the training set for the policy student network into the policy network and the policy student network respectively in batches of 32 data each time. Substitute the outputs of the two networks into the cross-entropy loss function to calculate the loss value, and use the gradient backpropagation method to update the weight parameters of the policy student network until the loss value converges, obtaining the distilled policy sub-network;

[0024] Step 7, plan the flight path of the drone:

[0025] Step 7.1, input the state space set of the drone at the current moment of the path to be planned into the distilled policy sub-network, output the action space set at the current moment. The drone acts according to the action space set, generating the state space set at the next moment and the movement path of the drone at the current moment. Splice the movement path at the current moment to the movement path generated at the previous moment according to the position information;

[0026] Step 7.2, determine whether the drone has reached the target location. If so, execute Step 8; otherwise, execute Step 7.3.

[0027] Step 7.3, determine whether the number of drone actions has reached the preset upper limit. If so, prompt that the drone path planning fails; otherwise, repeat Step 7.1;

[0028] Step 8, take the path spliced with whether the drone has reached the target location as the planned action path of the drone.

[0029] Compared with the prior art, the present invention has the following advantages:

[0030] First, since the present invention adopts the most advanced SAC algorithm in the field of deep reinforcement learning, adds a prioritized experience replay mechanism on this basis, and independently designs a potential-based reward function, it effectively avoids the deficiencies of overestimation and misjudgment in the case of more noise, as well as poor training efficiency and stability. This makes the present invention have the characteristics of strong robustness, stable training, and fast convergence speed, which helps the unmanned aerial vehicle (UAV) to make more accurate decisions in different environments, with lower training costs and higher efficiency.

[0031] Second, since the present invention uses continuous actions to control the flight of the UAV when designing the action space of reinforcement learning, it overcomes the defect of poor smoothness and sometimes even spinning in place in the prior art, making the present invention have the characteristics of strong control flexibility and smooth flight path of the UAV.

[0032] Third, since the present invention performs model distillation on the model after the model training is completed, realizes the lightweight work of the model, speeds up the inference speed of the model, and overcomes the deficiency of low model response speed in the prior art, making the present invention have the characteristics of strong real-time performance and fast response speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is the overall implementation flowchart of the present invention;

[0034] Figure 2 is the schematic diagram of the flight model of the UAV established by the present invention in a three-dimensional coordinate system;

[0035] Figure 3 is the schematic diagram of the autonomous navigation destination range in the design of the reward function of the present invention;

[0036] Figure 4 is the schematic diagram of the deep reinforcement learning network model constructed by the present invention based on the SAC algorithm;

[0037] Figure 5 is the change diagram of the reward function during the training process of the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0038] The following further describes the present invention in detail with reference to the accompanying drawings and specific embodiments.

[0039] Refer to Figure 1 to further describe the implementation steps of the embodiment of the present invention in detail.

[0040] Step 1, construct a three-degree-of-freedom flight model of the UAV motion mode:

[0041] According to the flight control principle of the drone, based on three quantities: tangential overload, normal overload, and roll angle, the track yaw angle, track pitch angle, and speed of the drone are controlled respectively, and the three-degree-of-freedom flight control mathematical model of the drone in three-dimensional space is constructed as follows:

[0042]

[0043] Referring to Figure 2 the schematic diagram of the flight model of the drone in the three-dimensional coordinate system shown, where g represents the gravitational acceleration of the drone; t represents the unit time during the flight of the drone; n x represents the tangential overload of the drone; n z represents the normal overload of the drone; μ represents the roll angle of the drone; v represents the speed of the drone, satisfying the constraint condition v ∈ [v min , v max , v min represents the minimum flight speed of the drone, v max represents the maximum flight speed of the drone; γ represents the track pitch angle of the drone, that is, the angle between the speed direction and the horizontal plane, satisfying the constraint condition γ ∈ [-π / 2, π / 2]; ψ represents the track yaw angle of the drone, that is, the angle between the projection of the speed direction on the horizontal plane and the y-axis, satisfying the constraint condition ψ ∈ [-π, π]; x, y, and z respectively represent the coordinate values of the drone in the three-dimensional space coordinate system.

[0044] Step 2, based on the drone flight control mathematical model and the Markov decision process, independently design the state space set, continuous action space set, and reward function:

[0045] The independently designed state space set includes the coordinate values when the drone is at a certain position in three-dimensional space, the speed v of the drone, the track pitch angle γ of the drone, the track yaw angle ψ of the drone, the radius d of the safe approach range of the target position in , the radius d of the determination range for successfully approaching the target position out , and the coordinate values of the target point position in three-dimensional space.

[0046] The continuous action space set includes the tangential overload n of the drone x ; the normal overload n of the drone z ; the roll angle μ of the drone.

[0047] The reward function is: R total = r res + ω d r d + ω a r a + ω h r h , where rres , r d , r a , r h respectively represent the result reward function, distance reward function, angle reward function, and altitude reward function, ω d , ω a , ω h respectively represent the coefficients of the distance reward function, angle reward function, and altitude reward function;

[0048] The result reward function r res is as follows:

[0049]

[0050] Referring to Figure 3 the schematic diagram of the autonomous navigation destination range shown, where d in represents the safe distance between the UAV and the target location, d out represents the boundary of the range where the UAV successfully approaches the target location, and D represents the Euclidean distance between the UAV and the target location in three-dimensional space;

[0051] The distance reward function r d is as follows:

[0052]

[0053] where D max represents the flight boundary of the UAV relative to the target position in the three-dimensional coordinate system, and e (·) represents the exponential function with the natural constant e as the base;

[0054] The angle reward function r a is as follows:

[0055]

[0056] where θ represents the angular difference between the yaw angle ψ of the UAV's flight path and the straight-line direction between the UAV and the target point;

[0057] The altitude reward function r h is as follows:

[0058]

[0059] where H represents the flight altitude of the UAV, and h low represents the minimum safe altitude of the UAV's flight, and h high represents the maximum safe altitude of the UAV.

[0060] Step 3, referring to Figure 4Schematic diagram of the shown model structure, constructing a deep reinforcement learning neural network based on the SAC algorithm:

[0061] Step 3.1, build a policy sub-network, the structure of which is composed of six fully connected layers connected in series in sequence. Set the number of nodes of the six fully connected layers to 14, 512, 512, 512, 512, and 3 respectively;

[0062] Step 3.2, build two action-value evaluation sub-networks with the same structure. Each sub-network is composed of six fully connected layers connected in series in sequence. Set the number of nodes of the six fully connected layers to 17, 512, 512, 512, 512, and 1 respectively;

[0063] Step 3.3, build a state-value evaluation sub-network and a state-value evaluation target sub-network with the same structure. Each sub-network is composed of six fully connected layers connected in series in sequence. Set the number of nodes of the six fully connected layers to 14, 512, 512, 512, 512, and 1 respectively;

[0064] Step 3.4, connect the two action-value evaluation sub-networks in parallel and then connect them to the policy sub-network and the state-value sub-network respectively to form a deep reinforcement learning neural network;

[0065] In a specific embodiment, a policy network π, two action-value evaluation networks Q1 and Q2, and two state-value evaluation networks V1 and V2 are built;

[0066] Step 4, generate an experience data set:

[0067] Form the quadruple experience data corresponding to the action by the state space set, action space set, obtained reward information, and state space set after executing the action of the drone each time, store it in the experience pool, and form an experience data set with the quadruple experience data of at least 10,000 actions stored in the experience pool;

[0068] Step 5, train the deep neural network using the SAC algorithm:

[0069] Step 5.1, randomly initialize a state space set and input it into the policy sub-network. The policy sub-network outputs an action space set, makes the drone act according to the action space set given by the policy sub-network, and stores the quadruple composed of the state set of the drone before the action, the action space set given by the policy sub-network, the reward value generated by the reward function, and the state set of the drone after the action in the experience pool. Assign the highest weight to the data currently stored in the experience pool, and decay the weights of the remaining data in the order of storage;

[0070] In the embodiment of the present invention, the state s is randomly initialized t , when the drone transitions from the current state s t to the next state st+1 At this time, combine it with the action a performed at the current moment t and the reward r obtained at the current moment t to form a quadruple (s t , a t , r t , s t+1 ). Input it into the experience pool P. P will assign a weight ω to each input quadruple;

[0071] Step 5.2: Extract a subset of 128 experience data from the updated experience pool according to the prioritized experience replay and input it into the deep neural network, outputting two action values, a state value 1, a state value 2, and a reward value. Substitute the two current output action values and the state value 1 into the loss function L1; substitute the smaller value of the two current output action values and the state value 2 into the loss function L2; substitute the smaller value of the two current output action values into the loss function L3. Use the gradient backpropagation method to update the weight parameters of the action value evaluation sub-network, the state value evaluation sub-network, and the policy network respectively, and use the exponential decay averaging method to update the weight parameters of the state value evaluation target sub-network to obtain the updated weight parameters of the deep neural network;

[0072] In the embodiment of the present invention, 128 experience data subsets are extracted according to the data probability in the experience pool P according to the size of ω. The input of the policy network π is the state information s of the UAV at the current moment t , and the output is the action a of the UAV generated by the decision t . The inputs of Q1 and Q2 are the state information s at the current moment t and the action a taken by the UAV t , and the outputs are the scores q1 and q2 for the currently taken action. The inputs of the V1 and V2 networks are the state information s at the current moment t , and the outputs are the scores v1 and v2 for the current state information; Update Q1 and Q2 using q1, q2, v1 and the loss function L1, update V1 using min(q1, q2), v2 and the loss function L2, update π using min(q1, q2) and the loss function L3, and finally update V2 using the exponential decay averaging method.

[0073] The loss function L1 is as follows:

[0074]

[0075] where E(·) represents the expectation function, Q θ (·) represents the output value of the action value evaluation network, θ represents the parameters of the action value evaluation network, s t represents the state of the UAV before performing this action, a tDenote the action executed at the current moment, r(·) denote the reward value output by the reward function, ξ denote the discount factor, Denote the output value of the state value evaluation target network, Denote the parameters of the state value evaluation target network, s t+1 Denote the state of the UAV after executing this action.

[0076] The loss function L2 is as follows:

[0077]

[0078] Among them, V ψ (·) denote the output value of the state value evaluation network, ψ denote the parameters of the state value evaluation network, π φ (·) denote the output value of the policy network, φ denote the parameters of the policy network.

[0079] The loss function L3 is as follows:

[0080]

[0081] Among them, the Is a random action obtained by sampling in the Gaussian distribution.

[0082] The exponential decay averaging method is as follows:

[0083]

[0084] Among them, ψ denote the parameters of the state value evaluation network, τ denote the probability hyperparameter.

[0085] Step 5.3, determine whether the currently output reward value converges. If so, after obtaining the trained deep neural network, execute Step 6; otherwise, execute Step 5.1;

[0086] In the embodiment of the present invention, the convergence situation of the reward function is as Figure 5 Shown in the reward function change diagram during the training process of the embodiment;

[0087] Step 6, perform model distillation on the policy sub-network:

[0088] Step 6.1, randomly extract the action space set from 10,000 pieces of experience data in the experience pool as the training set of the policy student network;

[0089] Step 6.2: Input the data in the policy student network training set in batches of 32 into the policy network and the policy student network respectively. Substitute the outputs of the two networks into the cross-entropy loss function to calculate the loss value. Use the gradient backpropagation method to update the weight parameters of the policy student network until the loss value converges, and obtain the distilled policy sub-network.

[0090] Step 7: Plan the flight path of the drone:

[0091] Step 7.1: Input the current moment state space set of the drone whose path is to be planned into the distilled policy sub-network, and output the current moment action space set. The drone acts according to the action space set to generate the next moment state space set and the movement path of the drone at the current moment. The drone path set represents the overall path obtained by connecting the paths generated at each moment in sequence according to the state space set at the current moment and the state space set at the next moment.

[0092] Step 7.2: Determine whether the drone has reached the target location. If so, execute Step 8; otherwise, execute Step 7.3.

[0093] Step 7.3: Determine whether the number of drone actions has reached the preset upper limit. If so, prompt that the drone path planning has failed; otherwise, add the movement path at the current moment to the overall planned path, and repeat Step 7.1.

[0094] Step 8: Use the path spliced with whether the drone has reached the target location as the planned drone action path.

Claims

1. An autonomous path planning method for unmanned aerial vehicles based on a lightweight continuous SAC algorithm, characterized in that Using a deep reinforcement learning neural network model with SAC algorithm and prioritized experience replay, autonomously design the state space, continuous action space, and potential-based reward function based on the Markov decision process, and perform model distillation on the trained model; the steps of this planning method are as follows: Step 1, construct a three-degree-of-freedom flight model of the UAV's motion mode; Step 2, autonomously design the state space set, continuous action space set, and reward function based on the UAV flight control mathematical model and the Markov decision process; Step 3, construct a deep reinforcement learning neural network based on the SAC algorithm: Step 3.1, build a policy sub-network, the structure of which is composed of six fully connected layers connected in series in turn, and set the nodes of the six fully connected layers to 14, 512, 512, 512, 512, 3 respectively; Step 3.2, build two action value evaluation sub-networks with the same structure, each sub-network is composed of six fully connected layers connected in series in turn, and set the nodes of the six fully connected layers to 17, 512, 512, 512, 512, 1 respectively; Step 3.3, build a state value evaluation sub-network and a state value evaluation target sub-network with the same structure, each sub-network is composed of six fully connected layers connected in series in turn, and set the nodes of the six fully connected layers to 14, 512, 512, 512, 512, 1 respectively; Step 3.4, connect the two action value evaluation sub-networks in parallel and then connect them to the policy sub-network and the state value sub-network respectively to form a deep reinforcement learning neural network; Step 4, generate an experience data set: Form the quadruple experience data corresponding to this action by the state space set, action space set, obtained reward information, and state space set after executing the action of the UAV each time, store it in the experience pool, and form the experience data set by the quadruple experience data of at least 10,000 actions stored in the experience pool; Step 5, use the SAC algorithm to train the deep neural network: Step 5.1, randomly initialize a state space set and input it into the policy sub-network. The policy sub-network outputs an action space set, makes the UAV act according to the action space set given by the policy sub-network, and stores the quadruple composed of the state set of the UAV before the action, the action space set given by the policy sub-network, the reward value generated by the reward function, and the state set of the UAV after the action in the experience pool, assign the highest weight to the data currently stored in the experience pool, and decay the weights of the remaining data according to the storage order; Step 5.2, extract a subset of 128 experience data from the updated experience pool according to prioritized experience replay and input it into the deep neural network, output two action values, state value 1, state value 2, and reward value, and substitute the two current output action values and state value 1 into the loss function L1; substitute the smaller value of the two current output action values and state value 2 into the loss function L2; substitute the smaller value of the two current output action values into the loss function L3, and use the gradient backpropagation method to update the weight parameters of the action value evaluation sub-network, state value evaluation sub-network, and policy network respectively, and use the exponential decay averaging method to update the weight parameters of the state value evaluation target sub-network to obtain the updated weight parameters of the deep neural network; Step 5.3, determine whether the currently output reward value converges. If so, after obtaining the trained deep neural network, execute Step 6; otherwise, execute Step 5.1; Step 6, perform model distillation on the policy sub-network: Step 6.1, randomly extract the action space set from 10,000 experience data in the experience pool as the training set for the policy student network; Step 6.2, input the training set of the policy student network into the policy network and the policy student network respectively in batches of 32 data each time, substitute the outputs of the two networks into the cross-entropy loss function to calculate the loss value, and use the gradient backpropagation method to update the weight parameters of the policy student network until the loss value converges to obtain the distilled policy sub-network; Step 7, plan the flight path of the drone: Step 7.1, input the current moment state space set of the drone whose path is to be planned into the distilled policy sub-network, output the current moment action space set, the drone acts according to the action space set, generate the next moment state space set and the movement path of the drone at the current moment, and splice the movement path at the current moment to the movement path generated at the previous moment according to the position information; Step 7.2, determine whether the drone has reached the target location. If so, execute Step 8; otherwise, execute Step 7.3; Step 7.3, determine whether the number of drone actions has reached the preset upper limit. If so, prompt that the drone path planning fails; otherwise, repeat Step 7.1; Step 8, use the path spliced with whether the drone has reached the target location as the planned action path of the drone.

2. The UAV autonomous path planning method based on the lightweight continuous SAC algorithm according to claim 1, characterized in that The construction of the drone flight control mathematical model described in Step 1 refers to, according to the flight control principle of the drone, based on three quantities of tangential overload, normal overload, and roll angle, respectively control the track yaw angle, track pitch angle, and speed of the drone, and construct the three-degree-of-freedom flight control mathematical model of the drone in three-dimensional space as follows: Among them, g represents the gravitational acceleration of the drone; t represents the unit time during the flight of the drone; n x represents the tangential overload of the drone; n z represents the normal overload of the drone; μ represents the roll angle of the drone; v represents the speed of the drone, satisfying the constraint condition v ∈ [v min , v max , v min represents the minimum flight speed of the drone, v max represents the maximum flight speed of the drone; γ represents the track pitch angle of the drone, that is, the angle between the speed direction and the horizontal plane, satisfying the constraint condition γ ∈ [-π / 2, π / 2]; ψ represents the track yaw angle of the drone, that is, the angle between the projection of the speed direction on the horizontal plane and the y-axis, satisfying the constraint condition ψ ∈ [-π, π]; x, y, and z respectively represent the coordinate values of the drone in the three-dimensional space coordinate system.

3. The UAV autonomous path planning method based on the lightweight continuous SAC algorithm according to claim 1, characterized in that, The self-designed state space set described in step 2 includes the coordinate values when the UAV is at a certain position in the three-dimensional space, the speed v of the UAV, the track pitch angle γ of the UAV, the track yaw angle ψ of the UAV, and the radius d of the safe approach range of the target position in , the radius d of the determination range for successfully approaching the target position out , and the coordinate values of the target point position in the three-dimensional space.

4. The method for autonomous path planning of an unmanned aerial vehicle based on the lightweight continuous SAC algorithm according to claim 1, wherein The set of continuous action spaces described in step 2 includes the tangential overload n of the UAV x , the normal overload n of the UAV z , and the roll angle μ of the UAV.

5. The drone autonomous path planning method based on the lightweight continuous SAC algorithm according to claim 1, wherein The reward function R described in Step 2 total = r res + ω d r d + ω a r a + ω h r h , where r res , r d , r a , r h represent the result reward function, distance reward function, angle reward function, and height reward function respectively, and ω d , ω a , ω h represent the coefficients of the distance reward function, angle reward function, and height reward function respectively; The result reward function r res is as follows: Among them, d in represents the safety distance between the UAV and the target location, d out represents the boundary of the range where the UAV successfully approaches the target location, and D represents the Euclidean distance between the UAV and the target location in three-dimensional space; The distance reward function r d is as follows: Among them, D max represents the flight boundary of the UAV relative to the target position in a three-dimensional coordinate system, and e() represents the exponential function with the natural constant e as the base; The angle reward function r a is as follows: Among them, θ represents the angle difference between the track yaw angle ψ of the drone and the straight line direction between the drone and the target point; The height reward function r h is as follows: Among them, H represents the flight altitude of the UAV, and h low represents the minimum safe altitude of the UAV's flight, and h high represents the maximum safe altitude of the UAV.

6. The UAV autonomous path planning method based on the lightweight continuous SAC algorithm according to claim 1, characterized in that The loss function L1 described in Step 5.2 is as follows: Among them, E(·) represents the expectation function, Q θ (·) represents the output value of the action-value evaluation network, θ represents the parameters of the action-value evaluation network, s t represents the state of the UAV before performing this action, a t represents the action performed at the current moment, r(·) represents the reward value output by the reward function, ξ represents the discount factor, represents the output value of the state-value evaluation target network, represents the parameters of the state-value evaluation target network, s t+1 represents the state of the UAV after performing this action.

7. The method for autonomous path planning of an unmanned aerial vehicle based on the lightweight continuous SAC algorithm according to claim 6, wherein The loss function L2 described in Step 5.2 is as follows: Among them, V ψ (·) represents the output value of the state value evaluation network, ψ represents the parameters of the state value evaluation network, and π φ (·) represents the output value of the policy network, and φ represents the parameters of the policy network.

8. The method for autonomous path planning of an unmanned aerial vehicle based on the lightweight continuous SAC algorithm according to claim 7, wherein The loss function L3 described in Step 5.2 is as follows: Among them, the is a random action obtained by sampling in a Gaussian distribution.

9. The method for autonomous path planning of an unmanned aerial vehicle based on the lightweight continuous SAC algorithm according to claim 7, wherein The exponential decay averaging method described in Step 5.2 is as follows: Among them, ψ represents the parameter of the state value evaluation network, and τ represents the probability hyperparameter.

10. The method for autonomous path planning of an unmanned aerial vehicle based on the lightweight continuous SAC algorithm according to claim 1, characterized in that The set of UAV paths described in step 7.3 represents the overall path obtained by sequentially connecting the paths generated at each moment according to the state space sets of the current moment and the next moment.

Images