DDPG unmanned aerial vehicle landing method based on expert experience for mobile platform

Abstract

The application discloses a mobile platform-based DDPG unmanned plane landing method based on expert experience, which comprises the following steps: task decomposition; building a landing scene in a simulation environment; initializing an environment state; initializing algorithm neural network and other hyperparameters; designing a landing function for training the designed deep reinforcement learning neural network; and training the designed deep reinforcement learning neural network based on the simulation environment with the unmanned plane landing scene, and guiding the unmanned plane with expert experience in the training process. The application can accelerate the convergence speed of the algorithm and greatly solve the problem of low sampling efficiency in the early stage of deep reinforcement learning.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of autonomous control of unmanned aerial vehicles, and relates to a DDPG unmanned aerial vehicle landing method based on expert experience for a mobile platform. BACKGROUND

[0002] With the continuous updating and development of technology, the application scenarios of rotary-wing unmanned aerial vehicles are becoming more and more extensive, and the coverage is also becoming more and more extensive. Small unmanned aerial vehicles are used for delivering food and taking photos, and large unmanned aerial vehicles are used in surveying and mapping engineering, security, rescue, and map surveying. However, in order to further improve the autonomy of unmanned aerial vehicles in completing tasks, it is necessary to consider many aspects, such as the takeoff, navigation, landing of unmanned aerial vehicles, and autonomous landing in relatively complex environments. However, in the accidents of unmanned aerial vehicles, the accidents caused by the takeoff and landing of unmanned aerial vehicles account for 60%, so it is very important to study the takeoff and landing of unmanned aerial vehicles. In addition, in the cooperation of multiple robots, such as unmanned ground vehicles (UGV), unmanned aerial vehicles are used to land and refuel, which also provides a higher level of autonomy required for multi-rotor unmanned aerial vehicles in long-term tasks.

[0003] In 2019, Shi Haobin et al. designed a rotary-wing unmanned aerial vehicle tracking control method based on reinforcement learning in view of the problems of the classic PID control and image-based visual servoing control in the process of tracking and controlling the rotary-wing unmanned aerial vehicle, such as poor control accuracy. The method uses image-based visual servoing for closed-loop control and uses the Sarsa learning algorithm to adjust the gain. The feature points extracted from the visual information transmitted by the camera of the unmanned aerial vehicle are verified with the feature points of the target image, and the distance of the unmanned aerial vehicle from the target is used as the reward for learning. The experimental results verify the feasibility of the method in the task scenario. However, due to the limitations of the Sarsa algorithm itself, the training speed and training effect are not good enough. A kind of unmanned aerial vehicle end-to-end control method based on deep reinforcement learning (CN111460650A) uses the images obtained by the unmanned aerial vehicle on-board camera after processing as input, and directly obtains the unmanned aerial vehicle control instruction after processing by a deep reinforcement learning neural network, so as to realize the autonomous landing of the unmanned aerial vehicle. However, due to the random exploration in the early stage of reinforcement learning, the training time is too long and the efficiency is low. SUMMARY

[0004] The technical problem this invention aims to solve is as follows: Addressing the existing technical issues, and based on the aforementioned background, this invention proposes a DDPG (Deep Deterministic Policy Gradient) UAV landing method for mobile platforms based on expert experience. The Deep Deterministic Policy Gradient (DDPC) algorithm not only utilizes the superior performance of the experience pool and dual neural network structure in deep Q-network algorithms, improving problems such as data explosion in traditional reinforcement learning, but also possesses the advantages of policy gradient algorithms, effectively processing continuous domain data and enabling rapid neural network convergence. Furthermore, to accelerate the training process, this invention changes its initial random exploration behavior by employing an expert algorithm for guidance. Under the guidance of early expert experience, the Actor policy network is continuously trained, enabling it to master expert actions. This significantly saves training time and cost, and improves the convergence time of the network and model. Therefore, this DDPG UAV landing method for mobile platforms based on expert experience is of great significance for the realization of UAV applications in related fields, accelerating reinforcement learning training speed, improving training efficiency, and possessing high application value.

[0005] The present invention is achieved by at least one of the following technical solutions.

[0006] A DDPG drone landing method based on expert experience for mobile platforms includes the following steps:

[0007] Step 1: Construct a Markov model for the landing of the drone mobile platform;

[0008] Step 2: Construct the neural network in the DDPG algorithm;

[0009] Step 3: Update the status of the drone and the target;

[0010] Step 4: Construct a drone landing scenario in a simulation simulator and train the drone to land on the mobile platform using a deterministic policy gradient method based on expert experience.

[0011] Furthermore, the Markov model is (S,A,O,R,γ), where S is the input state of the neural network, A is the output action of the UAV, O is the observation space of the UAV sensor, R is the reward function, and γ is the discount factor.

[0012] Further, in step 1-1, define the state space of the Markov model, with input state S:

[0013] Based on the drone status and mobile platform status, the input status is set as follows:

[0014] S = [S uav S platform ]

[0015] The drone status x uav 、y uav 、z uav represents the position of the UAV in the three-dimensional plane, is the velocity component of the UAV along the X, Y, Z axes;

[0016] Mobile platform state: S platform = [x platform , y platform , z platform ], x platform , y platform , z platform represents the position of the UAV in the three-dimensional plane;

[0017] Step 1-2, define the action space of the Markov model, i.e. the output action A of the UAV:

[0018] The output action A represents the feedback signal received by the UAV, i.e. the distance between the UAV and the moving target, and the action set collected for its own state; Set the output action as:

[0019]

[0020] where, is the velocity component of the UAV along the X, Y, Z axes, and the velocity of the UAV is constrained in combination with the actual application: where represent the minimum and maximum speed of the UAV, respectively;

[0021] Step 1-3, define the observation space of the Markov model, i.e. the observation space O of the sensor:

[0022] Use the radar sensor to determine and obtain the position of the UAV and the target; Set the observation space as:

[0023] O = [D]

[0024] where the relative distance D between the UAV and the target is:

[0025]

[0026] Step 1-4, define the reward function R:

[0027] Use the sensor to obtain the position information of the UAV and the mobile platform, and through distance reward and punishment, collision reward and punishment, and time reward and punishment of the UAV, the reward function R is obtained, which represents the feedback value obtained by the UAV when selecting a certain action in the current state: The reward function R is obtained by combining the distance reward and collision avoidance reward of the UAV:

[0028]

[0029] wherein λ4, λ5 represent the weight coefficients of the distance reward collision avoidance reward ;

[0030] Step 1-5, define the discount factor γ:

[0031] Set the discount factor 0 < γ < 1 for calculating the cumulative value of the return in the whole process; the greater the γ value, the more emphasis on long-term income.

[0032] Further, the distance reward function is:

[0033]

[0034] wherein is the distance reward function, λ1, λ2 are the weight values of the two rewards; D t represents the Euclidean distance between the unmanned aerial vehicle and the mobile platform at the current t time; L is a constant value set; if D t ≥ L, a negative punishment reward proportional to the distance D t is given; if D t < L, a positive reward inversely proportional to the distance D t is given on the basis of the normal number c1;

[0035] collision avoidance reward is:

[0036]

[0037] wherein D t is the Euclidean distance between the unmanned aerial vehicle and the mobile platform at the current t time, and the collision penalty reward is 0 when no collision occurs; if the unmanned aerial vehicle collides, a negative punishment reward proportional to the distance of the mobile platform is given on the basis of the negative punishment of the constant c2, and λ3 is the collision coefficient constant.

[0038] Further, step 2 includes the following steps:

[0039] Step 2-1, construct the policy network in the DDPG algorithm, namely the Actor policy network;

[0040] For the input state S, the output vector U of the policy network is represented as:

[0041] U = μ actor (s)

[0042] Step 2-2, construct the evaluation network in the DDPG algorithm, namely the Critic evaluation network;

[0043] The output of the evaluation network is a state-action value Q μ (s, u) is represented as:

[0044]

[0045] where k is a summation variable, E[·] represents a mathematical expectation; s t+k+1 , u t+k+1 respectively represent a state input vector and an action output vector at t+k+1; γ is a discount factor, r is a reward of the UAV, s represents a state of the UAV, s t is a state of the UAV at t, u represents an action policy of the UAV, u t represents an action policy of the UAV at t;

[0046] Step 2-3, constructing a target neural network, i.e., a target network;

[0047] The weights of the policy network μ actor and the evaluation network Q μ (s, u) are copied into the respective target networks, i.e., θ μ → θ μ′ , θ Q → θ Q′ , wherein θ μ , θ Q respectively represent parameters of the current policy network and the evaluation network, and θ μ′ , θ Q′ respectively represent parameters of the current target policy network and the target evaluation network.

[0048] Further, the structure of the Actor policy network includes an input layer, a hidden layer and an output layer, one layer of feature extraction layer, and three layers of fully connected layers, and the sizes of the three layers of fully connected layers are 400, 300 and 9 respectively.

[0049] Further, the structure of the Critic evaluation network includes one layer of feature extraction layer and three layers of fully connected layers, and the sizes of the three layers of fully connected layers are 400, 300 and 1 respectively.

[0050] Further, the target network includes an Actor target network and a Critic target network, the structure of the Actor target network includes one layer of feature extraction layer and three layers of fully connected layers, and the sizes are 400, 300 and 9 respectively; the structure of the Critic target network includes one layer of feature extraction layer and three layers of fully connected layers, and the sizes are 400, 300 and 1 respectively.

[0051] Further, step 3 includes the following steps:

[0052] Step 3-1, constructing the state update equation of the UAV at time t:

[0053]

[0054] where x uav (t), y uav (t), and z uav (t) represent the coordinate values of the UAV at time t, represents the velocity of the UAV at time t, represents the acceleration of the UAV at time t; and Δt represents the duration of each action of the UAV.

[0055] Step 3-2, constructing the state update equation of the mobile platform at time t:

[0056]

[0057] where x platform (t), y platform (t), and z platform (t) represent the coordinate values of the mobile platform at time t, represents the velocity of the mobile platform at time t, c3 represents the height of the mobile platform, and c4 represents the speed of the mobile platform.

[0058] Further, step 4 includes the following steps:

[0059] Step 4-1, setting the maximum number of training rounds E, the maximum number of steps S per round, setting the experience pool size M, setting the update proportion coefficient τ of the target neural network, setting the learning rate l of the neural network, setting the discount factor γ, and initializing the network parameters;

[0060] Step 4-2, initializing the state space S and initializing the simulation environment;

[0061] Step 4-3, when the number of UAV training steps is less than s direct_by_expert , according to the expert algorithm, selecting the action of the UAV at the current state s t :

[0062] a t = a expert + M

[0063] where a expert is the velocity calculated based on the state at time t according to the expert algorithm, and M is a Gaussian noise vector.

[0064] When the number of UAV training steps is greater than or equal to s direct_by_expert , according to the policy network, selecting the action of the UAV at the current state s t :

[0065] at =μ d (s t |θ μ )+M

[0066] Where μ d (·) represents the deterministic policy function, and M is the Gaussian noise vector; θ μ Indicates the current policy network of the drone;

[0067] Step 4-4: The UAV performs actions based on its relative distance to the target (mobile platform). t The reward value r at time t is calculated using the reward function. t Then, the state s at the next moment is obtained from step 3. t+1 Then the quadruple t a t r t s t+1 Stored in the experience pool;

[0068] Steps 4-5: When the number of training steps for the drone is less than s direct_by_expert At that time, a small batch of samples B is randomly drawn from the experience pool for learning, the loss function is calculated, and the policy network is updated according to the minimum loss function. The learning process of the policy network can be represented as follows:

[0069]

[0070] L A Let N represent the Actor loss function, N represent the number of samples used to update the policy network at time t, and a represent the loss function of the Actor. expert It is based on the state s at time t by experts t The action at the calculation point, μ(s) t |θ μ ) is the policy network at time t with parameter θ μ The actions are calculated, and then the policy network parameters θ are updated using gradient descent based on minimizing the loss function. μ ;

[0071] When the number of training steps for the drone is greater than or equal to s direct_by_expert At that time, a small batch of samples B is randomly drawn from the experience pool for learning. The learning process can be represented as follows:

[0072] y t =r t +γQ′(s t+1 ,μ′(s t+1 |θ μ′ )|θ Q′ )

[0073] Where y t Denotes the target network, r t Let θ be the reward value at time t.​μ′ and θ Q′ represent the parameters of the target evaluation network and the target policy network respectively, Q' represents the state-action value obtained by taking μ' at time s t+1 ; γ represents a discount factor, and μ' represents the target policy network.

[0074] Step 4-6, update the policy network according to the minimum loss function:

[0075]

[0076] L C represents the critic loss function, N represents the number of samples used to update the network, a t represents the action taken by the UAV at time t, θ Q represents the evaluation network.

[0077] Step 4-7, update the policy gradient:

[0078]

[0079] wherein, represents the policy gradient under the policy network parameter θ μ , and respectively represent the evaluation network state-action value function gradient and the policy network policy function gradient, μ(s t ) represents the action policy selected at the policy network state s t , and respectively represent the state-action value of the evaluation network when the action a = μ(s t ) is taken at the state s t and the action value of the policy network at the state.

[0080] Step 4-8, update the weights of the target evaluation network and the target policy network according to the following formula:

[0081]

[0082] wherein τ is a soft update proportion coefficient.

[0083] Step 4-9, perform k = k + 1 for the iteration step length k and make a judgment. If k < S, perform t = t + Δt and return to step 4-3. Otherwise, go to step 4-10.

[0084] Step 4-10, make a judgment on the number of rounds e. If e < E, return to step 4-2. If e ≥ E, save the best network model parameters in the training process.

[0085] Compared with the prior art, the present application has the beneficial effects that:

[0086] The present application provides a mobile platform-based DDPG unmanned aerial vehicle landing method based on expert experience. The method does not rely on an environment model, establishes a deep neural network, uses sensor information such as positions and speeds of the unmanned aerial vehicle and the target as inputs of the neural network, uses speeds of each direction of the unmanned aerial vehicle as outputs, then trains a policy network based on an expert experience-based DDPG algorithm, and continues training in an AC framework, so that the mobile platform-based DDPG unmanned aerial vehicle landing method based on expert experience in a complex environment is completed, and the advantages are specifically embodied in that:

[0087] (1) The present application realizes landing of the unmanned aerial vehicle on the ground mobile platform under the condition that the environment model is unknown, adopts a deep deterministic policy gradient (DDPG) method based on expert experience, and through sampling data of the unmanned aerial vehicle, the optimal evaluation and policy network reaching the target can be autonomously learned under the guidance of the expert and the powerful fitting capability of the neural network, so that the landing task is completed.

[0088] (2) The present application uses expert experience to guide learning, greatly improves the convergence speed while ensuring the accuracy of the algorithm, saves the engineering development and model training cost, uses the Actor network trained by the expert experience for later learning, increases the possibility of the policy network outputting correct actions, thereby indirectly causes the Critic network training to accelerate, and a good cycle is formed between the two, the training time and convergence speed of the model are improved, and the cost is greatly reduced.

[0089] (3) In the unmanned aerial vehicle autonomous landing control method provided by the present application, the deep neural network after training can directly convert the position information between the unmanned aerial vehicle and the target into continuous control instructions required for landing of the unmanned aerial vehicle. The present application eliminates the state estimation and path planning modules in the traditional method, is more in line with the human thinking mode, and can improve the autonomy and landing efficiency of the unmanned aerial vehicle. BRIEF DESCRIPTION OF DRAWINGS

[0090] Figure 1 It is a mobile platform-based DDPG unmanned aerial vehicle landing method training flowchart based on expert experience;

[0091] Figure 2 It is a mobile platform-based DDPG unmanned aerial vehicle landing method algorithm structure schematic diagram based on expert experience;

[0092] Figure 3 It is a curve graph of average reward change obtained by the unmanned aerial vehicle at each step in the DDPG algorithm training process;

[0093] Figure 4The average reward change curve diagram of the unmanned aerial vehicle at each step in the training process of the DDPG algorithm based on expert experience is obtained.

[0094] Figure 5 The trajectory display diagram of the DDPG algorithm based on expert experience for completing the landing task is shown. DETAILED DESCRIPTION

[0095] The present application will be further described in conjunction with the embodiments, the drawings, but the embodiments of the present application are not limited to this embodiment.

[0096] Embodiment 1

[0097] The DDPG unmanned aerial vehicle landing method based on expert experience for mobile platform proposed in the present application, as shown in the figure, includes the following steps: Figure 1

[0098] Step 1: Constructing the Markov model (S, A, O, R, γ) for the unmanned aerial vehicle mobile platform landing, wherein S is the input state of the neural network, A is the output action of the unmanned aerial vehicle, O is the observation space of the unmanned aerial vehicle sensor, R is the reward function, and γ is the discount factor.

[0099] Step 1-1: Defining the state space of the Markov model, input state S:

[0100] In combination with the state of the unmanned aerial vehicle and the state of the mobile platform, the input state is set as:

[0101] S=[S uav ,S platform ]

[0102] Wherein: the state of the unmanned aerial vehicle x uav , y uav , z uav represents the position of the unmanned aerial vehicle in the three-dimensional plane, is the velocity component of the unmanned aerial vehicle along the X, Y, Z axes;

[0103] The state of the mobile platform S platform =[x platform , y platform , z platform ], x platform , y platform , z platform represents the position of the unmanned aerial vehicle in the three-dimensional plane;

[0104] Step 1-2: Defining the action space of the Markov model, i.e. the output action A of the unmanned aerial vehicle:

[0105] The output action A represents that the unmanned aerial vehicle receives the feedback signal from the outside world, i.e. the distance of the unmanned aerial vehicle from the mobile target, and the action set collected according to its own state; the output action is set as:​

[0106]

[0107] wherein, The speed components of the UAV along the X, Y, and Z axes are combined to constrain the speed of the UAV in actual applications:

[0108] Step 1-3: Define the observation space of the Markov model, i.e., the observation space O of the sensor:

[0109] The positions of the UAV and the target are determined and obtained using the radar sensor; the observation space is set as:

[0110] O = [D]

[0111] wherein, the relative distance D between the UAV and the target is:

[0112]

[0113] Step 1-4: Define the reward function R:

[0114] The position information of the UAV and the mobile platform is obtained using the sensor, and the reward function R is obtained by combining the distance reward and punishment, collision reward and punishment, and time reward, which represents the feedback value obtained by the UAV when selecting a certain action in the current state;

[0115] In this embodiment, L is set to 4 meters, and the distance reward function r t is:

[0116]

[0117] wherein is the distance reward function, D t represents the Euclidean distance between the UAV and the mobile platform at the current time t; if D t ≥ L, a negative punishment reward proportional to the distance D t is given; if D t <L, a positive reward proportional to the distance D t is given in addition to the normal number 1;

[0118] In this embodiment, the collision avoidance reward is set as:

[0119]

[0120] wherein D tis the Euclidean distance between the UAV and the mobile platform at the current time t, the collision penalty reward is 0 when no collision occurs, if the UAV collides, a negative penalty reward proportional to the distance from the mobile platform is given on the basis of a penalty of -10, and λ3 is a collision coefficient constant;

[0121] The reward function R is obtained by combining the UAV distance reward and the collision avoidance reward:

[0122]

[0123] Step 1-5: Define the discount factor γ:

[0124] The discount factor 0 < γ < 1 is set to calculate the cumulative value of the return in the whole process; the greater the γ value, the more emphasis on long-term income; in this embodiment, γ = 0.95 is set.

[0125] Step 2: Construct the neural network in the DDPG algorithm

[0126] Step 2-1: Construct the policy network in the DDPG algorithm, that is, the Actor policy network;

[0127] The policy network μ actor is composed of an input layer, a hidden layer and an output layer, and the output vector U of the policy network for the input state S is represented as:

[0128] U = μ actor (s)

[0129] Step 2-2: Construct the evaluation network in the DDPG algorithm, that is, the Critic evaluation network;

[0130] The output of the evaluation network is the state-action value Q μ (s, u), which is represented as:

[0131]

[0132] Where k is the summation variable, E[·] represents the mathematical expectation; s t+k+1 , u t+k+1 represent the state input vector and the action output vector at t+k+1, respectively, γ is the discount factor, r is the reward of the UAV, s represents the state of the UAV, s t is the state of the UAV at t, u represents the action policy of the UAV, and u t represents the action policy of the UAV at t;

[0133] Step 2-3: Construct the target neural network, that is, the target network, mainly including the Actor target network and the Critic target network;

[0134] The policy network μ actor and the evaluation network Q μ (s, u) are copied to the respective target networks, i.e. θ μ → θ μ′ , θ Q → θ Q′ , where θ μ , θ Q denote the parameters of the current policy network and evaluation network, respectively, and θ μ , θ Q′ denote the parameters of the current target policy network and target evaluation network, respectively.

[0135] It should be noted that in the present embodiment, the policy network, the evaluation network, and the target neural network each have three layers of neural networks, with 100 hidden layer neurons, and use the Relu activation function, and the output layer uses the tanh function.

[0136] Step 3: Update the state of the UAV and the target

[0137] Step 3-1: Construct the state update equation of the UAV at time t:

[0138]

[0139] where x uav (t), y uav (t), z uav (t) represent the coordinate values of the UAV at time t, represents the velocity of the UAV at time t, represents the acceleration of the UAV at time t; the minimum speed of the UAV is 0 m / s, and the maximum speed is 2 m / s; in the present embodiment, the simulation time interval is set to Δt = 0.5 seconds.

[0140] Step 3-2: Construct the state update equation of the mobile platform at time t:

[0141]

[0142] where x platform (t), y platform (t), z patform (t) represent the coordinate values of the mobile platform at time t, and the size of the mobile platform in the Z direction is 0, represents the velocity of the mobile platform at time t, and the velocity of the mobile platform is 1 m / s.

[0143] Step 4: Construct a UAV landing scene in the simulation simulator, and use the deterministic policy gradient method based on expert experience to train the UAV to land facing the mobile platform:

[0144] Step 4-1: Set the maximum number of training rounds E = 800, the maximum number of steps per round S = 25, set the experience pool size M = 10000, set the soft update proportion coefficient of the target neural network τ = 0.9, set the learning rate of the neural network l = 0.001, set the discount factor γ = 0.98, and initialize the network parameters;

[0145] Step 4-2: Initialize the state space S and initialize the simulation environment;

[0146] Set the initial state of the unmanned aerial vehicle Initial state of the target The starting position of the target is randomly distributed within a radius of 5 meters from the origin;

[0147] Step 4-3: In this embodiment, when the number of unmanned aerial vehicle training steps is less than 1000 steps, according to the expert algorithm, the action of the unmanned aerial vehicle is selected at the current state s t :

[0148] a t = a expert + M

[0149] Where a expert is the speed calculated based on the state at time t according to the expert algorithm, and M is a Gaussian noise vector;

[0150] In this embodiment, when the number of unmanned aerial vehicle training steps is greater than or equal to 1000 steps, according to the strategy network, the action of the unmanned aerial vehicle is selected at the current state s t :

[0151] a t = μ d (s t | θ μ ) + M

[0152] Where μ d (·) represents a deterministic policy function, and M is a Gaussian noise vector;

[0153] Step 4-4: The unmanned aerial vehicle executes the action a t According to step 1-3, the relative distance between the unmanned aerial vehicle and the moving platform is obtained, and the reward value r t at time t is calculated by the reward function in step 1-4. Then, the next state s t+1 is obtained by step 3, and then the four-tuple <s t , a t , r t , s t+1 > is stored in the experience pool;

[0154] Step 4-5: In this embodiment, when the number of training steps of the UAV is less than 1000, a small batch of samples B is randomly extracted from the experience pool for learning, a loss function is calculated, and the policy network is updated according to the minimum loss function. The learning process of the policy network is represented as:

[0155]

[0156] L A represents the Actor loss function, N represents the number of samples used to update the policy network at time t, a expert is the action calculated by the expert according to the state s t at time t, μ(s t |θ μ ) is the action calculated by the policy network at time t under the parameter θ μ , and the network parameter θ μ is updated by gradient descent according to the minimization of the loss function.

[0157] In this embodiment, when the number of training steps of the UAV is greater than or equal to 1000, a small batch of samples B = 32 is randomly extracted from the experience pool for learning, and the learning process is represented as:

[0158] y t = r t + γQ'(s t+1 , μ'(s t+1 |θ μ′ )|θ Q′ )

[0159] where y t represents the target network, r t is the reward value at time t, θ μ′ and θ Q′ represent the parameters of the target evaluation network and the target policy network respectively, Q' represents the state-action value obtained by taking μ' at time s t+1 ; γ represents the discount factor, μ' represents the target policy network.

[0160] Step 4-6: Update the policy network according to the minimum loss function:

[0161]

[0162] L C represents the critic loss function, and N represents the number of samples used to update the network.

[0163] Step 4-7: Update the policy gradient:

[0164]

[0165] where, denotes the policy gradient under the policy network parameter μ , and denote the evaluation network state-action value function gradient and the policy network policy function gradient, respectively, μ(s t ) denotes the action policy selected at the policy network state s t , and denote the evaluation network state-action value and the policy network action value at the state s t when the action a = μ(s t ) is taken;

[0166] Step 4-8: The weights of the target evaluation network and the target policy network are updated according to the following formula:

[0167]

[0168] wherein τ is a soft update proportion coefficient;

[0169] Step 4-9: Perform k = k + 1 for the iteration step length k and make a judgment. If k < S, perform t = t + Δt and return to step 4-3. Otherwise, go to step 4-10;

[0170] Step 4-10: Make a judgment on the number of rounds e. If e < E, return to step 4-2. If e ≥ E, save the best model parameters in the training process.

[0171] In this embodiment, when training, the unmanned aerial vehicle is designed to start landing from (0, 0, 5) m, and the mobile platform is located at a distance from the origin (0, 0, 0). The starting point is randomly distributed on a circle with a radius of 5 m. The average reward change curve of the unmanned aerial vehicle obtained in each round during the training process is shown in Figure 1 , and the original algorithm DDPG can obtain a high and stable reward in each round after about 8000 steps of training, as shown in Figure 2 , the improved algorithm Ex-DDPG of the present application can obtain a high and stable reward in each round after about 3000 steps of training, indicating that the DDPG algorithm based on expert guidance proposed by the present method can greatly improve the convergence speed of the original DDPC, and further improve the efficiency and stability of the autonomous decision-making process of the unmanned aerial vehicle. The simulation results are shown in Figure 5 , it can be seen that the unmanned aerial vehicle trained based on the DDPC algorithm can efficiently complete the landing task.

[0172] Embodiment 2

[0173] The structure of the Actor target network in this embodiment is one feature extraction layer and three fully connected layers with sizes of 400, 300 and 9 respectively, and the structure of the Critic target network is one feature extraction layer and three fully connected layers with sizes of 400, 300 and 1 respectively.

[0174] Embodiment 3

[0175] The structure of the Actor policy network in this embodiment is one feature extraction layer and three fully connected layers with sizes of 400, 300 and 9 respectively, and the structure of the Critic evaluation network is one feature extraction layer and three fully connected layers with sizes of 400, 300 and 1 respectively.

[0176] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application should be equivalent replacement methods and should be included in the protection scope of the present application.

Claims

1. A DDPG drone landing method based on expert experience for mobile platforms, characterized in that, Comprising the following steps: Step 1, constructing a Markov model of the unmanned aerial vehicle mobile platform landing; Step 2, constructing a neural network in the DDPG algorithm; Step 3, updating the state of the unmanned aerial vehicle and the target; Step 4, constructing an unmanned aerial vehicle landing scene in a simulation simulator, training the unmanned aerial vehicle to land on a mobile platform using a deterministic policy gradient method based on expert experience, comprising the following steps: Step 4-1, setting the maximum number of training rounds E, the maximum number of steps S per round, setting the experience pool size M, setting the target neural network update proportion coefficient τ, setting the neural network learning rate l, setting the discount factor γ, and initializing the network parameters; Step 4-2, initializing the state space S and initializing the simulation environment; Step 4-3, when the number of UAV training steps is less than s direct_by_expert , according to the expert algorithm, select the action of the UAV at the current state s t : a t = a expert + M where a expert is the velocity calculated based on the state at time t according to the expert algorithm, and M is a Gaussian noise vector; When the number of UAV training steps is greater than or equal to s direct_by_expert , according to the policy network, an action of the UAV is selected at the current state s t . a t = μ d (s t |θ μ )+ M where μ d (·) represents a deterministic policy function, M is a Gaussian noise vector; θ μ represents the current policy network of the UAV; Step 4-4, the UAV performs action a according to the relative distance between the UAV and the target mobile platform t The reward value r at time t is calculated by the reward function t The state s at the next time is obtained again by step 3 t+1 Then the four-tuple <s t , a t , r t , s t+1 > is stored in the experience pool Step 4-5, when the number of UAV training steps is less than s direct_by_expert When the number of UAV training steps is less than s, a small batch of samples B is randomly extracted from the experience pool, the loss function is calculated, and the strategy network is updated according to the minimum loss function. The learning process of the strategy network is represented as: L A represents the Actor loss function, N represents the number of samples used to update the policy network at time t, a expert is the action calculated by the expert according to the state s t at time t, μ(s t | θ μ ) is the action calculated by the policy network at time t with parameters θ μ , and the policy network parameters θ μ are updated by gradient descent according to the minimization of the loss function. When the number of UAV training steps is greater than or equal to s direct_by_expert When the number of UAV training steps is greater than or equal to s When the number of UAV training steps is greater than or equal to s y t = r t + γQ'(s t+1 , μ'(s t+1 | θ μ′ )| θ Q′ ) where y t represents the target network, r t is the reward value at time t, θ μ′ and θ Q′ represent the parameters of the target evaluation network and the target policy network, respectively, Q' represents the state-action value obtained by taking μ' at time s t+1 ; γ represents the discount factor, and μ' represents the target policy network. Step 4-6, updating the policy network according to the minimum loss function: L C critic loss function, N represents the number of samples used to update the network, a t represents the action made by the UAV at time t, θ Q represents the evaluation network; Step 4-7, updating the policy gradient: wherein, represents the policy gradient under the policy network parameters θ μ and represent the evaluation network state-action value function gradient and the policy network policy function gradient, respectively, μ(s t ) represents the action policy chosen at state s t under the policy network, and represent the evaluation network state-action value and the policy network action value at state s t for taking action a = μ(s t ), respectively.​ Step 4-8, updating the weights of the target evaluation network and the target policy network according to the following formula: Where τ is the soft update proportion coefficient; Step 4-9, executing k = k + 1 for the iteration step length k and making a judgment, if k < S, then t = t + Δt and return to step 4-3, otherwise go to step 4-10; Step 4-10, judging the number of rounds e, if e < E, return to step 4-2; if e ≥ E, save the best network model parameters in the training process. 2.The mobile platform facing expert experience based DDPG method for unmanned aerial vehicle landing according to claim 1, wherein, The Markov model is where S is the input state of the neural network, A is the output action of the UAV, O is the observation space of the UAV sensor, R is the reward function, is a discount factor. 3.The mobile platform facing expert experience based DDPG method for unmanned aerial vehicle landing according to claim 1, wherein, Step 1-1, define the state space of the Markov model, input state S: In combination with the state of the unmanned aerial vehicle and the mobile platform, the input state is set as: S = [S uav , S platform ] wherein the state of the drone x uav , y uav , z uav represent the position of the drone in a three-dimensional plane, are the velocity components of the drone along the X, Y, Z axes; Mobile platform state: S platform = [x platform , y platform , z platform ], x platform , y platform , z plqtform represent the position of the drone in a three-dimensional plane; Step 1-2, define the action space of the Markov model, i.e. the output action A of the unmanned aerial vehicle: The output action A represents the feedback signal received by the unmanned aerial vehicle from the outside world, i.e. the distance of the unmanned aerial vehicle from the mobile target, and the action set collected according to its own state; the output action is set as: wherein, is the velocity component of the UAV along the X, Y, Z axis, and the velocity of the UAV is constrained in combination with the actual application: wherein respectively represent the minimum and maximum speed of the UAV; Step 1-3, define the observation space of the Markov model, i.e. the observation space O of the sensor: Use the radar sensor to judge and obtain the position of the unmanned aerial vehicle and the target; the observation space is set as: O = [D] Where the relative distance D between the unmanned aerial vehicle and the target is: Step 1-4, define the reward function R: Use the sensor to obtain the position information of the unmanned aerial vehicle and the mobile platform, obtain the reward function R by rewarding and punishing the distance of the unmanned aerial vehicle, collision reward and time reward, which represents the feedback value obtained by the unmanned aerial vehicle when selecting a certain action in the current state: combine the distance reward and collision reward of the unmanned aerial vehicle to obtain the reward function R: wherein λ4, λ5 represent the weight coefficients of the distance reward collision avoidance reward respectively. Step 1-5, defining discount factor Setting discount factor for calculating the cumulative value of returns throughout the process; when The greater the value, the greater the emphasis on long-term returns.

4. The mobile platform expert experience oriented DDPG drone landing method of claim 3, wherein, Distance reward function is: wherein is the distance reward function, λ1, λ2 are the weight values of the two rewards; D t represents the Euclidean distance between the unmanned aerial vehicle and the mobile platform at the current time t; L is a constant value set; if D t ≥ L, a negative punishment reward proportional to the distance D t is given; if D t <L, a positive reward inversely proportional to the distance D t is given on the basis of the normal number c1. Collision avoidance reward For: where D t is the Euclidean distance between the UAV and the mobile platform at the current time t, the collision penalty reward is 0 when no collision occurs, if the UAV collides, a negative penalty reward proportional to the distance from the mobile platform is given in addition to a negative penalty of a constant c2, and λ3 is a collision coefficient constant. 5.The mobile platform-oriented DDPG drone landing method based on expert experience of claim 1, wherein, Step 2 comprises the following steps: Step 2-1, constructing a policy network in the DDPG algorithm, i.e. an Actor policy network; For the input state S, the output vector U of the policy network is represented as: U = μ actor (s) Step 2-2, constructing an evaluation network in the DDPG algorithm, i.e. a Critic evaluation network; The output of the evaluation network is a state-action value Q μ (s, u), denoted as: wherein k is a summation variable, E[·] represents a mathematical expectation; s t+k+1 , u t+k+1 respectively represent a state input vector and an action output vector at a t+k+1 moment; γ is a discount factor, r is a reward of the UAV, s represents a state of the UAV, s t is a state of the UAV at a t moment, u represents an action policy of the UAV, u t represents an action policy of the UAV at a t moment; Step 2-3, constructing a target neural network, i.e. a target network; The policy network μ actor and the evaluation network Q μ are copied to the respective target networks, i.e. θ μ → θ μ′ , θ Q → θ Q′ , where θ μ , θ Q denote the parameters of the current policy network and evaluation network, respectively, and θ μ′ , θ Q′ denote the parameters of the current target policy network and target evaluation network, respectively. 6.The mobile platform-oriented DDPG drone landing method based on expert experience of claim 1, wherein, The structure of the Actor policy network includes an input layer, a hidden layer and an output layer, one feature extraction layer and three fully connected layers, with the sizes of the three fully connected layers being 400, 300 and 9 respectively.

7. The mobile platform expert experience oriented DDPG drone landing method of claim 1, wherein, The critic evaluation network structure includes a feature extraction layer, three fully connected layers, and the sizes of the three fully connected layers are 400, 300, and 1 respectively. 8.The mobile platform-oriented DDPG drone landing method based on expert experience of claim 1, wherein, The target network includes an Actor target network and a Critic target network. The structure of the Actor target network includes a feature extraction layer, three fully connected layers, and the sizes of the three fully connected layers are 400, 300, and 9 respectively. The structure of the Critic target network includes a feature extraction layer, three fully connected layers, and the sizes of the three fully connected layers are 400, 300, and 1 respectively. 9.The mobile platform-oriented DDPG drone landing method based on expert experience of claim 1, wherein, Step 3 includes the following steps: Step 3-1, constructing a state update equation of the unmanned aerial vehicle at time t: wherein x uav (t), y uav (t), z uav (t) represents the coordinate value of the UAV at time t, represents the velocity of the UAV at time t, represents the acceleration of the UAV at time t; Δt represents the time duration for the UAV to perform one action. Step 3-2, constructing a state update equation of the mobile platform at time t: wherein x platform (t), y platform (t), z platform (t) represents the coordinate value of the mobile platform at time t, represents the velocity of the mobile platform at time t, c3 represents the height of the mobile platform, and c4 represents the velocity of the mobile platform.

Images