Unmanned aerial vehicle agent task execution and optimal trajectory generation method based on deep reinforcement learning

Abstract

The invention belongs to the field of computer software, and provides an unmanned aerial vehicle intelligent body task execution and optimal trajectory generation method based on deep reinforcement learning. The unmanned aerial vehicle intelligent body task execution model constructed through the TD3 algorithm can realize more accurate unmanned aerial vehicle action value evaluation according to the unmanned aerial vehicle observation value. During iterative training of an unmanned aerial vehicle agent task execution model, task execution actions of an unmanned aerial vehicle are constrained by using an observation space and an action space, and it can be ensured that each decision in task execution of the unmanned aerial vehicle is beneficial for the unmanned aerial vehicle to complete a task execution target; the unmanned aerial vehicle agent task execution model is trained through the unmanned aerial vehicle agent task execution function, and trajectory prediction of task execution of the unmanned aerial vehicle is improved. Clustering analysis is carried out on trajectory information of all execution tasks of the unmanned aerial vehicle intelligent body through a K-Medoids center clustering algorithm, the limitation of single reasoning decision can be effectively avoided, and the task completion reliability is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer software, and particularly relates to a method for task execution and optimal trajectory generation of a UAV agent based on deep reinforcement learning. BACKGROUND

[0002] With the popularization of the application of UAVs, the strategy and path planning selected by the UAV when performing a task have become a focus of current research. In the existing solutions, the commonly used methods include A* search algorithm, genetic algorithm, ant colony algorithm and expert experience. Among them, the A* algorithm is a heuristic search algorithm, and its performance depends largely on the quality of the heuristic function. In addition, the A* algorithm is suitable for static discrete search space, and is not suitable for the dynamic state space of the UAV. The genetic algorithm also has poor adaptability in the dynamic environment of the UAV task execution and the like. There is no mature and effective quantitative analysis method in terms of algorithm accuracy, computational complexity and the like, which brings certain difficulties to the optimization and improvement of the algorithm. The ant colony algorithm has certain global search ability, but it is easy to fall into a local optimal solution. Although the expert experience can play a good role in a specific scene, it requires a large amount of expert work, and further development is needed to face different scenes. In addition, although some research work combines reinforcement learning with UAV task execution, it is still in the initial stage, and there are problems such as lack of consideration of anti-interference information, coarse path granularity, and lack of use of real coordinate information of the UAV. SUMMARY

[0003] In order to solve the problem that the UAV agent in the prior art has poor dynamic adaptability, poor anti-interference performance and weak generalization ability when performing a task, resulting in the inability to generate an optimal trajectory for the UAV agent to perform a task, a method for task execution and optimal trajectory generation of a UAV agent based on deep reinforcement learning is proposed, which solves the above problems.

[0004] A method for task execution and optimal trajectory generation of a UAV agent based on deep reinforcement learning, comprising the following steps:

[0005] S1: setting and loading a UAV task document: setting a UAV task document according to the task scene of the UAV agent and the task target of the UAV agent, and loading the UAV task document in a simulator;

[0006] S2: constructing an observation space according to the real-time characteristics of the UAV agent, the latitude and longitude coordinates and the orientation angle of all obstacles on the ground; setting an action space according to the value range of the UAV agent action; setting a UAV agent task execution function according to the UAV task document, including a reward function, an end function, a reset function and a step function;

[0007] S3: constructing a UAV agent task execution model: constructing a UAV agent task execution model based on a double-delay deep deterministic policy gradient algorithm, including a policy network and a value function network;

[0008] S4: training the UAV agent: inputting the UAV observation value in the observation space into the policy network to obtain a UAV action value in the action space; inputting the UAV action value into the UAV agent task execution function to obtain UAV state feedback data; inputting the UAV state feedback data into the value function network to calculate a loss function value, and performing gradient update on the UAV agent task execution model according to the loss function value to obtain an optimal UAV agent task execution model;

[0009] S5: generating an optimal trajectory of the UAV agent: performing a UAV reasoning task through the optimal UAV agent task execution model, and saving all trajectory information in the UAV reasoning task process; clustering the all trajectory information through a K-Medoids center clustering algorithm to obtain an optimal trajectory.

[0010] Preferably, the range of the UAV action value output by the policy network is limited by an action space limitation strategy; the range of information that the UAV can receive during task execution, i.e., the UAV observation value, is limited; if the corresponding value of the UAV observation value is not within the set range of the observation space, the task is terminated.

[0011] Preferably, the reward function is set according to whether the task execution action of the UAV agent is beneficial to completing the task target of the UAV agent; if the task execution action of the UAV is beneficial to completing the task target of the UAV agent, a positive number is returned, otherwise a negative number or zero is returned.

[0012] Preferably, the UAV observation value includes but is not limited to: real-time latitude and longitude coordinates of the UAV, a heading angle, latitude and longitude coordinates of a ground obstacle detected by the UAV, and an orientation; the UAV action value includes: latitude and longitude coordinates of the UAV, a flight direction, a speed, and a time for a further step.

[0013] Preferably, the method for training the UAV agent in step S4 is:

[0014] A1: inputting the UAV observation value into the policy network to obtain a UAV action value;

[0015] A2: inputting the UAV action value into a step function for action judgment to obtain UAV state feedback data; the UAV state feedback data includes: a new UAV observation value, a reward value, and a task termination Boolean value;

[0016] A3: inputting the new observation value of the UAV and the reward value into the value function network to obtain a value function estimation value; and calculating a loss function value according to the value function estimation value;

[0017] A4: calculating a gradient of the UAV intelligent agent task execution model according to the loss function value; performing gradient update on the value function network according to the gradient; when the number of times of gradient update of the value function network reaches a preset number of times, performing gradient update on the policy network, and resetting the number of times of update of the value function network;

[0018] A5: determining whether to end the training according to the task termination Boolean value; if the task termination Boolean value is 0, determining whether the reward value is greater than an optimal reward value; if the reward value is greater than the optimal reward value, taking the current model parameter as a parameter of an optimal UAV intelligent agent task execution model, and taking the current reward value as the optimal reward value; if the task termination Boolean value is 1, ending the training, outputting the optimal UAV intelligent agent task execution model, and resetting and initializing all parameters in the UAV intelligent agent task execution function through a reset function.

[0019] Preferably, the step in the step function is:

[0020] B1: communicating with a simulator; and the simulator updating an observation value of the UAV according to the action value of the UAV to obtain a new observation value of the UAV;

[0021] B2: calculating a distance between the latitude and longitude coordinates in the new observation value of the UAV and a target; and if the distance is less than a preset radius of the UAV, starting a preset program of the UAV;

[0022] B3: calculating a reward value generated by the new observation value of the UAV and starting the preset program of B2 through a reward function, and simultaneously increasing a task execution step number by 1;

[0023] B4: calling an end function to determine whether the task executed in the current round reaches an end state to obtain a task termination Boolean value; and outputting the new observation value of the UAV, the reward value and the task termination Boolean value.

[0024] Preferably, the loss function comprises a policy gradient loss, a value function loss and a KL divergence, and the formula is:

[0025] (1)

[0026] (2)

[0027] (3)

[0028] (4)

[0029] wherein, represents a policy gradient loss; represents a parameter of a policy network; VF represents a value function loss; and KL represents a KL divergence; is an i-th value function network; and a both represent a UAV action value output by a policy network; D is a set of UAV observation values; s is a current UAV observation value; E[] represents a mathematical expectation; and b represents a target value; represents a value function estimation value output by an i-th value function network; represents a true probability distribution of y given a condition x; represents an approximate probability distribution of y given a condition x; x represents a UAV observation value; and y represents a UAV action value output by a decision network.

[0030] Preferably, the method for performing model inference on the optimal UAV intelligent agent task execution model is:

[0031] C1: inputting a UAV observation value into a policy network in the optimal UAV intelligent agent task execution model to obtain a UAV action value;

[0032] C2: inputting the UAV action value into a step function to obtain a new UAV observation value, a reward value, and a task termination Boolean value;

[0033] C3: if the task termination Boolean value is 1, ending the UAV task, saving a current task execution step number and a task execution trajectory, and jumping to C4; otherwise, jumping to C1;

[0034] C4: increasing a task execution number by 1 to obtain a current task execution number; determining whether the current task execution number reaches a maximum task execution number; if the current task execution number reaches the maximum task execution number, ending the inference; if the current task execution number does not reach the maximum task execution number, resetting parameters through a reset function, and jumping to C1 to perform the next task execution.

[0035] Preferably, the method for obtaining the optimal trajectory through K-Medoids center clustering in step S5 is:

[0036] D1: counting a number of latitude-longitude coordinate pairs in all trajectory information in the UAV inference task process, and calculating an average value K of a number of latitude-longitude coordinate pairs contained in a task execution trajectory;

[0037] D2: selecting K sample points as center points from the all trajectory information through a K-Medoids center clustering method, and assigning the remaining sample points to a cluster in which a center point located closest to the remaining sample points; the sample points are latitude-longitude coordinate pairs;

[0038] D3: Calculate the distance sum of each sample point in each cluster to other sample points, select the sample point with the minimum distance sum as the new center point; assign the remaining sample points to the cluster where the nearest new center point is located;

[0039] D4: If the center point is not changed or reaches the maximum clustering number, the current center point is taken as the optimal trajectory, otherwise, repeat the operation of D3.

[0040] Beneficial effects:

[0041] The application belongs to the field of computer software, and proposes a method for executing tasks and generating optimal trajectories of unmanned aerial vehicle agents based on deep reinforcement learning, which combines reinforcement learning algorithm (TD3 algorithm) and unsupervised learning algorithm (K-Medoids center clustering algorithm), improves the dynamic adaptability, anti-interference and generalization ability of decision-making, can more efficiently and accurately guide the unmanned aerial vehicle agent to complete the specified execution task, and realizes the selection of the optimal trajectory when the unmanned aerial vehicle agent executes the task.

[0042] The unmanned aerial vehicle task document is set according to the task environment of the unmanned aerial vehicle agent and the task target of the unmanned aerial vehicle agent, the unmanned aerial vehicle task parameters and success conditions can be systematically defined, the planning of the unmanned aerial vehicle agent in the execution process can be effectively constrained, the problems of poor dynamic adaptability, weak anti-interference and insufficient generalization ability can be effectively solved, and the execution trajectory of the task is optimized.

[0043] The unmanned aerial vehicle agent task execution model based on the double-delay deep deterministic policy gradient algorithm is constructed, the unmanned aerial vehicle agent can be effectively trained according to the observation value of the unmanned aerial vehicle, more accurate evaluation of the action value of the unmanned aerial vehicle is realized, the policy deviation caused by overestimation of rewards is avoided, the optimal trajectory of the unmanned aerial vehicle when completing the task can be obtained, and the reliability of the unmanned aerial vehicle in improving the completion of the task is improved.

[0044] In the iterative training of the unmanned aerial vehicle agent task execution model, the execution task action of the unmanned aerial vehicle is constrained by the observation space and the action space, so that each decision in the execution task of the unmanned aerial vehicle can be beneficial to the completion of the execution task target of the unmanned aerial vehicle; the unmanned aerial vehicle agent task execution function is used to train the unmanned aerial vehicle agent task execution model, so that the unmanned aerial vehicle agent task execution model can more accurately evaluate the next action according to the current observation value of the unmanned aerial vehicle, and the trajectory prediction of the unmanned aerial vehicle in executing the task is improved.

[0045] The K-Medoids center clustering algorithm is used to cluster and analyze the trajectory information of all execution tasks of the unmanned aerial vehicle agent, which can effectively avoid the limitations of single inference decision (such as the possible success or failure deviation based on a single trajectory), so as to select the optimal trajectory from the global trajectory data and improve the reliability of task completion. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating a method for a deep reinforcement learning-based unmanned aerial vehicle (UAV) agent to perform tasks and generate optimal trajectories.

[0047] Figure 2 System architecture diagram for training unmanned aerial vehicle agents.

[0048] Figure 3 A flowchart for training unmanned aerial vehicle agents.

[0049] Figure 4 This is a structural diagram of an optimal trajectory generation system for unmanned aerial vehicle (UAV) intelligent agents.

[0050] Figure 5 A flowchart for generating the optimal trajectory for a drone intelligent agent.

[0051] Figure 6 A schematic diagram illustrating the decision-making process when an untrained drone agent performs a task.

[0052] Figure 7 A schematic diagram illustrating the decision-making process of a trained unmanned aerial vehicle (UAV) agent when performing a task.

[0053] Figure 8 The change in loss rate during the training of the drone intelligent agent task execution model.

[0054] Figure 9 The change in reward value during the training of the drone intelligent agent task execution model. Detailed Implementation

[0055] The following description, in conjunction with the accompanying drawings and specific implementation details, provides further explanation.

[0056] Example 1:

[0057] like Figure 1 As shown, a method for UAV intelligent agents to perform tasks and generate optimal trajectories based on deep reinforcement learning is described, with the following steps:

[0058] S1: Set and load the UAV mission document: Set the UAV mission document according to the mission scenario and mission objectives of the UAV agent, and load the UAV mission document in the simulator;

[0059] S2: Construct an observation space based on the real-time characteristics of the UAV agent, the latitude and longitude coordinates and orientation angles of all obstacles on the ground; set the action space based on the value range of the UAV agent's actions; set the UAV agent's task execution functions based on the UAV task document, including: reward function, termination function, reset function and step function;

[0060] Specifically, the observation space is a constraint on the range of real-time information values that the UAV can receive, i.e., a constraint on the UAV observation values;

[0061] Specifically, the action space is a constraint on the range of response values that the UAV can make, i.e., a constraint on the range of UAV action values;

[0062] Specifically, the UAV agent task execution function is called at each step of the UAV executing a task, and the task state and whether to reward or punish the UAV agent are determined by the UAV agent task execution function;

[0063] The reward function has the effect that when the UAV gradually approaches the target or starts the preset program for the target, the system gives the UAV agent a positive reward; when the UAV moves away from the target or collides with an obstacle, the system gives the UAV agent a negative reward;

[0064] The end function has the effect of determining in real time whether the UAV ends the task; if the UAV successfully executes the task or collides with an obstacle, the task ends;

[0065] The reset function has the effect that if the UAV ends the task, all states of the system are reset through the reset function, and a new task execution can be started.

[0066] S3: Constructing a UAV agent task execution model: constructing a UAV agent task execution model based on a double-delay deep deterministic policy gradient algorithm, including a policy network and a value function network;

[0067] S4: Training the UAV agent: inputting the UAV observation value in the observation space into the policy network to obtain the UAV action value in the action space; inputting the UAV action value into the UAV agent task execution function to obtain UAV state feedback data; inputting the UAV state feedback data into the value function network to calculate a loss function value, and performing gradient update on the UAV agent task execution model according to the loss function value to obtain an optimal UAV agent task execution model;

[0068] S5: Generating an optimal trajectory of the UAV agent: performing a UAV reasoning task through the optimal UAV agent task execution model, and saving all trajectory information in the UAV reasoning task process; clustering the all trajectory information through a K-Medoids center clustering algorithm to obtain an optimal trajectory.

[0069] Embodiment Two

[0070] I. UAV agent training phase:

[0071] As Figure 2As shown, the unmanned aerial vehicle task execution training system based on a double-delay deep deterministic policy gradient (TD3) algorithm mainly comprises the following modules:

[0072] An unmanned aerial vehicle agent module:

[0073] The unmanned aerial vehicle agent module is the core module for the unmanned aerial vehicle to execute tasks; the unmanned aerial vehicle agent module needs to be provided with an observation space, an action space, a step function, a reset function, an end function, a reward function, a rough region for the unmanned aerial vehicle to execute tasks, and a task target to be executed, etc. The observation space is a real number vector composed of real-time latitude and longitude coordinates of the unmanned aerial vehicle, a heading angle, latitude and longitude coordinates of a ground obstacle detected by the unmanned aerial vehicle, and a heading, etc. The action space is a vector composed of a range of angles that can be changed by the unmanned aerial vehicle, whether a preset program is started, etc. The step function is a logical combination of state feedback data of the unmanned aerial vehicle obtained by the unmanned aerial vehicle according to an action value of the unmanned aerial vehicle output by a decision network (actor), and real-time communication with a simulator to obtain real-time simulation situation information. The main function of the reset function is to reset the parameters for the unmanned aerial vehicle to execute tasks. The rough region for the unmanned aerial vehicle to execute tasks and the task target to be executed are set in a task document of the unmanned aerial vehicle. The rough region for the unmanned aerial vehicle to execute tasks is a range of latitude and longitude coordinates for the unmanned aerial vehicle to execute tasks.

[0074] Specifically, the decision network refers to an actor network in a task execution model of the unmanned aerial vehicle agent, which outputs an action value to be executed by the unmanned aerial vehicle at the next step according to an input observation value of the unmanned aerial vehicle.

[0075] Specifically, the real-time simulation situation information is an observation value of the unmanned aerial vehicle.

[0076] A simulator module:

[0077] The simulator is a deduction system composed of a third party to perform real-time animation simulation, and common deduction systems include Mozi, MaxSim, and Zhankun, etc.

[0078] A buffer module:

[0079] The buffer stores real-time data interacted by the unmanned aerial vehicle agent and the simulator in the memory. When the unmanned aerial vehicle executes tasks, the trainer directly reads data in batches from the buffer module, which not only shortens the time for obtaining situation information, but also uses batch data to train and update the gradient of the task execution model of the unmanned aerial vehicle agent, which not only effectively prevents the model from falling into a local optimum, but also accelerates the convergence speed of the model.

[0080] Specifically, the data for training refers to the observation value, action value and reward value of the UAV agent at each step during the execution of the task. Saving the observation value, action value and reward value at each step during the execution of the task for training the UAV agent can enable the UAV to make better decisions based on the latest situation, so that the UAV agent can perform better actions to complete the set task target.

[0081] Specifically, the situation information refers to the UAV observation value, and the UAV observation value includes the longitude and latitude, orientation, longitude and latitude of the target and obstacles, orientation, etc. of the UAV; the obstacles are divided into static obstacles and dynamic obstacles, such as buildings, airplanes, mountains, birds, etc.

[0082] The trainer and evaluator module:

[0083] Specifically, the trainer in the trainer and evaluator module is used to control the entire training process of the UAV during the execution of the task. After starting the training according to the set hyperparameters, the trainer constantly reads the UAV observation value from the buffer, and transmits the UAV observation value to the actor network and critic network of the UAV agent task execution model, and then calculates the loss function according to the output of the critic network, performs model gradient update, etc. While training, the current model can also be evaluated according to the evaluation frequency set in the hyperparameters. During evaluation, the gradient of the model is not updated, and the best model and related log information are saved to a file in a timely manner according to the evaluation result.

[0084] Specifically, the trainer includes a loss function, an optimizer and a UAV agent task execution model constructed based on the TD3 algorithm.

[0085] The loss function includes a policy gradient loss, a value loss function and a KL divergence; the formula of the loss function is:

[0086] (1)

[0087] Policy gradient loss (policy loss): used to measure the error of the current policy relative to the expected behavior, which is usually updated by maximizing or minimizing the expected return to update the parameters of the policy network; the formula of the policy gradient loss is:

[0088] (2)

[0089] Where, is the i-th value function network (critic); is the UAV action value output by the current policy network; D is the experience replay buffer; s is the current UAV observation value; E[] represents the mathematical expectation; represents the value function estimation value corresponding to the output of the ith value function network when the current UAV observation value and UAV action value are inputted;

[0090] Value function loss (vf loss): it measures the error between the value function estimation value and the true value (target value); the formula of the value function loss is:

[0091] (3)

[0092] (5)

[0093] (6)

[0094] wherein, and are two value function networks (critics); is a policy network (actor); is the clipped noise, used for target policy smoothing; represents the next step UAV observation value; represents the action value outputted by the policy network when the next step UAV observation value is inputted; r represents the reward value of the UAV after executing the action value a; E[] represents the mathematical expectation; b is the target value; c represents the noise clipping range, represents that the added noise conforms to the normal distribution with a standard deviation of () represents a noise clipping function, which limits the noise in the range of-c to c;

[0095] Kullback-Leibler divergence: Kullback-Leibler divergence, also known as relative entropy, is a method for describing the gap between two probability distributions P and Q. The formula of Kullback-Leibler divergence is:

[0096] (4)

[0097] wherein, represents the true probability distribution or target distribution of y given the condition x; represents the approximate probability distribution or model distribution of y given the condition x; x represents the conditional variable, i.e. the observation value obtained by the UAV; y represents the target variable to be modeled or predicted, i.e. the UAV action value outputted by the decision network;

[0098] The loss function value is calculated in the optimizer, the gradient is obtained by deriving the model parameters according to the loss function value, and the gradient update of the model parameters is completed according to the gradient.

[0099] The UAV intelligent agent task execution model constructed based on the TD3 algorithm comprises:

[0100] ​Twin Delayed Deep Deterministic policy gradient (TD3) is an online off-policy deep reinforcement learning algorithm for solving continuous control problems, which is improved on the basis of Deep Deterministic policy Gradient (DDPG) algorithm. The TD3 algorithm proposes three key techniques on the basis of the DDPG algorithm:

[0101] Double network: two sets of Critic networks are used, and the smaller value of the two is taken when calculating the target value, so as to suppress the network overestimation problem;

[0102] Target policy smoothing regularization: when calculating the target value y, a disturbance is added to the action value of the next state , which enhances the adaptability to small environmental changes, so that the value evaluation is more accurate;

[0103] Delayed update: the value function network (Critic network) is updated multiple times, and then the policy network (Actor network) is updated, so as to ensure that the training of the Actor network is more stable; at the same time, the problem of frequent adjustment of the policy is avoided.

[0104] Specifically, the UAV agent task execution model constructed based on the TD3 algorithm effectively solves the overestimation problem of traditional reinforcement learning in continuous action space through the double network structure, delayed update and target policy smoothing regularization, and significantly improves the policy stability and convergence efficiency. The UAV agent task execution model constructed based on the TD3 algorithm can more accurately learn the task policy in a complex environment and realize more reliable and accurate path planning.

[0105] Specifically, as shown in Figure 3 , the training process of the UAV agent executing tasks is as follows:

[0106] 1) Define the UAV task document, that is, set the UAV task document:

[0107] Set the UAV task document, which includes the execution environment and execution target of the UAV:

[0108] For example, assuming that the UAV executes a task, the purpose of the UAV is to avoid obstacles 1-3, find out the specific position (target) of obstacle 4, and plan an optimal path to approach obstacle 4 and start a preset program (such as cleaning obstacle 4).

[0109] Define the activity area of the UAV, the longitude range is [42.7864, 45.9286], and the latitude range is [32.0139, 34.2072]. In the training phase, if the activity range of the UAV exceeds the specified area, it is considered that the UAV has deviated too far from the target, and the task should be ended and all environmental variables should be reset for the next task.

[0110] 2) Load the UAV task document:

[0111] The defined UAV task document needs to be loaded in the simulator, and the initialization work of the animation simulation resource loading has been completed. At the same time, the following UAV task execution parameters need to be set, that is, the parameters in the UAV agent task execution function:

[0112] Initialize the object in the UAV main code;

[0113] Set the maximum number of task execution steps of the UAV: max_step = 500;

[0114] Set the current task execution step of the UAV: step = 0;

[0115] Set the reward value of the UAV: reward = 0;

[0116] Initialize the UAV connection handle with the simulator;

[0117] 3) Define the observation space:

[0118] The observation space is the value range of the UAV's own real-time features and the information features detected, which is a 2 x 15 continuous value two-dimensional array, specifically:

[0119] [[42.77, 32.012, 0, 0, 0, 0, 0, 0, 0, 0 0, 0, 0, 0, 0],[45.93, 34.3,360, 45.93, 34.3, 360, 45.93, 34.3, 360, 45.93, 34.3, 360, 45.93, 34.3, 360]]

[0120] The first list indicates that the observation value range of the UAV's longitude coordinate is [42.77, 45.93];

[0121] The second list indicates that the observation value range of the UAV's latitude coordinate is [32.012, 34.3];

[0122] The third list indicates that the angle between the UAV's orientation and the north direction is [0, 360];

[0123] The fourth list represents the longitude coordinate range of the obstacle 1 as [0, 45.93];

[0124] The fifth list represents the latitude coordinate range of the obstacle 1 as [0, 34.3];

[0125] The sixth list represents the angle range between the orientation of the obstacle 1 and the north direction as [0, 360];

[0126] The seventh list represents the longitude coordinate range of the obstacle 2 as [0, 45.93];

[0127] The eighth list represents the latitude coordinate range of the obstacle 2 as [0, 34.3];

[0128] The ninth list represents the angle range between the orientation of the obstacle 2 and the north direction as [0, 360];

[0129] The tenth list represents the longitude coordinate range of the obstacle 3 as [0, 45.93];

[0130] The eleventh list represents the latitude coordinate range of the obstacle 3 as [0, 34.3];

[0131] The twelfth list represents the angle range between the orientation of the obstacle 3 and the north direction as [0, 360];

[0132] The thirteenth list represents the longitude coordinate range of the obstacle 4 as [0, 45.93];

[0133] The fourteenth list represents the latitude coordinate range of the obstacle 4 as [0, 34.3];

[0134] The fifteenth list represents the angle range between the orientation of the obstacle 4 and the north direction as [0, 360].

[0135] 4) Define the action space:

[0136] The action space is the value range of the action value given by the decision network to the UAV, which is specifically defined as a discrete integer value ranging from 0 to 360, representing the angle value that the UAV can modify the flight direction each time.

[0137] 5) Set the reward function:

[0138] The reward function is used to judge whether the action performed by the UAV is beneficial to the further execution of the task. If it is beneficial, a positive number is returned, otherwise a negative number or zero is returned. The initial reward value reward = 0, and the specific calculation logic is:

[0139] The closer the UAV is to the target, the greater the reward value, i.e. reward += 10;

[0140] When the UAV flies towards the target, the reward value is increased by 20, i.e. reward += 20;

[0141] When the UAV flies away from the target, the reward value is decreased by 20, i.e. reward -= 20;

[0142] When the UAV detects the target, the reward value is increased by 50, i.e. reward += 50;

[0143] When the UAV starts the preset program, the reward value is increased by 150, i.e. reward += 150.

[0144] 6) Set the end function:

[0145] The function of the end function is to judge whether the current simulation reaches the end state, and if it reaches the end state, it returns True, otherwise it returns False. The specific judgment logic is:

[0146] Judge whether the current task execution step step is greater than the maximum task execution step max_step, if yes, directly return True, otherwise proceed to the next step;

[0147] Judge whether the UAV flies out of the specified area, i.e. whether the UAV's latitude and longitude exceeds the latitude and longitude range set in the observation space, if yes, directly return True, otherwise proceed to the next step;

[0148] Judge whether the UAV has started the preset program, if yes, directly return True, otherwise proceed to the next step;

[0149] If the above conditions are not met, return False.

[0150] 7) Set the reset function:

[0151] The UAV needs to call the reset function to initialize the UAV task execution parameters before each task execution, and return the observation value feature vector (list) and other information dictionary (dict). The other information dictionary mainly records whether the UAV has started the preset program during task execution and the specific reason for the end of the task, which has no substantial impact on the entire task execution process, and is convenient for checking detailed information.

[0152] Initialize the UAV task execution parameters; the UAV task execution parameters include:

[0153] The maximum number of task executions: max_step=500;

[0154] The current task execution step: step=0;

[0155] The reward value: reward=0;

[0156] Reconnect to the simulator handle.

[0157] 8) Set the step function:

[0158] The step function is the specific logic implementation part of each step in the execution of the task of the UAV, taking the action value given by the decision network as input, and the UAV performs the following related operations according to the action value:

[0159] Calculate the observation value of the next path point of the UAV according to the input action value; the observation value of the next path point of the UAV is calculated in the simulator; in the simulator, the initial latitude and longitude coordinates of the UAV, the flight direction, the speed, and the time for the next step are combined to calculate the real-time observation value of the UAV, which can be directly obtained through the simulator.

[0160] Determine the distance between the UAV coordinate point in the new observation value and the target. If the distance is less than the preset radius of the UAV, the UAV starts the preset program.

[0161] Call the reward function to calculate the reward value generated after the execution of the new action;

[0162] Task execution step number step += 1;

[0163] Call the end function to determine whether the task executed this round reaches the end state;

[0164] The return value of the step function is: the new UAV observation value (list), the reward value obtained after the execution of the action (reward), the task termination Boolean value, and the other information dictionary (dict).

[0165] 9) Define the TD3 algorithm: (input is the observation value of the UAV)

[0166] In this module, the TD3 algorithm is initialized with hyperparameters. The main hyperparameters set are:

[0167] twin_q = True whether to use double Q function;

[0168] policy_delay = 2 delay policy update step number;

[0169] smooth_target_policy = True whether to perform smoothing operation on the target policy;

[0170] l2_reg = 0.0 L2 regularization factor;

[0171] tau = 5e-3 target network update factor;

[0172] train_batch_size = 100 sample dataset batch size;

[0173] target_noise = 0.2 target noise factor;

[0174] target_noise_clip = 0.5 target noise clipping threshold;

[0175] actor_hiddens = [400, 300] width of two hidden layers in actor network;

[0176] actor_hidden_activation = "relu" activation function used in actor network is relu;

[0177] actor_lr = 1e-3 actor network learning rate;

[0178] critic_hiddens = [400, 300] width of two hidden layers in critic network;

[0179] critic_hidden_activation = "relu" activation function used in critic network is relu function;

[0180] critic_lr = 1e-3 critic network learning rate.

[0181] 10) Agent training:

[0182] Specifically, the agent training needs to define and initialize the functions related to the unmanned aerial vehicle agent first, and then start the iterative training of the unmanned aerial vehicle agent. The parameters related to the overall training process are mainly:

[0183] num_cpus_for_main_process = 0 The number of CPUs used in the training process is 0, which means that the system will apply for CPU resources according to the actual server idle situation;

[0184] num_gpus = 1 The number of GPUs used in the training process is 1, using 1 GPU;

[0185] evaluation_interval = 3 Model evaluation frequency, the model is evaluated once every 3 training times;

[0186] output_dir = “. / model” Model and log saving path;

[0187] num_train_steps = 100 total number of model training iterations;

[0188] best_policy_reward_mean = -99999 best model evaluation policy network reward mean value;

[0189] Specifically, the process of training the agent is to input the observation value into the policy network (action) to obtain the UAV action value; after the UAV executes the action value, it will obtain the new observation value of the UAV, determine the reward value corresponding to the action value through the reward function, and then send the new UAV observation value and the reward value into the value function network (critic) to evaluate the goodness of the action value (calculate the loss function). According to the loss function, the gradient of the UAV agent task execution model is calculated, and the system is then adjusted according to the gradient to update the parameters in the action network and the critic network, so that the action value given by the action next time is more accurate.

[0190] Specifically, in the training iteration process, the system will call the agent step function, loss value calculation, and gradient update.

[0191] Specificly, through the above continuous iteration optimization, the action value given by the agent can finally reasonably control the UAV to execute the task.

[0192] Specifically, during the training process of the UAV agent, there is no output, only the parameters of the optimal UAV agent model in the training process are saved.

[0193] 11) Evaluate and save the model:

[0194] During the training process, when the number of training iterations reaches the set evaluation frequency, the model will be evaluated. The function call process during evaluation is consistent with the call process during training, but there is no gradient update operation during evaluation.

[0195] The evaluation will return the policy network reward mean value policy_reward_mean. If the current policy_reward_mean is greater than the best reward value best_policy_reward_mean, the following operations will be performed:

[0196] (1) best_policy_reward_mean = policy_reward_mean;

[0197] (2) save the model to the specified path;

[0198] (3) judge whether the maximum number of training is reached:

[0199] If the number of training iterations reaches the set maximum value, the entire training process for the drone agent ends; otherwise, training continues.

[0200] like Figure 8 As shown, the light-colored line represents the original loss transformation value during the training process of the drone agent, and the dark-colored line represents the loss transformation value after smoothing the original loss value. Figure 8 The graph shows the change in the loss function value of the UAV agent during the training process. The loss value of the UAV agent decreases during the training process, indicating that the UAV agent can better learn environmental features (such as the location of the target and obstacle 4 and the latitude, longitude and orientation of obstacles 1-3), thereby generating a more accurate path to perform the task.

[0201] like Figure 9 As shown, the light-colored line represents the original reward value transformation during model training, and the dark-colored line represents the reward value transformation after smoothing the original reward value. Figure 9 The graph shows the change in reward value of the UAV agent during the training process, where the reward value of the UAV agent shows an upward trend during the training process; this indicates that each decision made by the UAV agent in performing the task can effectively promote the achievement of the task objective.

[0202] II. Determining the optimal trajectory for the UAV:

[0203] Once the model training converges, the route for the drone to perform the mission will be relatively fixed. However, this route may not be the optimal route. Therefore, in order to find an optimal route for performing the mission, it is necessary to analyze and process the trajectory of the drone's mission.

[0204] like Figure 4 As shown, the optimal trajectory generation system for unmanned aerial vehicles mainly includes the following modules:

[0205] Actor: This module is a trained policy network that has reached a certain convergence level. It executes the model inference function in this function and can provide specific action values ​​for each step in the UAV's mission execution. The policy network is the policy network in the optimal UAV agent mission execution model.

[0206] Drone Agent: This drone module is the same agent as the drone used during training. It saves all relevant information such as latitude and longitude, orientation angle, and key event logs to a file when the drone performs a task.

[0207] Simulator: The simulator used during training is the same third-party simulation system.

[0208] File storage: Stores the trajectory information of each mission performed by the drone.

[0209] Clustering: Clustering all the trajectory information stored in the file to calculate an optimal path for the UAV to perform the task.

[0210] Specifically, the clustering algorithm used in the clustering is K-Medoids center clustering, K-Medoids is also called PAM (Partitioning Around Medoids), which is a variant of K-Means clustering algorithm, aiming to improve the stability of clustering and tolerance to outliers. Unlike K-Means, K-Medoids selects actual points in the data set as cluster centers, i.e. latitude and longitude coordinate pairs in the UAV task execution trajectory as cluster centers, rather than calculating the mean of all points in the cluster as the cluster center. By taking latitude and longitude coordinate pairs in the UAV task execution trajectory as cluster centers, K-Medoids performs better when dealing with data sets with noise and outliers, and can obtain high-precision and high-accuracy task execution trajectories.

[0211] As shown in Figure 5 The processing flow of the optimal trajectory generation of the UAV agent is as follows:

[0212] Step 1: Load the UAV task document:

[0213] The defined UAV task document needs to be loaded in the simulator, and the initialization work of the animation simulation resource loading has been completed. The UAV task document here is the same as the UAV task document in the training.

[0214] Step 2: Load the decision network model:

[0215] Load the trained decision network model to provide action values for each step of the UAV agent's task execution. The decision network model is the policy network in the optimal UAV agent task execution model;

[0216] And initialize the following parameters:

[0217] Set the number of UAV task executions: max_eval_steps = 200;

[0218] Set the number of current task executions: eval_steps = 0;

[0219] Set an empty array: steps_list = []; the empty array is used to store the number of steps executed by the UAV from departure to starting the preset program or reaching the task termination state;

[0220] Step 3: UAV reasoning task:

[0221] The UAV reasoning task is a process of executing a series of task execution steps, and the execution logic of each step in the task execution is:

[0222] The UAV observation value is input into the decision network model to generate the current UAV action value;

[0223] The current UAV action value is input into the step function to update the UAV state, and the current observation value, reward value and task termination Boolean value of the UAV are obtained;

[0224] The latitude and longitude information and key event information of the UAV at the current step are recorded;

[0225] According to the task termination Boolean value returned by the step function, it is judged whether the current task execution reaches the end state, if the end state is reached, the current task execution is ended, otherwise the task is continued.

[0226] Step 4: Save the trajectory:

[0227] 1) Data saving:

[0228] The task execution steps performed at the task termination (task end) in the third step are stored in steps_list; at the same time, the UAV latitude and longitude coordinate pair corresponding to each step of the UAV in the task execution in the third step and key events and other information are saved in the designated execution task record file;

[0229] 2) Determine whether to end the UAV reasoning task:

[0230] The current task execution number is calculated, and the formula is: task execution number eval_steps += 1;

[0231] Determine whether the current task execution number reaches the maximum task execution number, and the judgment rule is:

[0232] According to whether the value of eval_steps is greater than max_eval_steps, it is judged whether to end the UAV reasoning task; if it is greater, the UAV reasoning task is ended, and the next step operation is entered, otherwise jump to the third step to continue the UAV reasoning task.

[0233] Specifically, the maximum task execution number is not the same as the maximum task execution step number;

[0234] The maximum task execution step number refers to the maximum number of steps that the UAV can advance when executing a task once, and the real-time step number will increase by one each time;

[0235] The maximum number of task executions max_eval_steps refers to the number of experiments of task execution; each time the task is executed, the UAV will perform many steps (here, many steps refer to the number of task execution steps, and each step will have latitude and longitude coordinates); in each task execution experiment, there are two possibilities of task success and failure, and each task execution experiment will form a path trajectory (the path trajectory is composed of many pairs of latitude and longitude coordinates), and the system saves max_eval_steps path trajectories of the UAV in total; through clustering, the optimal task execution trajectory information is found from the max_eval_steps path trajectories.

[0236] Step 5: Load the trajectory:

[0237] Load the trajectory data of 200 task executions saved in the task execution record file into the memory.

[0238] Step 6: Find the optimal trajectory through K-Medoids center clustering:

[0239] In the 200 task executions of the UAV, there are both cases of completing the target task and cases of not completing the task. Moreover, the latitude and longitude in each task execution trajectory are not the same, so it is necessary to use the K-Medoids center clustering method to analyze the trajectory data and find the optimal task execution path.

[0240] Each task execution trajectory is composed of several latitude and longitude coordinate pairs. According to the statistics and calculation of all task execution trajectories, the average value k of the number of latitude and longitude coordinate pairs in the task execution trajectory is calculated. Through the K-Medoids center clustering method, k sample points are selected as initial center points from all task execution trajectories, and the remaining samples are assigned to the cluster where the nearest center point is located. For each cluster, calculate the total distance of all points to other points, select the point with the smallest total distance as the new center point, and repeat the above process until the center point no longer changes or the maximum number of clustering is reached. The center point obtained through K-Medoids center clustering is the optimal trajectory.

[0241] The related parameters in K-Medoids center clustering include:

[0242] n_clusters number of cluster centers, here the average value of the number of steps in the 200 task executions stored in steps_list is used as the number of cluster centers, and the number of cluster centers is used as the number of steps of the UAV in the optimal trajectory;

[0243] metric = “euclidean” distance measurement method, here the Euclidean distance is used;

[0244] init = "heuristic" clustering center initialization method, here the "heuristic" selection has the minimum sum distance with other points n cluster points; the "heuristic" selection refers to using an experience rule-based, non-exhaustive intelligent search method to initialize the cluster center point;

[0245] max_iter = 500 the number of iterations of clustering data.

[0246] Step 7: Save the optimal trajectory:

[0247] The optimal task execution path found by the K-Medoids center clustering in the sixth step is saved for later use by the unmanned aerial vehicle agent in the task execution scenario, and thus the process of unmanned aerial vehicle task execution and optimal trajectory generation is completed.

[0248] The above unmanned aerial vehicle can observe the specific process of each step of the unmanned aerial vehicle in the simulator in real time. Taking the Mozi simulator as an example, the effect is shown:

[0249] Among them, Figure 6 and Figure 7 the red airplane-shaped icon in and the red semi-circular icon is the starting point of the unmanned aerial vehicle agent, and the blue icon in the large red circle is the obstacle 1-3, and the blue icon in the small red circle is the target obstacle; when the target obstacle enters the range of the unmanned aerial vehicle agent judging whether to start the preset program (i.e. when the target obstacle enters the red circle corresponding to the unmanned aerial vehicle agent), the unmanned aerial vehicle agent starts the preset program; the red circle corresponding to the obstacle indicates the range of the obstacle for the unmanned aerial vehicle agent to end the task, i.e. when the unmanned aerial vehicle agent enters the red circle corresponding to the obstacle, it directly ends the task; the white circle indicates the interference range of obstacles 1-3 to the unmanned aerial vehicle agent when executing the task (such as affecting the observation value received by the unmanned aerial vehicle).

[0250] As shown in Figure 6 , the performance of the unmanned aerial vehicle agent without training when executing the task is shown. Figure 6 (a) shows that the distance between the unmanned aerial vehicle agent and the target obstacle is far; and Figure 6 (b) shows that the agent does not move towards the target obstacle. From Figure 6 , it can be found that the unmanned aerial vehicle agent without training has difficulty in making effective task execution decisions. The direction of the sharp head of the unmanned aerial vehicle agent is the direction of flight of the unmanned aerial vehicle agent.

[0251] As shown in Figure 7 , the schematic diagram of the unmanned aerial vehicle executing the task after the above training is shown. Figure 7As can be seen, the trained UAV can perform the task according to the optimal trajectory, wherein:

[0252] Figure 7 (a) indicates that the UAV is flying towards the target according to the optimal trajectory; the flight direction of the UAV agent is towards the target obstacle.

[0253] Figure 7 (b) indicates that the target is within the preset radius, and the UAV agent initiates the preset program; in Figure 7 In (b), when the target obstacle enters the red circle detection range of the UAV agent, the UAV agent immediately starts the preset program. The UAV agent first marks the target obstacle with a yellow star, and then moves towards the target obstacle from the red arrow of the UAV. Finally, the red arrow reaches the position of the yellow star, and the UAV agent completes the removal of the target obstacle.

[0254] Figure 7 (c) indicates that the UAV agent returns after the completion of the preset program. Figure 7 (b) and Figure 7 (c) compared, Figure 6 In (c), the target obstacle marked by the small red circle no longer exists, indicating that the UAV agent has removed the target obstacle, i.e., the UAV agent has completed the specified task target.

[0255] As Figure 7 and ​ indicate, by combining the TD3 algorithm and the K-Medoids center clustering algorithm, the optimal trajectory can be determined, so that the UAV agent can more efficiently and accurately complete the specified task target.

[0256] It should be noted that the above specific embodiments can enable those skilled in the art to more fully understand the present invention, but in no way limit the present invention. Therefore, although the present invention has been described in detail with reference to the drawings and examples, those skilled in the art should understand that modifications or equivalent replacements can still be made to the present invention, in short, all technical solutions and improvements that do not deviate from the spirit and scope of the present invention should be covered within the protection scope of the patent of the present invention.

Claims

1. A method for task execution and optimal trajectory generation of a UAV intelligent agent based on deep reinforcement learning, characterized in that, S1: Set and load the UAV mission document: Set the UAV mission document according to the mission scenario and mission objectives of the UAV agent, and load the UAV mission document in the simulator; S2: Construct an observation space based on the real-time characteristics of the UAV agent and the latitude, longitude coordinates and orientation angles of all obstacles on the ground; The action space is set according to the range of values ​​of the UAV agent's actions; the UAV agent's task execution functions are set according to the UAV task document, including: reward function, termination function, reset function and step function; S3: Constructing a UAV agent task execution model: Constructing a UAV agent task execution model based on a dual-delay deep deterministic policy gradient algorithm, including a policy network and a value function network; S4: Training the UAV agent: Input the UAV observation values ​​in the observation space into the policy network to obtain the UAV action values ​​in the action space; input the UAV action values ​​into the UAV agent task execution function to obtain UAV state feedback data; input the UAV state feedback data into the value function network to calculate the loss function value, and perform gradient updates on the UAV agent task execution model based on the loss function value to obtain the optimal UAV agent task execution model; S5: Optimal trajectory generation of UAV agent: Perform UAV inference tasks using the optimal UAV agent task execution model and save all trajectory information during the UAV inference task; cluster all trajectory information using the K-Medoids centroid clustering algorithm to obtain the optimal trajectory.

2. A method for task execution and optimal trajectory generation of a UAV agent based on deep reinforcement learning, as described in claim 1, characterized in that, The action space limits the range of drone action values ​​output by the policy network; the observation space limits the range of information the drone can receive during mission execution, i.e., limits the drone's observation values; if the corresponding value of the drone's observation value is not within the set range of the observation space, the mission is terminated.

3. A method for task execution and optimal trajectory generation of a UAV agent based on deep reinforcement learning, as described in claim 1, characterized in that... The reward function is set based on whether the drone agent's task execution actions are conducive to achieving the drone agent's task objective; if the drone's task execution actions are conducive to achieving the drone agent's task objective, a positive number is returned, otherwise a negative number or zero is returned.

4. A method for task execution and optimal trajectory generation of a UAV agent based on deep reinforcement learning, as described in claim 1, characterized in that, The UAV observation values ​​include, but are not limited to: the UAV's real-time latitude and longitude coordinates, orientation angle, and the latitude and longitude coordinates and orientation of ground obstacles detected by the UAV; the UAV action values ​​include: the UAV's latitude and longitude coordinates, flight direction, speed, and time to move forward one step.

5. A method for task execution and optimal trajectory generation of a UAV agent based on deep reinforcement learning, as described in claim 1, characterized in that, The method for training the UAV agent in step S4 is as follows: A1: Input the UAV observation values ​​into the policy network to obtain the UAV action values; A2: Input the UAV action value into the step function to judge the action and obtain the UAV status feedback data; The UAV status feedback data includes: new UAV observations, reward values, and mission termination Boolean values; A3: Input the new observation value of the UAV and the reward value into the value function network to obtain the value function estimate; calculate the loss function value based on the value function estimate; A4: Calculate the gradient of the UAV agent task execution model based on the loss function value; update the value function network based on the gradient; when the number of gradient updates of the value function network reaches a preset number, update the policy network and reset the number of updates of the value function network. A5: Determine whether to end training based on the task termination Boolean value: If the task termination Boolean value is 0, determine whether the reward value is greater than the optimal reward value. If the reward value is greater than the optimal reward value, the current model parameters are used as the parameters of the optimal UAV agent task execution model, and the optimal reward value is equal to the current reward value; if the task termination Boolean value is 1, end training, output the optimal UAV agent task execution model, and reset and initialize all parameters in the UAV agent task execution function through the reset function.

6. The method for task execution and optimal trajectory generation of a UAV agent based on deep reinforcement learning as described in claim 5, wherein the steps in the step function are as follows: B1: Communicate with the simulator; the simulator updates the UAV observation values ​​based on the UAV action values ​​to obtain new UAV observation values; B2: Calculate the distance between the latitude and longitude coordinates of the new observation value of the UAV and the target; if the distance is less than the preset radius of the UAV, the UAV starts the preset program; B3: Calculate the new observation value of the UAV and the reward value generated by whether the preset program is started in B2 through the reward function, and increment the task execution step count by one; B4: Call the termination function to determine whether the task executed in this round has reached the end state, and get the task termination boolean value; output the new observation value of the drone, the reward value and the task termination boolean value.

7. A method for task execution and optimal trajectory generation of a UAV intelligent agent based on deep reinforcement learning, as described in claim 1, characterized in that, The loss function includes: policy gradient loss, value function loss, and KL divergence, as shown in the formula: （1） （2） （3） （4） in, Represents the policy gradient loss; The parameters of the policy network are represented by VF; the value function loss is represented by KL; and the KL divergence is represented by KL divergence. It is the i-th value function network; Both 'a' and 'b' represent the drone action values ​​output by the policy network; D is the set of drone observations; s is the current drone observation value; E[] represents the mathematical expectation; b represents the target value; This represents the estimated value of the value function output by the i-th value function network. This represents the true probability distribution of y given condition x; Let y represent the approximate probability distribution of y given condition x; x represents the drone observation value; and y represents the drone action value output by the decision network.

8. A method for task execution and optimal trajectory generation of a UAV intelligent agent based on deep reinforcement learning, as described in claim 1, characterized in that, The method for performing model inference on the optimal UAV agent task execution model is as follows: C1: Input the UAV observations into the policy network of the optimal UAV agent task execution model to obtain the UAV action values; C2: Input the UAV action value into the step function to obtain the new observation value, reward value and mission termination Boolean value of the UAV; C3: If the task termination Boolean value is 1, the drone task ends, and the current number of steps and the trajectory of the task are saved, and then jump to C4; otherwise, jump to C1. C4: Increment the number of task executions by one to get the current number of task executions; determine whether the current number of task executions has reached the maximum number of task executions. If the maximum number of task executions is reached, the inference process ends; if the maximum number of task executions is not reached, the parameters are reset using the reset function, and the process jumps to C1 to execute the next task.

9. A method for task execution and optimal trajectory generation of a UAV agent based on deep reinforcement learning, as described in claim 1, characterized in that, The method for obtaining the optimal trajectory through K-Medoids centroid clustering described in step S5 is as follows: D1: Count the number of latitude and longitude coordinate pairs in all trajectory information during the UAV inference mission, and calculate the average number K of latitude and longitude coordinate pairs contained in the trajectory of the mission execution. D2: Select K sample points as center points from all the trajectory information using the K-Medoids centroid clustering method, and assign the remaining sample points to the clusters containing the nearest center points; the sample points are latitude and longitude coordinate pairs. D3: Calculate the sum of distances from each sample point in each cluster to other sample points, and select the sample point with the smallest sum of distances as the new center point; The remaining sample points are assigned to the cluster containing the nearest new center point; D4: If the center point does not change or the maximum number of clustering steps is reached, then the current center point is taken as the optimal trajectory; otherwise, repeat the D3 operation.

Images