A UAV path planning method based on pulse distributed reinforcement learning

Abstract

The present invention discloses a UAV path planning method based on pulse distributed reinforcement learning, which belongs to the field of machine intelligent decision-making and control technology. The method of the present invention includes constructing a distributed reinforcement learning neural network model based on a pulse neural network. The model predicts the reward distribution of actions rather than the traditional reward expectation during the UAV path planning process, and captures more effective information based on the reward distribution to more stably select high-quality UAV control actions when facing complex scenarios. The model architecture based on the pulse neural network of the present invention reduces the calculation frequency, and the method based on distributed reinforcement learning makes up for the problem of the single focus and loss of valuable information of the existing UAV path planning method, enhances the ability of the present invention to deal with uncertainty and randomness, and expands the applicability of the method of the present invention when facing complex and changeable real UAV path planning tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine intelligent decision-making and control, and in particular relates to a UAV path planning method based on pulse distributed reinforcement learning. Background Art

[0002] The UAV path planning task is to find a safe, dynamically feasible, and optimal path for the UAV to fly from the starting point to the target point under a certain environment through a certain algorithm and control method. At present, the algorithms for the UAV path planning task can be divided into four categories: graph search-based methods, random sampling-based methods, optimization problem solving-based methods, and reinforcement learning-based methods.

[0003] The graph search-based method regards the UAV path planning task as a graph search problem in mathematics, constructs the map of the UAV workspace as a graph, regards the starting point and target point of the UAV as nodes in the graph, and regards the flight path of the UAV as edges in the graph. After the construction is completed, the algorithm searches by continuously expanding the neighbor nodes of the current node until the target location is found.

[0004] The random sampling based method can be regarded as an improvement on the graph search based method. This method does not need to traverse the entire UAV workspace, but only uniformly randomly samples the state space to construct a connected graph and perform collision detection. The sampling ends when both the starting point and the target point appear in the graph, which improves the planning efficiency.

[0005] The method based on solving optimization problems regards the UAV path planning task as a nonlinear constrained optimization problem. After determining the optimization goal and constraints of the task, it uses the optimization algorithm to solve it.

[0006] The reinforcement learning method is a new UAV path planning method that has emerged in recent years with the development of artificial intelligence technology. The reinforcement learning method is based on the Markov decision process, which treats the control problem of the UAV as a Markov decision process. By modeling the process and constructing a neural network for training, the optimal trajectory planning strategy is finally obtained.

[0007] However, the problem of drone planning in practical applications is becoming increasingly complex, and multiple constraints and various goals such as dynamics, energy consumption, and air traffic control need to be considered, and these constraints are often difficult to describe with clear mathematical expressions. For methods based on graph search and random sampling, it is difficult to apply them to today's actual task requirements because they focus on path planning and ignore constraints; for methods based on solving optimization problems, the complexity of numerical solutions is extremely high and it is difficult to obtain the optimal solution, which requires a huge amount of calculation; for methods based on reinforcement learning, training and executing neural network controllers also require a lot of computing resources, which may not be available in some real-life scenarios and practical problems, which is a problem for real-time control tasks.

[0008] Looking at the above methods, the common problem of existing UAV path planning algorithms that can be applied to actual tasks is that they require a lot of computing resources. This is due to the complexity of the model on the one hand, and on the other hand, it is due to the fact that most of the existing reinforcement learning methods are based on artificial neural networks. Combining pulse neural networks with reinforcement learning, pulse reinforcement learning provides a low-energy solution for UAV path planning tasks. Its sparse activation and event-driven characteristics greatly reduce the frequency of calculations. However, due to the loss of a large amount of reward distribution information, the pulse reinforcement learning method lacks the ability to deal with uncertainty and randomness, and still has defects when facing real-world UAV path planning tasks. In addition, most of the existing pulse reinforcement learning methods are completed by converting the trained artificial neural network into a pulse neural network, which loses the energy consumption advantage of the pulse neural network to a certain extent, and is very dependent on the training of the artificial neural network. When the artificial neural network training is imperfect or errors accumulate during the conversion process, it will have a greater impact on the final result. Summary of the invention

[0009] The purpose of the present invention is to propose a UAV path planning method based on pulse distributed reinforcement learning, aiming to solve the problems of high computing resource requirements of existing methods, poor model generalization ability, weak model ability to deal with uncertainty and randomness, and high dependence of existing pulse reinforcement learning models on pre-trained artificial neural networks.

[0010] The technical solution adopted by the present invention is:

[0011] A UAV path planning method based on pulse distributed reinforcement learning, the method comprising the following steps:

[0012] Step 1: Select the training data for UAV path planning and build a training set;

[0013] Using Markov process Describe the UAV path planning process. The training data of the training set uses Markov sequence to describe the path of the UAV;

[0014] in, Represents the state space, which is used to characterize all possible states of the drone;

[0015] Represents the action space, which is used to characterize all the control actions that the drone can take;

[0016] Represents the state transition probability, which is used to characterize the state Select Control Action Transfer to a certain state probability;

[0017] Represents the reward function, which is used to characterize the state Next select action The immediate rewards received;

[0018] Represents the decay coefficient, which is used to adjust the reward weight, discounting future rewards to focus more on recent rewards;

[0019] Step 2: construct a feature extraction network based on a pulse neural network to extract features from the input drone state. Perform feature extraction to output the state State Embedding ,in, is the mean value of the output of the spiking neural network in the entire simulation time window;

[0020] Step 3: Build a pulse distributed reinforcement learning network whose input is the state embedding , based on the quantile distribution, the reward distribution that can be obtained by taking the control action is estimated, and the action value is characterized based on the expected reward corresponding to the reward distribution to select the optimal control action;

[0021] Step 4: construct a loss function to measure the gap between the expected return predicted by the pulse distributed reinforcement learning network and the actual return;

[0022] Step 5: Train the feature extraction network and the pulse distributed reinforcement learning network based on the training set, optimize the network parameters with the purpose of minimizing the constructed loss function, and stop when the preset training end condition is reached. Based on the trained feature extraction network and the pulse distributed reinforcement learning network, a UAV path planning model for autonomous path planning of the UAV is obtained.

[0023] Furthermore, in step 1, each state of the state space is described by establishing a three-dimensional coordinate system for the operating space of the UAV.

[0024] Furthermore, the network structure of the feature extraction network includes a stacked structure consisting of alternating convolutional layers and pulse neuron layers in sequence, and a fully connected layer is connected after the stacked structure as the output layer of the feature extraction network.

[0025] Furthermore, the control strategy of the pulse distributed reinforcement learning network to select the optimal control action is:

[0026]

[0027]

[0028] in, represents the control strategy of the spiking distributed reinforcement learning network, From the action space Any control action in Indicates the current state action pair Expected return, Express expectations, Indicates the current state action pair The return distribution, expected return By the attenuation coefficient Adjust the expected cumulative rewards in the future.

[0029] Furthermore, the return distribution The method of obtaining is: select a certain number of quantiles and output the quantile values ​​corresponding to the quantiles , establish the quantile distribution To estimate the return distribution .

[0030] Furthermore, the network structure of the pulse distributed reinforcement learning network is any one of the following two structures:

[0031] Structure 1: includes a first stacking structure, a second stacking structure and a fully connected layer in sequence, wherein the first stacking structure is composed of alternating convolutional layers and spiking neuron layers, and the second stacking structure is composed of alternating fully connected layers and spiking neuron layers;

[0032] Structure 2: includes a second stacking structure and a fully connected layer in sequence.

[0033] Furthermore, the loss function is the Huber quantile loss function.

[0034] Furthermore, in step 5, the training end conditions include but are not limited to: the number of training times reaches a preset upper limit, the loss function value converges, and the path planning capability reaches the expected target.

[0035] Furthermore, the path planning ability is measured by the probability of the UAV reaching the destination and / or the time step length.

[0036] Furthermore, the UAV path planning method based on pulse distributed reinforcement learning of the present invention also includes step 6, a path planning execution step based on the UAV path planning model obtained in step 5, specifically including the following sub-steps:

[0037] Step 6-1, using Markov process to describe the current UAV path planning task;

[0038] Step 6-2, input the current state of the drone to be controlled into the drone path planning model, and obtain the optimal control action under the current state based on the output of the model; that is, extract effective information (the state embedding corresponding to the current state) based on the feature extraction network of the model, and then input it into the pulse distributed reinforcement learning network to select and output the optimal control action through the control strategy;

[0039] Step 6-3, the controlled UAV executes the optimal control action output by the model, and obtains the state at the next moment after the execution is completed;

[0040] Step 6-4, return to step 6-1 with the state obtained in step 6-3 as the current state;

[0041] Repeat steps 6-2 to 6-4 until the UAV path planning task is completed.

[0042] The technical solution provided by the present invention brings at least the following beneficial effects:

[0043] The present invention proposes a UAV path planning method based on pulse distributed reinforcement learning by combining distributed reinforcement learning with pulse neural networks. The method reduces the calculation frequency based on the model architecture of the pulse neural network, improves the deficiency of the existing UAV path planning method that requires a large amount of computing resources, and provides a low-energy solution for the UAV path planning task; the method based on distributed reinforcement learning makes up for the problems of the existing UAV path planning method that focuses on a single point and loses valuable information, enhances the ability of the present invention to deal with uncertainty and randomness, and expands the applicability of the present invention in the face of complex and changeable real UAV path planning tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1A processing diagram of a UAV path planning method based on pulse distributed reinforcement learning provided by an embodiment of the present invention;

[0046] Figure 2 A control flow chart of a UAV path planning method based on pulse distributed reinforcement learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] To make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described in detail and completely in conjunction with the drawings in the implementation of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present invention.

[0048] The embodiment of the present invention provides a UAV path planning method based on pulse distributed reinforcement learning, such as Figure 1 As shown, it includes the following steps:

[0049] Step S1, selecting training data for UAV path planning and constructing a training set;

[0050] In this step, the Markov process is used Describe the UAV path planning process, and describe the path of the UAV through the Markov sequence. represents the state space, i.e., all possible states of the drone; represents the action space, i.e., all the control actions that the drone can take; represents the state transition probability, which is expressed as , indicating that in the state Select Control Action Then transfer to a certain state The probability of Represents the reward function, which is represented by , that is, in the state Select Control Action The instant rewards you receive, Describes the corresponding state-action pairs; Represents the decay coefficient, which is used to adjust the reward weight and discount future rewards to make the system pay more attention to recent rewards.

[0051] In one embodiment, the state space Indicates the possible location of the drone, and describes it by establishing a three-dimensional coordinate system for the drone's workspace; action space Indicates the various control actions that the drone can take, such as forward, backward, left turn, right turn, etc.; state transition probability Indicates the current state Take control action , thus entering the next state The probability of reward function Provide instant rewards to the network model based on the state of the drone and the control actions taken, such as rewards for keeping the drone flying normally through control actions , reward for causing the drone to collide by controlling the action , Reward for controlling the drone to avoid the aviation control area .

[0052] In one embodiment, the training data of this step can be derived from the drone control process completed by human experts, that is, from the optimal flight path prepared by human experts in advance; when human expert data is lacking, it can also be derived from the drone control process completed by other high-performance drone path planning algorithms. For example, a human expert prepares an optimal flight path for a drone in advance and uses it as a sequence. Indicates that, where the subscript is the time point identifier, for each moment in the sequence , This is the state of the drone at that moment, which can be obtained by the camera or positioning system on the drone. is the control action taken at that moment, is the reward obtained at that moment, and the constructed training data set consists of multiple such sequences.

[0053] Step S2: construct a feature extraction network based on a pulse neural network to extract the state of the input drone. Perform feature extraction to output the state State Embedding That is, the UAV status is preprocessed based on the constructed feature extraction network.

[0054] In this step, the feature extraction network is based on a spiking neural network. is the mean value of the pulse neural network output in the entire simulation time window. The function of the feature extraction network is to extract local and global features of the environment from the image, including edges, textures, shapes, structures, etc. For example, the drone’s surrounding environment can be observed through the camera and other equipment carried by the drone to determine the drone’s current state. , as the input of the feature extraction network; the feature extraction network focuses on obstacles, other flying objects, height from the ground and other information in the captured image and finally outputs a vector as a representation of the state, i.e., state embedding .

[0055] In one embodiment, the parameters of the feature extraction network in this step are randomly initialized, and then can be updated through gradient back propagation based on a preset loss function during training.

[0056] In one embodiment, the feature extraction network of this step includes a stacked structure consisting of alternating convolutional layers and spiking neuron layers, and a fully connected layer is connected after the stacked structure as the output layer of the feature extraction network. That is, the feature extraction network includes K (K≥2) convolutional layers, a fully connected layer, and K spiking neuron layers, each convolutional layer is connected to a spiking neuron layer, and then a fully connected layer is connected after the last spiking neuron layer. These functional layers enable the feature extraction network to capture the features of the environment from the input image to better describe the environment.

[0057] Step S3, establishing a pulse distributed reinforcement learning network, which describes the rewards that can be obtained from the control action based on the quantile distribution to select the optimal control action;

[0058] In this step, the input of the pulse distributed reinforcement learning network is the state embedding , based on the quantile distribution, the reward distribution that can be obtained by taking the control action is estimated, and the action value is characterized based on the expected reward corresponding to the reward distribution to select the optimal control action;

[0059] In one embodiment, the expression of the pulse distributed reinforcement learning network established in step S3 is:

[0060]

[0061]

[0062] in, represents the control strategy of the pulse distributed reinforcement learning network, that is, in a certain state When , select the control action to be taken, Control strategy for short ; From the action space Any control action in is the action value of the current state-action pair, also known as the reward, which is represented by The expected cumulative reward in the future is adjusted, that is, the drone is in the current state Take control action , future discounts and rewards, Action Value ; express expectations; represents the reward distribution of the current state-action pair, that is, the reward is regarded as a random variable, Return Distribution .

[0063] In the embodiment of the present invention, the pulse distributed reinforcement learning network maximizes the action value To select the most appropriate control action, due to the current state Take control action The returns are not always the same, so the expected return distribution is used as For example, if you control a drone to turn left, there is a 90% probability that the drone will fly normally, and there is also a 10% probability that it will collide, so the reward is not fixed. By treating the reward as a random variable and modeling it, the network model has a stronger ability to handle randomness and uncertainty. The input of the pulse distributed reinforcement learning network is the state embedding output of the feature extraction network in the previous step , the output is the predicted Value and predicted optimal control action .

[0064] In one embodiment, the reward distribution is estimated by quantile distribution to calculate the action value and select the optimal control action.

[0065] Since the real reward distribution of the control task is unknown in real scenarios, a pulse neural network is constructed according to the expression of the pulse distributed reinforcement learning network, a certain number of quantiles are selected, the quantile values ​​corresponding to the quantiles are output, and the quantile distribution is established. To estimate the return distribution ,in Represents the quantile value, that is, sampling state-action pairs in the training set and establishing a quantile distribution for them, estimating the return corresponding to the state-action pair by the established quantile distribution, and realizing the quantile value pair return distribution Estimate and represent. At the same time, through the quantile distribution right Calculate and output The largest optimal control action .

[0066] Preferably, the quantile distribution The expression is:

[0067]

[0068] in:

[0069]

[0070]

[0071] in, is a hyperparameter, indicating the number of selected quantiles, is the number of the quantile point. In this embodiment, ; is the impact function; It is the distribution function, that is, the integral of the probability density function, which describes the probability distribution of a random variable; Indicates The quantile value of the state-action pair, For the return distribution The inverse function of the distribution function is output by the pulse neural network, let represents the last layer of the spiking neural network (i.e., the last spiking neuron layer in the spiking distributed reinforcement learning network), Represents the spiking neural network The parameters of the layers, in this embodiment, the pulse neural network includes 4 layers, so . represents the simulation time window length of the spiking neural network. In this embodiment, let . Indicates the second-to-last layer of the spiking neural network at simulation time Output at the moment; is the quantile, and the quantile is chosen by OK, for , The probability density function of the return distribution equal parts, , that is, in In the probability density function of , the area divided between every two quantiles is For simplicity, let . Quantile distribution The expression means that each quantile The probability of occurrence is approximately , each quantile Corresponding to a quantile , indicating that in the return distribution middle .

[0072] In one embodiment, the network structure of the pulse distributed reinforcement learning network is: a first stacking structure is formed by alternating convolutional layers and pulse neuron layers, a second stacking structure is formed by alternating fully connected layers and pulse neuron layers, the output of the first stacking structure is used as the input of the second stacking structure, and a fully connected layer is connected after the second stacking structure as the output layer of the pulse distributed reinforcement learning network. Alternatively, the network structure is set to: a fully connected layer is directly connected after the second stacking structure as the output layer of the pulse distributed reinforcement learning network. That is, in this embodiment, the pulse distributed reinforcement learning network includes convolutional layer, Layer fully connected layer and Each convolutional layer is followed by a layer of spiking neurons. Except for the last fully connected layer, each fully connected layer is followed by a layer of spiking neurons.

[0073] Step S4, constructing a loss function to measure the gap between the reward calculated by the quantile value estimated by the pulse distributed reinforcement learning network and the actual reward, so as to optimize and update the network parameters by minimizing the loss function during the training process.

[0074] In one embodiment, the loss function constructed in step S4 is specifically a Huber quantile loss function.

[0075] The Huber quantile loss function is a hyperparameter In this embodiment, control In the UAV path planning problem, suppose that at a certain moment The drone status and control actions are: , the corresponding return distribution is , the next moment The drone status is , the corresponding return distribution is , then the Huber quantile loss is expressed as follows:

[0076] ;

[0077] in:

[0078] ;

[0079] ;

[0080] in, is the auxiliary amount, Indicated in Instant rewards at all times, Represents the attenuation coefficient, which is used to adjust the reward weight. Indicates that the control strategy The optimal control action is chosen. , They are the quantile values ​​corresponding to the system state-action pair at the current moment and the quantile values ​​corresponding to the system state-action pair at the next moment, respectively. and the next moment return distribution Establish quantile distribution to represent and calculate, represents the Huber loss function, Represents the Huber quantile loss function.

[0081] Step S5, using the training set to train the feature extraction network and the pulse distributed reinforcement learning network, determine the hyperparameters, which include the maximum number of model iterations, the model convergence threshold, and the model trade-off parameters; stop training until the prediction performance reaches the preset value, and obtain the UAV path planning model based on the trained feature extraction network and the pulse distributed reinforcement learning network to implement autonomous path planning of the UAV. Among them, the model trade-off parameters refer to the learning rate of the neural network and the relevant parameters of the pulse neuron, such as the simulation time window length of the pulse neural network, the membrane potential threshold of the pulse neuron, the membrane time constant, the initial membrane potential, the learning rate of the feature extraction network and the pulse distributed reinforcement learning network, etc.

[0082] In one embodiment, the training process of the feature extraction network and the pulse distributed reinforcement learning network specifically includes the following steps:

[0083] (1) Obtain training data from human experts or other high-performance UAV path planning algorithms to build a training dataset;

[0084] (2) Set the number of quantiles N, the spiking neural network simulation time window T, and the Huber quantile loss parameter, hyperparameters ;

[0085] (3) Initialize the network parameters of the feature extraction network and the pulse distributed reinforcement learning network;

[0086] (4) Obtaining data from the training set , the current moment and the next moment Status , Input the feature extraction network respectively, and get Moment and State Embedding , ;

[0087] (5) Embedding the state , Input pulse distributed reinforcement learning network to obtain the corresponding quantile value and quantile distribution ;

[0088] (6) Pulse Distributed Reinforcement Learning Network Calculation of Quantile Distribution The expectation of is taken as the expected return, and the expected returns of all state-action pairs are output;

[0089] (7) The pulse distributed reinforcement learning network selects the control action that maximizes the expected return as the optimal control action. ;

[0090] (8) For state-action pairs , Calculate the corresponding quantile value and obtain the Huber quantile loss based on the quantile position;

[0091] (9) Backward gradient propagation based on Huber quantile loss to update the network parameters of the feature extraction network and the pulse distributed reinforcement learning network;

[0092] (10) Whether the preset training end condition is met, if so, then end; otherwise, return to step (4); wherein the training end condition is set as the network model prediction performance reaching a preset value, which can be set as: the number of training times reaches a preset upper limit, the Huber quantile loss converges, and the path planning ability based on the metric detects whether the prediction performance reaches a preset value, etc. Preferably, the path planning ability can be measured by the probability of the drone reaching the destination and / or the time step length.

[0093] In one embodiment, the process of implementing the UAV path planning based on the trained UAV path planning model is as follows: Figure 2 Specifically, it includes:

[0094] Step 1: For the drone to be controlled, the Markov process is used to describe the system according to the possible states of the drone and the control actions that can be selected; that is, the Markov process is used to describe the drone path planning task, and the camera and other equipment carried by the drone observe the drone's surrounding environment to determine the drone's current state. ;

[0095] Step 2: Set the current state of the drone to be controlled Input the drone path planning model, and use the model's feature extraction network to focus on obstacles, other flying objects, height from the ground, and other information in the captured image, and output the state embedding ;

[0096] Step 3: Embed the state The pulse distributed reinforcement learning network of the input model is distributed by quantiles Calculating Action Value , through the control strategy Select the largest The action corresponding to the value is the optimal control action And output;

[0097] Step 4: For the drone to be controlled, execute the selected optimal control action Control and execute optimal control actions The state of the drone changes, and the state of the drone to be controlled at the next moment is obtained ;

[0098] By status As the current state, repeat steps 2, 3, and 4 until the drone path planning process is completed.

[0099] In one embodiment, in the above step 3, the control strategy The selection process is: for any input state embedding , respectively for all possible control actions , predicted by the model’s spiking distributed reinforcement learning network The corresponding return distribution quantile, and then the return distribution is described by the quantile ,based on The expected return is obtained by the mathematical expectation, and then the control action corresponding to the maximum expected return is selected using the expected return as the metric value. As the optimal control action , in order to maximize the expected return.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0101] The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the creative concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A UAV path planning method based on pulse distributed reinforcement learning, characterized in that: The following steps are involved: Step 1: Select the training data for UAV path planning and build a training set; Using Markov process Describe the UAV path planning process. The training data of the training set uses Markov sequence to describe the path of the UAV; in, Represents the state space, which is used to characterize all possible states of the drone; Represents the action space, which is used to characterize all the control actions that the drone can take; Represents the state transition probability, which is used to characterize the state Select Control Action Transfer to a certain state probability; Represents the reward function, which is used to characterize the state Next select action The immediate rewards received; Represents the decay coefficient, which is used to adjust the reward weight; Step 2: Construct a feature extraction network based on a pulse neural network. The feature extraction network is used to extract the state of the input Perform feature extraction to output the state State Embedding ,in, is the mean value of the output of the spiking neural network in the entire simulation time window; Step 3: Build a pulse distributed reinforcement learning network whose input is the state embedding , based on the quantile distribution, the reward distribution that can be obtained by taking the control action is estimated, and the action value is characterized based on the expected reward corresponding to the reward distribution to select the optimal control action; Step 4: construct a loss function to measure the gap between the expected return predicted by the pulse distributed reinforcement learning network and the actual return; Step 5: Train the feature extraction network and the pulse distributed reinforcement learning network based on the training set, optimize the network parameters with the purpose of minimizing the constructed loss function, stop when the preset training end condition is reached, and obtain the UAV path planning model for autonomous path planning of the UAV based on the trained feature extraction network and the pulse distributed reinforcement learning network; Step 6, performing path planning based on the UAV path planning model obtained in step 5, which specifically includes the following sub-steps: Step 6-1, using Markov process to describe the current UAV path planning task; Step 6-2, input the current state of the drone to be controlled into the drone path planning model, and obtain the optimal control action under the current state based on the output of the model; Step 6-3, the controlled UAV executes the optimal control action output by the model, and obtains the state at the next moment after the execution is completed; Step 6-4, return to step 6-1 with the state obtained in step 6-3 as the current state; Repeat steps 6-2 to 6-4 until the UAV path planning task is completed.

2. A UAV path planning method based on pulse distributed reinforcement learning as claimed in claim 1, characterized in that: In step 1, each state of the state space is described by establishing a three-dimensional coordinate system for the UAV's operating space.

3. A UAV path planning method based on pulse distributed reinforcement learning as claimed in claim 1, characterized in that: The network structure of the feature extraction network includes a stacked structure consisting of alternating convolutional layers and pulse neuron layers, and a fully connected layer is connected after the stacked structure as the output layer of the feature extraction network.

4. A UAV path planning method based on pulse distributed reinforcement learning as claimed in claim 1, characterized in that: The control strategy of the impulse distributed reinforcement learning network to select the optimal control action is: ； ； in, represents the control strategy of the spiking distributed reinforcement learning network, From the action space Any control action in Indicates the current state action pair Expected return, Express expectations, Indicates the current state action pair The return distribution, expected return By the attenuation coefficient Adjust the expected cumulative rewards in the future.

5. A UAV path planning method based on pulse distributed reinforcement learning as claimed in claim 4, characterized in that: Return Distribution The method of obtaining is: select a certain number of quantiles and output the quantile values ​​corresponding to the quantiles , establish the quantile distribution To estimate the return distribution .

6. A method for UAV path planning based on pulse distributed reinforcement learning as claimed in claim 1, characterized in that: The network structure of the pulse distributed reinforcement learning network is structure 1 or structure 2; Structure 1 includes: a first stacking structure, a second stacking structure and a fully connected layer in sequence; Structure 2 includes: a second stacking structure and a fully connected layer in sequence; The first stacked structure is composed of alternating convolutional layers and spiking neuron layers, and the second stacked structure is composed of alternating fully connected layers and spiking neuron layers.

7. A method for UAV path planning based on pulse distributed reinforcement learning as claimed in claim 1, characterized in that: The loss function is the Huber quantile loss function.

8. The method for UAV path planning based on pulse distributed reinforcement learning as claimed in claim 1, characterized in that: In step 5, the training end conditions are: the number of training times reaches a preset upper limit, the loss function value converges and / or the path planning capability reaches the expected target.

9. A method for UAV path planning based on pulse distributed reinforcement learning as claimed in claim 8, characterized in that: The path planning ability is measured by the probability of the UAV reaching the destination and / or the time step length.

Images