Unmanned aerial vehicle path planning method based on pulse distributed reinforcement learning
By adopting pulse distributed reinforcement learning method in drone path planning, combining Markov process and pulse neural network, the problems of high computing resources and poor generalization capabilities are solved, and low energy consumption and efficient path planning are achieved. Enhanced applicability to complex environments.
Patent Information
- Application Number
- CN202411422856.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-12
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-10-12
AI Technical Summary
The existing UAV path planning algorithm has high demand for computing resources, poor generalization capabilities of the model, and it is difficult to cope with uncertainty and randomness in complex environments. It depends on artificial neural network training and is easily affected by training errors.
Using a method based on pulsed distributed reinforcement learning, the UAV path planning is described through the Markov process, and the pulsed neural network feature extraction network and pulsed distributed reinforcement learning network are constructed, and the return distribution is estimated using quantile distribution to select the optimal control. Action, reduce the calculation frequency and enhance the processing power of uncertainty and randomness.
Low-energy consumption drone path planning is realized, the applicability to complex environmental tasks and the efficiency of path planning is improved, the demand for computing resources is reduced, and the generalization ability of the model is enhanced.
Smart Images

Figure CN118913295A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of machine intelligent decision-making and control technology, and in particular relates to a UAV path planning method based on pulse distributed reinforcement learning. Background Art
[0002] The UAV path planning task involves finding a safe, dynamically feasible, and optimal path for a given starting and destination point in a specific environment using algorithms and control methods. Currently, algorithms for UAV path planning tasks can be categorized into four main categories: graph search-based methods, random sampling-based methods, optimization problem-solving-based methods, and reinforcement learning-based methods.
[0003] Graph search-based methods treat the drone path planning task as a mathematical graph search problem. They construct a map of the drone's workspace as a graph, treating the drone's starting and destination points as nodes in the graph, and the drone's flight path as edges. Once constructed, the algorithm searches by continuously expanding the neighboring nodes of the current node until the target location is found.
[0004] The random sampling-based method can be seen as an improvement on the graph search-based method. This method does not need to traverse the entire UAV workspace, but only uniformly randomly samples the state space to construct a connected graph and perform collision detection. Sampling ends when both the starting point and the target point appear in the graph, which improves planning efficiency.
[0005] The method based on solving optimization problems regards the UAV path planning task as a nonlinear constrained optimization problem. After determining the optimization objectives and constraints of the task, it uses the optimization algorithm to solve it.
[0006] Reinforcement learning is a new approach to drone path planning that has emerged in recent years with the development of artificial intelligence. Based on the Markov decision process, reinforcement learning approaches treat drone control problems as a Markov decision process. By modeling this process and constructing a neural network for training, they ultimately achieve an optimal trajectory planning strategy.
[0007] However, practical drone planning problems are becoming increasingly complex, requiring consideration of multiple constraints and objectives, such as dynamics, energy consumption, and air traffic control. These constraints often cannot be described using explicit mathematical expressions. Graph-search and random sampling-based methods, due to their focus on path planning and neglect of constraints, are difficult to apply to today's practical mission requirements. Optimization-based methods suffer from the extreme complexity of numerical solutions, making it difficult to find the optimal solution and requiring a significant computational effort. Reinforcement learning-based methods also require significant computational resources for training and executing neural network controllers, which may not be available in certain real-world scenarios and practical problems, posing a challenge for real-time control tasks.
[0008] A review of the aforementioned methods reveals that a common problem with existing drone path planning algorithms applicable to practical tasks is their significant computational resource requirements. This is partly due to model complexity and partly because the majority of existing reinforcement learning methods are based on artificial neural networks. Combining spiking neural networks with reinforcement learning offers a low-energy solution for drone path planning tasks. Its sparse activation and event-driven nature significantly reduces computational frequency. However, due to the loss of significant reward distribution information, spiking reinforcement learning methods lack the ability to cope with uncertainty and randomness, and remain deficient in real-world drone path planning tasks. Furthermore, most existing spiking reinforcement learning methods convert pre-trained artificial neural networks into spiking neural networks, which negates the energy efficiency advantages of spiking neural networks and is highly dependent on their training. Incomplete training of the artificial neural networks or accumulated errors during the conversion process can significantly impact the final results. Summary of the Invention
[0009] The purpose of this invention is to propose a UAV path planning method based on pulse distributed reinforcement learning, aiming to solve the problems of high computing resource requirements, poor model generalization ability, weak model ability to cope with uncertainty and randomness, and high dependence of existing pulse reinforcement learning models on pre-trained artificial neural networks.
[0010] The technical solution adopted in the present invention is:
[0011] A UAV path planning method based on pulse distributed reinforcement learning, the method comprising the following steps:
[0012] Step 1: Select the training data for UAV path planning and build a training set;
[0013] Using Markov process Describe the UAV path planning process. The training data of the training set uses Markov sequence to describe the UAV path;
[0014] in, Represents the state space, which is used to characterize all possible states of the drone;
[0015] Represents the action space, which is used to characterize all the control actions that the drone can take;
[0016] Represents the state transition probability, which is used to characterize the state Select Control Action Transfer to a certain state probability;
[0017] Represents the reward function, which is used to characterize the state Next select action The immediate rewards received;
[0018] Represents the decay coefficient, which is used to adjust the reward weight, discounting future rewards to focus more on recent rewards;
[0019] Step 2: Construct a feature extraction network based on a pulse neural network to extract features of the input drone state. Perform feature extraction to output state State Embedding ,in, is the mean output of the spiking neural network in the entire simulation time window;
[0020] Step 3: Build a pulse distributed reinforcement learning network whose input is the state embedding , based on the quantile distribution, the reward distribution that can be obtained by taking the control action is estimated, and the action value is characterized based on the expected reward corresponding to the reward distribution to select the optimal control action;
[0021] Step 4: Construct a loss function to measure the gap between the expected return predicted by the pulse distributed reinforcement learning network and the actual return;
[0022] Step 5: Train the feature extraction network and the pulse distributed reinforcement learning network based on the training set, optimize the network parameters with the goal of minimizing the constructed loss function, and stop training when the preset end condition is reached. Based on the trained feature extraction network and the pulse distributed reinforcement learning network, a UAV path planning model for autonomous path planning of the UAV is obtained.
[0023] Furthermore, in step 1, each state of the state space is described by establishing a three-dimensional coordinate system for the operating space of the UAV.
[0024] Furthermore, the network structure of the feature extraction network includes a stacked structure consisting of alternating convolutional layers and pulse neuron layers, and a fully connected layer is connected after the stacked structure as the output layer of the feature extraction network.
[0025] Furthermore, the control strategy of the pulse distributed reinforcement learning network to select the optimal control action is:
[0026]
[0027]
[0028] in, represents the control strategy of the impulse distributed reinforcement learning network, From the action space Any control action in Indicates the current state action pair expected return, Express expectations, Indicates the current state action pair The return distribution, expected return By the attenuation coefficient Adjust the expected cumulative rewards in the future.
[0029] Furthermore, the return distribution The way to obtain is: select a certain number of quantiles and output the quantile values corresponding to the quantiles , establish the quantile distribution To estimate the return distribution .
[0030] Furthermore, the network structure of the pulse distributed reinforcement learning network is either of the following two structures:
[0031] Structure 1: It includes a first stacked structure, a second stacked structure, and a fully connected layer. The first stacked structure consists of alternating convolutional layers and spiking neuron layers, and the second stacked structure consists of alternating fully connected layers and spiking neuron layers.
[0032] Structure 2: includes a second stacked structure and a fully connected layer in sequence.
[0033] Furthermore, the loss function is the Huber quantile loss function.
[0034] Furthermore, in step 5, the training termination conditions include but are not limited to: the number of training times reaches a preset upper limit, the loss function value converges, and the path planning capability reaches the expected target.
[0035] Furthermore, the path planning ability is measured by the probability of the UAV reaching the destination and / or the time step length.
[0036] Furthermore, the UAV path planning method based on pulse distributed reinforcement learning of the present invention further includes step 6, a path planning execution step based on the UAV path planning model obtained in step 5, specifically including the following sub-steps:
[0037] Step 6-1, using Markov process to describe the current UAV path planning task;
[0038] In step 6-2, the current state of the drone to be controlled is input into the drone path planning model, and the optimal control action under the current state is obtained based on the output of the model. That is, the feature extraction network based on the model extracts effective information (the state embedding corresponding to the current state), which is then input into the pulse distributed reinforcement learning network to select and output the optimal control action through the control strategy.
[0039] Step 6-3: The controlled UAV executes the optimal control action output by the model and obtains the state at the next moment after the execution is completed;
[0040] Step 6-4, return to step 6-1 with the state obtained in step 6-3 as the current state;
[0041] Repeat steps 6-2 to 6-4 until the UAV path planning task is completed.
[0042] The technical solution provided by the present invention brings at least the following beneficial effects:
[0043] The present invention combines distributed reinforcement learning with pulse neural networks to propose a UAV path planning method based on pulse distributed reinforcement learning. This method reduces the calculation frequency based on the model architecture of the pulse neural network, improves the deficiency of existing UAV path planning methods that require a large amount of computing resources, and provides a low-energy solution for UAV path planning tasks; the method based on distributed reinforcement learning makes up for the problems of existing UAV path planning methods that have a single focus and lose valuable information, enhances the ability of the present invention to deal with uncertainty and randomness, and expands the applicability of the present invention when facing complex and changeable real-life UAV path planning tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0045] Figure 1A processing diagram of a UAV path planning method based on pulse distributed reinforcement learning provided by an embodiment of the present invention;
[0046] Figure 2 A control flow chart of a UAV path planning method based on pulse distributed reinforcement learning provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0047] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described in detail and completely in conjunction with the drawings in the implementation of the present invention. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present invention.
[0048] The embodiment of the present invention provides a UAV path planning method based on pulse distributed reinforcement learning, such as Figure 1 As shown, it includes the following steps:
[0049] Step S1, selecting training data for UAV path planning and constructing a training set;
[0050] In this step, the Markov process is used Describe the UAV path planning process and describe the UAV path through Markov sequence. represents the state space, i.e., all possible states of the drone; represents the action space, i.e., all the control actions that the drone can take; Represents the state transition probability, which is represented by , indicating that in the state Select Control Action Then transfer to a certain state probability; Represents the reward function, which is represented by , that is, in the state Select Control Action The instant rewards you receive, Describes the corresponding state-action pairs; Represents the decay coefficient, which is used to adjust the reward weight and discount future rewards to make the system focus more on recent rewards.
[0051] In one embodiment, the state space Indicates the possible location of the drone, and describes it by establishing a three-dimensional coordinate system for the drone's workspace; action space Indicates the various control actions that the drone can take, such as forward, backward, left turn, right turn, etc.; state transition probability Indicates the current state Take control action , thus entering the next state The probability of reward function Provide instant rewards to the network model based on the state of the drone and the control actions taken, such as rewards for keeping the drone flying normally through control actions , reward for causing the drone to collide by controlling the action , reward for controlling the drone to avoid the aviation control area .
[0052] In one embodiment, the training data for this step can be derived from the drone control process completed by human experts, that is, from the optimal flight path previously determined by human experts. When human expert data is lacking, the training data can also be derived from the drone control process completed by other high-performance drone path planning algorithms. For example, a human expert may determine the optimal flight path for a drone in advance and use it as a sequence. Indicates that, where the subscript is the time point identifier, for each moment in the sequence , This is the state of the drone at that moment, which can be obtained by the camera or positioning system on the drone. is the control action taken at that moment, is the reward obtained at that moment, and the constructed training dataset consists of multiple such sequences.
[0053] Step S2: construct a feature extraction network based on a pulse neural network to extract the state of the input drone. Perform feature extraction to output state State Embedding ; That is, the drone status is preprocessed based on the constructed feature extraction network.
[0054] In this step, the feature extraction network is based on the spiking neural network. is the mean value of the pulse neural network output in the entire simulation time window. The function of the feature extraction network is to extract local and global features of the environment from the image, including edges, textures, shapes, structures, etc. For example, the camera and other equipment on the drone can be used to observe the surrounding environment of the drone and determine the current state of the drone. , as the input of the feature extraction network; the feature extraction network focuses on information such as obstacles, other flying objects, and height from the ground in the captured image and finally outputs a vector as the representation of the state, that is, the state embedding .
[0055] In one embodiment, the parameters of the feature extraction network in this step are randomly initialized, and then can be updated through gradient back propagation based on a preset loss function during training.
[0056] In one embodiment, the feature extraction network in this step comprises a stacked structure consisting of alternating convolutional layers and spiking neuron layers, followed by a fully connected layer as the output layer of the feature extraction network. Specifically, the feature extraction network comprises K (K ≥ 2) convolutional layers, one fully connected layer, and K spiking neuron layers. Each convolutional layer is followed by a spiking neuron layer, and the last spiking neuron layer is followed by a fully connected layer. These functional layers enable the feature extraction network to capture environmental features from the input image to better describe the environment.
[0057] Step S3, establishing a pulse distributed reinforcement learning network, which describes the rewards that can be obtained from the control action based on the quantile distribution to select the optimal control action;
[0058] In this step, the input of the pulse distributed reinforcement learning network is the state embedding , based on the quantile distribution, the reward distribution that can be obtained by taking the control action is estimated, and the action value is characterized based on the expected reward corresponding to the reward distribution to select the optimal control action;
[0059] In one embodiment, the expression of the pulse distributed reinforcement learning network established in step S3 is:
[0060]
[0061]
[0062] in, Represents the control strategy of the pulse distributed reinforcement learning network, that is, in a certain state When , select the control action to be taken, Control strategy for short ; From the action space Any control action in is the action value of the current state-action pair, also known as the reward, which is represented by The expected cumulative reward in the future is adjusted, that is, the drone is in the current state Take control action , future discount rewards and, Action Value ; express expectations; represents the reward distribution of the current state-action pair, that is, the reward is regarded as a random variable, Return Distribution .
[0063] In the embodiment of the present invention, the pulse distributed reinforcement learning network maximizes the action value To select the most appropriate control action, due to the current state Take control action The returns are not always the same, so the expected return distribution is used as For example, when controlling a drone to turn left, there is a 90% probability that the drone will fly normally, but there is also a 10% probability that it will collide, so the reward is not fixed. By treating the reward as a random variable and modeling it, the network model has a stronger ability to handle randomness and uncertainty. The input of the pulse distributed reinforcement learning network is the state embedding output of the feature extraction network in the previous step. , the output is the predicted Optimal control actions based on values and predictions .
[0064] In one embodiment, the reward distribution is estimated by quantile distribution to calculate the action value and select the optimal control action.
[0065] Since the real reward distribution of the control task is unknown in real scenarios, a pulse neural network is constructed based on the expression of the pulse distributed reinforcement learning network, a certain number of quantiles are selected, the quantile values corresponding to the quantiles are output, and the quantile distribution is established. To estimate the return distribution ,in Represents the quantile value, that is, sampling state-action pairs in the training set and establishing a quantile distribution for them, estimating the return corresponding to the state-action pair by the established quantile distribution, and realizing the quantile value pair return distribution At the same time, through the quantile distribution right Calculate and output Maximum optimal control action .
[0066] Preferably, quantile distribution The expression is:
[0067]
[0068] in:
[0069]
[0070]
[0071] in, is a hyperparameter, indicating the number of selected quantiles, is the number of the quantile. In this embodiment, let ; is the shock function; It is the distribution function, that is, the integral of the probability density function, which describes the probability distribution of a random variable; Indicates the The quantile value of the state-action pair, Return distribution The inverse function of the distribution function is output by the pulse neural network, let represents the last layer of the spiking neural network (i.e., the last layer of spiking neurons in the spiking distributed reinforcement learning network), Represents the spiking neural network The parameters of the layers, in this embodiment, the pulse neural network includes 4 layers, so . Represents the simulation time window length of the pulse neural network. In this embodiment, let . Indicates the second to last layer of the spiking neural network at simulation time Output at the moment; is the quantile, and the quantile is chosen by OK, for , The probability density function of the return distribution equal parts, , that is, in In the probability density function of , the area divided between every two quantiles is For simplicity, let Quantile distribution The expression means that each quantile The probability of occurrence is approximately , each quantile Corresponding to a quantile , which means that in the return distribution middle .
[0072] In one embodiment, the network structure of the pulse distributed reinforcement learning network is: a first stacking structure is formed by alternating convolutional layers and pulse neuron layers, a second stacking structure is formed by alternating fully connected layers and pulse neuron layers, the output of the first stacking structure is used as the input of the second stacking structure, and a fully connected layer is connected after the second stacking structure as the output layer of the pulse distributed reinforcement learning network. Alternatively, the network structure is set to: a fully connected layer is directly connected after the second stacking structure as the output layer of the pulse distributed reinforcement learning network. That is, in this embodiment, the pulse distributed reinforcement learning network includes convolutional layers, Layer fully connected layer and Each convolutional layer is followed by a spike neuron layer. Except for the last fully connected layer, each fully connected layer is followed by a spike neuron layer.
[0073] Step S4: construct a loss function to measure the gap between the rewards calculated by the estimated quantile values of the pulse distributed reinforcement learning network and the actual rewards, so as to optimize and update the network parameters by minimizing the loss function during the training process.
[0074] In one embodiment, the loss function constructed in step S4 is specifically a Huber quantile loss function.
[0075] The Huber quantile loss function is a hyperparameter In this embodiment, control In the UAV path planning problem, suppose that at a certain moment The drone status and control actions are , the corresponding return distribution is , the next moment The drone status is , the corresponding return distribution is , then the Huber quantile loss is expressed as follows:
[0076] ;
[0077] in:
[0078] ;
[0079] ;
[0080] in, is the auxiliary amount, Indicates Instant rewards at all times, Represents the attenuation coefficient, which is used to adjust the reward weight. Indicates control strategy based on The optimal control action is selected, , They are the percentile values corresponding to the system state action pair at the current moment and the percentile values corresponding to the system state action pair at the next moment, respectively. and the next moment return distribution Establish quantile distribution to represent and calculate, represents the Huber loss function, Represents the Huber quantile loss function.
[0081] Step S5: Use the training set to train the feature extraction network and the pulse distributed reinforcement learning network to determine hyperparameters, including the maximum number of model iterations, the model convergence threshold, and the model trade-off parameters. Training is terminated until the prediction performance reaches a preset value. Based on the trained feature extraction network and the pulse distributed reinforcement learning network, a UAV path planning model is obtained to implement autonomous UAV path planning. The model trade-off parameters refer to the learning rate of the neural network and related parameters of the spiking neurons, such as the simulation time window length of the spiking neural network, the membrane potential threshold, membrane time constant, and initial membrane potential of the spiking neurons, and the learning rates of the feature extraction network and the pulse distributed reinforcement learning network.
[0082] In one embodiment, the training process of the feature extraction network and the pulse distributed reinforcement learning network specifically includes the following steps:
[0083] (1) Obtain training data from human experts or other high-performance UAV path planning algorithms to construct a training dataset;
[0084] (2) Set the number of quantiles N, the pulse neural network simulation time window T, and the Huber quantile loss parameter, hyperparameters ;
[0085] (3) Initialize the network parameters of the feature extraction network and the pulse distributed reinforcement learning network;
[0086] (4) Obtaining data from the training set , the current moment and the next moment Status 、 Input the feature extraction network separately and get Moment and State Embedding 、 ;
[0087] (5) Embedding the state 、 Input pulse distributed reinforcement learning network to obtain the corresponding quantile value and quantile distribution ;
[0088] (6) Pulse Distributed Reinforcement Learning Network Calculation of Quantile Distribution The expectation of is taken as the expected return, and the expected return of all state-action pairs is output;
[0089] (7) The pulse distributed reinforcement learning network selects the control action that maximizes the expected reward as the optimal control action ;
[0090] (8) For state-action pairs 、 Calculate the corresponding quantile value and obtain the Huber quantile loss based on the quantile position;
[0091] (9) Backward gradient propagation based on Huber quantile loss updates the network parameters of the feature extraction network and the pulse distributed reinforcement learning network;
[0092] (10) Whether the preset training end condition is met, if so, then end; otherwise, return to step (4); wherein the training end condition is set as the network model prediction performance reaching a preset value, which can be set as: the number of training times reaches a preset upper limit, the Huber quantile loss converges, and the metric-based path planning ability detection prediction performance reaches a preset value. Preferably, the path planning ability can be measured by the probability of the UAV reaching the destination and / or the time step length.
[0093] In one embodiment, the process of implementing the UAV path planning based on the trained UAV path planning model is as follows: Figure 2 Specifically, the following are included:
[0094] Step 1: For the drone to be controlled, use the Markov process to describe the system according to the possible states of the drone and the control actions that can be selected; that is, the Markov process is used to describe the drone path planning task, and the camera and other equipment carried by the drone observe the drone's surroundings and determine the drone's current state. ;
[0095] Step 2: Set the current state of the drone to be controlled Input the drone path planning model, and use the model's feature extraction network to focus on obstacles, other flying objects, height from the ground, and other information in the captured image, and output state embedding ;
[0096] Step 3: Embed the state The pulse distributed reinforcement learning network of the input model is distributed by quantiles Calculating action value , through the control strategy Select the largest The action corresponding to the value is the optimal control action And output;
[0097] Step 4: For the drone to be controlled, execute the selected optimal control action Control and execute optimal control actions The state of the drone changes, and the state of the drone to be controlled at the next moment is obtained ;
[0098] By status As the current state, repeat steps 2, 3, and 4 until the drone path planning process is completed.
[0099] In one embodiment, in step 3 above, the control strategy The selection process is: for any input state embedding , respectively for all possible control actions , predicted by the model’s pulse distributed reinforcement learning network The corresponding return distribution percentile, and then the return distribution is described by the percentile ,based on The expected return is obtained from the mathematical expectation, and then the control action corresponding to the maximum expected return is selected using the expected return as the metric value. As the optimal control action , to maximize expected returns.
[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.
[0101] The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the inventive concept of the present invention, which all fall within the scope of protection of the present invention.
Claims
1. A UAV path planning method based on pulse distributed reinforcement learning, characterized in that: The following steps are involved: Step 1: Select the training data for UAV path planning and build a training set; Using Markov process Describe the UAV path planning process. The training data of the training set uses Markov sequence to describe the path of the UAV; in, Represents the state space, which is used to characterize all possible states of the drone; Represents the action space, which is used to characterize all the control actions that the drone can take; Represents the state transition probability, which is used to characterize the state Select Control Action Transfer to a certain state probability; Represents the reward function, which is used to characterize the state Next select action The immediate rewards received; Represents the decay coefficient, which is used to adjust the reward weight; Step 2: Construct a feature extraction network based on a pulse neural network. The feature extraction network is used to extract the state of the input Perform feature extraction to output the state State Embedding ,in, is the mean value of the output of the spiking neural network in the entire simulation time window; Step 3: Build a pulse distributed reinforcement learning network whose input is the state embedding , based on the quantile distribution, the reward distribution that can be obtained by taking the control action is estimated, and the action value is characterized based on the expected reward corresponding to the reward distribution to select the optimal control action; Step 4: construct a loss function to measure the gap between the expected return predicted by the pulse distributed reinforcement learning network and the actual return; Step 5: Train the feature extraction network and the pulse distributed reinforcement learning network based on the training set, optimize the network parameters with the purpose of minimizing the constructed loss function, and stop when the preset training end condition is reached. Based on the trained feature extraction network and the pulse distributed reinforcement learning network, a UAV path planning model for autonomous path planning of the UAV is obtained.
2. A UAV path planning method based on pulse distributed reinforcement learning as claimed in claim 1, characterized in that: In step 1, each state of the state space is described by establishing a three-dimensional coordinate system for the UAV's operating space.
3. A UAV path planning method based on pulse distributed reinforcement learning as claimed in claim 1, characterized in that: The network structure of the feature extraction network includes a stacked structure consisting of alternating convolutional layers and pulse neuron layers, and a fully connected layer is connected after the stacked structure as the output layer of the feature extraction network.
4. A UAV path planning method based on pulse distributed reinforcement learning as claimed in claim 1, characterized in that: The control strategy of the impulse distributed reinforcement learning network to select the optimal control action is: ; ; in, represents the control strategy of the spiking distributed reinforcement learning network, From the action space Any control action in Indicates the current state action pair Expected return, Express expectations, Indicates the current state action pair The return distribution, expected return By the attenuation coefficient Adjust the expected cumulative rewards in the future.
5. A UAV path planning method based on pulse distributed reinforcement learning as claimed in claim 4, characterized in that: Return Distribution The method of obtaining is: select a certain number of quantiles and output the quantile values corresponding to the quantiles , establish the quantile distribution To estimate the return distribution .
6. A method for UAV path planning based on pulse distributed reinforcement learning as claimed in claim 1, characterized in that: The network structure of the pulse distributed reinforcement learning network is structure 1 or structure 2; Structure 1 includes: a first stacking structure, a second stacking structure and a fully connected layer in sequence; Structure 2 includes: a second stacking structure and a fully connected layer in sequence; The first stacked structure is composed of alternating convolutional layers and spiking neuron layers, and the second stacked structure is composed of alternating fully connected layers and spiking neuron layers.
7. A method for UAV path planning based on pulse distributed reinforcement learning as claimed in claim 1, characterized in that: The loss function is the Huber quantile loss function.
8. The method for UAV path planning based on pulse distributed reinforcement learning as claimed in claim 1, characterized in that: In step 5, the training end conditions are: the number of training times reaches a preset upper limit, the loss function value converges and / or the path planning capability reaches the expected target.
9. A method for UAV path planning based on pulse distributed reinforcement learning as claimed in claim 8, characterized in that: The path planning ability is measured by the probability of the UAV reaching the destination and / or the time step length.
10. A UAV path planning method based on pulse distributed reinforcement learning according to any one of claims 1 to 9, characterized in that: The method further includes executing step 6 of path planning based on the UAV path planning model obtained in step 5, which specifically includes the following sub-steps: Step 6-1, using Markov process to describe the current UAV path planning task; Step 6-2, input the current state of the drone to be controlled into the drone path planning model, and obtain the optimal control action under the current state based on the output of the model; Step 6-3, the controlled UAV executes the optimal control action output by the model, and obtains the state at the next moment after the execution is completed; Step 6-4, return to step 6-1 with the state obtained in step 6-3 as the current state; Repeat steps 6-2 to 6-4 until the UAV path planning task is completed.
Citation Information
Patent Citations
Deep and reinforcement learning-based real-time online path planning method of
CN106970615A
AGV path planning based on deep reinforcement learning
CN114355915A
Path planning method and device and automatic driving vehicle control system
CN116558517A
Real-time path optimization and maneuvering avoidance action planning method and device
CN118089768A
Unmanned ship risk adaptive navigation algorithm based on distributed reinforcement learning
CN118747519A
Cited By
End-to-end automatic driving decision-making method and system based on brain-like reinforcement learning
CN119928898A
