Unmanned aerial vehicle obstacle avoidance method, system and device based on reinforcement learning and differential programming and medium

Abstract

The invention discloses an unmanned aerial vehicle obstacle avoidance method, system and device based on reinforcement learning and differential programming and a medium. The obstacle avoidance method comprises the following steps: constructing an unmanned aerial vehicle state matrix s; constructing a task neural network module, and obtaining an expected flight vector fexp of the unmanned aerial vehicle according to the unmanned aerial vehicle state matrix s; constructing a differentiable safety module, correcting an expected flight vector fexp of the unmanned aerial vehicle according to the obstacle position, and obtaining an actual flight direction vector ffly of the unmanned aerial vehicle; the task neural network module and the differentiable safety module are combined in a mode that the task neural network module is in front of the differentiable safety module, an unmanned aerial vehicle obstacle avoidance strategy network is constructed, an unmanned aerial vehicle simulation flight environment is established, the unmanned aerial vehicle obstacle avoidance strategy network is trained, and an available unmanned aerial vehicle obstacle avoidance strategy is obtained; the system, the device and the medium are used for bearing and implementing the method. Safe flight of the unmanned aerial vehicle is achieved in the simulation environment, the advantages of an intelligent algorithm and a traditional obstacle avoidance algorithm are integrated, and therefore the safety of the unmanned aerial vehicle in the task execution process is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicle control, and in particular relates to a method, system, device and medium for unmanned aerial vehicle obstacle avoidance based on reinforcement learning and differentiable programming. Background Art

[0002] In recent years, commercial drones have gradually become less expensive, and the number of drones is increasing. In some planning blueprints, automatically controlled drones may take on the duties of unmanned logistics, emergency communication base stations, border patrols, etc., and the demand for intelligent drones is showing an upward trend. At present, there are a large number of intelligent algorithms based on neural networks that can improve the intelligence level of drones and achieve better working results in complex dynamic environments, but the safety is often poor. There are often static obstacles such as buildings and trees in the working environment of drones, and even dynamic obstacles between drones. The existence of these obstacles may cause serious safety hazards to the flight of drones.

[0003] Existing obstacle avoidance algorithms for drones often run based on pre-defined rules and have good safety performance. However, the parameters of these algorithms are determined in advance by the designers, and the program logic is not differentiable, resulting in the inability to transfer gradients. These defects make it difficult to combine existing obstacle avoidance algorithms with intelligent methods based on neural networks, and it is even more impossible to perform end-to-end joint parameter optimization.

[0004] The patent application document with publication number CN118628944A discloses a reinforcement learning drone obstacle avoidance method that integrates event streams and event frames. The event streams and event frame information provided by the time camera are trained using reinforcement learning to avoid obstacles. However, since there is no special safety design in the reinforcement learning algorithm, trial and error learning is still required, and there is a risk of collision during the training phase.

[0005] The patent application document with publication number CN118760235A discloses a drone autonomous obstacle avoidance system based on deep Q network and dynamic SARSA algorithm, which guides the drone cluster to achieve efficient collaborative obstacle avoidance through obstacle information exchange in the drone cluster. However, there is no safety design in the reinforcement learning algorithm, and it is difficult to avoid collisions between drones and obstacles during training. Summary of the invention

[0006] In order to overcome the deficiencies of the above-mentioned prior art, the purpose of the present invention is to provide a method, system, device and medium for obstacle avoidance of unmanned aerial vehicles based on reinforcement learning and differentiable programming. By embedding the path planning algorithm into the neural network in a differentiable form, dangerous actions can be eliminated in real time in the decision space of the unmanned aerial vehicle, and the smooth propagation of the training gradient can be maintained; the present invention improves the safety of trajectory decision-making of the unmanned aerial vehicle during the execution of the mission, and realizes the fusion of the advantages of intelligent algorithms and traditional obstacle avoidance algorithms.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] A drone obstacle avoidance method based on reinforcement learning and differentiable programming includes the following steps:

[0009] Step 1, construct the drone state matrix s;

[0010] Step 2: Build the task neural network module and obtain the expected flight vector f of the drone based on the drone state matrix s built in step 1. exp ;

[0011] Step 3: Construct a differentiable safety module to correct the expected flight vector f of the drone obtained in step 2 according to the obstacle position. exp , get the actual flight direction vector f of the drone fly ;

[0012] Step 4: With the task neural network module constructed in step 2 in front and the differentiable safety module constructed in step 3 in the back, the two modules are combined to construct a UAV obstacle avoidance strategy network, build a UAV simulation flight environment, and train the UAV obstacle avoidance strategy network to obtain a usable UAV obstacle avoidance strategy.

[0013] The specific method of step 1 is:

[0014] Step 1.1, according to the task that the drone needs to perform, obtain task-related information;

[0015] Step 1.2, obtain the drone's own status information, including coordinates, attitude and motion status;

[0016] Step 1.3, the drone state information obtained in step 1.2 and the task related information obtained in step 1.1 are expressed in matrix form to form the drone state matrix s.

[0017] The specific method of step 2 is:

[0018] The task neural network module consists of several deep neural network layers, where the first layer is called the input layer and the last layer is called the output layer;

[0019] Step 2.1, the input layer of the task neural network module uses the dimension of the drone state matrix s constructed in step 1.3 as the input dimension size;

[0020] Step 2.2: Input the drone state matrix s constructed in step 1.3 into the input layer of the task neural network module constructed in step 2.1, perform forward propagation calculations on all layers of the task neural network module, and obtain the expected flight vector of the drone in the output layer.

[0021]

[0022] The specific method of step 3 is:

[0023] Step 3.1, get the coordinates of all obstacles z(z∈[1,2,…,Z]) And the influence range R of all obstacles z ;

[0024] Step 3.2, calculate the current coordinates of the drone The distance vector v = q to all obstacles z uav -q z and the distance d = ||v||2;

[0025] Step 3.3: The influence range R of all obstacles obtained in step 3.1 z And the distance vector v=q of all obstacles z of the drone obtained in step 3.2 uav -q z And the distance d = ||v||2, calculate and record the repulsive force exerted by all obstacles z on the drone Among them, φ(x) = 1 / (1+exp(-xβ0)) is the sigmoid function, β0 is the temperature coefficient, k rep is the repulsion coefficient;

[0026] Step 3.4: The repulsive force f exerted on the drone by the obstacle z obtained in step 3.3 z rep , calculate the sum of the repulsive forces of all obstacles z on the drone

[0027] Step 3.5: Substitute the desired flight vector f of the drone obtained in step 2 exp The sum of the repulsive force f calculated in step 3.4 rep Find the vector sum and normalize it to get the actual flight direction vector of the drone after correction That is, a differentiable security module is obtained.

[0028] The specific method of step 4 is:

[0029] Step 4.1, with the task neural network module constructed in step 2 in front and the differentiable safety module constructed in step 3 in the back, the two modules are combined to form the network structure of the drone obstacle avoidance strategy;

[0030] Step 4.2, build the UAV simulation flight environment, including time and space parameters, UAV missions, UAV motion model and obstacles;

[0031] Step 4.3, select any deterministic policy gradient reinforcement learning algorithm framework, replace the Actor network in the algorithm with the drone obstacle avoidance strategy network in step 4.1, and obtain a deterministic policy gradient reinforcement learning algorithm;

[0032] Step 4.4, design the reward function required for training for the UAV mission;

[0033] Step 4.5, set training hyperparameters;

[0034] Step 4.6, based on the UAV simulation flight environment built in step 4.2, the reward function required for training designed in step 4.4, and the training hyperparameters set in step 4.5, use the deterministic policy gradient reinforcement learning algorithm obtained in step 4.3 to train the UAV obstacle avoidance strategy network obtained in step 4.1. When the preset training reaches the preset training rounds, the UAV obstacle avoidance strategy network training is completed.

[0035] In step 2, the deep neural network layer includes but is not limited to one or more of a multi-layer fully connected network, a convolutional network, a recurrent neural network, or an attention network.

[0036] In step 3, the influence range of the obstacle is determined by the outline of the specific obstacle. When the influence range is set, the influence range is greater than the distance from the obstacle coordinates to its farthest boundary.

[0037] The present invention also provides a UAV obstacle avoidance system based on reinforcement learning and differentiable programming, comprising:

[0038] UAV state matrix s construction module, used to construct the UAV state matrix s;

[0039] Task neural network module building module, used to obtain the expected flight vector f of the drone based on the constructed drone state matrix s exp ;

[0040] Differentiable safety module building block to correct the desired flight vector f of the drone based on the obstacle position exp , get the actual flight direction vector f of the drone fly ;

[0041] The UAV obstacle avoidance strategy network construction and training module is used to combine the two modules in a way that the task neural network module is in front and the differentiable safety module is in the back, to construct the UAV obstacle avoidance strategy network, build a UAV simulation flight environment, and train the UAV obstacle avoidance strategy network to obtain a usable UAV obstacle avoidance strategy.

[0042] The present invention also provides a drone obstacle avoidance device based on reinforcement learning and differentiable programming, comprising:

[0043] Memory: a computer program for storing the above-mentioned obstacle avoidance method for unmanned aerial vehicles based on reinforcement learning and differentiable programming, which is a computer-readable device;

[0044] Processor: used to implement the drone obstacle avoidance method based on reinforcement learning and differentiable programming when executing the computer program.

[0045] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the drone obstacle avoidance method based on reinforcement learning and differentiable programming.

[0046] Compared with the prior art, the present invention has the following advantages:

[0047] 1. In step 3.3 of the present invention, the idea of ​​differentiable programming is adopted, and the repulsive force calculation of all obstacles on the drone is relaxed by the Sigmoid function, so that the originally non-differentiable artificial potential field algorithm is differentiated, so that the algorithm can be directly combined with the deep neural network, and the smooth propagation of the gradient can be guaranteed. Therefore, the tasks that the drone itself needs to perform and the obstacle avoidance requirements can be taken into account, and the integrated design of the task strategy and the obstacle avoidance strategy is realized. This method overcomes the problem that the existing obstacle avoidance algorithm is difficult to combine with the intelligent method based on the neural network.

[0048] 2. The root of the obstacle avoidance method adopted in step 3 of the present invention is the artificial potential field method, and the influence range and repulsion coefficient of all obstacles can be adjusted manually. Compared with the use of neural network obstacle avoidance, it is more explainable and reliable. And the obstacle coordinates, obstacle influence range and repulsion coefficient parameters in the artificial potential field method are quantitatively adjustable, and the obstacle avoidance effect can be quantitatively adjusted according to demand, which is more reliable than the qualitative adjustment method of the neural network. This method can avoid wrong actions during the training stage, and no trial and error learning is required, which significantly improves the reliability of obstacle avoidance.

[0049] 3. In step 4.1 of the present invention, a drone obstacle avoidance strategy network is formed by combining a task neural network module and a differentiable safety module, wherein the task neural network module is in front and the differentiable safety module is closely combined in the back, and the gradient can be smoothly propagated, so most of the parameters in the network can be optimized by the gradient, so spontaneous optimization and adjustment can be performed in different environments, showing better adaptability. This technology integrates the design of the task network and the differentiable safety module, solving the problem of difficulty in end-to-end joint parameter optimization.

[0050] In summary, the present invention embeds the artificial potential field method into the neural network in a differentiable form to form a differentiable strategic obstacle avoidance network, which can achieve collision avoidance of obstacles by drones. This method has the advantages of strong interpretability, high reliability and quantitative tuning. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a complete flow chart of the present invention.

[0052] Figure 2 This is a network structure diagram of the drone obstacle avoidance strategy constructed by the present invention.

[0053] Figure 3 This is a simulated flight trajectory diagram of an embodiment of the present invention.

[0054] Figure 4 This is a safety performance diagram of the training process of an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. The embodiments described with reference to the accompanying drawings are exemplary and intended to be used to explain the present application, but should not be understood as limiting the present application.

[0056] The technical idea of ​​the present invention is: first, a UAV obstacle avoidance strategy network is constructed, which includes a task neural network module for outputting the expected flight direction of the UAV, and a differentiable safety module for correcting the flight direction of the UAV; then, a simulation environment for UAV flight is built, and the designed UAV obstacle avoidance strategy network is trained through the deterministic policy gradient method in reinforcement learning.

[0057] Experimental conditions

[0058] This experiment was run on a desktop computer equipped with an Intel i9 CPU, Nvidia 4090 GPU, and 64G memory, and Pytorch was used for simulation.

[0059] like Figure 1 As shown in the figure, a drone obstacle avoidance method based on reinforcement learning and differentiable programming is shown in the figure. The specific steps are as follows:

[0060] Step 1, construct the drone state vector s;

[0061] Step 1.1, according to the task that the drone needs to perform, obtain task-related information. The task-related information refers to the various target tasks that the drone may perform. During the flight of the task, obstacles may be encountered and obstacle avoidance is required. For example, suppose the current drone is performing a data collection task, and the drone needs to collect data back and forth between three data collection target locations. The coordinates of the three data collection target locations are Mission information can be set as the real-time relative position relationship vector between the drone and the location to be collected in, is the current coordinate of the drone.

[0062] Step 1.2, obtain the drone's own status information, for example, select the drone's current x-axis coordinate x uav , y-axis coordinate y uav , z-axis coordinate z uav , speed v uav , acceleration a uav , flight direction f uav , then the drone's own state information forms a vector

[0063]

[0064] Step 1.3, concatenate the drone's own state information obtained in step 1.2 into a vector and the task-related information obtained in step 1.1 in vector form to form the drone state vector

[0065]

[0066] Step 2: Build the task neural network module and obtain the expected flight vector f of the drone based on the drone state vector s built in step 1. exp ;

[0067] Step 2.1, as a specific embodiment, the task neural network module is composed of multiple layers of fully connected layers, wherein the number of dimensions of the drone state vector formed in step 1.3 is used as the number of neurons in the first layer, the number of neurons in the last layer is 3, and the output value of the last layer is further linearly mapped to a reasonable value range after being constrained by the Sigmoid activation function, and the reasonable value range is a maximum value that is significantly smaller than the sum of the repulsive forces of obstacles on the drone f rep The maximum value of the repulsive force f rep It has a significant impact on the output value of the task neural network module;

[0068] The deep neural network that constitutes the task neural network module can be a common neural network structure such as a multi-layer fully connected network, a convolutional network, a recurrent neural network, or an attention network, or a combination of them. The specific structure is determined by the drone state vector and the task requirements.

[0069] Step 2.2: Input the drone state vector s constructed in step 1.3 into the input layer of the task neural network module constructed in step 2.1, perform forward propagation calculations on all layers of the task neural network module, and output the expected flight vector of the drone.

[0070] Step 3: Construct a differentiable safety module to correct the expected flight vector f of the drone obtained in step 2 according to the obstacle position. exp , get the actual flight direction vector f of the drone fly ;

[0071] The Differentiable Safety Module is a differentiable artificial potential field algorithm;

[0072] Step 3.1, get the coordinates of all obstacles z(z∈[1,2,…,Z]) And the influence range R of all obstacles z As a specific example, assume that there are three cylindrical obstacles in the scene, and the coordinates of the ground center are The radius of each cylinder is 50 meters, and the repulsive force has a range of R z = 100, because the cylindrical obstacle is at the height z of the drone uav There is a threat to the UAV, so the coordinates of each cylindrical obstacle can be recorded as

[0073] The influence range of an obstacle is mainly determined by the outline of the specific obstacle. When setting the influence range, it is necessary to ensure that it is greater than the distance from the obstacle coordinates to its farthest boundary.

[0074] Step 3.2, calculate the current coordinates of the drone Distance vector to all cylindrical obstacles z and the distance d z =||v z ||2,z=1,2,3;

[0075] Step 3.3, based on the influence range R of the cylindrical obstacle z obtained in step 3.1 z And the distance vector between the drone and the cylindrical obstacle z obtained in step 3.2 and the distance d z =||v z ||2,z=1,2,3, calculate and record the repulsive force of all cylindrical obstacles z on the drone Among them, φ(x)=1 / (1+exp(-xβ0)) is the Sigmoid function, β0 is the temperature coefficient, k rep is the repulsion coefficient. As a specific example, β0=5, k rep =400;

[0076] Step 3.4: The repulsive forces exerted on the drone by all cylindrical obstacles z obtained in step 3.3 Calculate the sum of the repulsive forces of all cylindrical obstacles z on the drone

[0077]

[0078] Step 3.5: Substitute the desired flight vector f of the drone obtained in step 2 exp The sum of the repulsive force f calculated in step 3.4 rep , find the vector sum and normalize it to get the actual flight direction vector of the drone after correction That is, a differentiable security module is obtained.

[0079] like Figure 2 As shown, in step 4, the two modules are combined in such a way that the task neural network module constructed in step 2 is in front and the differentiable safety module constructed in step 3 is in the back, a UAV obstacle avoidance strategy network is constructed, a UAV simulation flight environment is built, and the UAV obstacle avoidance strategy network is trained to obtain a usable UAV obstacle avoidance strategy.

[0080] Step 4.1, such as Figure 2 As shown, the task neural network module constructed in step 2 is placed in front and the differentiable safety module constructed in step 3 is placed in the back, and the two modules are combined to form the network structure of the drone obstacle avoidance strategy;

[0081] Step 4.2, build a simulated flight environment for drones. As a specific embodiment, assume that the scene range is a square area of ​​one square kilometer. The drone flies at a constant speed of 20m / s at an altitude of 75 meters and makes a flight direction decision every second. There are three cylindrical obstacles z in the center of the scene, and all the obstacles are higher than the flight altitude of the drone. The radius of the obstacles is 50 meters. There are three data collection target locations outside the obstacle influence range in the scene. The drone needs to fly back and forth between the three data collection target locations for data collection. The horizontal coordinate position of the drone is randomly initialized each time the flight environment is reset. The simulation time of each search and rescue mission is 120 seconds.

[0082] Step 4.3, select any deterministic policy gradient reinforcement learning algorithm framework. As a specific embodiment, select the TD3 framework, replace the Actor network in the algorithm with the drone obstacle avoidance strategy network described in step 4.1, and obtain a deterministic policy gradient reinforcement learning algorithm;

[0083] The deterministic policy gradient reinforcement learning algorithm framework refers to a type of reinforcement learning algorithm that outputs a certain action value, rather than a specific reinforcement learning algorithm.

[0084] Step 4.4, design the reward function required for training for the drone mission. As a specific embodiment, the real-time reward of the drone is designed to be the negative of the average AoI (Age of Information) between the data collection target locations to minimize the global AoI.

[0085] Step 4.5, set training hyperparameters, such as learning rate, number of training steps, experience pool size, etc. As a specific embodiment, the learning rate of the drone obstacle avoidance strategy network and the Critic network in TD3 are set to 0.0005, and the learning rate gradually decreases with the number of training steps. During the training, 5000 data collection tasks are simulated, and the experience pool size is set to 40000;

[0086] Step 4.6, based on the UAV simulation flight environment built in step 4.2, the reward function required for training designed in step 4.4, and the training hyperparameters set in step 4.5, the UAV obstacle avoidance strategy network obtained in step 4.1 is trained using the TD3 framework selected in step 4.3. When the preset training reaches the preset training rounds, the UAV obstacle avoidance strategy network training is completed. As a specific embodiment, the end condition can be set to reach the preset number of search and rescue mission simulations, or the reward has stably converged.

[0087] Experimental Results

[0088] The trajectory of the UAV data collection mission is as follows Figure 3 As shown, it can be found that the UAV does not enter the cylindrical no-fly zone once when flying back and forth between various data collection target locations (SN1, SN2 and SN3 in the figure), indicating that the obstacle avoidance method of the present invention is effective.

[0089] Figure 4 The number of times the UAV enters the no-fly zone during reinforcement learning training when the method proposed in the present invention (Ours in the figure) is compared with the number of times the UAV enters the no-fly zone when the method proposed in the present invention is used and when the method proposed in the present invention is not used (Traditional in the figure). It can be seen that after using the method proposed in the present invention, the trajectory of the UAV is safe throughout the entire training process and has never entered the no-fly zone.

[0090] The present invention also provides a UAV obstacle avoidance system based on reinforcement learning and differentiable programming, comprising:

[0091] The drone state matrix s construction module is used to implement the construction of the drone state matrix s in step 1;

[0092] Task neural network module building module, used to implement the drone state matrix s constructed in step 2 according to step 1, and obtain the drone's expected flight vector f exp ;

[0093] Differentiable safety module building block, used to implement the desired flight vector f obtained in step 3 based on the obstacle position correction of the drone in step 2 exp , get the actual flight direction vector f of the drone fly ;

[0094] The UAV obstacle avoidance strategy network construction and training module is used to implement the task neural network module constructed in step 2 in front and the differentiable safety module constructed in step 3 in the back in step 4, combine the two modules, build a UAV obstacle avoidance strategy network, build a UAV simulation flight environment, and train the UAV obstacle avoidance strategy network to obtain a usable UAV obstacle avoidance strategy.

[0095] The present invention also provides a drone obstacle avoidance device based on reinforcement learning and differentiable programming, comprising:

[0096] Memory: a computer program for storing the above-mentioned obstacle avoidance method for unmanned aerial vehicles based on reinforcement learning and differentiable programming, which is a computer-readable device;

[0097] Processor: used to implement the drone obstacle avoidance method based on reinforcement learning and differentiable programming when executing the computer program.

[0098] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the drone obstacle avoidance method based on reinforcement learning and differentiable programming.

[0099] The above description is only a specific example of the present invention and does not constitute any limitation to the present invention. It is obvious that for professionals in this field, after understanding the content and principles of the present invention, it is possible to make various modifications and changes in form and details without departing from the principles and structures of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

Claims

1. A drone obstacle avoidance method based on reinforcement learning and differentiable programming, characterized in that: The following steps are involved: Step 1, construct the drone state matrix s; Step 2: Build the task neural network module and obtain the expected flight vector f of the drone based on the drone state matrix s built in step 1. exp ; Step 3: Construct a differentiable safety module to correct the expected flight vector f of the drone obtained in step 2 according to the obstacle position. exp , get the actual flight direction vector f of the drone fly ; Step 4: With the task neural network module constructed in step 2 in front and the differentiable safety module constructed in step 3 in the back, the two modules are combined to construct a UAV obstacle avoidance strategy network, build a UAV simulation flight environment, and train the UAV obstacle avoidance strategy network to obtain a usable UAV obstacle avoidance strategy.

2. The unmanned aerial vehicle obstacle avoidance method based on reinforcement learning and differentiable programming according to claim 1, characterized in that: The specific method of step 1 is: Step 1.1, according to the task that the drone needs to perform, obtain task-related information; Step 1.2, obtain the drone's own status information, including coordinates, attitude and motion status; Step 1.3, the drone state information obtained in step 1.2 and the task related information obtained in step 1.1 are expressed in matrix form to form the drone state matrix s.

3. The unmanned aerial vehicle obstacle avoidance method based on reinforcement learning and differentiable programming according to claim 1, characterized in that: The specific method of step 2 is: The task neural network module consists of several deep neural network layers, where the first layer is called the input layer and the last layer is called the output layer; Step 2.1, the input layer of the task neural network module uses the dimension of the drone state matrix s constructed in step 1.3 as the input dimension size; Step 2.2: Input the drone state matrix s constructed in step 1.3 into the input layer of the task neural network module constructed in step 2.1, perform forward propagation calculations on all layers of the task neural network module, and obtain the expected flight vector of the drone in the output layer.

4. The unmanned aerial vehicle obstacle avoidance method based on reinforcement learning and differentiable programming according to claim 1, characterized in that: The specific method of step 3 is: Step 3.1, get the coordinates of all obstacles z(z∈[1,2,…,Z]) And the influence range R of all obstacles z ; Step 3.2, calculate the current coordinates of the drone The distance vector v = q to all obstacles z uav -q z and the distance d = ||v||2; Step 3.3: The influence range R of all obstacles obtained in step 3.1 z And the distance vector v=q of all obstacles z of the drone obtained in step 3.2 uav -q z And the distance d = ||v||2, calculate and record the repulsive force exerted by all obstacles z on the drone Among them, φ(x)=1 / (1+exp(-xβ0)) is the Sigmoid function, β0 is the temperature coefficient, k rep is the repulsion coefficient; Step 3.4, the repulsive force exerted on the drone by the obstacle z obtained in step 3.3 Calculate the sum of the repulsive forces of all obstacles z on the drone Step 3.5: Substitute the desired flight vector f of the drone obtained in step 2 exp The sum of the repulsive force f calculated in step 3.4 rep Find the vector sum and normalize it to get the actual flight direction vector of the drone after correction That is, a differentiable security module is obtained.

5. The unmanned aerial vehicle obstacle avoidance method based on reinforcement learning and differentiable programming according to claim 1, characterized in that: The specific method of step 4 is: Step 4.1, with the task neural network module constructed in step 2 in front and the differentiable safety module constructed in step 3 in the back, the two modules are combined to form the network structure of the drone obstacle avoidance strategy; Step 4.2, build the UAV simulation flight environment, including time and space parameters, UAV missions, UAV motion model and obstacles; Step 4.3, select any deterministic policy gradient reinforcement learning algorithm framework, replace the Actor network in the algorithm with the drone obstacle avoidance strategy network in step 4.1, and obtain a deterministic policy gradient reinforcement learning algorithm; Step 4.4, design the reward function required for training for the UAV mission; Step 4.5, set training hyperparameters; Step 4.6, based on the UAV simulation flight environment built in step 4.2, the reward function required for training designed in step 4.4, and the training hyperparameters set in step 4.5, use the deterministic policy gradient reinforcement learning algorithm obtained in step 4.3 to train the UAV obstacle avoidance strategy network obtained in step 4.

1. When the preset training reaches the preset training rounds, the UAV obstacle avoidance strategy network training is completed.

6. The unmanned aerial vehicle obstacle avoidance method based on reinforcement learning and differentiable programming according to claim 3 is characterized in that: In step 2, the deep neural network layer includes but is not limited to one or more of a multi-layer fully connected network, a convolutional network, a recurrent neural network, or an attention network.

7. The unmanned aerial vehicle obstacle avoidance method based on reinforcement learning and differentiable programming according to claim 4, characterized in that: In step 3, the influence range of the obstacle is determined by the outline of the specific obstacle. When the influence range is set, the influence range is greater than the distance from the obstacle coordinates to its farthest boundary.

8. A UAV obstacle avoidance system based on reinforcement learning and differentiable programming based on the method according to any one of claims 1 to 7, characterized in that: include: UAV state matrix s construction module, used to construct the UAV state matrix s; Task neural network module building module, used to obtain the expected flight vector f of the drone based on the constructed drone state matrix s exp ; Differentiable safety module building block to correct the desired flight vector f of the drone based on the obstacle position exp , get the actual flight direction vector f of the drone fly ; The UAV obstacle avoidance strategy network construction and training module is used to combine the two modules in a way that the task neural network module is in front and the differentiable safety module is in the back, to construct the UAV obstacle avoidance strategy network, build a UAV simulation flight environment, and train the UAV obstacle avoidance strategy network to obtain a usable UAV obstacle avoidance strategy.

9. An obstacle avoidance device for unmanned aerial vehicles based on reinforcement learning and differentiable programming, characterized in that: include: Memory: a computer program storing a method for avoiding obstacles for unmanned aerial vehicles based on reinforcement learning and differentiable programming as described in any one of claims 1 to 7, which is a computer-readable device; Processor: used to implement the drone obstacle avoidance method based on reinforcement learning and differentiable programming as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, can implement a drone obstacle avoidance method based on reinforcement learning and differentiable programming as described in any one of claims 1 to 7.

Images