Multi-unmanned aerial vehicle track dynamic planning and collaborative obstacle avoidance method based on reinforcement learning

Abstract

The invention discloses a multi-unmanned aerial vehicle track dynamic planning and collaborative obstacle avoidance method based on reinforcement learning, and the method comprises the steps: 1, initial environment construction: constructing an urban low-altitude environment three-dimensional grid map, and planning an initial four-dimensional track for each unmanned aerial vehicle; 2, conflict perception: in the operation process of the unmanned aerial vehicle, calculating conflict information between the unmanned aerial vehicle and surrounding obstacles and other unmanned aerial vehicles in real time; 3, conflict resolution: training an intelligent agent to learn an optimal avoidance strategy by using a time sequence neural network model and a reinforcement learning algorithm, and dynamically adjusting a track; and 4, track adjustment: after successful conflict resolution, backtracking an unmanned aerial vehicle action sequence, recording the position of the unmanned aerial vehicle in each time step, and obtaining a new unmanned aerial vehicle four-dimensional track. And finally, a flight route set meeting the unmanned aerial vehicle task requirement and the safety interval meeting the requirement is obtained. According to the invention, the operation efficiency of the unmanned aerial vehicle is improved, and a more intelligent, efficient, understandable and assessed cooperative obstacle avoidance strategy is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-UAV autonomous flight and coordinated obstacle avoidance, and specifically relates to a multi-UAV autonomous flight and coordinated obstacle avoidance method in structured airspace. Background Art

[0002] With the rapid advancement of drone technology, its application in multiple key areas such as logistics, rescue and disaster response has become increasingly in-depth, greatly promoting the prosperity of the drone industry and the diversification of flight needs. However, the contradiction between the surge in the number of drones and limited airspace resources has become increasingly prominent. In the management of urban low-altitude airspace, the traditional point-to-point free flight mode can no longer effectively cope with the increasingly complex safety and efficiency challenges. For this reason, drone trajectory planning and collaborative obstacle avoidance technology have become the key to solving the problem.

[0003] However, most of the existing UAV trajectory planning and obstacle avoidance methods have limitations, such as only considering point-to-point flight of UAVs, relying on trajectory prediction results to deal with conflicts between UAVs in isolation, large consumption of computing resources, insufficient detection sensitivity, trajectory jitter, and weak interpretability of obstacle avoidance behavior, etc., and less consideration is given to the coordination of autonomous trajectory planning, coordinated obstacle avoidance, and low-altitude airspace structure of UAVs. These problems limit the efficient and safe operation of UAVs in urban low-altitude areas with high obstacle density and complex and changing environment. Summary of the invention

[0004] In view of the above problems, the present invention proposes a method for dynamic trajectory planning and collaborative obstacle avoidance of multiple UAVs based on reinforcement learning, which aims to solve the problems of dynamic trajectory planning, collaborative operation, conflict detection and collaborative obstacle avoidance of large-scale UAVs in low-altitude complex operating environments. This method realizes efficient trajectory planning and autonomous obstacle avoidance of UAVs in low-altitude route networks by constructing three-dimensional grid maps, generating four-dimensional trajectories, real-time obstacle detection and reciprocal speed obstacle calculation, constructing a time series deep neural network model, executing and adjusting collaborative obstacle avoidance strategies, and dynamically adjusting and optimizing initial trajectories.

[0005] In order to achieve the above technical objectives and the above technical effects, the present invention is implemented through the following technical solutions:

[0006] A multi-UAV trajectory dynamic planning and collaborative obstacle avoidance method based on reinforcement learning, characterized by comprising the following steps:

[0007] S1. Constructing a 3D grid map: Conduct 3D modeling of the urban area, model the buildings and tree obstacles as their minimum enveloping cylinders, and construct a low-altitude 3D grid map of the city, including: a flyable grid and an obstacle grid;

[0008] S2. Generate initial four-dimensional trajectory: In the constructed urban low-altitude three-dimensional grid map, based on the UAV flight mission requirements, use the path planning algorithm to generate the initial four-dimensional trajectory of the UAV. The trajectory contains the sequence of the position of the UAV in three-dimensional space changing over time, that is, it takes into account both space and time dimensions;

[0009] S3. Real-time obstacle detection and reciprocal speed obstacle calculation: During the operation of the drone, the obstacle information within a certain range around each drone is obtained in real time, and the reciprocal speed obstacles between drones are calculated to predict potential collision risks; at the same time, conflicting buildings and reciprocal speed obstacles are prioritized according to the expected collision time to guide subsequent obstacle avoidance decisions;

[0010] S4. Construction and training of time series neural network model: Construct a time series neural network model, take the current state vector (including position, speed, acceleration, etc.) of the UAV, obstacle vector and the detected reciprocal speed obstacle vector as input; output the optimal control instructions (such as speed, heading angle, etc.) of the UAV in the next time step through multi-layer neural network processing to ensure that the safety interval between the UAV and obstacles and other UAVs meets the requirements; use reinforcement learning algorithm to train the model, and guide the model to learn the optimal obstacle avoidance strategy by defining a series of reward and punishment functions (such as rewards for reaching the target point, penalties for deviating from the route, severe penalties for mid-air collisions, and slight penalties for trajectory jitter);

[0011] S5. Collaborative obstacle avoidance strategy execution and adjustment: Use the trained neural network model to adjust the obstacle avoidance strategy and coordinate the obstacle avoidance actions according to the current status, priority, and relative position relationship of each drone to ensure that there is no collision between drones;

[0012] S6. Dynamic adjustment and optimization of the initial track: After the UAV performs the obstacle avoidance operation, the actual action of the UAV at each time step is traced back according to the output instructions of the time series neural network, including the adjustment of position, speed, heading, etc. By recording the adjusted state, the four-dimensional track of the UAV is reconstructed, that is, the complete sequence of the UAV position changes over time; finally, a track set is obtained that meets the requirements of the UAV mission while the safety interval between each UAV meets the requirements.

[0013] Optionally, in step S1, the method for constructing a low-altitude three-dimensional grid map of a city includes:

[0014] S11. Based on laser radar, geographic information system GIS data, and GPS data, the coordinate information of three-dimensional objects in the airspace is collected, and the collected data is converted into a coordinate system;

[0015] S12. Model the obstacles in the airspace as their minimum envelope cylinder to facilitate subsequent calculations and reduce the complexity of conflict detection;

[0016] S13, determining the grid size, dividing the three-dimensional space into a number of cubic grids of equal size; each grid represents a spatial unit, and is used to store the environmental information in the unit;

[0017] S14. Assign values ​​to the grid, where 0 represents the flyable airspace, 1 represents the obstacle area, and the value between 0 and 1 represents the risk value of the grid.

[0018] Optionally, in step S2, the method for constructing the initial four-dimensional trajectory of the UAV includes:

[0019] S21. Obtain the drone operation plan, including the start and end points, expected take-off time, expected arrival time, risk preference (shortest path, lowest risk, minimum delay, etc.);

[0020] S22. Select different trajectory planning algorithms (A*, Theta*, etc.), consider the flight performance constraints of the UAV, set different weight heuristic functions according to risk preferences, and link the start and end points of the route in the three-dimensional grid map;

[0021] S23. Calculate the time t when the UAV arrives at the waypoint on the route according to the expected running speed v_del of the UAV, and generate a waypoint coordinate-arrival time sequence (x, y, z, t).

[0022] Optionally, in step S3, the real-time obstacle detection and reciprocal speed obstacle calculation method specifically includes:

[0023] S31, selecting obstacles within a certain range of the current drone and all drones within the range;

[0024] S32, respectively calculating the speed obstacle vectors of the current UAV and all UAVs within the range;

[0025] The area where a drone may collide with other drones when it maintains its current speed or moves within a certain speed range is defined as a set of velocity vectors. When a drone adopts a velocity vector within this set, a collision will occur. Specifically:

[0026] Select obstacle information within the range of 45° to the left and right of the (x,y) plane projection vector, 50m away, and 10m in height difference from the current UAV's running direction;

[0027] Select a drone within 50m of the current drone and calculate its speed obstacle vector:

[0028]

[0029] Among them, P A , PB represents the position vector of the current UAV and other UAVs, pra and prb represent the priority of the two aircraft, V A 、V B represents the velocity vectors of the two aircraft, t is the time step, a is the time scaling factor, r A 、r B Represents the collision radius of the two aircraft, P vo is the starting point of the speed barrier cone. is the center vector of the speed obstacle cone, α is the opening angle of the speed obstacle cone, and the speed obstacle vector of the UAV can be expressed as:

[0030]

[0031] The obstacle vector can be expressed as: Where (x, y) is the coordinate of the obstacle, h is the height of the obstacle, and r is the radius of the obstacle after cylindrical modeling;

[0032] S33, calculating the estimated collision time between the current UAV and the obstacle and all speed obstacles, and sorting the obstacles according to the urgency of collision avoidance.

[0033] Among them, the calculation method of the expected collision time between drones can be calculated as follows:

[0034] Assume that the initial relative position of the two drones is (relx, rely, relz), the initial relative speed is (relvx, relvy, relvz), and the radius is r a 、r b , the relative distance at time t is d(t), then:

[0035]

[0036] At the moment of collision, d(t) = r = r a +r b , then:

[0037]

[0038] Square both sides to get:

[0039] r 2 =(relx+relvx*t) 2 +(rely+relvy*t) 2 +(relz+relvz*t) 2

[0040] Arranged to get the quadratic equation about t:

[0041] at 2 +bt+c=0

[0042] Where a = relvx 2 +relvy 2 +relvz 2 , b=2(relx*relvx+rely*relvy+relz*relvz), c=relx 2 +rely 2 +relz 2 -r 2 .

[0043] If c≤0, the two aircraft are flying in opposite directions. If the discriminant b 2 -4ac≤0, there is no risk of collision. If it is greater than 0, use the root-finding formula Solve, choose the smallest value greater than or equal to 0 among the two roots as the expected collision time.

[0044] Optionally, in step S4, the time series neural network model takes into account time dependence and spatial correlation, and captures the dynamic changes of the drone state and observation space over time by introducing structures such as recurrent neural network RNN ​​or long short-term memory network LSTM, and outputs drone control instructions; the model training process adopts a combination of simulated environment and actual flight data to improve the generalization ability and robustness of the model; at the same time, through continuous iterative optimization, the model can achieve efficient and safe autonomous navigation and obstacle avoidance in complex and changeable urban environments.

[0045] Among them, the method for building a deep learning model based on time series neural network is:

[0046] S4.1.1, data acquisition: obtain the current state observation vector of the drone obs_self = [p, v, r, pr, v_des], where p is the current three-dimensional position coordinate of the drone, v is the current speed of the drone, r is the collision detection radius of the drone, pr is the priority of the drone, and v_des is the expected running speed of the drone;

[0047] Get all obstacle observation vectors within the drone detection range in, is the single obstacle observation vector, and n is the total number of obstacles in the observation area;

[0048] Get the speed obstacle observation vector of other drones around the drone in, is a single speed obstacle vector, and m is the total number of speed obstacles in the observation area;

[0049] Link the drone's current state observation vector, obstacle observation vector, and speed obstacle observation vector to obtain the drone's observation vector of the environment at that moment:

[0050] obs=[obs_self,obs_barrier,obs_vo]

[0051] S4.1.2. Construct a time series neural network: Select a neural network suitable for processing time series data (such as LSTM or BiGRU) and use the environmental observation vector as the input of the time series neural network, obtain the forward information h_forward or reverse information h_back (if any) of the time series neural network, and concatenate them to form an output containing complete context information;

[0052] S4.1.3. Deep neural network construction: Use the concatenated time series neural network output as the input of the deep neural network;

[0053] The deep neural network should contain: normalization layer, fully connected layer, and optional convolution layer;

[0054] The output of the deep neural network is the acceleration of the drone in three dimensions (ax, ay, az).

[0055] Optionally, use a reinforcement learning algorithm to train a deep neural network model as follows:

[0056] S4.2.1. Obtaining the state of the drone and the observation vector: The drone’s current state observation vector obs_self, obstacle observation vector obs_barrier and speed obstacle observation vector obs_vo are used as the state input of the reinforcement learning model;

[0057] S4.2.2, Policy Network and Value Network: Use the above-mentioned time series neural network as part of the policy network to process time series data and capture environmental dynamics. Receive environmental observations as input and output the control action of the drone (such as acceleration). If a value-based reinforcement learning algorithm is used, in addition to the policy network, a value network is also required to evaluate the future expected reward for taking an action in a given state. It also receives environmental observations as input and outputs the value estimate of the state-action pair;

[0058] S4.2.3, Penalty items: According to the operation objectives of the UAV, speed obstacle zone intrusion, route deviation, delay, collision are selected as penalties and waypoint arrival and destination arrival are selected as rewards; since delay, collision and arrival at the target position can be calculated by time and coordinate distance difference, only the calculation methods for speed obstacle zone intrusion and route deviation are given.

[0059] (1) The calculation method of the invasion speed barrier area is as follows:

[0060] Assume the current drone is A, the related drone is B, and the speed obstacle of B relative to A is defined as:

[0061]

[0062] Among them, P vo is the starting point of the speed barrier cone. is the center vector of the speed barrier cone, and α is the opening angle of the speed barrier cone; the calculation method is:

[0063]

[0064] Among them, P A , P B represents the position vector of the current UAV and other UAVs, pra and prb represent the priority of the two aircraft, V A 、V B represents the velocity vectors of the two aircraft, t is the time step, a is the time scaling factor, r A 、r B Represents the radius of the collision zone between the two aircraft.

[0065] Assume that the current state vector of drone A is: state = [p, v, r, pr, v_des], where p is the current three-dimensional position coordinate of the drone, v is the current speed of the drone, r is the collision detection radius of the drone, pr is the priority of the drone, and v_des is the expected running speed of the drone.

[0066] Then the position of drone A after at time steps is: p new =p+v*at, at this time p new The starting vector of the speed obstacle is The angle β between it and the center vector of the speed obstacle cone can be calculated as:

[0067]

[0068] The sizes of the speed obstacle cone angles α and β are judged: if α≥β, it is considered that the two aircraft have a risk of intruding into each other's speed obstacle; if α<β, it is considered that there is no risk of collision between the two aircraft.

[0069] (2) The method for calculating the degree of deviation from the route is as follows:

[0070] Assume that a direct flight route is defined by the starting point s = (x1, y1, z1) and the end point e = (x2, y2, z2). At this moment, the position of the drone p = (x0, y0, z0). The route direction vector can be calculated as:

[0071] d=es=(x2-x1,y2-y1,z2-z1)=(dx,dy,dz)

[0072] Normalize the direction vector:

[0073]

[0074] in, Calculate the projection length from the drone position to the route starting point vector:

[0075]

[0076] Therefore, the closest point to the drone on the route is:

[0077]

[0078] The deviation degree of the drone is the distance from the drone to the nearest point:

[0079]

[0080] S4.2.4, Strategy optimization: Assign different weights to the penalty terms and add them up (or other processing methods) as the penalty function, with the goal of minimizing the penalty function and maximizing the cumulative reward, and use the policy gradient algorithm to update the strategy.

[0081] If you use a value-based reinforcement learning algorithm, you need to update the value network. The update of the value network depends on the action output by the policy network and the reward and next state obtained from the environment;

[0082] S4.2.5, Experience replay: During the interaction between the drone and the environment, the state, action, reward and next state of each time step are stored in the experience pool. Then, a batch of experience is randomly sampled during training to update the network;

[0083] S4.2.6, Model deployment: Train the model in a predetermined simulation scenario, and deploy it on a drone when the model performance meets the requirements, and train it in the actual environment.

[0084] S4.2.7, trajectory adjustment: When all UAVs have completed a route without collision, delay, and deviation from the scheduled route to the extent that meets the requirements, the status of all UAVs is backtracked to form a new position-time four-dimensional trajectory set.

[0085] Optionally, in step S6, the specific method for dynamically adjusting and optimizing the initial track is: when the UAV is running, the position of the UAV is recorded at each time step. If, after all UAVs arrive at the target point, there is no collision between UAVs in the route network or the safety interval does not meet the requirements, and the arrival time of the UAV and the degree of deviation from the initial track meet the requirements, then the position-time series of all UAVs are used as a new four-dimensional track set.

[0086] Compared with the prior art, the present invention has the following beneficial effects:

[0087] The present invention constructs an urban low-altitude route network to plan the optimal route for drones, and at the same time combines deep neural networks with speed obstacle methods to build a conflict detection-release model, making full use of information such as route network structure and drone operation plan, aiming to improve the sensitivity and efficiency of conflict detection, reduce the computational complexity of release action, and use reinforcement learning algorithm to train the model to achieve collaborative obstacle avoidance among multiple drones and optimization of the overall flight path, and provide interpretability and evaluability for drone release behavior. While improving the operating efficiency of drones, a more intelligent, efficient, understandable and evaluable collaborative obstacle avoidance strategy is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor, among which:

[0089] Figure 1 A flowchart of a multi-UAV trajectory dynamic planning and collaborative obstacle avoidance method based on reinforcement learning according to an embodiment of the present invention;

[0090] Figure 2 A schematic diagram of a time series neural network structure according to an embodiment of the present invention;

[0091] Figure 3 This is the initial environment constructed in one embodiment of the present invention, where the blue cylinder represents the modeled obstacle, the green dot represents the starting point of the drone, the red dot represents the end point of the drone, the black x represents the waypoint on the track, and the black line segment represents the track of the drone;

[0092] Figure 4 Schematic diagram of a scenario in which a reinforcement learning method is used to train an intelligent agent in an embodiment of the present invention, wherein modeled obstacles are represented by cylinders, waypoints are represented by red dots, labels such as 0, 1, 2, etc. represent the order of waypoints, and a green triangle represents the current position of the drone. DETAILED DESCRIPTION

[0093] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0094] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0095] Example 1

[0096] In an embodiment of the present invention, a multi-UAV trajectory dynamic planning and collaborative obstacle avoidance method based on reinforcement learning is provided. Figure 1 As shown, the following steps are included:

[0097] (1) Generate a low-altitude three-dimensional grid map of the city based on geographic information data or sensor data;

[0098] (2) Based on the UAV flight mission requirements and performance constraints, a path planning algorithm is selected to generate a four-dimensional trajectory.

[0099] (3) Obtain obstacle information within a certain range around the UAV in real time, calculate the reciprocal speed obstacle model with other UAVs, and arrange the order of reciprocal speed obstacles according to the expected collision time.

[0100] (4) Construct a time series neural network model, use the current state vector of the UAV, obstacle vector, and reciprocal speed obstacle vector as the input of the time series neural network, and output the control instructions for the next time step of the UAV. Use the reinforcement learning algorithm to train the model and guide the model to learn the optimal obstacle avoidance strategy;

[0101] (5) Use the trained neural network model to adjust the obstacle avoidance strategies between drones and coordinate the obstacle avoidance actions to ensure that there are no collisions between drones. By backtracking the collision avoidance trajectories of the drones, the four-dimensional trajectory of the drones is reconstructed, that is, the complete sequence of the drone position changes over time. Finally, a set of trajectories is obtained that meets the requirements of the drone mission while ensuring that the safety intervals between drones meet the requirements.

[0102] In a specific embodiment of the present invention, the method for constructing a low-altitude three-dimensional grid map of a city includes:

[0103] Based on geographic information data, the urban low-altitude airspace is divided into flyable airspace, obstacles, dangerous areas, etc.;

[0104] The obstacle is modeled as its minimum envelope cylinder, which is represented by (x, y, h, r), where x and y are two-dimensional plane coordinates, h is the height, and r is the radius;

[0105] The three-dimensional map is rasterized, the flyable airspace is assigned a value of 0, obstacles are assigned a value of 1, and other grids are assigned a value of 0-1 (representing their risk value).

[0106] The four-dimensional trajectory construction method is:

[0107] Based on the flight performance constraints of the UAV, a three-dimensional path planning algorithm suitable for discretized airspace is selected to connect the starting point and the end point of the UAV. Based on the flight plan and the expected flight speed of the UAV, the time it takes for the UAV to reach a series of waypoints is calculated to form a waypoint coordinate-time series (x, y, z, t).

[0108] The current state vector, obstacle vector and reciprocal speed obstacle vector are obtained by:

[0109] Current state vector: Get the current state observation vector of the drone obs_self = [p, v, r, pr, v_des], where p is the current three-dimensional position coordinate of the drone, v is the current speed of the drone, r is the collision detection radius of the drone, pr is the priority of the drone, and v_des is the expected running speed of the drone;

[0110] Obstacle observation vector: Select obstacle information within the range of 45° to the left and right of the current UAV running direction and the (x, y) plane projection vector, 50m away, and 10m in height difference. The obstacle vector can be expressed as: Where (x, y) is the coordinate of the obstacle, h is the height of the obstacle, and r is the radius of the obstacle after cylindrical modeling;

[0111] All obstacle observation vectors in, is the single obstacle observation vector, and n is the total number of obstacles in the observation area;

[0112] Reciprocal speed obstacle vector: Select a drone within 50m of the current drone and calculate its speed obstacle vector as follows:

[0113]

[0114] Among them, P A , P B represents the position vector of the current UAV and other UAVs, pra and prb represent the priority of the two aircraft, V A 、V B represents the velocity vectors of the two aircraft, t is the time step, a is the time scaling factor, r A 、r BRepresents the collision radius of the two aircraft, P vo is the starting point of the speed barrier cone. is the center vector of the speed obstacle cone, and α is the opening angle of the speed obstacle cone.

[0115] Then the UAV speed obstacle vector can be expressed as:

[0116]

[0117] The method for calculating the estimated collision time between the drones can be calculated as follows:

[0118] Assume that the initial relative position of the two drones is (relx, rely, relz), the initial relative speed is (relvx, relvy, relvz), and the radius is r a 、r b , the relative distance at time t is d(t), then:

[0119]

[0120] At the moment of collision, d(t) = r = r a +r b , then:

[0121]

[0122] Square both sides to get:

[0123] r 2 =(relx+relvx*t) 2 +(rely+relvy*t) 2 +(relz+relvz*t) 2

[0124] Arranged to get the quadratic equation about t:

[0125] at 2 +bt+c=0

[0126] Where a = relvx 2 +relvy 2 +relvz 2 , b=2(relx*relvx+rely*relvy+relz*relvz), c=relx 2 +rely 2 +relz 2 -r 2 .

[0127] If c≤0, the two aircraft are flying in opposite directions. If the discriminant b 2-4ac≤0, there is no risk of collision. If it is greater than 0, use the root-finding formula Solve, choose the smallest value greater than or equal to 0 among the two roots as the expected collision time.

[0128] like Figure 2 As shown, the basic structure of the time series neural network is:

[0129] Link the drone's current state observation vector, obstacle observation vector, and speed obstacle observation vector to obtain the drone's observation vector of the environment at that moment:

[0130] obs=[obs_self,obs_barrier,obs_vo]

[0131] Select a neural network suitable for processing time series data, use the above-mentioned environmental observation vector obs as the input of the time series neural network, obtain the forward information h_forward or reverse information h_back (if any) of the time series neural network, and concatenate them to form observation information o = [h_forward, h_back] containing complete context information as the input of another deep neural network. The deep neural network should include: a normalization layer, a fully connected layer, and an optional convolutional layer, and the output is the acceleration of the drone in three-dimensional directions (ax, ay, az).

[0132] The method of training a deep learning neural network model using a reinforcement learning method includes:

[0133] (1) The current state observation vector of the UAV, the obstacle observation vector, and the speed obstacle observation vector are used as the state input of the reinforcement learning model.

[0134] (2) The above time series neural network is used as part of the policy network to process time series data and capture environmental dynamics. The policy network receives environmental observations as input and outputs the control action (acceleration) of the drone.

[0135] If you use a value-based reinforcement learning method, in addition to the policy network, you also need a value network to evaluate the future expected reward of taking an action in a given state. The value network receives environmental observations as input and outputs an estimate of the value of the state-action pair.

[0136] (3) Using the experience replay method, the state, action, reward, and next state of each time step are stored in the experience pool during the interaction between the drone and the environment. During the training process, a batch of experience is randomly sampled to update the network.

[0137] (4) Considering the deviation penalty, collision penalty, and arrival reward, the reward function is set. According to the goal of reinforcement learning, that is, maximizing the cumulative reward, the parameters of the policy network are updated through the policy gradient algorithm.

[0138] If you use a value-based reinforcement learning method, in addition to updating the policy network, you also need to update the value network. The update of the value network depends on the action output by the policy network and the reward and next state obtained from the environment.

[0139] (5) Set up a simulation scenario based on the planned four-dimensional route network, deploy the model in the scenario for training, and regularly evaluate the performance of the model on the test set to ensure that it can generalize to unseen environments. If the model performance meets the requirements, it can be deployed on the drone to make real-time decisions in the actual environment.

[0140] The intrusion speed obstacle zone is determined as follows:

[0141] Assume the current drone is A, the related drone is B, and the speed obstacle of B relative to A is defined as:

[0142]

[0143] Among them, P vo is the starting point of the speed barrier cone. is the center vector of the speed barrier cone, and α is the opening angle of the speed barrier cone; the calculation method is:

[0144]

[0145] Among them, P A , P B represents the position vector of the current UAV and other UAVs, pra and prb represent the priority of the two aircraft, V A 、V B represents the velocity vectors of the two aircraft, t is the time step, a is the time scaling factor, r A 、r B Represents the radius of the collision zone between the two aircraft.

[0146] Assume that the current state vector of drone A is: state = [p, v, r, pr, v_des], where p is the current three-dimensional position coordinate of the drone, v is the current speed of the drone, r is the collision detection radius of the drone, pr is the priority of the drone, and v_des is the expected running speed of the drone.

[0147] Then the position of drone A after at time steps is: p new =p+v*at, at this time p new The starting vector of the speed obstacle is The angle β between it and the center vector of the speed obstacle cone can be calculated as:

[0148]

[0149] The sizes of the speed obstacle cone angles α and β are judged: if α≥β, it is considered that the two aircraft have a risk of intruding into each other's speed obstacle; if α<β, it is considered that there is no risk of collision between the two aircraft.

[0150] The method for calculating the degree of deviation from the route is as follows:

[0151] Assume that a direct flight route is defined by the starting point s = (x1, y1, z1) and the end point e = (x2, y2, z2). At this moment, the position of the drone p = (x0, y0, z0). The route direction vector can be calculated as:

[0152] d=es=(x2-x1,y2-y1,z2-z1)=(dx,dy,dz)

[0153] Normalize the direction vector:

[0154]

[0155] in, Calculate the projection length from the drone position to the route starting point vector:

[0156]

[0157] Therefore, the closest point to the drone on the route is:

[0158]

[0159] The deviation degree of the drone is the distance from the drone to the nearest point:

[0160]

[0161] The method in the embodiment of the present invention is described in detail below in conjunction with a specific implementation manner.

[0162] (1) The urban low-altitude airspace of 500*500*50 (unit: meter) is processed into a grid environment of 100*100*10. The four drones performing missions in this airspace and their expected take-off and arrival schedules are shown in the table:

[0163] serial number starting point end 1 (3，3，3) (95，85，8) 2 (95，5，5) (25，83，6) 3 (92，27，7) (3，26，3) 4 (33，89，6) (5，25，3)

[0164] (2) Use the Theta* algorithm to link the starting point and end point of the drone in the grid map, such as Figure 3 shown.

[0165] (3) Taking DJI Matrice 600Pro as an example, some of its parameters are as follows:

[0166] Table 1 UAV parameters and protection box dimensions

[0167]

[0168]

[0169] In order to facilitate calculation and improve the running speed of the program, the positioning accuracy and external dimensions are comprehensively considered, and the size of the protection box is rounded to an integer, and the maximum external protection sphere diameter r = 5m is obtained, and the running speed is taken as 15m / s.

[0170] (4) Extract waypoints to enable each drone to take off according to the schedule, such as Figure 4 As shown, the green triangle represents the drone, the red dot represents the waypoint, the number represents the order of the waypoint, and the cylinder represents the obstacle after cylindrical modeling.

[0171] (5) Get the current drone observation vector. Take the drone numbered 1 in the second time step after the program starts as an example: its position vector is: [19, 21, 30], its velocity vector is [1.34, 1.48, 0.1] (unit: 5m), the radius is 5m, the priority is 1, and the expected velocity vector is [1.34, 1.5, 0.1] (unit: 5m). Therefore, its own observation vector is:

[0172] obs_self=[19.0,21.0,3.0,1.34,1.48,0.1,1.0,5.0,1.32,1.5,0.1]

[0173] Since other drones are out of the observation range, the speed obstacle observation vector is empty at this time:

[0174]

[0175] The obstacle observation vector is:

[0176] obs_barrier = [10,17,8,4]

[0177] Link the above observation vectors and input them into Figure 2 In the LSTM-based neural network shown:

[0178] obs=[obs_self,obs_barrier,obs_vo]

[0179] At the same time, the drone has not collided with obstacles or other drones, and only the penalty caused by the difference between its velocity vector and the expected velocity vector is calculated.

[0180] (6) Update the drone position based on the neural network output action.

[0181] (7) Update drone observations and update drone penalties until the drone reaches the target location / collides with an obstacle or other drones.

[0182] (8) Repeat the above process, perform multiple sampling, put the samples into the deep learning algorithm experience pool, and use the deep learning algorithm to update the neural network parameters until the drone can reasonably avoid obstacles and other drones and the operating cost is acceptable.

[0183] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

[0184] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A multi-UAV trajectory dynamic planning and collaborative obstacle avoidance method based on reinforcement learning, characterized in that: The steps include: S1. Constructing a 3D grid map: Conduct 3D modeling of the urban area, model the buildings and tree obstacles as their minimum enveloping cylinders, and construct a low-altitude 3D grid map of the city, including: a flyable grid and an obstacle grid; S2. Generate initial four-dimensional trajectory: In the constructed urban low-altitude three-dimensional grid map, based on the UAV flight mission requirements, use the path planning algorithm to generate the initial four-dimensional trajectory of the UAV. The trajectory contains the sequence of the position of the UAV in three-dimensional space changing over time, that is, it takes into account both space and time dimensions; S3. Real-time obstacle detection and reciprocal speed obstacle calculation: During the operation of the drone, the obstacle information within a certain range around each drone is obtained in real time, and the reciprocal speed obstacles between drones are calculated to predict potential collision risks; at the same time, conflicting buildings and reciprocal speed obstacles are prioritized according to the expected collision time to guide subsequent obstacle avoidance decisions; S4. Construction and training of time series neural network model: Construct a time series neural network model, take the current state vector of the UAV, obstacle vector and the detected reciprocal speed obstacle vector as input; output the optimal control command of the UAV in the next time step through multi-layer neural network processing to ensure that the safety interval between the UAV and obstacles and other UAVs meets the requirements; use reinforcement learning algorithm to train the model to guide the model to learn the optimal obstacle avoidance strategy; S5. Collaborative obstacle avoidance strategy execution and adjustment: Use the trained neural network model to adjust the obstacle avoidance strategy and coordinate the obstacle avoidance actions according to the current status, priority, and relative position relationship of each drone to ensure that there is no collision between drones; S6. Dynamic adjustment and optimization of the initial track: After the UAV performs the obstacle avoidance operation, the actual action of the UAV at each time step is traced back according to the output instructions of the time series neural network. By recording the adjusted state, the four-dimensional track of the UAV is reconstructed, that is, the complete sequence of the UAV position changes over time. Finally, a set of tracks is obtained that meets the requirements of the UAV mission while the safety interval between each UAV meets the requirements.

2. According to the method of claim 1, the method is characterized by: In step S1, the method for constructing a low-altitude three-dimensional grid map of a city includes: S11. Based on laser radar, geographic information system GIS data, and GPS data, the coordinate information of three-dimensional objects in the airspace is collected, and the collected data is converted into a coordinate system; S12. Model the obstacles in the airspace as their minimum envelope cylinder to reduce the complexity of conflict detection; S13, determining the grid size, dividing the three-dimensional space into a number of cubic grids of equal size; each grid represents a spatial unit, and is used to store the environmental information in the unit; S14. Assign values ​​to the grid, where 0 represents the flyable airspace, 1 represents the obstacle area, and the value between 0 and 1 represents the risk value of the grid.

3. The method for dynamic trajectory planning and coordinated obstacle avoidance of multiple UAVs based on reinforcement learning according to claim 1 is characterized in that: In step S2, the method for constructing the initial four-dimensional trajectory of the UAV includes: S21. Obtain the drone operation plan, including the start and end points, expected take-off time, expected arrival time, and risk preference; S22. Select different trajectory planning algorithms, consider the flight performance constraints and mission requirements of the UAV, set different weight heuristic functions according to risk preferences, and link the starting and ending points of the route in the three-dimensional grid map; S23. Calculate the time t when the UAV arrives at the waypoint on the route according to the expected running speed v_del of the UAV, and generate a waypoint coordinate-arrival time sequence (x, y, z, t).

4. The method for dynamic trajectory planning and coordinated obstacle avoidance of multiple UAVs based on reinforcement learning according to claim 1, characterized in that: In step S3, the real-time obstacle detection and reciprocal speed obstacle calculation method specifically includes: S31, selecting obstacles within a certain range of the current drone and all drones within the range; S32, respectively calculating the speed obstacle vectors of all UAVs within the range relative to the current UAV; S33, calculating the estimated collision time between the current UAV and the obstacle and all speed obstacles, and sorting the obstacles according to the urgency of collision avoidance.

5. The method for dynamic trajectory planning and coordinated obstacle avoidance of multiple UAVs based on reinforcement learning according to claim 4 is characterized in that: In step S32, the area where the drone may collide with other drones when maintaining the current speed or moving within a certain speed range is defined as a series of speed vector sets. When the drone adopts the speed vector in the set, it will cause a collision; The velocity obstacle vector is calculated as follows: Among them, P A , P B represents the position vector of the current UAV and other UAVs, pra and prb represent the priority of the two aircraft, V A 、V B represents the velocity vectors of the two aircraft, t is the time step, a is the time scaling factor, r A 、r B Represents the radius of the collision area between the two aircraft, P vo is the starting point of the speed barrier cone. is the center vector of the speed obstacle cone, and α is the opening angle of the speed obstacle cone. Therefore, the speed obstacle vector of the drone is expressed as: The obstacle vector is expressed as: Among them, (x, y) is the coordinate of the obstacle, h is the height of the obstacle, and r is the radius of the obstacle after cylindrical modeling.

6. The method for dynamic trajectory planning and coordinated obstacle avoidance of multiple UAVs based on reinforcement learning according to claim 1, characterized in that: In step S4, the time series neural network model takes into account time dependence and spatial correlation, and captures the dynamic changes of the drone state and observation space over time by introducing structures such as recurrent neural network RNN ​​or long short-term memory network LSTM, and outputs drone control instructions; the model training process adopts a combination of simulated environment and actual flight data to improve the generalization ability and robustness of the model; at the same time, through continuous iterative optimization, the model can achieve efficient and safe autonomous navigation and obstacle avoidance in complex and changeable urban environments.

7. The method for dynamic trajectory planning and coordinated obstacle avoidance of multiple UAVs based on reinforcement learning according to claim 1, characterized in that: In step S6, the specific method of dynamically adjusting and optimizing the initial track is: when the UAV is running, the position of the UAV is recorded at each time step. If, after all UAVs arrive at the target point, there is no collision between UAVs in the route network or the safety interval does not meet the requirements, and the arrival time of the UAV and the degree of deviation from the initial track meet the requirements, then the position-time series of all UAVs are used as a new four-dimensional track set.

Images