Unmanned aerial vehicle double-layer conflict resolution method for urban low-altitude dense traffic flow airspace

Abstract

The application discloses a UAV double-layer conflict resolution method for urban low-altitude dense traffic flow airspace, comprising the following steps: before the takeoff of an intelligent UAV, a mixed integer nonlinear programming model based on a conflict resolution strategy is constructed, and the mixed integer nonlinear programming model is optimized by using an improved random fractal search algorithm; after the takeoff of the intelligent UAV, a double deep Q network is constructed and network training is performed on the double deep Q network, and the trained double deep Q network is used to make the intelligent UAV output conflict-free maneuvering actions at each time step. The application proposes a double-layer conflict resolution framework in combination with deep reinforcement learning, combines pre-flight conflict resolution with real-time conflict resolution, greatly reduces the occurrence of air collisions compared with any single method, enables the intelligent UAV to avoid current obstacles in an ultra-low-altitude urban dense traffic flow environment, reduces secondary conflicts, and significantly shortens the training time of a real-time conflict resolution method based on reinforcement learning.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and urban air traffic systems, and particularly to a UAV double-layer conflict resolution method for urban low-altitude dense traffic flow airspace. BACKGROUND

[0002] Unmanned aerial vehicles are also known as drones. The development of drone technology has enabled its application to extend from the military field to entertainment, agriculture, logistics, and even construction and other fields. These applications have brought many benefits to society, but at the same time, they have also introduced a large amount of air traffic into the urban area of the ultra-low airspace. Without proper traffic management, heavy air traffic in the ultra-low airspace can pose a great risk to property and pedestrians on the ground. Major aviation regulatory agencies have proposed to develop traffic management frameworks aimed at ensuring the safe and efficient operation of drones in the ultra-low airspace. These frameworks under development all have some common key modules, one of which is autonomous conflict and resolution. There are currently many studies dedicated to developing the functional module of autonomous conflict resolution, but there are still three major limitations.

[0003] (1) Most existing studies focus on solving primary conflicts, that is, simply avoiding obstacles. However, in the future ultra-low airspace, traffic throughput is expected to be much higher than current manned air traffic, possibly by several orders of magnitude. When air traffic density increases, solving primary conflicts alone can cause more secondary conflicts, that is, drones may encounter new conflicts after avoiding the current obstacles. In addition, this means that drones need to perform more obstacle avoidance operations, thereby increasing the instability of the entire airspace.

[0004] (2) Secondary conflicts are an important indicator of airspace stability, but so far there has been no real-time quantitative modeling method for secondary conflicts. At the same time, most of the obstacle avoidance functional modules currently rely on traditional optimization methods or geometric algorithms, which limits their processing capacity when faced with a large number of obstacles and makes it difficult to effectively reduce secondary conflicts.

[0005] (3) Most current conflict resolution methods focus on a single flight phase, such as before or after takeoff, and there is no research on how to integrate conflict resolution in both phases into an integrated framework to improve the overall safety of drone flights in the ultra-low airspace. According to existing research, relying solely on conflict detection and resolution before takeoff, complex weather conditions (such as strong winds) or mechanical limitations can cause drones to deviate from the planned path, thereby increasing the risk of conflict. If conflict detection and resolution after takeoff are relied upon, learning-based algorithms are usually used to ensure computational efficiency. However, for such a complex and highly uncertain ultra-low airspace, the time required for pre-training increases exponentially with the size of the airspace, and it is difficult to achieve effective conflict resolution. SUMMARY

[0006] The present application aims at providing a UAV double-layer conflict resolution method for urban low-altitude dense traffic airspace.

[0007] Technical scheme: The UAV double-layer conflict resolution method for urban low-altitude dense traffic airspace comprises the following steps:

[0008] Before the intelligent UAV takes off, a mixed integer nonlinear programming model based on conflict resolution strategy is constructed, and the mixed integer nonlinear programming model is optimized by using an improved random fractal search algorithm to obtain the initial parameters of the intelligent UAV taking off;

[0009] After the intelligent UAV takes off, a double deep Q network is constructed and network training is performed, the influence of surrounding neighbors on the intelligent UAV is simulated through an introduced attention mechanism, and the intelligent UAV outputs conflict-free maneuvering actions at each time step by using the trained double deep Q network until the intelligent UAV reaches the final destination.

[0010] Further, the process of constructing the mixed integer nonlinear programming model based on conflict resolution strategy comprises the following steps:

[0011] Based on the flight plan and real-time flight data of the intelligent UAV, an initial four-dimensional flight path is constructed for all UAVs in the airspace;

[0012] Whether there is a flight conflict is detected for the four-dimensional flight path, and a solution strategy is assigned to each detected flight conflict;

[0013] The optimization objective function of the i-th flight conflict is defined as f i (ζ i ), wherein ζ i is a feasible solution in the search space, wherein is a decision variable of the conflict resolution strategy, respectively representing the position, take-off time and cruising speed of the intelligent UAV, and the expression of the objective function is:

[0014] min:f obj =ω risk R CP +ω t_delay T delay +ω t_air T air

[0015] In the formula, ω risk , ω t_delay , ω t_air are weight factors of risk, delay and flight time respectively, and ω risk +ωt_delay +ω t_air =1,R CP represents the potential conflict risk, T delay represents the total delay time of all UAVs, T air represents the total flight time of all UAVs;

[0016] The constraint condition for constructing the objective function is represented as:

[0017]

[0018]

[0019] In the formula, represents the actual take-off time of the UAV U g , represents the actual flight time of the UAV U g , represents the scheduled take-off time of the UAV U g , represents the scheduled flight time of the UAV U g in the air, represents the delay time of the UAV U g , uav represents the total number of UAVs; N p represents the total number of adjacent waypoint pairs, d ij represents the Euclidean distance of the arc J ij , represents the battery endurance time, represents the number of UAVs starting from the starting point, represents the number of UAVs reaching the end point, represents the cruising speed of the UAV U g on the arc J ij , and respectively represent the minimum and maximum values of the cruising speed of the UAV;

[0020] The mixed integer nonlinear programming model is composed of the objective function and the constraint condition.

[0021] Further, the process of optimizing the mixed integer nonlinear programming model by using the improved random fractal search algorithm includes the following steps:

[0022] Step 101, assuming that a body particle is regarded as a potential solution, the initialization of the particle is randomly performed within the problem condition constraint, which is represented as:

[0023] P i =B lower +λ(B upper -B lower )

[0024] where P i represents the i th initial particle in the population, B lower and B upper represent the lower and upper limits of the constraint vector respectively, and λ is a random number and λ ∈ [0, 1];

[0025] Step 102, new particles are generated in the search space by Gaussian random walk distribution, denoted as:

[0026]

[0027] where P i η is the i th new particle generated by the particle P i , η is the number of new particles generated, μ P is the Gaussian distribution mean of the new particle position, μ P = |P i |, σ is the standard deviation, P best is the optimal particle in the current group, and are weight factors for adjusting the degree of development and exploration, obeying uniform distribution, ranging from [0, 1];

[0028] Step 103, all generated particles are evaluated by the fitness function, and the solution particles with fitness values greater than the threshold value are kept unchanged, and the remaining solution particles are updated when the following conditions are met, the conditions are expressed as:

[0029]

[0030] where P ai represents the probability of the particle being updated, rank(P i ) represents the ranking of the particle after being evaluated by the fitness function, n p represents the total number of particles;

[0031] Step 104, the position of the best particle in the population is recorded as X best , the distance value d Pi between the updated solution particle and the best particle is calculated, and all distance values are summarized as a distance vector D P = [d1 d2…d n ] T ;

[0032] Step 105, the fitness value of each solution particle is calculated by the fitness function, and all fitness values are summarized as a fitness vector F P = [f1 f2…f n ] T ;

[0033] Step 106, normalize the distance vector and the fitness vector, and calculate the score vector of each solution particle, expressed as:

[0034] S P = ω FDB F P_norm + (1-ω FDB )D P_norm

[0035] In the formula, F P_norm is the normalized value of the fitness vector, D P_norm is the normalized value of the distance vector, and ω FDB is a weight parameter.

[0036] Step 107, numerically sort all score vectors, select the top 5% score vectors as the dominant population, and update the probability of each particle being updated according to the following formula:

[0037]

[0038] In the formula, l rate represents the update rate, P dominant represents the probability matrix of particles in the dominant population, and N dominant represents the total number of particles in the dominant population.

[0039] Step 108, repeat steps 101 to 107 until the maximum number of iterations is reached, and select the solution particle with the highest fitness function value as the final solution of the optimization problem.

[0040] Further, the fitness function is composed of an objective function and a penalty function, expressed as:

[0041] f value = f obj (x,t,v)+ψ(x,t,v)

[0042] In the formula, ψ(w,v,t) is a penalty function, expressed as:

[0043]

[0044] In the formula, ω fc represents the weight factor of flight conflict, and N fc represents the total number of flight conflicts in a solution.

[0045] Further, the double deep Q network includes a policy network and a target network, and the policy network and the target network have the same structure and are composed of multiple layers of perceptron and attention mechanism.

[0046] Further, the constructing and network training of the double deep Q network comprises the following steps:

[0047] Step 201, initializing the experience pool, and initializing the parameters θ of the policy network and the parameters θ' of the target network;

[0048] Step 202, generating an action by using the policy network and combining the ε-greedy algorithm in the simulation environment by the intelligent unmanned aerial vehicle, and calculating the reward of the current action by using the reward function;

[0049] Step 203, collecting the experience of the intelligent unmanned aerial vehicle joining the experience pool, when the number of experiences in the experience pool is greater than or equal to the batch size, randomly sampling a batch from the experience pool, and updating the parameters θ of the policy network according to the loss function; wherein represents the complete observation vector of the intelligent unmanned aerial vehicle at time t, represents the action of the intelligent unmanned aerial vehicle at time t, r t represents the reward obtained by the intelligent unmanned aerial vehicle at time t;

[0050] Step 204, updating the iteration number, and periodically copying the parameters of the policy network to the parameters of the target network;

[0051] Step 205, repeating steps 202 to 204 until the maximum iteration number is reached, and stopping iteration.

[0052] Further, the intelligent unmanned aerial vehicle outputs a conflict-free maneuvering action at each time step by using the trained double deep Q network, which comprises:

[0053] Step 301, constructing and initializing the simulation environment;

[0054] Step 302, creating a pre-planned path for each unmanned aerial vehicle;

[0055] Step 303, the intelligent unmanned aerial vehicle obtains the current complete observation vector which includes the state vector of the intelligent unmanned aerial vehicle itself the state vector of the K adjacent unmanned aerial vehicles around and the state vector related to the static obstacles around the intelligent unmanned aerial vehicle

[0056] Step 304, inputting the complete observation vector into the double deep Q network to obtain all actions and corresponding Q values of the intelligent unmanned aerial vehicle under the current observation state;

[0057] Step 305, the intelligent unmanned aerial vehicle selects the action corresponding to the maximum Q value to execute.

[0058] Further, step 304 comprises:

[0059] First, the complete observation vector of the intelligent unmanned aerial vehicle is abstracted into a vector with a length of 256 by using a multilayer perception machine, and then the abstract vector representing the state of the intelligent unmanned aerial vehicle itself is taken as a query vector of an attention mechanism, and the abstract vector representing the state of other unmanned aerial vehicles around the intelligent unmanned aerial vehicle is taken as a key vector and a value vector, the weight ω of the value vector is calculated from the key vector and the query vector, and the expression is:

[0060]

[0061] In the formula, MLP represents a multilayer perception machine, and dim represents the dimension of input and output;

[0062] The context vector V is calculated according to the weight c , which represents the influence of the surrounding neighbor unmanned aerial vehicles on the current main unmanned aerial vehicle, and is expressed as:

[0063] V c =ωV t neighbor

[0064] In the formula, V t neighbor represents a feature vector extracted from the vector representing the surrounding neighbor unmanned aerial vehicles by a multilayer perception machine, and the expression is:

[0065]

[0066] The context vector V c and the abstract vector of the static obstacles around the unmanned aerial vehicle are linked, and then pass through a multilayer perception machine to obtain the final Q value, and the optimal action of the next step is selected from the Q value, and the expression is:

[0067]

[0068] Advantages: Compared with the prior art, the present application has the following advantages:

[0069] 1. The present application provides a quantitative method, which can measure secondary conflicts in real time, and integrates the quantitative index into the reward function of deep reinforcement learning, so that the deep reinforcement learning method can take the optimization action of reducing secondary conflicts and avoiding primary conflicts when performing tasks;

[0070] 2. The present application introduces a new optimization method in the pre-tactical stage of the flight to provide more reasonable initial parameters, thereby accelerating the convergence speed in the training process and improving the training efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0071] Figure 1 is a flow chart of the unmanned aerial vehicle double-layer conflict resolution method for urban low-altitude dense traffic flow airspace.​

[0072] Figure 2 a schematic diagram of a simulation airspace;

[0073] Figure 3 a meshed schematic diagram of a simulation airspace;

[0074] Figure 4 a network structure diagram of a double deep Q network;

[0075] Figure 5 a network structure diagram of an attention mechanism;

[0076] Figure 6 a schematic diagram of an environment used in computer simulation;

[0077] Figure 7 a comparison diagram of the number of secondary conflicts. DETAILED DESCRIPTION

[0078] In order to make the purposes, technical solutions and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and examples.

[0079] The UAV double-layer conflict resolution method for the urban low-altitude dense traffic flow airspace described in the embodiment has a flow chart as shown in the figure, and includes: Figure 1

[0080] Before the intelligent UAV takes off, a mixed integer nonlinear programming model based on a conflict resolution strategy is constructed, and the mixed integer nonlinear programming model is optimized by using an improved random fractal search algorithm to obtain initial parameters of the intelligent UAV taking off;

[0081] After the intelligent UAV takes off, a double deep Q network is constructed and network training is performed, the influence of surrounding neighbors on the intelligent UAV is simulated by introducing an attention mechanism, and the intelligent UAV outputs a conflict-free maneuvering action at each time step by using the trained double deep Q network until the intelligent UAV reaches a final destination.

[0082] ​The embodiment is specifically for ultra-low altitude urban environment, assuming that the sky traffic flow is very dense, a double-layer conflict resolution framework is proposed combining deep reinforcement learning, which includes two stages of conflict resolution before and during flight of intelligent unmanned aerial vehicles. Before flight, the improved random fractal search algorithm is used to find the best take-off time and cruising speed of each unmanned aerial vehicle, which are used as the initial parameters after take-off; then the double deep Q network framework is used as the learning framework, so that the intelligent unmanned aerial vehicle can output conflict-free maneuvering actions at each time step; then the attention mechanism is used to construct the influence of surrounding neighboring unmanned aerial vehicles on the intelligent unmanned aerial vehicle, so that the learned conflict resolution strategy can adapt to any number of neighboring unmanned aerial vehicles. In this embodiment, the pre-flight conflict resolution and real-time conflict resolution are combined, which not only greatly reduces the occurrence of air collision compared with using any method alone, but also enables the intelligent unmanned aerial vehicle to avoid the current obstacles in the ultra-low altitude urban dense traffic flow environment, reduces secondary conflicts, and significantly shortens the training time of the real-time conflict resolution method based on reinforcement learning.

[0083] Further, the process of constructing a mixed integer nonlinear programming model based on the conflict resolution strategy includes:

[0084] (1) Based on the flight plan and real-time flight data of the intelligent unmanned aerial vehicle, an initial four-dimensional flight path is constructed for all unmanned aerial vehicles in the airspace.

[0085] (2) Detect whether there is a flight conflict in the four-dimensional flight path, and assign a resolution strategy to each detected flight conflict.

[0086] (3) The optimization objective function of the i-th flight conflict is defined as f i (ζ i ), where ζ i is a feasible solution in the search space, where is the decision variable of the conflict resolution strategy, representing the position, take-off time and cruising speed of the intelligent unmanned aerial vehicle respectively. It should be noted that the position change is a discrete change based on the grid space, which means that the position can only jump between predefined grid nodes, but cannot be continuously displaced. In addition, hovering actions are not allowed, i.e. continuous decision variables cannot point to the same position point, to ensure that each action reflects clear mobility and directionality. The expression of the objective function is:

[0087] min:f obj =ω risk R CP +ω t_delay T delay +ω t_air T air

[0088] In the formula, ωrisk ω t_delay ω t_air These are the weighting factors for risk, delay, and flight time, respectively, and ω risk +ω t_delay +ω t_air =1. Simply put, the goal is to solve for f1, f2, ... f F In order to find a better strategy i .

[0089] Among them, R CP Indicating potential conflict risks, related to trajectory intersections, can be written as:

[0090]

[0091] In the formula, Indicates intelligent drone U g Distance from the route intersection, W cp This indicates the distance the drone travels through all the route intersections;

[0092] T delay The total latency of all drones is represented as:

[0093]

[0094] In the formula, Indicates intelligent drone U g The actual takeoff time Indicates the scheduled departure time. T air The total flight time of all drones is expressed as:

[0095]

[0096] In the formula, w i and w j It is an adjacent waypoint, W r This represents the set of waypoints, where adjacency is defined by the absolute difference between their indices i and j, denoted as |ij| = 1; d ij Indicates arc J ij (w i ,w j The Euclidean distance of ) U-shaped drone g In arc J ij The cruising speed.

[0097] (4) Construct the constraints of the objective function, expressed as:

[0098]

[0099] In the formula, denotes the actual take-off time of the UAV U g denotes the actual flight time of the UAV U g denotes the scheduled take-off time of the UAV U g denotes the scheduled air flight time of the UAV U g denotes the delay time of the UAV U g N uav denotes the total number of UAVs; N p denotes the total number of adjacent waypoint pairs, d ij denotes the Euclidean distance of the arc J ij denotes the battery endurance time, denotes the number of UAVs departing from the start point, denotes the number of UAVs arriving at the end point, denotes the cruise speed of the UAV U g on the arc J ij and denote the minimum and maximum values of the UAV cruise speed, respectively.

[0100] The first constraint limits the arrival delay time of the intelligent UAVs, which constrains the time of the intelligent UAVs to arrive at the destination not to exceed a threshold The second constraint ensures that the flight time of the intelligent UAVs does not exceed its battery limit; the third constraint is used to ensure that the traffic flow remains balanced, the number of intelligent UAVs departing from the start point needs to be balanced with the number of intelligent UAVs arriving at the end point ; the fourth constraint is to ensure that the cruise speed of the intelligent UAVs does not exceed the physical limit of the UAV.

[0101] (5) The mixed integer nonlinear programming model is composed of the objective function and the constraints.

[0102] In one example, at any time, the flight plan inside the airspace is visible in real time, based on the flight plan and the real-time flight data of the UAV, a four-dimensional flight trajectory can be built for all UAVs inside the airspace, and potential conflicts may occur at the waypoints where the trajectories intersect. The four-dimensional waypoint can be defined as a directed graph G=(W, J), where W={0, 2,..., w+1} is the set of waypoints, and waypoint 0 represents the starting position and waypoint w+1 represents the destination. The set is the arc set of adjacent waypoints, where and are the three-dimensional grid coordinates after the airspace is gridded, N​​​​​​node denotes the total number of nodes i, j in the directed graph. The waypoint of trajectory intersection is defined as the intersection point of at least two arcs, denoted as CWP k denotes the waypoint of trajectory intersection. The set of predicted arrival time of any waypoint for each UAV is denoted as The set of all UAVs in airspace is denoted as set where N uav denotes the total number of UAVs. The set of four-dimensional waypoints is denoted as where N UAV denotes the total number of UAVs in airspace, denotes the four-dimensional waypoint of UAV U g , where is a three-dimensional vector, denoting the three-dimensional grid coordinates of the four-dimensional waypoint, denotes the time of UAV arriving at the four-dimensional waypoint.

[0103] Flight conflicts are identified by finding intersection waypoints, which are formed by a pair of flight trajectories, when the time difference of two UAVs arriving at the intersection waypoint is less than the conflict threshold Δt conflict , i.e., a conflict is considered to exist, which can be represented as:

[0104]

[0105] In the formula, denotes the time of a single UAV arriving at the intersection waypoint; in this example, Δt conflict can be set to 10 seconds.

[0106] The set of detected flight conflicts is denoted as C = {c1, c2,..., c p}, c p denotes the pth flight conflict, p ∈ N fc , N fc denotes the total number of flight conflicts currently checked.

[0107] Further, the process of optimizing the mixed integer nonlinear programming model using the improved random fractal search algorithm includes the following steps:

[0108] Step 101, assuming that a body particle is regarded as a potential solution, the initialization of the particle is randomly performed within the constraints of the problem conditions, denoted as:

[0109] P i = B lower + λ (B upper - B lower )

[0110] In the formula, Pi represents the i-th initial particle in the population, B lower and B upper respectively represent the lower and upper limits of the constraint vector, λ is a random number, and λ ∈ [0, 1];

[0111] Step 102, generating new particles in the search space by Gaussian random walk distribution, denoted as:

[0112]

[0113] In the formula, P i η is the i-th new particle generated by the particle P i η is the number of new particles generated, μ P is the Gaussian distribution mean of the new particle position, μ P = |P i |, σ is the standard deviation, P best is the optimal particle in the current group, and represent the weight factors adjusting the development and exploration degree, obeying uniform distribution, ranging from [0, 1];

[0114] The calculation formula of the standard deviation σ is:

[0115]

[0116] In the formula, N Iter represents the total number of iterations;

[0117] Step 103, evaluating all generated particles by the fitness function, keeping the solution particles with the fitness value greater than the threshold value unchanged, and updating the remaining solution particles when the following conditions are met:

[0118]

[0119] In the formula, represents the probability of updating the particle, rank(P i ) represents the ranking of the particle after being evaluated by the fitness function, used to determine the relative superiority or inferiority of the particle in the population, n p represents the total number of particles;

[0120] Among them, the updating process of the remaining solution particles is divided into two stages, the first stage updating is represented as:

[0121]

[0122] In the formula, P r and P t are other particles randomly selected from the current population;

[0123] Then the second stage update is performed as follows, denoted as:

[0124]

[0125] where P t and P r are other particles randomly selected from the current population;

[0126] Step 104, record the position of the best particle in the population as X best , calculate the distance value between the updated solution particle and the best particle Collect all distance values into a distance vector D P = [d1d2…d n ] T ;

[0127] where the calculation expression of the distance value is:

[0128]

[0129] where v is the dimension space of the particle, denoted as the value of the particle P i in the dimension v, denoted as the value of the best particle P best in the dimension v;

[0130] Step 105, calculate the fitness value of each solution particle through the fitness function, and collect all fitness values into a fitness vector F P = [f1 f2…f n ] T ;

[0131] Step 106, normalize the distance vector and the fitness vector, and calculate the score vector of each solution particle, expressed as:

[0132] S P = ω FDB F P_norm +(1-ω FDB )D P_norm

[0133] where F P_norm is the normalized value of the fitness vector, D P_norm is the normalized value of the distance vector, and ω FDB is a weight parameter used to adjust the weight of the fitness value and the distance value during the update process, with a value range of [0, 1]. The score vector of each particle is determined by the fitness function and the distance, combining the needs of development and exploration;

[0134] Step 107, numerically sort all score vectors, select the top 5% score vectors as the dominant population, and update the probability of each particle being updated according to the following formula:

[0135]

[0136] In the formula, l rate represents the update rate, P dominant represents the probability matrix of particles in the dominant population, N dominant represents the total number of particles in the dominant population.

[0137] Step 108, repeat steps 101 to 107 until the maximum number of iterations is reached, and select the solution particle with the highest fitness function value as the final solution of the optimization problem.

[0138] Further, the fitness function is composed of the objective function and the penalty function, and is expressed as:

[0139] f value = f obj (x, t, v) + ψ(x, t, v)

[0140] In the formula, ψ(w, v, t) is a penalty function, and the expression is:

[0141]

[0142] In the formula, ω fc represents the weight factor of flight conflict, N fc represents the total number of flight conflicts in a solution.

[0143] Further, the double deep Q network includes a policy network and a target network, and the policy network and the target network have the same structure and are composed of multiple layers of perception and attention mechanisms.

[0144] Further, constructing a double deep Q network and training the network includes the following steps:

[0145] Step 201, initialize the experience pool, and initialize the parameters θ of the policy network and the parameters θ' of the target network;

[0146] Step 202, generate actions in the simulation environment using the policy network and combining the ε-greedy algorithm using intelligent unmanned aerial vehicles, and calculate the rewards of the current actions using the reward function;

[0147] Where the reward function is defined as:

[0148]

[0149] Where, the parameter r crossThe calculation expression of r is:

[0150]

[0151] In the formula, represents the track deviation of the intelligent unmanned aerial vehicle relative to its own reference track, and a and β are constant coefficients, which are respectively 4000 and 3.5 in the embodiment;

[0152] The calculation expression of r goal is:

[0153]

[0154] In the formula, and respectively represent the Euclidean distance between the position of the intelligent unmanned aerial vehicle at the previous time and the next time and the destination, and μ is a constant coefficient, which is 6 in the embodiment;

[0155] The calculation expression of r intru is:

[0156]

[0157] In the formula, rh NMAC represents the radius of the intelligent unmanned aerial vehicle, which can be 2.5 meters; d near represents the Euclidean distance between the intelligent unmanned aerial vehicle and the nearest unmanned aerial vehicle; and ρ is a constant coefficient, which can be 50;

[0158] The calculation expression of r dec is:

[0159]

[0160] In the formula, and respectively represent the potential conflict number of the unmanned aerial vehicle under the current action and the potential conflict number under the next step given subsequent action; h is a constant coefficient, which can be 2; and dec t is a modeling of a real-time quantitative definition of the secondary conflict, and the expression is:

[0161]

[0162] The calculation expression of r pen is:

[0163]

[0164] In the above reward function, r goal and r intruThe main role of r is to guide the intelligent UAV to move towards the target without collision; dec The r is responsible for guiding the UAV to learn to minimize the secondary collision while taking evasive action. When only using these three parts as rewards, the strategy of the intelligent UAV will cause it to move to a corner of the map and take small and gradual actions, waiting for all other UAVs to leave the airspace before moving towards the target, so r cross and r pen are needed to prevent the intelligent UAV from learning such an extremely inefficient strategy.

[0165] Step 203, collecting the experience of the intelligent UAV Join the experience pool, when the number of experiences in the experience pool is greater than or equal to the batch size, randomly sample a batch from the experience pool, and update the parameters θ of the policy network according to the loss function; wherein represents the complete observation vector of the intelligent UAV at time t, represents the action of the intelligent UAV at time t, r t represents the reward obtained by the intelligent UAV at time t;

[0166] Wherein, the expression of the loss function is:

[0167]

[0168] In the formula, represents the expected value calculation of the sampled experience data in the experience replay pool D, r t represents the reward obtained by the current intelligent UAV at time t, and γ∈[0,1) represents the discount factor, represents the estimated Q value, which is calculated by the target network, and the parameters of the target network are represented by θ'; represents the complete state after the UAV selects the current action Q represents the current Q value, which is calculated by the policy network, and the parameters of the policy network are represented by θ.

[0169] Step 204, update the number of iterations, and periodically copy the parameters of the policy network to the parameters of the target network;

[0170] Step 205, repeat steps 202 to 204 until the maximum number of iterations is reached, and stop iteration.

[0171] After the loss function is derived, the gradient of the loss function with respect to the parameters of the policy network is calculated using backpropagation, and the parameters of the policy network are updated by the optimization algorithm Adam to minimize the loss. Finally, every certain training round, such as every 50 updates, the parameters of the policy network are copied to the target network to maintain the stability of the target network. When the return of each round tends to be stable, it represents that the training of the policy network is completed.

[0172] Further, the trained double deep Q network is used to make the intelligent unmanned aerial vehicle output conflict-free maneuvering actions at each time step, which includes:

[0173] Step 301, constructing and initializing a simulation environment;

[0174] Step 302, creating a pre-planned path for each unmanned aerial vehicle;

[0175] Step 303, the intelligent unmanned aerial vehicle acquires the current complete observation vector which includes the state vector of the intelligent unmanned aerial vehicle itself the state vector of the K neighboring unmanned aerial vehicles around and the state vector related to the static obstacles around the intelligent unmanned aerial vehicle

[0176] Step 304, inputting the complete observation vector into the double deep Q network to obtain all actions and corresponding Q values of the intelligent unmanned aerial vehicle in the current observation state;

[0177] Step 305, the intelligent unmanned aerial vehicle selects the action corresponding to the maximum Q value to execute.

[0178] Further, step 304 includes:

[0179] First, the complete observation vector of the intelligent unmanned aerial vehicle is abstracted into a vector with a length of 256 by using a multi-layer perception machine, and then the abstract vector representing the state of the intelligent unmanned aerial vehicle itself is used as the query vector of the attention mechanism, and the abstract vector representing the state of the other unmanned aerial vehicles around the intelligent unmanned aerial vehicle is used as the key vector and the value vector. The weight ω of the value vector is calculated from the key vector and the query vector, and the expression is:

[0180]

[0181] In the formula, MLP represents a multi-layer perception machine, and dim represents the dimension of the input and output;

[0182] According to the weight, a context vector V c is calculated, which represents the influence of the surrounding neighbor unmanned aerial vehicles on the current intelligent unmanned aerial vehicle, and is expressed as:

[0183] V c = ωV tneighbor

[0184] where V t neighbor represents the feature vector extracted from the vector representing the surrounding neighbors of the UAV by the multi-layer perceptron neural network, expressed as:

[0185]

[0186] The context vector V c and the abstract vector of static obstacles around the UAV are linked, and then passed through a multi-layer perceptron to obtain the final Q value, and the optimal action for the next step is selected therefrom, expressed as:

[0187]

[0188] where represents finding the action that can maximize the objective function in a set of possible action collections a, and the output of the objective function is also the final Q value;

[0189] In an example, as Figure 2 shown, a 210x130m airspace is selected from a city map as the research airspace, including the detection range of the UAV, the starting and ending points, the waypoints, and the intersection waypoints. In this example, only the cruising state is considered, so the height of the airspace is not considered, and the airspace is discretized into an equidistant grid with a degree of Figure 6 As shown in Figure 2 , the UAV is modeled as a circle with a radius of 2.5m, and 25 non-intelligent UAVs are generated, which have their own starting and ending points, but will not autonomously avoid obstacles; the intelligent UAV is generated and the starting and ending points are generated, Figure 6 The environment used in the actual computer simulation includes the modeling of buildings after discretization of the entire environment, and the layout of the initial flight routes of the 25 non-intelligent UAVs and one intelligent UAV in the airspace. After the simulation environment is constructed, the double deep Q network offline training phase can be started, with a batch size of 64, a learning rate of 0.00001, a total number of training rounds of 50000, and a maximum of 45 time steps per round. When using the ε-greedy algorithm, ε starts from 1 in the first round and decreases linearly in each round, and decreases to 0.05 in the 25000th round, and then does not continue to decrease. The skip point algorithm is used to create a pre-planned path for each UAV, and the turning points on the path are used as the waypoints, and the planned path is analyzed to find intersection points, which are identified and set as intersection waypoints. Then set the value range of the decision variable, the constraint threshold and the weight factor. The potential conflict risk R CPThe risk value of each position in the airspace that can be used as a waypoint before takeoff is higher if it is closer to the position of the intersection waypoint, as shown in Figure 3 After the airspace is gridded using the concept of AirMatrix, the airspace selectable positions are x grid ∈ [1, 21], y grid ∈ [1, 13]. The decision variable value range of the takeoff time (unit: seconds) and the cruise speed (unit: meters / second) of each UAV is t etd ∈ [1, 30] and v level ∈ [5, 15], respectively. In addition, the delay time threshold is set to 10 seconds, and the battery endurance time threshold is set to 900 seconds. Before takeoff, the definition of conflict resolution is that the minimum time difference of consecutive UAVs passing through the intersection waypoint is at least 3 seconds. The weight ω risk of the potential conflict before takeoff is 0.5, the flight delay weight ω t_delay and the air time weight ω t_air are both 0.25, and the conflict penalty weight ω fc is set to 0.1. The improved type random fractal search algorithm is used before the takeoff of the Z intelligent UAV to optimize the mixed integer nonlinear programming model, and the takeoff time and cruise speed of each aircraft are found, which will be used as input parameters for the following deep reinforcement learning.

[0190] The dual deep Q network that has been trained is loaded, and the environment perceived by the intelligent UAV includes the state vector of the intelligent UAV itself, the state vector from the neighboring UAV, and the state vector of the static obstacles within the detection range. The complete state space of the intelligent UAV is defined as where represents the state vector of the intelligent UAV itself, which is represented as: where e t c represents the lateral deviation between the position of the intelligent UAV and the reference path, and represent the difference between the current speed and the previous speed of the intelligent UAV in the x-axis and y-axis, respectively, and represent the current speed of the intelligent UAV in the x-axis and y-axis, respectively, and represent the Euclidean distance between the intelligent UAV and its next waypoint in the x-axis and y-axis directions, respectively. There are a total of K neighboring UAVs within the detection range of the intelligent UAV, represented as i = {1, 2,..., K}. The state of a single i-th neighboring UAV is represented as: where, and ​are the Euclidean distances between the intelligent UAV and the ith neighboring UAV in the x-axis and y-axis directions, respectively, and are the velocities of the ith neighboring UAV in the x-axis and y-axis directions, respectively. is the state vector related to static obstacles around the UAV. With the spatial data of buildings, which contains the top-view contour of the buildings (in the form of polygons) and the height from the ground to the highest point, the airspace is discretized into cubes of equal size using the concept of AirMatrix. When a cube intersects or overlaps with any polygon, the position of the cube in the matrix will be changed to 1. Therefore, a three-dimensional binary matrix can represent the entire airspace, and since only horizontal flight is considered, a two-dimensional matrix can be used to represent the occupancy of the surrounding area by buildings within the detection range.

[0191] The complete observation vector of the intelligent UAV will be used to derive the Q-values of all actions of the intelligent UAV in the current observation state by the trained double deep Q-network, Figure 4 is the network structure diagram of the double deep Q-network used in this embodiment, which also contains an attention mechanism module, as shown in Figure 5 Three MLP structures are used to extract features from the three parts of the observation vector of the intelligent UAV, respectively. Each MLP consists of one fully connected layer, including 256 neurons. The state vector related to static obstacles around the intelligent UAV needs to be flattened first before being sent to the MLP structure, and the vector and extract features, and then the context vector V c representing the influence of the surrounding neighbor UAVs on the current main UAV is extracted through the attention mechanism. Then the extracted features of , the context vector V c and features are spliced together, and then the final MLP structure is used to output the Q-values of all actions of the intelligent UAV in the current state. The final MLP consists of three fully connected layers with 512, 512 and 9 neurons, respectively. Except for the last fully connected layer with 9 neurons without an activation function, the activation functions of the other fully connected layers are all rectified linear units.

[0192] The intelligent UAV selects the action corresponding to the maximum Q-value and executes it. The action space of the intelligent UAV is discrete, with 9 choices, each consisting of a fixed value of acceleration in the x-axis and y-axis directions, as follows:

[0193]

[0194] where h a is the physical limit of the acceleration of the intelligent UAV in this embodiment, which is a constant and can be set to 4 m / s 2 The state transition model of the intelligent UAV is: It is a conditional probability that describes the possibility of transitioning to another state given the current state and the selected action. The actual state transition model is hidden to the intelligent UAV, but it can be derived from the dynamics given to the intelligent UAV. The dynamics used by the intelligent UAV in this embodiment are defined as follows:

[0195]

[0196] When the current speed of the intelligent UAV exceeds the maximum speed allowed by its physical performance, i.e. The speed of the intelligent UAV is then changed to:

[0197]

[0198] where v max is the maximum speed allowed by the physical performance of the intelligent UAV in this embodiment, which is set to 15 m / s, is the heading direction of the current intelligent UAV derived from the velocity vector.

[0199] The intelligent UAV will fly steadily to its terminal point while avoiding collisions and reducing secondary conflicts by performing the action corresponding to the maximum Q value output by the current maximum policy network at each time step. As shown in Figure 7 the number of secondary conflicts accumulated by the intelligent UAV after reaching the terminal point under different degrees of congestion in the airspace (5, 15, and 25 other UAVs, respectively) using the deep reinforcement learning algorithm proposed in this application. At the same time, the number of secondary conflicts accumulated when using the current most advanced non-learning algorithm, Optimal Reciprocal Collision Avoidance (ORCA), is also counted for comparison. Through comparison, it can be seen that when the airspace is more crowded, the use of the deep reinforcement learning algorithm DDQN-attention proposed in this application can more effectively reduce the number of secondary conflicts accumulated, thereby balancing the stability of the airspace.

Claims

1. A method for solving the conflict of UAVs in the airspace of urban low-altitude dense traffic flow, characterized in that, The application relates to an intelligent unmanned aerial vehicle (UAV) conflict resolution method. Before the intelligent UAV takes off, a mixed integer nonlinear programming model based on a conflict resolution strategy is constructed, and an improved random fractal search algorithm is used to optimize the mixed integer nonlinear programming model to obtain initial parameters of the intelligent UAV taking off. The process of optimizing the mixed integer nonlinear programming model by using the improved random fractal search algorithm comprises the following steps. In step 101, a particle is assumed to be a potential solution, and the particle is initialized randomly within the problem condition constraints, and is expressed as: P i = B lower + λ(B upper - B lower ) where P i represents the ith initial particle in the population, B lower and B upper respectively represent the lower and upper limits of the constraint vector, and λ is a random number and λ ∈ [0, 1]; In step 102, new particles are generated in the search space through a Gaussian random walk distribution, and are expressed as: wherein, is the i-th new particle generated by the particle P i is the i-th new particle generated by the particle P P is the i-th new particle generated by the particle P P is the i-th new particle generated by the particle P i is the i-th new particle generated by the particle P best is the i-th new particle generated by the particle P and denote the weight factors adjusting the degree of exploitation and exploration, obeying uniform distribution, ranging from [0, 1]. In step 103, all generated particles are evaluated through a fitness function, and the solution particles with a fitness value greater than a threshold value are kept unchanged, and the rest of the solution particles are updated when the following conditions are met, and the conditions are expressed as: wherein represents the probability that a particle is updated, rank(P i ) represents the ranking of a particle after evaluation by the fitness function, n p represents the total number of particles; Step 104, record the position of the best particle in the population as X best , calculate the distance value between the updated solution particle and the best particle Collect all distance values into distance vector D P = [d1d2…d n ] T ; Step 105, calculate the fitness value of each solution particle by fitness function, and collect all fitness values into fitness vector F P = [f1f2…f n ] T ; In step 106, the distance vector and the fitness vector are normalized, and a score vector of each solution particle is calculated, and the expression is: S P = ω FDD F P_norm + (1-ω FDD )D P_norm In the formula, F P_norm D is the value after fitness vector normalization P_norm ω is the value after distance vector normalization FDD ω is the weight parameter In step 107, all score vectors are numerically sorted, the top 5% score vectors are selected as the dominant population, and the probability of each particle being updated is updated according to the following formula: wherein l r1te denotes the update rate, P domin1nt denotes the probability matrix of particles in the dominant population, N domin1nt denotes the total number of particles in the dominant population; In step 108, steps 101 to 107 are repeated until the maximum iteration number is reached, and the solution particle with the highest fitness function value is selected as the final solution of the optimization problem. After the intelligent UAV takes off, a double deep Q network is constructed and network training is performed, the influence of surrounding neighbors on the intelligent UAV is simulated through an introduced attention mechanism, and the intelligent UAV outputs a conflict-free maneuvering action at each time step through the trained double deep Q network until the intelligent UAV reaches a final destination. The double deep Q network comprises a policy network and a target network, and the policy network and the target network have the same structure and are both composed of a multilayer perceptron and an attention mechanism. The process of constructing the double deep Q network and performing network training comprises the following steps. In step 201, an experience pool is initialized, and the parameters theta of the policy network and the parameters theta' of the target network are initialized. In step 202, the policy network is used by the intelligent UAV in the simulation environment, an action is generated by combining an epsilon-greedy algorithm, and a reward function is used to calculate the reward of the current action. Step 203, collecting experience of the intelligent unmanned aerial vehicle Joining to the experience pool, when the number of experiences in the experience pool is greater than or equal to the batch size, randomly sampling a batch from the experience pool, and updating the parameters θ of the policy network according to the loss function; wherein represents the complete observation vector of the intelligent unmanned aerial vehicle at t moment, represents the action of the intelligent unmanned aerial vehicle at t moment, r t represents the reward obtained by the intelligent unmanned aerial vehicle at t moment; In step 204, the iteration number is updated, and the parameters of the policy network are periodically copied to the parameters of the target network. In step 205, steps 202 to 204 are repeated until the maximum iteration number is reached, and the iteration is stopped.

2. The method according to claim 1, wherein, The process of constructing the mixed integer nonlinear programming model based on the conflict resolution strategy comprises the following steps. Based on the flight plan and real-time flight data of the intelligent UAV, an initial four-dimensional flight path is constructed for all UAVs in the airspace. Whether a flight conflict exists in the four-dimensional flight path is detected, and a solution strategy is assigned to each detected flight conflict. The optimization objective function of the ith flight conflict is defined as f i (ζ i ), where ζ i is a feasible solution in the search space, where is the decision variable of the conflict resolution strategy, respectively representing the position, take-off time and cruising speed of the intelligent UAV, and the expression of the objective function is: min:f obj = ω risk R XP + ω t_del1y T del1y + ω t_1ir T 1ir where ω risk , ω t_del1y , and ω t_1ir are weight factors for risk, delay, and flight time, respectively, and ω risk + ω t_del1y + ω t_1ir = 1, R XP denotes the potential conflict risk, T del1y denotes the total delay time of all drones, and T 1ir denotes the total flight time of all drones. The constraint conditions of the objective function are constructed and are expressed as: wherein denotes the actual take-off time of the UAV U g , denotes the actual flight time of the UAV U g , denotes the scheduled take-off time of the UAV U g , denotes the scheduled air flight time of the UAV U g , denotes the delay time of the UAV U g , N u1v denotes the total number of UAVs; N p denotes the total number of adjacent waypoint pairs, d ij denotes the Euclidean distance of the arc J ij , denotes the battery endurance time, denotes the number of UAVs starting from the start point, denotes the number of UAVs reaching the end point, denotes the cruising speed of the UAV U g on the arc J ij , and denote the minimum and maximum values of the UAV cruising speed, respectively; The mixed integer nonlinear programming model is composed of the objective function and the constraint conditions. 3.The method of claim 1, wherein, The fitness function is composed of the objective function and a penalty function, and is expressed as: f v1lue = f obj (x, t, v) + ψ(x, t, v) In the formula, the penalty function is expressed as: where ω op represents a weight factor for flight conflicts, N op represents the total number of flight conflicts in one solution.

4. The method of claim 1, wherein, The intelligent UAV outputs a conflict-free maneuvering action at each time step through the trained double deep Q network. Step 301, constructing and initializing the simulation environment; Step 302, creating a pre-planned path for each unmanned aerial vehicle; Step 303, the intelligent unmanned plane acquires a current complete observation vector which includes a self state vector of the intelligent unmanned plane state vectors of K surrounding neighboring unmanned planes and a state vector related to static obstacles around the intelligent unmanned plane Step 304, inputting the complete observation vector into the double deep Q network to obtain all actions and corresponding Q values of the intelligent unmanned aerial vehicle in the current observation state; Step 305, the intelligent unmanned aerial vehicle selects the action corresponding to the maximum Q value to execute.

5. The method of claim 4, wherein, Step 304 includes: First, the complete observation vector of the intelligent unmanned aerial vehicle is abstracted into a vector with a length of 256 The abstract vector representing the state of the intelligent unmanned aerial vehicle is taken as a query vector of the attention mechanism, and the abstract vector representing the state of the other unmanned aerial vehicles around the intelligent unmanned aerial vehicle is taken as a key vector and a value vector. The weight ω of the value vector is calculated from the key vector and the query vector, and the expression is as follows: In the formula, MLP represents a multi-layer perceptron, and dim represents the dimension of input and output; According to the weight, a context vector V is calculated c , representing the influence of the surrounding neighbor drones on the current master drone, is expressed as: V c = ωV t neighbor where V t neighbor is a feature vector extracted from the vector representing the surrounding neighbor drones by a multi-layer perceptron, expressed as: The context vector V c and the abstract vector of static obstacles around the UAV are linked, and then pass through a multi-layer perception machine to obtain the final Q value, and the optimal action of the next step is selected therefrom, denoted as:

Images