Multi-target adaptive constraint unmanned aerial vehicle path planning method based on reinforcement learning

Abstract

The invention discloses a multi-target adaptive constraint unmanned aerial vehicle path planning method based on reinforcement learning. The method comprises the following steps: initializing an unmanned aerial vehicle path population; determining a population state, wherein the population state comprises a ratio of feasible individuals in the main population, a ratio of feasible non-dominated individuals to non-dominated individuals, a central point, a standard deviation and a state flag determined based on the auxiliary population; according to the population state, using a reinforcement learning network to obtain an action, and storing the action to an output population; and judging whether a termination condition is met or not, if not, selecting an adaptive constraint strategy to update the population by taking the action as an index and continuing iteration, and if so, outputting all non-dominated individuals in the population as an optimized unmanned aerial vehicle flight path. According to the method, multiple constraint challenges in the unmanned aerial vehicle path planning process can be effectively coped with, and the constraint strategy is dynamically updated through online feedback of reinforcement learning, so that the efficiency and adaptability of path planning are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle path planning, and more specifically to a multi-objective adaptive constraint unmanned aerial vehicle path planning method based on reinforcement learning. Background Art

[0002] Due to their excellent maneuverability and stable performance, drones have great application prospects in many fields. For example, in logistics distribution, urban management, emergency rescue and other fields, the application of drone technology can not only greatly improve operational efficiency and response speed, but also reduce operating costs. In addition, drones have shown great potential in environmental monitoring, traffic management, disaster assessment and other aspects. How to efficiently and safely plan the flight path of drones in complex and diverse flight environments is one of the core issues in related research fields.

[0003] At present, the methods for solving UAV path planning problems can be mainly divided into the following three categories: path planning algorithms based on node optimization, path planning algorithms based on sampling, and path planning algorithms based on artificial intelligence.

[0004] Node optimization-based algorithms construct loss functions and use heuristic information or traversal to find the optimal path. This type of algorithm has high computational complexity, especially when there are many nodes and complex networks, the amount of calculation will increase dramatically, resulting in reduced efficiency in planning paths.

[0005] Sampling-based algorithms randomly sample the environment and generate a roadmap, then search for a feasible path from the starting point to the end point. As the complexity of the environment increases, the quality of its path may further deteriorate, and there are also problems with high computational complexity and the possibility of not finding the optimal solution.

[0006] Path planning algorithms based on artificial intelligence, such as swarm intelligence algorithms and reinforcement learning, are an important direction in the field of drone path planning because of their environmental interactivity, which enables them to adapt to complex flight environments. Existing mainstream methods often model drone path planning as a constrained multi-objective optimization problem and then use algorithms to solve it. However, when this method faces complex environments, it becomes difficult for drones to meet constraints, that is, the algorithm has difficulty finding the optimal flight path.

[0007] Therefore, a new algorithm needs to be developed to adjust the constraint handling method according to the environment so that the algorithm can focus on finding the optimal flight path. Summary of the invention

[0008] In view of this, the present invention provides a multi-objective adaptive constraint UAV path planning method based on reinforcement learning; mainly based on the general framework of the constrained multi-objective evolutionary algorithm, an innovative adaptive selection constraint processing technology strategy is introduced to effectively cope with the multiple constraint challenges in the UAV path planning process.

[0009] In order to achieve the above object, the present invention adopts the following technical solution:

[0010] A multi-objective adaptive constraint UAV path planning method based on reinforcement learning, comprising:

[0011] Initialize the drone path population, including the main population, auxiliary population and output population;

[0012] Determine the population status, including the proportion of feasible individuals in the main population, the proportion of feasible non-dominated individuals to non-dominated individuals, the center point, the standard deviation, and the status flag determined based on the auxiliary population;

[0013] According to the population state, the action is obtained using the reinforcement learning network and stored in the output population;

[0014] Determine whether the termination condition is reached. If not, select the adaptive constraint strategy based on the action as the index to update the population and continue iterating. If yes, output all non-dominated individuals in the population as the optimized UAV flight path.

[0015] As a preferred method, the feasible individual proportion calculation formula is:

[0016]

[0017] Among them, P1 represents the main population, ν is the number of feasible individuals in P1, |P1| is the size of population P1; the ratio of feasible non-dominated individuals to non-dominated individuals is expressed as:

[0018]

[0019] in, is the number of feasible non-dominated individuals in P1, κ is the number of non-dominated individuals in P1;

[0020] The center point is the average value of the objective function of all individuals in the main population;

[0021] The formula for calculating the standard deviation is:

[0022]

[0023] in, represents the center point, y represents all individual x i The corresponding target value, is y and The Euclidean distance of

[0024] The calculation formula for the status flag determined based on the auxiliary population is:

[0025]

[0026] Among them, μ j is the rate of updating individuals in the auxiliary population P2, μ thr is a preset parameter, and J is a set consisting of 10 consecutive generations.

[0027] Preferably, the reinforcement learning network calculates the reward according to the following formula:

[0028]

[0029] In the formula, R' represents the intermediate variable, IGD represents the reverse intergenerational distance of the main population, and IGD' represents the reverse intergenerational distance of the main population after update. It indicates the proportion of feasible individuals in the main population after update, rounded down, χ indicates the number of individuals in the auxiliary population after update, λ indicates the standard deviation of all individuals in the main population in the target space after update, λ' indicates the standard deviation of all individuals in the output population in the target space after update, and flag indicates the state mark.

[0030] As a preferred method, the reverse generational distance calculation formula is:

[0031]

[0032] Where y is the target value of individual x, md(y, P1) is the Euclidean distance of individual x from the nearest individual in the main population P1 in the target space, P2 represents the auxiliary population, and P'1 represents the updated main population.

[0033] As a preference, the following loss function is used to train the reinforcement learning network:

[0034]

[0035] In the formula, R represents the reward, λ represents the discount factor of future rewards, It is the value of the action a' with the largest output after inputting the state St' into the target network Q2. St' represents the updated population state, Q2 represents the target network, Q1 represents the main network, and Q1(St, a) is the action value corresponding to action a after inputting the state St into the main network Q1.

[0036] Preferably, when updating the main population, the constraint processing update mode is selected using the action as an index; the constraint processing categories include target value priority, boundary value priority and constraint dominance principle;

[0037] When selecting the target value first, obtain the offspring population of the main population and merge it with the main population, determine the number of other individuals that each individual in the merged population can dominate without constraints, calculate the fitness of all individuals in the merged population, and remove the individual with the largest fitness;

[0038] When the boundary value is selected first, the offspring population of the main population is obtained and merged with the main population, the number of other individuals dominated by the boundary constraints of each individual in the merged population is determined, the fitness of all individuals in the merged population is calculated, and the individual with the largest fitness is removed;

[0039] Boundary constraint dominance is defined as: Calculate the individual x according to the constraint function i The constraint violation value G i , for two individuals x1 and x2, if G1 and G2 are both less than or equal to the boundary value ε or G1 is equal to G2, and all objective function values ​​of x1 are not worse than x2, and at least in one objective function, the value of x1 is strictly better than x2, then the x1 boundary constraint dominates x2; if G2 is greater than the boundary value ε, and G1 is less than G2, then the x1 boundary constraint dominates x2;

[0040] When the constraint dominance principle is selected, the offspring population of the main population is obtained and merged with the main population, the number of constraints and dominance of other individuals by each individual in the merged population is determined, the fitness of all individuals in the merged population is calculated, and the individual with the largest fitness is removed;

[0041] Constraint dominance is defined as: Calculate the individual x according to the constraint function i The constraint violation value G i For two individuals x1 and x2, if G1 is equal to G2, and all objective function values ​​of x1 are not worse than x2, and at least in one objective function, the value of x1 is strictly better than x2, then the x1 constraint dominates x2; if G1 is less than G2, then the x1 constraint dominates x2.

[0042] Preferably, the boundary value ε is determined as follows:

[0043]

[0044] In the formula, ε0 is the maximum constraint violation value among all individuals in the initial population, t represents the current iteration number, and T max Indicates the maximum number of iterations.

[0045] Preferably, the offspring population of the main population is obtained by the following steps:

[0046] The main population and the output population are merged to obtain the secondary main population, and a neighbor set is assigned to each individual in the secondary main population;

[0047] For each individual x i, select two individuals from the corresponding neighbor set, take the individual with small unconstrained dominance fitness as the parent individual, traverse the neighbor set, and determine the final parent individual x r1 and x r2 ;

[0048] For parent individual x r1 and x r2 Perform mutation to obtain individual u i ;

[0049] For individual u i Each component u' i Restored to individual x with probability (1-CR) i The corresponding component x' i ,get

[0050] As a preference, the neighbor set is allocated according to the following formula:

[0051]

[0052] Where σ represents the maximum neighbor angle, y i The individual x i The target value of θ(y i ,y j ) is the vector y i and i The angle formed, P4 represents the secondary main population.

[0053] Preferably, the steps of calculating individual fitness include:

[0054] Fit i =R i +D i

[0055]

[0056] In the formula, Fit i represents individual fitness, R i represents the original fitness of the individual, D i represents individual density, P represents the combined population, x j >x i Represents x j Dominate x i , S i Represents individual x i The number of other individuals in the combined population is controlled in the desired way, and τ represents x i Distance The Euclidean distance between individuals.

[0057] It can be seen from the above technical solutions that the present invention discloses a multi-objective adaptive constraint UAV path planning method based on reinforcement learning. Compared with the prior art,

[0058] 1) This method utilizes the online feedback mechanism of deep reinforcement learning, which can dynamically update the constraint selection strategy according to different scenarios. It has the characteristics of strong adaptability and high efficiency.

[0059] 2) The present invention proposes an adaptive constraint processing technology to solve the multiple constraint problems of UAV path planning in complex environments, thereby improving the efficiency of path planning.

[0060] 3) This method uses B-spline curve to ensure the smoothness of the UAV's flight path, meets the UAV's flight requirements, and is practical. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0062] Figure 1 A flow chart of the multi-objective adaptive constraint UAV path planning method based on reinforcement learning provided by the present invention;

[0063] Figure 2 is a schematic diagram of the deep reinforcement learning used in the present invention;

[0064] Figure 3 It is the three-dimensional environment terrain in the embodiment of the present invention. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] The embodiment of the present invention discloses a multi-objective adaptive constraint UAV path planning method based on reinforcement learning. The method introduces an innovative adaptive selection constraint processing technology strategy based on the general framework of the constrained multi-objective evolutionary algorithm to effectively deal with the multiple constraint challenges in the UAV path planning process. At the same time, the method uses the online feedback mechanism of deep reinforcement learning to dynamically update the optimal selection strategy, thereby improving the efficiency and adaptability of path planning.

[0067] Embodiment 1

[0068] Initialize the drone path population. In this embodiment, the drone path is determined based on a series of path points; and the path point acquisition step includes:

[0069] 1. Obtain the three-dimensional environment terrain of the flight area; that is, import terrain data or construct the three-dimensional environment terrain according to the terrain generation function;

[0070] 2. Construct a path representation model based on B-spline curves based on the three-dimensional terrain environment; specifically including:

[0071] 2.1 The flight path is represented by a B-spline curve. The B-spline curve is generated by a set of control points. Each control point corresponds to a coordinate in the three-dimensional Cartesian coordinate system. The control points are represented as: CP0(x0, y0, z0), CP1(x1, y1, z1),…, CP s-1 (x s-1 ,y s-1 ,z s-1 ), s represents the number of control points.

[0072] 2.2 Using uniform sampling, a flight trajectory can be constructed based on the control points. The path points of the flight trajectory are represented as: FP0(x'0,y'0,z'0), FP1(x'1,y'1,z'1),…,FP n (x' n ,y' n ,z' n ), n+1 represents the number of path points. The path points are obtained by the following formula:

[0073]

[0074] Among them, E j,d (·) represents the basis function of the jth segment of the d-order B-spline curve, τ i is the i+1th sample value in the domain [0,s+d], satisfying τ0=0, B j,d (·) The recursive definition is as follows:

[0075]

[0076]

[0077] Among them, r j ∈R={r0,r1,…,r k}, represents the jth node of the B-spline curve, satisfying:

[0078]

[0079] Embodiment 2

[0080] In this embodiment, a multi-objective adaptive constraint UAV path planning method based on reinforcement learning is as follows: Figure 1 , the steps include:

[0081] Initialize the drone path population, including the main population, auxiliary population and output population; in one embodiment, including the initialization population size N, the main population P1, the auxiliary population P2, the output population P3, the maximum neighbor angle σ, the main network Q1 and the target network Q2 in the deep Q network, the maximum number of iterations T max wait.

[0082] Determine the population status, including the proportion of feasible individuals in the main population, the proportion of feasible non-dominated individuals to non-dominated individuals, the center point, the standard deviation, and the status flag determined based on the auxiliary population;

[0083] According to the population state, the action is obtained by using the reinforcement learning network and stored in the output population; in one embodiment, the reinforcement learning network includes a main network Q1 and a target network Q2, and the structures of the main network Q1 and the target network Q2 are as follows:

[0084] The number of nodes in the network input layer is the dimension of the state St; the number of nodes in the output layer is the number of candidate constraint processing technologies, which is set to 3; the number of hidden layers is 5, and the number of nodes in each layer is 8, 16, 32, 16, and 8 respectively; Exponential Linear Units (ELU) is used as the activation function, and all nodes in each layer are fully connected. The network weights are randomly generated.

[0085] Determine whether the termination condition is reached. If not, update the main population, auxiliary population and output population respectively and continue to iterate. If yes, use all non-dominated individuals in the output population as the optimized drone path.

[0086] In one embodiment, calculating the population state includes:

[0087] 1) Calculate the proportion of feasible individuals γ in the main population P1:

[0088]

[0089] Among them, P1 represents the main population, v is the number of feasible individuals in P1, and |P1| is the size of population P1;

[0090] 2) Calculate the ratio ψ of feasible non-dominated individuals to non-dominated individuals in the main population P1. Non-dominated individuals refer to individuals that are not dominated by other individuals. The expression is:

[0091]

[0092] in, is the number of feasible non-dominated individuals in P1, κ is the number of non-dominated individuals in P1;

[0093] 3) Calculate the center point of the main population P1 Center Point is the average value of the objective function of all individuals in the main population; when calculating, first determine the target value of all individuals in the main population P1, and then form a vector based on the average value of each objective function

[0094] 4) Calculate the standard deviation of the main population P1 First, calculate all individuals x in the main population P1 i The target value y i , and then calculate according to the formula as follows:

[0095]

[0096] in, represents the center point, y represents all individual x i The corresponding target value, is y and The Euclidean distance of

[0097] 5) The calculation formula of the status flag determined based on the auxiliary population is:

[0098]

[0099] Among them, μ j is the rate of updating individuals in the auxiliary population P2, μ thr is a preset parameter, and J is a set consisting of 10 consecutive generations.

[0100] In one embodiment, if Figure 2 ,

[0101] According to the population status Use the reinforcement learning network to get actions and store them in the output population;

[0102] In this embodiment, the action a is randomly selected with a probability of ξ, that is, with a probability of (1-ξ), the state St is input into the main network Q1 in the deep Q network, and the value Q of all actions is output. The action corresponding to the maximum Q value is the selected action a.

[0103]

[0104] Where A is a set of candidate constraint processing technology indices.

[0105] In a preferred embodiment, when updating the reinforcement learning network, the reward is calculated according to the following formula:

[0106]

[0107] In the formula, R' represents the intermediate variable, IGD represents the reverse intergenerational distance of the main population, and IGD' represents the reverse intergenerational distance of the main population after update. It represents the proportion of feasible individuals in the main population after update, rounded down, χ represents the number of individuals in the auxiliary population after update, θ and θ' are the sum of the constraint violation values ​​of all individuals in the next generation main population P1' and the next generation output population P3' respectively, λ represents the standard deviation of all individuals in the main population in the target space after update, λ' represents the standard deviation of all individuals in the output population in the target space after update, and flag represents the state mark.

[0108] The reverse generational distance calculation formula is:

[0109]

[0110] In the formula, y is the target value of individual x, md(y ， P1) is the Euclidean distance of individual x from the nearest individual in the main population P1 in the target space, P2 represents the auxiliary population, and P'1 represents the updated main population.

[0111] In one embodiment, it is determined whether the termination condition is met. If not, the main population P1 is updated to the next generation main population P1', the auxiliary population P2 is updated to the next generation auxiliary population P2', and the output population P3 is updated to the next generation output population P'3, and iteration continues; if all non-dominated individuals in the output population are used as a set of optimized drone flight paths.

[0112] When continuing to iterate, first calculate the state St' of the next generation main population P1' to obtain the training sample {St, a, R, St'}. This embodiment stores it in the experience pool, which is a queue. When the queue is full, the oldest training sample will be deleted and then a new training sample will be added.

[0113] Then, train and update the deep Q network. Using the gradient descent method, randomly select b training samples from the experience pool E and train w times. Use the following loss function loss to train the main network Q1,

[0114]

[0115] In the formula, R represents the reward, λ represents the discount factor of future rewards, It is the value of the action a' with the largest output after inputting the state St' into the target network Q2. St' represents the updated population state, Q2 represents the target network, Q1 represents the main network, and Q1(St, a) is the action value corresponding to action a after inputting the state St into the main network Q1.

[0116] Furthermore, the target network Q2 is updated periodically, and every 10 generations, Q2 is replaced by the main network Q1.

[0117] Embodiment 3

[0118] This embodiment further provides different population update methods, including:

[0119] First, when updating the main population, the constraint processing update method is selected using the action as an index; the constraint processing categories include target value priority, boundary value priority, and constraint dominance principle; in one embodiment, the corresponding constraint processing method is determined based on the range of action value.

[0120] (1) When selecting target value priority (ICV), obtain the offspring population P of the main population o And merge with the main population P1, determine the number of other individuals that each individual in the merged population dominates without constraint, calculate the fitness of all individuals in the merged population, and take the population after removing the individual with the largest fitness as the next generation main population P1';

[0121] Unconstrained dominance is defined as follows:

[0122] For two individuals x1 and x2, if all objective function values ​​of x1 are not worse than those of x2, and on at least one objective function, the value of x1 is strictly better than that of x2, then x1 dominates x2 without constraints.

[0123] (2) When the boundary value ε is selected first, the offspring population of the main population is obtained and merged with the main population, the number of other individuals dominated by the boundary constraints of each individual in the merged population is determined, and the fitness of all individuals in the merged population is calculated, and the population after removing the individual with the largest fitness is used as the next generation main population P1';

[0124] Boundary constraint dominance is defined as: Calculate the individual x according to the constraint function i The constraint violation value G i , for two individuals x1 and x2, if G1 and G2 are both less than or equal to the boundary value ε or G1 is equal to G2, and all objective function values ​​of x1 are not worse than x2, and at least in one objective function, the value of x1 is strictly better than x2, then the x1 boundary constraint dominates x2; if G2 is greater than the boundary value ε, and G1 is less than G2, then the x1 boundary constraint dominates x2;

[0125] In this embodiment, the boundary value ε is determined as follows:

[0126]

[0127] In the formula, ε0 is the maximum constraint violation value among all individuals in the initial population, t represents the current iteration number, and T max Indicates the maximum number of iterations.

[0128] (3) When the constraint dominance principle (CDP) is selected, the offspring population of the main population is obtained and merged with the main population, the number of other individuals constrained and dominated by each individual in the merged population is determined, and the fitness of all individuals in the merged population is calculated, and the population after removing the individual with the largest fitness is used as the next generation main population P1';

[0129] Constraint dominance is defined as: Calculate the individual x according to the constraint function i The constraint violation value G i For two individuals x1 and x2, if G1 is equal to G2, and all objective function values ​​of x1 are not worse than x2, and at least in one objective function, the value of x1 is strictly better than x2, then the x1 constraint dominates x2; if G1 is less than G2, then the x1 constraint dominates x2.

[0130] According to an embodiment of the present invention, during the above population update process, the fitness calculation method is:

[0131] Fit i =R i +D i

[0132]

[0133] In the formula, Fit i represents individual fitness, R i represents the original fitness of the individual, D i represents individual density, P represents the combined population, x j >x i Represents x j Dominate x i ,s i Represents individual x i The number of other individuals in the combined population is controlled in the desired way, and τ represents x i Distance The Euclidean distance between individuals.

[0134] To further optimize the above technical solution, the offspring population of the main population is obtained through the following steps:

[0135] ①Merge the main population P1 and the output population P3 to obtain the secondary main population P4. Based on each individual in the secondary main population, the neighbor set is assigned according to the following formula;

[0136]

[0137] Where σ represents the maximum neighbor angle, yi The individual x i The target value of is composed of the target vector, P4 represents the secondary main population, θ(y i ,y j ) is the vector y i and i The angle formed is calculated as follows:

[0138]

[0139] ②For each individual x i , select two individuals from the corresponding neighbor set, and use the individual with small unconstrained dominance fitness as the parent individual, that is, randomly select two individuals from the neighbor set Nei i Select 2 individuals x j and x k , compared with its fitness Fit j and Fit k , if Fit j <Fit k , select x j As the parent individual; otherwise, select x k As the parent individual. Repeat the above steps and select two parent individuals x r1 and x r2 .

[0140] In this embodiment, we first calculate each individual x in the main population P1 i The number of other individuals S in P4 without constraint domination 1i ;

[0141] Then calculate x i The original fitness R i ;

[0142]

[0143] And calculate individual x i The density D i In this embodiment, individual x i The Euclidean distances to all individuals in P4 are stored in a list. After sorting the list in ascending order, x i Distance The Euclidean distance between individuals is τ, |P4| is the number of individuals in population P4; at this time, the density D i for:

[0144]

[0145] Finally, calculate the fitness Fit i , the formula is as follows:

[0146] Fiti =R i +D i

[0147] ③ For the parent individual x r1 and x r2 Perform mutation to obtain individual u i ; The formula is as follows:

[0148] u i =x i +F×(x r1 -x r2 )

[0149] Among them, F is the coefficient that controls the degree of mutation.

[0150] ④ For individual u i Each component u' i Restored to individual x with probability (1-CR) i The corresponding component x' i ,get And each individual x in the main population P1 i The resulting offspring Put into set P o , where CR is the mutation rate.

[0151] Preferably, if the individual If a component of violates the boundary condition, the component is repaired. The repair rules are as follows:

[0152]

[0153] Among them, L j and R j are the lower and upper bounds of the j-th decision variable, respectively. Represents an individual is the jth decision variable.

[0154] Second, update the auxiliary population P2. o All individuals in Do the following:

[0155] Individual Merge with auxiliary population P2 to get the population. Calculate each individual x in the merged population i Constrain the number of other individuals S 2i .

[0156] According to S 2i Get the fitness Fit of all individuals in the merged population.

[0157] Remove the individual with the highest fitness in the merged population and use it as the auxiliary population P2' for the next generation.

[0158] Third, update the output population P3. o All individuals in Do the following:

[0159] Individual Merge with the output population P3 to obtain the merged population, and calculate each individual x in the merged population i Constrain the number of other individuals S 3i .

[0160] According to S 3i Calculate the fitness Fit of all individuals in the merged population.

[0161] Remove the individual with the highest fitness from population P4 and use it as the output population P3' for the next generation.

[0162] Embodiment 4

[0163] This embodiment provides an optional constrained multi-objective optimization problem model for drone path planning, including two objective functions and four constraint functions, as follows:

[0164] (1) The first objective function is the UAV path length, which is formulated as follows:

[0165]

[0166] Among them, FP j represents the j-th path point of the UAV.

[0167] (2) The second objective function is the distance between the drone and the obstacle, and the formula is as follows:

[0168]

[0169] Among them, ds is the safety distance, d j,k is the distance between the jth path point and the kth grid point, and sp is the distance between a path point FP j And the number of all terrain grid points within ds from the path point after all terrain grid points are projected onto the horizontal plane.

[0170] (3) The first constraint function is used to constrain the flight slope of the UAV. The formula is as follows:

[0171]

[0172]

[0173] in, They are the maximum climb and minimum dive slopes allowed between adjacent path points of the drone.

[0174] (4) The second constraint function is used to constrain the turning angle of the drone. The formula is as follows:

[0175]

[0176] in, is the maximum turning angle, v j,j-1 is the waypoint FP j To FP j-1 The vector of θ(v j,j-1 ,v j+1,j ) is the vector v j,j-1 and v j+1,j The angle is as follows:

[0177]

[0178] (5) The third constraint function is used to constrain the minimum flight altitude of the UAV. The formula is as follows:

[0179]

[0180] Among them, z j , They represent the flight altitude of the UAV at the jth path point and the height of the terrain below the path point respectively, and h is the minimum flight altitude of the UAV.

[0181] (6) The fourth constraint function is used to constrain the maximum flight altitude of the UAV. The formula is as follows:

[0182]

[0183] Among them, H is the maximum flight altitude of the drone.

[0184] In this embodiment, the parameters of the multi-objective optimization problem are set, including the number of control points s of the B-spline curve, the safety distance d safe , minimum flight altitude h, maximum flight altitude H, maximum turning angle Set the starting and ending coordinates of the drone flight.

[0185] Based on the general framework of the constrained multi-objective evolutionary algorithm, this paper introduces an adaptive selection constraint processing technology strategy to deal with the multiple constraint problems of UAV path planning and ensure the feasibility of the flight path. At the same time, the online feedback mechanism of deep reinforcement learning is used to dynamically update the selection strategy according to the actual feedback information, which can significantly improve the efficiency and adaptability of UAV path planning.

[0186] In order to further demonstrate the advantages of the adaptive constrained multi-objective evolutionary algorithm UAV path planning method based on deep reinforcement learning proposed in the present invention, the results of the comparative experiments between the method of the present invention and other algorithms are briefly described below.

[0187] The terrain generation function is designed as follows:

[0188]

[0189] Among them, ρ1, ρ2, ρ3, ρ4, ρ5, ρ6 and ρ7 are control parameters for simulating three-dimensional environmental terrain. They work together to generate a smooth terrain surface with features such as mountains and valleys. (x, y, z) are the coordinates of terrain points in the ground coordinate system. In this embodiment, ρ1, ρ2, ρ3, ρ4, ρ5, ρ6 and ρ7 are set to 1.2, 2, 0.3, 0, 0.2, 2 and 0.1 respectively, and the sampling accuracy span of the terrain points is set to 0.25 units. At this time, the determined three-dimensional environmental terrain is as shown in the attached figure. Figure 3 shown.

[0190] As shown in Table 1, multiple test problems are generated by adjusting the parameters of the generated terrain function and the multi-objective optimization problem.

[0191] Table 1

[0192]

[0193] This application will use a proposed adaptive constrained multi-objective evolutionary algorithm based on deep reinforcement learning to conduct experimental comparisons with the CMODQLMT, CCMO, and cDPEA algorithms on a test set, where the initialization parameters of these original algorithms are set consistently.

[0194] Table 2 shows the average HV and its STD obtained by running each algorithm 30 times independently on three test problems. The bold part represents the optimal value in the same test problem.

[0195] Table 2

[0196]

[0197]

[0198] The hypervolume matrix (HV) and its standard deviation (STD) are used to evaluate the performance of each algorithm. The larger the HV and the smaller the STD, the better the algorithm performance.

[0199] Table 2 shows the HV and STD values ​​obtained after running each algorithm and the algorithm using the method of the present invention on each test problem. Table 3 shows the proportion of the number of runs of each algorithm that obtained a feasible solution after running 30 times independently on the three test problems.

[0200] The HV in the above experimental results shows that the algorithm method of the present invention exhibits good convergence performance in the UAV path planning, that is, the flight path is shorter under the premise of meeting the feasibility. The STD shows that the algorithm method of the present invention exhibits good stability in the UAV path planning.

[0201] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0202] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-objective adaptive constraint UAV path planning method based on reinforcement learning, characterized in that: Initialize the drone path population, including the main population, auxiliary population and output population; Determine the population status, including the proportion of feasible individuals in the main population, the proportion of feasible non-dominated individuals to non-dominated individuals, the center point, the standard deviation, and the status flag determined based on the auxiliary population; According to the population state, the action is obtained using the reinforcement learning network and stored in the output population; Determine whether the termination condition is reached. If not, select the adaptive constraint strategy based on the action as the index to update the population and continue iterating. If yes, output all non-dominated individuals in the population as the optimized UAV flight path.

2. The path planning method according to claim 1, characterized in that: The center point is the average value of the objective function of all individuals in the main population; The formula for calculating the standard deviation is: in, represents the center point, y represents all individual x i The corresponding target value, is y and The Euclidean distance of The calculation formula for the status flag determined based on the auxiliary population is: Among them, μ j is the rate of updating individuals in the auxiliary population P2, μ thr is a preset parameter, and J is a set consisting of 10 consecutive generations.

3. The path planning method according to claim 1, characterized in that: The reinforcement learning network calculates the reward as follows; R=max(-20,min(20,R')) In the formula, R' represents the intermediate variable, IGD represents the reverse intergenerational distance of the main population, and IGD' represents the reverse intergenerational distance of the main population after update. represents the proportion of feasible individuals in the main population after the update, rounded down, χ represents the number of individuals in the auxiliary population after the update, θ represents the sum of the constraint violation values ​​of all individuals in the main population after the update, θ' represents the sum of the constraint violation values ​​of all individuals in the output population after the update, represents the standard deviation of all individuals in the main population in the target space after update, It indicates the standard deviation of all individuals in the updated output population in the target space, and flag indicates the state mark.

4. The path planning method according to claim 3, characterized in that: The reverse generational distance calculation formula is: Where y is the target value of individual x, md(y, P1) is the Euclidean distance of individual x from the nearest individual in the main population P1 in the target space, and P2 represents the auxiliary population.

5. The path planning method according to claim 1, characterized in that: The reinforcement learning network is trained using the following loss function: In the formula, R represents the reward, λ represents the discount factor of future rewards, It is the value of the action a' with the largest output after inputting the state St' into the target network Q2. St' represents the updated population state, Q2 represents the target network, Q1 represents the main network, and Q1(St, a) is the action value corresponding to action a after inputting the state St into the main network Q1.

6. The path planning method according to claim 1, characterized in that: When updating the main population, the constraint processing update mode is selected using the action as an index; the constraint processing categories include target value priority, boundary value priority and constraint dominance principle; When selecting the target value first, obtain the offspring population of the main population and merge it with the main population, determine the number of other individuals that each individual in the merged population can dominate without constraints, calculate the fitness of all individuals in the merged population, and remove the individual with the largest fitness; When the boundary value is selected first, the offspring population of the main population is obtained and merged with the main population, the number of other individuals dominated by the boundary constraints of each individual in the merged population is determined, the fitness of all individuals in the merged population is calculated, and the individual with the largest fitness is removed; Boundary constraint dominance is defined as: Calculate the individual x according to the constraint function i The constraint violation value G i , for two individuals x1 and x2, if G1 and G2 are both less than or equal to the boundary value ε or G1 is equal to G2, and all objective function values ​​of x1 are not worse than x2, and at least in one objective function, the value of x1 is strictly better than x2, then the x1 boundary constraint dominates x2; if G2 is greater than the boundary value ε, and G1 is less than G2, then the x1 boundary constraint dominates x2; When the constraint dominance principle is selected, the offspring population of the main population is obtained and merged with the main population, the number of constraints and dominance of other individuals by each individual in the merged population is determined, the fitness of all individuals in the merged population is calculated, and the individual with the largest fitness is removed; Constraint dominance is defined as: Calculate the individual x according to the constraint function i The constraint violation value G i For two individuals x1 and x2, if G1 is equal to G2, and all objective function values ​​of x1 are not worse than x2, and at least in one objective function, the value of x1 is strictly better than x2, then the x1 constraint dominates x2; if G1 is less than G2, then the x1 constraint dominates x2.

7. The path planning method according to claim 6, characterized in that: The boundary value ε is determined as follows: In the formula, ε0 is the maximum constraint violation value among all individuals in the initial population, t represents the current iteration number, and T max Indicates the maximum number of iterations.

8. The path planning method according to claim 6, characterized in that: The offspring population of the main population is obtained through the following steps: The main population and the output population are merged to obtain the secondary main population, and a neighbor set is assigned to each individual in the secondary main population; For each individual x i , select two individuals from the corresponding neighbor set, take the individual with small unconstrained dominance fitness as the parent individual, traverse the neighbor set, and determine the final parent individual x r1 and x r2 ; For parent individual x r1 and x r2 Perform mutation to obtain individual u i ; For individual u i Each component u' i Restored to individual x with probability (1-CR) i The corresponding component x' i ,get 9. The path planning method according to claim 8, characterized in that: The neighbor set is allocated according to the following formula: Where σ represents the maximum neighbor angle, y i The individual x i The target value of θ(y i ,y j ) is the vector y i and i The angle formed, P4 represents the secondary main population.

10. The path planning method according to claim 6 or 8, characterized in that: The calculation steps of individual fitness include: Fit i =R i +D i In the formula, Fit i represents individual fitness, R i represents the original fitness of the individual, D i represents individual density, P represents the combined population, Represents x j Dominate x i , S i Represents individual x i The number of other individuals in the combined population is controlled in the desired way, and τ represents x i Distance The Euclidean distance between individuals.

Images