Multi-objective adaptive constraint unmanned aerial vehicle path planning method based on reinforcement learning

Abstract

The application discloses a kind of multi-objective adaptive constraint unmanned aerial vehicle path planning methods based on reinforcement learning, steps include initializing unmanned aerial vehicle path population;Determine population state, including the proportion of feasible individual in main population, the proportion of feasible non-dominated individual and non-dominated individual, center point, standard deviation and the state flag determined based on auxiliary population;According to population state, action is obtained using reinforcement learning network, and is stored to output population;Judge whether to reach termination condition, if no with action as index, select adaptive constraint strategy to update population and continue iteration, if yes with all non-dominated individuals in output population as optimized unmanned aerial vehicle flight path.The application can effectively deal with multiple constraint challenges in unmanned aerial vehicle path planning process, dynamically update constraint strategy through reinforcement learning online feedback, so as to improve the efficiency and adaptability of path planning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle path planning, and more particularly to a multi-target adaptive constraint unmanned aerial vehicle path planning method based on reinforcement learning. BACKGROUND

[0002] Unmanned aerial vehicles have great application prospects in many fields due to their excellent maneuverability and stable performance, for example, in logistics distribution, city management, emergency rescue and many other fields. The application of unmanned aerial vehicle technology can not only greatly improve the operation efficiency and response speed, but also reduce the operating cost. In addition, unmanned aerial vehicles have great potential in environmental monitoring, traffic management, disaster assessment and many other aspects. How to efficiently and safely plan the flight path of unmanned aerial vehicles in complex and diverse flight environments is one of the core problems in the current related research field.

[0003] Currently, the methods for solving the unmanned aerial vehicle path planning problem can be mainly divided into the following three categories: node optimization-based path planning algorithm, sampling-based path planning algorithm and artificial intelligence-based path planning algorithm.

[0004] The node optimization-based algorithm constructs a loss function and uses heuristic information or traversal to find the optimal path. This kind of algorithm has high computational complexity, especially in the case of numerous nodes and complex network, the calculation amount will increase sharply, resulting in reduced efficiency of planning path.

[0005] The sampling-based algorithm randomly samples the environment and generates a route map, and then searches for a feasible path from the starting point to the ending point. As the complexity of the environment increases, the quality of the path may further decrease, and there are problems of high computational complexity and possible failure to find the optimal solution.

[0006] The artificial intelligence-based path planning algorithm, such as swarm intelligence algorithm and reinforcement learning, has environmental interaction, which enables it to adapt to complex flight environments, and is an important direction in the field of unmanned aerial vehicle path planning. The existing mainstream method models the path planning of unmanned aerial vehicles as a constrained multi-objective optimization problem, and then uses algorithms to solve it, but this method makes it difficult for unmanned aerial vehicles to meet the constraints when facing complex environments, i.e. the algorithm is difficult to find the optimal flight path.

[0007] Therefore, a new algorithm needs to be developed to adjust the constraint processing method according to the environment, so that the algorithm focuses on finding the optimal flight path. SUMMARY

[0008] Therefore, the application provides a multi-objective adaptive constraint UAV path planning method based on reinforcement learning.

[0009] In order to achieve the above object, the application adopts the following technical scheme:

[0010] A multi-objective adaptive constraint UAV path planning method based on reinforcement learning comprises:

[0011] Initializing a UAV path population, including a main population, an auxiliary population and an output population;

[0012] Determining a population state, including a feasible individual proportion in the main population, a feasible non-dominated individual and a non-dominated individual proportion, a center point, a standard deviation and a state flag determined based on the auxiliary population;

[0013] According to the population state, an action is obtained by using a reinforcement learning network and stored in the output population;

[0014] It is judged whether a termination condition is reached, if not, the population is updated by using the action as an index to select an adaptive constraint strategy and iteration is continued, if yes, all non-dominated individuals in the output population are used as optimized UAV flight paths.

[0015] Preferably, the feasible individual proportion calculation formula is:

[0016]

[0017] Wherein, P1 represents the main population, v is the number of feasible individuals in P1, and |P1| is the size of the population P1; the feasible non-dominated individual and the non-dominated individual proportion is expressed as:

[0018]

[0019] Wherein, is the number of feasible non-dominated individuals in P1, and κ is the number of non-dominated individuals in P1;

[0020] The center point is the average value of all individual objective functions in the main population;

[0021] The calculation formula of the standard deviation is:

[0022]

[0023] Wherein, represents the center point, y represents all individual x i corresponding target value, is the Euclidean distance of y and .

[0024] The state flag is determined based on the auxiliary population, and the calculation formula is:

[0025]

[0026] wherein μ j is the ratio of updating individuals of the auxiliary population P2, μ thr is a preset parameter, and J is a set consisting of every 10 continuous generations.

[0027] As preferred, the reinforcement learning network calculates the reward according to the following formula:

[0028]

[0029] wherein R' represents an intermediate variable, IGD represents the inverse intergenerational distance of the main population, IGD' represents the inverse intergenerational distance of the updated main population, represents the proportion of feasible individuals of the updated main population, χ represents the number of individuals of the updated auxiliary population, λ represents the standard deviation of all individuals in the target space of the updated main population, λ' represents the standard deviation of all individuals in the target space of the output population, and flag represents the state flag.

[0030] As preferred, the inverse intergenerational distance calculation formula is:

[0031]

[0032] wherein y is the target value of individual x, md(y, P1) is the Euclidean distance of individual x in the target space from the nearest individual in the main population P1, P2 represents the auxiliary population, and P'1 represents the updated main population.

[0033] As preferred, the reinforcement learning network is trained using the following loss function:

[0034]

[0035] wherein R represents the reward, λ represents the discount factor of future rewards, is the value of the action a' with the maximum output after inputting the state St' into the target network Q2, St' represents the state of the updated population, Q2 represents the target network, Q1 represents the main network, and Q1(St, a) is the action value corresponding to the action a after inputting the state St into the main network Q1.

[0036] As preferred, when updating the main population, the constraint processing update mode is selected by taking the action as an index; the constraint processing categories include target value priority, boundary value priority, and constraint dominance principle.

[0037] When the target value is prioritized, the offspring population of the main population is obtained and merged with the main population. The number of unconstrained dominance of each individual in the merged population is determined, and the fitness of all individuals in the merged population is calculated, and the individual with the largest fitness is removed.

[0038] When the boundary value is prioritized, the offspring population of the main population is obtained and merged with the main population. The number of other individuals dominated by the boundary constraints of each individual in the merged population is determined, and the fitness of all individuals in the merged population is calculated, and the individual with the largest fitness is removed.

[0039] Boundary constraint dominance is defined as: calculating the individual x according to the constraint function i The constraint violation value G i , for two individuals x1 and x2, if G1 and G2 are both less than or equal to the boundary value ε or G1 is equal to G2, and all objective function values ​​of x1 are not worse than x2, and on at least one objective function, the value of x1 is strictly better than x2, then the x1 boundary constraint dominates x2; if G2 is greater than the boundary value ε, and G1 is less than G2, then the x1 boundary constraint dominates x2;

[0040] When the constraint dominance principle is selected, the offspring population of the main population is obtained and merged with the main population. The number of constraints and dominance of other individuals by each individual in the merged population is determined, and the fitness of all individuals in the merged population is calculated, and the individual with the largest fitness is removed.

[0041] Constraint dominance is defined as: calculating the individual x according to the constraint function i The constraint violation value G i For two individuals x1 and x2, if G1 is equal to G2, and all objective function values ​​of x1 are not worse than x2, and on at least one objective function, the value of x1 is strictly better than x2, then the x1 constraint dominates x2; if G1 is less than G2, then the x1 constraint dominates x2.

[0042] Preferably, the boundary value ε is determined as follows:

[0043]

[0044] Where ε0 is the maximum constraint violation value among all individuals in the initial population, t represents the current iteration number, and T max Indicates the maximum number of iterations.

[0045] Preferably, the offspring population of the main population is obtained by the following steps:

[0046] Merge the main population and the output population to obtain the secondary main population, and assign a neighbor set to each individual in the secondary main population;

[0047] For each individual x iFrom the optional two individuals in the corresponding neighbor set, the individual with the smallest unconstrained fitness is selected as the parent individual, the neighbor set is traversed to determine the final parent individual x r1 and x r2 ;

[0048] The parent individual x r1 and x r2 is mutated to obtain individual u i ;

[0049] Each component u' i in the individual u i is restored to the corresponding component x' i in the individual x i with a probability of (1-CR) to obtain

[0050] As a preference, the neighbor set is allocated as follows,

[0051]

[0052] In the formula, σ represents the maximum neighbor angle, y i is a target vector composed of the target values of the individual x i , θ(y i , y j ) is the included angle formed by the vectors y i and y i , and P4 represents the sub-main population.

[0053] As a preference, the step of calculating the individual fitness includes:

[0054] Fit i = R i + D i

[0055]

[0056] In the formula, Fit i represents the individual fitness, R i represents the original fitness of the individual, D i represents the individual density, P represents the merged population, x j > x i represents that x j dominates x i , S i represents the number of other individuals in the merged population that are dominated by the individual x i in the required constraint manner, and τ represents the Euclidean distance of x i from the first individual.

[0057] Via the technical solution, the application discloses a multi-target adaptive constraint UAV path planning method based on reinforcement learning,

[0058] 1) The method utilizes the online feedback mechanism of deep reinforcement learning, can dynamically update the constraint selection strategy according to different scenes, and has the characteristics of strong adaptability and high efficiency.

[0059] 2) The application proposes an adaptive constraint processing technology for solving the multiple constraint problems of UAV path planning in complex environments, thereby improving the efficiency of path planning.

[0060] 3) The method uses B-spline curves to ensure the smoothness of the flight path of the UAV, meets the flight requirements of the UAV, and has practicality. BRIEF DESCRIPTION OF DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0062] Figure 1 The application provides a multi-target adaptive constraint UAV path planning method based on reinforcement learning.

[0063] Figure 2 It is the principle diagram of deep reinforcement learning used in the application.

[0064] Figure 3 It is a three-dimensional environment terrain in the embodiment of the application. DETAILED DESCRIPTION

[0065] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, not all. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0066] The embodiment of the application discloses a multi-target adaptive constraint UAV path planning method based on reinforcement learning. The method introduces an innovative adaptive constraint processing technology strategy on the general framework of constraint multi-target evolutionary algorithm, which is used to effectively cope with the multiple constraint challenges in the UAV path planning process. At the same time, the method utilizes the online feedback mechanism of deep reinforcement learning to dynamically update the optimal selection strategy, thereby improving the efficiency and adaptability of path planning.

[0067] Embodiment one

[0068] Initialize the population of UAV paths, in this embodiment, the UAV paths are determined based on a series of path points; and the path point obtaining step comprises:

[0069] 1. Obtain the three-dimensional environmental terrain of the flight area; that is, import the terrain data or construct the three-dimensional environmental terrain according to a terrain generation function;

[0070] 2. Construct a path representation model based on B-spline curve based on the three-dimensional terrain environment; specifically comprising:

[0071] 2.1. Represent the flight path by using B-spline curve, the B-spline curve is generated by a group of control points, each control point corresponds to a coordinate on the three-dimensional Cartesian coordinate system, and the control point is represented as: CP0(x0, y0, z0), CP1(x1, y1, z1), …, CPs(xs, ys, zs), s represents the number of control points. s-1 (x s-1 ,y s-1 ,z s-1 ), s represents the number of control points.

[0072] 2.2. A flight trajectory can be constructed according to the control points by using uniform sampling. The path points of the flight trajectory are represented as: FP0(x'0, y'0, z'0), FP1(x'1, y'1, z'1), …, FPn+1(x'n+1, yn+1, zn+1), n+1 represents the number of path points. The path points are derived from the following formula: n (x' n ,y' n ,z' n ), n+1 represents the number of path points. The path points are derived from the following formula:

[0073]

[0074] Wherein, E j,d (·) represents the base function of the jth segment of the d-order B-spline curve, τ i is the i+1th sampling value in the domain [0, s+d], which satisfies τ0=0, B j,d (·) is recursively defined as follows:

[0075]

[0076]

[0077] Wherein, r j ∈R={r0, r1, …, r k}, represents the jth node of the B-spline curve, which satisfies:

[0078]

[0079] Embodiment Two

[0080] In this embodiment, the multi-objective adaptive constraint UAV path planning method based on reinforcement learning includes the following steps: Figure 1

[0081] Initializing the UAV path population, including the main population, the auxiliary population and the output population; in one embodiment, it includes initializing the population size N, the main population P1, the auxiliary population P2, the output population P3, the maximum neighbor angle σ, the main network Q1 and the target network Q2 in the deep Q network, the maximum iteration number T max , and the like.

[0082] Determining the population state, including the proportion of feasible individuals in the main population, the proportion of feasible non-dominated individuals and non-dominated individuals, the center point, the standard deviation and the state flag determined based on the auxiliary population;

[0083] According to the population state, the action is obtained by using the reinforcement learning network, and is stored to the output population; in one embodiment, the reinforcement learning network includes the main network Q1 and the target network Q2, and the structures of the main network Q1 and the target network Q2 are as follows:

[0084] The number of nodes in the input layer of the network is the dimension of the state St; the number of nodes in the output layer is the number of candidate constraint processing technologies, that is, 3; the number of nodes in each layer is 8, 16, 32, 16 and 8 respectively; the Exponential Linear Units (ELU) is used as the activation function, and full connection is performed between the nodes of each layer. The network weights are randomly generated.

[0085] It is judged whether the termination condition is reached, if not, the main population, the auxiliary population and the output population are updated respectively, and then the iteration is continued, if yes, all the non-dominated individuals in the output population are taken as the optimized UAV path.

[0086] In one embodiment, calculating the population state includes:

[0087] 1) Calculating the proportion γ of feasible individuals in the main population P1:

[0088]

[0089] Wherein, P1 represents the main population, v is the number of feasible individuals in P1, and |P1| is the size of the population P1;

[0090] 2) Calculating the proportion ψ of feasible non-dominated individuals and non-dominated individuals in the main population P1, the non-dominated individual refers to the individual that is not dominated by other individuals, and the expression is:

[0091]

[0092] ​in, is the number of feasible non-dominated individuals in P1, κ is the number of non-dominated individuals in P1;

[0093] 3) Calculate the center point of the main population P1 Center Point is the average value of the objective function of all individuals in the main population; when calculating, first determine the target value of all individuals in the main population P1, and then form a vector based on the average value of each objective function

[0094] 4) Calculate the standard deviation of the main population P1 First calculate all individuals x in the main population P1 i The target value y i , and then calculate according to the formula as follows:

[0095]

[0096] in, represents the center point, y represents all individual x i The corresponding target value, is y and The Euclidean distance of

[0097] 5) The calculation formula of the status flag determined based on the auxiliary population is:

[0098]

[0099] Among them, μ j is the rate of updating individuals in the auxiliary population P2, μ thr is a preset parameter, and J is a set consisting of every 10 consecutive generations.

[0100] In one embodiment, if Figure 2 ,

[0101] According to population status Use the reinforcement learning network to obtain actions and store them in the output population;

[0102] In this embodiment, the action a is randomly selected with a probability of ξ, that is, with a probability of (1-ξ), the state St is input into the main network Q1 in the deep Q network, and the value Q of all actions is output. The action corresponding to the maximum Q value is the selected action a.

[0103]

[0104] Where A is a set of candidate constraint processing technology indices.

[0105] In a preferred embodiment, when updating the reinforcement learning network, the reward is calculated according to the following formula:

[0106]

[0107] wherein R' represents an intermediate variable, IGD represents the inverse generational distance of the main population, IGD' represents the inverse generational distance of the updated main population, represents the proportion of feasible individuals in the updated main population rounded down, χ represents the number of individuals in the updated auxiliary population, θ and θ' are the sum of constraint violation values of all individuals in the next generation main population P1' and the output population P3' of the next generation respectively, λ represents the standard deviation of all individuals in the updated main population in the target space, λ' represents the standard deviation of all individuals in the output population in the target space, and flag represents a state flag.

[0108] wherein the inverse generational distance is calculated by the following formula:

[0109]

[0110] wherein y represents the target value of individual x, md(y ， P1) represents the Euclidean distance of individual x in the target space from the nearest individual in the main population P1, P2 represents the auxiliary population, and P1' represents the updated main population.

[0111] In an embodiment, it is determined whether the termination condition is reached. If not, the main population P1 is updated to the next generation main population P1', the auxiliary population P2 is updated to the next generation auxiliary population P2', and the output population P3 is updated to the next generation output population P'3, and the iteration is continued. If yes, all non-dominated individuals in the output population are taken as a set of optimized UAV flight paths.

[0112] In the iteration, first, the state St' of the next generation main population P1' is calculated, and a training sample {St, a, R, St'} is obtained. In this embodiment, the training sample is stored in an experience pool, which is a queue. When the queue is full, the oldest training sample is deleted, and a new training sample is added.

[0113] Then, the deep Q network is trained and updated. Gradient descent method is used to randomly select b training samples from the experience pool E, and train w times. The following loss function loss is used to train the main network Q1,

[0114]

[0115] wherein R represents a reward, λ represents a discount factor of future rewards, is the value of the action a' with the maximum output when the state St' is input into the target network Q2, St' represents the state of the updated population, Q2 represents the target network, Q1 represents the main network, and Q1(St, a) is the action value corresponding to the action a when the state St is input into the main network Q1.

[0116] Further, the target network Q2 is periodically updated, every 10 generations, by replacing Q2 with the master network Q1.

[0117] Embodiment Three

[0118] The embodiment further provides an updating method for different populations, including:

[0119] First, when updating the master population, the action is used as an index to select a constraint processing updating method; the constraint processing categories include target value priority, boundary value priority, and constraint domination principle; in one embodiment, the corresponding constraint processing method is determined according to the range of action value.

[0120] (1) When selecting target value priority (ICV), the offspring population P2 of the master population P1 is obtained o and merged with the master population P1 to determine the number of individuals in the merged population that are not constrained to dominate other individuals, and to calculate the fitness of all individuals in the merged population, and the population after removing the individual with the maximum fitness is used as the next generation master population P1';

[0121] The definition of unconstrained domination is as follows:

[0122] For two individuals x1 and x2, x1 is not worse than x2 in all objective function values, and x1 is strictly better than x2 in at least one objective function, then x1 unconstrainedly dominates x2.

[0123] (2) When selecting boundary value ε priority, the offspring population of the master population is obtained

[0124] The boundary constraint domination is defined as follows: the constraint violation value G i of individual x i is calculated according to the constraint function, and for two individuals x1 and x2, if G1 and G2 are both less than or equal to the boundary value ε or G1 is equal to G2, and x1 is not worse than x2 in all objective function values, and x1 is strictly better than x2 in at least one objective function, then x1 boundary-constrainedly dominates x2; if G2 is greater than the boundary value ε and G1 is less than G2, then x1 boundary-constrainedly dominates x2.

[0125] In this embodiment, the boundary value ε is determined as follows:

[0126]

[0127] Wherein, ε0 is the maximum constraint violation value of all individuals in the initial population, t represents the current iteration number, T max represents the maximum iteration number.

[0128] (3) When the constraint dominance principle (CDP) is selected, the offspring population of the main population is obtained and merged with the main population, the number of constraint domination of each individual in the merged population over other individuals is determined, the fitness of all individuals in the merged population is calculated, and the population after removing the individual with the maximum fitness is taken as the next generation main population P1';

[0129] The constraint domination is defined as follows: according to the constraint function, the constraint violation value G i of the individual x i is calculated, for two individuals x1 and x2, if G1 is equal to G2, at the same time, all objective function values of x1 are not worse than those of x2, and at least in one objective function, the value of x1 is strictly better than that of x2, then x1 constraint dominates x2; if G1 is less than G2, then x1 constraint dominates x2.

[0130] According to the embodiment of the present application, in the above population updating process, the fitness calculation method is as follows:

[0131] Fit i = R i + D i

[0132]

[0133] Wherein, Fit i represents the individual fitness, R i represents the original fitness of the individual, D i represents the individual density, P represents the merged population, x j > x i represents that x j dominates x i , s i represents the number of other individuals in the merged population that are dominated by the individual x i in the required constraint manner, and τ represents the Euclidean distance of x i from the first individual.

[0134] In order to further optimize the above technical solution, the offspring population of the main population is obtained by the following steps:

[0135] ① Merging the main population P1 and the output population P3 to obtain the secondary main population P4, and assigning a neighbor set to each individual in the secondary main population based on the following formula;

[0136]

[0137] Wherein, σ represents the maximum neighbor angle, yi is a target vector composed of target values of individuals x i , P4 represents a sub-main population, θ(y i , y j ) is an included angle formed by vectors y i and y i , and a calculation formula is as follows:

[0138]

[0139] ②For each individual x i , two individuals are selected at random from the corresponding neighbor set, and the individual with a smaller non-constrained dominated fitness is selected as a parent individual, i.e. two individuals x i and x j are selected from the neighbor set Nei k , and their fitnesses Fit j and Fit k are compared, if Fit j < Fit k , x j is selected as the parent individual; otherwise, x k is selected as the parent individual. The above steps are repeated to select two parent individuals x r1 and x r2 .

[0140] In this embodiment, firstly, the number S i of other individuals in P4 that are non-constrained dominated by each individual x 1i in the main population P1 is calculated;

[0141] then the original fitness R i of x i is calculated;

[0142]

[0143] and the density D i of the individual x i is calculated, in this embodiment, the Euclidean distances of the individual x i to all individuals in P4 are calculated and stored in a list, after the list is sorted in ascending order, the Euclidean distance of x i to the first individual is τ, and |P4| is the number of individuals in the population P4; at this time, the density D i is as follows:

[0144]

[0145] finally, the fitness Fit i is calculated, and the formula is as follows:

[0146] Fiti = R i + D i

[0147] III. Mutation of parent individual x r1 and x r2 to obtain individual u i ; the formula is as follows:

[0148] u i = x i + F x (x r1 - x r2 )

[0149] wherein F is a coefficient for controlling the mutation degree.

[0150] IV. Each component u' i in individual u i is restored to the corresponding component x' i in individual x i with a probability of (1-CR) to obtain and the offspring of each individual x i in main population P1 is put into set P o , wherein CR is the mutation rate.

[0151] As a preference, if a certain component of individual violates the boundary condition, the component is repaired, and the repair rule is as follows:

[0152]

[0153] wherein L j and R j are the lower and upper bounds of the jth decision variable respectively, denotes the jth decision variable of individual .

[0154] Second, update the auxiliary population P2. Perform the following operations on all individuals in offspring population P o :

[0155] merge individual with auxiliary population P2 to obtain population. Calculate the fitness Fit of each individual x i in the merged population, and constrain the number S 2i of individuals dominated by other individuals.

[0156] According to S 2i , the fitness Fit of all individuals in the merged population is obtained.

[0157] The individual with the highest fitness in the merged population is removed and then used as the auxiliary population P2' of the next generation.

[0158] Third, update the output population P3. For all individuals in the offspring population P o The following operations are performed:

[0159] The individual x is merged with the output population P3 to obtain a merged population, and the fitness Fit of each individual x i in the merged population is calculated. 3i .

[0160] The fitness of all individuals in the merged population is calculated according to S 3i .

[0161] The individual with the highest fitness in the population P4 is removed and then used as the output population P3' of the next generation.

[0162] Embodiment Four

[0163] This embodiment provides an optional constraint multi-objective optimization problem model for unmanned aerial vehicle path planning, including two objective functions and four constraint functions, as follows:

[0164] (1) The first objective function is the length of the unmanned aerial vehicle path, and the formula is as follows:

[0165]

[0166] Where FP j represents the jth path point of the unmanned aerial vehicle.

[0167] (2) The second objective function is the distance between the unmanned aerial vehicle and the obstacle, and the formula is as follows:

[0168]

[0169] Where ds is the safety distance, d j,k is the distance between the jth path point and the kth grid point, and sp is the number of all terrain grid points within ds from the path point FP j projected onto the horizontal plane.

[0170] (3) The first constraint function is a constraint function for constraining the flight slope of the unmanned aerial vehicle, and the formula is as follows:

[0171]

[0172]

[0173] Where,​ respectively are the maximum climb and minimum dive slopes allowed by the UAV between adjacent waypoints.

[0174] (4) The second constraint function is a constraint function for restricting the turning angle of the UAV, and the formula is as follows:

[0175]

[0176] wherein, is the maximum turning angle, v j,j-1 is the vector from the waypoint FP j to FP j-1 , and θ(v j,j-1 , v j+1,j ) is the included angle between the vectors v j,j-1 and v j+1,j , and the formula is as follows:

[0177]

[0178] (5) The third constraint function is a constraint function for restricting the minimum flight height of the UAV, and the formula is as follows:

[0179]

[0180] wherein, z j , respectively represent the flight height of the UAV at the jth waypoint and the height of the terrain below the waypoint, and h is the minimum flight height of the UAV.

[0181] (6) The fourth constraint function is a constraint function for restricting the maximum flight height of the UAV, and the formula is as follows:

[0182]

[0183] wherein, H is the maximum flight height of the UAV.

[0184] In this embodiment, the parameters of the multi-objective optimization problem are set. They include the number of control points s of the B-spline curve, the safety distance d safe , the minimum flight height h, the maximum flight height H, and the maximum turning angle The starting point and the ending point coordinates of the UAV flight are set.

[0185] The present application introduces an adaptive selection constraint processing technology strategy on the basis of the general framework of the multi-objective evolutionary algorithm, so as to cope with the multiple constraint problems of the UAV path planning and guarantee the feasibility of the flight path. Meanwhile, the online feedback mechanism of the deep reinforcement learning is utilized to dynamically update the selection strategy according to the actual feedback information, so as to significantly improve the efficiency and adaptability of the UAV path planning.

[0186] To further demonstrate the advantages of the proposed adaptive constraint multi-objective evolutionary algorithm for unmanned aerial vehicle path planning based on deep reinforcement learning, the results of the proposed method and other algorithms in the comparative experiment will be briefly described below.

[0187] The terrain function is designed as follows:

[0188]

[0189] where ρ1, ρ2, ρ3, ρ4, ρ5, ρ6 and ρ7 are control parameters for simulating the terrain of a three-dimensional environment, which jointly generate a smooth terrain surface with features such as mountains and valleys, and (x, y, z) is the coordinate of the terrain point in the ground coordinate system. In this embodiment, ρ1, ρ2, ρ3, ρ4, ρ5, ρ6 and ρ7 are set to 1.2, 2, 0.3, 0, 0.2, 2 and 0.1 respectively, and the sampling precision span of the terrain point is set to 0.25 units. The determined three-dimensional environment terrain is shown in FIG. 2. Figure 3

[0190] As shown in Table 1, a plurality of test problems are generated by adjusting the parameters of the generated terrain function and the multi-objective optimization problem, and the maximum turning angle

[0191] Table 1

[0192]

[0193] The proposed adaptive constraint multi-objective evolutionary algorithm for unmanned aerial vehicle path planning based on deep reinforcement learning will be used for experimental comparison with CMODQLMT, CCMO and cDPEA algorithms on the test set, where the initialization parameters of these original algorithms are set uniformly.

[0194] Table 2 shows the average HV value and its STD obtained by running each algorithm independently for 30 times in three test problems, and the bold part represents the optimal value in the same test problem.

[0195] Table 2

[0196]

[0197]

[0198] The performance of each algorithm is evaluated using two indicators, hyper volume matrix (HV) and its standard deviation (STD), where the larger the HV and the smaller the STD, the better the algorithm performance.

[0199] ​Table 2 shows the HV and STD values of each algorithm and the algorithm using the method of the present application after running in each test problem. Table 3 shows the proportion of running times of each algorithm to obtain feasible solutions after running 30 times independently in 3 test problems.

[0200] The HV in the above experimental results shows that the algorithm method of the present application exhibits good convergence performance in unmanned aerial vehicle path planning, that is, the flight path is shorter under the premise of meeting the feasibility. The STD shows that the algorithm method of the present application exhibits good stability in unmanned aerial vehicle path planning.

[0201] Each of the embodiments in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between each embodiment can be referred to each other. For the device disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0202] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-objective adaptive constraint UAV path planning method based on reinforcement learning, characterized by: Initialize the UAV path population, including the main population, auxiliary population and output population; Determine the population status, including the proportion of feasible individuals in the main population, the ratio of feasible non-dominated individuals to non-dominated individuals, the center point, the standard deviation, and the status flag determined based on the auxiliary population; Based on the population state, the reinforcement learning network is used to obtain actions and store them in the output population; The reinforcement learning network calculates the reward as follows; R=max(-20,min(20,R')) In the formula, R' represents the intermediate variable, IGD represents the reverse intergenerational distance of the main population, and IGD' represents the reverse intergenerational distance of the main population after update. represents the proportion of feasible individuals in the main population after the update, rounded down, χ represents the number of individuals in the auxiliary population after the update, represents the sum of constraint violation values ​​of all individuals in the main population after update, represents the sum of constraint violation values ​​of all individuals in the output population after update, represents the standard deviation of all individuals in the main population in the target space after update, It represents the standard deviation of all individuals in the updated output population in the target space, and flag represents the status mark; The formula for calculating the reverse generational distance is: Where y is the target value of individual x, md(y, P1) is the Euclidean distance of individual x from the nearest individual in the main population P1 in the target space, and P2 represents the auxiliary population; Determine whether the termination condition is met. If not, select the adaptive constraint strategy based on the action as the index to update the population and continue iteration. If so, use all non-dominated individuals in the output population as the optimized UAV flight path. When updating the main population, use the action as the index to select the constraint processing update method. The constraint processing categories include target value priority, boundary value priority, and constraint dominance principle.

2. The path planning method according to claim 1, characterized in that: The center point is the average value of the objective function of all individuals in the main population; The formula for calculating the standard deviation is: in, represents the center point, y represents all individual x i The corresponding target value, is y and The Euclidean distance of The calculation formula for the status flag determined based on the auxiliary population is: Among them, μ j is the rate of updating individuals in the auxiliary population P2, μ thr is a preset parameter, and J is a set consisting of every 10 consecutive generations.

3. The path planning method according to claim 1, wherein: The reinforcement learning network is trained using the following loss function: Where R represents the reward, λ represents the discount factor for future rewards, It is the value of the action a' that outputs the maximum value after the state St' is input into the target network Q2. St' represents the updated population state, Q2 represents the target network, Q1 represents the main network, and Q1(St, a) is the action value corresponding to action a after the state St is input into the main network Q1.

4. The path planning method according to claim 1, wherein: When the target value is prioritized, the offspring population of the main population is obtained and merged with the main population. The number of unconstrained dominance of each individual in the merged population is determined, and the fitness of all individuals in the merged population is calculated, and the individual with the largest fitness is removed. When the boundary value is prioritized, the offspring population of the main population is obtained and merged with the main population. The number of other individuals dominated by the boundary constraints of each individual in the merged population is determined, and the fitness of all individuals in the merged population is calculated, and the individual with the largest fitness is removed. Boundary constraint dominance is defined as: calculating the individual x according to the constraint function i The constraint violation value G i , for two individuals x1 and x2, if G1 and G2 are both less than or equal to the boundary value ε or G1 is equal to G2, and all objective function values ​​of x1 are not worse than x2, and on at least one objective function, the value of x1 is strictly better than x2, then the x1 boundary constraint dominates x2; if G2 is greater than the boundary value ε, and G1 is less than G2, then the x1 boundary constraint dominates x2; When the constraint dominance principle is selected, the offspring population of the main population is obtained and merged with the main population. The number of constraints and dominance of other individuals by each individual in the merged population is determined, and the fitness of all individuals in the merged population is calculated, and the individual with the largest fitness is removed. Constraint dominance is defined as: calculating the individual x according to the constraint function i The constraint violation value G i For two individuals x1 and x2, if G1 is equal to G2, and all objective function values ​​of x1 are not worse than x2, and on at least one objective function, the value of x1 is strictly better than x2, then the x1 constraint dominates x2; if G1 is less than G2, then the x1 constraint dominates x2.

5. The path planning method according to claim 4, characterized in that: The boundary value ε is determined as follows: Where ε0 is the maximum constraint violation value among all individuals in the initial population, t represents the current iteration number, and T max Indicates the maximum number of iterations.

6. The path planning method according to claim 4, characterized in that: The offspring population of the main population is obtained through the following steps: Merge the main population and the output population to obtain the secondary main population, and assign a neighbor set to each individual in the secondary main population; For each individual x i , select two individuals from the corresponding neighbor set, take the individual with the smallest unconstrained dominant fitness as the parent individual, traverse the neighbor set, and determine the final parent individual x r1 and x r2 ; For parent individual x r1 and x r2 Perform mutation to obtain individual u i ; For individual u i Each component u' i Restored to individual x with a probability of 1-CR i The corresponding component x' i ,get CR is the mutation rate.

7. The path planning method according to claim 6, characterized in that: The neighbor set is allocated according to the following formula: Where σ represents the maximum neighbor angle, y i The individual x i The target value of the target vector θ(y i ,y j ) is the vector y i and y i The angle formed, P4 represents the secondary main population.

8. The path planning method according to claim 4 or 6, characterized in that: The calculation steps of individual fitness include: Fit i =R i +D i Where, Fit i represents individual fitness, R i represents the individual’s original fitness, D i represents individual density, P represents the combined population, x j >x i Represents x j Dominate x i , S i Represents individual x i The number of other individuals in the combined population is controlled in the desired way, and τ represents x i Distance The Euclidean distance of individuals.

Images