Unmanned aerial vehicle path planning method fusing reinforcement learning and artificial bee colony algorithm

Abstract

The invention provides an unmanned aerial vehicle path planning method fusing reinforcement learning and an artificial bee colony algorithm. The method comprises the following steps: projecting a starting point and a destination point to a bottom plane of a flight space to obtain a connecting straight line between the starting point and the destination point on the bottom plane of the flight space; n points are randomly selected on the connecting line straight line to serve as route points of the flight path, the z coordinate value of each route point is compared with the height value of the terrain at the position of the route, and whether the route point collides with the obstacle or not is judged; calculating the distance between the collision point and the highest point of each obstacle one by one, and replacing the route point with the highest point of the obstacle closest to the collision route point; a new nectar source is searched through a multi-strategy double-population artificial bee colony algorithm flow of improved reinforcement learning, and a three-dimensional unmanned aerial vehicle flight path passing through a waypoint is generated based on nectar source position interpolation fitting. A better scheme is provided for efficient and safe path planning of the unmanned aerial vehicle under the complex terrain, and the method has important practical application value.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle path planning, and particularly relates to an unmanned aerial vehicle path planning method fusing reinforcement learning and artificial bee colony algorithm. BACKGROUND

[0002] The unmanned aerial vehicle path planning problem has been a research problem of great value and has been concerned by domestic and foreign experts and scholars. In the research on the unmanned aerial vehicle path planning problem, the commonly used path planning techniques are mainly based on classical path planning algorithms and biological intelligent optimization algorithms.

[0003] The central idea of the classical path planning Dijkstra algorithm is to take the path starting point as the center point, and constantly spread to the adjacent vertices until the end point is reached. The main process of the Dijkstra algorithm is to constantly create labels related to the vertices. The labels are divided into two types, temporary labels and permanent labels. The temporary labels are mainly assigned to vertices that have not been reached, while the permanent labels are assigned to vertices that have been reached. The algorithm ends until the end point is assigned a permanent label. Some scholars use the Voronoi diagram method to model the three-dimensional map environment, and take the center point of the Voronoi diagram as the obstacle point. Due to the related properties of the Voronoi diagram, the distance from any point in the polygon to the base point is the shortest. Therefore, the flight path of the unmanned aerial vehicle is defined on the edge of the polygon, and the Dijkstra algorithm is used to define the edge point as the waypoint for unmanned aerial vehicle flight path planning.

[0004] Based on the Dijkstra algorithm, some scholars have introduced heuristic strategies into the algorithm and proposed the A* algorithm. The A* algorithm is a heuristic search algorithm based on static road network. By adjusting the weight of the heuristic function, the search behavior of the algorithm is controlled, thereby greatly accelerating the search efficiency of the algorithm. Due to the traditional A* algorithm, when facing complex map or multi-dimensional path planning problems, the time complexity is high, and an effective solution may not be obtained.

[0005] Genetic algorithm is an intelligent optimization algorithm proposed by American scholars in the 1970s based on the principles of "genetic inheritance and variation" and "natural selection" in the biological evolution process in nature. Since its proposal, it has been widely applied in various fields. Eun et al. introduced a genetic algorithm for cooperative task allocation and path planning, which uses Voronoi diagram to avoid threats and obstacles, and generates an unmanned aerial vehicle flight path under time and path constraints.

[0006] The basic idea of the particle swarm algorithm is derived from the behavior of birds flocking, by abstracting the birds as particles without mass and shape, through the cooperation and information sharing between particles, the better solution is constantly searched in the solution space. Since the particle swarm algorithm is easy to implement, has fewer parameters, and does not require much information for optimization problems, it has been widely applied since its inception. Like other biological intelligent algorithms, the particle swarm algorithm also has the problems of premature convergence and easy to fall into local optimum, therefore, since the particle swarm algorithm was proposed, many scholars have made various effective improvements on the basis of the basic particle swarm algorithm. Foo et al. combined the particle swarm algorithm with the B-spline curve technique, generated multiple candidate paths by using the particle swarm algorithm, and described the paths by using the B-spline curve technique, so that the path planning is more flexible and efficient.

[0007] The artificial bee colony algorithm was first proposed by Karaboga in 2005, which is a bionic algorithm that simulates the behavior of bees searching for high-quality honey sources to solve practical engineering optimization problems. Compared with other algorithms, the artificial bee colony algorithm has the advantages of low complexity, strong adaptability, flexibility and efficiency when solving practical problems. The classical artificial bee colony algorithm has been well applied in many classical theories and practical problems in the field of optimization scheduling, such as numerical optimization, constrained multi-objective optimization, flow shop scheduling problem, distributed integrated process planning and scheduling problem, and path planning. However, the classical artificial bee colony algorithm also has the disadvantages of slow convergence speed and low utilization rate. Therefore, in recent years, the improvement of the artificial bee colony algorithm has attracted widespread attention.

[0008] The classical path planning algorithms in the above prior art, such as A* algorithm, Dijkstra algorithm, artificial potential field algorithm, etc., generally have high computational complexity and the problem of target unreachability, and it is difficult to efficiently cope with the demand for unmanned aerial vehicle path planning in complex terrain; and heuristic algorithms such as particle swarm algorithm, genetic algorithm, ant colony algorithm, etc. are prone to fall into local optimum, affecting the global optimality of the planning result. In addition, the standard artificial bee colony algorithm has the problems of low efficiency in the early stage of search, low search efficiency due to single search equation, slow convergence speed and easy to fall into local optimum, and the greedy strategy in the standard reinforcement learning may lead to non-convergence or local optimum due to excessive inclination to known high reward behavior, which limits their application effect in unmanned aerial vehicle path planning in complex environments such as mountainous areas. SUMMARY

[0009] The present application provides an unmanned aerial vehicle path planning method combining reinforcement learning and artificial bee colony algorithm to effectively plan paths for unmanned aerial vehicles in complex terrain.

[0010] In order to achieve the above purpose, the present application adopts the following technical solutions.

[0011] A method for unmanned aerial vehicle path planning by fusing reinforcement learning and artificial bee colony algorithm, comprising:

[0012] Step S1: connecting the starting point and the destination point of the unmanned aerial vehicle, and obtaining a spatial straight line analytical expression between the starting point and the destination point;

[0013] Step S2: projecting the starting point and the destination point to the bottom plane of the flight space to obtain a straight line between the starting point and the destination point on the bottom plane of the flight space;

[0014] Step S3: randomly taking n points on the straight line as the waypoints of the flight trajectory, and corresponding the positions of the honey sources to the coordinates of the waypoints in the improved artificial bee colony algorithm;

[0015] Step S4: according to the x and y coordinates of each randomly taken waypoint, obtaining the z coordinate of each waypoint by using the spatial straight line analytical expression;

[0016] Step S5: comparing the z coordinate value of each waypoint with the height value of the terrain at the waypoint to determine whether the waypoint collides with an obstacle;

[0017] Step S6: if collision occurs, performing step S7, otherwise directly performing step S8;

[0018] Step S7: calculating the distance between the collision point and the highest point of each obstacle one by one, and replacing the collision waypoint with the highest point of the obstacle closest to the collision waypoint;

[0019] Step S8: obtaining the initial waypoint coordinates as the initial honey source positions, embedding a reinforcement learning mechanism into the dynamic selection process of the honey source search strategy based on the initial waypoint coordinates, searching for new honey sources by using the improved multi-strategy double-population artificial bee colony algorithm of reinforcement learning, interpolating and fitting the smooth curve of the unmanned aerial vehicle flight path passing through the waypoints based on all the honey source positions, and taking the smooth curve as the specific three-dimensional unmanned aerial vehicle flight path.

[0020] Preferably, the method for obtaining the initial waypoint coordinates as the initial honey source positions, embedding a reinforcement learning mechanism into the dynamic selection process of the honey source search strategy based on the initial waypoint coordinates, and searching for new honey sources by using the improved multi-strategy double-population artificial bee colony algorithm of reinforcement learning comprises:

[0021] adopting multiple search strategies to guide the foraging bees to find new honey sources based on the initial honey source positions to obtain a search population, the multiple search strategies including an optimal individual oriented exploration equation, a local circular region search strategy search equation, and a spiral shape guided search strategy search equation,

[0022]

[0023] In the formula, represent a certain dimension of the solution, represent the current foraging bee, is a certain foraging bee number other than is the optimal individual in the current population, is a random number between [0, 1], representing the foraging bee No. the disturbance degree of the forage No. to the forage No. ; represents the position coordinates of the foraging bee No. in the population in the dimension No. ;

[0024] Based on the search population, a supplementary population is obtained by guiding the observation bee to search through a double population, the search population and the supplementary population are merged to obtain an updated supplementary population, and an intelligent agent action is selected based on the updated supplementary population through a greedy decision mechanism based on dynamic adjustment of an iteration cycle:

[0025]

[0026] The greedy selection coefficient ∈ (0, 1), the random number ∈ (0, 1), when is greater than , the action with the maximum Q value in state S is selected; when is less than , an action in state S is randomly selected;

[0027] The calculation of the greedy selection coefficient decreases from generation to generation as follows:

[0028]

[0029] In the formula, and represent the greedy coefficient at the beginning and at the end, represents the current iteration number, represents the number of attenuations, represents the step size of the attenuation;

[0030] A ternary probability decision model is constructed, the Q value is updated using the Bellman formula, the selection probability corresponding to each search strategy is calculated using the Q value, the dynamic adjustment of the selection probability of each search strategy is realized through the iterative update of the Q value function, and the agent selects the search strategy used next according to the new probability to form the final forage.

[0031] Preferably, the supplementary population obtained by guiding the observation bee to search through a double population based on the search population comprises:

[0032] Step 1: randomly generate N source positions as a supplementary population P2;

[0033] Step 2: combine the initial population P1 and the supplementary population P2 to obtain an initial supplementary population;

[0034] Step 3: calculate the fitness of each source position in the initial supplementary population;

[0035] Step 4: calculate the selection probability of each source by using the formula , wherein represents the selection probability of each source, indicates the fitness of the i-th source.

[0036] Step 5: perform a judgment for each source;

[0037] Step 5.1: generate a random number between 0 and 1, if the selection probability of the source is greater than the random number, then perform 5.2; otherwise, go to Step 5 to judge the next source;

[0038] Step 5.2: update the supplementary population, update the size of the supplementary population, and go to Step 5 to judge the next source;

[0039] Step 6: if the size of the supplementary population reaches the specified maximum population size, stop the judgment; otherwise, go to Step 5;

[0040] Step 7: if each source position in the initial supplementary population has completed the judgment, and the size of the supplementary population still does not reach the specified size, then add the optimal source in the supplementary population to the supplementary population to obtain a supplementary population;

[0041] Step 8: output the supplementary population.

[0042] As can be seen from the technical solutions provided by the above embodiments of the present application, the present application effectively solves the defects of the existing algorithm, and exhibits higher efficiency, better convergence and stronger global optimization ability in mountain unmanned aerial vehicle path planning, thereby providing a more optimal scheme for efficient and safe path planning of unmanned aerial vehicles in complex terrains, and having important practical application value.

[0043] Additional aspects and advantages of the application will be set forth in part in the description which follows, and will in part be apparent from the description, or can be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0044] ​In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description only some of the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0045] Figure 1 The processing flowchart of the application provided by the embodiment of the present application searches for a new honey source by improving the multi-strategy double-population artificial bee colony algorithm flow. DETAILED DESCRIPTION

[0046] The embodiments of the present application will be described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain the present application, and cannot be interpreted as a limitation on the present application.

[0047] Those skilled in the art can understand that, unless specifically stated, the singular forms "a", "an" and "the" used herein also include the plural forms. It should be further understood that the use of the phrase "comprises" in the specification of the present application means that the features, integers, steps, operations, elements and / or components exist, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we say an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be an intermediate element. In addition, "connected" or "coupled" used herein can include wireless connection or coupling. The phrase "and / or" used herein includes any one of the associated listed items and all combinations thereof.

[0048] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as that generally understood by those skilled in the art to which the present application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have meanings consistent with those in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as such.

[0049] In order to facilitate the understanding of the embodiments of the present application, the following will be further explained and described with several specific embodiments as examples in conjunction with the drawings, and each embodiment does not constitute a limitation on the embodiments of the present application.

[0050] The embodiment of the application proposes a multi-strategy double-population artificial bee colony algorithm based on improved reinforcement learning, generates initial honey source positions based on path heuristics to accelerate algorithm convergence, introduces three search equations and selects them in combination with improved reinforcement learning strategies to improve local exploration capability, adopts a double-population mechanism to guide onlooker bee search, increases population diversity to jump out of local optimum, and proposes a generation-by-generation recession greedy strategy to balance early exploration and late convergence.

[0051] The embodiment of the application proposes an initial population enhancement strategy, adopts a suboptimal path guided initialization mechanism to replace traditional random individual generation, considers that a suboptimal path may have terrain interference risks, designs a collision point dynamic adjustment scheme, and realizes obstacle avoidance optimization by replacing a conflict waypoint with the highest point of the nearest mountain. The improved mechanism ensures the safety of a three-dimensional space path while effectively accelerating the algorithm convergence process.

[0052] The embodiment of the application provides a step of generating an initial honey source position strategy based on path heuristics, including the following processing steps:

[0053] Step S1: connecting the starting point and the destination point of the unmanned aerial vehicle, and using a formula to obtain the spatial straight line analytical expression between the starting point and the destination point.

[0054] Step S2: projecting the starting point and the destination point to the bottom plane of the flight space, and connecting the starting point and the destination point.

[0055] Step S3: randomly taking n points on the straight line as the waypoints of the flight trajectory. In the improved artificial bee colony algorithm, the honey source position is one-to-one corresponding to the waypoint coordinates.

[0056] Step S4: according to the x and y coordinates of each randomly taken waypoint, the z coordinates of each waypoint are obtained by using the spatial straight line.

[0057] Step S5: comparing the z coordinate value of each waypoint with the height value of the terrain at the waypoint to determine whether the waypoint collides with an obstacle.

[0058] Step S6: if collision occurs, step S7 is executed, otherwise, step S8 is directly executed.

[0059] Step S7: the distance between the collision point and the highest point of each obstacle is calculated one by one, and the highest point of the obstacle closest to the collision waypoint is used to replace the waypoint.

[0060] Step S8: obtain initial waypoint coordinates as initial honey source positions, after the initial waypoint coordinates, build a multi-strategy search equation library, embed a reinforcement learning mechanism into the dynamic selection process of the honey source search strategy, enable the algorithm to autonomously decide to search for new honey sources, interpolate and fit a smooth curve of the UAV flight path passing through these waypoints based on all the honey source positions, and use the smooth curve as a specific three-dimensional UAV flight path.

[0061] The above step S8 specifically includes:

[0062] The processing flow of the multi-strategy double-population artificial bee colony algorithm for searching new honey sources by improving reinforcement learning provided by the embodiment of the present application is as shown in the figure. Figure 1 The processing flow of the multi-strategy double-population artificial bee colony algorithm for searching new honey sources by improving reinforcement learning provided by the embodiment of the present application is as shown in the figure.

[0063] The multi-strategy search equation library is innovatively constructed, and the reinforcement learning mechanism is embedded into the dynamic selection process of the search strategy, and the algorithm is enabled to autonomously make decisions to improve the local optimization precision and convergence adjustment efficiency.

[0064] Optimal individual-oriented exploration equation:

[0065] In swarm intelligence algorithms, global exploration and local mining are necessary measures. Global exploration refers to finding a global optimal solution in an unknown solution space, and local mining refers to exploring a local optimal solution in a known solution space. In order to improve the search effect of the algorithm, the two search capabilities should be balanced. In the traditional artificial bee colony algorithm, the generation of a new solution is affected by another randomly generated solution. Since the randomly generated solution has great uncertainty, it is easy to mislead the exploration direction. Therefore, in the present application, the following formula is introduced:

[0066]

[0067] In the formula, represents a certain dimension of the solution, represents the current foraging bee, is a certain foraging bee number other than . is the optimal individual in the current population. is a random number between 0 and 1, representing the disturbance degree of the jth honey source to the ith honey source. is a random number between 0 and 1, representing the disturbance degree of the jth honey source to the ith honey source. is a random number between 0 and 1, representing the disturbance degree of the jth honey source to the ith honey source. (taking 2) represents the influence of the global optimal individual on the generation of a new solution. If the value is too large, the new solution will easily fall into a local optimum, and if the value is too small, the influence of the global optimal individual on the generation of a new solution can be ignored.

[0068] Local circular area search strategy search equation:

[0069] The search process of the above-mentioned honey bee is carried out in a straight line direction, in order to improve the local search ability of the algorithm, a local circular region search strategy is introduced in the application:

[0070]

[0071] In the formula, represents the number of directions of division, represents the first direction, represents the first direction, represents the search step length. The honey bee searches in directions with the honey source position as the center, and the search step length as the radius. Then the search equation can be expressed as: the position of the first honey bee in the first direction in the space of the first dimension after moving forward a distance.

[0072] Spiral shape guiding search strategy search equation:

[0073] In the search process of the meta-heuristic algorithm, the spiral shape search process is widely concerned by various algorithms. By setting the correlation coefficient to control the amplitude and slope of the spiral shape, the direction and amplitude of the search can be better defined. For example, a large amplitude spiral can make the algorithm explore a wider area in the search space, and a small amplitude spiral can concentrate the search process in a small area. The embodiment of the application introduces two famous spiral models, the Archimedes spiral model and the logarithmic spiral model, to guide the search of the honey bee.

[0074] Archimedes spiral model: Archimedes curve, also known as "constant speed spiral", refers to a moving point moving along a straight line at a constant speed, while the straight line rotates around a point on the line at a constant angular velocity, that is, each turn of the spiral has a fixed distance. Its polar equation is:

[0075]

[0076] Logarithmic spiral model, also known as equiangular spiral, the distance between the arms of the logarithmic spiral increases in a geometric progression, and the intersection angle between any straight line and the logarithmic spiral is always equal. Its polar equation is:

[0077]

[0078] Wherein, represents the radius of the spiral, represents the polar angle, which can be any constant.

[0079] ​​​​​The polar equation of two spiral curves is converted into Cartesian coordinate equation, and then the spiral approximation search strategy equation can be expressed as:

[0080]

[0081] A random number of epsilon (-1, 1), A constant parameter, The current global optimal solution. A random number of epsilon (0, 1).

[0082] The population obtained by employing the bee stage search is used as the search population (P1).

[0083] In order to solve the problem that the classical artificial bee colony algorithm does not perform well in search convergence and has a slow convergence speed, the application adopts a double population mode to realize honey source search in the observation bee stage. The supplementary population is used to increase the diversity of the population, so that the algorithm can jump out of the local optimum in time. The search population and the supplementary population are combined to obtain an updated supplementary population, and the observation bee completes the local search of the honey source in the updated supplementary population.

[0084] The specific steps of the double population guided observation bee search method of the embodiment of the application are as follows:

[0085] Step 1: randomly generate n honey source positions as a supplementary population (P2).

[0086] Step 2: combine the initial population and the supplementary population (P1+P2) to obtain an initial supplementary population.

[0087] Step 3: calculate the fitness of each honey source position in the initial supplementary population.

[0088] Step 4: calculate the selection probability of each honey source by using the formula

[0089] Step 5: perform judgment for each honey source.

[0090] Step 5.1: generate a random number of 0-1, if the selection probability of the honey source is greater than the random number, then execute step 5.2; otherwise, go to step 5 to judge the next honey source.

[0091] Step 5.2: update the supplementary population, and go to step 5 to judge the next honey source.

[0092] Step 6: if the size of the supplementary population reaches the specified maximum population size, stop the judgment; otherwise, go to step 5.

[0093] ​Step 7: If the size of the supplementary population still does not reach the required size after all the nectar source locations in the initial supplementary population have been determined, then the best nectar source in the supplementary population will be added to the supplementary population.

[0094] Step 8: Output the supplementary population.

[0095] After obtaining a supplementary population, the bees were observed searching for a better solution near the nectar source using search equation one.

[0096]

[0097] The pseudocode for the dual-population guided observation bee search is shown in Table 1.

[0098] The process of generating a supplementary population using a supplementary population is as follows:

[0099] Algorithm table: Generating a supplementary population using the supplementary population

[0100] Input: Initial population nec_p, replenished population size Np, replenished population size Nk

[0101] Output: Supplement population Can_p

[0102] Initialize algorithm parameters

[0103] Set p_i=0, loc1=1, loc2=1,

[0104] rec_p.pos=[],rec_p.fitness=[],rec_p.path = [],rec_p.limitNum=1, Ini_p=[],Eli_p=[]

[0105] For i = 1: Np

[0106] rec_p.pos, rec_p.fitness ← Randomly generate n nectar source locations and calculate their fitness.

[0107] End For

[0108] Ini_p ← Merges the initial population and the supplementary population (rec_p, nec_p)

[0109] For i = 1: Ne + Np

[0110] Calculate the selection probability p_i for each honey source.

[0111] End For

[0112] For i = 1: Ne + Np

[0113] C ← Generates a random number between (0, 1).

[0114] If p_i>C

[0115] Can_p(loc1) = Ini_p(i)

[0116] If loc1 <Nk

[0117] loc1 = loc1 + 1

[0118] Else

[0119] Break

[0120] End If

[0121] Else

[0122] Eli_p(loc2)=Ini_p(i)

[0123] loc2 = loc2 + 1

[0124] End If

[0125] End For

[0126] The eliminated populations are ranked according to their fitness (Eli_p.fitness).

[0127] If loc1-1 < Nk

[0128] Can_p ← Selects well-fitting individuals from the eliminated population to add to the replenished population (Eli_p).

[0129] End If

[0130] Improved reinforcement learning guidance using bee selection search equation

[0131] Improved reinforcement learning for greedy strategies

[0132] In traditional reinforcement learning frameworks, this invention proposes a greedy decision-making mechanism based on dynamically adjusted iteration cycles. This strategy achieves an exploration-exploitation balance by introducing a decay factor: in the early stages of the algorithm, it enhances the ability to explore the unknown state space to discover potentially high-reward regions; in the later stages of iteration, it gradually focuses on known optimal actions to improve convergence efficiency. However, static greed coefficients can easily lead to path dependence on locally high-reward actions, limiting the search range for the global optimum. To address this drawback, this invention constructs an agent action selection formula that incorporates a dynamic decay factor:

[0133]

[0134] Greedy selection coefficient ∈ (0, 1), random number ∈ (0, 1). When Greater than When, select the action with the largest Q value in state S; when Less than At that time, randomly select an action from state S.

[0135] The greedy selection coefficient decays generation by generation. The calculation is as follows:

[0136]

[0137] In the formula, and This represents the greed coefficient at the beginning and end. This represents the current iteration number. Indicates the number of decays. This indicates the step size of the decay.

[0138] During the nectar extraction stage, this invention innovatively introduces a reinforcement learning mechanism incorporating a dynamic decay factor to guide the adaptive selection of multi-strategy search equations by the honeybee population. Considering the inherent random perturbation characteristics of each search strategy, the result of a single iteration is insufficient to judge the merits of a strategy—short-term optimization performance may mask its global search potential. Therefore, this invention constructs a ternary probabilistic decision model (p1 / p2 / p3), dynamically adjusting the selection probability of each search strategy through iterative updates of the Q-value function, ultimately forming an adaptive search equation decision mechanism based on environmental feedback.

[0139] Represents the state At that time, the intelligent agent takes action. The cumulative value of the reward received.

[0140] Represents the state At that time, the intelligent agent takes action. The cumulative value of the reward received.

[0141] This invention uses the Bellman formula to update the Q-value and uses the Q-value to calculate the selection probability corresponding to the three search strategies. The agent then selects the next search strategy based on the new probabilities. The probability calculation formula is as follows:

[0142] .

[0143] In summary, the multi-strategy dual-population artificial bee colony algorithm based on improved reinforcement learning proposed in this invention has significant advantages: It solves the problem of low initial search efficiency in the standard artificial bee colony algorithm by generating initial nectar source locations through path-based heuristics, and can quickly generate effective initial waypoints and accelerate convergence by combining actual terrain; it introduces three search equations and uses an improved reinforcement learning strategy (including a greedy coefficient with generational decay) for selection, overcoming the limitations of a single search equation, improving local exploration and global optimization capabilities, and avoiding getting trapped in local optima; the dual-population mechanism used in the observation bee stage increases population diversity, further helping the algorithm to escape local optima in a timely manner and accelerate convergence. These improvements enable the algorithm to outperform classic algorithms (such as A*, Dijkstra) and other heuristic algorithms (such as particle swarm optimization, standard artificial bee colony, etc.) in UAV path planning in complex environments such as mountainous areas in terms of planning accuracy, convergence speed, operating efficiency, and adaptability to complex terrain, and can more efficiently plan safe and optimal paths.

[0144] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0145] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0146] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for apparatus or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The apparatus and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0147] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for UAV path planning that integrates reinforcement learning and artificial bee colony algorithm, characterized in that, include: Step S1: Connect the starting point and the destination point of the UAV with a line and find the analytical expression of the spatial straight line between the starting point and the destination point; Step S2: Project the starting point and the destination point onto the bottom plane of the flight space to obtain a straight line connecting the starting point and the destination point on the bottom plane of the flight space; Step S3: Randomly select n points on the straight line connecting the points as waypoints for the flight trajectory. In the improved artificial bee colony algorithm, the location of the nectar source is matched one-to-one with the coordinates of the waypoints. Step S4: Based on the randomly selected x and y coordinates of each waypoint, use the spatial linear equation to calculate the z coordinate of each waypoint respectively; Step S5: Compare the z-coordinate value of each waypoint with the elevation value of the terrain at that waypoint to determine whether the waypoint has collided with an obstacle; Step S6: If a collision occurs, proceed to step S7; otherwise, proceed directly to step S8. Step S7: Calculate the distance between the collision point and the highest point of each obstacle, and replace the waypoint with the highest point of the obstacle closest to the collision waypoint; Step S8: Obtain the initial waypoint coordinates as the initial nectar source location. Based on the initial waypoint coordinates, embed the reinforcement learning mechanism into the dynamic selection process of the nectar source search strategy. Use the improved reinforcement learning multi-strategy dual-population artificial bee colony algorithm to search for new nectar sources. Based on the interpolation fitting of all nectar source locations, generate a smooth curve of the UAV flight path passing through these waypoints. Use the above smooth curve as the specific three-dimensional UAV flight path.

2. The method according to claim 1, characterized in that, The process of obtaining the initial waypoint coordinates as the initial nectar source location, embedding a reinforcement learning mechanism into the dynamic selection process of the nectar source search strategy based on the initial waypoint coordinates, and using an improved reinforcement learning multi-strategy dual-population artificial bee colony algorithm to search for new nectar sources includes: Based on the initial nectar source location, multiple search strategies are used to guide foraging bees to find new nectar sources, resulting in a search population. The multiple search strategies include: the optimal individual-oriented exploration equation, the local circular region search strategy equation, and the spiral-shaped guided search strategy equation. In the formula, Represents a certain dimension of the solution. Representing current beekeeping, Is it except A honeybee ID other than that of a foraging bee. It is the best individual in the current group. It is a random number between [0,1], representing the nth The first honey source for the second The degree of disturbance to each nectar source; Indicates the first in the population Only the bees collect honey on the first day Position coordinates in a dimension; A supplementary population is obtained by guiding observation bees to search the search population. The search population and the supplementary population are then merged to obtain an updated supplementary population. Based on this updated supplementary population, an agent action is selected using a greedy decision-making mechanism that is dynamically adjusted based on the iteration cycle. Greedy selection coefficient ∈ (0, 1), random number ∈ (0, 1), when Greater than When, select the action with the largest Q value in state S; when Less than At that time, randomly select an action from state S; The greedy selection coefficient decays generation by generation. The calculation is as follows: In the formula, and This represents the greed coefficient at the beginning and end. This represents the current iteration number. Indicates the number of decays. Indicates the step size of the decay; A ternary probabilistic decision-making model is constructed, and the Bellman formula is used to update the Q value. The selection probability corresponding to various search strategies is calculated using the Q value. The selection probability of various search strategies is dynamically adjusted through the iterative update of the Q value function. The agent selects the next search strategy to be used based on the new probability, thus forming the final honey source.

3. The method according to claim 2, characterized in that, The method of obtaining a supplementary population based on a search population through dual-population guided observation bee search includes: Step 1: Randomly generate N nectar source locations to supplement the population P2; Step 2: Merge the initial population P1 and the supplementary population P2 to obtain the initial supplementary population; Step 3: Calculate the fitness of each nectar source location in the initial supplementary population; Step 4: Using the formula Calculate the probability of selecting each honey source; where, This represents the probability of selecting each honey source. Indicates the first The adaptability of each nectar source; Step 5: Perform a judgment for each nectar source; Step 5.1: Generate a random number between 0 and 1. If the probability of selecting a honey source is greater than the random number, proceed to step 5.2; otherwise, go to step 5 to judge the next honey source. Step 5.2: Update and supplement the population. After updating the size of the supplementary population, proceed to Step 5 to identify the next nectar source. Step 6: If the size of the supplemented population reaches the maximum prescribed population size, stop the judgment; otherwise, proceed to step 5. Step 7: If the size of the supplementary population still does not reach the required size after each nectar source location in the initial supplementary population has been determined, then the best nectar source in the supplementary population is added to the supplementary population to obtain the supplementary population. Step 8: Output the supplementary population.

Images