Reinforcement learning method and system for three-dimensional path planning of unmanned aerial vehicle based on hybrid ant colony algorithm

Abstract

The application discloses a reinforcement learning unmanned aerial vehicle three-dimensional path planning method and system based on a hybrid ant colony algorithm, and belongs to the technical field of unmanned aerial vehicle autonomous flight control and intelligent navigation. The method comprises the following steps: S1, constructing a three-dimensional flight environment hybrid model; S2, extracting internal operation state and external environment complexity characteristics of an optimization group, and generating a comprehensive state vector; S3, inputting the comprehensive state vector into a double-delay deep deterministic policy gradient network, and outputting a control parameter action vector; S4, performing space node search and extreme value escape, and generating a candidate three-dimensional route; S5, evaluating the candidate three-dimensional route, updating double-delay deep deterministic policy gradient network parameters, judging whether a maximum iteration number or a convergence condition is reached, if yes, outputting a smoothed global optimal route, and if no, increasing the iteration number by 1 and returning to S2. The application improves system robustness in a complex dynamic environment, reduces invalid node exploration overhead, and improves obstacle avoidance safety.

Description

Technical Field

[0001] This invention belongs to the field of autonomous flight control and intelligent navigation technology for unmanned aerial vehicles (UAVs), specifically involving a reinforcement learning-based UAV 3D path planning method and system based on a hybrid ant colony algorithm. Background Technology

[0002] Currently, unmanned aerial vehicles (UAVs) play a crucial role in tasks such as complex terrain reconnaissance and near-field restricted inspection. Three-dimensional path planning is a core component ensuring autonomous flight of UAVs. Traditional swarm intelligence optimization algorithms are typically applied to such spatial optimization tasks. However, existing path planning technologies exhibit numerous systemic flaws when dealing with complex, non-convex three-dimensional flight environments: (1) Existing swarm intelligence algorithms typically employ fixed values ​​or preset simple linear decay strategies for their control parameters. When faced with complex three-dimensional spaces with drastic changes in obstacle density and terrain undulations, fixed parameter configurations cannot adapt to the real-time fluctuations in the algorithm's internal search state and external environmental characteristics. This leads to the algorithm's difficulty in converging quickly in the early stages of the search and its tendency to lose population diversity in the later stages of iteration, resulting in poor overall robustness and difficulty in maintaining stable optimization performance in variable flight mission scenarios.

[0003] (2) Existing heuristic path planning methods mainly rely on the passive penalty mechanism in the multi-objective cost function when dealing with obstacles. The algorithm usually filters and eliminates the tracks that cross obstacles only after generating a complete trial path. This passive evaluation method lacks the active avoidance guidance of the physical space repulsion field, which causes the algorithm to frequently generate invalid collision nodes during the space exploration process, which greatly consumes computing resources and reduces the efficiency of the system in finding the global optimum.

[0004] (3) Some schemes that introduce artificial potential fields to enhance obstacle avoidance capabilities generally face the technical bottleneck of local minima stagnation. When the repulsive force of obstacles and the attractive force of the target point in a complex environment reach a local equilibrium, the optimization individual is very likely to get stuck in a deadlock state. The existing algorithm architecture lacks a real-time detection mechanism for this local minima state and a long-distance escape strategy linked to it, which makes the planning system often fail to plan or generate extremely tortuous invalid routes when facing non-convex restricted spaces.

[0005] (4) Some technologies attempt to introduce machine learning to assist path optimization, but they are often limited by the discrete action space design, resulting in insufficient adjustment accuracy of control parameters, which makes it difficult to meet the needs of continuous dynamic three-dimensional space. Existing deep reinforcement learning-assisted architectures generally suffer from slow cold start in new environments and fail to achieve effective decoupling between macro-state perception and the underlying optimization engine, which limits the generalization ability of the overall algorithm and makes it impossible to ensure the flight safety and planning quality of UAVs when the environment changes suddenly. Summary of the Invention

[0006] To address the aforementioned shortcomings in existing technologies, the present invention provides a reinforcement learning-based UAV 3D path planning method and system using a hybrid ant colony algorithm, which solves the following technical problems: (1) Existing swarm intelligence algorithms have fixed control parameters or use simple linear decay, which makes them unable to adapt to changes in internal and external states in complex three-dimensional space, resulting in poor system optimization robustness.

[0007] (2) Heuristic planning methods rely too much on the ex-post passive punishment of multi-objective cost functions and lack the active avoidance guidance of spatial physical repulsion, resulting in ineffective system exploration and huge computational overhead.

[0008] (3) After the introduction of artificial potential field repulsion mechanism, it is very easy to cause local balance between attraction and repulsion, which leads to the optimization individual getting stuck in spatial deadlock and unable to plan a globally feasible route.

[0009] (4) The discrete action space of existing machine learning-assisted architectures leads to insufficient adjustment accuracy and extremely slow cold start when facing unknown environments.

[0010] To achieve the aforementioned objectives, the present invention employs the following technical solution: a reinforcement learning-based UAV 3D path planning method using a hybrid ant colony algorithm, comprising the following steps: S1. Construct a 3D flight environment hybrid model and initialize the number of iterations; S2. Based on the three-dimensional flight environment hybrid model and candidate three-dimensional routes, extract the internal operating state and external environment complexity features of the optimization group to generate a comprehensive state vector; S3. Input the comprehensive state vector into the dual-delay deep deterministic policy gradient network, and output the continuous swarm intelligence algorithm control parameter action vector. S4. Based on the control parameter action vector, a smart optimization engine that integrates artificial potential field repulsion force is used to search for spatial nodes and escape extreme values ​​to generate candidate three-dimensional routes. S5. Evaluate candidate 3D routes based on multi-objective composite cost function, update the parameters of dual-delay deep deterministic policy gradient network, and determine whether the maximum number of iterations or convergence condition has been reached. If yes, output the smoothed global optimal route; otherwise, increment the iteration count by 1 and return to S2.

[0011] The beneficial effects of this invention are as follows: (1) Improve the robustness of the system under complex dynamic environments. By relying on deep reinforcement learning networks to perform high-precision continuous closed-loop intervention on the action vector of the underlying optimization control parameters, the planning system can accurately adapt to internal and external feature fluctuations, effectively avoid premature convergence or divergence caused by fixed parameters, and enable the algorithm to continuously stagnate algebraically. On average, it decreased by more than 40%.

[0012] (2) Reduce the overhead of exploring invalid nodes and improve obstacle avoidance safety. Physical repulsion potential energy is directly integrated into the underlying heuristic selection process, giving the optimizing individual the ability to actively avoid high-risk areas. This fundamentally avoids the computational waste of generating complete routes containing collisions and then discarding them. Experiments show that this mechanism reduces the number of invalid paths generated. Reduce collision probability by 65% The value is reduced to 0, which significantly improves the efficiency of optimization calculation.

[0013] (3) Solving the bottleneck of local extreme value stagnation under complex non-convex terrain. With the construction of potential energy state detection and Levy long jump linkage escape mechanism, the system can quickly identify spatial deadlock caused by the balance of gravity and repulsion and force individuals to escape local obstacle traps, and the success rate of local extreme value escape is high. This ensures a high success rate in generating globally feasible flight paths within confined spaces.

[0014] (4) Accelerate the online deployment convergence process. Leveraging an offline pre-training architecture covering typical terrains and an online network fine-tuning architecture, the system can quickly learn the parameter mapping patterns of the currently unknown spatial domain within a very short number of iterations. The initial cost after pre-training... 30% reduction compared to random initialization, convergence iterations It reduces learning speed by 25%-35%, significantly overcoming the engineering application limitations of traditional unsupervised learning schemes, such as slow cold start.

[0015] Furthermore: In S2, the three-dimensional flight environment hybrid model is specifically as follows: Construct a three-dimensional rectangular coordinate system Static obstacles are represented by cylinders. The obstacle model includes an inner spatial boundary and an outer spatial boundary, with the inner spatial boundary corresponding to the collision radius. This represents an absolute physical obstacle avoidance space that is inaccessible to aircraft; the outer space boundary corresponds to the threat radius. This indicates the effective area of ​​action for triggering the artificial potential field repulsion force and the multi-target threat cost penalty. If the horizontal geometric distance from the current exploration node of the aircraft to the central axis of the obstacle is within... When the range is reached, initiate the calculation of repulsive potential energy; In S2, the method for generating the comprehensive state vector is as follows: The relative cost change rate of the optimization group, global convergence progress, number of consecutive stagnation algebras, mean Hamming distance of the population path, pheromone distribution entropy, environmental obstacle density and terrain undulation are min-max normalized to obtain the comprehensive state vector.

[0016] Furthermore, in S3, the dual-delay deep deterministic policy gradient network includes an Actor policy network and a Critic value network, which are responsible for the high-dimensional mapping of state features and the evaluation of action value. The dual-delay deep deterministic policy gradient network forward propagates the comprehensive state vector and outputs the continuous swarm intelligence algorithm control parameter action vector, including pheromone guiding weights, heuristic function weights, pheromone evaporation factors, pheromone enhancement coefficients, artificial potential field repulsion coefficients, and walk mode switching thresholds.

[0017] Furthermore, S4 includes the following sub-steps: S41. Initialize the spatial coordinates of each individual at the current spatial node, and each individual corresponds to a three-dimensional path sequence; S42. Calculate the comprehensive heuristic information based on the pheromone guidance weight and the heuristic function weight. Calculate the node transition probability of each individual based on the transition weight and the comprehensive heuristic information. S43. Determine whether the total potential energy change of an individual over three consecutive steps is less than a preset threshold. If yes, then trigger the Lévy distribution long-distance jump, generate the long-distance jump step size, verify the validity of the spatial coordinates of the new spatial node, and update the individual's current spatial node as the new spatial node; if no, then select the next candidate spatial node according to the individual's node transition probability, and update the individual's current spatial node as the next candidate spatial node. S44. After all individuals have completed the iteration, several candidate 3D routes are generated. The global optimal path and the worst path are extracted, and elite pheromone preservation update is performed.

[0018] Further: In S42, the calculation of the first... Synthetic Heuristic Information The specific expression is: In the formula, For distance heuristics, For artificial potential field heuristics; In the formula, For the current spatial node i To any spatial node in the three-dimensional path sequence j The three-dimensional Euclidean distance, where the first... k Individual corresponding three-dimensional path sequence , The coordinates of the first waypoint The coordinates of the second waypoint For the first Waypoint coordinates , For the first The x-coordinate of each waypoint For the first The ordinate of each waypoint For the first Vertical coordinates of each waypoint; In the formula, For spatial nodes j The total potential energy formed by the superposition of the gravitational force at the target point and the repulsive force of the obstacle; Definition of the first k Individual from the current spatial node i To space node j transfer weights : In the formula, For the first Pheromones guide weights, For the first Heuristic function weights, For the first Current spatial node i To space node j pheromone concentration; Calculate the first k Individual node transition probability The specific expression is: In the formula, For the current spatial node i The sum of the transition weights of all accessible nodes.

[0019] Furthermore: In S43, a long-distance jump in the Lévy distribution is triggered, generating a long-distance jump step size. The specific expression is: In the formula, Let be a random variable that follows a standard normal distribution. It is a random variable that follows a specific normal distribution.

[0020] Furthermore: In S44, elite-preserving pheromone updates are performed to generate the next generation of pheromone concentrations, and the pheromone concentration is calculated. Current spatial node i To space node j pheromone concentration The specific expression is: In the formula, Additional volatility coefficient for the worst-case path. For the first Pheromones volatile factors, For the first Pheromones enhancement coefficient The cost of the current group's globally optimal path. The cost of the worst path for the current group.

[0021] Furthermore, S5 includes the following sub-steps: S51. Calculate the total path cost based on the multi-objective composite cost function. The specific expression is: In the formula, For the cost of flight distance, As a threat of cost, The cost of high constraints, For path smoothness cost, The weighting coefficient for the flight distance cost. The weighting coefficient for the threat cost. The weighting coefficients for the cost of high constraints The weighting coefficients for path smoothness cost; S52. Calculate the composite reward value based on the decrease in the total path cost, the safety margin, and the number of stalls. The specific expression is: In the formula, The immediate reward item corresponding to the decrease in value. To meet the reward requirements corresponding to the spatial safety distance margin, A speed reward term to encourage rapid convergence of the algorithm. The penalty term for continuous stagnation in the optimization process. To test the path, there are collision penalties for touching obstacles. S53. Based on the composite reward value, perform gradient backpropagation and soft update on the gradient network of the dual-delay deep deterministic policy to optimize the network parameters. S54. Determine whether the maximum number of iterations or the convergence condition has been reached. If so, output the global optimal path and use cubic B-spline interpolation to smooth it, generating the smoothed global optimal route. In the formula, To smooth out the independent parameter variables The generated 3D flight path coordinates point For cubic B-spline basis functions, These are control points used for curve interpolation. The index number of the control point. For the independent parameter variables of the spline curve, This is the starting index value of the control point in the global control point sequence; If not, increment the iteration count by 1 and return to S2.

[0022] A reinforcement learning-based UAV 3D path planning system using a hybrid ant colony algorithm, comprising a decision layer, an execution layer, and a perception layer; The decision layer is equipped with a dual-delay deep deterministic policy gradient network, which receives the comprehensive state vector and outputs the control parameter action vector of the swarm intelligence algorithm. The execution layer includes an enhanced node transfer module, a minimum value linkage escape module, and an elite pheromone preservation update module. The enhanced node transfer module performs active obstacle avoidance selection, the minimum value linkage escape module performs Levy distribution long-distance jump anti-deadlock, and the elite pheromone preservation update module performs survival of the fittest to generate candidate three-dimensional routes. The perception layer includes a 3D environment hybrid modeling module, a multi-objective cost evaluation module, and a real-time state feature extraction module. The 3D environment hybrid modeling module and the multi-objective cost evaluation module calculate the losses of distance, threat, altitude, and smoothness to quantify and score candidate 3D routes. The real-time state feature extraction module extracts the convergence, diversity, and environmental complexity features of the optimization group and performs normalization processing to generate a comprehensive state vector and a composite reward value.

[0023] The beneficial effects of this invention are as follows: (1) This invention constructs a parameter adaptive mapping mechanism with a seven-dimensional state vector as input and a six-dimensional action vector as output. A dual-delay deep deterministic strategy gradient network is used to replace the traditional fixed or linear decay configuration of parameters, realizing high-precision continuous closed-loop intervention on the core optimization variables of the swarm intelligence algorithm, and solving the inherent defect of poor robustness of the algorithm in complex three-dimensional space.

[0024] (2) An artificial potential field repulsion force calculation model is directly embedded in the bottom-level intelligent optimization node selection stage. This changes the passive mode of existing heuristic algorithms that rely solely on multi-objective evaluation functions for post-collision penalties, and empowers the optimization individual to actively avoid high-risk areas during the exploration stage, significantly reducing the computational resource consumption of invalid search nodes.

[0025] (3) A potential energy change monitoring model was constructed to accurately identify the local deadlock state caused by the mutual cancellation of attraction and repulsion. The system uses the Lévy power law exponential distribution to perform forced long-distance spatial jumps, which solves the technical bottleneck of traditional artificial potential field methods that are prone to getting trapped in local minima, and effectively ensures the generation rate of globally feasible routes in dense obstacle spaces.

[0026] (4) A multi-dimensional evaluation model including incentives for decreasing cost, rewards for safe distance, and penalties for stagnation was established to guide the updating of reinforcement learning network parameters. At the same time, a transfer learning framework of offline pre-training in typical scenarios and online freezing of the front network layers and fine-tuning of the final output layer was established, which significantly overcomes the engineering application limitations of machine learning schemes that have extremely slow cold starts when facing unknown spatial domains. Attached Figure Description

[0027] Figure 1 This is a flowchart of the reinforcement learning UAV 3D path planning method based on the hybrid ant colony algorithm of the present invention.

[0028] Figure 2 This is a schematic diagram of a three-dimensional flight environment hybrid model.

[0029] Figure 3 This is a schematic diagram of the reinforcement learning UAV 3D path planning system based on the hybrid ant colony algorithm of the present invention.

[0030] Figure 4 This is the path planning result of the algorithm of this invention in case one.

[0031] Figure 5 This is a diagram showing the path planning result of the algorithm of this invention in case two.

[0032] Figure 6 This is a diagram showing the path planning result of the algorithm of this invention in case three.

[0033] Figure 7 This is a diagram showing the path planning result of the algorithm of this invention in case four.

[0034] Figure 8 This is a test graph showing the total cost of the algorithm of this invention in Case 1.

[0035] Figure 9 This is a test graph showing the total cost of the algorithm of this invention in case two.

[0036] Figure 10 This is a test graph showing the total cost of the algorithm of this invention in case three.

[0037] Figure 11 This is a test graph showing the total cost of the algorithm of this invention in case four. Detailed Implementation

[0038] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0039] like Figure 1 As shown, in one embodiment of the present invention, the reinforcement learning UAV 3D path planning method based on hybrid ant colony algorithm includes the following steps: S1. Construct a 3D flight environment hybrid model and initialize the number of iterations; S2. Based on the three-dimensional flight environment hybrid model and candidate three-dimensional routes, extract the internal operating state and external environment complexity features of the optimization group to generate a comprehensive state vector; S3. Input the comprehensive state vector into the dual-delay deep deterministic policy gradient network (TD3) and output the control parameter action vector of the swarm intelligence algorithm. S4. Based on the control parameter action vector, a smart optimization engine that integrates artificial potential field repulsion force is used to search for spatial nodes and escape extreme values ​​to generate candidate three-dimensional routes. S5. Evaluate candidate 3D routes based on multi-objective composite cost function, update the parameters of dual-delay deep deterministic policy gradient network, and determine whether the maximum number of iterations or convergence condition has been reached. If yes, output the smoothed global optimal route; otherwise, increment the iteration count by 1 and return to S2.

[0040] like Figure 2 As shown, in S2, the three-dimensional flight environment hybrid model is specifically as follows: Construct a three-dimensional rectangular coordinate system Static obstacles are represented by cylinders. The obstacle model includes an inner spatial boundary and an outer spatial boundary, with the inner spatial boundary corresponding to the collision radius. This represents an absolute physical obstacle avoidance space that is inaccessible to aircraft; the outer space boundary corresponds to the threat radius. This indicates the effective area of ​​action that triggers the artificial potential field repulsion force and the multi-target threat cost penalty, if the spacecraft is currently exploring a node. The horizontal geometric distance to the center axis of the obstacle is When the range is reached, the repulsive potential energy calculation is initiated, where, for x-axis coordinates for The vertical axis coordinate, for The vertical axis coordinates, This is the current spatial node; In S2, the method for generating the comprehensive state vector is as follows: The relative cost change rate of the optimization group Global convergence progress Continuous stationary algebra Mean Hamming distance of population path Pheromones distribution entropy Environmental obstacle density and terrain relief Perform min-max normalization to obtain the comprehensive state vector; Among them, the Synthetic state vector The specific expression is: In the formula, Index for iteration count, It is the transpose symbol. For the first The rate of change of relative value of generations For the first The global convergence progress. For the first Subsequent continuous stationary algebras, For the first Mean Hamming distance of the population path For the first Pheromones distribution entropy For the first Density of environmental obstacles For the first Depth of topographic relief.

[0041] In S3, the dual-delay deep deterministic policy gradient network includes an Actor policy network and a Critic value network, responsible for high-dimensional mapping of state features and evaluation of action value. The dual-delay deep deterministic policy gradient network forward propagates the comprehensive state vector and outputs a continuous swarm intelligence algorithm control parameter action vector, including pheromone-guided weights. Heuristic function weights pheromone volatile factors Pheromones enhancement coefficient Artificial potential field repulsion coefficient Threshold for switching between roaming modes .

[0042] In this embodiment, the first Replace control parameters action vector The specific expression is: In the formula, For the first Pheromones guide weights, For the first Heuristic function weights, For the first Pheromones volatile factors, For the first Pheromones enhancement coefficient For the first The repulsive coefficient of the artificial potential field, For the first Threshold for switching between proxy roaming modes; This invention proposes a deep reinforcement learning adaptive parameter tuning mechanism based on a continuous state-action space. This mechanism takes a comprehensive state vector as input and outputs a control parameter action vector, replacing the traditional fixed parameter method compared to existing methods. Alternatively, a linear decay parameter configuration can be used to achieve real-time closed-loop adjustment of the core parameters of the ant colony algorithm. As the initial pheromone guiding weight, For the initial heuristic function weights, This is the initial pheromone volatile factor.

[0043] S4 includes the following steps: S41. Initialize the spatial coordinates of each individual at the current spatial node, and each individual corresponds to a three-dimensional path sequence; S42. Calculate the comprehensive heuristic information based on the pheromone guidance weight and the heuristic function weight. Calculate the node transition probability of each individual based on the transition weight and the comprehensive heuristic information. S43. Determine whether the total potential energy change of an individual over three consecutive steps is less than a preset threshold. ; If yes, then trigger the Lévy distribution long-distance jump, generate the long-distance jump step size, verify the validity of the spatial coordinates of the new spatial node, and update the individual's current spatial node as the new spatial node; if no, then select the next candidate spatial node according to the individual's node transition probability, and update the individual's current spatial node as the next candidate spatial node. S44. After all individuals have completed the iteration, several candidate 3D routes are generated, and the globally optimal path is extracted. With the worst path The elites retain pheromone updates.

[0044] In this embodiment, based on the control parameter action vector, the optimizing individual first calculates comprehensive heuristic information, which is obtained by multiplying the spatial geometric distance heuristic term and the artificial potential field heuristic term. After selecting the next spatial node based on the transition probability, the system detects the total potential energy change of the individual over multiple consecutive steps. If the total potential energy change is continuously lower than a set threshold, the exploring individual is determined to be trapped in a local minimum deadlock state. At this time, a long-distance random jump mechanism based on the Lévy distribution is triggered to change the spatial coordinates. After a single round of group search is completed, the system evaluates the optimal and worst paths based on the composite cost and updates the elite retention pheromone.

[0045] In S42, calculate the... Synthetic Heuristic Information The specific expression is: In the formula, For distance heuristics, This invention uses an artificial potential field as a heuristic term; in this embodiment, the invention deeply integrates the active repulsive force of the artificial potential field into the heuristic exploration function, replacing the traditional method that only considers the total path cost. A mechanism for post-collision passive penalty. This invention incorporates artificial potential field heuristics. The first step of directly embedding the necessary technical features defined Synthetic Heuristic Information In this way, the individual seeking the best has the ability to proactively avoid high-risk areas during the node selection stage.

[0046] In the formula, For the current spatial node i To any spatial node in the three-dimensional path sequence j The three-dimensional Euclidean distance, where the first... k Individual corresponding three-dimensional path sequence , The coordinates of the first waypoint The coordinates of the second waypoint For the first Waypoint coordinates , For the first The x-coordinate of each waypoint For the first The ordinate of each waypoint For the first Vertical coordinates of each waypoint; In the formula, For spatial nodes j The total potential energy formed by the superposition of the attraction at the target point and the repulsion of the obstacle; in this embodiment, the calculation range of the artificial potential field heuristic term is set with an effective repulsion safety radius. Only when exploring the individual and the first m Distance between the centers of the obstacles When the radius is smaller than the effective repulsion safety radius, the repulsion potential energy is introduced into the calculation, that is: , For the first An obstacle at a spatial node j The repulsive potential energy component contributed at the point.

[0047] Definition of the first k Individual from the current spatial node i To space node j transfer weights : In the formula, For the first Pheromones guide weights, For the first Heuristic function weights, For the first Current spatial node i To space node j pheromone concentration; Calculate the first k Individual node transition probability The specific expression is: In the formula, For the current spatial node i The sum of the transition weights of all accessible nodes.

[0048] In S43, the Lévy distribution long-distance jump is triggered, generating the long-distance jump step size. The specific expression is: In the formula, Let be a random variable that follows a standard normal distribution. It is a random variable that follows a specific normal distribution.

[0049] This invention constructs a dynamic detection and long-step jump linkage escape mechanism for local equilibrium states in potential fields, solving the inherent problem of deadlock in traditional potential field methods. This mechanism detects continuous... K Total potential energy change If a local deadlock is detected, a long-distance jump in the Lévy distribution is triggered, forcing the individual to leave the local extreme region.

[0050] In S44, the elite-preserving pheromone update is performed to generate the next generation of pheromone concentrations, and the pheromone concentration of the next generation is calculated. Current spatial node i To space node j pheromone concentration The specific expression is: In the formula, Additional volatility coefficient for the worst-case path. For the first Pheromones volatile factors, For the first Pheromones enhancement coefficient The cost of the current group's globally optimal path. The cost of the worst path for the current group.

[0051] S5 includes the following steps: S51. Calculate the total path cost based on the multi-objective composite cost function. The specific expression is: In the formula, For the cost of flight distance, As a threat of cost, The cost of high constraints, For path smoothness cost, The weighting coefficient for the flight distance cost. The weighting coefficient for the threat cost. The weighting coefficients for the cost of high constraints The weighting coefficients for path smoothness cost; S52. Calculate the composite reward value based on the decrease in the total path cost, the safety margin, and the number of stalls. The specific expression is: In the formula, The immediate reward item corresponding to the decrease in value. To meet the reward requirements corresponding to the spatial safety distance margin, A speed reward term to encourage rapid convergence of the algorithm. The penalty term for continuous stagnation in the optimization process. To test the path, there are collision penalties for touching obstacles. S53. Based on the composite reward value, perform gradient backpropagation and soft update on the gradient network of the dual-delay deep deterministic policy to optimize the network parameters. S54. Determine whether the maximum number of iterations or the convergence condition has been reached. If so, output the global optimal path and use cubic B-spline interpolation to smooth it, generating the smoothed global optimal route. In the formula, To smooth out the independent parameter variables The generated 3D flight path coordinates point For cubic B-spline basis functions, These are control points used for curve interpolation. The index number of the control point. For the independent parameter variables of the spline curve, This is the starting index value of the control point in the global control point sequence; If not, increment the iteration count by 1 and return to S2.

[0052] like Figure 3 As shown, the reinforcement learning UAV 3D path planning system based on hybrid ant colony algorithm includes a decision layer, an execution layer, and a perception layer. The layers form a complete closed-loop control link through data interaction. The decision layer is equipped with a dual-delay deep deterministic policy gradient network, which receives the comprehensive state vector and outputs the control parameter action vector of the swarm intelligence algorithm. The execution layer, as an artificial potential field-enhanced ant colony optimization engine, includes an enhanced node transfer module, a minimum value linkage escape module, and an elite pheromone preservation update module. It performs active obstacle avoidance selection based on the enhanced node transfer module, performs Levy distribution long-distance jump anti-deadlock through the minimum value linkage escape module, and performs survival of the fittest through the elite pheromone preservation update module, thus completing the search for nodes in three-dimensional space and generating candidate three-dimensional routes. The perception layer comprises a 3D environment hybrid modeling module, a multi-objective cost evaluation module, and a real-time state feature extraction module. The 3D environment hybrid modeling module and the multi-objective cost evaluation module calculate the losses of distance, threat, altitude, and smoothness, quantifying and scoring candidate 3D routes. The real-time state feature extraction module extracts and normalizes features of the optimization group's convergence, diversity, and environmental complexity, generating a comprehensive state vector and a composite reward value reflecting cost reduction and safety margin. The comprehensive state vector and composite reward value are transmitted back to the decision layer along the feedback link, thus achieving a closed-loop system encompassing environmental state perception, continuous parameter decision-making, underlying proactive optimization, and cost feedback.

[0053] To verify the beneficial effects of the present invention, the following experimental data are provided in this embodiment: Figures 4 to 7 This diagram illustrates the comparative results of trajectory planning for three-dimensional flight missions in the same scenario, under different obstacle information and different endpoint positions, in scenarios one through four. The diagram shows the globally optimal flight paths generated by the TD3-ACO-APF algorithm of this invention and various comparative algorithms (HLOA, CCO-AVOA, EEFO, and ACOPAR) within a three-dimensional confined space containing static obstacles. Different line types distinguish the planning results of each algorithm, intuitively demonstrating the safety margin advantage and flight path spatial smoothness advantage of the algorithm of this invention when traversing dense obstacle gaps after incorporating an artificial potential field mechanism.

[0054] Figures 8 to 11 This is a diagram comparing the total cost convergence curves of various algorithms. Figures 4 to 7This corresponds to the scenario setting. The horizontal axis of the two-dimensional coordinate system represents the number of optimization iterations, and the vertical axis represents the total cost comprised of flight distance, threat penalty, altitude constraint, and smoothness loss. Multiple broken lines in the figure correspond to the decreasing trend of optimization costs for different planning algorithms, reflecting the technical performance of the algorithm of this invention under the guidance of continuous adaptive parameter tuning in deep reinforcement learning, in terms of early convergence speed, ability to escape local extreme plateaus, and final global optimal optimization accuracy.

[0055] In the description of this invention, the above are merely preferred embodiments and are not intended to limit the scope of protection of this invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A reinforcement learning-based 3D path planning method for unmanned aerial vehicles using a hybrid ant colony algorithm, characterized in that, Includes the following steps: S1. Construct a 3D flight environment hybrid model and initialize the number of iterations; S2. Based on the three-dimensional flight environment hybrid model and candidate three-dimensional routes, extract the internal operating state and external environment complexity features of the optimization group to generate a comprehensive state vector; S3. Input the comprehensive state vector into the dual-delay deep deterministic policy gradient network, and output the control parameter action vector of the swarm intelligence algorithm. S4. Based on the control parameter action vector, a smart optimization engine that integrates artificial potential field repulsion force is used to search for spatial nodes and escape extreme values ​​to generate candidate three-dimensional routes. S5. Evaluate candidate 3D routes based on multi-objective composite cost function, update the parameters of dual-delay deep deterministic policy gradient network, and determine whether the maximum number of iterations or convergence condition has been reached. If yes, output the smoothed global optimal route; otherwise, increment the iteration count by 1 and return to S2.

2. The reinforcement learning UAV 3D path planning method based on the hybrid ant colony algorithm according to claim 1, characterized in that, In S2, the three-dimensional flight environment hybrid model is specifically as follows: Construct a three-dimensional rectangular coordinate system Static obstacles are represented by cylinders. The obstacle model includes an inner spatial boundary and an outer spatial boundary, with the inner spatial boundary corresponding to the collision radius. This represents an absolute physical obstacle avoidance space that is inaccessible to aircraft; the outer space boundary corresponds to the threat radius. This indicates the effective area of ​​action for triggering the artificial potential field repulsion force and the multi-target threat cost penalty. If the horizontal geometric distance from the current exploration node of the aircraft to the central axis of the obstacle is within... When the range is reached, initiate the calculation of repulsive potential energy; In S2, the method for generating the comprehensive state vector is as follows: The relative cost change rate of the optimization group, global convergence progress, number of consecutive stagnation algebras, mean Hamming distance of the population path, pheromone distribution entropy, environmental obstacle density and terrain undulation are min-max normalized to obtain the comprehensive state vector.

3. The reinforcement learning UAV 3D path planning method based on the hybrid ant colony algorithm according to claim 1, characterized in that, In S3, the dual-delay deep deterministic policy gradient network includes an Actor policy network and a Critic value network, which are responsible for high-dimensional mapping of state features and evaluation of action value. The dual-delay deep deterministic policy gradient network forward propagates the comprehensive state vector and outputs the continuous swarm intelligence algorithm control parameter action vector, including pheromone guiding weights, heuristic function weights, pheromone evaporation factors, pheromone enhancement coefficients, artificial potential field repulsion coefficients, and walk mode switching thresholds.

4. The reinforcement learning UAV 3D path planning method based on the hybrid ant colony algorithm according to claim 2, characterized in that, S4 includes the following steps: S41. Initialize the spatial coordinates of each individual at the current spatial node, and each individual corresponds to a three-dimensional path sequence; S42. Calculate the comprehensive heuristic information based on the pheromone guidance weight and the heuristic function weight. Calculate the node transition probability of each individual based on the transition weight and the comprehensive heuristic information. S43. Determine whether the total potential energy change of an individual over three consecutive steps is less than a preset threshold. If yes, then trigger the Lévy distribution long-distance jump, generate the long-distance jump step size, verify the validity of the spatial coordinates of the new spatial node, and update the individual's current spatial node as the new spatial node; if no, then select the next candidate spatial node according to the individual's node transition probability, and update the individual's current spatial node as the next candidate spatial node. S44. After all individuals have completed the iteration, several candidate 3D routes are generated. The global optimal path and the worst path are extracted, and elite pheromone preservation update is performed.

5. The reinforcement learning UAV 3D path planning method based on the hybrid ant colony algorithm according to claim 4, characterized in that, In S42, calculate the... Synthetic Heuristic Information The specific expression is: In the formula, For distance heuristics, For artificial potential field heuristics; In the formula, For the current spatial node i to any spatial node in the three-dimensional path sequence j The three-dimensional Euclidean distance, where the first... k Individual corresponding three-dimensional path sequence , The coordinates of the first waypoint The coordinates of the second waypoint For the first Waypoint coordinates , For the first The x-coordinate of each waypoint For the first The ordinate of each waypoint For the first Vertical coordinates of each waypoint; In the formula, For spatial nodes j The total potential energy formed by the superposition of the gravitational force at the target point and the repulsive force of the obstacle; Definition of the first k Individual from the current spatial node i To space node j Transfer weights : In the formula, For the first Pheromones guide weights, For the first Heuristic function weights, For the first Current spatial node i To space node j pheromone concentration; Calculate the first k Individual node transition probability The specific expression is: In the formula, For the current spatial node i The sum of the transition weights of all accessible nodes. , For the first k Individual from the current spatial node i To space node s The transfer weights, For the first k The set of legal nodes that an individual can access next.

6. The reinforcement learning UAV 3D path planning method based on the hybrid ant colony algorithm according to claim 4, characterized in that, In S43, the Lévy distribution long-distance jump is triggered, generating the long-distance jump step size. The specific expression is: In the formula, Let be a random variable that follows a standard normal distribution. It is a random variable that follows a specific normal distribution.

7. The reinforcement learning UAV 3D path planning method based on the hybrid ant colony algorithm according to claim 5, characterized in that, In S44, the elite-preserving pheromone update is performed to generate the next generation of pheromone concentrations, and the pheromone concentration of the next generation is calculated. Current spatial node i To space node j pheromone concentration The specific expression is: In the formula, Additional volatility coefficient for the worst-case path. For the first Pheromones volatile factors, For the first Pheromones enhancement coefficient The cost of the current group's globally optimal path. The cost of the worst path for the current group.

8. The reinforcement learning UAV 3D path planning method based on the hybrid ant colony algorithm according to claim 7, characterized in that, S5 includes the following steps: S51. Calculate the total path cost based on the multi-objective composite cost function. The specific expression is: In the formula, For the cost of flight distance, As a threat of cost, The cost of high constraints, For path smoothness cost, The weighting coefficient for the flight distance cost. The weighting coefficient for the threat cost. The weighting coefficients for the cost of high constraints The weighting coefficients for path smoothness cost; S52. Calculate the composite reward value based on the decrease in the total path cost, the safety margin, and the number of stalls. The specific expression is: In the formula, The immediate reward item corresponding to the decrease in value. To meet the reward requirements corresponding to the spatial safety distance margin, A speed reward term to encourage rapid convergence of the algorithm. The penalty term for continuous stagnation in the optimization process. To test the path, there are collision penalties for touching obstacles. S53. Based on the composite reward value, perform gradient backpropagation and soft update on the gradient network of the dual-delay deep deterministic policy to optimize the network parameters. S54. Determine whether the maximum number of iterations or the convergence condition has been reached. If so, output the global optimal path and use cubic B-spline interpolation to smooth it and generate the smoothed global optimal route. In the formula, To smooth out the independent parameter variables The generated 3D flight path coordinates point, For cubic B-spline basis functions, These are control points used for curve interpolation. The index number of the control point. For the independent parameter variables of the spline curve, This is the starting index value of the control point in the global control point sequence; If not, increment the iteration count by 1 and return to S2.

9. A reinforcement learning UAV 3D path planning system based on a hybrid ant colony algorithm, applied to the reinforcement learning UAV 3D path planning method based on a hybrid ant colony algorithm as described in any one of claims 1 to 8, characterized in that, It includes the decision-making layer, the execution layer, and the perception layer; The decision layer is equipped with a dual-delay deep deterministic policy gradient network, which receives the comprehensive state vector and outputs the control parameter action vector of the swarm intelligence algorithm. The execution layer includes an enhanced node transfer module, a minimum value linkage escape module, and an elite pheromone preservation update module. The enhanced node transfer module performs active obstacle avoidance selection, the minimum value linkage escape module performs Levy distribution long-distance jump anti-deadlock, and the elite pheromone preservation update module performs survival of the fittest to generate candidate three-dimensional routes. The perception layer includes a 3D environment hybrid modeling module, a multi-objective cost evaluation module, and a real-time state feature extraction module. The 3D environment hybrid modeling module and the multi-objective cost evaluation module calculate the losses of distance, threat, altitude, and smoothness, and quantify and score the candidate 3D routes. The real-time state feature extraction module extracts the convergence, diversity, and environmental complexity features of the optimization group and performs normalization processing to generate a comprehensive state vector and a composite reward value.

Images