A Cluster Path Planning Method and System Based on Improved A-star Algorithm and Reinforcement Learning

Abstract

The present invention provides a cluster path planning method and system based on an improved A-star algorithm and reinforcement learning, belonging to the field of UAV cluster control. To solve the problems of collision caused by the failure to consider the volume of the cluster and the failure in a high-dynamic environment in existing traditional path planning. The present invention performs path planning based on the improved A* path planning algorithm, generates an optimal path considering constraint conditions; through the setting of local target points, guides high-speed aircraft to fly along the shortest path; combines the SAC reinforcement learning algorithm to construct a reward function to guide high-speed aircraft to fly towards the target; designs a target flight reward according to the constraint conditions of high-speed aircraft; adopts a hierarchical control framework based on the improved A-star and deep reinforcement learning to further optimize the cluster flight path and obstacle avoidance ability. The present invention ensures the precise path planning and stable formation control of the cluster in a complex obstacle environment, and effectively improves the obstacle avoidance ability and cooperative control of the UAV cluster in a complex environment.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle (UAV) swarm control. Specifically, it relates to a swarm path planning method and system based on an improved A-star algorithm and reinforcement learning. Background Art

[0002] In the field of UAV swarm control and obstacle avoidance, traditional methods usually combine global path planning and local obstacle avoidance strategies to achieve path planning and obstacle avoidance control of UAVs in complex environments. The global path planning method can plan an optimal path from the starting point to the target point for a single UAV in a static and known environment, ensuring the feasibility and optimality of the path. In actual UAV swarm tasks, the local obstacle avoidance strategy dynamically adjusts the flight trajectory of the UAV by real-time sensing the changes in the surrounding environment through the UAV's sensors, avoiding collisions with dynamic obstacles. Such methods have certain effects in low-speed and sparse obstacle environments, especially in the path planning and obstacle avoidance control of a single UAV, and can achieve good path tracking and obstacle avoidance responses. At the same time, as a data-driven control method, deep reinforcement learning (DRL) has shown good effects in the path tracking and local obstacle avoidance tasks of single UAVs, and can achieve adaptive control through a policy network in a dynamic and uncertain environment, further improving the intelligence level of obstacle avoidance.

[0003] However, although traditional methods perform well in the path planning and obstacle avoidance control of a single UAV, they still face many challenges when dealing with the formation control and obstacle avoidance problems of high-speed UAV swarms. First, the path planning does not consider the volume constraints of the swarm. Traditional path planning algorithms (such as A* and Dijkstra) are mainly for the path planning of a single UAV and do not consider the volume limitations of multiple UAVs in the swarm. When multiple UAVs fly together, path overlaps and mutual interference between UAVs may occur, and even collisions between UAVs may result. Second, the obstacle avoidance reaction in a dynamic environment is not timely enough. The local obstacle avoidance algorithm usually relies on information in the neighboring environment for local adjustment, but in the face of high-density obstacles or high-speed dynamic threats, the reaction speed and computational complexity of traditional methods are difficult to meet the real-time requirements. Summary of the Invention

[0004] The technical problem to be solved by the present invention is:

[0005] To solve the problems of collisions caused by the lack of consideration of the swarm volume in existing traditional path planning and its failure in a highly dynamic environment.

[0006] The technical solution adopted by the present invention to solve the above technical problem:

[0007] The present invention provides a cluster path planning method based on an improved A-star algorithm and reinforcement learning, including the following steps:

[0008] S100, UAV modeling and environment modeling, including kinematic modeling of high-speed UAVs and modeling of task optimization problems;

[0009] S200, A formation control framework using a hierarchical controller. The upper layer uses an improved A* algorithm to control the situation in the obstacle-dense area, and constructs a heuristic function considering the volume of the UAV swarm and the influence range of obstacles, the overload constraint of high-speed aircraft, and the smoothness of the flight trajectory;

[0010] S300, After the planning algorithm completes the preliminary planning, generate the shortest path to the target point in the map, and select local target points on the shortest path at a specified search step size;

[0011] S400, The lower layer realizes the local path tracking control and dynamic obstacle avoidance control of the UAV through the SAC reinforcement learning algorithm, including designing the observation space, action space, and reward function.

[0012] Further, in step S100, it includes

[0013] S110, To accurately describe the flight state of a powerless high-speed UAV, kinematic modeling is required. The state variables of the UAV swarm are defined as:

[0014] [V γ ψ R E θ φ] T

[0015] Among them, V represents the flight speed of the aircraft; γ represents the flight path angle; ψ represents the flight heading angle; R E represents the geocentric distance; θ represents the geocentric longitude; φ represents the geocentric latitude;

[0016] Write the UAV motion equations according to the above state variables as follows:

[0017]

[0018] Among them, n x represents the acceleration along the flight direction, and g represents the acceleration due to gravity; n y represents the acceleration of the aircraft perpendicular to the velocity direction; n z represents the lateral acceleration of the aircraft;

[0019] S120, After completing the UAV modeling, it is also necessary to model the environmental constraints; the goal of the formation control of the aircraft swarm is to enable the swarm to avoid obstacles and reach the target point while maintaining the formation, and model this task as an optimization problem:

[0020] 1) Meet the overload constraint of the high-speed aircraft, where the actual overload n of the aircraft is less than the maximum overload constraint n max ;

[0021] 2) Maintain formation flight and ensure that the actual state u of the high-speed aircraft in the formation follows the desired motion state u d ;

[0022] 3) Avoid the obstacle area and ensure that the minimum distance between the position of the high-speed aircraft and the position of the nearest obstacle is not less than the set safety distance;

[0023] 4) The UAV cluster flies to the specified target position, ensuring that the distance difference between the position of the high-speed aircraft and the target position approaches 0;

[0024] The above optimization problem is expressed by the following formula:

[0025]

[0026] where α represents the weight coefficient of the formation optimization target, u i represents the state information of the i-th UAV in the UAV cluster, and β represents the weight coefficient of the distance constraint between the UAV and the target point position; d safe represents the safety distance between the high-speed UAV cluster and the obstacle area; F1 represents the UAV desired state decision function, c iN represents the weight coefficients of different UAV states, and F2 represents the controller of the aircraft itself.

[0027] Furthermore, in the path planning process of step S200, considering the overload constraint of the high-speed aircraft and the smoothness of the flight trajectory, after the initial planning based on the A* algorithm is completed, the nodes at both ends of several intervals are connected as the local target points of the high-speed aircraft cluster; the cluster will autonomously complete the flight mission from one target point to the next target point;

[0028] The action space of the agent is expressed as follows:

[0029] a = [a x , a y

[0030] a x , a y ∈[-2, -1, 0, 1, 2]

[0031] a x + a y ≥2, a x × a y ≠1

[0032] where a x , a​y Represent the actions in the x - direction and y - direction respectively;

[0033] Set the obstacle as a circular obstacle. If the agent collides with the obstacle, remove the path point in this moving direction and stop further exploration of the collision point position;

[0034] To check whether the path intersects with the obstacle, connect the starting point of the previous moment and the current target point. If the projection of this connection line does not contain the obstacle, the path should satisfy the following constraints:

[0035] ||(x next ,y next )-(x obs ,y obs )||2>r obs

[0036] where, (x next ,y next ) represents the local target point of the current path, (x obs ,y obs ) represents the coordinate of the position of the obstacle closest to the current position of the UAV. Then the distance between the two coordinates needs to be greater than the radius r of the obstacle obs ;

[0037] If the projection of the local path line contains the obstacle, the path satisfies the following constraints:

[0038] d((x obs ,y obs )-line((x current ,y current ),(x next ,y next )))<r obs

[0039] where, line((x current ,y current ),(x next ,y next )) represents the local path line, the path line from the current position to the next local target point, and (x obs ,y obs ) represents the obstacle position coordinate;

[0040] If the distance from the obstacle position to the path line is less than the preset radius of the obstacle, it is considered that the path line at this time intersects with the obstacle cylinder. If the UAV cluster travels along this route, it will collide with the obstacle;

[0041] Conduct constraints on the collective volume. Assume the radius of the aircraft cluster is r formation , and the radius of the virtual obstacle is:

[0042] r = r obs + r formation 。

[0043] Furthermore, in step S300, by selecting local target points, it is used to achieve flexible path following of the cluster within a local range, and the flight direction and speed are dynamically adjusted according to real-time sensor data, so as to effectively avoid obstacles;

[0044] (x t , y t ), t ∈ T

[0045] (x kt , y kt ), kt ∈ T

[0046] (k - 1)t

[0047] Among them, (x t , y t ) represents all path points, and these path points form the shortest path obtained by the upper-layer planning. (x kt , y kt ) represents the local path target point obtained according to the step size of (k - 1)t.

[0048] Furthermore, in step S400, the input of the SAC reinforcement learning algorithm is the observation data collected in the observation space and outputs the actions that the UAV should execute selected from the action space.

[0049] Each agent learns the policy through interaction with the environment. The goal of the aircraft is to obtain the maximum total long-term expected reward in a single episode:

[0050]

[0051] Among them, γ ∈ [0, 1] is the discount factor. When γ → 1, it allows the learning policy to pay more attention to the long-term reward. R t+k represents the reward obtained by the agent at the (t + k)-th step;

[0052] The observation space of the high-speed aircraft includes its own state, the states of surrounding aircraft, obstacle information, and target state; the state of the aircraft itself includes the aircraft's own position information [x, y, z], and the aircraft's flight speed information [v x , v y , v z ; the state information of surrounding aircraft includes the relative position relationship [Δx, Δy, Δz] and relative speed relationship [Δv x , Δv y , Δv z; The obstacle information includes the relative position information between the obstacle and the aircraft [Δx o , Δy o ; And obtain the relative position information between the target and the aircraft [Δx t , Δy t , Δz t ;

[0053] The action space of the high-speed aircraft consists of the normal overload of the aircraft, and the normal overload includes two directions [n y , n z . The overload magnitude output by the neural network needs to be limited to ensure that the actual overload of the aircraft is less than its allowable maximum overload value.

[0054] Furthermore, by designing a reward function for the path planning and control of the high-speed aircraft, it is used to guide the high-speed aircraft to fly efficiently towards the predetermined target area or track the target, including,

[0055] r t = k t × ||Δv t ‖ × sgn(Δv t )

[0056] where k t is the weight coefficient of this reward, ‖Δv t ‖ represents the absolute value of the relative velocity between the high-speed aircraft and the target; when the aircraft moves towards the target point, sgn(Δv t ) = 1, when the aircraft moves away from the target point, sgn(Δv t ) = -1;

[0057] r f = k f × ‖ΔR‖

[0058] where k f is the weight coefficient of this reward, ||ΔR|| represents the average deviation of the distance between a high-speed aircraft and several surrounding aircraft. The goal is to achieve a reward function that remains zero when the high-speed aircraft perfectly maintains its original formation;

[0059] r o = k o × ||Δr o || 2

[0060] where k o is the weight coefficient of this reward, ||Δr o|| represents the relative distance between the high-speed aircraft and the obstacle area, and the value of this reward item is proportional to the square of this relative distance; the reward function is designed such that when the aircraft approaches the obstacle area, the penalty increases sharply; when the aircraft moves away from the obstacle, the penalty decreases rapidly;

[0061] By summing up the aforementioned rewards, the total reward function can be obtained:

[0062] r = r t + r f + r o .

[0063] A cluster path planning system based on an improved A-star algorithm and reinforcement learning, the system has program modules corresponding to the above steps, and when running, it executes the steps in the above-mentioned cluster path planning method based on the improved A-star algorithm and reinforcement learning.

[0064] A computer-readable storage medium, the computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the cluster path planning method based on the improved A-star algorithm and reinforcement learning when called by a processor.

[0065] Compared with the prior art, the beneficial effects of the present invention are:

[0066] The present invention proposes an obstacle avoidance control method for intelligent formation of high-speed aircraft clusters based on a hierarchical control framework, which combines an improved A* path planning algorithm with deep reinforcement learning (SAC). Compared with traditional single path planning or single reinforcement learning control methods, the present invention has the following significant advantages in terms of path planning accuracy, cluster formation stability, and dynamic obstacle avoidance response speed: 1) Significantly improved path planning accuracy and safety: The present invention introduces the concept of collective volume constraint in A-star path planning, considering the collective volume characteristics and space occupancy of multiple unmanned aerial vehicles (UAVs), so that the planned path not only meets the passage requirements of a single UAV, but also ensures the passability and safety of the entire cluster. 2) Faster dynamic obstacle avoidance response speed and higher obstacle avoidance success rate: The present invention applies the SAC deep reinforcement learning algorithm to local control and dynamic obstacle avoidance, greatly improving the response speed and obstacle avoidance success rate of high-speed aircraft in dealing with dynamic obstacles. Traditional local obstacle avoidance methods often have difficulty achieving fast response when dealing with dynamic obstacles and are prone to path failure. In contrast, the SAC algorithm can generate more robust control strategies in an uncertain environment by introducing an entropy regularization mechanism, enabling the UAV to quickly adjust the path when encountering dynamic obstacles, avoid collisions, and ensure the smoothness of path adjustment and the stability of the formation. 3) Stronger formation stability and more stable formation maintenance: In traditional multi-UAV control systems, it is difficult to maintain the stability of the cluster solely relying on path planning methods. Especially during path adjustment and obstacle avoidance, problems such as UAVs deviating from the formation, uneven paths, and formation disbandment may occur in the cluster. The hierarchical control architecture of the present invention combines the global planning of A* and the local control of SAC, enabling the UAVs to always maintain the stability of the relative formation while following the path. During the formation process, the SAC controller adjusts the actions of the local UAV based on the state information of neighboring UAVs to ensure that the formation shape of the entire cluster remains unchanged and avoid formation disbandment or formation failure caused by path changes.

[0067] The present invention can ensure accurate path planning and stable formation control of the UAV cluster in the UAV cluster scenario, effectively improving the obstacle avoidance ability and cooperative control of the UAV cluster in a complex environment. Brief Description of the Drawings

[0068] Figure 1 It is the structure diagram of the layer controller framework in the embodiment of the present invention;

[0069] Figure 2 It is the schematic diagram of the movement direction of the intelligent agent of the improved A-star planning algorithm in the embodiment of the present invention;

[0070] Figure 3 It is the scenario diagram of the intelligent agent encountering an obstacle in the embodiment of the present invention;

[0071] Figure 4This is the path planning result diagram of the A-star algorithm with constraints in the embodiments of the present invention;

[0072] Figure 5 This is the comparison diagram of the simulation results of the method proposed in the embodiments of the present invention, the unimproved A-star algorithm, the reinforcement learning method, and the method that only uses the reinforcement learning method. Detailed implementation manners

[0073] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following detailed description of the specific embodiments of the present invention will be given with reference to the accompanying drawings.

[0074] To solve the problems faced by UAV swarms in path planning and obstacle avoidance, the present invention proposes a swarm path planning method based on an improved A-star algorithm and reinforcement learning. On the basis of the original formation control method for high-speed aircraft swarms based on deep reinforcement learning, an upper-layer constrained A* planning algorithm is introduced to control the situation in the dense obstacle area. It provides a safe flight path for the high-speed aircraft swarm in the environment, effectively improves the flight safety of the high-speed aircraft swarm under overload constraints, and can support the swarm to complete the formation flight task. The hierarchical controller design adopted by the present invention uses the upper-layer improved A-star algorithm to plan the global shortest path and the lower-layer SAC deep reinforcement learning algorithm as the control strategy to track the generated shortest path and perform local path planning, realizing obstacle avoidance control while ensuring the tracking of the shortest path.

[0075] Specific implementation manner 1: Combining Figure 1 As shown in the figure, the present invention provides a swarm path planning method based on an improved A-star algorithm and reinforcement learning, including the following steps:

[0076] S100. UAV modeling and environment modeling, specifically including,

[0077] According to the embodiments of the present invention, the research object is a powerless high-speed UAV; due to insufficient engine thrust, the UAV cannot achieve maneuverability by controlling the magnitude of the flight direction overload; different from traditional UAVs with a power system, the powerless high-speed UAV does not rely on the engine to provide thrust, and its flight mainly relies on the initial speed and the action of gravity, and uses overload control to adjust the flight attitude and trajectory;

[0078] S110. To accurately describe the flight state of the powerless high-speed UAV, kinematic modeling needs to be carried out on it. The state variables of the UAV swarm are defined as:

[0079] [VγψR E θφ] T

[0080] Among the above state variables, V represents the flight speed of the aircraft; γ represents the flight path angle; ψ represents the flight heading angle; R E represents the geocentric distance; θ represents the geocentric longitude; φ represents the geocentric latitude;

[0081] Based on the above state variables, the motion equations of the UAV are written as follows:

[0082]

[0083] The first equation in the above equations describes the rate of change of the aircraft speed over time, affected by gravity and the normal acceleration component; where n x represents the acceleration along the flight direction, and g represents the acceleration due to gravity;

[0084] The second equation in the above equations indicates that the change in the flight path angle is affected by the vertical acceleration and the curvature of the Earth, where n y represents the acceleration of the aircraft perpendicular to the velocity direction;

[0085] The third equation is the heading angle rate of change equation, which describes the change in the heading angle, mainly affected by the lateral acceleration and the curvature of the Earth. In high-latitude regions, the change in latitude has a greater impact on the heading angle; where n z represents the lateral acceleration of the aircraft;

[0086] The fourth equation represents the geocentric distance rate of change equation, which describes the change in the geocentric distance of the aircraft related to the aircraft speed and the flight path angle;

[0087] The fifth and sixth equations respectively represent the latitude and longitude rate of change equations, and their changes are mainly related to state variables such as speed, flight path angle, and heading angle;

[0088] S120. After completing the modeling of the UAV, it is also necessary to model the environmental constraints; the goal of the formation control of the aircraft cluster is to enable the cluster to avoid obstacles and reach the target point while maintaining the formation, and this task can be modeled as an optimization problem:

[0089] 1) Satisfy the overload constraint of the high-speed aircraft. The actual overload n of the aircraft is less than the maximum overload constraint n max ;

[0090] 2) Maintain formation flight and ensure that the actual state u of the high-speed aircraft in the formation follows the desired motion state u d ;

[0091] 3) Avoid the obstacle area and ensure that the minimum distance between the position of the high-speed aircraft and the position of the nearest obstacle is not less than the set safety distance;

[0092] 4) The UAV swarm flies to the specified target position, ensuring that the distance difference between the position of the high-speed aircraft and the target position approaches 0;

[0093] The above optimization problem is expressed by the following formula:

[0094]

[0095] Among them, α represents the weight coefficient of the formation optimization target, and u i represents the state information of the i-th UAV in the UAV swarm, and β represents the weight coefficient of the distance constraint between the UAV and the target point position; d safe represents the safety distance between the high-speed UAV swarm and the obstacle area; F1 represents the UAV expected state decision function, which determines the expected state of the current aircraft according to the input states of other aircraft in the swarm, and c iN represents the weight coefficient of different UAV states; F2 represents the controller of the aircraft itself, and calculates the UAV overload control command according to the expected motion state of the UAV obtained from the previous equation;

[0096] S200, Path planning area division and start / end point setting;

[0097] According to the embodiments of the present invention, the initial position of the UAV swarm and the position of the target point are set in the environment, and dense obstacles are randomly generated;

[0098] S300, Construct a heuristic function by comprehensively considering the volume of the UAV swarm, the influence range of obstacles, the overload constraint of the high-speed aircraft, and the flight trajectory smoothness;

[0099] The present invention adopts a hierarchical formation control framework. The upper layer uses path planning based on the A* algorithm to provide local target points in a complex obstacle environment; the lower layer uses intelligent formation control based on reinforcement learning to enable each UAV to autonomously form a formation, fly to the local target point, and avoid nearby obstacles;

[0100] Combined with Figure 1 As shown, after determining the initial position and target position of the UAV swarm, the improved A* algorithm starts searching from the starting point to find the shortest path to the target; along this path, a point is selected as the local target point of the swarm within the specified search step; when the UAV swarm reaches this target point, the next local target point is updated as the new target of the swarm;

[0101] The upper-layer controller uses the improved A* algorithm for path planning. The A* algorithm is one of the best-known path planning algorithms and can be applied to metric or topological configuration spaces; this algorithm combines heuristic search and shortest-path-based search, evaluates the value of each cell in the configuration space, and the cost function formula of the A* algorithm is as follows:

[0102] f(v) = g(v) + h(v)

[0103] Among them, g(v) represents the cost from the starting point to the current position of the UAV, and h(v) represents the cost from the current point of the UAV to the target position. By selecting the lowest cost and f(v), the shortest path of the UAV from the starting position to the target point in the whole process is obtained;

[0104] In the present invention, during the path planning process, the intelligent agent will consider circular obstacles at any position. Different from the previous practice of regarding the obstacle area as a square, obstacle avoidance is achieved through collision detection between the moving direction and the circular obstacle; replacing the obstacle with a circle can be closer to the real environment, simplify the collision detection calculation, improve the obstacle avoidance accuracy and path planning efficiency, and avoid the redundant interference of the square boundary, thereby achieving a more flexible and efficient obstacle avoidance ability;

[0105] Preferably, the planning algorithm will consider the overload constraint of the high-speed aircraft and the smoothness of the flight trajectory; after the initial planning is completed, the nodes at both ends of several intervals are connected as the local target points of the high-speed aircraft cluster; the cluster will autonomously complete the flight mission from one target point to the next target point;

[0106] Preferably, during the path planning process of the UAV, each intelligent agent can select 16 different motion modes. The motion modes are as Figure 2 shown in the improved A-star planning algorithm intelligent agent motion direction, and the action space of the intelligent agent is expressed as follows using the formula:

[0107] a = [a x , a y

[0108] a x , a y ∈[-2, -1, 0, 1, 2]

[0109] a x + a y ≥2, a x × a y ≠1

[0110] Among them, a x , a y respectively represent the actions of the intelligent agent in the x direction and the y direction;

[0111] During the path planning process, if the intelligent agent collides with an obstacle, such as Figure 3 ​As shown in the scenario where the agent encounters an obstacle, it is necessary to remove the path points in the moving direction and stop further exploration of the collision point position. To check whether the path intersects with the obstacle, it is necessary to calculate the local path line, that is, calculate the line connecting the starting point at the previous moment and the current target point. If the projection of this line does not contain an obstacle, the path should satisfy the following constraints:

[0112] ||(x next ,y next )-(x obs ,y obs )||2>r obs

[0113] where (x next ,y next ) represents the local target point of the current path, and (x obs ,y obs ) represents the position coordinates of the obstacle closest to the current position of the UAV. Then the distance between the two coordinates needs to be greater than the obstacle radius r obs ;

[0114] If the projection of the local path line contains an obstacle, the path satisfies the following constraints:

[0115] d((x obs ,y obs )-line((x current ,y current ),(x next ,y next )))<r obs

[0116] where line((x current ,y current ),(x next ,y next )) represents the local path line, the path line from the current position to the next local target point, and (x obs ,y obs ) represents the obstacle position coordinates. If the distance from the obstacle position to the path line is less than the preset obstacle radius, it is considered that the path line intersects with the obstacle cylinder at this time. If the UAV cluster travels along this route, it will collide with the obstacle;

[0117] Preferably, since the research object of the present invention is a high-speed aircraft cluster, it is necessary to constrain the volume of the cluster in the A* algorithm path planning. Assume that the radius of the aircraft cluster is r formation . To ensure that the cluster does not collide with obstacles due to the cluster volume when tracking the shortest path, a virtual obstacle radius is set here, and its calculation formula is as follows:

[0118] r=robs +r formation

[0119] By setting the virtual obstacle radius through the above formula, it can be ensured that when the cluster moves along the planned path, the cluster will not collide with any obstacles. This part is different from the traditional single UAV path planning;

[0120] S400. The planning algorithm completes the preliminary planning, generates the shortest path to the target point in the map, and selects local target points on the shortest path with a specified search step size;

[0121] The shortest path obtained from the above planning aims to provide dynamic and efficient local target point information for the cluster, so as to guide the cluster to perform precise path following tasks in a complex obstacle environment; different from traditional global path planning, this method not only focuses on determining the global optimal path, but also guides the UAV cluster to adjust the flight direction in real time by selecting local target points to ensure the actual feasibility and dynamic adaptability of the path; these local target points are selected from the global path according to the preset step size and planning results, and gradually provide directional guidance information for the cluster;

[0122] During the actual execution process, the cluster UAVs do not strictly fly one by one according to the global path points generated by the upper-layer path planning algorithm, because under the conditions of numerous obstacles and dynamic environmental changes, strictly following the global path may lead to problems such as non-smooth paths and untimely obstacle avoidance, and even affect the overall flight efficiency and safety of the cluster; by selecting local target points, the cluster can achieve more flexible path following in the local range and dynamically adjust the flight direction and speed according to real-time sensor data, so as to effectively avoid obstacles and move forward along the optimal direction at the same time;

[0123] (x t ,y t ), t ∈ T

[0124] (x kt ,y kt ), kt ∈ T

[0125] (k - 1)t

[0126] Among them, (x t ,y t ) represents all path points, and these path points form the shortest path obtained from the upper-layer planning. (x kt ,y kt ) represents the local path target point obtained according to the (k - 1)t step size;

[0127] S500. Implement the local path tracking control and dynamic obstacle avoidance control of the UAV through the SAC reinforcement learning algorithm;

[0128] The optimization problem defined in step S100 of the present invention can be described using a Markov decision process, so as to solve the problem using a reinforcement learning framework; in the reinforcement learning framework, if a certain behavior strategy of the agent leads to a positive reward from the environment to the agent, the tendency of the agent to adopt this behavior strategy in the future will be strengthened; if the learning adopted by the agent to adapt to the environment has the following characteristics, it is called reinforcement learning: the agent is not static and passively waiting, but actively explores the environment;

[0129] In reinforcement learning, the agent selects an action a to act on the environment. The environment changes after receiving the action, and generates a reinforcement signal r to feed back to the agent. The agent then selects the next action based on the reinforcement signal and the current state of the environment. The principle of selection is to increase the probability of receiving a positive reward. The selected action not only affects the immediate reinforcement value, but also affects the state of the environment at the next moment and the final reinforcement value.

[0130] In the reinforcement learning framework of the present invention, each agent learns a strategy by interacting with the environment; the goal of the aircraft is to obtain the maximum long-term expected reward sum in a single round:

[0131]

[0132] In the above formula, γ∈[0,1] is a discount factor. When γ→1, the learning strategy is allowed to pay more attention to long-term rewards. t+k Represents the reward obtained by the agent at step t+k;

[0133] In the present invention, all drones in the cluster share the same network parameters during training and application, and make decisions by observing the status of other aircraft and the environment;

[0134] The observation space of a high-speed aircraft includes its own state, the state of surrounding aircraft, obstacle information, and target state; the state of the aircraft itself includes the aircraft's own position information [x, y, z], the aircraft's flight speed information [v x ,v y ,v z ]; the surrounding aircraft status information includes the relative position relationship [Δx, Δy, Δz] and relative speed relationship [Δv x ,Δv y ,Δv z ]; obstacle information is also obtained through the observation space, which includes the relative position information between the obstacle and the aircraft [Δx o ,Δy o, and since the present invention regards the obstacle as a cylinder rather than a cube, this relative position information is a two-dimensional vector; finally, the observation space also observes and obtains the relative position information of the target and the aircraft [Δx t , Δy t , Δz t ;

[0135] The action space of the high-speed aircraft consists of the normal overload of the aircraft, and the normal overload includes two directions [n y , n z ; the overload magnitude output by the neural network needs to be limited to ensure that the actual overload of the aircraft is less than its allowable maximum overload value;

[0136] In the design of the reward function of the present invention, by analyzing the physical constraints and mission requirements of the high-speed aircraft, a reward function applicable to aircraft path planning and control is established; the core goal of this reward function is to guide the high-speed aircraft to fly efficiently towards the predetermined target area or the tracking target; during the design process, the relative speed between the aircraft and the target, a key factor, is emphasized as an important indicator to measure the flight effect of the aircraft;

[0137] Specifically, to encourage the aircraft to continuously adjust its flight direction and accelerate towards the target during flight, a target flight reward mechanism is designed; the greater the speed of the aircraft relative to the target, especially the component of the speed in the target direction, the greater the reward value obtained; this design ensures that the aircraft can always maintain dynamic tracking of the target while flying at high speed, avoiding situations such as deviation from the course or waste of speed;

[0138] r t = k t × ||Δv t || × sgn(Δv t )

[0139] In this equation, k t is the weight coefficient of this reward, ||Δv t || represents the absolute value of the relative speed between the high-speed aircraft and the target; when the aircraft moves towards the target point, sgn(Δv t ) = 1, and when the aircraft moves away from the target point, sgn(Δv t ) = -1;

[0140] To maintain the formation flight stability within the high-speed aircraft cluster, each aircraft calculates the corresponding reward according to the position difference relative to the surrounding aircraft. The purpose of this part of the reward function design is to punish the scenarios of deviating from the initial formation or the desired formation, and its formula is as follows:

[0141] r f= k f × ||ΔR||

[0142] In the above expression, k f is the weight coefficient of this reward, and ||ΔR|| represents the average deviation of the distance between a high-speed aircraft and several surrounding aircraft. The goal is to achieve a reward function that remains zero when the high-speed aircraft perfectly maintains its original formation;

[0143] To ensure that the high-speed aircraft avoids the obstacle area during the learning process and maintains a safe distance from it, the present invention designs a reward to punish the aircraft when it approaches the obstacle area. The following is the method for formulating this reward:

[0144] r o = k o × ||Δr o || 2

[0145] In this formula, k o is the weight coefficient of this reward, and ||Δr o || represents the relative distance between the high-speed aircraft and the obstacle area. The value of this reward term is proportional to the square of this relative distance; the design of this reward function makes the punishment increase sharply when the aircraft approaches the obstacle area; when the aircraft is far from the obstacle, the punishment decreases rapidly;

[0146] By summing up the aforementioned rewards, the total reward function can be obtained:

[0147] r = r t + r f + r o

[0148] Simulation test

[0149] The present invention has preliminarily tested the path planning ability of the improved A* algorithm in various environments, and performed simulations by setting different initial positions and targets in the same environment, as Figure 4 shown in the path planning result of the constrained A-star algorithm; the gray area represents the virtual obstacle area considering the volume constraint of the aircraft group, the red dot and the green dot respectively represent the starting point and the ending point, and the blue line represents the initially planned path; it can be seen from the path planning simulation result that the path planning algorithm of the invention can achieve the preliminary planning of the global path on the premise of considering the volume constraint of the group;

[0150] Preferably, the cluster path planning method based on the improved A-star and reinforcement learning algorithms proposed by the present invention is compared with other methods; in this part of the experiment, the method proposed by the present invention, the unimproved A-star and reinforcement learning method, and the method using only reinforcement learning are compared. These methods will be tested in several identical simulation environments to evaluate their respective flight performances; this part will illustrate the deficiencies of traditional reinforcement learning in formation control and will demonstrate how to introduce the A* algorithm for upper-layer planning to improve the formation control performance;

[0151] The simulation experiment results of the three methods are as Figure 5 shown in the comparison chart of the simulation results of various methods. Among the first type of A-star and reinforcement learning algorithms considering constraints proposed by the present invention, the high-speed aircraft cluster can fly safely through the obstacle area and reach the specified target point. During this process, the cluster formation is well maintained, and there is no internal collision of the aircraft in the cluster due to formation change or collision with obstacles in the environment, achieving the expected effect of the cluster path planning method; the second method is the A-star and reinforcement learning method, and the difference between this method and the first method is that it does not consider the volume limitation of the aircraft cluster. It can be seen from the simulation results of the second method that its formation maintenance effect is poor, and two of the three experiments failed to achieve the expected cluster path planning effect; the last method only uses the deep reinforcement learning algorithm. Since the high-speed aircraft cannot predict the future obstacle area and there is no upper-layer path planning, it is easy to enter the obstacle area during the local obstacle avoidance process, resulting in task failure.

[0152] Specific implementation plan two: A cluster path planning system based on the improved A-star algorithm and reinforcement learning of the present invention. This system has program modules corresponding to the above steps and executes the steps in the cluster path planning method based on the improved A-star algorithm and reinforcement learning when running.

[0153] The other combinations and connection relationships of this implementation plan are the same as those of the first specific implementation plan.

[0154] Specific implementation plan three: A computer-readable storage medium of the present invention. The computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the cluster path planning method based on the improved A-star algorithm and reinforcement learning when called by a processor.

[0155] The other combinations and connection relationships of this implementation plan are the same as those of the first specific implementation plan.

[0156] The other combinations and connection relationships of this implementation plan are the same as those of the first specific implementation plan.

[0157] Although the present invention is disclosed as above, the scope of protection of the present invention is not limited thereto. Those skilled in the art of the present invention can make various changes and modifications without departing from the spirit and scope of the present disclosure, and these changes and modifications will all fall within the scope of protection of the present invention.

Claims

1. A cluster path planning method based on an improved A-star algorithm and reinforcement learning, characterized in that It includes the following steps: S100, UAV modeling and environment modeling, including kinematic modeling of high-speed UAVs and modeling of mission optimization problems; S200, A formation control framework using a hierarchical controller. The upper layer uses an improved A* algorithm to control the situation in the obstacle-dense area. A heuristic function is constructed considering the volume of the UAV swarm, the influence range of obstacles, the overload constraint of high-speed aircraft, and the flight trajectory smoothness; Considering the overload constraint of high-speed aircraft and the smoothness of the flight trajectory, after the initial planning based on the A* algorithm is completed, the nodes at both ends of several intervals are connected as the local target points of the high-speed aircraft swarm; the swarm will autonomously complete the flight mission from one target point to the next target point; The action space of the agent is expressed as follows: a = [a x , a y ​ a x ,a y ∈[-2,-1,0,1,2] a x +a y ≥2,a x ×a y ≠1 Among them, a x , a y respectively represent actions in the x - direction and the y - direction; The obstacle is set as a circular obstacle. If the agent collides with the obstacle, the path points in that direction are removed, and the further exploration of the collision point position is stopped; To check whether the path intersects with the obstacle, connect the starting point of the previous moment and the current target point. If the projection of this connection line does not contain the obstacle, the path should satisfy the following constraints: ||(x next ,y next )-(x obs ,y obs )||2 > r obs Among them, (x next , y next ) represents the local target point of the current path, and (x obs , y obs ) represents the coordinate of the obstacle position closest to the current position of the UAV. Then the distance between the two coordinates needs to be greater than the radius r obs ; If the projection of the local path line contains the obstacle, the path satisfies the following constraints: d((x obs ,y obs ) - line((x current ,y current ),(x next ,y next ))) < r obs Among them, line((x current , y current ), (x next , y next )) represents a local path line, the path line from the current position to the next local target point, and (x obs , y obs ) represents the obstacle position coordinates; If the distance from the obstacle position to the path line is less than the preset obstacle radius, it is considered that the path line intersects with the obstacle cylinder at this time, and if the UAV swarm travels along this route, it will collide with the obstacle; Constrain the volume of the swarm. Assume the radius of the aircraft swarm is r formation , and the radius of the virtual obstacle is: r＝r obs +r formatio\ S300, After the planning algorithm completes the preliminary planning, generate the shortest path to the target point in the map, and select the local target points on the shortest path with a specified search step size; S400, The lower layer realizes the local path tracking control and dynamic obstacle avoidance control of the UAV through the SAC reinforcement learning algorithm, including designing the observation space, action space, and reward function; The input of the SAC reinforcement learning algorithm is the observation data collected in the observation space and outputs the actions that the UAV should execute selected from the action space. Each agent learns the policy through interaction with the environment. The goal of the aircraft is to obtain the maximum total long-term expected reward in a single episode: Among them, γ ∈ [0, 1] is the discount factor, allowing the learning strategy to pay more attention to long-term rewards when γ → 1, and R t+k represents the reward obtained by the agent at the (t + k)-th step; The observation space of a high-speed aircraft includes its own state, the states of surrounding aircraft, obstacle information, and target state; the state of the aircraft itself includes the aircraft's own position information [x, y, z], the aircraft's flight speed information [v x , v y , v z ; the state information of surrounding aircraft includes the relative position relationship [Δx, Δy, Δz] and relative speed relationship [Δv x , Δv y , Δv z between the own aircraft and the surrounding aircraft; the obstacle information includes the relative position information [Δx o , Δy o between the obstacle and the aircraft; and the relative position information [Δx t , Δy t , Δz t of the target and the aircraft is obtained; The action space of a high-speed aircraft consists of the normal overload of the aircraft, and the normal overload includes two directions [n y , n z . The magnitude of the overload output by the neural network needs to be limited to ensure that the actual overload of the aircraft is less than its maximum allowable overload value.

2. The cluster path planning method based on the improved A-star algorithm and reinforcement learning according to claim 1, characterized in that: In step S100, it includes, S110, To accurately describe the flight state of the unpowered high-speed UAV, kinematic modeling is required. The state variables of the UAV swarm are defined as: [VγψR E θφ] T where, V represents the flight speed of the aircraft; γ represents the flight path angle; ψ represents the flight heading angle; R E represents the geocentric distance; θ represents the geocentric longitude; φ represents the geocentric latitude; Write the UAV motion equations according to the above state variables as follows: where n x represents the acceleration along the flight direction, and g represents the acceleration due to gravity; n y represents the acceleration of the aircraft perpendicular to the velocity direction; n z represents the lateral acceleration of the aircraft; S120, After completing the UAV modeling, it is also necessary to model the environmental constraints; the goal of the aircraft swarm formation control is to enable the swarm to avoid obstacles and reach the target point while maintaining the formation. This task is modeled as an optimization problem: 1) Meet the overload constraint of the high-speed aircraft, where the actual overload n of the aircraft is less than the maximum overload constraint n max ; 2) Maintain formation flight to ensure that the actual state u of the high-speed aircraft in the formation follows the desired motion state u d ; 3) Avoid the obstacle area to ensure that the minimum distance between the position of the high-speed aircraft and the position of the nearest obstacle is not less than the set safety distance; 4) The UAV swarm flies to the specified target position to ensure that the distance difference between the position of the high-speed aircraft and the target position approaches 0; The above optimization problem is expressed by the formula as follows: Among them, α represents the weight coefficient of the formation optimization objective, and u i represents the state information of the i-th unmanned aerial vehicle in the unmanned aerial vehicle cluster, and β represents the weight coefficient of the distance constraint between the unmanned aerial vehicle and the target point position; d safe represents the safe distance between the high-speed unmanned aerial vehicle cluster and the obstacle area; F1 represents the decision function of the expected state of the unmanned aerial vehicle, and c iN represents the weight coefficients of different unmanned aerial vehicle states, and F2 represents the controller of the aircraft itself.

3. A cluster path planning method based on an improved A-star algorithm and reinforcement learning according to claim 2, characterized in that: In step S300, by selecting local target points, it is used to achieve flexible path following of the cluster within a local range, and dynamically adjust the flight direction and speed according to real-time sensor data, so as to effectively avoid obstacles; (x t ,y t ),t∈T (x kt , y kt ), where \(t\in T\) (k - 1)t Among them, (x t , y t ) represents all the path points, and these path points form the shortest path obtained from the upper-level planning. (x kt , y kt ) represents the local path target point obtained according to the step size of (k - 1)t.

4. A cluster path planning method based on an improved A-star algorithm and reinforcement learning according to claim 3, characterized in that: Design a reward function through path planning and control of high-speed aircraft, which is used to guide the high-speed aircraft to fly efficiently towards a predetermined target area or a tracking target, including, r t = k t × ||Δv t || × sgn(Δv t ) where k t is the weight coefficient of the reward, and ||Δv t || represents the absolute value of the relative velocity between the hypersonic vehicle and the target; when the vehicle moves towards the target point, sgn(Δv t ) = 1, and when the vehicle moves away from the target point, sgn(Δv t ) = -1; r f = k f × ||ΔR|| where k f is the weight coefficient of the reward, and ||ΔR|| represents the average deviation of the distance between a high-speed aircraft and several surrounding aircraft. The goal is to achieve a reward function that remains zero while the high-speed aircraft perfectly maintains its original formation; r o = k o × ||Δr o || 2 where k o is the weight coefficient of the reward, ||Δr o || represents the relative distance between the high-speed aircraft and the obstacle area, and the value of this reward term is proportional to the square of this relative distance; the design of this reward function is such that when the aircraft approaches the obstacle area, the penalty increases sharply; when the aircraft moves away from the obstacle, the penalty decreases rapidly; By summing up the aforementioned rewards, the total reward function can be obtained: r＝r t +r f +r o 。 5. A cluster path planning system based on an improved A-star algorithm and reinforcement learning, characterized in that: This system has program modules corresponding to the steps of any one of the above claims 1-4, and when running, it executes the steps in the above-mentioned cluster path planning method based on the improved A-star algorithm and reinforcement learning.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the cluster path planning method based on the improved A-star algorithm and reinforcement learning described in any one of claims 1-4 when called by a processor.

Images