Cluster path planning method and system based on improved A-star algorithm and reinforcement learning

Abstract

The invention provides a cluster path planning method and system based on an improved A-star algorithm and reinforcement learning, and belongs to the field of unmanned aerial vehicle cluster control. The problems of collision and failure in a high-dynamic environment caused by the fact that cluster volume is not considered in existing traditional path planning are solved. According to the invention, path planning is carried out based on an improved A * path planning algorithm, and an optimal path is generated by considering constraint conditions; by setting a local target point, the high-speed aircraft is guided to fly along the shortest path; constructing a reward function in combination with an SAC reinforcement learning algorithm to guide the high-speed aircraft to fly to the target; designing a target flight reward according to the constraint condition of the high-speed aircraft; and a hierarchical control framework based on improved A-star and deep reinforcement learning is adopted to further optimize a cluster flight path and obstacle avoidance capability. Accurate path planning and stable formation control of the cluster are ensured in the complex obstacle environment, and the obstacle avoidance capability and cooperative control of the unmanned aerial vehicle cluster in the complex environment are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle cluster control, and in particular to a cluster path planning method and system based on an improved A-star algorithm and reinforcement learning. Background Art

[0002] In the field of control and obstacle avoidance of drone swarms, traditional methods usually adopt a combination of global path planning and local obstacle avoidance strategies to achieve path planning and obstacle avoidance control of drones in complex environments. The global path planning method can plan an optimal path from the starting point to the target point for a single drone in a static known environment, ensuring the feasibility and optimality of the path. In actual drone swarm tasks, the local obstacle avoidance strategy uses the drone's sensors to perceive changes in the surrounding environment in real time, dynamically adjust the motion trajectory of the aircraft, and avoid collisions with dynamic obstacles. This type of method has a certain effect in low-speed and sparse obstacle environments, especially in the path planning and obstacle avoidance control of a single drone, which can achieve good path tracking and obstacle avoidance response. At the same time, deep reinforcement learning (DRL), as a data-driven control method, has shown good results in the path tracking and local obstacle avoidance tasks of a single drone, and can achieve adaptive control through a policy network in a dynamic uncertain environment, further improving the intelligent level of obstacle avoidance.

[0003] However, traditional methods perform well in path planning and obstacle avoidance control of a single UAV, but still face many challenges when dealing with formation control and obstacle avoidance problems of high-speed UAV clusters. First, path planning does not take into account the volume constraints of the cluster. Traditional path planning algorithms (such as A* and Dijkstra) are mainly aimed at path planning of a single UAV, and do not consider the cluster volume constraints of multiple UAVs in the cluster. As a result, when multiple UAVs fly together, there may be path overlaps and interference between UAVs, and even collisions between UAVs. Secondly, the obstacle avoidance response in a dynamic environment is not timely. Local obstacle avoidance algorithms usually rely on information from the neighboring environment for local adjustments, but when faced with high-density obstacles or high-speed dynamic threats, the reaction speed and computational complexity of traditional methods are difficult to meet real-time requirements. Summary of the invention

[0004] The technical problems to be solved by the present invention are:

[0005] In order to solve the problem that the existing traditional path planning does not take the cluster volume into consideration, resulting in collision and failure in high dynamic environments.

[0006] The present invention adopts the following technical solutions to solve the above technical problems:

[0007] The present invention provides a cluster path planning method based on an improved A-star algorithm and reinforcement learning, comprising the following steps:

[0008] S100, UAV modeling and environment modeling, including kinematic modeling of high-speed UAVs and modeling of mission optimization problems;

[0009] S200, a formation control framework using a hierarchical controller, the upper layer uses an improved A* algorithm to control the situation in the obstacle-dense area, and considers the volume of the drone cluster and the impact range of obstacles, the overload constraint of high-speed aircraft and the smoothness of the flight trajectory to build a heuristic function;

[0010] S300, after the planning algorithm completes the preliminary planning, the shortest path to the target point is generated in the map, and the local target point on the shortest path is selected with a specified search step size;

[0011] S400, the lower layer realizes the local path tracking control and dynamic obstacle avoidance control of the UAV through the SAC reinforcement learning algorithm, including the design of observation space, action space and reward function.

[0012] Furthermore, in step S100, it includes:

[0013] S110. In order to accurately describe the flight state of unpowered high-speed drones, kinematic modeling is required. The state variables of the drone cluster are defined as:

[0014] [V γ ψ R E θ φ] T

[0015] Where V represents the flight speed of the aircraft; γ represents the flight path angle; ψ represents the flight heading angle; R E represents the distance from the center of the earth; θ represents the longitude of the earth; φ represents the latitude of the earth;

[0016] According to the above state variables, the UAV motion equations are listed as follows:

[0017]

[0018] Among them, n x represents the acceleration along the flight direction, g represents the acceleration due to gravity; n y Indicates the acceleration of the aircraft perpendicular to the velocity direction; n z Indicates the lateral acceleration of the aircraft;

[0019] S120. After completing the modeling of the UAV, it is also necessary to model the environmental constraints. The goal of the aircraft cluster formation control is to enable the cluster to avoid obstacles and reach the target point while maintaining the formation. This task is modeled as an optimization problem:

[0020] 1) The overload constraint of the high-speed aircraft is met, and the actual overload n of the aircraft is less than the maximum overload constraint n max ;

[0021] 2) Maintain formation flight to ensure that the actual state u of the high-speed aircraft in the formation follows the expected motion state u d ;

[0022] 3) Avoid obstacle areas and ensure that the minimum distance between the high-speed aircraft and the nearest obstacle is not less than the set safety distance;

[0023] 4) The UAV cluster flies to the designated target position, ensuring that the distance difference between the high-speed aircraft position and the target position is close to 0;

[0024] The above optimization problem is expressed using the formula as follows:

[0025]

[0026] Among them, α represents the weight coefficient of the formation optimization objective, u i represents the state information of the i-th UAV in the UAV cluster, β represents the weight coefficient of the distance constraint between the UAV and the target point; d safe represents the safe distance between the high-speed UAV cluster and the obstacle area; F1 represents the UAV expected state decision function, c iN Represents the weight coefficient of different drone states, and F2 represents the controller of the aircraft itself.

[0027] Furthermore, in the path planning process of step S200, the overload constraint of the high-speed aircraft and the smoothness of the flight trajectory are considered. After completing the initial planning based on the A* algorithm, the nodes at both ends of several intervals are connected as local target points of the high-speed aircraft cluster; the cluster will autonomously complete the flight mission from one target point to the next target point;

[0028] The action space of the agent is expressed as follows:

[0029] a=[a x ,a y ]

[0030] a x ,a y ∈[-2,-1,0,1,2]

[0031] a x +a y ≥2,a x ×a y ≠1

[0032] Among them, a x ,ay Represents the movement in the x direction and y direction respectively;

[0033] Set the obstacle to a circular obstacle. If the agent collides with the obstacle, remove the waypoint in the direction of movement and stop further exploration of the collision point.

[0034] In order to check whether the path intersects with obstacles, a line is drawn between the starting point at the last moment and the current target point. If the projection of the line does not contain obstacles, the path should satisfy the following constraints:

[0035] ||(x next ,y next )-(x obs ,y obs )||2>r obs

[0036] Among them, (x next ,y next ) represents the local target point of the current path, (x obs ,y obs ) represents the coordinates of the obstacle closest to the current position of the drone, and the distance between the two coordinates needs to be greater than the obstacle radius r obs ;

[0037] If the projection of the local path line contains obstacles, the path satisfies the following constraints:

[0038] d((x obs ,y obs )-line((x current ,y current ),(x next ,y next )))<r obs

[0039] Among them, line((x current ,y current ),(x next ,y next )) represents the local path line, the path line from the current position to the next local target point, (x obs ,y obs ) represents the obstacle position coordinates;

[0040] If the distance from the obstacle position to the path line is less than the preset obstacle radius, the path line is considered to intersect with the obstacle cylinder. If the drone cluster follows this route, it will collide with the obstacle.

[0041] Constrain the cluster volume, assuming that the radius of the aircraft cluster is r formation , the radius of the virtual obstacle is:

[0042] r=r obs +r formation .

[0043] Furthermore, in step S300, by selecting a local target point, the cluster can flexibly follow the path in the local range, and dynamically adjust the flight direction and speed according to the real-time sensor data, so as to effectively avoid obstacles;

[0044] (x t ,y t ),t∈T

[0045] (x kt ,y kt ),kt∈T

[0046] (k-1)t

[0047] Among them, (x t ,y t ) represents all the path points, which constitute the shortest path obtained by the upper-level planning. (x kt ,y kt ) represents the local path target point obtained according to the (k-1)t step size.

[0048] Furthermore, in step S400, the input of the SAC reinforcement learning algorithm is the observation data collected in the observation space and the output is the action that the drone should perform selected from the action space.

[0049] Each agent learns a strategy by interacting with the environment, and the goal of the robot is to obtain the maximum long-term expected reward sum in a single round:

[0050]

[0051] Where γ∈[0,1] is the discount factor. When γ→1, the learning strategy is allowed to pay more attention to long-term rewards. t+k Represents the reward obtained by the agent at step t+k;

[0052] The observation space of a high-speed aircraft includes its own state, the state of surrounding aircraft, obstacle information, and target state; the state of the aircraft itself includes its own position information [x, y, z], the flight speed information [v x ,v y ,v z ]; the surrounding aircraft status information includes the relative position relationship [Δx, Δy, Δz] and relative speed relationship [Δv x ,Δv y ,Δv z]; obstacle information includes the relative position information between the obstacle and the aircraft [Δx o ,Δy o ]; and obtain the relative position information between the target and the aircraft [Δx t ,Δy t ,Δz t ];

[0053] The action space of a high-speed aircraft is composed of the normal overload of the aircraft, which includes two directions [n y ,n z ], the overload size output by the neural network needs to be limited to ensure that the actual overload of the aircraft is less than its maximum allowable overload value.

[0054] Furthermore, a reward function is designed for the path planning and control of the high-speed aircraft to guide the high-speed aircraft to fly efficiently to a predetermined target area or tracking target, including:

[0055] r t =k t ×||Δv t ‖×sgn(Δv t )

[0056] Among them, k t is the weight coefficient of the reward, ‖Δv t || represents the absolute value of the relative speed between the high-speed aircraft and the target; when the aircraft moves toward the target point, sgn(Δv t )=1, when the aircraft moves away from the target point, sgn(Δv t )=-1;

[0057] r f =k f ×‖ΔR‖

[0058] Among them, k f is the weight coefficient of the reward, ||ΔR|| represents the average deviation of the distance between a high-speed aircraft and several surrounding aircraft. The goal is to achieve a reward function that remains zero while the high-speed aircraft perfectly maintains its original formation;

[0059] r o =k o ×||Δr o || 2

[0060] Among them, k o is the weight coefficient of the reward, ||Δr o|| represents the relative distance between the high-speed aircraft and the obstacle area. The value of this reward item is proportional to the square of the relative distance. The reward function is designed so that the penalty increases sharply when the aircraft approaches the obstacle area; when the aircraft moves away from the obstacle, the penalty decreases rapidly.

[0061] By summing up the above rewards, we can get the total reward function:

[0062] r=r t +r f +r o .

[0063] A cluster path planning system based on improved A-star algorithm and reinforcement learning, the system has a program module corresponding to the above steps, and executes the steps in the above cluster path planning method based on improved A-star algorithm and reinforcement learning during operation.

[0064] A computer-readable storage medium stores a computer program, wherein the computer program is configured to implement the steps of a cluster path planning method based on an improved A-star algorithm and reinforcement learning when called by a processor.

[0065] Compared with the prior art, the present invention has the following beneficial effects:

[0066] The present invention proposes an obstacle avoidance control method for intelligent formation of high-speed aircraft clusters based on a hierarchical control framework, which combines an improved A* path planning algorithm with deep reinforcement learning (SAC). Compared with traditional single path planning or single reinforcement learning control methods, the present invention has the following significant advantages in terms of path planning accuracy, cluster formation stability and dynamic obstacle avoidance response speed: 1) The accuracy and safety of path planning are significantly improved: The present invention introduces the concept of cluster volume constraint in A-star path planning, taking into account the cluster volume characteristics and space occupancy of multiple drones, so that the planned path not only meets the passage requirements of a single drone, but also ensures the overall passability and safety of the cluster. 2) The dynamic obstacle avoidance response speed is faster and the obstacle avoidance success rate is higher: The present invention greatly improves the response speed and obstacle avoidance success rate of high-speed aircraft in dealing with dynamic obstacles by applying the SAC deep reinforcement learning algorithm to local control and dynamic obstacle avoidance. Traditional local obstacle avoidance methods are often difficult to achieve rapid response when dealing with dynamic obstacles, and path failure is prone to occur. In contrast, the SAC algorithm can generate a more robust control strategy in an uncertain environment by introducing an entropy regularization mechanism, so that the drone can quickly adjust the path when encountering dynamic obstacles, avoid collisions, and ensure the smoothness of the path adjustment and the stability of the formation. 3) The formation is more stable and the formation is more stable: In traditional multi-drone control systems, it is difficult to maintain the stability of the cluster by relying solely on path planning methods. Especially in the process of path adjustment and obstacle avoidance, the cluster may have problems such as drones deviating from the formation, uneven paths, and formation dissolution. The hierarchical control architecture of the present invention combines the global planning of A* with the local control of SAC, so that the drone can always maintain the stability of the relative formation while following the path. During the formation process, the SAC controller adjusts the action of the machine through the status information of the neighboring machine to ensure that the formation shape of the entire cluster remains unchanged, avoiding the formation dissolution or formation failure caused by path changes.

[0067] The present invention can ensure accurate path planning and stable formation control of the cluster in a UAV cluster scenario, effectively improving the obstacle avoidance and collaborative control of the UAV cluster in a complex environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is a diagram of the framework structure of the middle-level controller in an embodiment of the present invention;

[0069] Figure 2 A schematic diagram of the movement direction of the intelligent agent in the improved A-star planning algorithm according to an embodiment of the present invention;

[0070] Figure 3 A scene diagram of an intelligent agent encountering an obstacle in an embodiment of the present invention;

[0071] Figure 4Result diagram of path planning of the A-star algorithm with constraints in an embodiment of the present invention;

[0072] Figure 5 It is a comparison chart of simulation results of the method proposed by the present invention, the unimproved A-star and reinforcement learning method, and the reinforcement learning method alone in the embodiments of the present invention. DETAILED DESCRIPTION

[0073] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0074] In order to solve the problems faced by drone clusters in path planning and obstacle avoidance, the present invention proposes a cluster path planning method based on an improved A-star algorithm and reinforcement learning. On the basis of the original high-speed aircraft cluster formation control method based on deep reinforcement learning, the upper-level constraint A* planning algorithm is introduced to control the situation in areas with dense obstacles. It provides a safe flight path for high-speed aircraft clusters in the environment, effectively improves the flight safety of high-speed aircraft clusters under overload constraints, and can support the cluster to complete formation flight missions. The hierarchical controller design adopted by the present invention uses the upper-level improved A-star algorithm to plan the global shortest path and the lower-level SAC deep reinforcement learning algorithm as the control strategy to track and plan the local path of the generated shortest path, while ensuring the tracking of the shortest path to achieve obstacle avoidance control.

[0075] Specific implementation plan 1: Combine Figure 1 As shown, the present invention provides a cluster path planning method based on improved A-star algorithm and reinforcement learning, comprising the following steps:

[0076] S100, UAV modeling and environment modeling, including:

[0077] According to the embodiment of the present invention, an unpowered high-speed UAV is taken as the research object; due to insufficient engine thrust, the UAV cannot achieve maneuverability by controlling the magnitude of overload in the flight direction; unlike traditional UAVs with power systems, unpowered high-speed UAVs do not rely on engines to provide thrust, and their flight mainly relies on the initial speed and gravity, and uses overload control to adjust the flight attitude and trajectory;

[0078] S110. In order to accurately describe the flight state of unpowered high-speed drones, kinematic modeling is required. The state variables of the drone cluster are defined as:

[0079] [VγψR E θφ] T

[0080] In the above state variables, V represents the flight speed of the aircraft; γ represents the flight path angle; ψ represents the flight heading angle; R E represents the distance from the center of the earth; θ represents the longitude of the earth; φ represents the latitude of the earth;

[0081] According to the above state variables, the UAV motion equations are listed as follows:

[0082]

[0083] The first equation in the above system of equations describes the rate of change of the aircraft's velocity with time, which is affected by gravity and normal acceleration components; where n x represents the acceleration along the flight direction, g represents the acceleration due to gravity;

[0084] The second equation in the above system of equations shows that the change in flight path angle is affected by the vertical acceleration and the curvature of the earth, where n y It represents the acceleration of the aircraft perpendicular to the velocity direction;

[0085] The third equation is the heading angle change rate equation, which describes the change of the heading angle, which is mainly affected by the lateral acceleration and the curvature of the earth. In high-latitude areas, the latitude change has a greater impact on the heading angle; where n z Indicates the lateral acceleration of the aircraft;

[0086] The fourth equation represents the rate of change of the distance from the center of the earth equation, which describes the change of the distance from the center of the earth of the aircraft as a function of the aircraft speed and the flight path angle;

[0087] The fifth and sixth equations represent the latitude and longitude change rate equations, respectively, and their changes are mainly related to state variables such as speed, flight path angle, and heading angle;

[0088] S120. After completing the modeling of the drone, it is also necessary to model the environmental constraints. The goal of the aircraft cluster formation control is to enable the cluster to avoid obstacles and reach the target point while maintaining the formation. This task can be modeled as an optimization problem:

[0089] 1) The overload constraint of the high-speed aircraft is met, and the actual overload n of the aircraft is less than the maximum overload constraint n max ;

[0090] 2) Maintain formation flight to ensure that the actual state u of the high-speed aircraft in the formation follows the expected motion state u d ;

[0091] 3) Avoid obstacle areas and ensure that the minimum distance between the high-speed aircraft and the nearest obstacle is not less than the set safety distance;

[0092] 4) The UAV cluster flies to the designated target position, ensuring that the distance difference between the high-speed aircraft position and the target position is close to 0;

[0093] The above optimization problem is expressed using the formula as follows:

[0094]

[0095] Among them, α represents the weight coefficient of the formation optimization objective, u i represents the state information of the i-th UAV in the UAV cluster, β represents the weight coefficient of the distance constraint between the UAV and the target point; d safe represents the safe distance between the high-speed UAV cluster and the obstacle area; F1 represents the UAV expected state decision function, which determines the expected state of the current aircraft based on the input state of other aircraft in the cluster, c iN Represents the weight coefficient of different UAV states; F2 represents the controller of the aircraft itself, and the UAV overload control command is calculated based on the expected motion state of the UAV obtained by the previous equation;

[0096] S200, dividing the route planning area and setting the starting and ending points;

[0097] According to an embodiment of the present invention, the initial position of the drone cluster and the position of the target point are set in the environment, and dense obstacles are randomly generated;

[0098] S300, constructs a heuristic function by comprehensively considering the volume of the drone cluster and the impact range of obstacles, the overload constraint of high-speed aircraft and the smoothness of the flight trajectory;

[0099] The present invention adopts a hierarchical formation control framework. The upper layer adopts path planning based on the A* algorithm to provide local target points in complex obstacle environments; the lower layer adopts intelligent formation control based on reinforcement learning, so that each drone can autonomously form a formation, fly to the local target point, and avoid nearby obstacles.

[0100] Combination Figure 1 As shown in the figure, after determining the initial position and target position of the drone cluster, the improved A* algorithm starts searching from the starting point to find the shortest path to the target; along this path, a point is selected as the local target point of the cluster within the specified search step; when the drone cluster reaches the target point, the next local target point is updated as the new target of the cluster;

[0101] The upper controller uses the improved A* algorithm for path planning. The A* algorithm is one of the best known path planning algorithms and can be applied to metric or topological configuration space. The algorithm combines heuristic search and shortest path-based search to evaluate the value of each cell in the configuration space. The cost function formula of the A* algorithm is as follows:

[0102] f(v)=g(v)+h(v)

[0103] Among them, g(v) represents the cost from the starting point to the current position of the drone, h(v) represents the cost from the current point to the target position of the drone, and the shortest path from the starting position to the target point of the drone in the whole process is obtained by selecting the minimum cost and f(v);

[0104] In the present invention, during the path planning process, the intelligent agent will consider circular obstacles at any position. Different from the previous practice of treating the obstacle area as a square, obstacle avoidance is achieved through collision detection between the movement direction and the circular obstacle. Replacing the obstacle with a circle can be closer to the real environment, simplify the collision detection calculation, improve the obstacle avoidance accuracy and path planning efficiency, avoid the redundant interference of the square boundary, and thus achieve a more flexible and efficient obstacle avoidance capability.

[0105] Preferably, the planning algorithm will consider the overload constraint of the high-speed aircraft and the smoothness of the flight trajectory; after completing the initial planning, the nodes at both ends of several intervals are connected as the local target points of the high-speed aircraft cluster; the cluster will autonomously complete the flight mission from one target point to the next target point;

[0106] Preferably, during the path planning process of the drone, each agent can choose 16 different movement modes, such as Figure 2 Improved A-star planning algorithm agent movement direction as shown, the agent's action space is expressed as follows using the formula:

[0107] a=[a x ,a y ]

[0108] a x ,a y ∈[-2,-1,0,1,2]

[0109] a x +a y ≥2,a x ×a y ≠1

[0110] Among them, a x ,a y Represent the actions of the agent in the x and y directions respectively;

[0111] During the path planning process, if the agent collides with an obstacle, such as Figure 3As shown in the scenario where the agent encounters an obstacle, it is necessary to remove the path points in the direction of movement and stop further exploration of the collision point position; in order to check whether the path intersects with the obstacle, it is necessary to calculate the local path line, that is, calculate the line connecting the starting point at the previous moment and the current target point. If the projection of the line does not contain the obstacle, the path should satisfy the following constraints:

[0112] ||(x next ,y next )-(x obs ,y obs )||2>r obs

[0113] Where (x next ,y next ) represents the local target point of the current path, (x obs ,y obs ) represents the coordinates of the obstacle closest to the current position of the drone, and the distance between the two coordinates needs to be greater than the obstacle radius r obs ;

[0114] If the projection of the local path line contains obstacles, the path satisfies the following constraints:

[0115] d((x obs ,y obs )-line((x current ,y current ),(x next ,y next )))<r obs

[0116] Among them, line((x current ,y current ),(x next ,y next )) represents the local path line, the path line from the current position to the next local target point, (x obs ,y obs ) represents the coordinates of the obstacle position. If the distance from the obstacle position to the path line is less than the preset obstacle radius, it is considered that the path line at this time intersects with the obstacle cylinder. If the drone cluster travels along this route, it will collide with the obstacle.

[0117] Preferably, since the research object of the present invention is a high-speed aircraft cluster, it is necessary to constrain the cluster volume in the A* algorithm path planning. Assume that the radius of the aircraft cluster is r formation To ensure that the cluster does not collide with obstacles due to the cluster volume when tracking the shortest path, the virtual obstacle radius is set here. The calculation formula is as follows:

[0118] r=robs +r formation

[0119] By setting the virtual obstacle radius using the above formula, it can be ensured that the cluster will not collide with any obstacles when driving along the planned path. This part is different from the traditional single UAV path planning;

[0120] S400, the planning algorithm completes the preliminary planning, generates the shortest path to the target point in the map, and selects the local target point on the shortest path with a specified search step size;

[0121] The shortest path obtained by the above planning is intended to provide the cluster with dynamic and efficient local target point information, so as to guide the cluster to perform accurate path following tasks in a complex obstacle environment. Different from the traditional global path planning, this method not only focuses on the determination of the global optimal path, but also guides the aircraft cluster to adjust the flight direction in real time by selecting local target points to ensure the actual feasibility and dynamic adaptability of the path. These local target points are selected from the global path according to the pre-set step size and planning results, and gradually provide directional guidance information for the cluster.

[0122] In actual execution, swarm aircraft do not strictly follow the global path points generated by the upper-level path planning algorithm to fly one by one, because under the conditions of numerous obstacles and dynamically changing environments, strictly following the global path may lead to problems such as an unsmooth path and untimely obstacle avoidance, and even affect the overall flight efficiency and safety of the swarm. By selecting local target points, the swarm can achieve more flexible path following in a local range, and dynamically adjust the flight direction and speed according to real-time sensor data, so as to effectively avoid obstacles and move forward in the optimal direction.

[0123] (x t ,y t ),t∈T

[0124] (x kt ,y kt ),kt∈T

[0125] (k-1)t

[0126] Among them, (x t ,y t ) represents all the path points, which constitute the shortest path obtained by the upper-level planning. (x kt ,y kt ) represents the local path target point obtained according to the (k-1)t step size;

[0127] S500, realizes local path tracking control and dynamic obstacle avoidance control of UAV through SAC reinforcement learning algorithm;

[0128] The optimization problem defined in step S100 of the present invention can be described using a Markov decision process, so as to solve the problem using a reinforcement learning framework; in the reinforcement learning framework, if a certain behavior strategy of the agent leads to a positive reward from the environment to the agent, the tendency of the agent to adopt this behavior strategy in the future will be strengthened; if the learning adopted by the agent to adapt to the environment has the following characteristics, it is called reinforcement learning: the agent is not static and passively waiting, but actively explores the environment;

[0129] In reinforcement learning, the agent selects an action a to act on the environment. The environment changes after receiving the action, and generates a reinforcement signal r to feed back to the agent. The agent then selects the next action based on the reinforcement signal and the current state of the environment. The principle of selection is to increase the probability of receiving a positive reward. The selected action not only affects the immediate reinforcement value, but also affects the state of the environment at the next moment and the final reinforcement value.

[0130] In the reinforcement learning framework of the present invention, each agent learns a strategy by interacting with the environment; the goal of the aircraft is to obtain the maximum long-term expected reward sum in a single round:

[0131]

[0132] In the above formula, γ∈[0,1] is a discount factor. When γ→1, the learning strategy is allowed to pay more attention to long-term rewards. t+k Represents the reward obtained by the agent at step t+k;

[0133] In the present invention, all drones in the cluster share the same network parameters during training and application, and make decisions by observing the status of other aircraft and the environment;

[0134] The observation space of a high-speed aircraft includes its own state, the state of surrounding aircraft, obstacle information, and target state; the state of the aircraft itself includes the aircraft's own position information [x, y, z], the aircraft's flight speed information [v x ,v y ,v z ]; the surrounding aircraft status information includes the relative position relationship [Δx, Δy, Δz] and relative speed relationship [Δv x ,Δv y ,Δv z ]; obstacle information is also obtained through the observation space, which includes the relative position information between the obstacle and the aircraft [Δx o ,Δy o], and since the present invention regards the obstacle as a cylinder rather than a cube, the relative position information is a two-dimensional vector; finally, the observation space also observes the relative position information [Δx t ,Δy t ,Δz t ];

[0135] The action space of a high-speed aircraft is composed of the normal overload of the aircraft, and the normal overload includes two directions [n y ,n z ], the overload size output by the neural network needs to be limited to ensure that the actual overload of the aircraft is less than its maximum allowed overload value;

[0136] The design part of the reward function of the present invention analyzes the physical constraints and mission requirements of the high-speed aircraft and establishes a reward function suitable for aircraft path planning and control; the core goal of the reward function is to guide the high-speed aircraft to fly efficiently to the predetermined target area or tracking target; in the design process, the key factor of the relative speed between the aircraft and the target is considered as an important indicator to measure the flight effect of the aircraft;

[0137] Specifically, a target flight reward mechanism is designed to motivate the aircraft to continuously adjust its flight direction and accelerate towards the target during flight. The greater the speed of the aircraft relative to the target, especially the higher the component of the speed in the target direction, the greater the reward value obtained. This design ensures that the aircraft can always keep dynamic tracking of the target while flying at high speed, avoiding deviation from the course or wasting speed.

[0138] r t =k t ×||Δv t ||×sgn(Δv t )

[0139] In this equation, k t is the weight coefficient of the reward, ||Δv t || represents the absolute value of the relative speed between the high-speed aircraft and the target; when the aircraft moves toward the target point, sgn(Δv t )=1, when the aircraft moves away from the target point, sgn(Δv t )=-1;

[0140] In order to maintain formation flight stability within a high-speed aircraft cluster, each aircraft calculates a corresponding reward based on its position difference relative to the surrounding aircraft. The purpose of this part of the reward function is to punish scenarios that deviate from the initial formation or the desired formation. The formula is as follows:

[0141] r f=k f ×||ΔR||

[0142] In the above expression, k f is the weight coefficient of the reward, ||ΔR|| represents the average deviation of the distance between a high-speed aircraft and several surrounding aircraft. The goal is to achieve a reward function that remains zero while the high-speed aircraft perfectly maintains its original formation;

[0143] In order to ensure that the high-speed aircraft avoids obstacles and maintains a safe distance from them during the learning process, the present invention designs a reward to punish the aircraft when it approaches the obstacle area. The following is the method for formulating this reward:

[0144] r o =k o ×||Δr o || 2

[0145] In this formula, k o is the weight coefficient of the reward, ||Δr o || represents the relative distance between the high-speed aircraft and the obstacle area. The value of this reward item is proportional to the square of the relative distance. The reward function is designed so that the penalty increases sharply when the aircraft approaches the obstacle area; when the aircraft moves away from the obstacle, the penalty decreases rapidly.

[0146] By summing up the above rewards, we can get the total reward function:

[0147] r=r t +r f +r o

[0148] Simulation test

[0149] This paper preliminarily tests the path planning ability of the improved A* algorithm in various environments, and sets different initial positions and targets for simulation in the same environment, such as Figure 4 The path planning results of the constrained A-star algorithm are shown in the figure. The gray area represents the virtual obstacle area considering the volume constraint of the aircraft cluster. The red and green dots represent the starting and ending points respectively. The blue line represents the initial planned path. The path planning simulation results show that the invented path planning algorithm can realize the preliminary planning of the global path under the premise of considering the cluster volume constraint.

[0150] Preferably, the cluster path planning method based on the improved A-star and reinforcement learning algorithm proposed in the present invention is compared with other methods; this part of the experiment compares the method proposed in the present invention, the unimproved A-star and reinforcement learning method, and the reinforcement learning method alone, and these methods will be tested in several identical simulation environments to evaluate their respective flight performances; this part will illustrate the shortcomings of traditional reinforcement learning in formation control, and will demonstrate how to introduce the A* algorithm for upper-level planning to improve formation control performance;

[0151] The simulation results of the three methods are shown in Figure 5 As shown in the comparison chart of simulation results of various methods, in the first A-star and reinforcement learning algorithm considering constraints proposed by the present invention, a high-speed aircraft cluster can safely fly through the obstacle area and reach the designated target point. In this process, the cluster formation is well maintained, and there is no internal collision of aircraft in the cluster and collision with obstacles in the environment due to formation changes, achieving the expected cluster path planning method effect; the second method is the A-star and reinforcement learning method. The difference between this method and the first method is that the volume limitation of the aircraft cluster is not considered. It can be seen from the simulation results of the second method that its formation maintenance effect is poor. Two out of three experiments failed, and the expected cluster path planning effect was not achieved; the last method only uses a deep reinforcement learning algorithm. Since high-speed aircraft cannot predict future obstacle areas and there is no upper-level path planning, it is easy to enter the obstacle area during local obstacle avoidance, resulting in mission failure.

[0152] Specific implementation scheme 2: The present invention provides a cluster path planning system based on improved A-star algorithm and reinforcement learning. The system has a program module corresponding to the above steps, and executes the steps in the above cluster path planning method based on improved A-star algorithm and reinforcement learning during operation.

[0153] The other combinations and connection relationships of this embodiment are the same as those of the first embodiment.

[0154] Specific implementation scheme three: The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of a cluster path planning method based on an improved A-star algorithm and reinforcement learning when called by a processor.

[0155] The other combinations and connection relationships of this embodiment are the same as those of the first embodiment.

[0156] The other combinations and connection relationships of this embodiment are the same as those of the first embodiment.

[0157] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. A cluster path planning method based on improved A-star algorithm and reinforcement learning, characterized in that: The following steps are involved: S100, UAV modeling and environment modeling, including kinematic modeling of high-speed UAVs and modeling of mission optimization problems; S200, a formation control framework using a hierarchical controller, the upper layer uses an improved A* algorithm to control the situation in the obstacle-dense area, and considers the volume of the drone cluster and the impact range of obstacles, the overload constraint of high-speed aircraft and the smoothness of the flight trajectory to build a heuristic function; S300, after the planning algorithm completes the preliminary planning, the shortest path to the target point is generated in the map, and the local target point on the shortest path is selected with a specified search step size; S400, the lower layer realizes the local path tracking control and dynamic obstacle avoidance control of the UAV through the SAC reinforcement learning algorithm, including the design of observation space, action space and reward function.

2. The cluster path planning method based on improved A-star algorithm and reinforcement learning according to claim 1, characterized in that: In step S100, it includes: S110. In order to accurately describe the flight state of unpowered high-speed drones, kinematic modeling is required. The state variables of the drone cluster are defined as: [VγψR E [th] T Where V represents the flight speed of the aircraft; γ represents the flight path angle; ψ represents the flight heading angle; R E represents the distance from the center of the earth; θ represents the longitude of the earth; φ represents the latitude of the earth; According to the above state variables, the UAV motion equations are listed as follows: Among them, n x represents the acceleration along the flight direction, g represents the acceleration due to gravity; n y Indicates the acceleration of the aircraft perpendicular to the velocity direction; n z Indicates the lateral acceleration of the aircraft; S120. After completing the modeling of the UAV, it is also necessary to model the environmental constraints. The goal of the aircraft cluster formation control is to enable the cluster to avoid obstacles and reach the target point while maintaining the formation. This task is modeled as an optimization problem: 1) The overload constraint of the high-speed aircraft is met, and the actual overload n of the aircraft is less than the maximum overload constraint n max ; 2) Maintain formation flight to ensure that the actual state u of the high-speed aircraft in the formation follows the expected motion state u d ; 3) Avoid obstacle areas and ensure that the minimum distance between the high-speed aircraft and the nearest obstacle is not less than the set safety distance; 4) The UAV cluster flies to the designated target position, ensuring that the distance difference between the high-speed aircraft position and the target position is close to 0; The above optimization problem is expressed using the formula as follows: Among them, α represents the weight coefficient of the formation optimization objective, u i represents the state information of the i-th UAV in the UAV cluster, β represents the weight coefficient of the distance constraint between the UAV and the target point; d safe represents the safe distance between the high-speed UAV cluster and the obstacle area; F1 represents the UAV expected state decision function, c iN Represents the weight coefficient of different drone states, and F2 represents the controller of the aircraft itself.

3. A cluster path planning method based on improved A-star algorithm and reinforcement learning according to claim 2, characterized in that: In the path planning process of step S200, the overload constraint of the high-speed aircraft and the smoothness of the flight trajectory are considered. After completing the initial planning based on the A* algorithm, the nodes at both ends of several intervals are connected as the local target points of the high-speed aircraft cluster; the cluster will autonomously complete the flight mission from one target point to the next target point; The action space of the agent is expressed as follows: a=[a x ,a y ] a x ,a y ∈[-2,-1,0,1,2] a x +a y ≥2,a x ×a y ≠1 Among them, a x ,a y Represents the movement in the x direction and y direction respectively; Set the obstacle to a circular obstacle. If the agent collides with the obstacle, remove the waypoint in the direction of movement and stop further exploration of the collision point. In order to check whether the path intersects with obstacles, a line is drawn between the starting point at the last moment and the current target point. If the projection of the line does not contain obstacles, the path should satisfy the following constraints: ||(x next ,y next )-(x obs ,y obs )||2>r obs Among them, (x next ,y next ) represents the local target point of the current path, (x obs ,y obs ) represents the coordinates of the obstacle closest to the current position of the drone, and the distance between the two coordinates needs to be greater than the obstacle radius r obs ; If the projection of the local path line contains obstacles, the path satisfies the following constraints: d((x obs ,y obs )-line((x current ,y current ),(x next ,y next )))<r obs Among them, line((x current ,y current ),(x next ,y next )) represents the local path line, the path line from the current position to the next local target point, (x obs ,y obs ) represents the obstacle position coordinates; If the distance from the obstacle to the path line is less than the preset obstacle radius, the path line is considered to intersect with the obstacle cylinder. If the drone cluster follows this route, it will collide with the obstacle. Constrain the cluster volume, assuming that the radius of the aircraft cluster is r formation , the radius of the virtual obstacle is: r＝r obs +r formation 。 4. The cluster path planning method based on improved A-star algorithm and reinforcement learning according to claim 3, characterized in that: In step S300, a local target point is selected to achieve flexible path following of the cluster within a local range, and the flight direction and speed are dynamically adjusted according to real-time sensor data, so as to effectively avoid obstacles; (x t ,and t ),t∈T (x kt ,and kt ),kt∈T (k-1)t Among them, (x t ,y t ) represents all the path points, which constitute the shortest path obtained by the upper-level planning. (x kt ,y kt ) represents the local path target point obtained according to the (k-1)t step size.

5. The cluster path planning method based on improved A-star algorithm and reinforcement learning according to claim 4, characterized in that: In step S400, the input of the SAC reinforcement learning algorithm is the observation data collected in the observation space and the output is the action that the drone should perform selected from the action space. Each agent learns a strategy by interacting with the environment, and the goal of the robot is to obtain the maximum long-term expected reward sum in a single round: Where γ∈[0,1] is the discount factor. When γ→1, the learning strategy is allowed to pay more attention to long-term rewards. t+k Represents the reward obtained by the agent at step t+k; The observation space of a high-speed aircraft includes its own state, the state of surrounding aircraft, obstacle information, and target state; the state of the aircraft itself includes its own position information [x, y, z], the flight speed information [v x ,v y ,v z ]; the surrounding aircraft status information includes the relative position relationship [Δx, Δy, Δz] and relative speed relationship [Δv x ,Δv y ,Δv z ]; obstacle information includes the relative position information between the obstacle and the aircraft [Δx o ,Δy o ]; and obtain the relative position information between the target and the aircraft [Δx t ,Δy t ,Δz t ]; The action space of a high-speed aircraft is composed of the normal overload of the aircraft, which includes two directions [n y ,n z ], the overload size output by the neural network needs to be limited to ensure that the actual overload of the aircraft is less than its maximum allowable overload value.

6. The cluster path planning method based on improved A-star algorithm and reinforcement learning according to claim 5, characterized in that: By designing reward functions for high-speed aircraft path planning and control, it is used to guide the high-speed aircraft to fly efficiently to the predetermined target area or tracking target, including: r t =k t ×‖Δv t ‖×sgn(Δv t ) Among them, k t is the weight coefficient of the reward, ‖Δv t || represents the absolute value of the relative speed between the high-speed aircraft and the target; when the aircraft moves toward the target point, sgn(Δv t )=1, when the aircraft moves away from the target point, sgn(Δv t )=-1; r f =k f ×||ΔR|| Among them, k f is the weight coefficient of the reward, ||ΔR|| represents the average deviation of the distance between a high-speed aircraft and several surrounding aircraft. The goal is to achieve a reward function that remains zero while the high-speed aircraft perfectly maintains its original formation; r o =k o ×||Δr o || 2 Among them, k o is the weight coefficient of the reward, ||Δr o || represents the relative distance between the high-speed aircraft and the obstacle area. The value of this reward item is proportional to the square of the relative distance. The reward function is designed so that the penalty increases sharply when the aircraft approaches the obstacle area; when the aircraft moves away from the obstacle, the penalty decreases rapidly. By summing up the above rewards, we can get the total reward function: r＝r t +r f +r o 。 7. A cluster path planning system based on improved A-star algorithm and reinforcement learning, characterized by: The system has a program module corresponding to the steps of any one of claims 1 to 6, and executes the steps of the cluster path planning method based on the improved A-star algorithm and reinforcement learning during operation.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the cluster path planning method based on improved A-star algorithm and reinforcement learning described in any one of claims 1 to 6 when called by a processor.

Images