A method and system for cooperative pursuit of target track planning of a UAV cluster system

Abstract

The application discloses a UAV cluster system cooperative pursuit target path planning method and system, relates to the technical field of path planning, respectively takes the observation vector of each UAV in the UAV cluster system at the current time as input, determines the first action constant and the second action constant corresponding to each UAV by using a deep reinforcement learning network, and further obtains the observation vector of each UAV at the next time by calculating the heading angle of each UAV at the next time by using an artificial potential field method, and the above steps are cycled until the iteration termination condition is reached, so that the path planning of each UAV in the cooperative pursuit target process is completed. In the path planning process of the application, each UAV solves its own path separately, realizes effective decentralization, and considers the influence of surrounding obstacles in the path planning process, so that the decentralized path planning in a complex environment is realized.

Description

Technical Field

[0001] This invention relates to the field of trajectory planning technology, and in particular to a method and system for collaborative target pursuit trajectory planning in a fixed-wing unmanned aerial vehicle (UAV) swarm system. Background Technology

[0002] The application of unmanned aerial vehicle (UAV) swarm systems in areas such as regional coverage has attracted widespread attention. Inter-UAV swarm system adversarial operations, by decomposing the task, ultimately evolve into multiple UAVs engaging the same target, thus simplifying the task to the pursuit of a given target. The trajectory planning problem for UAV swarm systems pursuing a given target is currently a hot topic in the field. Many methods exist for solving this problem, such as potential field-based algorithms, mathematical programming methods, heuristic algorithms, and reinforcement learning-based methods. Reinforcement learning-based methods can handle more complex target pursuit scenarios compared to traditional methods. However, many current reinforcement learning-based trajectory planning methods fail to achieve effective decentralization, which can easily lead to task failure. Furthermore, the influence of surrounding obstacles on the UAVs during task execution is also a crucial consideration. Therefore, researching decentralized collaborative target pursuit trajectory planning techniques in complex environments has practical engineering significance. Summary of the Invention

[0003] The purpose of this invention is to provide a method and system for collaborative target pursuit trajectory planning in a drone swarm system, which can achieve decentralized trajectory planning in complex environments.

[0004] To achieve the above objectives, the present invention provides the following solution:

[0005] A method for collaborative target pursuit trajectory planning in an unmanned aerial vehicle (UAV) swarm system, the trajectory planning method comprising:

[0006] Obtain the observation vector of each drone in the drone swarm system at the current moment; the observation vector includes the distance of the drone to the nearest designated obstacle and the azimuth angle relative to the designated obstacle, the distance of the drone to the target and the azimuth angle relative to the target, and the distance of the drone to each other drone and the azimuth angle relative to each other drone;

[0007] Using the observation vector of each UAV at the current moment as input, a deep reinforcement learning network is used to determine the first action constant and the second action constant for each UAV.

[0008] For each UAV, using the first action constant and the second action constant as inputs, the attractive force of the UAV pointing towards the target, the repulsive force of the designated obstacle on the UAV, and the resultant force of the interaction between the UAV and the other UAVs are calculated using the artificial potential field method; the heading angle of the UAV at the next moment is determined based on the attractive force, the repulsive force, and the resultant force; the UAV is controlled to fly according to the heading angle of the UAV at the next moment to obtain the observation vector of each UAV in the UAV swarm system at the next moment;

[0009] Determine if the iteration termination condition has been met; if yes, end the iteration; otherwise, use the observation vector of each drone at the next time step as the observation vector of each drone at the current time step in the next iteration, and return to the step of "using the observation vector of each drone at the current time step as input, and using a deep reinforcement learning network to determine the first action constant and the second action constant corresponding to each drone".

[0010] A trajectory planning system for collaborative target pursuit in a drone swarm system, the trajectory planning system comprising:

[0011] The acquisition module is used to acquire the observation vector of each drone in the drone swarm system at the current moment; the observation vector includes the distance of the drone to the nearest designated obstacle and the azimuth angle relative to the designated obstacle, the distance of the drone to the target and the azimuth angle relative to the target, and the distance of the drone to each other drone and the azimuth angle relative to each other drone;

[0012] The action selection module is used to determine the first action constant and the second action constant for each UAV by taking the observation vector of each UAV at the current moment as input and using a deep reinforcement learning network.

[0013] The prediction module is used to calculate, for each UAV, the attractive force of the UAV pointing towards the target, the repulsive force of the designated obstacle on the UAV, and the resultant force of the interaction between the UAV and the other UAVs using an artificial potential field method, with the first action constant and the second action constant as input; determine the heading angle of the UAV at the next moment based on the attractive force, the repulsive force, and the resultant force; control the UAV to fly according to the heading angle of the UAV at the next moment, and obtain the observation vector of each UAV in the UAV swarm system at the next moment;

[0014] The iteration module is used to determine whether the iteration termination condition has been met; if yes, the iteration ends; if no, the observation vector of each UAV at the next moment is used as the observation vector of each UAV at the current moment in the next iteration, and the steps of "using the observation vector of each UAV at the current moment as input, and using a deep reinforcement learning network to determine the first action constant and the second action constant corresponding to each UAV" are returned.

[0015] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0016] This invention provides a method and system for collaborative target pursuit trajectory planning in a drone swarm system. Taking the observation vector of each drone in the swarm system at the current moment as input, a deep reinforcement learning network is used to determine the first and second action constants for each drone. Then, an artificial potential field method is used to calculate the attractive force of each drone pointing towards the target, the repulsive force of a designated obstacle on the drone, and the resultant force of the interaction between the drone and other drones to determine the heading angle of each drone at the next moment. This further yields the observation vector of each drone in the swarm system at the next moment. The above steps are repeated until the iteration termination condition is met, thereby completing the trajectory planning for each drone in the collaborative target pursuit process. In the trajectory planning process of this invention, each drone solves its own trajectory independently, achieving effective decentralization. Furthermore, the influence of surrounding obstacles is considered during the trajectory planning process, thus achieving decentralized trajectory planning in complex environments. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the design framework of the trajectory planning method provided in Embodiment 1 of the present invention;

[0019] Figure 2 This is a schematic diagram of the algorithm framework of the DDPG-based cooperative target pursuit algorithm provided in Embodiment 1 of the present invention;

[0020] Figure 3 This is a flowchart of the trajectory planning method provided in Embodiment 1 of the present invention;

[0021] Figure 4 This is a schematic diagram of the DDPG network structure provided in Embodiment 1 of the present invention;

[0022] Figure 5 This is a schematic diagram of the wall following principle provided in Embodiment 1 of the present invention;

[0023] Figure 6 This is a schematic diagram of the training reward convergence process provided in Embodiment 1 of the present invention;

[0024] Figure 7 This is a schematic diagram of the cooperative target pursuit process 1 provided in Embodiment 1 of the present invention;

[0025] Figure 8 This is a schematic diagram of the cooperative target pursuit process 2 provided in Embodiment 1 of the present invention;

[0026] Figure 9 This is a system block diagram of the trajectory planning system provided in Embodiment 2 of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] The purpose of this invention is to provide a method and system for collaborative target pursuit trajectory planning in a drone swarm system, which can achieve decentralized trajectory planning in complex environments.

[0029] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] Example 1:

[0031] like Figure 1As shown, the design framework of the UAV swarm system cooperative target pursuit trajectory planning method in this embodiment includes: (1) constructing the UAV swarm system cooperative target pursuit trajectory planning problem, determining the UAV motion equation and the conditions for successful target pursuit; (2) determining the target attraction, obstacle repulsion and interaction forces between individual UAVs in the artificial potential field, and introducing the wall following principle to solve the local minima problem of the artificial potential field method, thus obtaining an improved artificial potential field method; (3) encoding and feature extraction of the observation vectors observed by the UAVs; (4) converting the process of individual UAVs pursuing the target into a Markov decision process, introducing a deep reinforcement learning network framework for solving the problem, and designing the reward function; (5) combining steps (2) to (4) to propose a decentralized cooperative target pursuit algorithm to solve the UAV swarm system cooperative target pursuit trajectory planning problem; (6) using the designed decentralized cooperative target pursuit algorithm to solve the UAV swarm system cooperative target pursuit trajectory planning problem, and verifying the effectiveness of the above algorithm using simulation examples.

[0032] The problem of collaborative target pursuit trajectory planning in a drone swarm system refers to the pursuit of a faster target by multiple isomorphic, slower-moving pursuers (fixed-wing drones) in an environment with obstacles. The multiple pursuers form a group and move together to restrict the target's movement space, thereby enabling one of the pursuers to capture the target in a shorter time.

[0033] The equations of motion for the UAV, that is, the kinematic model of the UAV as a two-dimensional planar mass, are as follows:

[0034]

[0035] Where x and y represent the drone's position coordinates, and v and ψ represent the drone's velocity and heading angle, respectively. Given the drone's velocity and heading angle, the drone's position coordinates can be calculated using the above formula.

[0036] The condition for successfully pursuing a target is: the distance between the target and any pursuer is less than a preset value d. cap This enables the drone swarm system to collaboratively pursue targets.

[0037] The drone swarm system's collaborative target pursuit trajectory planning problem in this embodiment also needs to meet the following conditions:

[0038] (1) The flight speed v of each UAV in the UAV swarm system pursuer All are identical and remain constant during trajectory planning; the target's flight speed v evader Maintain constancy during the trajectory planning process.

[0039] (2) The target’s location information can be acquired by all UAVs, and each UAV can acquire the location information of all other UAVs.

[0040] The improved artificial potential field method in step (2) is used to calculate the resultant force on the UAV, thereby determining the heading angle ψ of the UAV at the next moment. The two parameters (λ, η) are designed as the action space of the deep reinforcement learning network in step (4), and are determined according to the deep reinforcement learning network in step (4). Therefore, the algorithm framework of the decentralized cooperative target pursuit algorithm based on DDPG is as follows: Figure 2 As shown, firstly, the observation vectors observed by the UAV are extracted using step (3). The observation vectors are then processed in a high dimension through a designed fully connected layer, and a fixed-length feature vector is output to the DDPG action network (i.e., deep reinforcement learning network) in step (4). The deep reinforcement learning network continuously optimizes its parameters during the interaction with the environment and learning from the experience pool, and outputs the parameters (λ, η) to step (2). The resultant force on each UAV in the UAV swarm system is calculated, thereby determining the forward direction of each UAV at the next moment and obtaining the observation vector of each UAV at the next moment. The above steps are repeated until the conditions for successfully chasing the target are met, thus completing the trajectory planning process of each UAV in the process of collaboratively chasing the target.

[0041] Based on the above design framework, this embodiment provides a method for collaborative target pursuit trajectory planning in a drone swarm system, such as... Figure 3 As shown, the trajectory planning method includes:

[0042] S1: Obtain the observation vector of each drone in the drone swarm system at the current moment; the observation vector includes the distance of the drone to the nearest designated obstacle and the azimuth angle relative to the designated obstacle, the distance of the drone to the target and the azimuth angle relative to the target, and the distance of the drone to each other drone and the azimuth angle relative to each other drone;

[0043] The observation vector of each UAV consists of two parts: distance and orientation angle. The observation vector of the i-th UAV (i.e., UAV i) can be written in the following form:

[0044] o i =(d o,i ,ψ o,i ,d e,i ,ψ e,i ,d 1,i ,ψ 1,i ,d 2,i ,ψ 2,i ,...,d M,i ,ψ M,i);

[0045] Among them, o i d is the observation vector of UAV i; o,i ψ is the distance from drone i to the nearest specified obstacle; o,i , where is the azimuth angle of UAV i relative to the specified obstacle closest to UAV i; in this embodiment, the obstacle is a static obstacle, and each UAV can obtain the position information of each obstacle in advance, so the distance d can be calculated based on the position coordinates of UAV i and the position coordinates of each obstacle. o,i and direction angle ψ o,i ,;d e,i Let ψ be the distance from UAV i to the target; e,i Let be the heading angle of UAV i relative to the target; since UAV i can obtain the target's position information, this distance d can be calculated based on the position coordinates of UAV i and the target's position coordinates. e,i and direction angle ψ e,i M represents the number of drones in the drone swarm system other than drone i; d 1,i ,d 2,i ,...,d M,i Let ψ be the distance from drone i to every other drone; 1,i ,...,ψ M,i Let be the heading angle of UAV i relative to each other UAV; since UAV i can obtain the position information of each other UAV, the distance d can be calculated based on the position coordinates of UAV i and the position coordinates of each other UAV. 1,i ,d 2,i ,...,d M,i and direction angle ψ 1,i ,...,ψ M,i .

[0046] Considering that the sheer number of drones in a drone swarm system can cause a "dimensionality explosion" problem in reinforcement learning, thus affecting the target tracking effect, this embodiment extracts features from the observation vectors observed by the drones. Specifically, a neural network including a fully connected layer is used to extract high-order information from the observed vectors and output a fixed-length feature vector to the deep reinforcement learning network. As the number of other drones M changes, the dimension of the observation vector of drone i will also change. Therefore, the dimension of the fully connected layer is selected as (2M+4)×128, and a column vector (feature vector) of fixed length 128 is output to the deep reinforcement learning network for learning and optimization to determine the next action of the drone.

[0047] Specifically, the feature extraction process of the observation vector in this embodiment may include: for each UAV, performing feature extraction on the observation vector of the UAV at the current moment to obtain the UAV's feature vector, and using the UAV's feature vector as the new observation vector of the UAV at the current moment, and then executing the subsequent S2. The feature extraction of the UAV's observation vector at the current moment may include: inputting the UAV's observation vector at the current moment into a neural network for feature extraction, the neural network including a fully connected layer.

[0048] S2: Using the observation vector of each UAV at the current moment as input, a deep reinforcement learning network is used to determine the first action constant and the second action constant for each UAV.

[0049] The process of a single drone collaboratively pursuing a target can be constructed as a Markov decision process, which contains {S, A, R, P, γ}, where s t ∈S,a t ∈A,r t ∈R represent the state at time t (in this embodiment, the state is the observation vector observed by the UAV), action, and reward, respectively, and P represents the state s from time t. t Transition to state s at time t+1 t+1 Take action a at the time t The unknown probability, γ, represents the discount factor, and the action taken is determined by the neural network policy layer, i.e., a. t ~π θ (s t The cooperative target pursuit trajectory planning problem is a continuous action space problem. Therefore, value-based methods that solve the state-action value function are no longer applicable. Policy-based methods, which select the next action based on probability, are suitable for controlling the motion of UAVs. Therefore, the deep reinforcement learning network in this embodiment adopts the Deep Deterministic Policy Gradient (DDPG) framework, such as... Figure 4 As shown, the system includes an environment, an experience pool, an Actor network, and a Critic network. The environment and experience pool are used to generate and store experience, respectively. During interaction with the environment, the drone gains experience and stores it in the experience pool for the neural network to learn from. The Actor network is used to determine the probability of the drone choosing various actions. When the drone interacts with the environment, it will choose the next action based on the action probabilities provided by the Actor network. The Critic network will evaluate the action chosen by the drone based on the state of the environment. Then, the Actor network will modify the probability of choosing the action based on the evaluation results.

[0050] Figure 4In this system, each Actor network and Critic network has a two-layer network, and the functions of the four networks are as follows:

[0051] (1) Actor Current Network: Responsible for iteratively updating the policy network parameters θ, and responsible for updating the parameters based on the current state s. t Select the current action a t Used to interact with the environment to generate s t+1 ,r t ;

[0052] (2) Actor Target Network: Responsible for determining the next state s sampled from the experience pool. t+1 Select the optimal next action a t+1 Network parameters θ' are periodically copied from θ;

[0053] (3) Critic current network: responsible for the iterative update of the value network parameters w, and responsible for calculating the current Q value Q(s). t ,a t ,w) and the target Q value y=r t +γQ'(s t+1 ,a t+1 ,w');

[0054] (4) Critic target network: responsible for calculating Q'(s) in the target Q value. t+1 ,a t+1 The network parameter w' is copied from w periodically.

[0055] Based on the structure of the deep reinforcement learning network described above, for each UAV, this embodiment directly uses the observation vector of the UAV at the current moment as the state and inputs it into the deep reinforcement learning network to determine the corresponding action of the UAV. The action includes the first action constant and the second action constant in the artificial potential field method, thereby determining the first action constant and the second action constant corresponding to each UAV.

[0056] S3: For each UAV, using the first action constant and the second action constant as input, calculate the attractive force of the UAV pointing towards the target, the repulsive force of the designated obstacle on the UAV, and the resultant force between the UAV and the other UAVs using the artificial potential field method; determine the heading angle of the UAV at the next moment based on the attractive force, the repulsive force, and the resultant force; control the UAV to fly according to the heading angle of the UAV at the next moment to obtain the observation vector of each UAV in the UAV swarm system at the next moment;

[0057] To fully utilize potential field-related methods in this embodiment to achieve obstacle avoidance, target pursuit, and collision avoidance between individuals, it is necessary to first determine the calculation formulas (i.e., expressions) for attractive force, repulsive force, and inter-individual interaction force in the artificial potential field method.

[0058] The formula for calculating the attractive force of the drone pointing at the target (in this embodiment, a unit vector pointing at the target) is as follows:

[0059]

[0060] Among them, F a,i The attractive force of drone i pointing towards the target; x e x represents the target's position coordinates; i Let be the position coordinates of UAV i.

[0061] The formula for calculating the repulsive force exerted on drone i by a specified obstacle (i.e., the obstacle closest to drone i) is as follows:

[0062]

[0063] Among them, F r,i η is the repulsive force exerted on drone i by the designated obstacle closest to drone i; η is the first action constant, which is determined by the deep reinforcement learning network; ρ0 is the radius of the obstacle's force range; x o,i x represents the coordinates of the nearest specified obstacle to drone i; i Let be the position coordinates of UAV i.

[0064] The formula for calculating the resultant force of the interaction between drones and other drones (i.e., the resultant force of the interaction between individual drones) is as follows:

[0065]

[0066] Among them, F in,i Let be the resultant force between drone i and other drones; P(i) be the set of neighboring drones of drone i, including all other drones in the drone swarm system except drone i; λ is the second action constant, which is determined by the deep reinforcement learning network; x j x represents the position coordinates of other UAV j; i Let be the position coordinates of UAV i.

[0067] Since the above position coordinates are all two-dimensional coordinates, including the x and y directions, the calculated attractive force, repulsive force, and resultant force are all vectors with direction.

[0068] In the three calculation formulas of the artificial potential field method, the two action constants (λ, η) are designed as the continuous action space of the deep reinforcement learning network, determined by the deep reinforcement learning network based on the observation vector of the UAV. Specifically, the parameter η in the repulsive force expression can adjust the effective range of the obstacle, and the parameter λ in the resultant force expression between UAVs can adjust the closeness between them. A larger λ can form a larger encirclement, compressing the target's escape space and facilitating the completion of the target pursuit mission.

[0069] This embodiment can calculate the resultant force of attraction, repulsion, and action on each drone based on the three calculation formulas mentioned above. Therefore, the resultant force F acting on the i-th drone... i The expression is: F i =F a,i +F r,i +F in,i .

[0070] To avoid local minima in traditional artificial potential field methods, this embodiment introduces a wall-following principle to determine the UAV's direction of travel near obstacles, thus improving the artificial potential field method and obtaining an improved artificial potential field method. Figure 5 As shown, when the target attraction F a,i and the repulsive force F of the obstacle r,i The resultant force F ar,i The direction of the resultant force and the target attraction F a,i When the angle between the directions is greater than 90°, the next forward direction of the UAV is selected as n1( Figure 5 (middle horizontal direction to the left) or n2 ( Figure 5 (in the direction to the right of the horizontal plane), in this case, when the resultant force F between the individuals... in,i When the preset threshold is exceeded, the forward direction n1 is selected so that its resultant force F with the inter-individual forces is combined. in,i If the angle between the two directions is smaller, then choose the forward direction n2 such that its angle with the current velocity direction ψ is smaller. i The included angle is smaller.

[0071] Based on the aforementioned wall-following principle, in S3, determining the drone's heading angle at the next moment based on the resultant force of attraction, repulsion, and action can include: calculating the resultant force of attraction and repulsion to obtain a first resultant force; calculating the resultant force of attraction, repulsion, and action to obtain a second resultant force; determining whether the angle between the direction of the first resultant force and the direction of attraction is greater than 90 degrees to obtain a first judgment result; if the first judgment result is yes, then determining whether the magnitude of the resultant force is greater than a preset threshold; if yes, then selecting the direction parallel to the specified obstacle and with an angle less than 90 degrees to the direction of the resultant force as the drone's heading angle at the next moment; otherwise, selecting the direction parallel to the specified obstacle and with an angle less than 90 degrees to the direction of the drone's heading angle at the current moment as the drone's heading angle at the next moment; if the first judgment result is no, then selecting the direction of the second resultant force as the drone's heading angle at the next moment.

[0072] By controlling the drone to fly according to its heading angle at the next moment, the position coordinates of the drone at the next moment can be obtained according to the drone's motion equation, and thus the observation vector of each drone in the drone swarm system at the next moment can be obtained.

[0073] S4: Determine if the iteration termination condition has been met; if yes, end the iteration; if no, use the observation vector of each drone at the next moment as the observation vector of each drone at the current moment in the next iteration, and return to the step of "using the observation vector of each drone at the current moment as input, and using a deep reinforcement learning network to determine the first action constant and the second action constant corresponding to each drone".

[0074] The iteration termination condition in this embodiment is that the distance from any UAV to the target is less than a preset value d. cap .

[0075] Preferably, after obtaining the observation vector of each UAV in the UAV swarm system at the next moment, the trajectory planning method of this embodiment further includes: for each UAV, calculating the reward of the UAV at the current moment using the reward function, and storing the observation vector of the UAV at the current moment, the first action constant and the second action constant corresponding to the UAV, the reward of the UAV at the current moment and the observation vector of the UAV at the next moment as experience into the experience pool of the deep reinforcement learning network. Every N iterations, the deep reinforcement learning network is trained based on the data in the experience pool, and the trained deep reinforcement learning network is used as the deep reinforcement learning network for the N+1th iteration. This training method can adopt existing commonly used training methods, which will not be described in detail here.

[0076] This embodiment sets the reward function to mainly include five parts: heading angle transformation reward, collision avoidance reward between individual UAVs, obstacle avoidance reward, distance-to-target potential field function reward, and cooperative reward.

[0077] The expression for the reward function is as follows:

[0078] r = r heading +r collide +r obstacle +r potential +r formation ;

[0079] Where r is the drone's reward at the current moment; r heading Bonus for heading angle transformation; r collide Rewards for collision avoidance between individual drones; obstacle Rewards for obstacle avoidance; potential The reward is the potential field function at the distance from the target; r formation For collaborative rewards.

[0080] The heading angle transformation reward r in this embodiment heading The main consideration is the dynamic characteristics of fixed-wing UAVs. When the heading angle of the UAV exceeds a first preset value (which can be 40°, or other values ​​as needed) at adjacent moments, a first penalty value (which can be -5, or other values ​​as needed) is applied. Specifically, the calculation method for the heading angle change reward includes: determining whether the angle between the UAV's heading angle at the next moment and its heading angle at the current moment is greater than the first preset value; if so, the heading angle change reward is the first penalty value; otherwise, the heading angle change reward is 0.

[0081] Inter-drone collision avoidance reward r collide The calculation principle is as follows: when a collision occurs, a second penalty value (which can be -30, or other values ​​as needed) is applied; otherwise, it is 0. Specifically, the calculation method for the collision avoidance reward between individual drones is as follows: determine whether a drone will collide with another drone in the next moment; if so, the collision avoidance reward is the second penalty value; otherwise, the reward is 0. The likelihood of a drone colliding with another drone in the next moment can be determined based on the position coordinates of all drones in that moment.

[0082] Obstacle avoidance reward r obstacleThe calculation principle is as follows: when a drone directly collides with an obstacle, a third penalty value (which can be -20, or other values ​​can be selected as needed) is applied; when the drone approaches an obstacle, a fourth penalty value (which can be -2, or other values ​​can be selected as needed) is applied. Specifically, the obstacle avoidance reward is calculated as follows: First, determine if the distance between the drone and the designated obstacle at the next moment is less than a third preset value, obtaining a second judgment result. If the second judgment result is yes, the obstacle avoidance reward is the third penalty value. If the second judgment result is no, determine if the distance between the drone and the designated obstacle at the next moment is less than a fourth preset value, and if the fourth preset value is greater than the third preset value. If yes, the obstacle avoidance reward is the fourth penalty value; otherwise, the obstacle avoidance reward is 0.

[0083] When the third preset value is d p The fourth preset value is 1.5d. p When the third penalty value is -20 and the fourth penalty value is -2, the formula for calculating the obstacle avoidance reward is as follows:

[0084]

[0085] Where d0 is the distance from the drone to the designated obstacle at the next moment, which can be calculated based on the drone's position coordinates at the next moment and the position coordinates of each obstacle. The designated obstacle is the obstacle closest to the drone.

[0086] The distance-to-target potential field function reward is used to encourage the drone to approach the target as quickly as possible, and its calculation formula is as follows:

[0087] r potential =γΦ(s)-Φ(s″);

[0088] Where, r potential Φ(()) is the potential field function reward for distance from the target; γ is the discount factor; Φ(()) is the potential field function; s is the observation vector of the UAV at the next time step, i.e., the state of the UAV at the next time step; s” is the observation vector of the UAV at the current time step, i.e., the state of the UAV at the current time step.

[0089] The potential field function is:

[0090]

[0091] Where, d e d1 is the distance from the drone to the target; d2 is the fifth preset value; d3 is the sixth preset value; d4 is the seventh preset value.

[0092] The collaborative reward is used to encourage drone swarms to cooperate and form formations to surround the target, reduce the target's escape space, and capture the target as quickly as possible. Its calculation formula is as follows:

[0093]

[0094] Where, r formation For collaborative rewards; r captor The reward obtained by the drone for successfully tracking down the target; d i,T Let d be the distance from UAV i to the target; cap This is the eighth preset value; r helper The cooperative reward obtained by other drones in tracking down the target; d j,T The distance from the drone j to the target; w q q is the first weighting factor; w is the intermediate parameter; q is the first weighting factor; w is the second weighting factor d It is the second weighting factor.

[0095]

[0096] Where n is the number of drones in the drone swarm system; The unit vector pointing from the drone closest to the target to the target; Let be the unit vector pointing from UAV i to the target, calculated using the following formula:

[0097]

[0098] Where, x e x represents the target's position coordinates; i Let be the position coordinates of UAV i.

[0099] This embodiment adopts a framework of centralized training and distributed execution of the same network. The state-action-reward experience obtained by all drones in the drone swarm system will be used to train the same network. This facilitates the faster collection of the required amount of experience and improves the training speed of the network. The relevant algorithm training process is as follows:

[0100]

[0101] This embodiment provides a method for collaborative target pursuit trajectory planning in a fixed-wing unmanned aerial vehicle (UAV) swarm system. It mainly solves the trajectory planning problem of collaborative target pursuit in a static obstacle environment. On the one hand, by combining artificial potential fields and reinforcement learning algorithms, it enables the pursuit of faster targets and obstacle avoidance within a limited area. On the other hand, by adopting a decentralized reinforcement learning algorithm framework, the UAV swarm system is trained centrally and executed in a distributed manner using the same network, so that each UAV can solve its own trajectory independently during the pursuit process, thereby improving computational efficiency.

[0102] Compared with the prior art, the beneficial effects of this embodiment are as follows:

[0103] (1) Considering the existence of obstacles in the given area, the drone is guided to avoid obstacles by introducing a repulsive force through an artificial potential field, while the wall following criterion is used to avoid falling into the minimum value of the artificial potential field method.

[0104] (2) By combining artificial potential field methods with deep reinforcement learning, the parameters in the potential field function are fully utilized to adjust the spatial distribution of the UAV swarm system, thereby achieving target pursuit;

[0105] (3) A decentralized trajectory planning algorithm framework was designed. The same set of neural networks is used for centralized training and distributed execution, so that each UAV can solve its own trajectory independently, thereby improving computational efficiency and achieving effective decentralization.

[0106] The following embodiment considers a multi-fixed-wing UAV swarm system, including 3 fixed-wing UAVs, and performs numerical simulation on their cooperative pursuit of target trajectory planning to verify the effectiveness of the proposed trajectory planning method. The simulation is trained using PyTorch 1.71 and Python 3.8.

[0107] The training parameters are set as follows: the flight area is 5000×5000m, the flight speed of each drone in the drone swarm system is 20m / s, the target's flight speed is 22m / s, and the action space is selected as η∈(0,3×10). 8 The training iterations are 9000, with a maximum step size of 1000 steps per run. The Actor and Critic networks are trained using the Adam optimizer. The experience pool size is 1e6. The target network is updated every 200 runs. The learning rates for the Actor and Critic networks are 10. -4 10 -3 The discount factor γ = 0.99, the greedy exploration strategy ε changes linearly from 1 to 0.01 after 6000 training iterations, the drone radius is 50m, and the radius parameters are d1 = 400m, d2 = 600m, d3 = 800m, and the obstacle radius parameter d p It is 20m.

[0108] For the drone swarm cooperative target pursuit algorithm, the change in total reward during the training process is as follows: Figure 6 As shown, Figure 6 The ordinate represents the training reward, and the x-axis represents the number of iterations. Figure 6In the training process, network parameters were recorded every 1000 iterations, and the reward converged around 6000 iterations, demonstrating the effectiveness of the proposed cooperative target pursuit algorithm. At the beginning of the training process, the amount of sample data affected the training effect; the policy network could not select appropriate actions based on the actual state, resulting in largely negative rewards and an experimental process. As the amount of data in the experience pool increased, the policy network continuously learned, and the UAV cooperative pursuit strategy was continuously improved and optimized. After 6000 training iterations, the multi-UAV pursuit strategy was roughly determined and stabilized in the training environment. Figure 7 and Figure 8 It can be seen that the pursuers formed an encircling formation around the target. Due to the target's high speed, the drones flew further away after a few steps to gain an advantage over the target in time. This indicates that the reward function encourages the drones to cooperate in chasing a given target.

[0109] Example 2:

[0110] This embodiment provides a collaborative target pursuit trajectory planning system for a drone swarm system, such as... Figure 9 As shown, the trajectory planning system includes:

[0111] The acquisition module M1 is used to acquire the observation vector of each drone in the drone swarm system at the current moment; the observation vector includes the distance of the drone to the nearest designated obstacle and the azimuth angle relative to the designated obstacle, the distance of the drone to the target and the azimuth angle relative to the target, and the distance of the drone to each other drone and the azimuth angle relative to each other drone;

[0112] The action selection module M2 is used to determine the first action constant and the second action constant for each UAV by taking the observation vector of each UAV at the current moment as input and using a deep reinforcement learning network.

[0113] The prediction module M3 is used to calculate, for each UAV, the attractive force of the UAV pointing towards the target, the repulsive force of the designated obstacle on the UAV, and the resultant force of the interaction between the UAV and the other UAVs using the artificial potential field method, with the first action constant and the second action constant as input; determine the heading angle of the UAV at the next moment based on the attractive force, the repulsive force, and the resultant force; control the UAV to fly according to the heading angle of the UAV at the next moment, and obtain the observation vector of each UAV in the UAV swarm system at the next moment;

[0114] The iteration module M4 is used to determine whether the iteration termination condition has been met; if yes, the iteration ends; if no, the observation vector of each UAV at the next moment is used as the observation vector of each UAV at the current moment in the next iteration, and the step of "using the observation vector of each UAV at the current moment as input, and using a deep reinforcement learning network to determine the first action constant and the second action constant corresponding to each UAV" is returned.

[0115] Each embodiment in this specification focuses on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be found in the method section.

[0116] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for cooperative pursuit of target trajectory planning for a UAV swarm system, characterized in that, The path planning method comprises: obtaining an observation vector of each unmanned aerial vehicle in the unmanned aerial vehicle cluster system at a current time; the observation vector comprises a distance of the unmanned aerial vehicle to a nearest designated obstacle and a directional angle of the unmanned aerial vehicle relative to the designated obstacle, a distance of the unmanned aerial vehicle to a target and a directional angle of the unmanned aerial vehicle relative to the target, and a distance of the unmanned aerial vehicle to each other unmanned aerial vehicle and a directional angle of the unmanned aerial vehicle relative to each other unmanned aerial vehicle; respectively inputting the observation vector of each unmanned aerial vehicle at the current time into a deep reinforcement learning network to determine a first action constant and a second action constant corresponding to each unmanned aerial vehicle; for each unmanned aerial vehicle, inputting the first action constant and the second action constant into an artificial potential field method to calculate an attractive force of the unmanned aerial vehicle pointing to the target, a repulsive force of the designated obstacle to the unmanned aerial vehicle, and a resultant force of the unmanned aerial vehicles and the other unmanned aerial vehicles; determining a heading angle of the unmanned aerial vehicle at a next time based on the attractive force, the repulsive force, and the resultant force; controlling the unmanned aerial vehicle to fly according to the heading angle of the unmanned aerial vehicle at the next time, to obtain an observation vector of each unmanned aerial vehicle in the unmanned aerial vehicle cluster system at the next time; determining whether an iteration termination condition is reached; if yes, ending the iteration; if no, taking the observation vector of each unmanned aerial vehicle at the next time as the observation vector of each unmanned aerial vehicle at the current time in the next iteration, and returning to the step of respectively inputting the observation vector of each unmanned aerial vehicle at the current time into the deep reinforcement learning network to determine the first action constant and the second action constant corresponding to each unmanned aerial vehicle; the determination of the heading angle of the unmanned aerial vehicle at the next time based on the attractive force, the repulsive force, and the resultant force specifically comprises: calculating a resultant force of the attractive force and the repulsive force to obtain a first resultant force; calculating a resultant force of the attractive force, the repulsive force, and the resultant force of the unmanned aerial vehicles and the other unmanned aerial vehicles to obtain a second resultant force; determining whether an included angle between a direction of the first resultant force and a direction of the attractive force is greater than 90 degrees to obtain a first determination result; if the first determination result is yes, determining whether a size of the resultant force of the unmanned aerial vehicles and the other unmanned aerial vehicles is greater than a preset threshold; if yes, selecting a direction parallel to the designated obstacle and having an included angle less than 90 degrees with a direction of the resultant force as the heading angle of the unmanned aerial vehicle at the next time; otherwise, selecting a direction parallel to the designated obstacle and having an included angle less than 90 degrees with a direction of the heading angle of the unmanned aerial vehicle at the current time as the heading angle of the unmanned aerial vehicle at the next time; if the first determination result is no, selecting a direction of the second resultant force as the heading angle of the unmanned aerial vehicle at the next time.

2. The method of claim 1, wherein, Before the step of respectively inputting the observation vector of each unmanned aerial vehicle at the current time into the deep reinforcement learning network to determine the first action constant and the second action constant corresponding to each unmanned aerial vehicle, the path planning method further comprises: For each unmanned aerial vehicle, feature extraction is performed on an observation vector of the unmanned aerial vehicle at a current time to obtain a feature vector of the unmanned aerial vehicle, and the feature vector of the unmanned aerial vehicle is taken as a new observation vector of the unmanned aerial vehicle at the current time.

3. The method of claim 2, wherein, The feature extraction on the observation vector of the unmanned aerial vehicle at the current time specifically comprises: inputting the observation vector of the unmanned aerial vehicle at the current time into a neural network for feature extraction; and the neural network comprises a fully connected layer.

4. The method of claim 1, wherein, The formula for calculating the attraction of the unmanned aerial vehicle to the target is: ； wherein, a drone i attractiveness to a target; coordinates of a location of a target; a drone i coordinates of a location of a target; The formula for calculating the repulsion of the designated obstacle to the unmanned aerial vehicle is: ； wherein, is the distance from the drone i to the nearest specified obstacle i ; is the first action constant; is the obstacle force radius range; is the distance from the drone i to the nearest specified obstacle ; i is the position coordinates of the drone The formula for calculating the resultant force between the unmanned aerial vehicle and the other unmanned aerial vehicles is: ； Wherein, is the unmanned aerial vehicle i the resultant force between the unmanned aerial vehicle is the unmanned aerial vehicle i a neighbor set of unmanned aerial vehicles, including other unmanned aerial vehicles in the unmanned aerial vehicle cluster system except for the unmanned aerial vehicle i ; is a second action constant; is the position coordinates of the other unmanned aerial vehicle j ; is the position coordinates of the unmanned aerial vehicle i .

5. The method of claim 1, wherein, After obtaining the observation vector of each unmanned aerial vehicle in the unmanned aerial vehicle swarm system at a next time, the path planning method further comprises: For each unmanned aerial vehicle, a reward function is used to calculate a reward of the unmanned aerial vehicle at the current time, and the observation vector of the unmanned aerial vehicle at the current time, the first and second action constants corresponding to the unmanned aerial vehicle, the reward of the unmanned aerial vehicle at the current time, and the observation vector of the unmanned aerial vehicle at the next time are stored in an experience pool of the deep reinforcement learning network. Every N iterations, the deep reinforcement learning network is trained based on the data in the experience pool, and the trained deep reinforcement learning network is taken as the deep reinforcement learning network at the N+1 iteration.

6. The method of claim 5, wherein, The reward function is: ； wherein, is a reward of the UAV at the current time; is a heading angle transformation reward; is a collision avoidance reward between UAV individuals; is an obstacle avoidance reward; is a distance-to-target potential field function reward; is a coordination reward.

7. The path planning method according to claim 6, characterized in that, whether an included angle between a heading angle of the unmanned aerial vehicle at the next time and a heading angle of the unmanned aerial vehicle at the current time is greater than a first preset value is judged; if yes, a heading angle transformation reward is a first penalty value; otherwise, the heading angle transformation reward is 0; whether the unmanned aerial vehicle collides with the other unmanned aerial vehicles at the next time is judged; if yes, an unmanned aerial vehicle individual inter-collision avoidance reward is a second penalty value; otherwise, the unmanned aerial vehicle individual inter-collision avoidance reward is 0; whether a distance from the unmanned aerial vehicle to the designated obstacle at the next time is less than a third preset value is judged to obtain a second judgment result; if the second judgment result is yes, an obstacle avoidance reward is a third penalty value; if the second judgment result is no, whether the distance from the unmanned aerial vehicle to the designated obstacle at the next time is less than a fourth preset value is judged, the fourth preset value being greater than the third preset value; if yes, the obstacle avoidance reward is a fourth penalty value; otherwise, the obstacle avoidance reward is 0.

8. The method of claim 6, wherein, The formula for calculating the distance target potential field function reward is: ； wherein, is a distance to goal potential field function; is a discount factor; is a potential field function; s is an observation vector of the UAV at the next time instant; is an observation vector of the UAV at the current time instant; The potential field function is: ； wherein, is a distance from the UAV to the target; d 1 is a fifth preset value; d 2 is a sixth preset value; d 3 is a seventh preset value; The formula for calculating the cooperative reward is: ； wherein, is a cooperative reward; is a reward obtained by the UAV chasing to the target; is a UAV i distance to the target; is an eighth preset value; is a cooperative reward obtained by other UAVs chasing to the target; is a UAV j distance to the target; is a first weight factor; q is an intermediate parameter; is a second weight factor; ； wherein, n is the number of drones in the drone swarm system; is a unit vector pointing towards the target for the drone closest to the target; is the drone i is a unit vector pointing towards the target. 9.A system for cooperative pursuit of target trajectory planning for a UAV swarm system, characterized in that, The path planning system comprises: An acquisition module is configured to acquire an observation vector of each unmanned aerial vehicle in a unmanned aerial vehicle swarm system at a current time; the observation vector comprises a distance from the unmanned aerial vehicle to a designated obstacle closest to the unmanned aerial vehicle and a direction angle relative to the designated obstacle, a distance from the unmanned aerial vehicle to a target and a direction angle relative to the target, and a distance from the unmanned aerial vehicle to each other unmanned aerial vehicle and a direction angle relative to each other unmanned aerial vehicle; The action selection module is configured to determine, by using a deep reinforcement learning network, a first action constant and a second action constant corresponding to each unmanned aerial vehicle respectively, with an observation vector of each unmanned aerial vehicle at a current time as input; The prediction module is configured to calculate, by using a potential field method, an attractive force of the unmanned aerial vehicle pointing to the target, a repulsive force of the specified obstacle to the unmanned aerial vehicle, and a resultant force of the unmanned aerial vehicle and the other unmanned aerial vehicles, with the first action constant and the second action constant as input; determine a heading angle of the unmanned aerial vehicle at a next time based on the attractive force, the repulsive force, and the resultant force; and control the unmanned aerial vehicle to fly according to the heading angle of the unmanned aerial vehicle at the next time, to obtain an observation vector of each unmanned aerial vehicle at the next time in the unmanned aerial vehicle swarm system. The iteration module is configured to determine whether an iteration termination condition is reached; if yes, end the iteration; and if no, return to the step of determining, by using the deep reinforcement learning network, the first action constant and the second action constant corresponding to each unmanned aerial vehicle respectively, with the observation vector of each unmanned aerial vehicle at the next time as the observation vector of each unmanned aerial vehicle at the current time in the next iteration. The determination of the heading angle of the unmanned aerial vehicle at the next time based on the attractive force, the repulsive force, and the resultant force specifically includes: calculating a resultant force of the attractive force and the repulsive force to obtain a first resultant force; calculating a resultant force of the attractive force, the repulsive force, and the resultant force of the action to obtain a second resultant force; determining whether an included angle between a direction of the first resultant force and a direction of the attractive force is greater than 90 degrees to obtain a first determination result; if the first determination result is yes, determining whether a size of the resultant force of the action is greater than a preset threshold; if yes, selecting a direction parallel to the specified obstacle and having an included angle less than 90 degrees with the direction of the resultant force of the action as the heading angle of the unmanned aerial vehicle at the next time; otherwise, selecting a direction parallel to the specified obstacle and having an included angle less than 90 degrees with a direction of the heading angle of the unmanned aerial vehicle at the current time as the heading angle of the unmanned aerial vehicle at the next time; and if the first determination result is no, selecting a direction of the second resultant force as the heading angle of the unmanned aerial vehicle at the next time.

Images