A multi-unmanned aerial vehicle target pursuit method based on reinforcement learning improvement

Abstract

The application discloses a multi-unmanned aerial vehicle target pursuit method based on reinforcement learning improvement, and comprises the following steps: S1, modeling the pursuit problem; S2, constructing a training network framework, adopting a multi-level course learning mode to complete training of a pursuit strategy, and obtaining a trained intelligent agent; and S3, inputting an observation space of a current pursuit unmanned aerial vehicle into the trained strategy network, acquiring a corresponding pursuit strategy, and completing pursuit of a target object.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicles, and particularly relates to a multi-unmanned aerial vehicle target pursuit method based on improved reinforcement learning. BACKGROUND

[0002] Unmanned aerial vehicles (UAVs) have attracted extensive attention due to their miniaturization, intelligence, long endurance and other advantages, and have shown great potential in various fields, and are widely used in search and rescue, surveillance and tracking, target attack and other tasks. It is difficult to cope with complex task requirements relying on a single unmanned aerial vehicle, and the introduction of multi-unmanned aerial vehicle cooperative control can further expand its application range and more efficiently and flexibly perform tasks. In the problem of cooperative pursuit, domestic and foreign scholars have conducted extensive research and proposed a series of methods for multi-unmanned aerial vehicle cooperative decision-making. Deep reinforcement learning optimizes its action strategy through repeated attempts to interact with the environment, and multi-agent deep reinforcement learning (MADRL) can enable unmanned aerial vehicles to learn excellent maneuvering decision-making ability and environmental adaptability in solving multi-unmanned aerial vehicle cooperative control and sequential decision-making tasks, so it is necessary to use MADRL to solve the target pursuit task. Some research sets the scene in a two-dimensional simple environment, and based on the Multi-agent Deep Deterministic Policy Gradient (MADDPG) algorithm and other algorithms, the agent's strategy learning is carried out, and good performance is achieved. In addition, the existing technology introduces the CommNet network based on the improved communication mechanism in the deep reinforcement learning algorithm, realizes information sharing between multiple unmanned aerial vehicles, and the technology effectively reduces the difficulty of strategy learning and improves the task adaptability by splitting the cooperative hunting task into two sub-tasks of agent navigation obstacle avoidance and navigation collision avoidance, and proposes a hierarchical deep reinforcement learning method that can realize autonomous switching of sub-strategies, and scholars have proposed a multi-environment asynchronous double Q network (MAEDDQN) with priority experience replay for unmanned aerial vehicle strategy learning, but these methods only consider learning the maneuvering decision of multiple unmanned aerial vehicles in the scene of no obstacles or static obstacles in the external environment.

[0003] Multi-agent proximal policy optimization (MAPPO) algorithm is an extension of proximal policy optimization (PPO) in the multi-agent field. During training, the state feedback of all agents is fed back to the critic network for policy evaluation, so that each agent can adjust its own policy according to the environmental information and the state of other agents, which can solve the non-stationarity of MADRL environment and thus complete the cooperation task of multi-agent in a complex scene. However, in the problem of multi-UAV target pursuit, the traditional MAPPO algorithm has certain limitations: on the one hand, learning in a complex environment with dynamic obstacles may slow down the learning process, and on the other hand, in the setting of shared experience, the agent may be influenced by early successful experience and take similar actions, and the pursuit strategy is single. SUMMARY

[0004] Therefore, the present application provides a multi-UAV target pursuit method based on improved reinforcement learning, which at least solves the problem of slow learning process when the MAPPO algorithm is applied to the field of multi-UAV target pursuit in the prior art.

[0005] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0006] A multi-UAV target pursuit method based on improved reinforcement learning, comprising the following steps:

[0007] S1. Model the pursuit problem; in a three-dimensional scene of multi-UAV cooperative target pursuit in a dynamic obstacle environment, construct the observation space of the pursuit UAV based on the real-time state information of the pursuit UAV ;

[0008] S2. Construct a training network framework and complete the training of the pursuit strategy in a multi-level curriculum learning manner to obtain a trained agent;

[0009] Obtain the initial position coordinates of the pursuit UAV according to the initial positions of the preset target and dynamic obstacles, preset the action strategy of the pursuit UAV, the target and the dynamic obstacles, train the agent based on the observation space of the pursuit UAV; if the pursuit success rate corresponding to the strategy network of the current agent reaches the preset condition for ending the current curriculum, perform the training of the next curriculum until the trained agent is obtained by meeting the preset optimization condition;

[0010] S3. Input the observation space of the current pursuit UAV into the trained strategy network to obtain the corresponding pursuit strategy and complete the pursuit of the target.

[0011] Preferably, the number of pursuit UAVs is , the number of dynamic obstacles is , and the four-tuple is ​Describe the drone pursuit process, in which, This is the state transition function. For observation space, For the action space, For the reward function, any pursuit drone Observations This includes real-time status information. , , and Indicates drone coordinates , and They represent drones The velocity, pitch angle, and yaw angle; the state transition function specifically includes:

[0012] ;

[0013] ;

[0014] In the formula, , and Representing the three-dimensional coordinate system respectively Changes in coordinates , and They represent speeds respectively. Pitch angle and heading angle The change in; among which The tangential overload of a drone represents the ratio of the thrust acting on it in the direction of its forward velocity to its own weight, and is used to control acceleration and deceleration. For overloads that are perpendicular to the plane of the fuselage and directed upwards. Indicates its roll angle, Represents gravitational acceleration;

[0015] drones observation space The specific content includes:

[0016] ;

[0017] In the formula, , and These represent the target to the drone respectively. The relative distance between the positions in the 3D coordinate direction. , and These represent other pursuit drones. To drones the relative distance between the obstacle and the UAV, wherein , , and represent the relative position of the obstacle to the UAV , wherein ; , and represent the coordinates of the UAV , , and represent the velocity, pitch angle and heading angle of the UAV , respectively;

[0018] The specific content of the action space A includes:

[0019] The action space of each pursuit UAV includes discrete actions , each of which is composed of a control amount , by selecting different to control the change of the UAV state;

[0020] The reward function includes a capture success reward , a distance reward and a collision reward , and each UAV uses a reward value to guide the learning of the policy;

[0021] The success reward is:

[0022] ;

[0023] wherein S is a positive constant, representing the reward given when the distance between the pursuit UAV and the target is less than a threshold , that is, when the capture is successful;

[0024] The distance reward is:

[0025] ;

[0026] The distance reward is defined as the difference between the distance of the UAV to the target at the previous time step and the distance of the UAV to the target at the current time step , and a positive reward is given when the distance becomes smaller, and vice versa;

[0027] The collision reward is:

[0028] ;

[0029] wherein, C represents the punishment given when the current UAV collides, the UAV that collides will exit the task; represents the closest distance of the UAV to the obstacle or other UAVs, represents the collision threshold, represents the safety threshold, when the collision probability is large, and the strategy needs to be adjusted, represents the punishment value when the UAV is in a dangerous area but has not collided, and the punishment value is calculated using the product of and to calculate the punishment value of the UAV at this time.

[0030] Preferably, the specific content of S2 includes:

[0031] S21. The initial positions of the target object and the dynamic obstacle are preset, and the initial position of the preset target object is taken as the initial position of the pursuit UAV to obtain a Brown position set S0 of the first level course. Multiple Brown motions are performed, and the position information after each Brown motion is used to update the Brown position set S0 corresponding to the last Brown motion. A position is randomly extracted from the Brown position set each time as the initial coordinates of the pursuit UAV. Each pursuit UAV starts from the preset initial coordinates, performs b-step random Brown motion, and randomly completes any one discrete action in the action space during each step of the random Brown motion to obtain the state information of the current pursuit UAV. The position information is added to the Brown position set S0 corresponding to the last Brown motion to update and obtain the initial position set of the pursuit UAV for the training of the level course.

[0032] S22. A position is randomly extracted from the initial position set as the initial coordinates of the pursuit UAV.

[0033] S23. The action strategy of the pursuit UAV, the target object and the dynamic obstacle is preset.

[0034] S24. The agent is trained by introducing an adversarial network. After the action strategy output by the policy network is completed, the state information of the environment in which the current agent is located is obtained, the reward corresponding to the current action strategy is obtained according to the reward function, the value of the current action strategy is evaluated by the value network in the agent, the policy network updates the strategy according to the value provided by the policy network, and the pursuit UAV p corresponding observation space is taken as the input of the adversarial network to obtain the predicted action, the difference between the policy network and the adversarial network is increased while the return is maximized to complete the optimization of the policy network.

[0035] S25. If the pursuit success rate corresponding to the current policy network reaches the preset condition for ending the level, the initial position set of the pursuit UAV of the level is taken as the Brown position set of the pursuit UAV of the next level, a new initial position set is generated again according to S21, and S22-S24 are performed again until the preset optimization condition is met to obtain the trained agent.

[0036] Preferably, the specific content of S21 includes:

[0037] pursuit UAV from the preset position , performs b-step random Brown motion, and randomly selects a discrete action from the action space at each time step . Based on the current , a new state after the execution of the action is obtained through the state transition function, and the position coordinates corresponding to each new state are added to the current initial position set.

[0038] Preferably, the preset action strategies of the pursuit UAV, the target object and the dynamic obstacle are:

[0039] The pursuit UAV makes decisions through the policy network in the trained agent: according to the positions of the target object and the dynamic obstacle at time -1, the observation space corresponding to the pursuit UAV is obtained and taken as the input of the policy network, and the policy network outputs the decision action after the UAV executes; when the distance between the pursuit UAV and the dynamic obstacle or other pursuit UAVs is less than a safety threshold, the artificial potential field method is used to assist in decision-making. t p

[0040] Action strategy of the target object: the new positions of the target object after taking each action in the action space A are calculated respectively, the distances between each new position and the current pursuit UAV are obtained, and the action corresponding to the farthest distance is selected to update the state information of the target object.

[0041] Action strategy of the dynamic obstacle: uniform motion is taken, and the dynamic obstacle moves back and forth along the y axis direction.

[0042] Preferably, the specific content of the artificial potential field method for assisting decision-making in S23 includes:

[0043] For each discrete action in the action space, the velocity vector is calculated, where , and represent the velocities in the x, y and z axis directions, respectively.

[0044] According to the velocity vector and the artificial potential field force vector​​ The cosine similarity is calculated, wherein , and are respectively the components of the artificial potential force vector in the x, y and z axis directions:

[0045] ;

[0046] With the defined action space A, the calculated different velocity vectors , the corresponding , the vector corresponding to the action space A , the observation of the pursuit UAV is input into the strategy network to output the vector ; ;

[0047] The original is weighted to obtain the weighted , wherein , using the weighted , the action probability distribution is obtained by softmax, and then the final action is sampled.

[0048] Preferably, the specific content of the artificial potential force vector includes:

[0049] The target object and the dynamic obstacle are respectively modeled as a gravitational source and a repulsive source, and the motion direction and speed of the pursuit UAV are adjusted under the action of the combined force of the gravity and the repulsion. The distance between the UAV and the escape target determines the repulsive potential field. In the pursuit problem, there is one escape target and n obstacles, so the artificial potential force vector of the pursuit UAV is:

[0050] ;

[0051] Wherein, represents the gravitational force received by the pursuit UAV , which is the negative gradient of the gravitational potential field, represents the repulsive force of the obstacle:

[0052] ;

[0053] ;

[0054] In the formula, represents the gravitational potential field function, ​Indicates the gravitational coefficient. From drones Pointing to target The distance vector;

[0055] ;

[0056] ;

[0057] In the formula, The function representing the repulsive potential field of the obstacle. Indicates the repulsion coefficient. It is the radius of influence of the obstacle. From the obstacle Pointing to drones p The distance vector.

[0058] Preferably, the specific content of S24 includes:

[0059] ① Initialize network parameters: The policy network is used to generate actions, the value network is used to estimate state values, and the adversarial network is designed to mimic the behavior of the policy network;

[0060] ②In the environment, with The policy network model can learn parameters, and the current policy network is used to track the drone p. The action is interactively sampled to collect experience data at different time steps, including state, action, reward and next state;

[0061] ③ Adversarial Networks ,by The adversarial network model can be trained by using KL divergence as the loss term and aiming to minimize KL divergence to continuously mimic the policy network.

[0062] ④ When updating the policy in the policy network, the value network first uses the generalized dominance estimation method to calculate the dominance term corresponding to the current policy. To avoid being imitated by adversarial networks, the policy network computes the old policy network. Estimated actions action probability and the old adversarial network Estimated actions action probability The logarithmic difference, i.e. ,by and These represent the parameters of the old model, which are added to the advantages section. After acquiring new advantages, the modified advantages are denoted as... The policy network updates its parameters by minimizing the negative value of the loss.

[0063] V. Value network update: compute the difference between the actual collected reward and the estimated reward value output by the value network, take the mean square error as the loss term, and then update the parameters;

[0064] VI. Repeat iteration: constantly repeat the process of ②-⑤.

[0065] Preferably, the adversarial network The policy network Both take the observation of the UAV p as input Output the action of the UAV, where the policy network outputs the decision action of the UAV The adversarial network outputs the predicted action The purpose is to imitate the action of the policy network;

[0066] The adversarial network is trained by minimizing the KL divergence, and the loss function of the adversarial network Is:

[0067] ;

[0068] Where And Respectively represent the probability distribution of all possible actions given the observation When using the old policy network parameters And the current adversarial network parameters , Represents any one action, Indicates the KL divergence, which measures the difference between the distributions; And Respectively represent the network parameters of the old policy network and the current adversarial network, and the adversarial network imitates the past policy rather than the current policy, thereby maintaining relative stability;

[0069] The loss function of the policy network Is:

[0070] ;

[0071] Where, Is the expectation, Is the importance sampling ratio, Represents the clip function, Is the limit parameter, which clips To the range of Through the clip operation, control the update amplitude of the policy, and c is a hyperparameter, which linearly decays with training, thereby reducing the attention of the policy network to the adversarial network.

[0072] According to the above technical solution, compared with the prior art, the following beneficial effects are obtained:

[0073] The application provides a multi-UAV target pursuit method based on a multi-agent reinforcement learning method MAPPO, in order to train a pursuit strategy of a UAV in a dynamic complex environment, an adversarial network is introduced to promote the UAV to learn a diversified and robust pursuit strategy through simulating adversarial interaction; while the intelligent decision of the neural network is exerted, the real-time obstacle avoidance capability of the artificial potential field method is combined to correct the local path, so that the threat of dynamic obstacles can be coped with; and in order to improve sample efficiency, a curriculum training method is used during training, the distance between the initial position and the target is from near to far, and the progressive learning can help the UAV to obtain more effective experience in the early stage. BRIEF DESCRIPTION OF DRAWINGS

[0074] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, below the drawings needed to be used in the embodiments or prior art description will be briefly introduced, obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0075] Figure 1 A flowchart of the multi-UAV target pursuit method based on the improved reinforcement learning provided by the present application is shown in the figure.

[0076] Figure 2 A curriculum learning initial position set provided by the present application is shown in the figure.

[0077] Figure 3 An agent training framework in the multi-UAV target pursuit method based on the improved reinforcement learning provided by the present application is shown in the figure.

[0078] Figure 4 An auxiliary decision process schematic diagram of the artificial potential field method in the multi-UAV target pursuit method based on the improved reinforcement learning provided by the present application is shown in the figure.

[0079] Figure 5 A UAV motion model schematic diagram in the multi-UAV target pursuit method based on the improved reinforcement learning provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0080] The technical solutions in the embodiments of the present application will be described clearly and completely below with the drawings in the embodiments of the present application, obviously, the described embodiments are only some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0081] The present application provides a multi-UAV target pursuit method based on the improved reinforcement learning, as shown in the figure. Figure 1As shown, comprising the following steps:

[0082] S1. Model the pursuit problem; in a three-dimensional scene of multi- UAV cooperative pursuit of a target object in a dynamic obstacle environment, based on the real-time state information of the pursuit UAV p , the observation space o p of the pursuit UAV is constructed;

[0083] S2. Construct a training network framework, and complete the training of the pursuit strategy in a multi-level curriculum learning manner to obtain a trained agent;

[0084] According to the initial positions of the preset target object and dynamic obstacles, the initial position coordinates of the pursuit UAV are obtained, the action strategies of the pursuit UAV, the target object and the dynamic obstacles are preset, and based on the observation space of the pursuit UAV, an adversarial network is introduced to train the agent; if the pursuit success rate corresponding to the strategy network of the current agent reaches the preset condition for ending the current curriculum, the training of the next curriculum is performed, and the trained agent is obtained until the preset optimization condition is met;

[0085] S3. Input the observation space of the current pursuit UAV into the trained strategy network to obtain the corresponding pursuit strategy and complete the pursuit of the target object.

[0086] It should be noted that:

[0087] In a three-dimensional scene of multi- UAV cooperative pursuit of an escape UAV in a dynamic obstacle environment, the number of pursuit UAVs is m, the number of dynamic obstacles is n, and the number of escape UAVs is 1. A circular column is used to simulate the obstacle. The goal of the problem is to give the maneuvering strategy of the m pursuit UAVs, so that they can finally capture the target while avoiding obstacles. In the game process, the target escape path is unknown, but the pursuit UAV can obtain real-time position information of the obstacles and the target. In order to facilitate the use of reinforcement learning to solve the problem, this paper uses Markov Decision Process (MDP) to model the pursuit problem. MDP is generally described by four tuples (P, O, A, R).

[0088] Multi- UAV uses multi-level curriculum training method: at the beginning of training, set the initial position of the pursuit UAV close to the target; after mastering some successful pursuit experience, gradually increase the curriculum difficulty, that is, set the distance between the initial position of the UAV and the target to increase.

[0089] As shown in Figure 2 , the first pursuit UAV randomly selects a position pos i from the initial point set S and sets its initial state. Figure 2The diagram illustrates the initial positions from the XY plane during the 2nd and 5th iterations. The initial positions of the obstacles, the target, and the multiple sets of initial positions for the three pursuit drones are shown in the diagram. The initial positions of the obstacles and the target remain unchanged throughout the iterations. At the start of each training round, one set of initial positions for the three pursuit drones is randomly selected as the initial positions for that round. In the 2nd iteration, the initial drone positions are closer to the target; in the 5th iteration, the initial positions are more dispersed, thus gradually increasing the training difficulty.

[0090] After each round, the network is updated using data collected at each time step, including observations, drone actions, and environmental rewards. This network training improves the drone's decision-making capabilities. Figure 3 As shown, the agent training framework includes a policy network, a value network, and an adversarial network, which are updated simultaneously. The adversarial network and the policy network each consist of m instances, denoted as the adversarial network. p Policy Network p The value of p ranges from 1 to m. Observe o. p Input to the policy network respectively p Corresponding to each action a p Value network output advantages A p .

[0091] During the learning process, when the distance between the pursuit drone and obstacles or other pursuit drones is less than a safety threshold, an artificial potential field method is introduced to assist in decision-making. Figure 4 As shown;

[0092] When performing target pursuit missions after the drone has completed its learning process, it can select and execute actions in dynamic obstacle environments based on the current state, the obtained target and obstacle positions, and the positions of other drones as observation information. It can also utilize the trained policy network and the calculated artificial potential force under the current state to complete the target pursuit while avoiding obstacles.

[0093] To further implement the above technical solution, the number of drones used for pursuit was [number missing]. The number of dynamic obstacles is Through quadruples Describe the drone pursuit process, in which, This is the state transition function. For observation space, For the action space, For the reward function, any pursuit drone Observations This includes real-time status information. , , and represents the coordinates of the UAV , , and represent the velocity, pitch angle and heading angle of the UAV , respectively; the specific content of the state transition function includes:

[0094] ;

[0095] ;

[0096] wherein , and represent the change amount of the coordinates of the UAV in the three-dimensional space coordinate system, , and represent the change amount of the velocity , pitch angle and heading angle , respectively; wherein is the tangential overload of the UAV, representing the ratio of the thrust in the forward velocity direction and the gravity of the UAV itself, used for controlling acceleration and deceleration, is the overload perpendicular to the body plane and in the upward direction, represents the roll angle thereof, represents the gravity acceleration;

[0097] The specific content of the observation space of the UAV includes:

[0098] ;

[0099] wherein , and represent the relative distance of the position of the target to the UAV in the 3D coordinate direction, , and represent the relative distance of the other pursuit UAV to the UAV , wherein , , and represent the relative position of the obstacle to the UAV , wherein ; , and represent the coordinates of the UAV , , and The tables represent drones. Speed, pitch angle, and yaw angle;

[0100] The specific contents of action space A include:

[0101] The motion space of each pursuit drone includes Discrete actions Each discrete action consists of a control variable. By choosing different To control changes in the drone's status;

[0102] The reward function includes a capture success reward. Distance Rewards and collision rewards Each drone uses reward points To guide the learning of strategies;

[0103] Success reward for:

[0104] ;

[0105] in S A positive constant indicates the distance between the tracking drone and the target. Less than the threshold The reward given upon successful capture;

[0106] Distance Rewards for:

[0107] ;

[0108] Distance reward is defined as the distance the drone traveled to the target in the previous time step. Distance from the target at the current time step The difference in distance is rewarded positively if the distance decreases, and negatively if the distance increases.

[0109] Collision Rewards for:

[0110] ;

[0111] In the formula, C This indicates the penalty imposed when a drone collides with another drone; the drone involved in the collision will be removed from the mission. This indicates the closest distance between the drone and an obstacle or another drone. Indicates the collision threshold. Indicates the safety threshold, when The collision probability is large when less than the safety threshold, and the strategy needs to be adjusted, The penalty value when the UAV is in a dangerous area but has not collided yet is represented by The penalty value of the UAV at this time is calculated using the product of and

[0112] It should be noted that:

[0113] Considering the characteristics of the actual decision-making of the UAV and the convenience of solving, in the embodiment, as shown in Figure 5 , in a three-dimensional coordinate system with O as the origin, and are the speed of the UAV and the speed projection of the horizontal plane, , represent the pitch angle and the heading angle of the UAV, is the tangential overload of the UAV, is the normal overload, represents the roll angle. Each UAV adopts an action space containing 15 discrete actions, each action is composed of control quantities , and the change of the state of the UAV is controlled by selecting different . The action space is shown in Table 1.

[0114] Table 1 Action space

[0115] ;

[0116] In order to further implement the above technical solutions, the specific content of S2 includes:

[0117] S21. The initial positions of the target object and the dynamic obstacle are preset, and the initial position of the preset target object is taken as the initial position of the pursuit UAV to obtain a Brown position set S0 of the first level course. Multiple Brown motions are performed, and the position information after each Brown motion is used to update the Brown position set S0 corresponding to the last Brown motion. Each time a position is randomly extracted from the Brown position set as the initial coordinates of the pursuit UAV, each pursuit UAV starts from the preset initial coordinates, performs b-step random Brown motion, and randomly completes any one discrete action in the action space during each step of the random Brown motion to obtain the state information of the current pursuit UAV. The position information is added to the Brown position set S0 corresponding to the last Brown motion to update and obtain the initial position set of the pursuit UAV for the training of the level course;

[0118] S22. A position is randomly extracted from the initial position set as the initial coordinates of the pursuit UAV.

[0119] S23. preset the action strategy of the pursuit UAV, the target object and the dynamic obstacle;

[0120] S24. introduce the adversarial network to train the agent; after completing the action strategy output by the policy network, the state information of the environment where the agent is currently located is obtained, the reward corresponding to the current action strategy is obtained according to the reward function, the value of the current action strategy is evaluated through the value network in the agent, the policy network updates the policy according to the value provided by the policy network, and the pursuit UAV p corresponding observation space as the input of the adversarial network to obtain the predicted action, maximize the return while increasing the difference between the policy network and the adversarial network to complete the optimization of the policy network;

[0121] S25. if the pursuit success rate corresponding to the current policy network reaches the preset condition for ending the level, the initial position set of the pursuit UAV of this level is taken as the initial position set of the pursuit UAV of the next level, a new initial position set is generated according to S21, and S22-S24 are performed again until the trained agent is obtained by meeting the preset optimization condition.

[0122] In order to further implement the above technical solutions, the specific content of S21 includes:

[0123] the pursuit UAV starts from the preset position , performs b-step random Brownian motion, and randomly selects a discrete action from the action space at each time step , based on the current , obtains the new state after executing the action through the state transition function, and adds the position coordinates corresponding to each new state to the current initial position set.

[0124] It should be noted that:

[0125] In this embodiment, the discrete action a p is any action selected from the action space containing 15 discrete actions described in the above embodiment.

[0126] In order to further implement the above technical solutions, the preset action strategy of the pursuit UAV, the target object and the dynamic obstacle includes:

[0127] The pursuit UAV makes decisions through the policy network in the trained agent: according to the positions of the target object and the dynamic obstacle at time -1, the pursuit UAV t performs b-step random Brownian motion, and at each time step, a discrete action a pThe corresponding observation space is obtained and input into a policy network as an input, and the policy network outputs a decision action which is executed by the UAV; when the distance between the pursuit UAV and the dynamic obstacle or other pursuit UAVs is less than a safety threshold, an artificial potential field method is used to assist decision-making;

[0128] Action strategy of the target object: the corresponding new position of the target object after taking each action in the action space A is calculated respectively, and the distance between each new position and the current pursuit UAV is obtained, and the action corresponding to the farthest distance is selected to update the state information of the target object;

[0129] Action strategy of the dynamic obstacle: uniform motion is taken, and the dynamic obstacle moves back and forth along the y axis direction.

[0130] In order to further implement the above technical solutions, the specific content of the artificial potential field method assisted decision-making in S23 includes:

[0131] For each discrete action in the action space, the velocity vector is calculated, wherein , and represent the velocities in the x, y and z axis directions respectively;

[0132] The cosine similarity is calculated according to the velocity vector and the artificial potential field force vector , wherein , and are the components of the artificial potential field force vector in the x, y and z axis directions respectively:

[0133] ;

[0134] With the defined actions in the action space A, the different velocity vectors are calculated, the corresponding are obtained, the vectors corresponding to the actions in the action space A are obtained, the observation of the pursuit UAV is input into the policy network, and the vector is output; ;

[0135] The original z is weighted to obtain the weighted , wherein , the weighted is used, the action probability distribution is obtained through softmax, and then the final action is sampled.

[0136] In order to further implement the above technical solutions, the artificial potential field force vector The specific content includes:

[0137] The target and the dynamic obstacle are modeled as a gravitational source and a repulsive source respectively, and the motion direction and speed of the pursuit UAV are adjusted under the action of the resultant force of gravity and repulsion. In the pursuit problem, there is one escaping target and n obstacles, so the pursuit UAV The artificial potential field force vector is:

[0138] ;

[0139] Where, represents the gravitational force received by the pursuit UAV , which is the negative gradient of the gravitational potential field, represents the repulsive force of the obstacle:

[0140] ;

[0141] ;

[0142] In the formula, represents the gravitational potential field function, represents the gravitational coefficient, is the distance vector from the UAV to the target ;

[0143] ;

[0144] ;

[0145] In the formula, represents the repulsive potential field function of the obstacle, represents the repulsive coefficient, is the influence radius of the obstacle, is the distance vector from the obstacle to the UAV p .

[0146] In order to further implement the above technical solutions, the specific content of S24 includes:

[0147] ①Initialize network parameters: the policy network is used to generate actions, the value network is used to estimate state values, and the adversarial network aims to imitate the behavior of the policy network;

[0148] ②In the environment, let represent the learnable parameters of the policy network model, and the pursuit UAV uses the current policy network to act to interact with sampling, and collects experience data at different time steps, including state, action, reward and next state;

[0149] ③ Adversarial Networks ,by The adversarial network model can be trained by using KL divergence as the loss term and aiming to minimize KL divergence to continuously mimic the policy network.

[0150] ④ When updating the policy in the policy network, the value network first uses the generalized dominance estimation method to calculate the dominance term corresponding to the current policy. To avoid being imitated by adversarial networks, the policy network computes the old policy network. Estimated actions action probability and the old adversarial network Estimated actions action probability The logarithmic difference, i.e. ,by and These represent the parameters of the old model, which are added to the advantages section. After acquiring new advantages, the modified advantages are denoted as... The policy network updates its parameters by minimizing the negative value of the loss.

[0151] ⑤ Value Network Update: Calculate the difference between the actual collected reward and the estimated reward value output by the value network, using the mean squared error as the loss term, and then update the parameters;

[0152] ⑥ Repeated iteration: Continuously repeat processes ②-⑤.

[0153] To further implement the above technical solutions, adversarial networks Same-policy network All observations were made using drones (p). As input, the output is the drone's actions, where the policy network outputs the drone's decision actions. Actions to counteract network output prediction The purpose is to mimic strategic network actions;

[0154] The adversarial network is trained by minimizing the KL divergence, and the loss function of the adversarial network is... for:

[0155] ;

[0156] In the formula and These represent the network parameters when using the old strategy. and current adversarial network parameters At that time, given observations The probability distribution of time for all possible actions, where represents any one action, represents KL divergence, measuring the difference between distributions; and respectively represent the network parameters of the old policy network and the current adversarial network, the adversarial network imitates the past policy rather than the current policy, thereby maintaining relative stability;

[0157] Loss function of the policy network is:

[0158] ;

[0159] In the formula, is the expectation, is the importance sampling ratio, represents the clip function, is the limit parameter, and is clipped to the range of by the clip operation, controlling the update amplitude of the policy, and c is a hyperparameter, which is linearly decayed with training, thereby reducing the attention of the policy network to the adversarial network.

[0160] The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A multi-UAV target tracking method based on reinforcement learning, characterized in that, Includes the following steps: S1. Model the pursuit problem; In a 3D scenario of multi-UAV cooperative pursuit of a target in a dynamic obstacle environment, based on the pursuit UAV Its observation space is constructed using real-time status information. ; S2. Construct a training network framework and use a multi-level course learning method to complete the training of the pursuit strategy to obtain a trained agent; The initial position coordinates of the pursuit drone are obtained based on the initial positions of the preset target and dynamic obstacles. The action strategies of the pursuit drone, target and dynamic obstacles are preset. Based on the observation space of the pursuit drone, an adversarial network is introduced to train the agent. If the pursuit success rate corresponding to the current agent's strategy network reaches the preset condition for the end of the course, the training of the next course is carried out until the preset optimization conditions are met and a well-trained agent is obtained. The specific content of S2 includes: S21. Preset the initial positions of the target and dynamic obstacles, and use the preset initial position of the target as the initial position of the pursuit drone to obtain the Brownian position set for the first level course. Perform multiple Brownian motions and update the Brownian position set corresponding to the previous Brownian motion with the position information after each Brownian motion. Each time, a position is randomly selected from the Brownian position set as the initial coordinates of the pursuit drone. Each pursuit drone starts from the preset initial coordinates and performs b steps of random Brownian motion. During each step of random Brownian motion, it randomly completes any discrete action in the action space, obtains the current state information of the pursuit drone, and adds the position information to the Brownian position set corresponding to the previous Brownian motion. The initial set of positions for the pursuit drone to conduct training at this level is updated; S22. Randomly select a location from the initial location set as the initial coordinates for the pursuit drone; S23. Pre-set action strategies for pursuing drones, targets, and dynamic obstacles; S24. Introduce an adversarial network to train the agent; after completing the action policy output by the policy network, obtain the state information of the current environment in which the agent is located, obtain the reward corresponding to the current action policy according to the reward function, evaluate the value of the current action policy through the value network in the agent, update the policy based on the value it provides, and then track the drone. The corresponding observation space serves as the input to the adversarial network, which then predicts actions. This maximizes the reward while increasing the difference between the policy network and the adversarial network to optimize the policy network. S25. If the pursuit success rate corresponding to the current policy network reaches the preset condition for the end of this course, then the initial position set of the pursuit drone in this course is used as the Brownian position set of the pursuit drone in the next course. A new initial position set is generated according to S21, and S22-S24 are repeated until the preset optimization conditions are met and a trained agent is obtained. Pre-defined operational strategies for pursuing drones, targets, and dynamic obstacles: The pursuit drone makes decisions through a policy network in a pre-trained agent: based on the target and dynamic obstacles... Location at any given time, obtain the tracking drone The corresponding observation space is used as the input to the policy network. After the policy network outputs the decision action, the UAV executes it. When the distance between the pursuing UAV and dynamic obstacles or other pursuing UAVs is less than the safety threshold, the artificial potential field method is used to assist in decision-making. Action strategy for the target: Calculate the new position of the target after each action in action space A, and obtain the distance between each new position and the current pursuit drone. Select the action corresponding to the farthest distance to update the state information of the target. Movement strategy for dynamic obstacles: Adopt uniform speed movement, along... Move back and forth along the axis; The specific content of S24 includes: ① Initialize network parameters: The policy network is used to generate actions, the value network is used to estimate state values, and the adversarial network is designed to mimic the behavior of the policy network; ②In the environment, with The policy network model can learn parameters, and the current policy network is used to track the drone p. The action is interactively sampled to collect experience data at different time steps, including state, action, reward and next state; ③ Adversarial Networks ,by The adversarial network model can be trained by using KL divergence as the loss term and aiming to minimize KL divergence to continuously mimic the policy network. ④ When updating the policy in the policy network, the value network first uses the generalized dominance estimation method to calculate the dominance term corresponding to the current policy. To avoid being imitated by adversarial networks, the policy network computes the old policy network. Estimated actions action probability and the old adversarial network Estimated actions action probability The logarithmic difference, i.e. ,by and These represent the parameters of the old model, which are added to the advantages section. After acquiring new advantages, the modified advantages are denoted as... The policy network updates its parameters by minimizing the negative value of the loss. ⑤ Value Network Update: Calculate the difference between the actual collected reward and the estimated reward value output by the value network, using the mean squared error as the loss term, and then update the parameters; ⑥ Repeated iteration: Continuously repeat processes ②-⑤; S3. Input the observation space of the current pursuit drone into the trained policy network, obtain the corresponding pursuit policy, and complete the pursuit of the target.

2. The multi-UAV target tracking method based on reinforcement learning as described in claim 1, characterized in that, The number of drones that completed the pursuit was The number of dynamic obstacles is Through quadruples Describe the drone pursuit process, in which, This is the state transition function. For observation space, For the action space, For the reward function, any pursuit drone observation space This includes real-time status information. , , and Indicates drone coordinates , and They represent drones Speed, pitch angle, and yaw angle; The state transition function specifically includes: ； ； In the formula, , and Representing the three-dimensional coordinate system respectively Changes in coordinates , and They represent speeds respectively. Pitch angle and heading angle The change in; among which The tangential overload of a drone represents the ratio of the thrust acting on it in the direction of its forward velocity to its own weight, and is used to control acceleration and deceleration. For overloads that are perpendicular to the plane of the fuselage and directed upwards. Indicates its roll angle, Represents gravitational acceleration; drones observation space The specific content includes: ； In the formula, , and These represent the target to the drone respectively. The relative distance between the positions in the 3D coordinate direction. , and These represent other pursuit drones. To drones The relative distance, where , , and They represent obstacles. To drones The relative positions, of which ; , and Indicates drone coordinates , and The tables represent drones. Speed, pitch angle, and yaw angle; The specific contents of action space A include: The motion space of each pursuit drone includes Discrete actions Each discrete action consists of a control variable. By choosing different To control changes in the drone's status; The reward function includes a capture success reward. Distance Rewards and collision rewards Each drone uses reward points To guide the learning of strategies; Success reward for: ； in S A positive constant indicates the distance between the tracking drone and the target. Less than the threshold The reward given upon successful capture; Distance Rewards for: ； Distance reward is defined as the distance the drone traveled to the target in the previous time step. Distance from the target at the current time step The difference in distance is rewarded positively if the distance decreases, and negatively if the distance increases. Collision Rewards for: ； In the formula, C This indicates the penalty imposed when a drone collides with another drone; the drone involved in the collision will be removed from the mission. This indicates the closest distance between the drone and an obstacle or another drone. Indicates the collision threshold. Indicates the safety threshold, when The probability of a collision is high when the value is below the safety threshold, necessitating adjustments to the strategy. This represents the penalty value when a drone is in a danger zone but has not yet collided with another drone. and The product is used to calculate the penalty value of the drone at this time.

3. The multi-UAV target tracking method based on reinforcement learning as described in claim 1, characterized in that, The specific content of S21 includes: Drone Hunt From the preset location Starting from point B, perform b-step random Brownian motion, and at each time step... The inner random selects a discrete action from the action space. Based on the current The new state after the action is executed is obtained through the state transition function, and the position coordinates pos corresponding to each new state are recorded. t Add to the current initial position set.

4. The multi-UAV target tracking method based on reinforcement learning as described in claim 1, characterized in that, The specific content of the artificial potential field method for decision-making in S23 includes: For each discrete action in the action space, calculate the velocity vector. ,in , and These represent the velocities along the x, y, and z axes, respectively. Based on the velocity vector and the artificial potential force vector Calculate cosine similarity, where , and These are the components of the artificial potential force vector along the x, y, and z axes, respectively: ； In the defined action space A Each action, calculated different velocity vectors , obtain the corresponding , obtain the action space A Vector corresponding to each action To hunt down drones Observation The input policy network outputs a vector. ; For the original Weighted result ,in, Using weighted average The action probability distribution is obtained through softmax, and then the final action is obtained through sampling. .

5. The multi-UAV target tracking method based on reinforcement learning as described in claim 4, characterized in that, Obtain the force vector of the artificial potential field The specific content includes: The target and dynamic obstacles are modeled as a source of gravity and a source of repulsion, respectively. The direction and speed of the pursuit drone are adjusted under the combined force of gravity and repulsion. In the pursuit problem, there is one escaping target and n obstacles. Therefore, the pursuit drone... Artificial potential force vector for: ； in, Indicating the pursuit of drones The gravitational force experienced is the negative gradient of the gravitational potential field. Repulsive force representing an obstacle: ； ； In the formula, Represents the gravitational potential field function. Indicates the gravitational coefficient. From drones Pointing to target The distance vector; ； ； In the formula, The function representing the repulsive potential field of the obstacle. Indicates the repulsion coefficient. It is the radius of influence of the obstacle. From the obstacle Pointing to drones The distance vector.

6. The multi-UAV target tracking method based on reinforcement learning as described in claim 1, characterized in that, adversarial networks Same-policy network All observations were made using drones (p). As input, the output is the drone's actions, where the policy network outputs the drone's decision actions. Actions to counteract network output prediction The purpose is to mimic strategic network actions; The adversarial network is trained by minimizing the KL divergence, and the loss function of the adversarial network is... for: ； In the formula and These represent the network parameters when using the old strategy. and current adversarial network parameters At that time, given observations The probability distribution of time for all possible actions, where Represents any action. KL divergence is used to measure the difference between distributions; and These represent the network parameters of the old policy network and the current adversarial network, respectively. The adversarial network imitates the past policy rather than the current policy, thus maintaining relative stability. Loss function of policy network for: ； In the formula, For the expectation, The importance sampling ratio, This refers to the clip function. To limit the parameters, the clip operation is used to... Crop to Within the range, the policy update magnitude is controlled, and c is a hyperparameter that decreases linearly with training, thereby reducing the policy network's focus on the adversarial network.

Images