An unmanned aerial vehicle target tracking method based on game theory
By using a game theory-based unmanned aerial vehicle (UAV) target tracking method, a mathematical model of the UAV is established and a reward function is designed. By utilizing a policy network with a Soft Actor-Critic architecture, efficient collaborative tracking of multiple UAV systems in complex scenarios is achieved, solving the problem of single UAV load limitations and improving the system's intelligence and collaborative combat capabilities.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINESE PEOPLES LIBERATION ARMY UNIT 32180
- Filing Date
- 2025-04-09
- Publication Date
- 2026-05-19
AI Technical Summary
Individual drones have limited payload capacity and cannot independently complete complex and time-consuming tasks. There is an urgent need for intelligent multi-drone systems when tracking and avoiding obstacles, and existing technologies are unable to achieve efficient collaborative operations.
A game theory-based unmanned aerial vehicle (UAV) target tracking method is adopted. A mathematical model of the UAV is established, the game equilibrium point is defined, and a reward function is designed. The policy network and value network of the Soft Actor-Critic architecture are used to realize intelligent decision-making and collaborative tracking of multiple UAVs.
It enables multi-UAV systems to quickly and effectively pursue non-cooperative UAVs in complex scenarios, improving the system's intelligence level and collaborative combat capabilities.
Smart Images
Figure CN120295364B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of high-end equipment manufacturing technology, especially the field of unmanned aerial vehicle (UAV) technology, and specifically discloses a UAV target tracking method based on game theory. Background Technology
[0002] Unmanned aerial vehicles (UAVs) have seen significant development in recent years. In the civilian sector, their diverse functions have led to their widespread application in agriculture, transportation, and exploration. In the military field, UAVs offer advantages such as unmanned operation and lower cost, enabling them to perform a variety of more dangerous flight missions, making them a key development focus for many countries. However, the limited payload of a single UAV restricts its functionality, making it difficult to independently complete complex and time-consuming tasks. Therefore, the idea of multi-UAV systems or UAV swarms has emerged.
[0003] A swarm is a biological cluster similar to a bee colony or an ant colony. Individual members can communicate with each other, share information, and thus cooperate to complete various tasks. With the advantage of their large scale and mutual communication, swarms have the characteristics of high intelligence, rapid attack, strong survivability, and flexible use, making them a powerful weapon that can change the future of warfare.
[0004] In future air combat, due to the increasing intelligence of unmanned systems, non-cooperative drones will possess stronger anti-tracking capabilities. Multi-drone systems, with their large numbers and strong coordination, will have a greater advantage than single drones in scenarios involving the tracking and engagement of non-cooperative drones. In reality, while engaging enemy aircraft, drone swarms need to avoid obstacles and collisions in real time, which requires a sufficiently high level of intelligence within the swarm. Summary of the Invention
[0005] To address the aforementioned technical problems, this invention provides a target tracking method for unmanned aerial vehicles based on game theory, comprising the following steps:
[0006] S1. Establish a mathematical model for the unmanned aerial vehicle (UAV);
[0007] S2. Using the drone mathematical model in S1, define the game equilibrium point;
[0008] S3. Based on the definition in S2, define the reward function R;
[0009] S4. Design the policy network and value network based on the Soft Actor-Critic architecture;
[0010] S5. Deploy an intelligent agent network on the drone platform. The drone extracts state information based on the environment and its own state. tAs input to the policy network, the drone action decision 'a' is generated at each time step. t And control the drone to track the target.
[0011] Furthermore, S1 specifically includes:
[0012] Assuming the UAV is a two-dimensional motion model, the UAV state X is defined as including its two-dimensional position and heading angle: Define the control variable u as a two-dimensional velocity: u = [ω, v], where ω is the velocity in the x-direction and v is the velocity in the y-direction; the rate of change of the UAV's state X. Represented as: in, Let θ be the yaw angle of the drone, and let θ be the angle between the drone's velocity direction and the line connecting the tracker and the escapee. The relative position vector is: r = [d, θ], where d is the distance between the tracker and the escapee. Let θ be the angle between the drone and the escapee. Obtain the line-of-sight angle θ: Projecting the velocity vectors of the UAV and the target onto the line connecting them, we obtain the rate of change of the relative position vector r: X and r in the simulation are updated as follows:
[0013]
[0014] Where t is time, the drone's action space is a continuous two-dimensional space, and the tracker and the escapee have the same action space: A i =[ω i v i The drone's remaining energy and velocity are used as state inputs, and the drone's state space is:
[0015] Furthermore, S2 specifically includes:
[0016] Define the equilibrium point of a game as any strategy π for player i in the game. i All conditions are met:
[0017]
[0018] Where J is the payoff function for a particular tracker or escapee, representing the cost incurred during the pursuit. The drone target tracking task is a zero-sum game for both the tracker and the escapee, i.e.:
[0019]
[0020] Simplified to a two-player zero-sum game, the equilibrium point is represented as:
[0021]
[0022] In reinforcement learning, the payoff function is represented as:
[0023]
[0024] Furthermore, in S3, the reward function R includes a tracker reward function and an escapee reward function. Both the tracker reward function and the escapee reward function include a distance reward function, an energy reward function, a time reward function, a rate of change of action reward function, and a boundary reward function.
[0025] The distance reward function is:
[0026]
[0027] Where D is the boundary between continuous rewards and the final reward; when the distance between the drone and the target is greater than D, a reward that gradually increases as the distance decreases is given; when the distance between the drone and the target is less than D, it is initially determined that the drone has caught up, and the reward is doubled; β is a constant greater than 0, used to prevent the reward function from approaching infinity when d approaches 0, ensuring that the reward function satisfies the boundedness condition; W distance These are the weighting coefficients of the distance reward function;
[0028] The energy reward function is:
[0029]
[0030] Among them, W energy For positive constants; threshold E threshold Satisfy 0 <E threshold <1, when the remaining energy is less than the threshold E threshold At that time, a punishment shall be given;
[0031] The time reward function is:
[0032]
[0033] Among them, W time It is a positive constant; when the distance between the drone and the target is less than the threshold D threshold If the score is high enough, it is considered that the opponent has basically caught up and no time penalty is imposed; otherwise, it is considered that the opponent has not caught up and a time penalty is imposed.
[0034] The reward function for the rate of change of action is:
[0035] R uav_action =-W action (|v t-1 -v t |+|ω t-1 -ω t |) / dt,
[0036] Among them, Waction It is a positive number;
[0037] The boundary reward function is:
[0038]
[0039] Among them, W boarder It is a positive constant; d boarder ε is the shortest distance between the drone and the boundary; ε is a small quantity used to prevent the reward function from tending to infinity, and a penalty is given when the drone or target leaves the square area.
[0040] Furthermore, S3 also includes the reward for a single step taken by a single drone:
[0041] R uav =R uav_d +R uav_energy +R uav_time +R uav_action +R uav_boarder ,
[0042] In a zero-sum game, the sum of the payoff functions of both parties is 0, that is:
[0043]
[0044] The payoff function for a drone is equal to its long-term cumulative reward, i.e.:
[0045]
[0046] The game begins when the tracker is detected, and the reward function is:
[0047]
[0048] Where, d min The closest distance between all trackers and the escapee;
[0049] The escapee's single-step reward is:
[0050] R target =R target_pedg +R target_action .
[0051] Furthermore, in S4, both the policy network and the value network employ multi-layer fully connected neural networks to realize the mapping relationship between states and actions or cumulative reward values, constructing an agent network that includes one policy network, two softq networks, and two target networks.
[0052] This invention leverages the advantages of multiple drones pursuing a single drone, combining complex scenarios, multiple targets, and multiple constraints to achieve rapid pursuit of a non-cooperative single drone by multiple drones. This invention selects the soft actor-critc (SAC) algorithm as the deep reinforcement learning algorithm for the AC architecture and incorporates the policy entropy into the value function, enhancing the policy's exploratory nature and resulting in superior performance across many tasks. Attached Figure Description
[0053] Figure 1 This is a diagram defining the coordinate system of the UAV in this invention.
[0054] Figure 2 The distance reward function graph;
[0055] Figure 3 For measuring network architecture diagram;
[0056] Figure 4 Value network architecture diagram;
[0057] Figure 5 Here is a diagram of the soft actor-critic algorithm framework;
[0058] Figure 6 Flowchart for parameter update;
[0059] Figure 7 This is a diagram of a single-agent network structure. Detailed Implementation
[0060] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments. Unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by those skilled in the art to which this invention pertains.
[0061] In a specific embodiment of the present invention, the following steps are included:
[0062] S1. Establish a mathematical model of the UAV, simplifying the UAV into a two-dimensional motion model through assumptions;
[0063] The definition of a drone's state includes three dimensions: two-dimensional position and heading angle.
[0064]
[0065] Define the control variable as velocity in two dimensions;
[0066] u = [ω, v]
[0067] The rate of change of the drone's state can be expressed as:
[0068]
[0069] in The yaw angle of the drone is the angle between its direction of travel (velocity) and north, with clockwise being positive. Figure 1 As shown.
[0070] Next, we define the relative position vector. First, we define the line-of-sight angle θ as the angle between the drone's velocity direction and the line connecting the tracker and the escapee. The relative position vector is as follows:
[0071] r = [d, θ]
[0072] Define angle The line-of-sight angle θ can be obtained:
[0073]
[0074] Projecting the velocity vectors of the UAV and the target onto the line connecting them yields the rate of change of their relative positions:
[0075]
[0076] This yields a kinematic model between the two drones, which can be used between pairs of drones in a multi-drone system.
[0077] The position and relative position in the simulation are updated as follows:
[0078]
[0079] The relative position update can be achieved using a simpler method: first update the drone's position, then directly calculate the relative position relationship at the new moment using geometric relationships based on the new position. Therefore, the drone's control variable is u = [ω, v], and the actions are continuous. Thus, the drone's action space is a continuous two-dimensional space, and the pursuer and evader share the same action space.
[0080] A i =[ω i v i ]
[0081] The drone's remaining energy and speed are used as state inputs, enabling the drone to plan its subsequent actions more rationally. Therefore, the drone's state space is:
[0082]
[0083] S2. Using the mathematical model from step S1, define the game equilibrium point as: for any strategy π of player i in the game. i All of the following conditions are met:
[0084]
[0085] The right side of the inequality is the payoff function for all players under their optimal strategies. Since this task is also a zero-sum game, we have:
[0086]
[0087] This problem can be simplified by considering the tracker as an individual; it can be reduced to a two-person zero-sum game, the equilibrium of which can be expressed as:
[0088]
[0089] In reinforcement learning, the reward function is represented as:
[0090]
[0091] S3. Based on the definition in S2, design the reward function R, including the tracker reward function and the escapee reward function. The reward function includes distance reward, energy reward, time reward, rate of change of action reward, and boundary reward, etc., mainly as follows:
[0092] (1) Distance Reward
[0093] The tracker needs to catch up with the target, so a reward for catching the target must first be designed. To avoid convergence difficulties caused by sparse rewards, a continuous reward based on the distance between the tracker and the target should be designed, with the reward increasing the closer to the target:
[0094]
[0095] Here, D is the boundary between continuous rewards and the final reward. When the distance between the drone and the target is greater than D, a reward that gradually increases as the distance decreases is given. When the distance is less than D, it is initially determined that the drone has caught up, and the reward is doubled. β is in the denominator. To prevent the reward function from approaching infinity when d approaches 0, the reward function is bounded. W distance These are the weighting coefficients of the distance to the reward function. The curve of the reward function is as follows: Figure 2 As shown, W distance =0.6, β=0.05, D=0.05.
[0096] (2) Energy reward function
[0097] To minimize energy consumption during tracking, it is desirable for the remaining energy to exceed a certain threshold E. threshold (0 <E threshold <1), When the remaining energy is less than the threshold, a penalty is applied:
[0098]
[0099] Among them, W energy It is a positive number.
[0100] (3) Time reward function
[0101] To enable the drone to catch up with the target as quickly as possible, a time reward function is designed, which rewards the drone when the distance between it and the target is less than a threshold D. threshold If a player is considered to have essentially caught up, no time penalty is applied; otherwise, if they are considered not to have caught up, a time penalty is applied.
[0102]
[0103] Among them, W time It is a positive number.
[0104] (4) Rate of change of motion reward function
[0105] In the simulation, energy consumption calculation only considers the current speed and does not account for the energy consumption during acceleration and braking. Therefore, adding a motion change rate bonus here not only takes into account the energy consumption of acceleration and deceleration, but also makes the drone's motion changes as smooth as possible, resulting in a smoother trajectory.
[0106] R uav_action =-W action (|v t-1 -v t |+|ω t-1 -ω t |) / dt
[0107] Among them, W action It is a positive number.
[0108] (5) Boundary reward function
[0109] This mission is designed within a square area. A penalty will be imposed if the drone or target leaves this area.
[0110]
[0111] Among them, W boarder d is a positive constant. boarder ε is the shortest distance between the drone and the boundary; ε is a small quantity to prevent the reward function from tending to infinity.
[0112] In summary, the reward for a single step by a single drone is:
[0113] R uav =R uav_d +R uav_energy +R uav_time +R uav_action +R uav_boarder
[0114] The escapee's reward depends primarily on the pursuer. The cumulative long-term reward is the quantity the agent needs to maximize. In differential games, the cumulative long-term reward is equivalent to the payoff function. In a zero-sum game, the sum of the payoff functions of both parties is 0, i.e.:
[0115]
[0116] The payoff function for drones is equal to their long-term cumulative reward, that is:
[0117]
[0118] Therefore, the escapee's payoff function is the negative of the drone's total long-term cumulative reward, and the escapee's single-step reward function (PEDG part) is equal to the negative of the sum of the tracker's single-step rewards. For the escapee, the game begins when the tracker is detected, so the reward function is:
[0119]
[0120] Where, d min The closest distance between all trackers and the escapee.
[0121] At the same time, the escapee must also consider the range of their own movements, so the escapee's single-step reward is:
[0122] R target =R target_pedg +R target_action .
[0123] S4. Based on the Soft Actor-Critic (SAC) architecture, the policy network and value network are designed. Both employ multi-layer fully connected neural networks to realize the mapping relationship between states and actions or cumulative reward values, as shown below. Figure 3 , Figure 4 As shown.
[0124] Each drone agent has two softq-networks, and a corresponding target-network is set for each softq-network. The relationship between these softq-networks is as follows: Figure 5 As shown.
[0125] The joint state and joint action are input into the softq-network to obtain a predicted Q-value. The idea of updating the softq-network is to use the TD error as the loss function and perform backpropagation to update the network parameters. In order to stabilize the training, the Q-value of the next time step in the TD error is predicted by the target-network. The joint action and state of the next time step are input into two target-softq-networks. The smaller predicted value is used to calculate the target Q-value. Then, the difference between the predicted Q-value and the target Q-value is obtained to obtain the loss function. Then, backpropagation is performed to complete one parameter update of the softq-network. After a certain number of steps, the parameters of the softq-network are assigned to the target-network, thereby completing the parameter update of the target network.
[0126] Based on the loss function of the policy network:
[0127]
[0128] After using the reparameterization trick for improvement, we can directly differentiate the expected value with respect to the policy parameters to obtain the new loss function:
[0129]
[0130] The process of updating the policy network is as follows Figure 6 As shown.
[0131] The current state is input into the policy network, which outputs an action and a log probability. This action is the result of the reparameterization trick, i.e., f. φ (ε;s t The logarithmic probability corresponds to logπ. φ (f φ (ε;s t )|s t This action is input into two Q networks, and the smaller predicted value is taken and fed into the loss function of the policy network. Then, backpropagation is performed to update the parameters.
[0132] In summary, each agent has five neural networks: one policy network, two softQ networks, and their corresponding two target networks. The network structure is as follows: Figure 7 As shown.
[0133] S5. Utilizing a pre-trained agent network, deploy it on a drone platform. The drone extracts state information s based on the environment and its own state. tAs input to the policy network, the drone action decision 'a' is generated at each time step. t It controls the drone's actuators to fly and track targets.
[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.
Claims
1. A target tracking method for unmanned aerial vehicles based on game theory, characterized in that, Includes the following steps: S1. Establish a mathematical model for the unmanned aerial vehicle (UAV), specifically including: Assuming the UAV is a two-dimensional motion model, define the UAV state. Including two-dimensional position and heading angle: Define control variables Velocity in two dimensions: ,in, yes x Directional velocity, yes y Directional speed; UAV status rate of change Represented as: ,in, Define the line-of-sight angle for the drone's yaw angle. Let the angle between the drone's velocity direction and the line connecting the tracker and the escapee be given, and the relative position vector be: ,in, It is the distance between the pursuer and the escapee, defined as the angle. Obtain the viewing angle : Projecting the velocity vectors of the drone and the target onto the line connecting them yields the relative position vector. r rate of change: In simulation X and r Updated to: ， ， in, In terms of time, the drone's action space is a continuous two-dimensional space, and both the tracker and the escapee have the same action space: The drone's remaining energy and velocity are used as state inputs, and the drone's state space is: ; S2. Using the drone mathematical model in S1, define the game equilibrium point, specifically including: Define the equilibrium point of a game as the equilibrium point for the players in the game. i any strategy All conditions are met: ， in, This is a payoff function for a specific pursuer or escapee, representing the cost incurred during the pursuit. The drone target tracking task is a zero-sum game for both the pursuer and the escapee, i.e.: ， Simplified to a two-player zero-sum game, the equilibrium point is represented as: ， In reinforcement learning, the payoff function is represented as: ； S3. Based on the definition in S2, define a reward function R. R This includes a tracker reward function and an escapee reward function, each of which comprises a distance reward function, an energy reward function, a time reward function, a rate of change of motion reward function, and a boundary reward function. The distance reward function is: in, D It marks the boundary between continuous rewards and the final reward; when the distance between the drone and the target is greater than... D When the distance between the drone and the target is less than a certain value, a reward is given that gradually increases as the distance decreases; when the distance between the drone and the target is less than a certain value... D If the time is right, it is initially determined that the team has caught up, and the reward is doubled; It is a constant greater than 0, in order to prevent in d As the value approaches 0, the reward function approaches infinity, thus satisfying the bounded condition of the reward function; These are the weighting coefficients of the distance reward function; The energy reward function is: in, For positive constants; threshold Satisfying 0< <1, when the remaining energy is less than the threshold At that time, a punishment shall be given; The time reward function is: in, This is a normal value; when the distance between the drone and the target is less than the threshold. If the score is high enough, it is considered that the opponent has basically caught up and no time penalty is imposed; otherwise, it is considered that the opponent has not caught up and a time penalty is imposed. The reward function for the rate of change of action is: ， in, It is a positive number; The boundary reward function is: ， in, It is a positive number; The shortest distance between the drone and the boundary; A small amount is used to prevent the reward function from tending to infinity. In a square area, a penalty is given when the drone or target leaves the square area. S4. Design the policy network and value network based on the Soft Actor-Critic architecture; S5. Deploy an intelligent agent network on the drone platform, and the drone extracts state information based on the environment and its own state. s t As input to the policy network, it generates drone action decisions for each time step. And control the drone to track the target.
2. The unmanned aerial vehicle target tracking method based on game theory according to claim 1, characterized in that, S3 also includes a reward for each step taken by a single drone: ， In a zero-sum game, the sum of the payoff functions of both parties is 0, that is: The payoff function for a drone is equal to its long-term cumulative reward, i.e.: The game begins when the tracker is detected, and the reward function is: in, The closest distance between all trackers and the escapee; The escapee's single-step reward is: 。 3. The unmanned aerial vehicle target tracking method based on game theory according to claim 2, characterized in that, In S4, both the policy network and the value network use multi-layer fully connected neural networks to realize the mapping relationship from state to action or cumulative reward value, constructing an agent network, which includes one policy network, two softq networks, and two target networks.