PER-MATD3 deep reinforcement learning-based unmanned aerial vehicle dynamic game model and working method thereof

Abstract

The invention discloses an unmanned aerial vehicle dynamic game model based on PER-MATD3 deep reinforcement learning and a working method thereof, the unmanned aerial vehicle dynamic game model comprises an MATD3 algorithm module, a PER sorting module and a sample importance evaluation module, an algorithm framework of MATD3 is constructed, a Priority Expersion Replay (PER) method is introduced to improve the algorithm framework, and the algorithm framework of the MATD3 is optimized. According to the PER method, priority ranking is carried out on empirical data according to the importance of the empirical data, more important experiences can be selected for multiple times during sampling, and therefore the learning efficiency is improved, however, due to the fact that samples with high TD errors are frequently used in the PER method, even though the learning efficiency can be improved, deviation can be inevitably brought, and the error can be inevitably caused. In order to avoid oscillation and even discretization caused by deviation during training, an importance sampling method is introduced, and the sample weight with high TD error is properly reduced, so that the influence of each sample during gradient descent during training is the same, and the convergence of a training result is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicles, and in particular to a dynamic game model of unmanned aerial vehicles based on PER-MATD3 deep reinforcement learning and a working method thereof. Background Art

[0002] In recent years, the research on UAV swarm decision-making algorithms based on reinforcement learning has shown a booming development trend both at home and abroad, and has become one of the key technologies to improve the autonomous decision-making capabilities and collaborative combat efficiency of UAVs. In China, many research companies have invested in research in this cutting-edge field. They construct a multi-agent system model, regard a single UAV as an intelligent agent, and apply reinforcement learning algorithms to solve the sequential decision-making problem of UAV swarms in complex environments. These studies not only cover multiple aspects such as mission planning, path planning, and collaborative combat of UAV swarms, but also propose a variety of optimization algorithms, such as GOOSE, APO, DCS, etc., to cope with the challenges faced by UAV swarms in complex battlefield environments. Through these studies, China has achieved remarkable results in the autonomous decision-making and control of UAV swarms, laying a solid foundation for the widespread application of UAV swarms.

[0003] At the same time, foreign countries have also achieved fruitful results in the research of UAV swarm decision-making algorithms based on reinforcement learning. The U.S. Air Force Scientific Advisory Board and other institutions regard UAV swarms as an important trend in the future application of UAVs and have conducted in-depth exploration in this direction. Europe's Information Society Technology Program (IST) has also launched similar projects, such as COMETS, which aims to improve the collaborative combat capabilities of UAV swarms through real-time control technology. Foreign research has made important progress in hierarchical control, multi-objective genetic algorithms, and communication network topology, and has achieved efficient collaborative combat of UAV swarms in complex battlefield environments. These research results not only improve the operational flexibility and adaptability of UAV swarms, but also provide strong support for the widespread application of UAV swarms.

[0004] MATD3 is a classic deep reinforcement learning algorithm, which applies the TD3 algorithm to the field of multi-agents. The MATD3 algorithm draws on the core idea of ​​centralized training and distributed execution of Multi-Agent Deep Deterministic Policy Gradient (MADDPG). During the training phase, the algorithm can access the information of all agents, including the state and action of the agents, in order to learn more complex strategies, but in the execution phase, each agent makes independent decisions based only on its own observations and learned strategies. Therefore, during the training process of MATD3, the Critic not only needs to consider the state and action information of the current agent, but also needs to comprehensively consider the state and action information of other agents to estimate the action quality of the Actor network and guide the Actor network to produce better actions. In order to solve the problem of overestimation in MADDPG, the Critic in MATD3 uses two Q networks to estimate the value of the action. For the value estimation of each step, the smaller of the two Q values ​​is selected. The algorithm framework of MATD3 is as follows Figure 1 As shown in the figure. In order to reuse historical experience data, the DQN algorithm first introduced the experience replay method, which stores historical data in the experience pool, greatly improving the experience utilization rate. However, the random sampling method ignores the importance of experience data, which may lead to uneven quality of sampled experience data, thereby reducing learning efficiency.

[0005] In order to solve the problem of uneven quality of sampled empirical data in the prior art, the present invention introduces the PER method based on the MATD3 deep reinforcement learning algorithm to improve learning efficiency. Summary of the invention

[0006] The purpose of the present invention is to provide a UAV dynamic game model based on PER-MATD3 deep reinforcement learning to address the technical defects in the prior art.

[0007] Another object of the present invention is to provide a working method of a drone dynamic game model based on PER-MATD3 deep reinforcement learning.

[0008] The technical solution adopted to achieve the purpose of the present invention is:

[0009] A UAV dynamic game model based on PER-MATD3 deep reinforcement learning, including MATD3 algorithm module, PER sorting module and sample importance evaluation module:

[0010] The MATD3 algorithm module includes N agents and an experience pool D. Each agent represents a drone in a game model. Each agent participates in the calculation of the Critic1 network, the Critic2 network and the Actor network. The Critic1 network and the Critic2 network are composed of φ i1 and φ i2 Parameterization, the Actor network is parameterized by θ, the Actor network obtains the observation o of the i-th agent at this moment, and outputs the action a of the i-th agent at the next moment. The observation o and action a of the i-th agent interact with the Critic1 network and Critic2 network of the agent and other agents. The Actor network in which each agent participates transmits information to the experience pool D, and the Critic1 network and Critic2 network of each agent randomly obtain K samples from the experience pool D;

[0011] The experience pool D includes samples from all agents Among them, s1 is the state of the first sample at the current moment, a1 is the action of the first sample at the current moment, s′1 is the state of the first sample at the next moment, r1 is the reward of the first sample, and s K is the state of the Kth sample at the current moment, a K is the action of the Kth sample at the current moment, s′ K is the state of the Kth sample at the next moment, r K is the reward for the Kth sample, s Tadd For the T add The state of the sample at the current moment, For the T add The action of the sample at the current moment, For the T add The state of the sample at the next moment, For the T add The reward for each sample, T add is the total number of samples in the experience pool D;

[0012] The PER sorting module prioritizes K samples according to the importance of each sample. k Sorting, the priority p of samples with larger TD error values k The higher the priority p of the sample with smaller TD error value k Lower, calculate the sampling probability of the kth sample

[0013] The sample importance evaluation module calculates the weight ω of the kth sample k , to reduce the weight of samples with high TD error, and at the same time k Perform normalization.

[0014] In the above technical solution, the observation o in the MATD3 algorithm module includes state s, action a and reward r, the state s includes the drone position, yaw angle, pitch angle, and speed; the action a includes the drone's maneuvering strategy, corresponding to the pure strategy maneuvering action in the game model.

[0015] In the above technical solution, in the MATD3 algorithm module, for the i-th agent, a sample is randomly selected from the experience pool D, and the target action a′ i (s′ i ) is shown in the following formula:

[0016]

[0017] In the formula, a′ i (s′ i ) is the target state s′ of the i-th agent i The target action under i The target network strategy, s′ i is the target state after the transfer of the ith agent, clip is the maximum available tangential overload, and is a function that limits the target action to a certain range. In order to satisfy the random noise of normal distribution, the variance gradually decays from the initial value to the minimum value during the training process, and c is the noise edge value.

[0018] In the above technical solution, the loss function L(φi1) of the Critic1 network in the MATD3 algorithm module is as follows:

[0019]

[0020] The loss function of the Critic2 network in the MATD3 algorithm module As shown below:

[0021]

[0022] In the formula, and are the loss functions of the Critic1 network and the Critic2 network respectively, E(·) is the mathematical expectation, y i is the target Q value of the ith agent, s=(s1,s2,...,s N ) and s′=(s′1,s′2,...,s′ N ) are the joint states of N agents at the current moment and the next moment respectively; r = {r1, r2, ..., r N} represents the joint reward of N agents at the current moment, a=(a1,a2,...,aN ) and a′=(a′1,a′2,...,a′ N ) are the joint actions of N agents at the current moment and the next moment, μ={μ1,μ2,...,μ N}, μ is the parameter θ = {θ1, θ2,...,θ N}’s strategy set of N agents; and They are Critic1 network and Critic2 network respectively, and Parameterization, D is the experience pool, including samples from all agents K is the number of samples randomly drawn by the ith agent from the experience pool, ω k is the weight of the kth sample;

[0023] The target Q value y of the i-th agent i The calculation of is as follows:

[0024]

[0025] In the formula, r i is the joint reward of the i-th agent at the current moment, r i =r1,r2,...,r N , i=1-N; μ′={μ′1,μ′2,...,μ′ N}, μ′ is the parameter θ = {θ1, θ2,...,θ N}; a′=(a′1,a′2,...,a′ N ) is the joint action of N agents at the next moment, s′=(s′1,s′2,...,s′ N ) is the joint state of N agents at the next moment, and are the target network of Critic1 and the target network of Critic2 respectively, and parameterization, γ is the discount factor.

[0026] In the above technical solution, the parameters of the Critic1 network and the Critic2 network in the MATD3 algorithm module are and φ i2 Update the parameters of the Critic1 network by gradient descent The calculation of is as follows:

[0027]

[0028] In the formula, are the parameters of the Critic1 network, is the learning rate of the Critic1 network, is the loss function of the Critic1 network Relative to the parameters The gradient of

[0029] Parameters of Critic2 network The calculation is as follows:

[0030]

[0031] In the formula, are the parameters of the Critic2 network, is the learning rate of the Critic2 network, is the loss function of the Critic2 network Relative to the parameters gradient.

[0032] In the above technical solution, the parameter θ of the Actor network in the MATD3 algorithm module is i The calculation of is as follows:

[0033]

[0034] In the formula, θ i is the parameter of the Actor network, α A is the Actor network learning rate, is the policy gradient of the expected reward of the i-th agent;

[0035] The policy gradient of the expected reward of the i-th agent The calculation formula is as follows:

[0036]

[0037] In the formula, J(θ i ) is the cumulative expected reward of the ith agent at time T, E(·) is the mathematical expectation, s i =(s1,s2,...,s N ), is the joint state of N agents at time T; a i =(a1,a2,...,a N ), is the joint action of N agents at time T; μ i ={μ1,μ2,...,μ N},μ i The parameters are θ={θ1,θ2,...,θ N The strategy set of N agents in time T; is the Critic1 network, composed of Parameterization;

[0038] The cumulative expected reward J(θ) of the i-th agent at time T i ) is shown in the following formula:

[0039]

[0040] In the formula, t is time, E(·) is the mathematical expectation, γ t is the discount factor at time t, is the joint reward of the ith agent at time t.

[0041] In the above technical solution, the Actor target network parameter θ′ in the MATD3 algorithm module i and Critic target network parameter φ im The update formula is as follows:

[0042] θ′ i =τθ i +(1-τ)θ′ i

[0043]

[0044] In the formula, θ i ′ is the Actor target network parameter, φ im τ is the soft update coefficient of the Critic target network parameter, 0<τ≤1, m=1,2.

[0045] In the above technical solution, the priority p of sample k in the PER sorting module is k As shown below:

[0046]

[0047] In the formula, p k is the priority of the kth sample, are the TD errors of the Critic1 network and the Critic2 network for the i-th agent relative to the k-th sample;

[0048] The calculation is as follows:

[0049]

[0050] In the formula, and Respectively by and Parameterized Critic1 network and Critic2 network, yi is the target Q value of the i-th agent;

[0051] The sampling probability of the kth sample in the PER sorting module The calculation of is as follows:

[0052]

[0053] In the formula, is the sampling probability of the kth sample, p k is the priority of the kth sample, α is a 0-1 hyperparameter that affects the priority, and K is the number of samples randomly drawn from the experience pool.

[0054] In the above technical solution, in the sample importance evaluation module, the sampling weight ω of the kth sample k The calculation of is as follows:

[0055]

[0056] In the formula, ω k is the sampling weight of the kth sample, and β is a 0-1 hyperparameter;

[0057] For the sampling weight ω k Normalization is performed as shown below:

[0058]

[0059] In the formula, n is the sample number with the largest weight.

[0060] A working method of a UAV dynamic game model based on PER-MATD3 deep reinforcement learning, comprising the following steps:

[0061] Step 1, initialize network parameters:

[0062] Initialize the network parameters of each agent and create Critic1 network, Critic2 network and Actor network;

[0063] Step 2, initialize the target network:

[0064] Initialize the target network parameters of the i-th agent and θ i 'And create Critic1 target network, Critic2 target network and Actor target network;

[0065] Step 3, initialize hyperparameters:

[0066] Initialize the priority coefficient α, sampling weight coefficient β, experience pool D, soft update coefficient τ, discount factor γ, noise edge value c, number of samples extracted each time K, normal variance σ;

[0067] Step 4, training loop, performs the following steps in each Episode:

[0068] S41 generates random noise for motion exploration;

[0069] S42 obtains the initial state of each agent s=(s1,s2,...,s N ), N is the total number of agents;

[0070] S43 Execute action Select an action according to the current strategy and execute the target action a′ i (s′ i );

[0071] S44 will collect data from each interaction <s1,a1,s′1,r1,...,s K ,a K ,s′ K ,r K >Stored in the experience pool D, K is the number of randomly selected samples;

[0072] Step 5, sample selection and update:

[0073] S51, randomly select K samples from the experience pool D, and the probability of selecting the kth sample is p k ;

[0074] S52, calculate the sampling weight ω of the kth sample k and TD error δ k ;

[0075] S53, according to |δ k |Update the priority of the kth sample;

[0076] S54, calculating the target Q value;

[0077] S55, minimize the loss function and Update Critic1 network and Critic2 network, calculate the cumulative expected reward J(θ i );

[0078] Step 6: Update target network parameters and θ′ i ;

[0079] Step 7: Build a UAV dynamic game model based on steps 1-6, solve the perfect Bayesian-Nash equilibrium, and determine the UAV maneuver decision.

[0080] Compared with the prior art, the present invention has the following beneficial effects:

[0081] 1. The algorithm framework of MATD3 was constructed, and the Prioritized Experience Replay (PER) method was introduced to improve it. The PER method prioritizes each experience data according to its importance and can select more important experiences multiple times during sampling, thereby improving learning efficiency and further improving the efficiency of UAV air combat maneuver decision-making.

[0082] 2. Since the PER method frequently uses samples with high TD errors, although it can improve learning efficiency, it will inevitably bring deviations. In order to avoid deviations during training that lead to oscillation or even discreteness, the importance sampling method is introduced to appropriately reduce the weights of samples with high TD errors so that each sample has the same influence during gradient descent during training, thereby ensuring the convergence of the training results. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 Shown is the algorithm module framework diagram of the drone dynamic game model MATD3 based on PER-MATD3 deep reinforcement learning of the present invention.

[0084] Figure 2 The figure shows the average reward of the red drone using the drone dynamic game model based on PER-MATD3 deep reinforcement learning of the present invention.

[0085] Figure 3 The figure shows the average reward of the blue drone using the drone dynamic game model based on PER-MATD3 deep reinforcement learning of the present invention.

[0086] Figure 4 The figure shows the average success, failure and draw rates of the red drone using the drone dynamic game model based on PER-MATD3 deep reinforcement learning of the present invention. DETAILED DESCRIPTION

[0087] The present invention is further described in detail below in conjunction with specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0088] Example 1

[0089] A UAV dynamic game model based on PER-MATD3 deep reinforcement learning, including MATD3 algorithm module, PER sorting module and sample importance evaluation module:

[0090] The MATD3 algorithm module includes N agents and an experience pool D. Each agent represents a drone in a game model. Each agent includes a Critic1 network, a Critic2 network, and an Actor network. The Critic1 network and the Critic2 network are composed of φ i1 and φ i2 Parameterization, the Actor network is parameterized by θ, the Actor network obtains the observation o of the agent at this moment, and outputs the action a of the agent at the next moment. The observation o and action a of each agent interact with the Critic1 network and Critic2 network of the agent and other agents. The Actor network of each agent passes the information to the experience pool D, and the Critic1 network and Critic2 network of each agent randomly obtain K samples from the experience pool D.

[0091] The experience pool D includes samples from all agents Among them, s1 is the state of the first sample at the current moment, a1 is the action of the first sample at the current moment, s′1 is the state of the first sample at the next moment, r1 is the reward of the first sample, and s K is the current state of the kth sample, a K is the action of the kth sample at the current moment, s′ K is the state of the kth sample at the next moment, r K is the reward for the kth sample, For the T add The current state of the sample, a Tadd For the T add The current action of samples, For the T add The state of the sample at the next moment, For the T add The reward for each sample, T add is the total number of samples in the experience pool D;

[0092] The PER sorting module prioritizes K samples according to the importance of each sample. k Sorting, the priority p of samples with larger TD error values k The higher the priority p of the sample with smaller TD error value k Lower, calculate the sampling probability of sample k

[0093] The sample importance evaluation module calculates the weight ω of the kth sample k , to reduce the weight of samples with high TD error, and at the same time k Perform normalization.

[0094] For the i-th agent, randomly select samples from the experience pool D, and the target action a′ i (s′ i ) is calculated as follows:

[0095]

[0096] In the formula, a′ i (s′ i ) is the target state s′ of the i-th agent i The target action under i The target network strategy, s′ i is the target state after the transfer of the ith agent, clip is the maximum available tangential overload, and is a function that limits the target action to a certain range. In order to satisfy the random noise of normal distribution, the variance gradually decays from the initial value to the minimum value during the training process, and c is the noise edge value.

[0097] The loss function of the Critic1 network in the MATD3 algorithm module The calculation of is as follows:

[0098]

[0099] The loss function of the Critic2 network in the MATD3 algorithm module As shown below:

[0100]

[0101] In the formula, and are the loss functions of the Critic1 network and the Critic2 network respectively, E(·) is the mathematical expectation, y i is the target Q value of the ith agent, s=(s1,s2,...,s N ) and s′=(s′1,s′2,…,s′ N ) are the joint states of N agents at the current moment and the next moment respectively; r = {r1, r2, ..., r N} represents the joint reward of N agents at the current moment, a=(a1,a2,...,a N ) and a′=(a′1,a′2,...,a′ N ) are the joint actions of N agents at the current moment and the next moment, μ={μ1,μ2,...,μ N}, μ is the parameter θ = {θ1, θ2,...,θ N}’s strategy set of N agents; and They are Critic1 network and Critic2 network respectively, and Parameterization, D is the experience pool, including samples from all agents K is the number of samples randomly drawn from the experience pool.

[0102] The target Q value y of the i-th agent i The calculation of is as follows:

[0103]

[0104] In the formula, y i is the target Q value of the ith agent, r i is the joint reward of the i-th agent at the current moment, r i =r1,r2,...,r N , i=1~N; μ′={μ′1,μ′2,...,μ′ N}, μ′ is the parameter θ = {θ1, θ2,...,θ N}; a′=(a′1,a′2,...,a′ N ) is the joint action of N agents at the next moment, s′=(s′1,s′2,...,s′ N ) is the joint state of N agents at the next moment, and are the target network of Critic1 and the target network of Critic2 respectively, and parameterization, γ is the discount factor.

[0105] Parameters of Critic1 and Critic2 networks and Update the parameters of the Critic1 network by gradient descent The calculation of is as follows:

[0106]

[0107] In the formula, are the parameters of the Critic1 network, is the learning rate of the Critic1 network, is the loss function of the Critic1 network Relative to the parameters gradient.

[0108] Parameters of Critic2 network The calculation is as follows:

[0109]

[0110] In the formula, are the parameters of the Critic2 network, is the learning rate of the Critic2 network, is the loss function of the Critic2 network Relative to the parameters gradient.

[0111] The cumulative expected reward J(θ) of the i-th agent at time T i ) is shown in the following formula:

[0112]

[0113] In the formula, J(θ i ) is the cumulative expected reward of the ith agent at time T, t is time, E(·) is the mathematical expectation, γ t is the discount factor at time t, r i t is the joint reward of the ith agent at time t.

[0114] The policy gradient of the expected reward of the i-th agent The calculation formula is as follows:

[0115]

[0116] In the formula, is the policy gradient of the expected reward of the ith agent, E(·) is the mathematical expectation, s i =(s1,s2,…,s N ), is the joint state of N agents at time T; a i =(a1,a2,…,a N ), is the joint action of N agents at time T; μ i ={μ1,μ2,…,μ N},μ i The parameters are θ={θ1,θ2,…,θ N The strategy set of N agents in time T; is the Critic1 network, composed of Parameterized.

[0117] The parameters of the Actor network θ i The calculation of is as follows:

[0118]

[0119] In the formula, θ i is the parameter of the Actor network, αA is the Actor network learning rate, is the expected reward of the Actor network relative to the parameter θ i gradient.

[0120] In order to reuse historical experience data, the DQN algorithm first introduced the experience replay method, which stores historical data in the experience replay pool, greatly improving the experience utilization rate. However, the random sampling method ignores the importance of experience data, which may lead to uneven quality of sampled experience data, thereby reducing learning efficiency. In order to solve the above problems, this embodiment introduces the PER method, which prioritizes each experience data according to its importance, and can select more important experience multiple times when sampling, thereby improving learning efficiency.

[0121] The TD error is expressed as the difference between the estimated Q value and the target Q value, and is usually used to update the Critic network's estimate, so it can be used as a correction to the estimate, so the TD error can reflect the degree to which the algorithm evaluates the true value of the experience. Samples with larger TD error values ​​have higher priority, and conversely, samples with smaller TD error values ​​have lower priority. The TD error of the i-th agent relative to the k-th sample is shown below:

[0122]

[0123] In the formula, The TD errors of the Critc1 network and the Critic2 network are used respectively. The larger value between the two TD error absolute values ​​is selected as the priority of the kth sample. The priority of the kth sample p k As shown below:

[0124]

[0125] In the formula, p k is the priority of the kth sample, are the TD errors of the Critic1 network and the Critic2 network for the i-th agent relative to the k-th sample;

[0126] The kth sample sampling probability The calculation of is as follows:

[0127]

[0128] In the formula, is the sampling probability of the kth sample, p k is the priority of the kth sample, α is a 0-1 hyperparameter that affects the priority, and K is the number of samples randomly selected from the experience pool.

[0129] The use of the PER method will cause the problem of changing the state distribution. Before the PER method was proposed, the experience replay pool was used to solve the data correlation problem and ensure that the sample data was independent and identically distributed. However, the use of the PER method will frequently use samples with high TD errors. Although it can improve learning efficiency, it will inevitably bring deviations. In order to avoid oscillation or even discreteness caused by deviations during training, the importance sampling method is introduced to appropriately reduce the weight of samples with high TD errors so that each sample has the same influence during gradient descent during training, thereby ensuring the convergence of the training results. The sampling weight ω of the kth sample k The calculation of is as follows:

[0130]

[0131] In the formula, ω k is the sampling weight of the kth sample, β is a 0-1 hyperparameter that determines the influence of the PER method on the convergence result. The larger β is, the greater the sampling weight of the low priority sample is. k is the priority of the kth sample, K is the number of samples randomly selected from the experience pool;

[0132] In order to improve the stability of training, the weights are normalized as shown in the following formula:

[0133]

[0134] In the formula, ω k is the sampling weight of the kth sample, and n is the sample number with the largest weight.

[0135] Therefore, the loss function of the Critic1 network and the Critic2 network in the MATD3 algorithm module can be rewritten as:

[0136]

[0137] In the formula, and are the loss functions of the Critic1 network and the Critic2 network respectively, E(·) is the mathematical expectation, y i is the target Q value of the ith agent, s=(s1,s2,...,s N ) and s′=(s′1,s′2,...,s′ N ) are the joint states of N agents at the current moment and the next moment respectively; r = {r1, r2, ..., r N} is the joint reward of N agents at the current moment, a=(a1,a2,...,a N ) and a′=(a′1,a′2,...,a′ N) are the joint actions of N agents at the current moment and the next moment, μ={μ1,μ2,...,μ N}, μ is the parameter θ = {θ1, θ2,…,θ N}; μ′={μ′1,μ′2,...,μ′ N}, μ′ is the parameter θ={θ′1,θ′2,...,θ′ N The strategy set of the target network of N agents; and They are Critic1 network and Critic2 network respectively, and Parameterization, D is the experience pool, including samples from all agents

[0138] K is the number of samples randomly drawn from the experience pool, ω k is the weight of the kth sample.

[0139] Example 2

[0140] A working method of a UAV dynamic game model based on PER-MATD3 deep reinforcement learning, comprising the following steps:

[0141] Step 1, initialize network parameters:

[0142] Initialize the network parameters of each agent and θ i , and create Critic1 network, Critic2 network and Actor network;

[0143] Step 2, initialize the target network:

[0144] Initialize the target network parameters of each agent and θ′ i And create Critic1 target network, Critic2 target network and Actor target network;

[0145] Step 3, initialize hyperparameters:

[0146] Initialize the priority coefficient α, sampling weight coefficient β, experience pool D, soft update coefficient τ, discount factor γ, noise edge value c, number of samples extracted each time K, normal variance σ;

[0147] Step 4, training loop, performs the following steps in each Episode:

[0148] S41 generates random noise for motion exploration;

[0149] S42 obtains the initial state of each agent s=(s1,s2,...,s N ), N is the number of agents;

[0150] S43 Execute action Select an action according to the current strategy and execute the target action a′ i (s′ i );

[0151] S44 will collect data from each interaction <s1,a1,s′1,r1,...,s K ,a K ,s′ K ,r K >Stored in the experience pool D, K is the number of samples;

[0152] Step 5, sample selection and update:

[0153] S51, randomly select K samples from the experience pool D, and the probability of selecting the kth sample is p k ;

[0154] S52, calculate the sampling weight ω of the kth sample k and TD error δ k ;

[0155] S53, according to |δ k |Update the priority of the kth sample;

[0156] S54, calculating the target Q value;

[0157] S55, minimize the loss function and Update Critic1 network and Critic2 network, calculate the cumulative expected reward J(θ i );

[0158] Step 6: Update target network parameters and θ′ i ;

[0159] Step 7: Build a UAV dynamic game model based on steps 1-6, solve the perfect Bayesian-Nash equilibrium, and determine the UAV maneuver decision.

[0160] The PER-MATD3 network training model is built based on the proposed game model and is used to solve the perfect Bayesian-Nash equilibrium. In the PER-MATD3 model, each agent corresponds to a game participant, representing a drone in an air battle. The state space is set to the state information of the drone in the air battle, including the drone's position, yaw angle, pitch angle, and speed, and the action space is set to the drone's maneuvering strategy, corresponding to the pure strategy maneuvering action in the proposed game model. Therefore, the training result of the model corresponds to the perfect Bayesian-Nash equilibrium of the mixed strategy.

[0161] The reward of the PER-MATD3 model determines the expectation of iterative convergence of policy learning. The agent is trained through iterative learning to maximize the cumulative reward, which is consistent with the goal of finding a perfect Bayesian-Nash equilibrium for mixed strategies.

[0162] Example 3

[0163] Air combat scene and parameter settings

[0164] Assuming that the performance of both drones is the same, the minimum available tangential overload of the drone is 1, the maximum available tangential overload is 2, the minimum available normal overload is 0, and the maximum available normal overload is 8. The initial state settings of both drones are shown in 1.

[0165] Table 1 Initial situation information of the drones of both sides

[0166]

[0167] The hyperparameter settings of the PER-MATD3 algorithm are shown in Table 2.

[0168] Table 2 Hyperparameter settings

[0169]

[0170] The Actor network structure of each agent is the same, as shown in Table 3.

[0171] Table 3 Actor network structure of MATD3

[0172]

[0173] The critic network structure of each agent is the same, as shown in Table 4.

[0174] Table 4 Critic network structure of MATD3

[0175]

[0176] Simulation Results Analysis

[0177] The goal of reinforcement learning is to maximize the cumulative expected reward. Therefore, the average reward of the cumulative rounds is used as an indicator to measure the training effect. The average of the cumulative expected rewards of the last 200 rounds and 10 rounds is used as an indicator. First, the perfect Bayesian-Nash equilibrium of the drone 2v2 air combat scene is trained by the PER-MATD3 algorithm. The average reward of the red drone is as follows: Figure 2 In order to verify the effectiveness of the solution strategy for the perfect Bayesian-Nash equilibrium, the blue team uses the MATD3 algorithm as a comparison algorithm. The average value of the blue team's drone reward for the perfect Bayesian-Nash equilibrium strategy using the MATD3 algorithm is shown in Figure 3 As shown in the figure, the horizontal axis represents the number of training rounds, and the vertical axis represents the average cumulative expected reward value of the red drone. It can be seen that the average reward value in the initial stage of training is relatively low, and the drone has not learned experience through the algorithm. After a certain number of rounds, the average reward value of the drone gradually increases, and a more intelligent strategy is learned. The two algorithms can learn better strategies in about 12,000 rounds and 15,000 rounds respectively, and gradually reach convergence. Compared with the MATD3 algorithm, the PER-MATD3 algorithm converges to the perfect Bayesian-Nash equilibrium with fewer iterations and oscillations through training, and the average reward after convergence is the highest, the convergence speed is the fastest, and it has better stability.

[0178] The red team uses the PER-MATD3 algorithm, and the blue team uses the MATD3 algorithm for training. The changes in the success rate, failure rate, and draw rate of the red team's drones in the last 200 rounds are shown below: Figure 4 As shown in the figure, it can be seen that at the beginning of training, the success rate and failure rate of the red team are both around 20%, indicating that at this stage, the performance of the drone is relatively balanced, with no obvious advantages or disadvantages. As the training progresses, the success rate of the red team gradually increases, while the failure rate remains basically unchanged. In the end, the success rate of the red team's drones reaches about 60%. Therefore, under the same situation assessment function, the drone maneuver strategy obtained by solving the perfect Bayesian-Nash equilibrium based on the PER-MATD3 algorithm can effectively improve the decision-making ability of the drone, thereby expanding the drone's air combat advantage and improving the success rate of air combat.

[0179] The above is only a preferred embodiment of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A UAV dynamic game model based on PER-MATD3 deep reinforcement learning, characterized in that: Including MATD3 algorithm module, PER ranking module and sample importance evaluation module: The MATD3 algorithm module includes N agents and an experience pool D. Each agent represents a drone in a game model. Each agent participates in the calculation of the Critic1 network, the Critic2 network and the Actor network. The Critic1 network and the Critic2 network are composed of and Parameterization, the Actor network is parameterized by θ, the Actor network obtains the observation o of the i-th agent at this moment, and outputs the action a of the i-th agent at the next moment. The observation o and action a of the i-th agent interact with the Critic1 network and Critic2 network of the agent and other agents. The Actor network in which each agent participates transmits information to the experience pool D, and the Critic1 network and Critic2 network of each agent randomly obtain K samples from the experience pool D; The experience pool D includes samples from all agents Among them, s1 is the state of the first sample at the current moment, a1 is the action of the first sample at the current moment, s′1 is the state of the first sample at the next moment, r1 is the reward of the first sample, and s K is the state of the Kth sample at the current moment, a K is the action of the Kth sample at the current moment, s′ K is the state of the Kth sample at the next moment, r K is the reward for the Kth sample, For the T add The state of the sample at the current moment, For the T add The action of the sample at the current moment, For the T add The state of the sample at the next moment, For the T add The reward for each sample, T add is the total number of samples in the experience pool D; The PER sorting module prioritizes K samples according to the importance of each sample. k Sorting, the priority p of samples with larger TD error values k The higher the priority p of the sample with smaller TD error value k Lower, calculate the sampling probability of the kth sample The sample importance evaluation module calculates the weight ω of the kth sample k , to reduce the weight of samples with high TD error, and at the same time k Perform normalization.

2. The UAV dynamic game model based on PER-MATD3 deep reinforcement learning as claimed in claim 1, characterized in that: The observation o in the MATD3 algorithm module includes state s, action a and reward r. The state s includes the position, yaw angle, pitch angle and speed of the drone. The action a includes the maneuvering strategy of the drone, which corresponds to the pure strategy maneuvering action in the game model.

3. The UAV dynamic game model based on PER-MATD3 deep reinforcement learning as claimed in claim 1, characterized in that: In the MATD3 algorithm module, for the i-th agent, a sample is randomly selected from the experience pool D, and the target action a i ′(s i ′) is shown in the following formula: In the formula, a i ′(s i ′) is the i-th agent in the target state s i ′ is the target action under the condition, μ′ is the parameter θ i ′’s target network strategy, s i ′ is the target state after the transfer of the ith agent, clip is the maximum available tangential overload, and is a function that limits the target action to a certain range. In order to satisfy the random noise of normal distribution, the variance gradually decays from the initial value to the minimum value during the training process, and c is the noise edge value.

4. The UAV dynamic game model based on PER-MATD3 deep reinforcement learning as claimed in claim 1, characterized in that: The loss function of the Critic1 network in the MATD3 algorithm module The formula is as follows: The loss function of the Critic2 network in the MATD3 algorithm module As shown below: In the formula, and are the loss functions of the Critic1 network and the Critic2 network respectively, E(·) is the mathematical expectation, y i is the target Q value of the ith agent, s=(s1,s2,…,s N ) and s′=(s′1,s′2,…,s′ N ) are the joint states of N agents at the current moment and the next moment respectively; r = {r1, r2, …, r N } represents the joint reward of N agents at the current moment, a=(a1,a2,...,a N ) and a′=(a′1,a′2,...,a′ N ) are the joint actions of N agents at the current moment and the next moment, μ={μ1,μ2,...,μ N }, μ is the parameter θ = {θ1, θ2,…,θ N }’s strategy set of N agents; and They are Critic1 network and Critic2 network respectively, and Parameterization, D is the experience pool, including samples from all agents K is the number of samples randomly drawn by the ith agent from the experience pool, ω k is the weight of the kth sample; The target Q value y of the i-th agent i The calculation of is as follows: In the formula, r i is the joint reward of the i-th agent at the current moment, r i =r1,r2,...,r N , i=1-N; μ′={μ′1,μ′2,...,μ′ N }, μ′ is the parameter θ = {θ1, θ2,...,θ N }; a′=(a′1,a′2,...,a′ N ) is the joint action of N agents at the next moment, s′=(s′1,s′2,…,s′ N ) is the joint state of N agents at the next moment, and are the target network of Critic1 and the target network of Critic2 respectively, and parameterization, γ is the discount factor.

5. The UAV dynamic game model based on PER-MATD3 deep reinforcement learning as claimed in claim 4 is characterized in that: Parameters of Critic1 network and Critic2 network in the MATD3 algorithm module and Update the parameters of the Critic1 network by gradient descent The calculation of is as follows: In the formula, are the parameters of the Critic1 network, is the learning rate of the Critic1 network, is the loss function of the Critic1 network Relative to the parameters The gradient of Parameters of Critic2 network The calculation is as follows: In the formula, are the parameters of the Critic2 network, is the learning rate of the Critic2 network, is the loss function of the Critic2 network Relative to the parameters gradient.

6. The UAV dynamic game model based on PER-MATD3 deep reinforcement learning as claimed in claim 4, characterized in that: The parameter θ of the Actor network in the MATD3 algorithm module i The calculation of is as follows: In the formula, θ i is the parameter of the Actor network, α A is the Actor network learning rate, is the policy gradient of the expected reward of the i-th agent; The policy gradient of the expected reward of the i-th agent The calculation formula is as follows: In the formula, J(θ i ) is the cumulative expected reward of the ith agent at time T, E(·) is the mathematical expectation, si=(s1,s2,…,s N ), is the joint state of N agents at time T; a i =(a1,a2,…,a N ), is the joint action of N agents at time T; μ i ={μ1,μ2,…,μ N },μ i The parameters are θ={θ1,θ2,…,θ N The strategy set of N agents in time T; is the Critic1 network, composed of Parameterization; The cumulative expected reward J(θ) of the i-th agent at time T i ) is shown in the following formula: In the formula, t is time, E(·) is the mathematical expectation, γ t is the discount factor at time t, r i t is the joint reward of the ith agent at time t.

7. The UAV dynamic game model based on PER-MATD3 deep reinforcement learning as claimed in claim 1, characterized in that: The Actor target network parameter θ in the MATD3 algorithm module i ′ and Critic target network parameter φ im The update formula is as follows: i i ′=τθ i +(1-τ)θ i ′ In the formula, θ i ′ is the Actor target network parameter, φ im τ is the soft update coefficient of the Critic target network parameter, 0<τ≤1, m=1,2.

8. The UAV dynamic game model based on PER-MATD3 deep reinforcement learning as claimed in claim 1, characterized in that: The priority p of sample k in the PER sorting module k As shown below: In the formula, p k is the priority of the kth sample, are the TD errors of the Critic1 network and the Critic2 network for the i-th agent relative to the k-th sample; The calculation is as follows: In the formula, and Respectively by and Parameterized Critic1 network and Critic2 network, y i is the target Q value of the ith agent; The sampling probability of the kth sample in the PER sorting module The calculation of is as follows: In the formula, is the sampling probability of the kth sample, p k is the priority of the kth sample, α is a 0-1 hyperparameter that affects the priority, and K is the number of samples randomly drawn from the experience pool.

9. The UAV dynamic game model based on PER-MATD3 deep reinforcement learning as claimed in claim 1, characterized in that: In the sample importance evaluation module, the sampling weight ω of the kth sample k The calculation of is as follows: In the formula, ω k is the sampling weight of the kth sample, and β is a 0-1 hyperparameter; For the sampling weight ω k Normalization is performed as shown below: In the formula, n is the sample number with the largest weight.

10. The working method of the UAV dynamic game model based on PER-MATD3 deep reinforcement learning according to any one of claims 1 to 9, characterized in that: The following steps are involved: Step 1, initialize network parameters: Initialize the network parameters of each agent and create Critic1 network, Critic2 network and Actor network; Step 2, initialize the target network: Initialize the target network parameters of the i-th agent and θ i 'And create Critic1 target network, Critic2 target network and Actor target network; Step 3, initialize hyperparameters: Initialize the priority coefficient α, sampling weight coefficient β, experience pool D, soft update coefficient τ, discount factor γ, noise edge value c, number of samples extracted each time K, normal variance σ; Step 4, training loop, performs the following steps in each Episode: S41 generates random noise for motion exploration; S42 obtains the initial state of each agent s=(s1,s2,...,s N ), N is the total number of agents; S43 Execute action Select an action according to the current strategy and execute the target action a i ′(s i ′); S44 will collect data from each interaction <s1,a1,s′1,r1,...,s K ,a K ,s′ K ,r K >Stored in the experience pool D, K is the number of randomly selected samples; Step 5, sample selection and update: S51, randomly select K samples from the experience pool D, and the probability of selecting the kth sample is p k ; S52, calculate the sampling weight ω of the kth sample k and TD error δ k ; S53, according to |δ k |Update the priority of the kth sample; S54, calculating a target Q value; S55, minimize the loss function and Update Critic1 network and Critic2 network, calculate the cumulative expected reward J(θ i ); Step 6: Update target network parameters and θ i ′; Step 7: Build a UAV dynamic game model based on steps 1-6, solve the perfect Bayesian-Nash equilibrium, and determine the UAV maneuver decision.

Images