Unmanned aerial vehicle cluster cooperative combat game method and system based on deep reinforcement learning

Abstract

The application relates to a kind of unmanned aerial vehicle cluster cooperative combat game method and system based on deep reinforcement learning, which includes: constructing state space, action space in mixed relationship multi-unmanned aerial vehicle cluster decision model;Public goods game is introduced into the decision model, the reward function guiding the reward function of the free-rider behavior of unmanned aerial vehicle is constructed, the environment returns the corresponding reward according to the reward function after the action of unmanned aerial vehicle output;Set the initial parameters and network structure of MADDPG, train the decision model, and output the strategy.The application integrates public goods game, and balances the interest conflict problem in mixed relationship using deep reinforcement learning, which includes the confrontation relationship between unmanned aerial vehicle cluster and ground air defense force, the cooperation relationship between unmanned aerial vehicle cluster, and the competition relationship between unmanned aerial vehicles in intelligent cluster.The application can be applied in the field of multi-unmanned aerial vehicle cluster control.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multi-unmanned aerial vehicle cluster control, and particularly to a method and system for cooperative combat game of unmanned aerial vehicle cluster based on deep reinforcement learning. BACKGROUND

[0002] Intelligent unmanned aerial vehicles are suitable for application in high-speed, complex and variable modern military operations, and intelligent unmanned aerial vehicle groups have higher autonomy and can find the optimal decision in complex confrontation games. Putting unmanned aerial vehicles into the battlefield not only reduces the combat cost but also greatly reduces the personnel casualties, and the obvious combat advantage makes unmanned aerial vehicles more and more valued in confrontation, among which the attack mode of multi-cluster unmanned aerial vehicle cooperative combat can cope with more complex combat tasks and complete multi-target strikes at the same time. Therefore, improving the cooperation between multi-unmanned aerial vehicle groups and balancing the interest game between multi-unmanned aerial vehicles in the task become the key research to improve the combat efficiency and reduce the loss.

[0003] The research on decision-making problems in UAV and swarm confrontation can be mainly divided into two categories: one is the decision-making dominance problem of UAV's attitude, position and speed in air combat, and the other is the decision-making proportion problem of UAV in swarm. The first category of research is relatively rich, for example, using deep deterministic policy gradient (DDPG) algorithm to deal with the continuous space action problem of UAV, using DDPG and TD3 to solve the maneuvering decision-making problem of UAV, and improving the autonomous control accuracy of maneuvering decision-making. Based on the improvement of DDPG, SA-DDPG uses the maximum entropy inverse reinforcement learning algorithm to construct the reward structure, solves the problem of sparse reward in air combat decision-making, and introduces regularization into the policy network to strengthen the robustness of the decision-making model in the incomplete information environment.

[0004] The research results above are for one-on-one intelligent decision-making model in UAV air combat, and the improved multi-agent deep deterministic policy MADDPG (MADDPG(Lowe R, Wu Y I, Tamar A, et al. Multi-agent actor-critic for mixed cooperative-competitive environments[J]. Advances in neural information processing systems, 2017, 30.)) is more suitable for solving the problem of UAV cluster decision-making. Li Bo, Yue Kaiqiang, Gan Zhigang, et al. (Li Bo, Yue Kaiqiang, Gan Zhigang, et al. Multi-UAV cooperative task decision based on MADDPG[J]. Journal of Astronautics, 2021, 42(06): 757-765.) constructed a dynamic UAV group task decision-making model based on the MADDPG algorithm under the premise of mastering only part of the enemy firepower information before the war. MADDPG has higher stability in dealing with multi-machine decision-making problems.

[0005] In actual combat, the pros and cons of decision-making are also reflected in the control of own resources and losses. The task performed by the UAV may not be single, but may be a continuous multi-segment task. It is difficult for the UAV to replenish resources during the execution of the task, and allocating too many resources to one segment of the task will lead to a decrease in the individual survival ability. In addition, due to the improvement of the intelligence of the UAV, the intelligent UAV individual in the collective pursues individual benefits due to "selfishness", which conflicts with the collective interests, which is the research purpose and importance of the second type of UAV air combat decision-making research, but the research on the second type of decision-making problem is very few.

[0006] Currently, public goods game is widely used in social research, and rational individuals tend to maintain their own strength in single tasks. In order to make the UAV more intelligent, the game idea can also be introduced when the UAV cluster executes the task. The lower the contribution of the individual UAV to the collective, the smaller the loss of the individual, but it will have a negative impact on the completion of the task by the UAV cluster. The conflict between individual interests and collective interests of the UAV, and the reward and loss model of the individual can be regarded as a public goods game model. Due to the particularity of the UAV combat scene, related research is still relatively few, and the existing technology based on multi-element public goods evolutionary game is used to solve the proportion of individual selection as collective contributors in UAV group confrontation, but only the initial decision-making proportion is studied, without considering the process in UAV air combat.

[0007] Based on the above analysis, in order to improve the intelligence of the unmanned aerial vehicle decision, not only the problem of the combat success rate of the unmanned aerial vehicle should be considered, but also the resource saving decision proportion of the control individual should be considered to maintain the fighting capacity of the cluster, and the second type of research of the unmanned aerial vehicle cluster is particularly important. Therefore, an intelligent decision-making model of a multi-unmanned aerial vehicle cluster considering the long-time combat characteristics is urgently needed. SUMMARY

[0008] In view of the above problems, the purpose of the present application is to provide a method and system for cooperative combat game of unmanned aerial vehicle cluster based on deep reinforcement learning, which integrates public goods game and balances the interest conflict problem in mixed relationship by using deep reinforcement learning, the mixed relationship including the antagonistic relationship between unmanned aerial vehicle cluster and ground air defense force, the cooperative relationship between unmanned aerial vehicle clusters, and the competitive relationship between unmanned aerial vehicles in intelligent cluster.

[0009] To achieve the above purpose, the present application adopts the following technical scheme: a method for cooperative combat game of unmanned aerial vehicle cluster based on deep reinforcement learning, comprising: constructing state space and action space in mixed relationship multi-unmanned aerial vehicle cluster decision model; introducing public goods game into the decision model to construct a reward function guiding the free-riding behavior of unmanned aerial vehicles, and the environment returns the corresponding reward according to the reward function after the unmanned aerial vehicle outputs the action; setting the initial parameters and network structure of MADDPG, training the decision model, and outputting the strategy.

[0010] Further, the group decision model is: the unmanned aerial vehicle cooperative maneuvering strategy problem is regarded as a k-step Markov decision process, and N auxiliary unmanned aerial vehicles select unmanned aerial vehicle actions at discrete time t∈{1,2,…,k}

[0011]

[0012] Among them, the first action strategy is that the unmanned aerial vehicle selects to attack the air defense force of the red side, but will be attacked on the ground at the same time; the second action strategy is that the unmanned aerial vehicle selects to avoid the attack of the red side by releasing jamming bombs and moving, but at the same time, it cannot attack the red side.

[0013] Further, the state space includes the attack power of each combat unit at time t, the damage probability cumulative value at the previous t-1 time, and the corresponding value.

[0014] Further, the reward function for guiding the UAV "hitchhiking" behavior includes: obtaining a positive reward of the attack effect of each UAV on the air defense force after the attack at a certain step ends; obtaining a negative reward of the i-th UAV caused by the damage of the auxiliary UAV attack in the attack at the step; obtaining the benefit brought by the decision of the auxiliary UAV according to the superposition of the positive reward and the negative reward at the attack time; and determining whether the auxiliary UAV i is damaged after each attack, and determining the reward of each UAV according to the damage probability cumulative value.

[0015] Further, the reward of each UAV is determined according to the damage probability cumulative value, including: if the t-step damage probability cumulative value is less than or equal to 10%, the UAV i is damaged after the attack at the k-th step, and the UAV i obtains a reward of -50; determining whether the attack task is completed, if the damage is greater than 90%, the remaining attack UAVs have more than 10% of the combat effectiveness and can still fight, the task is successful, and each UAV obtains a reward of 100; if the damage is less than or equal to 90%, the remaining attack UAVs have more than 10% of the combat effectiveness and can still fight, the task is still continuing, and each UAV obtains a reward of 0; if the damage is less than or equal to 90%, the remaining attack UAVs have less than or equal to 10% of the combat effectiveness, the attack UAVs die, the attack task fails, and each UAV obtains a reward of -100.

[0016] Further, the training of the decision model includes: letting N intelligent agents in the model correspond to N auxiliary UAVs, each auxiliary UAV corresponding to a group of Actor-Critic networks, and updating the Actor network by using the policy gradient. The data generated after the auxiliary UAV interacts with the environment is stored in an experience replay pool N , and the data is spliced to form a set {x, x', a1,..., a N}, from randomly sampling the data at the same time, inputting x' into the Actor target network to obtain action a', inputting x', a' into the Critic target network to obtain the target action value at the next moment, and further obtaining the current target action value.

[0017] A UAV cluster cooperative combat game system based on deep reinforcement learning includes: a first processing module for constructing a state space and an action space in a mixed relationship multi-UAV cluster decision model; a second processing module for introducing a public goods game into the decision model, constructing a reward function for guiding the UAV "hitchhiking" behavior, and returning a corresponding reward to the environment according to the reward function after the UAV outputs an action; an output module for setting initial parameters and network structures of MADDPG, training the decision model, and outputting a strategy.

[0018] Further, the group decision model is: the unmanned aerial vehicle cooperative maneuvering strategy problem is regarded as a k-step Markov decision process, and N auxiliary unmanned aerial vehicles select unmanned aerial vehicle action at discrete time t in {1, 2,..., k}

[0019]

[0020] The first action strategy is that the unmanned aerial vehicle selects to attack the air defense force of the red side, but the attack will be subjected to ground attack at the same time. The second action strategy is that the unmanned aerial vehicle selects to avoid the attack of the red side by releasing a jamming bomb and moving.

[0021] A computer-readable storage medium storing one or more programs, the one or more programs comprising instructions which, when executed by a computing device, cause the computing device to perform any of the above-described methods.

[0022] A computing device comprising: one or more processors, memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing any of the above-described methods.

[0023] The present application has the following advantages due to the above technical solutions:

[0024] 1. The present application constructs a heterogeneous multi-unmanned aerial vehicle cluster air-to-ground decision problem, according to the characteristics and values of the unmanned aerial vehicle cluster, and uses a deep reinforcement learning algorithm to give an optimal decision method.

[0025] 2. The present application combines public goods game and unmanned aerial vehicle air combat decision problem, constructs an air combat decision model of a mixed relationship multi-unmanned aerial vehicle cluster, sets up a reward function to guide the free-rider behavior of the unmanned aerial vehicle, makes the unmanned aerial vehicle have the characteristics of a rational individual, and greatly improves the intelligence of the unmanned aerial vehicle.

[0026] 3. Since the number of participants involved in the unmanned aerial vehicle combat game is large and the confrontation process is long, the traditional game theory solving method is difficult to be directly applied. The present application balances the conflict between individual interests and collective interests in the game by using a deep reinforcement algorithm, finally converges to a stable equilibrium decision result, controls the "cost" and avoids the "tragedy of the commons", and successfully balances the relationship between resource input and combat success rate in the unmanned aerial vehicle air combat by introducing game theory. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is a flowchart of the unmanned aerial vehicle cluster cooperative combat game method based on deep reinforcement learning in the embodiment of the present application.​

[0028] Figure 2 is a model diagram of the UAV cluster cooperative attack in the embodiment of the present application;

[0029] Figure 3 is a model structure diagram based on MADDPG in the embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the embodiments of the present application will be described clearly and completely below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the described embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present application.

[0031] It should be noted that the terms used herein are only intended to describe specific embodiments, and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and furthermore, it should be understood that when the terms "comprise" and / or "include" are used in the specification, there is a feature, step, operation, device, component and / or combination thereof.

[0032] The present application provides a kind of UAV cluster cooperative combat game method and system based on deep reinforcement learning, it includes: the state space of mixed relationship multi-UAV cluster decision model, action space is constructed;Public goods game is introduced into decision model, the reward function that guides UAV " free ride " behavior is built, the corresponding reward is returned to environment according to reward function after UAV output action;The initial parameters and network structure of MADDPG are set, the decision model is trained, and the strategy is output.The present application is integrated into public goods game, and the interest conflict problem in mixed relationship is balanced using deep reinforcement learning, mixed relationship includes the confrontation relationship of UAV cluster and ground air defense force, the cooperation relationship between UAV cluster, and the competition relationship between UAVs in intelligent cluster.

[0033] Specifically, in one embodiment of the present application, a UAV cluster cooperative combat game method based on deep reinforcement learning is provided. As shown in the embodiment, Figure 1 The method comprises the following steps:

[0034] 1) the state space of mixed relationship multi-UAV cluster decision model, action space is constructed;

[0035] 2) public goods game is introduced into decision model, the reward function that guides UAV " free ride " behavior is built, the corresponding reward is returned to environment according to reward function after UAV output action;

[0036] 3) Set the initial parameters and network structure of MADDPG, train the decision model, and output the strategy.

[0037] In this embodiment, a multi-UAV cluster air-to-ground model is first constructed. Figure 2 As shown, the Blue team is the attacker, equipped with an air force consisting of drones carrying artillery shells. The Red team is the defender, equipped with ground-based air defenses capable of effectively striking drones. The Blue team's objective is to strike the Red team's bunker, which the Red team will counterattack. To minimize losses and ensure superior combat effectiveness after the mission, the Blue team deploys two drone teams with different payloads. One team is a strike drone swarm tasked with striking the target (bunker). While inherently low-value, it carries effective ammunition. The other team is a supporting drone swarm designed to contain the Red team's firepower. These drones are inherently more valuable and carry not only ammunition but also jamming flares. These jamming flares can be released to decoy enemy infrared-guided weapons from their true targets. In this invention, the Blue team's drones can avoid enemy attack damage by releasing jamming flares. The Red team lacks special weapons or air interception capabilities. Its primary air defenses consist of tank units (RT), anti-aircraft missile units (RD), and the bunker's (R) defense system.

[0038] In step 1) above, the group decision model is:

[0039] The UAV cooperative maneuvering strategy problem is regarded as a k-step Markov decision process, where N auxiliary UAVs make UAV action selections at discrete time t∈{1,2,…,k}.

[0040]

[0041] Among them, the first action strategy Choose to attack the Red Army's air defense force for the drone, but it will be attacked by the ground at the same time; the second action strategy The drone can choose to evade the red team's attacks by releasing jammers and moving, but it cannot attack the red team at the same time.

[0042] Assume that at time t, there are n drones choosing Strategy, Nn drone selection Strategy. In a confrontation, it is difficult to use reconnaissance means to conduct real-time damage assessment of intelligent agents. Therefore, a calculation method is invented that uses damage probability to represent the attack power and the degree of damage to the attacked individual. At the same time, it is considered that the attack power will decrease with the accumulation of damage. At time t, the state space includes the attack power of each combat unit at time t, the cumulative value of the damage probability at the previous time t-1, and its corresponding value. The probability of damage caused by the auxiliary drone group to RD and RT at time t Calculate Damage probability. Since the UAV will also be attacked by the red air defense system RT and RD when selecting an attack, the damage probability of the auxiliary UAV i at time t is The calculation is as follows:

[0043]

[0044] where, RD and RT at time t will cause damage to each auxiliary UAV. With the increase of confrontation time, the damage probability of UAV and red air defense forces will form accumulation, resulting in the decrease of its attack power. Assuming that all combat units ignore the time difference in each time when firing ammunition, the damage probability caused to the opponent at the first time is p 1 , S t-1 indicates the damage probability received in the previous state, then the calculation method of the damage probability caused to the opponent by each combat unit at time k is the cumulative value of the damage probability at the previous k-1 times multiplied by the initial attack power, and the iterative formula is calculated as follows:

[0045]

[0046] Strike-type UAV group attack model:

[0047] Assuming that there are m low-value UAVs attacking the bunker. Since attacking the bunker is the main task, only m UAVs attacking with full force can get the minimum loss. When m UAVs attack R at time t, the damage probability calculation method of R is as follows:

[0048]

[0049] where is the damage probability caused to R by a single strike-type UAV attack. Since the bunker is the protection object of the red side, the strike UAV will be attacked by the bunker at the same time, and will also be attacked by the air defense missile. The damage probability caused to the strike UAV by R at time t is The damage probability caused to it by RD at time t is Then the damage probability calculation of each strike UAV at time t is as follows:

[0050]

[0051] In the above step 1), the state space includes the attack power of each combat unit at time t, the cumulative value of the damage probability at the previous t-1 times, and its corresponding value.

[0052] In step 2) above, the public goods game: Traditional public goods games involve investing resources into a common pool, with participants sharing the results of their investment. This examines the relationship between investment and returns. Threshold public goods games set a threshold for achieving returns, similar to setting successful conditions for drone swarms to strike the enemy in combat. Intelligence often increases the selfishness of individuals within a collective. By incorporating public goods game concepts, while drone free-riding can lead to higher returns in a confrontation, appropriate reward and punishment mechanisms constrain their choices based on the ultimate effectiveness of cooperation.

[0053] In this embodiment, since the present invention utilizes a public goods game, rational drones want to retain their power and refuse to contribute, but the swarm needs drone contributions to complete its mission. Therefore, the present invention constructs a reward function that guides individual auxiliary drones to engage in "free-riding" behavior. Constructing a reward function that guides drone "free-riding" behavior includes the following steps:

[0054] 2.1) After a certain attack step is completed, obtain the positive reward for each drone's attack effect on the air defense force;

[0055] Specifically, after the attack in step t is completed, each auxiliary UAV’s attack on the RD not only reduces the threat to itself, but also reduces the threat to the attacking UAV, which helps to complete the bunker attack mission faster. RD Represents the value of RD assistance, and the positive reward formed by the attack effect of each auxiliary drone on RD

[0056] After the attack in step t, each auxiliary drone’s attack on RT reduces its threat to itself, v RT Represents the value of RT, forming a positive reward for the attack effect of RT

[0057] 2.2) In this attack step, the negative reward for the i-th drone caused by the auxiliary drone’s attack damage is obtained;

[0058] Specifically, in the t-th step attack, choose The auxiliary drone of the strategy will cause its own loss due to attack, so choose Strategy auxiliary drones will not be damaged by attacks, including v i Represents the value of the auxiliary drone, and the attack damage forms a negative reward for drone i

[0059] 2.3) At the attack moment, the benefit brought to the auxiliary drone by its decision is obtained based on the superposition of positive and negative rewards;

[0060] Specifically, in the public goods game, the individual who refuses to contribute can obtain greater benefits than the contributor by free riding. In each round of the game, the auxiliary UAV selects an action strategy represents the selection of cooperative behavior, and the action strategy represents free riding behavior. The benefits of the contributor are shared by all UAVs, and the individual loss caused by the attack is borne by the contributor alone, that is, the benefits of the contributor in one action are less than those of the non-contributor. In summary, the benefits brought to the auxiliary UAV i at time t are

[0061] 2.4) At the same time, it is judged whether the auxiliary UAV i is damaged after each step attack, and the reward of each UAV is determined according to the cumulative value of the damage probability;

[0062] Specifically, at the same time, it is judged whether the auxiliary UAV i is damaged after each step attack, and the reward of each UAV is determined according to the cumulative value of the damage probability; If the cumulative value of the damage probability at step t is represents that the UAV i is damaged after the k-step attack, and the UAV i obtains a reward of -50.

[0063] In the above step 2.4), the reward of each UAV is determined according to the cumulative value of the damage probability, and the condition for judging whether the task of attacking the bunker is completed is shown in Table 1.

[0064] Table 1. Table for judging the success or failure of the task after k-step attack

[0065] Decision condition State Reward φ > 90% and ψ > 10% Task success 100 φ ≤ 90% and ψ > 10% Task continues 0 φ ≤ 90% and ψ ≤ 10% Task failure -100

[0066] Let

[0067] Specifically, the following steps are included:

[0068] 2.4.1) If the cumulative value of the damage probability at step t is less than or equal to 10%, it represents that the UAV i is damaged after the k-step attack, and the UAV i obtains a reward of -50;

[0069] 2.4.2) It is judged whether the task of attacking the bunker is completed. If the damage of the bunker is more than 90%, the remaining attack UAVs have more than 10% combat effectiveness, which means that they can still fight, the task is successful, and each UAV obtains a reward of 100;

[0070] 2.4.3) If the damage of the bunker is less than or equal to 90%, the remaining attack UAVs have more than 10% combat effectiveness, which means that they can still fight, and the task is still continuing, and each UAV obtains a reward of 0;

[0071] 2.4.4) If the damage of the bunker is less than or equal to 90%, the remaining attack UAVs have less than or equal to 10% combat effectiveness, the attack UAVs are dead, the attack task fails, and each UAV obtains a reward of -100.

[0072] In step 3) above, training the decision model includes the following steps:

[0073] 3.1) If Figure 3 As shown in the figure, under the MADDPG algorithm, the N agents in the model correspond to N auxiliary drones, each auxiliary drone corresponds to a set of actor-critic networks, and each set of actor-critic networks contains two networks: a behavior network and a target network;

[0074] a i Represents the action of drone i. State S includes auxiliary drone group, attack drone group, red side air defense force, target point damage probability accumulation, attack power and each unit value. Action space After each round of attack, the state is updated S t →S t+1 , o i Represents the observation value of the auxiliary drone i on the environment. Since the drone's action in the present invention is a 2D discrete action, in order to maintain the DDPG style gradient approximator, the present invention uses Softmax to process the output strategy in the Actor network.

[0075] The set of policy parameters for all auxiliary UAVs is θ={θ1,θ2,…,θ N},Strategy Abbreviated as μ i , the set of all deterministic strategies of auxiliary UAVs is expressed as μ = {μ i ,μ2,…,μ N}.

[0076] 3.2) The data generated after the auxiliary drone interacts with the environment is stored in the experience replay pool In the data, the set {x,x′,a1,…,a N ,r1,…,r N},from Randomly sample data at the same moment, input x′ into the Actor target network to obtain action a′, input x′, a′ into the Critic target network to obtain the target action value at the next moment, and then obtain the current target action value.

[0077] Among them, the current target action value is calculated as follows:

[0078]

[0079] Where: μ′={μ′ i ,μ′2,…,μ′ N} is the parameter θ′ with delayed update iTarget policy The short form of r i represents the reward of the ith auxiliary UAV, and γ is the discount factor.

[0080] The critic network updates the network by minimizing the mean square error loss function as follows:

[0081]

[0082] The actor network is updated by policy gradient, and the deterministic policy μ i of the ith auxiliary UAV is updated as follows:

[0083]

[0084] where the objective function J(μ i ) is the cumulative expected return of the ith auxiliary UAV, and is calculated as follows: The policy parameters θ i are updated by gradient ascent. i represents the action of the auxiliary UAV i, o i represents the observation value of the auxiliary UAV i, and x={o1,o2…,o N} represents the observation values of all auxiliary UAVs. is the centralized action value function, since the input is the action of all auxiliary UAVs, it is considered that the influence between UAVs is reduced under the condition of known information, and the environment is considered to be stable and unchanged. is the sampling expectation, is the experience replay pool. is the gradient operator with respect to the policy parameters θ i , and is the gradient operator with respect to the deterministic action a i .

[0085] In addition, the target networks of the actor and the critic are updated in a soft update manner.

[0086] In the above embodiment, before step 3) is performed, initial parameter values also need to be set:

[0087] For example, the model environment set by the present application contains 4 auxiliary UAVs and 8 strike UAVs, and the ground forces of the red side: R, RT, and RD are regarded as a whole. The values of the auxiliary UAV, RD, and RT are v i =5, v RD =10, and v RT =15 respectively.= 8. The UAV swarm attack process includes multiple decision steps, and the initial damage probability parameter setting requires the overall strength of the UAV cluster to be higher than that of the ground air defense force, so that the UAV has a selective space. With the increase of the number of steps, the damage probability of the attacking unit decreases, so the initial parameter value is selected within a reasonable range as shown in Table 2.

[0088] Table 2 Initial damage probability caused by the combat unit to the corresponding attacked party

[0089]

[0090] In this embodiment, the MADDPG algorithm network structure adopted is as follows:

[0091] The Actor network structure is a fully connected neural network with [64; 64; 2], and the Critic network structure is a fully connected neural network with [64; 64; 1], the network structure representing the node numbers corresponding to the input layer, the hidden layer and the output layer. The network hidden layer adopts the ReLU function. The experience replay memory size is 1 million, the sampling mini-batch is 512 in the training process, the maximum number of rounds is 50,000, the maximum step length of each round is 100. The discount factor is 0.99, the soft update parameter of the target network is 0.0005, the learning rate of the Actor network is 0.005, the learning rate of the Critic network is 0.0005, and all of them are learned by using the Adam Optimizer optimizer.

[0092] Based on the above, the comparison experiment between the present application and the existing method is carried out, and the experimental results are as follows:

[0093] The TensorFlow framework is used to build the training network, and the hardware environment used for training is: AMD Ryzen95950X CPU, Nvidia RTX3090 graphics card, and 64G memory.

[0094] For comparison of experimental results, the application carries out comparison experiments based on three value-based deep reinforcement learning algorithms of IDQN (IDQN (Tampuu A, Matiisen T, Kodelja D, et al. Multiagent cooperation and competition with deep reinforcement learning [J]. PloS one, 2017, 12 (4): e0172395.)), IDDQN (IDDQN (Lv P, Wang X, Cheng Y, et al. Integrated double estimator architecture for reinforcement learning [J]. IEEE Transactions on Cybernetics, 2020.)), and ID3QN (ID3QN (Xiang J, Yan C, Wang Z, et al. Formation coordination control method of fixed-wing unmanned aerial vehicle based on deep reinforcement learning [J]. Acta Aeronautica et Astronautica, 2021, 42 (04): 420-433.)), and the experimental results are shown in Table 3.

[0095] Table 3 Training results of four algorithms for 50,000 rounds

[0096] Method MADDPG IDQN ID3QN ID3QN Convergence situation 5,000 episodes Not converged Not converged Converged in reverse Cumulative reward 515 Not converged Not converged Converged in reverse Blue side win rate 100％ 60％～80％ 0～75％ 0％

[0097] The application carries out multiple experiments based on the four algorithms, and it can be seen from Table 3 that the convergence effect and stability of the MADDPG algorithm are obviously better than those of the other three algorithms, the maximum reward can be learned after 5,000 rounds of training, and the learned decision result also reaches 100% in the combat win rate. When facing the public goods game environment of the multi-unmanned aerial vehicle cluster constructed by the application, the MADDPG realizes better experimental results with less model optimization work by using centralized training and decentralized execution, and successfully solves the problem of conflict between individual and collective benefits.

[0098] In an embodiment of the application, a deep reinforcement learning-based unmanned aerial vehicle cluster cooperative combat game system is provided, which comprises:

[0099] The first processing module constructs the state space and the action space in the mixed relationship multi-unmanned aerial vehicle cluster decision model;

[0100] The second processing module introduces the public goods game into the decision model, constructs a reward function for guiding the free-riding behavior of the unmanned aerial vehicle, and returns the corresponding reward according to the reward function after the unmanned aerial vehicle outputs an action;

[0101] An output module is configured to set initial parameters and network structures of the MADDPG, train the decision model, and output a strategy.

[0102] In the above embodiment, the group decision model is:

[0103] The unmanned aerial vehicle cooperative maneuver strategy problem is regarded as a k-step Markov decision process, and N auxiliary unmanned aerial vehicles select unmanned aerial vehicle actions at discrete time t in {1, 2,..., k}

[0104]

[0105] The first action strategy is: The unmanned aerial vehicle selects the attack air defense force of the red side, but the attack will be attacked on the ground at the same time. The second action strategy is:

[0106] The system provided in the embodiment is used for executing the above-mentioned method embodiments, and the specific process and detailed content are referred to the above-mentioned embodiments, which will not be repeated here.

[0107] In an embodiment of the present application, the computing device can be a terminal, which can include a processor, a communications interface, a memory, a display screen and an input device. The processor, the communications interface and the memory complete mutual communication through a communication bus. The processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program, which is executed by the processor to implement a deep reinforcement learning-based unmanned aerial vehicle cluster cooperative combat game method. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium. The communications interface is used for wired or wireless communication with external terminals. Wireless communication can be achieved through WIFI, management network, NFC (near field communication) or other technologies. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computing device. It can also be an external keyboard, touchpad or mouse, etc. The processor can call the logical instructions in the memory.

[0108] Moreover, the logic instructions in the storage medium of the above-mentioned embodiments can be implemented in the form of software function units and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or partially contribute to the prior art, or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0109] In an embodiment of the present application, a computer program product is provided, which includes a computer program stored on a non-transitory computer readable storage medium, and the computer program includes program instructions, which, when executed by a computer, enable the computer to perform the method provided by the above-mentioned method embodiments.

[0110] In an embodiment of the present application, a non-transitory computer readable storage medium is provided, which stores server instructions, and the computer instructions enable a computer to perform the method provided by the above-mentioned embodiments.

[0111] The computer readable storage medium provided by the above-mentioned embodiments has similar implementation principles and technical effects to the above-mentioned method embodiments, and will not be described here.

[0112] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The device that implements the functions specified in one flow or multiple flows and / or blocks Figure 1 The device that implements the functions specified in one flow or multiple flows and / or blocks

[0113] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow Figure 1 The functions specified in the flow or flows and / or blocks Figure 1 The functions specified in the flow or flows and / or blocks

[0114] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flow Figure 1 The functions specified in the flow or flows and / or blocks Figure 1 The functions specified in the flow or flows and / or blocks

[0115] Finally, it should be noted that the above-mentioned embodiments are merely used to illustrate the technical solutions of the present application, rather than limit the same; even though the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can be made to the technical solutions recorded in the foregoing embodiments, or equivalent replacements can be made to some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for cooperative combat game of a drone swarm based on deep reinforcement learning, characterized in that, The method comprises the following steps: constructing a state space and an action space in a multi-unmanned aerial vehicle (UAV) swarm decision model with mixed relationships; introducing a public goods game into the decision model to construct a reward function for guiding the "free-rider" behavior of the UAV, and returning a corresponding reward to the UAV according to the reward function after the UAV outputs an action; setting initial parameters and a network structure of a multi-agent deep deterministic policy gradient (MADDPG) algorithm, training the decision model, and outputting a strategy; the swarm decision model is as follows: The unmanned aerial vehicle cooperative maneuver strategy problem is regarded as a k-step Markov decision process, and N auxiliary unmanned aerial vehicles make unmanned aerial vehicle action selection at discrete time t∈{1, 2, …, k} The first action strategy selects an attack on the air defense forces of the red side for the UAV, but the attack will be subjected to ground attack at the same time; and the second action strategy selects evasion of the attack of the red side by releasing a jamming bomb and moving for the UAV, but the UAV cannot attack the red side at the same time. the reward function for guiding the "free-rider" behavior of the UAV comprises: After the end of the attack in step t, the positive reward of the attack effect of each UAV on the air defense force is obtained: let v RD The value of the auxiliary RD, the positive reward of the attack effect of each auxiliary UAV on the RD After the end of the attack in step t, the attack of each auxiliary UAV on the RT reduces its threat to itself, v RT The value of the RT, the positive reward of the attack effect on the RT Wherein, The damage probability of the auxiliary UAV group to the RD and RT at time t, respectively; And in the attack of this step, the negative reward of the ith UAV attacked by the auxiliary UAV attack damage formation is obtained: negative reward of the UAV attacked by the attack damage formation UAV i wherein, is the damage probability suffered by the auxiliary UAV i at time t, is the UAV action selection, v i is the value of the auxiliary UAV; At the attack moment of this step, the auxiliary unmanned plane's decision brings the profit to it according to the superposition of the positive reward and the negative reward: the auxiliary unmanned plane u's decision brings the profit to it at t time determining the reward of each UAV according to the cumulative value of the damage probability after judging whether the auxiliary UAV i is damaged after each attack; the training of the decision model comprises: N intelligent agents in the model correspond to N auxiliary UAVs, each auxiliary UAV corresponds to a group of Actor-Critic networks, and the Actor network is updated by using a policy gradient; Data generated after the auxiliary drone interacts with the environment is stored in an experience replay pool In this way, the data is spliced to form a set {x, x', a1, …, a N ,r1,…,r N} from which The data at the same time is randomly sampled, x' is input into the Actor target network to obtain an action a', x', a' is input into the Critic target network to obtain a target action value at the next moment, and then a current target action value is obtained. 2.The method of claim 1, wherein, the state space comprises the attack power of each combat unit at time t, the cumulative value of the damage probability at time t-1, and the corresponding value. 3.The method of claim 1, wherein, determining the reward of each UAV according to the cumulative value of the damage probability comprises: if the cumulative value of the damage probability at time t is less than or equal to 10%, the UAV i is damaged after the kth attack, and the UAV i obtains a reward of -50; judging whether the attack task is completed, if the damage exceeds 90%, the remaining attack UAVs have more than 10% of the combat power and can still fight, the task is successful, and each UAV obtains a reward of 100; if the damage is less than or equal to 90%, the remaining attack UAVs have more than 10% of the combat power and can still fight, and the task is still continuing, each UAV obtains a reward of 0; if the damage is less than or equal to 90%, the remaining attack UAVs have less than or equal to 10% of the combat power, the attack UAVs are dead, the attack task is failed, and each UAV obtains a reward of -100.

4. A deep reinforcement learning based UAV swarm cooperative combat game system for implementing the deep reinforcement learning based UAV swarm cooperative combat game method according to any one of claims 1 to 3, characterized in that, The method comprises the following steps: a first processing module constructs a state space and an action space in a multi-unmanned aerial vehicle (UAV) swarm decision model with mixed relationships; a second processing module introduces a public goods game into the decision model to construct a reward function for guiding the "free-rider" behavior of the UAV, and returns a corresponding reward to the UAV according to the reward function after the UAV outputs an action; an output module sets initial parameters and a network structure of a multi-agent deep deterministic policy gradient (MADDPG) algorithm, trains the decision model, and outputs a strategy.

5. The UAV swarm cooperative engagement game system based on deep reinforcement learning of claim 4, wherein, the swarm decision model is as follows: The unmanned aerial vehicle cooperative maneuver strategy problem is regarded as a k-step Markov decision process, and N auxiliary unmanned aerial vehicles make unmanned aerial vehicle action selection at discrete time t∈{1, 2, …, k} Among them, the first action strategy selects the attack of the red side's air defense force for the unmanned aerial vehicle, but the attack will be attacked on the ground at the same time; the second action strategy selects the red side's attack for the unmanned aerial vehicle by releasing a jamming bomb and moving to avoid, but at the same time, it cannot attack the red side.

6. A computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that when executed by a computer cause the computer to perform a method comprising: the one or more programs comprise instructions that, when executed by a computing device, cause the computing device to perform any one of the methods in claims 1 to 3.

7. A computing device, comprising: The method comprises the following steps: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs comprise instructions for performing any one of the methods in claims 1 to 3.

Images