A method for cooperative confrontation of a UAV cluster by introducing action interaction and credit allocation

Abstract

The application provides a UAV cluster cooperative confrontation method introducing action interaction and credit allocation, and relates to the UAV cluster air-to-air confrontation game decision technical field. The method firstly models the confrontation space, action space, state space and observation space of the UAV cluster; through an action matching mechanism, the subdivision of action influence is realized, the self-attention mechanism is used to process the mismatch between the observation value and the action, so that reliable information can be retained; by maximizing the joint action value of multiple agents, the best cooperation and strategy between UAVs are ensured; through a credit allocation mechanism, the UAVs can more accurately evaluate their own contributions in the cooperation process; the method can perform UAV cluster confrontation game under partial observability, improves the stability and confrontation performance of the agent learning strategy, and provides a new method and technology for UAV cluster confrontation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle cluster air-to-air confrontation game decision, and particularly relates to a method for unmanned aerial vehicle cluster cooperative confrontation by introducing action interaction and credit distribution. BACKGROUND

[0002] As a frontier field, unmanned aerial vehicle cluster technology is becoming a strategic highland that international military powers are competing to explore and deploy. China has also launched the research of unmanned aerial vehicle cluster confrontation concept, and is fully committed to the tide of key technology breakthrough, and through a series of cluster flight tests, the feasibility and effectiveness are constantly verified. At the same time, countries around the world have also followed suit, and in response to the diversified confrontation task requirements, they have carried out system integration and empirical exploration, striving to make breakthroughs in the application of unmanned aerial vehicle cluster technology. The in-depth research and test of this series not only reveals the unlimited potential of unmanned aerial vehicle cluster technology in modern warfare, but also marks the evolution of unmanned aerial vehicle technology from traditional single confrontation platform to intelligent, cooperative and clustered confrontation system, and its strategic importance is increasingly prominent. Unmanned aerial vehicle cluster technology, with its high flexibility, autonomy and survivability, is a new change in future warfare, which will bring unprecedented tactical advantages and strategic impact to military confrontation.

[0003] In the unmanned aerial vehicle cluster, there is cooperation and competition among multiple unmanned aerial vehicles. These cooperation and competition are not simply addition and subtraction, but a highly complex and dynamic system. The unmanned aerial vehicles in the system have different goals, capabilities and information resources, and how to effectively manage the interaction between unmanned aerial vehicles is a challenging problem. First, as the number of unmanned aerial vehicles increases, the coordination and communication costs will increase rapidly; second, under the condition of incomplete information, unmanned aerial vehicles need to learn global information from local observations in order to make reasonable decisions. In the unmanned aerial vehicle cluster system, it is a crucial problem to determine the contribution of each unmanned aerial vehicle to the global reward, that is, how to allocate credit. The purpose of credit allocation is to ensure that each unmanned aerial vehicle can obtain appropriate rewards in the process of cooperation or jointly achieving goals, in order to encourage cooperation and promote the improvement of system performance. However, traditional credit allocation methods often have the problem of unfair or low efficiency, especially when facing intelligent agents with different capabilities and contribution levels. The importance of this problem lies in that if there is no effective credit allocation, unmanned aerial vehicles may lack the motivation to actively participate in cooperation or share resources, thereby affecting the cooperation efficiency of the unmanned aerial vehicle cluster system. In addition, the interaction between unmanned aerial vehicles is also an important challenge. Unmanned aerial vehicles need to coordinate actions through effective interaction to achieve common goals. Different behaviors of unmanned aerial vehicles will have different effects on other intelligent agents, which may be direct, indirect or even long-term. However, existing methods often perform poorly in the interaction of unmanned aerial vehicles, and cannot fully utilize the information of each intelligent agent, resulting in unsatisfactory task completion. SUMMARY

[0004] The technical problem solved by the present application is to provide a method for introducing action interaction and credit distribution in cooperative confrontation of UAV clusters to realize simulation of cooperative confrontation of UAV clusters and provide a method for establishing confrontation game of UAV clusters in dynamic environment.

[0005] To solve the above technical problems, the technical solution adopted by the present application is: a method for introducing action interaction and credit distribution in cooperative confrontation of UAV clusters, comprising the following steps:

[0006] Step 1: model the confrontation space, action space, state space and observation space of the UAV cluster respectively, and set the confrontation parties of the UAV cluster;

[0007] The UAV cluster is composed of multiple UAVs; the UAVs of the confrontation parties in the confrontation scene all adopt a six-degree-of-freedom dynamics model, the motion of the UAVs in the continuous action space is described by using a dynamics equation, a continuous action set is adopted, and the continuous action parameters of the UAVs are represented by a vector;

[0008] The UAV cluster confrontation task is described as a tuple The composed decentralized partially observable Markov decision process is for a set of agents, for a set of states, for a set of actions, P is a state transition function, r is a reward function, Ω is a set of observations, O is an observation function, γ is a discount factor, and γ∈[0,1]; set s is a global state describing the environment, at each time step t, each agent selects an action and generates a joint action U is a set of joint actions; according to the state transition function P(s′|s,u): the environment enters the next state, so that the joint action u changes, where s′ is the state of the agent at the next time; all agents share the same reward function r(s,u): r(s,u) is a set of real numbers, the role of r(s,u) is to evaluate the state s and the joint action u and return a real value as a reward, which is used to measure the contribution of the state and action; the common goal of all factors in the tuple is to maximize the expected reward where G represents the discounted sum of future rewards t The expectation is calculated, and the specific formula is:

[0009]

[0010] where k is a step size after time t, r t+k is an immediate reward obtained at time t+k;

[0011] Step 2: considering the action interaction between UAVs, regarding the UAVs as intelligent agents independently and autonomously active, constructing an agent network training model, judging whether the agent action only affects the environmental information of the confrontation space or the agent action also affects other agents, and dynamically adjusting the decision of the agent according to the environmental information of the confrontation space; including the following steps:

[0012] Step 2.1: constructing an agent network training model for each agent for learning the strategy and decision of the agent;

[0013] Constructing the agent network training model, so that the agent learns adaptive strategy in confrontation and reaches the optimal strategy in the mutually influencing environment;

[0014] Dividing the observation value o and action u input into the agent network training model, dividing the initial observation value into three parts: the part related to the environment, the part related to itself, and the part related to other agents;

[0015] Converting the observation value o of the agent from a sequence form into a matrix form, so that the representation of the observation-state space is unified and comparable; dividing the action u into two parts: and Action only affects the environmental information and the agent itself; action affects other agents;

[0016] Step 2.2: dividing the agent network into different sub-modules, evaluating the local Q value of the agent action through the influence of the agent action on other agents, and calculating the global Q value of the agent network from the local Q values of all agents;

[0017] Dividing the agent network into different sub-modules, judging whether the agent action only affects the environmental information of the confrontation space or the agent itself also affects other agents through the influence of the agent action on other agents; if the agent action only affects the environmental information of the space and the agent itself, inputting the initial observation value of the agent into the embedding layer in the agent network training model, evaluating the local Q value of the agent action; if the agent action affects other agents, inputting the observation value o i,j of agent i to agent j and the action of agent i affecting agent j into the embedding layer in the agent network training model, evaluating the local Q value of the agent action; calculating the global Q value of the agent network from the local Q values of all agents;

[0018] The transformer is introduced into the agent network training model, the self-attention mechanism is used to identify and filter out useful information, and the observation value is weighted according to the obtained useful information; the attention is focused on the input features that are helpful to the agent, the attention on irrelevant features is suppressed, and the useless information that is not needed for decision-making is discarded;

[0019] The useful information identified and filtered out by the self-attention mechanism is observation data or features that are substantially helpful and valuable to the agent when making decisions or performing tasks, which are perceived by the agent in the environment of the confrontation space and can directly or indirectly affect the behavior, decision-making process or task completion of the agent;

[0020] Step 2.3: training the agent network training model using a reinforcement learning algorithm;

[0021] Step 2.4: the agent perceives and understands the actions of other agents, and dynamically adjusts the decision of the agent according to the environmental information of the confrontation space;

[0022] Step 3: the confrontation between the two parties in the UAV cluster is carried out, and the confrontation situation is evaluated and reasoned, the local Q value generated by the agent network training model is received as input, and a double-layer hybrid network structure is established by using the monotonic value function decomposition method to obtain the global Q value Q tot of the agent network, and then the optimal joint action of the agent is obtained;

[0023] The two parties of confrontation carry out confrontation through the set confrontation space and the agent network training model; both parties are committed to collecting confrontation data comprehensively, observing and recording the actions and environment of the confrontation party, estimating the state, and then evaluating the confrontation situation; based on the evaluation result of the confrontation situation, the reasoning and decision-making of the agent are dynamically adjusted;

[0024] The centralized training and distributed execution paradigm is adopted, the local Q value of each agent is integrated and constrained through a double-layer hybrid network to generate a global Q value; each agent has an independent Q network and can make independent decisions based on its local information, the output of the Q network is transmitted through the double-layer hybrid network to realize the mapping of the local Q value to the global Q value;

[0025] The double-layer hybrid network is responsible for credit allocation, taking the local Q value generated by the agent network as input, processing it through two separate super networks and activation functions contained in the double-layer hybrid network, and outputting the global Q value Q tot , the specific formula is:

[0026] Q tot (τ,u,s;θ)=W2R eLU (W1Q+b1)+b2

[0027] where, is the local Q-value output by the agent network, Q n (τ n ,u n ) is the local Q-value of the nth agent, τ n ,u n are the observation history and action of the nth agent respectively, τ is the observation history, s is the state of the environment, θ is the parameter of the target network, R eLU is the activation function of W1Q+b1, and are the weight matrix and bias term generated by the two separate hypernetworks respectively, and satisfy: is a real matrix, m and n are the row and column dimensions of the real matrix respectively;

[0028] The loss function of the double-layer hybrid network is updated by minimizing the squared TD error, and the loss function formula is as follows:

[0029]

[0030] where, is the TD target, τ', u', s' are the observation-action history, joint action and state at the next time respectively, θ - is the parameter of the target network, which is copied from θ regularly;

[0031] The two separate hypernetworks dynamically adjust their parameters or structures according to the input observation value to adapt to different environments;

[0032] Step 4: The two parties carry out cooperative task planning to provide the optimal strategy for the multi-agent; the local Q-value of the agent is updated, and the agent strategy is optimized by the method of credit assignment, and the specific method is:

[0033] According to the global Q-value Q tot obtained in step 3, the local Q-value of the agent is updated by using the back propagation of the agent network training model; the global Q-value of the agent is updated using gradient entropy to distinguish the credit assignment and optimize the strategy;

[0034] The concept of normalized gradient entropy is introduced in the double-layer hybrid network to measure the distributability of credit; according to the expansion result of the global Q-value Q tot , the gradient is calculated:

[0035]

[0036] Therefore, we get:

[0037] dQ tot = W2F W1dQ

[0038]

[0039] where, is a diagonal matrix, and x c is the c-th element of the output of the double-layer hybrid network, X is the input feature of the double-layer hybrid network, x1, x2, …, x m are all the outputs of the first layer linear transformation in the double-layer hybrid network, which are calculated by W1Q + b1;

[0040] Introducing action interaction and credit allocation loss function including the original TD loss and the regularization gradient entropy two parts, as shown in the following formula:

[0041]

[0042] where, λ is a hyperparameter used to balance the TD loss and the regularization term, and batchsize is the batch size of sampling;

[0043] Step 5: Repeat step 4 until the confrontation between the two parties is over, observe the confrontation performance of the introduced action interaction and credit allocation, and complete the confrontation simulation of the UAV cluster.

[0044] The beneficial effects produced by the above technical solutions are as follows:

[0045] (1) By defining a comprehensive confrontation space, action space, state space and observation space, the reality and complete modeling of the UAV cluster confrontation scene is ensured; such detailed modeling helps more effective strategy development and accurate performance evaluation;

[0046] (2) The action matching mechanism proposed by the action interaction realizes the subdivision of the action influence, and uses the self-attention mechanism to process the mismatch between the observation value and the action, so as to retain reliable information; the ability of the UAV to autonomously identify and adapt to the changing confrontation environment is strengthened; it enables the UAVs to have effective interaction to coordinate actions to achieve common goals;

[0047] (3) By maximizing the joint action value of multiple intelligent agents, the best cooperation and strategy between UAVs are ensured, resulting in a more effective and unified confrontation method;

[0048] (4) Through the unique credit allocation mechanism, the UAVs can more accurately evaluate their respective contributions in the process of cooperation;

[0049] (5) observing the performance of the action interaction and credit assignment distinguishable algorithm in the adversarial simulation, resulting in continuous improvement and adaptability of the strategy;

[0050] Each step of the method contributes uniquely to the overall effectiveness of the UAV swarm in the confrontation; by integrating advanced algorithms and learning techniques, the proposed method provides a complex and adaptive method for UAV swarm confrontation, enhancing their strategic ability, adaptability and effectiveness in various confrontation scenarios; UAV swarm confrontation games can be conducted under partial observability, improving the stability and confrontation performance of the agent learning strategy, providing a new method and technology for UAV swarm confrontation. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 A flowchart of a UAV swarm cooperative confrontation method with action interaction and credit assignment provided for an embodiment of the application;

[0052] Figure 2 A neural network model structure diagram adopted by the agent network training model provided for an embodiment of the application;

[0053] Figure 3 A credit assignment distinguishable network structure diagram provided for an embodiment of the application;

[0054] Figure 4 A network structure diagram of the action interaction and credit assignment distinguishable network provided for an embodiment of the application;

[0055] Figure 5 A UAV swarm confrontation game win rate diagram provided for an embodiment of the application;

[0056] Figure 6 A friendly force survival rate comparison diagram of a UAV swarm cooperative confrontation method with action interaction and credit assignment provided for an embodiment of the application and other algorithms;

[0057] Figure 7 An enemy force mortality rate comparison diagram of a UAV swarm cooperative confrontation method with action interaction and credit assignment provided for an embodiment of the application and other algorithms. DETAILED DESCRIPTION

[0058] The specific embodiments of the application will be further described in detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate the application, but are not used to limit the scope of the application.

[0059] In this embodiment, a UAV swarm cooperative confrontation method with action interaction and credit assignment, as shown in Figure 1 includes the following steps:

[0060] Step 1: Model the confrontation space, action space, state space, and observation space of the UAV cluster respectively, and set the confrontation between the UAV cluster, namely the red and blue sides;

[0061] For the UAV cluster, construct effective confrontation space, action space, state space, and observation space, mathematically describe the decision elements, attribute characteristics, and interaction relationships of the confrontation space, in order to design and evaluate different agent strategies;

[0062] The UAV cluster is composed of multiple UAVs and has cooperative confrontation capability. Every time interval, the UAV can accurately launch a missile at the enemy target to implement a devastating attack. If any UAV is unfortunately hit by an enemy missile, it is determined to be destroyed and cannot continue to participate in the battle. The core task of both sides is to completely eliminate all UAVs of the other side and seize air supremacy;

[0063] In the confrontation scenario, the UAVs of the confrontation parties (red and blue) use a six-degree-of-freedom dynamics model, use a dynamics equation to describe the motion of the UAV in the continuous action space, adopt a continuous action set, and use a vector to represent the continuous action parameters of the UAV. The dynamics model of each UAV is represented as:

[0064]

[0065] where v is the flight speed of the UAV, m is the mass of the UAV, t is the time, v x , v y are the X-axis and Y-axis velocity components, θ is the pitch angle, is the heading angle, φ is the roll angle, F is the resultant force of the UAV, x t , y t represent the positions of the UAV in the X and Y axes at time t, x t-1 , y t-1 represent the positions of the UAV in the X and Y axes at time t-1, r φ is the roll angular velocity, r θ is the pitch angular velocity, is the heading angular velocity, and these data types are all floating point types;

[0066] The continuous action space is represented as a multi-dimensional Euclidean space, where each dimension corresponds to the range of an action parameter. When an action needs to be selected, the optimal combination of action parameters is found by searching the continuous parameters in the action space to meet a specific goal;

[0067] The action space is a continuous action space, the motion of the UAV cluster in the continuous action space is described using a dynamic equation, a continuous action set is adopted, and the continuous action parameters of the UAV cluster are represented by a vector; then the continuous action space is discretized, and the dimension is expanded to 8 dimensions; the expression and control of the moving direction are more delicate and comprehensive;

[0068] In this embodiment, the continuous action space is comprehensively covered in a complete 360-degree cycle; the number of dimensions of the action space is directly related to the discretization precision of the continuous angle space; if the full 360-degree space is subdivided into n specific and non-overlapping direction intervals, then the dimension of the action space is equal to n; in order to enhance the accuracy and flexibility of direction selection, this embodiment adopts a more refined discretization strategy, which uniformly cuts the originally continuous 360-degree circumference into eight main directions: east, northeast, north, northwest, west, southwest, south, and southeast; this significantly expands the dimension of the action space to 8, thereby realizing more delicate and comprehensive expression and control of the moving direction;

[0069] The state space and the observation space comprehensively capture and utilize the global state information of the UAV cluster in the centralized training phase to optimize the cooperative performance of the entire UAV cluster; in the execution phase, each UAV needs to rely on its partial observable information to make independent decisions and perform corresponding actions;

[0070] In this embodiment, the global state information of the UAV cluster is shown in Table 1, which covers key elements such as the overall operating condition of the UAV cluster, environmental parameters, and the relative positions and states between members; each UAV can only obtain partial observation information within its field of view as shown in Table 2, which constitutes the basis for local decision-making and action of the UAV;

[0071] Table 1 Global state information

[0072]

[0073] Table 2 Partial observation information

[0074]

[0075] At each time step, the UAVs receive partial observations of the environment within their circular field of view; the radius of this field of view is equivalent to the maximum range of their line of sight, defining the boundary of the environment that the UAVs can perceive; from the perspective of the UAVs, their field of view encompasses not only the terrain features, but also the last actions of all friendly UAVs that are alive and within their field of view; the limitations of the UAVs in obtaining information are set: they cannot directly determine the status of friendly UAVs outside their field of view, whether they are at a greater distance or have been destroyed; the UAVs also have the ability to observe the terrain features around them, which helps them to better understand the environment they are in, and thus make more environmentally adaptive action choices;

[0076] The UAV swarm (i.e., multi-agent) confrontation task is described as a tuple The decentralized partially observable Markov decision process is composed of a set of agents, a set of states, a set of actions, P is a state transition function, r is a reward function, Ω is a set of observations, O is an observation function, and γ is a discount factor, γ ∈ [0, 1]; set s is the global state of the environment, at each time step t, each agent selects an action and generates a joint action U is a set of joint actions; according to the state transition function P(s'|s,u): the environment enters the next state, making the joint action u change, where s' is the state of the agent at the next time step; all agents share the same reward function r(s,u): r(s,u) is a set of real numbers, and the role of r(s,u) is to evaluate the state s and the joint action u, returning a real value as a reward, which measures the contribution of the state and action; the common goal of all factors in the tuple is to maximize the expected reward where G represents the discounted sum of future rewards t The expectation is calculated as follows:

[0077]

[0078] where k is the step size after time t, and r t+k is the immediate reward obtained at time t+k;

[0079] Step 2: Considering the action interaction between UAVs, the UAVs are regarded as independent and autonomous agents, an agent network training model is constructed to determine whether the agent action only affects the environmental information of the confrontation space or the agent action also affects other agents, and the agent's decision is dynamically adjusted according to the environmental information of the confrontation space; including the following steps:

[0080] Step 2.1: An agent network training model is constructed for each agent to learn the strategy and decision of the agent;

[0081] The agent network training model is constructed to enable the agent to learn adaptive strategies in confrontation and achieve optimal strategies in the mutually influencing environment; the neural network model structure used by the agent network training model is as shown in Figure 2 ;

[0082] The observation value o and the action u input into the agent network training model are divided, and the initial observation value is divided into three parts: the part related to the environment, the part related to itself, and the part related to other agents;

[0083] At time t, the initial observation value of the i-th agent is Wherein is the observation value related to the environment, is the inherent attribute value of the agent's attack, etc., is the observation related to other agents, and n is the number of agents;

[0084] The observation value o of the agent is converted from a sequence (concatenate) form to a matrix (matrix) form to make the representation of the observation-state space more unified and comparable; the action u is divided into two parts: and The action only affects the environmental information and the agent itself, such as movement, etc.; the action affects other agents, such as cooperative communication and attack, etc.

[0085] Step 2.2: The agent network is divided into different sub-modules, the local Q value of the agent action is evaluated through the influence of the agent action on other agents, and the global Q value of the agent network is calculated from the local Q value of all agents;

[0086] The agent network is divided into different sub-modules, and the influence of the action of the agent on other agents is judged, whether the action of the agent only affects the environmental information of the confrontation space or the agent itself also affects other agents; if the action of the agent only affects the environmental information of the space and the agent itself, the initial observation value of the agent is input into the embedding layer in the agent network training model, and the local Q value of the action of the agent is evaluated; if the action of the agent affects other agents, such as cooperative communication and attack, etc., the observation value o i,j of the agent i to the agent j is input into the embedding layer E in the agent network training model, and the action of the agent i affecting the agent j is input into the embedding layer E in the agent network training model, and the local Q value of the action of the agent is evaluated; the global Q value of the agent network is calculated from the local Q values of all agents;

[0087] In this embodiment, the agent network is divided into different sub-modules, the influence of the action of the agent on other agents is considered, and it is judged whether the action is or part of; if the action belongs to , the input initial observation value is generated through the embedding layer E , and then the local Q value of the action is evaluated; if the action belongs to , the observation value o i,j of the agent i to the agent j is input into the embedding layer E, and the action of the agent i affecting the agent j is input into the embedding layer E, and the local Q value of the agent i when the action belongs to is obtained The formula of

[0088]

[0089] Wherein, Q i is the local Q function of the i-th agent, is the hidden state of the agent i at the previous time step t-1, is the hidden state space, is the observation input, is the action at time step t;

[0090] The observation history is the observation history space;

[0091] The global Q value Q tot of the agent network is calculated from the local Q values of all agents;

[0092]

[0093] Wherein, τ t is the observation history at t, and u tFor the joint action at time t, F(·) is the agent credit allocation function, F is a monotonic function, which can be expressed as: Q i Q is the Q function of the i-th agent, is an agent set An agent in the agent set;

[0094] When matching the observation value of the agent with the action of the agent, there will always be a situation that the observation value of the agent does not match any action of the agent, that is, the observation value of the agent has no matching action of the agent; if these observation values of the agent are discarded, it will cause information loss to the agent;

[0095] The transformer is introduced into the agent network training model, the self-attention mechanism is used to identify and filter out useful information, and the observation value is weighted according to the obtained useful information; the attention is focused on the input features that are helpful to the agent, and the attention on irrelevant features is suppressed, and the useless information that is not needed for decision-making is discarded; the self-attention mechanism enables the agent to consider all observation values without direct matching, that is, the observation value and the action have no direct matching relationship, and the agent can still obtain useful information from these observation values, and the observation value is weighted according to the importance;

[0096] The useful information identified and filtered out by the self-attention mechanism is observation data or features that have substantial help and value to the agent when making decisions or performing tasks, which are perceived by the agent in the environment of the confrontation space and can directly or indirectly affect the behavior, decision-making process or task completion degree of the agent;

[0097] The network structure of the transformer is composed of the self-attention self-attention mechanism; first, each element in the input sequence is mapped to a query (Query) Q, a key (Key) K and a value (Value) V through linear transformation, and then the attention weight is weighted and summed, and the output is:

[0098]

[0099] Where, d k is a scaling factor related to K;

[0100] It is assumed that the Q, K and V matrices of each layer of the transformer are the same, that is, Where l is the number of layers of the transformer; the definition of the transformer is as follows:

[0101]

[0102] wherein MLP is a linear function for calculating Q, K, V, is the output calculated by the attention mechanism, using V i l to generate the input features of the next layer;

[0103] The last layer of the transformer is projected to the output space of the Q function Q i of the i-th agent by using an MLP:

[0104]

[0105] wherein Q i is the Q function of the i-th agent, is the hidden state of the agent i at the previous time step t-1, is the hidden state space, is the observation input, is the action at time step t;

[0106] Step 2.3: training the agent network training model by using a reinforcement learning algorithm;

[0107] Step 2.4: the agent perceives and understands the actions of other agents, and dynamically adjusts the decision of the agent according to the environmental information of the confrontation space;

[0108] Step 3: the confrontation parties (red and blue) in the UAV cluster are confronted, while the confrontation situation is evaluated and reasoned, and the local Q value generated by the agent network training model is received as input, so as to establish a double-layer hybrid network structure by using the monotonic value function decomposition method to obtain the global Q value Q tot of the agent network, and then obtain the optimal joint action of the agent;

[0109] The confrontation parties (red and blue) are confronted through the set confrontation space and the agent network training model; both confrontation parties strive to comprehensively collect confrontation data, observe and record the actions and environment of the confrontation party, estimate the state, and then evaluate the confrontation situation; based on the evaluation result of the confrontation situation, the reasoning and decision of the agent are dynamically adjusted;

[0110] The centralized training and distributed execution paradigm is adopted, the local Q values of each agent are integrated and constrained by a double-layer hybrid network to generate the global Q value; considering the additional state information available under the centralized training and distributed execution paradigm, the consistency and rationality of the value decomposition are ensured by using monotonicity constraint; each agent has an independent Q network and can make independent decisions according to its own local information, and the output of the Q network is transmitted through the double-layer hybrid network to realize the mapping of the local Q value to the global Q value Q tot ;

[0111] The double-layer hybrid network is responsible for credit allocation, taking the local Q value generated by the agent network as input, processing it through two separate supernetworks and activation functions contained in the double-layer hybrid network, and outputting the global Q value Q tot , and the specific formula is:

[0112] The weights of the double-layer hybrid network are generated by independent supernetworks using the real state s as input. Each supernetwork consists of a single linear layer followed by a ReLU function to ensure that the weights of the double-layer hybrid network are non-negative.

[0113] Q tot (τ,u,s;θ)=W2R eLU (W1Q+b1)+b2

[0114] wherein, is the Q value output by the agent network, Q n (τ n ,u n ) is the local Q value of the nth agent, τ n ,u n are the observation history and action of the nth agent, respectively, τ is the observation history, s is the environment state, θ is the parameter of the target network, R eLU is the activation function of W1Q+b1, and are the weight matrix and bias term generated by the two separate supernetworks, respectively, and satisfy: is a real matrix, and m and n are the row and column dimensions of the real matrix , respectively.

[0115] Since the monotonic value function decomposition algorithm QMIX has monotonicity, all elements in W1 and W2 are non-negative, i.e., satisfy

[0116] By ensuring that the agent only maximizes the local Q value based on its own local observation-action history, the global Q value is maximized, thereby ensuring that the joint action is the best action for the UAV swarm;

[0117] In this embodiment, the loss function of the double-layer hybrid network is updated by minimizing the squared TD error, and the loss function formula is as follows:

[0118]

[0119] wherein, is the TD target, τ', u', and s' are the observation-action history, joint action, and state at the next time, respectively, and θ -Copy from θ periodically for target network parameters;

[0120] The credit distribution is how to distribute the global credit or reward of the agent to each agent or local agent network in the cooperative confrontation of the UAV cluster or the multi-agent network training model; the credit is regarded as a quantitative evaluation of the contribution or performance of the agent, which reflects the importance and effectiveness of the agent in completing the task, cooperative combat or decision-making process;

[0121] The two separate super networks dynamically adjust their parameters or structures according to the input observation values to adapt to different environments;

[0122] Step 4: The two opposing parties (red and blue) plan the cooperative task to provide the optimal strategy for the multi-agent; update the local Q value of the agent, and optimize the agent strategy through the credit distribution method, the specific method is:

[0123] According to the global Q value Q tot obtained in step 3, the local Q value of the agent is updated by using the back propagation of the agent network training model; different agents have different degrees of influence on the confrontation game, and understanding the contribution of each agent to the global goal is the key to understanding the credit distribution, that is, the credit distribution is distinguishable; the global Q value Q tot of the agent is updated using gradient entropy to distinguish the credit distribution and optimize the strategy;

[0124] In this embodiment, in the value function decomposition paradigm, according to the Taylor expansion, the global Q value Q tot can be decomposed as:

[0125]

[0126] Wherein, represents the initial global Q value, o(Q T Q) represents a high-order small amount term, which is usually used to represent higher-order terms in Taylor expansion, [g1,…,g n ] T Q represents the gradient information of the output value of each agent network;

[0127] The influence of the local Q value on the global Q value Q tot depends on the corresponding gradient, the greater the gradient, the greater the influence on the global Q value Q tot , the greater the contribution to the entire agent network; since the credit distribution of the agent is different, the gradient distribution among the agents is also uneven, for this situation, the gradient is normalized, and the normalized gradient entropy is calculated, which measures the distinguishability of the contribution of the agent; the higher the gradient entropy, the more uniform the gradient distribution, and the more distinguishable the contribution among the agents;

[0128] Gradient normalization g i And the normalized gradient entropy is N orm :

[0129]

[0130] Where N orm As an evaluation index of distinguishability of credit distribution, logn is the maximum entropy of n values; g i represents the contribution of the ith agent to the global Q value Q tot If the credit distribution has high distinguishability, the gradient distribution will not be uniform; is the total gradient of all entities;

[0131] The gradient entropy is normalized using n random maximum entropies; The entropy is inversely proportional to the degree of gradient discrimination, that is, the lower the gradient entropy, the more uneven the gradient distribution, which will lead to the more difficult to distinguish the contribution between agents; The concept of normalized gradient entropy is introduced in the double-layer hybrid network to measure the allocability of credit; According to the expansion result of the global Q value Q tot , the gradient is calculated:

[0132]

[0133] So we get:

[0134] dQ tot = W2FW1dQ

[0135]

[0136] Where, is a diagonal matrix, and x c is the cth element of the output in the double-layer hybrid network, X is the input feature of the double-layer hybrid network, x1, x2, …, x m are the outputs of the first layer linear transformation in the double-layer hybrid network, which are calculated by W1Q+b1;

[0137] Introducing the loss function of action interaction and credit distribution Including two parts:

[0138] (1) Original TD loss That is, formula By maximizing the global Q value Q tot , each agent can learn the best strategy;

[0139] (2) Regularization gradient entropy

[0140]

[0141] wherein, λ is a hyper parameter used to balance the TD loss and the regularization term, and batchsize is the batch size of sampling;

[0142] When λ = 0, the gradient entropy is not calculated; the greater λ is, the stronger the punishment for indistinguishability is; the optimization of the regularization term only updates the parameters in the hybrid network, so that the double-layer hybrid network can more accurately measure the contribution of the agents; the obtained loss function is back-propagated to complete the update of the local Q value of the action value function and the policy, so that the agent can select a better policy to win the confrontation; the credit assignment is performed by introducing the gradient entropy, so that each agent can better understand the contribution of its action to the global reward, and the agents are prompted to better cooperate; the credit assignment distinguishable network structure is set as shown in Figure 3 The action interaction and credit assignment distinguishable network structure is introduced as shown in Figure 4 ;

[0143] Step 5: Repeat step 4 until the confrontation between the two parties ends, observe the confrontation performance of the action interaction and credit assignment, and complete the confrontation simulation of the UAV cluster;

[0144] In the embodiment, the confrontation game win rate curve of the UAV cluster confrontation between the two parties is shown in Figure 5 Through multiple experiments and observations, it is found that the win rate curve has limitations in some cases; in particular, the win rate may tend to be consistent at some time steps, and it is difficult to intuitively show the subtle differences and actual effects of the algorithm; in order to more accurately evaluate and compare the performance of different algorithms, the embodiment proposes to use the friendly party survival rate and the enemy party mortality rate as new evaluation indexes; wherein, the friendly party survival rate = the number of friendly party survivors / the original number of friendly parties; the enemy party mortality rate = the number of enemy party deaths / the original number of enemy parties;

[0145] In the embodiment, the friendly party survival rate and the enemy party mortality rate of the UAV cluster confrontation between the two parties using the method of the application and other algorithms of the multi-agent near-end policy optimization algorithm and the monotonic value function decomposition algorithm are shown in Figure 6 , 7 In the two graphs, the data in the left column are the friendly party survival rate and the enemy party mortality rate using the method of the application.

[0146] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the application, and not to limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent substitutions for some or all of the technical features; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the scope defined by the claims of the application.

Claims

1. A method for swarm cooperative confrontation of UAVs with action interaction and credit distribution, characterized in that: The method comprises the following steps: Step 1: modeling the confrontation space, action space, state space and observation space of the unmanned aerial vehicle cluster respectively, and setting the confrontation parties of the unmanned aerial vehicle cluster; Step 2: considering the action interaction between the unmanned aerial vehicles, regarding the unmanned aerial vehicles as intelligent agents independently and autonomously active, constructing an intelligent agent network training model, judging whether the intelligent agent action only affects the environmental information of the confrontation space or the intelligent agent action also affects other intelligent agents, and dynamically adjusting the decision of the intelligent agent according to the environmental information of the confrontation space; Step 3: the confrontation parties in the unmanned aerial vehicle cluster are in confrontation, and the confrontation situation is evaluated and reasoned, the local Q value generated by the intelligent agent network training model is received as input, a double-layer mixed network structure is established by using a monotonic value function decomposition method to obtain the global Q value of the intelligent agent network, and then the optimal joint action of the intelligent agent is obtained; Step 4: the confrontation parties perform cooperative task planning, and provide the optimal strategy for the multi-agent; the local Q value of the intelligent agent is updated, and the intelligent agent strategy is optimized by using a credit assignment method; Step 5: repeating step 4 until the confrontation between the confrontation parties ends, observing the confrontation performance of the action interaction and the credit assignment, and completing the confrontation simulation of the unmanned aerial vehicle cluster.

2. The method of claim 1, wherein: The step 1 comprises: The unmanned aerial vehicle cluster is composed of multiple unmanned aerial vehicles; the unmanned aerial vehicles of the confrontation parties in the confrontation scene adopt a six-degree-of-freedom dynamics model, the motion of the unmanned aerial vehicles in the continuous action space is described by using a dynamics equation, a continuous action set is adopted, and the continuous action parameters of the unmanned aerial vehicles are represented by a vector; A UAV swarm counter mission is described as a tuple The composed decentralized partially observable Markov decision process, wherein is a set of agents, is a set of states, is a set of actions, P is a state transition function, r is a reward function, Ω is a set of observations, O is an observation function, γ is a discount factor, γ ∈ [0, 1]; set is a global state describing the environment, at each time step t, each agent selects an action and generates a joint action U is a set of joint actions; according to the state transition function the environment enters the next state, so that the joint action u changes, wherein s' is the state of the agent at the next time; all agents share the same reward function is a set of real numbers, the role of r(s, u) is to evaluate the state s and the joint action u, and return a real value as a reward, to measure the contribution of the state and action; the common goal of all factors in the tuple is to maximize the expected reward wherein G represents the discounted sum of future rewards t find the expectation, the specific formula is: where k is a step size after time t, r t+k is the immediate reward obtained at time t+k.

3. The method of claim 2, wherein: The step 2 comprises: Step 2.1: constructing an intelligent agent network training model for each intelligent agent for learning the strategy and decision of the intelligent agent; Step 2.2: dividing the intelligent agent network into different sub-modules, evaluating the local Q value of the intelligent agent action through the influence of the intelligent agent action on other intelligent agents, and calculating the global Q value of the intelligent agent network from the local Q values of all intelligent agents; Step 2.3: training the intelligent agent network training model by using a reinforcement learning algorithm; Step 2.4: the intelligent agent perceives and understands the actions of other intelligent agents, and dynamically adjusts the decision of the intelligent agent according to the environmental information of the confrontation space.

4. The method of claim 3, wherein: The step 2.1 comprises: The intelligent agent network training model is constructed, so that the intelligent agent learns an adaptive strategy in the confrontation and reaches an optimal strategy in the mutually influencing environment; The observation value o and the action u input into the intelligent agent network training model are divided, and the initial observation value is divided into three parts: an environment-related part, a self-related part and an other-agent-related part; The observation o of the agent is converted from the sequence form to the matrix form, so that the representation of the observation-state space is unified and comparable; the action u is divided into two parts: and The action only affects the environment information and the agent itself; the action affects other agents.

5. The method of claim 4, wherein: The step 2.2 comprises: The agent network is divided into different sub-modules, and the influence of the agent action on other agents is judged to determine whether the agent action only affects the environmental information of the confrontation space or the agent itself also affects other agents; if the agent action only affects the environmental information of the space and the agent itself, the initial observation value of the agent is input into the embedding layer in the agent network training model, and the local Q value of the agent action is evaluated; if the agent action affects other agents, the observation value o i,j and the action of the agent i affecting the agent j is input into the embedding layer in the agent network training model, and the local Q value of the agent action is evaluated; the global Q value of the agent network is calculated from the local Q values of all agents; The transformer is introduced into the intelligent agent network training model, the self-attention mechanism is used to identify and filter out useful information, and the observation value is weighted according to the obtained useful information; attention is focused on the input features that are helpful to the intelligent agent, the attention on irrelevant features is suppressed, and useless information that is not needed for decision-making is discarded; The useful information identified and filtered by the self-attention mechanism is observation data or features that are substantially helpful and valuable for the agent when making decisions or performing tasks, which are perceived by the agent in the adversarial space and can directly or indirectly affect the behavior, decision-making process or task completion of the agent.

6. The method of claim 5, wherein: The step 3 comprises: The two parties of confrontation carry out confrontation through the set adversarial space and the agent network training model; both of the two parties of confrontation are committed to collecting confrontation data comprehensively, and estimate the state by observing and recording the actions and environment of the confrontation party, and then evaluate the confrontation situation; based on the evaluation result of the confrontation situation, the reasoning and decision of the agent are dynamically adjusted; The centralized training and distributed execution paradigm is adopted, a double-layer hybrid network is used to integrate and constrain the local Q values of each agent to generate global Q values; each agent has an independent Q network and can make independent decisions according to its own local information, the output of the Q network is transmitted through the double-layer hybrid network to realize the mapping of the local Q value to the global Q value.

7. The method of claim 6, wherein: The double-layer hybrid network is responsible for credit distribution, and takes the local Q value generated by the agent network as input, processes it through two separate supernetworks and activation functions contained in the double-layer hybrid network, and outputs a global Q value Q tot , and the specific formula is: Q tot (τ,u,s;θ) = W2R eLU (W1Q + b1) + b2 wherein, Q is the local Q-value output by the agent network, n (τ n , u n ) is the local Q-value of the nth agent, τ n , u n is the observation history and action of the nth agent, respectively, τ is the observation history, s is the environment state, θ is the parameter of the target network, R eLU is the activation function of W1Q + b1, and are the weight matrix and bias term generated by two separate hypernetworks, respectively, and satisfy: is a real matrix, m and n are the row and column dimensions of the real matrix , respectively. The loss function of the double-layer hybrid network is updated by minimizing the square TD error, and the loss function formula is as follows: where, are the observed-action history, joint action, and state at the next time step for the TD target, τ', u', s', respectively, and θ - are the target network parameters, which are periodically copied from θ; The two separate super networks dynamically adjust their parameters or structures according to the input observation values to adapt to different environments.

8. The method of claim 7, wherein: The step 4 comprises: According to the global Q value Q obtained in step 3 tot , the reverse transmission of the model trained by the agent network is used to update the local Q value of the agent; the global Q value of the agent is updated using gradient entropy to distinguish the credit assignment and optimize the strategy; The concept of normalized gradient entropy is introduced in the double-layer hybrid network to measure the allocability of credit. According to the expansion result of global Q value Q tot , the gradient is calculated: Thus, the following formula is obtained: dQ tot = W2F W1dQ wherein, is a diagonal matrix, and x c is the cth element of the output of the double-layer hybrid network, X is the input feature of the double-layer hybrid network, x1, x2, …, x m are all outputs of the first layer linear transformation in the double-layer hybrid network, and are calculated through W1Q+b1. Loss function introducing action interaction and credit assignment Including original TD loss And regularized gradient entropy Two parts, as shown in the following formula: Wherein, λ is a hyperparameter used to balance the TD loss and the regularization term, and batchsize is the batch size of sampling. The two separate super networks dynamically adjust their parameters or structures according to the input observation values to adapt to different environments. The step 4 comprises: Thus, the following formula is obtained: Wherein, λ is a hyperparameter used to balance the TD loss and the regularization term, and batchsize is the batch size of sampling.

Images