A multi-unmanned aerial vehicle cooperative confrontation method and system based on role representation and mutual information constraint

Abstract

The application relates to a multi-unmanned aerial vehicle cooperative confrontation method and system based on role representation and mutual information constraint, a multi-unmanned aerial vehicle cooperative confrontation scene is established; local observation information and joint action trajectory are acquired, a role coding network is trained by sampling batch data, potential semantic role representation of the unmanned aerial vehicle of the own side is extracted by using the role coding network after training; according to a dynamic reward threshold, the collected trajectory data and the corresponding potential semantic role representation are stored in a two-channel storage structure in a hierarchical manner, a mutual information estimator is synchronously trained; based on an internal reward and punishment mechanism and the mutual information estimator after training, a corrected total reward signal is calculated, a strategy network is evaluated and optimized, a strategy is updated through gradient back propagation of a mixed network, and the cooperative confrontation strategy of the multi-unmanned aerial vehicle of the own side is optimized; and an antagonistic system is realized based on the method. The application provides high-level semantic support for decision-making in a complex confrontation environment, improves convergence efficiency and robustness, and improves the accuracy of global value evaluation and credit distribution.

Description

Technical Field

[0001] This invention relates to the technical field of computer systems based on specific computational models, and particularly to a multi-UAV cooperative adversarial method and system based on role representation and mutual information constraints in the fields of machine learning and deep reinforcement learning. Background Technology

[0002] Unmanned Aerial Vehicles (UAVs) have become a research hotspot in both civilian and military fields due to their low maintenance costs and high mobility. Because individual UAVs have limited capabilities and are easily damaged, leading to a higher probability of mission failure, research has shifted towards UAV swarms to achieve collective intelligence and improve efficiency and performance. The development of UAV swarm technology has further promoted research on UAV swarm warfare. Multiple UAVs can perform tactical operations with high coordination, such as strikes, cover, and intelligence gathering, enhancing their application in the military and security fields and becoming an important research direction in these areas. Simultaneously, this technology has also promoted automated production and management in industry, logistics, and agriculture, fostering the development of intelligent application technologies. UAV swarm warfare requires UAVs to possess autonomous judgment, planning, and decision-making capabilities, as well as the ability to achieve information exchange and coordinated action among swarms.

[0003] Against this backdrop, intelligent decision-making algorithms based on game theory and reinforcement learning have gradually become an important research direction for collaborative decision-making in UAV swarms. Through continuous interaction with the environment and adaptive policy updates, these algorithms enable UAV swarms to autonomously select better penetration paths and combat strategies in complex, dynamic, and highly adversarial combat environments, thereby improving overall collaborative combat capabilities. However, existing technologies still have certain shortcomings.

[0004] Chinese patent CN120688830A discloses a dynamic allocation method for multi-UAV collaborative tasks. It constructs an initial allocation model for multi-UAV collaborative tasks based on static factors, uses an improved K-means clustering algorithm to optimally group task points, determines the optimal ranking of task points within each group using an ant colony algorithm, and solves the initial allocation problem using an improved genetic algorithm based on the initial allocation model and groupings. After initial allocation based on the solution, a dynamic redistribution model for multi-UAV collaborative tasks is constructed based on dynamic factors. Based on the dynamic redistribution model, a dynamic redistribution method is then employed.

[0005] Operations research approaches are used to solve dynamic redistribution problems, and dynamic redistribution is performed based on the results. While this method effectively solves the problem of rapidly allocating large-scale task points in physical space, its division of labor logic is still limited to the mechanical assignment of geographical coordinates, lacking explicit modeling and semantic representation of the deep tactical roles of UAVs.

[0006] Chinese patent CN120143849A discloses a cooperative adversarial method for UAV swarms based on BC-MAPPO. This method introduces behavioral cloning to initialize the policy network parameters using expert experience, and then further optimizes the network parameters through reinforcement learning, ultimately effectively improving the cooperative adversarial capabilities of the UAV swarm. However, because all UAVs share the same set of network parameters and lack explicit differentiated role guidance, the UAV swarm is prone to converge to a single homogeneous strategy in complex tasks.

[0007] Chinese patent CN119598825A discloses a method and system for multi-UAV cooperative adversarial decision-making. It constructs a multi-UAV cooperative air combat adversarial decision-making environment by establishing a multi-UAV air combat adversarial motion model and an air combat situation assessment model. Based on the action space, local observations, and states of each UAV in the adversarial decision-making environment, a distributed partially observable Markov decision process model for the multi-UAV cooperative adversarial decision-making problem is established. A multi-UAV cooperative adversarial reward function and a HASAC algorithm network space are designed. Based on the interaction between the HASAC algorithm network space and the multi-UAV cooperative adversarial decision-making environment, a multi-UAV cooperative adversarial strategy model is trained and generated. Compared to directly concatenating the observation vectors of each UAV, this method reduces the global state dimension and effectively improves training efficiency. However, when facing complex adversarial situations, the algorithm struggles to determine whether the current multi-UAV cooperation is in an optimal cooperative mode, resulting in a lack of clear, high-quality cooperative signals to guide strategy updates and a tendency to fall into suboptimal local equilibria. Summary of the Invention

[0008] To address the problems of unclear role division, homogenized strategies, and difficulty in assessing the quality of collaboration in complex adversarial environments, existing technologies for drone swarms provide a multi-drone cooperative adversarial method and system based on role representation and mutual information constraints.

[0009] The technical solution adopted in this invention is a multi-UAV cooperative adversarial method based on role representation and mutual information constraints. It establishes a multi-UAV cooperative adversarial scenario in which both sides contain at least two UAVs, defines an inherent reward and punishment mechanism, and initializes a role encoding network, a mutual information estimator, a dual-channel storage structure associated with the mutual information estimator, a policy network, and a hybrid evaluation network.

[0010] Control multiple friendly drones to interact with the adversarial scenario, obtain local observation information and joint action trajectory of each drone, sample batch data from them, train the role encoding network, and use the trained role encoding network to extract the potential semantic role representation of each friendly drone.

[0011] Based on the dynamic reward threshold, the collected trajectory data and its corresponding latent semantic role representations are hierarchically stored in the dual-channel storage structure, and the mutual information estimator is trained synchronously using the hierarchically stored information.

[0012] Based on the inherent reward and punishment mechanism and the trained mutual information estimator, the corrected total reward signal is calculated. The corrected total reward signal is used to evaluate and optimize the policy network. The gradient is backpropagated through the hybrid network to update the policy and optimize the cooperative combat strategy of the player's multiple UAVs.

[0013] Preferably, the role encoding network includes a sequentially connected temporal feature extraction network and a role mapping network; the temporal feature extraction network includes a gated recurrent unit, and the role mapping network includes a fully connected layer;

[0014] The current observation and the previous action of each of our drones are input into the gating loop unit to extract trajectory feature vectors containing historical behavior patterns, and then mapped into low-dimensional semantic role representations through a fully connected layer.

[0015] Preferably, the dual-channel storage structure includes an advantage experience buffer and a disadvantage experience buffer; the dynamic reward threshold is the historical moving average total return; the total return of the cooperative confrontation after each confrontation is compared with the dynamic reward threshold. If the total return is greater than the threshold, it is determined to be an advantageous strategy trajectory and stored in the advantageous experience buffer; otherwise, it is determined to be a disadvantageous strategy trajectory and stored in the disadvantageous experience buffer.

[0016] Preferably, the cosine similarity between the semantic role representations of any two friendly drones within a batch is calculated, a basic affinity matrix is ​​constructed, and a correction matrix containing a preset penalty factor is introduced to record and suppress pairings with poor historical performance, thereby training the role encoding network.

[0017] If two drones were previously assigned to the same role cluster in the historical trajectory of the disadvantage experience buffer, a preset penalty value is assigned to the corresponding position in the correction matrix; otherwise, it is 0. The correction matrix is ​​subtracted from the basic affinity matrix and nonnegated to obtain the corrected affinity matrix. The corrected affinity matrix is ​​input into the spectral clustering module to generate cluster labels. A contrastive loss function is constructed based on the cluster labels, and the role encoding network is iteratively trained.

[0018] Preferably, the mutual information estimator includes a lower bound estimator and an upper bound estimator; the lower bound estimator is configured with a dominant experience buffer, and the upper bound estimator is configured with a suboptimal experience buffer.

[0019] Preferably, the lower bound estimator includes a state feature branch and a role feature branch, used to map the global state and joint role representation to the same high-dimensional feature space, and calculate the joint statistic based on Fenchel duality principle to estimate the lower bound of mutual information; the lower bound estimator updates the parameters by minimizing the first loss function, which is constructed based on the advantageous sample pairs sampled in the advantage experience buffer and the unpaired samples in shuffled order;

[0020] The upper bound estimator is constructed as a conditional probability density prediction network comprising a mean prediction subnetwork and a variance prediction subnetwork. The mean prediction subnetwork and the variance prediction subnetwork output the predicted mean and predicted log-variance respectively based on the input global state to construct a Gaussian distribution that approximates the true conditional distribution, thereby estimating the upper bound of mutual information. The upper bound estimator updates the parameters by minimizing the gradient of a second loss function, which is the negative log-expectation of the conditional probability density function predicted by the network.

[0021] Preferably, the intrinsic reward and punishment mechanism is to use the trained mutual information estimator to calculate the lower bound of mutual information as a positive incentive, use the trained mutual information estimator to calculate the upper bound of mutual information as a negative penalty, add the original reward from the environmental feedback to the positive incentive, and subtract the negative penalty to obtain the corrected reward for policy updating.

[0022] Preferably, the hybrid evaluation network introduces an attention mechanism to generate weights for evaluating the value of joint actions, and the generated weights satisfy the following:

[0023]

[0024] in, The global state, after linear transformation, serves as the query vector. This is a representation of the roles of all drones after transformation, serving as key vectors. The feature dimension is used for nonlinear weighted fusion of the local Q-values ​​of each UAV.

[0025] Preferably, the parameters of the policy network are updated using the corrected total reward signal and the backpropagation gradient of the hybrid evaluation network;

[0026] During forward computation, the local Q-values ​​of each UAV are fused into a global joint action value using the hybrid evaluation network. During backward update, the time-series difference objective is calculated using the modified reward to construct a mean squared error loss function. The gradient is calculated by minimizing this loss function, and the gradient is backpropagated to the policy network of each UAV via the hybrid evaluation network and the weights generated according to the multi-head attention mechanism, thereby updating the parameters of the policy network.

[0027] A multi-UAV cooperative combat system based on role representation and mutual information constraints includes:

[0028] At least one processor; and

[0029] A memory communicatively connected to at least one of the processors; wherein,

[0030] The memory stores instructions that can be executed by the processor to implement the multi-UAV cooperative adversarial method based on role representation and mutual information constraints.

[0031] This invention relates to a multi-UAV cooperative adversarial method and system based on role representation and mutual information constraints. The method establishes a multi-UAV cooperative adversarial scenario, defines an inherent reward and punishment mechanism, and initializes a role encoding network, a mutual information estimator, a dual-channel storage structure associated with the mutual information estimator, a policy network, and a hybrid evaluation network. It controls the interaction of the friendly multi-UAVs with the adversarial scenario, acquiring local observation information and joint action trajectories of each UAV, sampling batch data from these, training the role encoding network, and using the trained role encoding network to extract latent semantic role representations of each friendly UAV. Based on a dynamic reward threshold, the collected trajectory data and their corresponding latent semantic role representations are hierarchically stored in the dual-channel storage structure, and the hierarchically stored information is used to synchronously train the mutual information estimator. Based on the inherent reward and punishment mechanism and the trained mutual information estimator, a corrected total reward signal is calculated, and the corrected total reward signal is used to evaluate and optimize the policy network. The gradient is backpropagated through the hybrid network to update the policy, optimizing the cooperative adversarial strategy of the friendly multi-UAVs. The adversarial system is implemented based on this method.

[0032] The beneficial effects of this invention are as follows:

[0033] (1) Break through the bottleneck of tactical homogenization in large-scale cluster collaboration;

[0034] By using a temporal encoder and a contrastive learning module, the observation-action trajectory of the UAV at the bottom layer is mapped into a potential role representation with clear semantics. This overcomes the problem of convergence of agent behavior in traditional methods, enabling UAV swarms to spontaneously emerge with differentiated tactical division of labor under unsupervised conditions, and providing high-level semantic support for decision-making in complex adversarial environments.

[0035] (2) Solve the problems of inaccurate selection of collaborative modes and difficulties in high-dimensional calculation;

[0036] By using a dual-channel experience replay and dual mutual information constraint mechanism, the algorithm accurately distinguishes and strengthens the advantageous collaborative mode of "high benefit" and suppresses the suboptimal grouping behavior of "high correlation but low efficiency", which significantly improves the convergence efficiency and robustness of the algorithm in strong adversarial scenarios.

[0037] (3) Improve the accuracy of overall value assessment and credit allocation;

[0038] By constructing an attention-oriented hybrid network and using the global battlefield situation as the query vector, the attention weights of different role representations are dynamically calculated. This allows the network to automatically focus on the most critical execution unit in the current tactic during the value decomposition process, effectively solving the problem of biased evaluation of the contribution of key agents in complex scenarios by traditional methods. Attached Figure Description

[0039] Figure 1 This is a flowchart of the method of the present invention;

[0040] Figure 2 This is a schematic diagram of the confrontation scenario of the present invention;

[0041] Figure 3 This is a diagram of the training algorithm framework based on role representation and mutual information constraints of the present invention.

[0042] Figure 4 This is a diagram showing the experimental application effect of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0044] This invention relates to a multi-UAV cooperative combat method based on role representation and mutual information constraints. In a specific implementation, a combat scenario involving red and blue UAVs in beyond-visual-range is constructed on the Mozi simulation platform. Each side deploys four UAVs equipped with radar systems capable of detecting enemy UAVs and weapons within their detection range and acquiring target coordinates in real time (initialized between the two sides). The weapons possess the capability to engage targets within their attack range. The two sides initially face each other and maintain the same altitude. In this embodiment, both UAVs fly at a speed of 648.2 km / h, an altitude of 10.973 km, and a distance of 720 km from the target point. Figure 2 As shown, this invention establishes a multi-UAV air combat scenario, acquires UAV observation information, mission status and adversarial feedback signals, and defines adversarial evaluation indicators to measure the overall combat performance of UAVs (such as attack success rate). It extracts temporal features of observation-action trajectories using a role encoder, and generates highly recognizable semantic roles through contrastive learning. Furthermore, it utilizes an attention mechanism to construct a hybrid network, enabling global state focus on key role information and addressing the problem of tactical homogenization in traditional methods. Finally, it constructs a dual-channel experience replay mechanism for advantages and disadvantages, combined with mutual information upper and lower bound estimation, to quantify the effectiveness of UAV role coordination and promote the stability and robustness of the strategy optimization process.

[0045] This invention includes the following steps:

[0046] (1) Establish a multi-drone cooperative combat scenario in which both sides have at least 2 drones, define the internal reward and punishment mechanism, and initialize the role coding network, mutual information estimator, dual-channel storage structure associated with the mutual information estimator, policy network and hybrid evaluation network.

[0047] (2) Control multiple UAVs of our own to interact with the adversarial scenario, obtain local observation information and joint action trajectory of each UAV, sample batch data from them, train the role encoding network, and use the trained role encoding network to extract the potential semantic role representation of each UAV of our own.

[0048] (3) Based on the dynamic reward threshold, the collected trajectory data and its corresponding latent semantic role representations are hierarchically stored in the dual-channel storage structure, and the mutual information estimator is trained synchronously using the hierarchically stored information.

[0049] (4) Based on the intrinsic reward and punishment mechanism and the trained mutual information estimator, the corrected total reward signal is calculated. The corrected total reward signal is used to evaluate and optimize the policy network. The gradient is backpropagated through the hybrid network to update the policy and optimize the cooperative confrontation strategy of our multi-UAV.

[0050] The method will be described below with reference to specific implementation methods.

[0051] (1) Establish a multi-drone cooperative combat scenario in which both sides have at least 2 drones, define the internal reward and punishment mechanism, and initialize the role coding network, mutual information estimator, dual-channel storage structure associated with the mutual information estimator, policy network and hybrid evaluation network.

[0052] (1-1) Cooperative confrontation scenario

[0053] First, a mathematical model of a cooperative combat scenario is constructed to obtain the situation of each friendly drone during a multi-drone cooperative combat process. With all enemy drones Distance vector between Euclidean distance vector Attack angle and the angle of attack :

[0054]

[0055] in, and The two-dimensional coordinates representing the red and blue drones, and The drones representing the red and blue teams respectively were in shaft and The speed on the axis can be controlled by adjusting the speed increment of each drone. and heading angle increment To perform real-time control, in this example , .

[0056] The cooperative adversarial task of the aforementioned drone swarm is modeled as a multi-agent Markov game, represented as a tuple. :

[0057]

[0058] in, This refers to the number of friendly drones; in this example, it is 4. This is battlefield situational information, presented in 33 dimensions in this embodiment. For each drone, a local collection of observations. For the actions of each drone, This is the state transition function. As a discount factor, it generally satisfies In this embodiment, 0.95 is used. The reward function for each drone.

[0059] In this embodiment, the reward function for each drone is based on a constructed multi-level scoring index system; the original environmental reward... It consists of survival rewards, kill rewards, and aerial combat advantage rewards. The aerial combat advantage reward... The calculation formula is:

[0060]

[0061] in, This represents the total time step; Represents "safety" (the angle between one's own line of sight and the enemy's attack direction; the larger the angle, the safer). Represents "aggressiveness" (the angle between your attack direction and the enemy's position; the smaller the angle, the more accurate). This constitutes the original environmental reward. Part of it.

[0062] (1-2) Role Encoder Network

[0063] A role encoding network is constructed to extract semantic role representations.

[0064] The role encoding network includes a sequentially connected temporal feature extraction network and a role mapping network; the temporal feature extraction network includes a gated recurrent unit, and the role mapping network includes a fully connected layer.

[0065] Specifically, with each of our own drones The current moment (Local) observations and the action at the previous moment (After splicing) Input gated loop unit, in this embodiment, local observation The dimension is 33, and the action at the previous moment... One-hot encoding is used, with a 9-dimensional action space. The total dimension of the concatenated input vector is 42. GRU is used to extract trajectory feature vectors containing historical behavior patterns. ;

[0066] The output will contain trajectory feature vectors that include historical behavior patterns. Mapped to low-dimensional semantic role representations through fully connected layers. .

[0067] (1-3) Dual-channel storage structure

[0068] The dual-channel storage structure includes an advantage experience buffer and a disadvantage experience buffer; the dynamic reward threshold is the historical moving average total return; the total return of the cooperative confrontation after each game is compared with the dynamic reward threshold. If the total return is greater than the threshold, it is determined to be an advantageous strategy trajectory and stored in the advantageous experience buffer; otherwise, it is determined to be a disadvantageous strategy trajectory and stored in the disadvantageous experience buffer.

[0069] (2) Control multiple UAVs of our own to interact with the adversarial scenario, obtain local observation information and joint action trajectory of each UAV, sample batch data from them, train the role encoding network, and use the trained role encoding network to extract the potential semantic role representation of each UAV of our own.

[0070] Based on complex and incomplete battlefield situation information To obtain a set of state information ,in, For drones Partial observation information, such as location coordinates ,speed and direction of action hour, , This is part of the observation information from our own drones. , This is part of the observation information obtained from detecting enemy drones. This is an attack method used by enemy drones;

[0071] The set of state information of all UAVs (if no enemy UAV is observed, the corresponding value is set to 0) is normalized to obtain normalized state information.

[0072] To ensure that the generated characters have clear tactical semantics (such as "main attack" and "feint attack") and are diverse, an adaptive spectral clustering method based on the correction matrix combined with contrastive learning is used for training.

[0073] Specifically, calculate the semantic role representations of any two drones within the batch. and Cosine similarity between them, constructing the basic affinity matrix The basic affinity matrix For a symmetric matrix, its first... Line number Column elements Represented as:

[0074]

[0075] In the formula, Represents the vector dot product. Represents the magnitude of a vector.

[0076] Introduce a correction matrix containing a preset penalty factor. The basic affinity matrix is ​​modified to record and suppress pairings that have historically performed poorly, resulting in the modified affinity matrix. ,Right now = - By subtracting, the similarity scores of those "historically poor combinations" are artificially reduced, ensuring that the system can proactively break inefficient and fixed tactics and force the agent to explore new combinations of role divisions.

[0077] If two drones were previously assigned to the same role cluster in the historical trajectory of the disadvantage experience buffer, a preset penalty value is assigned to the corresponding position in the correction matrix; otherwise, it is 0. The correction matrix is ​​subtracted from the basic affinity matrix and nonnegated to obtain the corrected affinity matrix. The corrected affinity matrix is ​​input into the spectral clustering module to generate cluster labels. A contrastive loss function is constructed based on the cluster labels, and the role encoding network is iteratively trained.

[0078] Will The input spectral clustering module uses K-Means to generate cluster labels during the feature vector clustering stage, and constructs a contrastive loss function based on the cluster labels. ,

[0079]

[0080] in, The cosine similarity function is used. For temperature coefficient, Positive samples within the same cluster, i.e., trajectories with similar tactical intentions. For negative samples, that is, for each role sample in the batch Other samples within the same cluster are considered positive samples. Samples from different clusters are considered negative samples. ;

[0081] By minimizing the contrastive loss, the character encoding network is trained to bring character vectors within the same cluster closer together and widen the distance between different clusters, thereby forming a clearly defined character distribution in the feature space.

[0082] (3) Based on the dynamic reward threshold, the collected trajectory data and its corresponding latent semantic role representations are hierarchically stored in the dual-channel storage structure, and the mutual information estimator is trained synchronously using the hierarchically stored information.

[0083] (3-1) Mutual Information Estimator

[0084] This invention no longer relies on manually designed static weights, but instead dynamically generates intrinsic guided rewards through dual-channel experience playback and mutual information estimators.

[0085] The mutual information estimator includes a lower bound estimator and an upper bound estimator; the lower bound estimator is coupled with a superior experience buffer. The upper bound estimator is configured in conjunction with a disadvantaged experience buffer. The current strategy's trajectory is determined by the total reward of the cooperative combat after each round and stored in either the advantage experience buffer or the disadvantage experience buffer.

[0086] Total reward for cooperative combat after each round of combat. Greater than the dynamic reward threshold When the current strategy's trajectory is determined, it is stored in the advantageous experience buffer; otherwise, it is stored in the disadvantageous experience buffer. The dynamic reward threshold is the historical moving average total return. ,in, It is the smoothing coefficient; that is, if > The current tactical coordination is determined to be "advantageous synergy," and the trajectory of this game is saved. This indicates that the character combination (e.g., "3 attack, 1 defense") has generated a positive benefit; conversely, if it has not, the current tactical combination is judged as "disadvantageous coordination," and the trajectory of this game is stored. This indicates that the combination of roles led to a suboptimal result.

[0087] A lower bound estimator is provided after incorporating the advantage experience buffer. An upper bound estimator is provided after incorporating a disadvantaged experience buffer. .

[0088] The lower bound estimator includes a state feature branch and a role feature branch, used to estimate the global state. and joint role representation Mapped to the same high-dimensional feature space, joint statistics are calculated based on Fenchel duality principle to estimate the lower bound of mutual information; the lower bound estimator updates the parameters by minimizing the first loss function, which is constructed based on the dominant sample pairs sampled in the dominant experience buffer and the shuffled unpaired samples.

[0089] Specifically, the input to the state feature branch is the global state vector. After passing through the first fully connected layer (output dimension set to 32), it is then connected to the Leaky ReLU activation function, outputting a state embedding vector. ;

[0090] The input to the character feature branch is the joint character representation vector. After passing through a second fully connected layer (with an output dimension of 32), the Leaky ReLU activation function is applied to output the character embedding vector. ;

[0091] The statistic is calculated as follows: and As input, the Jensen-Shannon divergence is calculated based on the Fenchel Dual principle, thereby outputting an estimate of the lower bound of mutual information.

[0092] The upper bound estimator is constructed as a conditional probability density prediction network containing a mean prediction subnetwork and a variance prediction subnetwork, used to approximate the true conditional distribution. The mean prediction subnetwork and variance prediction subnetwork output the predicted mean and predicted log-variance respectively based on the input global state to construct a Gaussian distribution to approximate the true conditional distribution, thereby estimating the upper bound of mutual information; the upper bound estimator updates the parameters by minimizing the second loss function, which is the negative log expectation of the conditional probability density function predicted by the network.

[0093] Specifically, assume that the posterior distribution follows a Gaussian distribution. The network consists of two subnetworks: mean prediction and variance prediction. The input to the mean prediction network is the global state vector. In the hidden layer, the first fully connected layer maps the input to half the dimension of the hidden layer and applies a ReLU activation function. In the output layer, the second fully connected layer maps the hidden layer features to the character representation dimension and outputs the predicted mean vector. The input to the variance prediction network is the global state vector. The hidden layer has the same structure as the mean prediction network, i.e., the first fully connected layer is followed by a ReLU activation function, and the second fully connected layer in the output layer maps the hidden layer features to the character representation dimension, followed by a Tanh activation function, outputting the predicted log-variance vector. Based on the predicted mean and variance Construct a Gaussian distribution and calculate the difference between the log-likelihood of positive sample pairs and the log-likelihood of negative sample pairs as an upper bound estimate of mutual information.

[0094] During policy updates, a positive incentive term is calculated using the trained lower bound estimator, and a negative penalty term is calculated using the trained upper bound estimator; these are combined to obtain the intrinsic reward. The intrinsic reward is then combined with the original environmental reward. Adjust the reward strategy.

[0095] Specifically, from Mid-sampling, by minimizing the first loss function parameters of the lower bound estimator Perform gradient updates to make the network output approximate the global state. With joint role representation The lower bound of mutual information,

[0096]

[0097] in, For the dominant sample pair, For unpaired samples in shuffled order, This is the Softplus activation function;

[0098] from Mid-sampling, by minimizing the second loss function upper bound estimator parameters Perform gradient updates to make the network output approximate the global state. With joint role representation The upper bound of mutual information (i.e. the conditional probability distribution of the prediction).

[0099]

[0100] in, This represents the conditional probability density function for network prediction.

[0101] (4) Based on the intrinsic reward and punishment mechanism and the trained mutual information estimator, calculate the corrected total reward signal, use the corrected total reward signal to evaluate and optimize the policy network, backpropagate gradient through the hybrid network to update the policy, and optimize the cooperative confrontation strategy of our multi-UAVs.

[0102] The intrinsic reward and punishment mechanism is to use the trained mutual information estimator to calculate the lower bound of mutual information as a positive incentive, and use the trained mutual information estimator to calculate the upper bound of mutual information as a negative penalty. The original reward from the environmental feedback is added to the positive incentive and the negative penalty is subtracted to obtain the corrected reward for policy updating.

[0103] Specifically, during policy updates, the trained estimator is used to calculate the intrinsic reward: a positive incentive term (a lower bound estimate of mutual information) is calculated. This encourages agents to reproduce highly cooperative patterns within the advantage buffer; it also calculates a negative penalty term (an estimate of the upper bound of mutual information). This suppresses inefficient collaborative patterns that agents repeatedly use in the disadvantage buffer; ultimately, the reward is adjusted to...

[0104]

[0105] in, As a natural reward for the environment, and This is the adjustment coefficient, in this example =0.01, =0.001.

[0106] In this invention, an attention-oriented mixing network is constructed to evaluate the value of joint actions. The hybrid evaluation network introduces an attention mechanism to generate weights for evaluating the value of joint operations. This attention mechanism enables dynamic weight allocation, incorporating global battlefield situational information. Mapping to query vectors to represent the semantic roles of all drones The weights of the hybrid network are generated by mapping key vectors (Key) and value vectors (Value) through a multi-head attention mechanism. ,

[0107]

[0108] in, The global state, after linear transformation, serves as the query vector. This is a representation of the roles of all drones after transformation, serving as key vectors. The feature dimension is used for nonlinear weighted fusion of the local Q-values ​​of each UAV.

[0109] The parameters of the policy network are updated by backpropagating the gradient using the corrected total reward signal and the hybrid evaluation network;

[0110] During forward computation, the local Q-values ​​of each UAV are fused into a global joint action value using the hybrid evaluation network. During backward update, the temporal difference objective is calculated using the modified reward, and a mean squared error loss function is constructed. The gradient is calculated by minimizing this loss function, and the gradient is backpropagated to the policy network of each UAV via the hybrid evaluation network and the weights generated according to the multi-head attention mechanism.

[0111] Specifically, when the situation indicates that the enemy's core has been exposed, the attention mechanism will automatically assign higher weight to the drones playing the "main attack" role, making them... Value in the global It occupies a dominant position.

[0112] During the network update phase, batch data is sampled from the experience pool;

[0113] During forward computation, a hybrid attention network is used to integrate the local data of each drone. value Integration into the value of overall joint action During reverse updates, the temporal difference objective is calculated using the modified reward function.

[0114]

[0115] in, This is to determine the overall battlefield situation in the next moment. For the coordinated actions of all drones in the next moment, As a representation of the joint roles in the next moment, As a discount factor, This indicates choosing the combined action that will maximize value in the next moment.

[0116] Construct the mean squared error loss function:

[0117]

[0118] The gradient is calculated by minimizing the loss function. The gradient is then backpropagated through the hybrid network and according to the attention weights to the policy network of each drone, thereby optimizing the cooperative adversarial strategy of the multiple drones.

[0119] Through end-to-end training, drone swarms will learn to dynamically adjust their roles based on the battlefield situation and effectively avoid inefficient coordination traps.

[0120] This invention also relates to a multi-UAV cooperative combat system based on role representation and mutual information constraints, comprising:

[0121] At least one processor; and

[0122] A memory communicatively connected to at least one of the processors; wherein,

[0123] The memory stores instructions that can be executed by the processor to implement the multi-UAV cooperative adversarial method based on role representation and mutual information constraints.

[0124] During the simulation, the server connects to the designed 4v4 air combat environment of the Mozi simulation platform via a fixed IP and port number; the proposed algorithm is deployed on the red team's UAV, and the blue team adopts a fixed direct flight strategy to conduct 1000 rounds of training and simulation until the reward function curve converges.

[0125] Experimental verification and comparative experiments were conducted, such as Figure 4 As shown, QMIX is the classic framework, Role is the role representation mechanism, and MI is the mutual information constraint mechanism. Through ablation experiments, the role representation module and the mutual information constraint module are fused with the QMIX algorithm. The experimental comparisons are the basic QMIX algorithm, the role representation QMIX algorithm, and the QMIX algorithm with added role representation and mutual information constraints. Through comparison and verification, each module has a certain improvement on the method, demonstrating the superiority of the method of this invention.

[0126] Finally, training can be conducted on both the red and blue drones in sequence to generate an air combat confrontation system between the two sides.

[0127] The present invention also relates to an application of the aforementioned multi-UAV cooperative adversarial method based on role representation and mutual information constraints, which is applied to the optimization strategy for obtaining multi-UAV adversarial scenarios in beyond-visual-range adversarial scenarios.

[0128] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0129] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0130] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0131] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0132] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0133] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A multi-UAV cooperative adversarial method based on role representation and mutual information constraints, characterized in that: Establish a multi-drone cooperative combat scenario in which both sides have at least two drones, define the inherent reward and punishment mechanism, and initialize the role encoding network, mutual information estimator, dual-channel storage structure associated with the mutual information estimator, policy network and hybrid evaluation network. Control multiple friendly drones to interact with the adversarial scenario, obtain local observation information and joint action trajectory of each drone, sample batch data from them, train the role encoding network, and use the trained role encoding network to extract the potential semantic role representation of each friendly drone. Based on the dynamic reward threshold, the collected trajectory data and its corresponding latent semantic role representations are hierarchically stored in the dual-channel storage structure, and the mutual information estimator is trained synchronously using the hierarchically stored information. Based on the inherent reward and punishment mechanism and the trained mutual information estimator, the corrected total reward signal is calculated. The corrected total reward signal is used to evaluate and optimize the policy network. The gradient is backpropagated through the hybrid network to update the policy and optimize the cooperative combat strategy of the player's multiple UAVs.

2. The multi-UAV cooperative adversarial method based on role representation and mutual information constraints according to claim 1, characterized in that: The role encoding network includes a sequentially connected temporal feature extraction network and a role mapping network; the temporal feature extraction network includes a gated recurrent unit, and the role mapping network includes a fully connected layer. The current observation and the previous action of each of our drones are input into the gating loop unit to extract trajectory feature vectors containing historical behavior patterns, and then mapped into low-dimensional semantic role representations through a fully connected layer.

3. The multi-UAV cooperative adversarial method based on role representation and mutual information constraints according to claim 1, characterized in that: The dual-channel storage structure includes an advantage experience buffer and a disadvantage experience buffer; the dynamic reward threshold is the historical moving average total return; the total return of the cooperative confrontation after each game is compared with the dynamic reward threshold. If the total return is greater than the threshold, it is determined to be an advantageous strategy trajectory and stored in the advantageous experience buffer; otherwise, it is determined to be a disadvantageous strategy trajectory and stored in the disadvantageous experience buffer.

4. The multi-UAV cooperative adversarial method based on role representation and mutual information constraints according to claim 3, characterized in that: Calculate the cosine similarity between the semantic role representations of any two friendly drones within a batch, construct a basic affinity matrix, introduce a correction matrix containing a preset penalty factor to record and suppress pairings with poor historical performance, and train the role encoding network. If two drones were previously assigned to the same role cluster in the historical trajectory of the disadvantage experience buffer, a preset penalty value is assigned to the corresponding position in the correction matrix; otherwise, it is 0. Subtract the correction matrix from the basic affinity matrix and perform nonnegation to obtain the corrected affinity matrix; The corrected affinity matrix is ​​input into the spectral clustering module to generate cluster labels. A contrastive loss function is constructed based on the cluster labels, and the role encoding network is iteratively trained.

5. The multi-UAV cooperative adversarial method based on role representation and mutual information constraints according to claim 3, characterized in that: The mutual information estimator includes a lower bound estimator and an upper bound estimator; the lower bound estimator is configured with a dominant experience buffer, and the upper bound estimator is configured with a suboptimal experience buffer.

6. The multi-UAV cooperative adversarial method based on role representation and mutual information constraints according to claim 5, characterized in that: The lower bound estimator includes a state feature branch and a role feature branch, which are used to map the global state and joint role representation to the same high-dimensional feature space. The joint statistics are calculated based on the Fenchel duality principle to estimate the lower bound of mutual information. The lower bound estimator updates the parameters by minimizing the first loss function, which is constructed based on the dominant sample pairs sampled in the dominant experience buffer and the unpaired samples in shuffled order. The upper bound estimator is constructed as a conditional probability density prediction network comprising a mean prediction subnetwork and a variance prediction subnetwork. The mean prediction subnetwork and the variance prediction subnetwork output the predicted mean and predicted log-variance respectively based on the input global state to construct a Gaussian distribution that approximates the true conditional distribution, thereby estimating the upper bound of mutual information. The upper bound estimator updates the parameters by minimizing the gradient of a second loss function, which is the negative log-expectation of the conditional probability density function predicted by the network.

7. A multi-UAV cooperative adversarial method based on role representation and mutual information constraints according to claim 6, characterized in that: The intrinsic reward and punishment mechanism is to use the trained mutual information estimator to calculate the lower bound of mutual information as a positive incentive, and use the trained mutual information estimator to calculate the upper bound of mutual information as a negative penalty. The original reward from the environmental feedback is added to the positive incentive and the negative penalty is subtracted to obtain the corrected reward for policy updating.

8. The multi-UAV cooperative adversarial method based on role representation and mutual information constraints according to claim 1, characterized in that: The hybrid evaluation network introduces an attention mechanism to generate weights for evaluating the value of joint actions. The generated weights satisfy the following conditions: ， in, The global state, after linear transformation, serves as the query vector. This is a representation of the roles of all drones after transformation, serving as key vectors. The feature dimension is used for nonlinear weighted fusion of the local Q-values ​​of each UAV.

9. A multi-UAV cooperative adversarial method based on role representation and mutual information constraints according to claim 8, characterized in that: The parameters of the policy network are updated by backpropagating the gradient using the corrected total reward signal and the hybrid evaluation network; During forward computation, the local Q-values ​​of each UAV are fused into a global joint action value using the hybrid evaluation network. During backward update, the temporal difference objective is calculated using the modified reward, and a mean squared error loss function is constructed. The gradient is calculated by minimizing this loss function, and the gradient is backpropagated to the policy network of each UAV via the hybrid evaluation network and the weights generated according to the multi-head attention mechanism.

10. A multi-UAV cooperative combat system based on role representation and mutual information constraints, characterized in that: include: At least one processor; and A memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by the processor to implement the multi-UAV cooperative adversarial method based on role representation and mutual information constraints as described in any one of claims 1 to 9.

Images