A method for cooperative confrontation of a UAV cluster based on graph attention reinforcement learning

Abstract

This invention relates to a method for cooperative combat among UAV swarms based on graph attention reinforcement learning, which enhances the cooperative combat capability among UAVs and has achieved experimental success in a simulated environment of UAV swarm cooperative combat: (1) A module for processing observation data of UAV swarms was designed to obtain training data of graph attention network; (2) Graph attention network was used to learn the actions that individuals in the UAV swarm should take in cooperative combat tasks; (3) Cooperative combat experiments were conducted with another UAV swarm in a simulation environment, and the performance was compared with other UAV swarm cooperative combat methods based on reinforcement learning.

Description

Technical Field

[0001] This invention relates to a cooperative adversarial method for drone swarms based on graph attention reinforcement learning, which belongs to the interdisciplinary field of aerospace and artificial intelligence. Background Technology

[0002] A multi-agent system refers to a system in which multiple agents exist in the same environment and can communicate with each other. Multi-agent systems often require these agents to collaborate on a specific task through communication. They have found wide application in solving complex real-world problems, including autonomous driving of multiple vehicles, distributed control of multiple robots, and cooperative combat using drone swarms. Specifically, cooperative combat using drone swarms involves two drone swarms existing in the same environment, where multiple drones within each swarm sense and coordinate with each other to jointly combat the other drone swarm.

[0003] Multi-agent reinforcement learning primarily studies how multi-agent systems, through acquiring environmental observation information, adopt certain strategies to take actions, interact with the environment through trial and error, and ultimately collaboratively complete tasks within the environment. In UAV swarm cooperative combat missions, each UAV is an agent. Applying multi-agent reinforcement learning to complete UAV swarm cooperative combat missions is currently a hot interdisciplinary research topic.

[0004] In cooperative adversarial tasks involving drone swarms, traditional CommNet multi-agent reinforcement learning algorithms attempt to share the hidden states of the policy, while PS-TRPO multi-agent reinforcement learning algorithms attempt to share the policy parameters. However, this sharing is not an effective way for agents to share information, and therefore cannot effectively guide reasonable cooperation among multiple agents. Some multi-agent reinforcement learning algorithms using value propagation link the updates of the value function and the policy function in reinforcement learning, but these methods struggle to adapt to dynamic changes among multiple agents.

[0005] Graph structures have great potential for sharing mutual information and adapting to dynamic changes: each agent is treated as a node in the graph, and edges are dynamically connected based on the distance between agents. The resulting graph can utilize the message propagation and message aggregation mechanisms of graph neural networks, enabling each agent to collect information from more distant agents, while direct communication of information only occurs between nearby agents.

[0006] Some existing technologies attempt to apply graph convolution-based multi-agent reinforcement learning algorithms to accomplish UAV swarm cooperative combat missions. This technology models the UAV swarm using a graph structure, treating messages from neighboring nodes as equally important and aggregating them with the same weight during message propagation on the graph. However, in UAV swarm cooperative combat missions, the information and importance of each UAV are often related to various factors such as the distance between UAVs, their current state, the rewards they have already obtained in the simulation environment, and the duration of interaction with the simulation environment. Existing technologies cannot achieve reasonable information propagation between UAVs, thus affecting the combat capability of UAV swarm cooperative combat.

[0007] Therefore, this invention designs a UAV swarm cooperative adversarial method based on graph attention reinforcement learning. It effectively models the information sharing mode among UAVs in the UAV swarm through graph structure, and reasonably adjusts the information and importance of each UAV in the UAV swarm through attention mechanism. Finally, it achieves good performance in the simulation experiment of UAV swarm cooperative adversarial task. Summary of the Invention

[0008] The technical problem solved by this invention is to overcome the shortcomings of existing technologies and provide a UAV swarm cooperative adversarial method based on graph attention reinforcement learning. By using graph structure to effectively model the information sharing mode among UAVs in the UAV swarm, and by using attention mechanism to reasonably adjust the information and importance of each UAV in the UAV swarm, good results are finally achieved in the simulation environment of UAV swarm cooperative adversarial task.

[0009] Technical Solution of the Invention: The present invention provides a method for cooperative adversarial warfare among drone swarms based on graph attention reinforcement learning, comprising the following steps:

[0010] Step 1: Repeat the cooperative combat simulation experiment in the simulated environment of UAV swarm cooperative combat, collect and save the data of each time stamp in each round of the experiment, including: the observation data of each UAV in the UAV swarm, the actions taken and the rewards obtained;

[0011] Step 2: Using the designed observation data processing module, feature extraction is performed on the observation data of each UAV in the UAV cluster collected in Step 1 to obtain the feature vector of each UAV; based on the coordinate position of each UAV in the environment in the observation data, each UAV is used as a node to construct a graph structure, and the feature vector of each UAV is used as the node feature on the graph structure, together with the graph structure as training data.

[0012] Step 3: Design a graph attention multi-agent reinforcement learning algorithm. Use the training data from Step 2 to train the graph attention network, learn the actions that each drone in the drone swarm should take at each time point, and the learning objective is for the drone swarm to obtain more rewards after each round of experiments, and finally obtain a fully trained graph attention network.

[0013] Step 4: Conduct experiments with another drone swarm in a simulated environment of drone swarm cooperative combat. Apply the graph attention network trained in Step 3. The graph attention network outputs the action that each drone in the drone swarm should take at each time point. Then the drone swarm executes the action and finally completes the cooperative combat mission.

[0014] To further optimize the above technical solution, the technical measures adopted by the present invention also include:

[0015] Furthermore, in step 3, the graph attention multi-agent reinforcement learning algorithm is specifically implemented as follows:

[0016] (1) A graph neural network is constructed by using one fully connected layer and one graph attention convolutional layer. The input dimension of each node of the graph neural network is consistent with the data dimension output by the observation data processing module, and the output dimension of each node is consistent with the number of actions that the UAV represented by the node can perform.

[0017] (2) Make a copy of the graph neural network, and keep the initial weights of the two graph neural networks consistent. The two graph neural networks approximate the action policy and evaluation policy in multi-agent reinforcement learning, respectively, and are called the current network and the target network.

[0018] (3) Input the data output by the observation data processing module into the current network, select and execute the action using the greedy algorithm, collect the feedback from the simulation environment to the UAV cluster, repeat until the experiment in the simulation environment ends, and store the collected data as an experience in the experience replay pool, i.e., a queue.

[0019] (4) Extract experience from the experience replay pool and input it into the current network and the target network respectively. Calculate the TD deviation of the output of the two networks. Use this deviation to backpropagate and train the current network. During this process, the parameters of the target network cannot be trained.

[0020] (5) Train the current network continuously according to steps (3) to (4), and copy the parameters in the current network to the target network to update its parameters every certain period until the TD deviation is reduced to the set threshold to terminate training.

[0021] Furthermore, in step 2, the designed observation data processing module specifically comprises:

[0022] (1) Input the original observation data of each UAV in the simulation environment, and the observation data processing module extracts the UAV's own state data and environmental observation data from it.

[0023] (2) Extract the position coordinates of the UAV in the simulation environment from its own state data. Using each UAV as a node, select the four nearest UAVs in the UAV cluster as adjacent UAVs according to their coordinates and establish edge connections to construct a topology graph;

[0024] (3) The environmental observation data and the remaining self-state data constitute the feature tensor of the UAV. A convolutional layer and a linear layer are used to extract features to obtain the environmental observation feature vector and the self-state feature vector.

[0025] (4) The environmental observation feature vector and the self-state feature vector are concatenated to obtain the UAV feature vector. The feature vector of each UAV is used as the feature of the graph node in the topology graph. Together with the topology graph, they form the training data. The observation data processing module outputs the training data to the graph neural network.

[0026] In a second aspect, the present invention provides an electronic device (computer, server, smartphone, etc.), characterized in that it includes a processor and a memory;

[0027] Memory, used to store computer programs;

[0028] The processor is used to execute the computer program stored in the memory, and when executing it, it implements the aforementioned method for cooperative adversarial tactics of drone swarms based on graph attention reinforcement learning.

[0029] Thirdly, the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) storing a computer program thereon, wherein the computer program, when executed by a processor, implements the aforementioned method for cooperative adversarial tactics of drone swarms based on graph attention reinforcement learning.

[0030] The advantages or contributions of this invention compared to the prior art are as follows:

[0031] (1) This invention designs an observation data processing module, which is responsible for converting the observation data of each UAV in the simulated environment of UAV swarm cooperative combat into the data required for training the network. Due to the introduction of trainable convolutional and linear layers, the input data of the graph neural network is enhanced, and the graph neural network can learn more abstract data features, thereby enabling UAVs to effectively utilize the observation data for cooperative combat.

[0032] (2) This invention proposes a UAV swarm cooperative adversarial method based on graph attention reinforcement learning. It effectively models the information sharing mode among UAVs in the UAV swarm through graph structure, and reasonably adjusts the information and importance of each UAV in the UAV swarm through attention mechanism. Finally, it achieves good experimental results in the simulation environment of UAV swarm cooperative adversarial task. Attached Figure Description

[0033] Figure 1 This is an illustration of the UAV swarm cooperative adversarial method based on graph attention reinforcement learning according to the present invention;

[0034] Figure 2 This is a diagram showing the internal structure of the observation data processing module in this invention.

[0035] Figure 3 This is a diagram of the graph attention network structure used in this invention;

[0036] Figure 4 The figure shows the experimental results in a simulated environment of cooperative combat by a swarm of drones in this invention. Detailed Implementation

[0037] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0038] like Figure 1 As shown, the UAV swarm cooperative adversarial method based on graph attention reinforcement learning of the present invention includes the following four steps:

[0039] Step 1: Repeat the cooperative combat simulation experiment continuously in the simulated environment of drone swarm cooperative combat, collecting and saving the data of each timestamp in each round of the experiment. Figure 1 The experience replay pool includes: observation data, actions taken, and rewards obtained by each drone in the drone swarm.

[0040] The environment consists of two drone swarms, red and blue. Each drone has 10 health points and takes 2 health points in damage each time it is attacked. Each drone acquires observation data as a 13×13×41 tensor, containing 13×13×7 environmental observation data and 13×13×34 self-state data. The first and second dimensions of the self-state data are redundant, meaning the actual valid data is repeated 13×13 times. This invention only uses the 34-dimensional data as the self-state data. The simulated environment for cooperative drone swarm combat assigns specific rewards to each drone's actions: if a drone attacks a drone in another drone swarm, it receives a 5-point reward from the simulation environment; due to ammunition consumption, the drone also receives a -0.01-point reward to represent the loss. If it defeats a drone in another drone swarm, it receives a 5-point reward from the environment; if it is defeated, it receives a -2-point reward from the environment.

[0041] Step 2: Process the observation data of each UAV in the UAV swarm using the designed observation data processing module. For each round of the UAV swarm cooperative combat simulation experiment, at the t-th experimental timestamp, the raw observation data acquired by the UAV swarm is denoted as... in is the raw observation data of the i-th drone, and N is the number of drones in the cluster.

[0042] like Figure 2 As shown, the observation data processing module obtains the UAV's own state data from the observation data, denoted as... And environmental observation data, denoted as

[0043] The state data of the drone cluster at the t-th experimental timestamp;

[0044] This refers to the environmental observation data of the UAV cluster at the t-th experimental timestamp.

[0045] in, This represents the state data of the i-th drone in the cluster. This represents the environmental observation data of the i-th drone in the cluster, where N is the number of drones in the cluster.

[0046] The drone's own status data includes information such as its ID, health points, and position coordinates in the simulated experimental environment; the environmental observation data includes information such as whether there are obstacles in the simulated experimental environment, whether there are drones in another drone cluster, and the health points of drones in another drone cluster.

[0047] The position coordinates of the drone in the simulated environment are extracted from its own state data. Using each drone as a node, the four nearest drones in the drone cluster are selected as adjacent drones based on their coordinates, and edges are established to connect them, constructing a topology graph, denoted as [node name missing]. Environmental observation data and the remaining self-state data constitute the UAV's feature tensor. Feature extraction is performed using one convolutional layer and one linear layer to obtain the environmental observation feature vector and the self-state feature vector, respectively. These two vectors are then concatenated to obtain the feature vector of the UAV swarm, denoted as [missing information]. in This represents the feature vector of the i-th drone in the cluster. It is an abstract high-level feature extracted from the raw observation data of the drone, which facilitates subsequent learning by the graph neural network. The feature vectors of each drone serve as features of nodes in the graph, and together with the graph structure, they form the training data.

[0048] Step 3: As Figure 3 As shown, a graph attention multi-agent reinforcement learning algorithm is designed. It uses a graph attention network to learn the actions that each drone in the drone swarm should take at each time point from the training data. The learning objective is to enable the drone swarm to obtain more rewards after each round of experiments.

[0049] Figure 3 middle, Right now At the t-th experimental timestamp, the observation data of the i-th UAV is sent to the observation data processing module. Figure 3 The symbol 't' is omitted for simplification;

[0050] Right now At the t-th experimental timestamp, the feature vector of the i-th UAV is output by the observation data processing module. Figure 3 The symbol 't' is omitted for simplification;

[0051] Right now At the t-th experimental timestamp, the i-th node on the graph represents the i-th drone. The output feature after processing by the graph neural network is the probability distribution of the actions that the i-th drone can perform.

[0052] Right now At the t-th experimental timestamp, the i-th drone... The action selected using an ε-greedy strategy is that the action is chosen with a probability of 1-ε. The action with the highest probability is selected, and other actions are randomly chosen with probability ε. Figure 3 The symbol 't' is omitted for simplification, and the action selected by the drone swarm is denoted as .

[0053] The specific implementation method is as follows:

[0054] ① Figure 3 The graph attention network shown is duplicated, with the initial weights remaining consistent. The two graph attention networks respectively approximate the action policy and evaluation policy in multi-agent reinforcement learning, denoted as Q of the current network. local and target network Q target .

[0055] ② The features output by the observation data processing module Input the current network, current network Q local The output is the score of each action selected by each drone. Each drone selects and executes actions using a greedy strategy. The actions selected by the drone swarm are... Subsequently, the total reward fed back to the drone swarm from the simulation environment is in is the reward obtained by agent i at the t-th experimental time stamp, and N is the number of drones in the cluster. At this point, the simulation environment moves to the next time stamp t+1.

[0056] ③ By repeating steps ① to ②, the graph attention multi-agent reinforcement learning algorithm designed in this invention repeatedly applies... This experience is stored as an experience in the experience replay pool, where and These are the raw observations and the constructed topology graph acquired by the drone swarm at timestamp t+1. During this period, the graph attention network was not trained.

[0057] ④ Then, if enough experience has been collected and stored in the drone swarm cooperative combat simulation environment, training the graph attention network can begin:

[0058] A. Extracting past data from the experience replay pool A total of 1024 items;

[0059] B. Calculation Where θ is the current network Q local The parameters in Actions selected for drone swarms The vector composed of each action;

[0060] C. Calculation:

[0061]

[0062] Where γ is the discount factor, which is taken as 0.99 in this invention, and θ′ is the target network Q. target Parameters in;

[0063] D. Calculate the TD deviation:

[0064]

[0065] E. Treat the TD bias as a loss function and backpropagate the gradient to optimize the parameters θ in the current network.

[0066] Repeat steps A through E above to train the graph attention network. The training objective is to make the output of the current network as close as possible to the output of the target network, reducing the TD bias. During training, the parameters θ′ of the target network are not updated. Every certain period, the parameters θ′ in the current network are copied to the target network to update θ′. In this invention, the period is one copy and update of the target network parameters θ′ after each round of experiments.

[0067] The computation of graph attention networks follows this process:

[0068] Record time t graph G t =(V t E t ), where V t Is it Figure G? t The set of all nodes, E t Is it Figure G? t The set of all edges above.

[0069] For each (i, j) ∈ E t , let e ij The correlation score between node i and node j is represented as follows:

[0070]

[0071] in, These are the data for node i and node j, respectively, R c This refers to a c-dimensional real vector space, where c = 2^33, and is called... is the feature vector of the node.

[0072] The inner product between features is used as a similarity measure. To adaptively learn better representations between nodes, the node features are first linearly transformed, and then the inner product of the transformed feature vectors is taken to calculate the relevance score.

[0073]

[0074] Among them, W q and W k It is a linear transformation matrix, and the elements in the matrix are the parameters to be trained.

[0075] Use the Softmax function to Regularization:

[0076]

[0077] in, The importance of the information transmitted by UAV j to UAV i at time t was measured, so that UAV i can take into account the status of its neighboring UAVs based on this importance and take actions that are more conducive to coordinating against another group of UAVs.

[0078] The aggregate representation of compute node i:

[0079]

[0080] Among them, W v It is a linear transformation matrix. It is attention weight.

[0081] Finally, the aggregated features and node features are combined. The fusion yields a more powerful feature representation based on target information:

[0082]

[0083] Here, || represents the vector concatenation operation (Concat).

[0084] For any node i∈V t calculate The final graph convolution operation. The length of the feature vector of the output node is the number of actions the agent can take, and each value represents the value function of the action taken in that state at time t.

[0085] Step 4: Conduct experiments with another drone swarm in a simulated drone swarm cooperative adversarial environment. Apply the graph attention network trained in Step 3, where each drone in the swarm takes action based on the network output at each timestamp to complete the drone swarm cooperative adversarial task. Compare the performance with other drone swarm cooperative adversarial methods based on multi-agent reinforcement learning in the simulated environment. The experiments show that in the drone swarm simulation environment, the graph attention multi-agent reinforcement learning algorithm of this invention achieves a higher average reward than the graph convolution-based multi-agent reinforcement learning algorithm GCN and the shared policy hidden state multi-agent reinforcement learning algorithm CommNet, demonstrating the effectiveness of the proposed graph attention reinforcement learning-based drone swarm cooperative adversarial method.

[0086] Figure 4The horizontal axis represents the number of experimental rounds in the simulation environment, and the vertical axis represents the mean reward (total reward divided by the number of drones in the cluster) obtained by our drone swarm in a certain round of the experiment. Figure 4 This indicates that in the initial experimental rounds (before round 50), the graph neural network was not trained, and the data was simply stored in the experience replay pool; subsequently, the graph attention multi-agent reinforcement learning algorithm began to train the graph attention network, and the average reward increased significantly; after the network was sufficiently trained, the average reward gradually stabilized.

[0087] Experiments show that the UAV swarm cooperative adversarial method based on graph attention reinforcement learning proposed in this invention guides the UAV swarm to achieve the best stable average return in the simulated experimental environment. This indicates that the observation data processing module proposed in this invention, combined with the graph attention multi-agent reinforcement learning algorithm, can effectively promote UAV swarm cooperative adversarial activities.

[0088] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (computer, server, smartphone, etc.) including a memory and a processor, wherein the memory stores a computer program configured to be executed by the processor, and the computer program includes instructions for performing the steps of the method of the present invention.

[0089] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk), which stores a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0090] It should be emphasized that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A method for cooperative adversarial tactics in drone swarms based on graph attention reinforcement learning, characterized in that, Includes the following steps: Step 1: Repeat the cooperative combat simulation experiment in the simulated environment of UAV swarm cooperative combat, collect and save the data of each time stamp in each round of the experiment, including: the observation data of each UAV in the UAV swarm, the actions taken and the rewards obtained; Step 2: Using the designed observation data processing module, feature extraction is performed on the observation data of each UAV in the UAV cluster collected in Step 1 to obtain the feature vector of each UAV; based on the coordinate position of each UAV in the environment in the observation data, each UAV is used as a node to construct a graph structure, and the feature vector of each UAV is used as the node feature on the graph structure, together with the graph structure as training data. Step 3: Design a graph attention multi-agent reinforcement learning algorithm. Use the training data from Step 2 to train the graph attention network, learn the actions that each drone in the drone swarm should take at each time point, and the learning objective is for the drone swarm to obtain more rewards after each round of experiments, and finally obtain a fully trained graph attention network. Step 4: Conduct an experiment with another drone swarm in a simulated environment of drone swarm cooperative combat. Apply the graph attention network trained in Step 3. The graph attention network outputs the action that each drone in the drone swarm should take at each time point. Then the drone swarm executes the action and finally completes the cooperative combat mission. In step 3, the graph attention multi-agent reinforcement learning algorithm is specifically implemented as follows: (1) A graph neural network is constructed by using one fully connected layer and one graph attention convolutional layer. The input dimension of each node of the graph neural network is consistent with the data dimension output by the observation data processing module, and the output dimension of each node is consistent with the number of actions that the UAV represented by the node can perform. (2) Make a copy of the graph neural network, and keep the initial weights of the two graph neural networks consistent. The two graph neural networks approximate the action policy and evaluation policy in multi-agent reinforcement learning, respectively, and are called the current network and the target network. (3) Input the data output from the observation data processing module into the current network, and use... The greedy algorithm selects and executes actions, collects feedback from the simulation environment to the drone swarm, and repeats this process until the experiment ends in the simulation environment. The collected data is stored as an experience in the experience replay pool, i.e., a queue. (4) Extract experience from the experience replay pool and input it into the current network and the target network respectively. Calculate the TD deviation of the output of the two networks. Use the deviation to backpropagate and train the current network. During this process, the parameters of the target network cannot be trained. (5) Train the current network continuously according to steps (3) to (4), and copy the parameters in the current network to the target network to update its parameters every certain period until the TD deviation is reduced to the set threshold to terminate training.

2. The UAV swarm cooperative adversarial method based on graph attention reinforcement learning according to claim 1, characterized in that: In step 2, the designed observation data processing module is specifically as follows: (1) Input the original observation data of each UAV in the simulation environment, and the observation data processing module extracts the UAV's own state data and environmental observation data from it; (2) Extract the position coordinates of the UAV in the simulation environment from its own state data; take each UAV as a node, select the four nearest UAVs in the UAV cluster as adjacent UAVs according to the coordinates and establish the edge connection to construct the topology graph; (3) The environmental observation data and the remaining self-state data constitute the feature tensor of the UAV. The features are extracted by a convolutional layer and a linear layer respectively, and the environmental observation feature vector and the self-state feature vector are obtained. (4) The environmental observation feature vector and the self-state feature vector are concatenated to obtain the UAV feature vector. The feature vector of each UAV is used as the feature of the graph node in the topology graph. Together with the topology graph, they form the training data. The observation data processing module outputs the training data to the graph neural network.

3. An electronic device, characterized in that, Including processor and memory; Memory, used to store computer programs; A processor for executing a computer program stored in memory, which, when executed, implements the method described in any one of claims 1-2.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-2.

Images