A multi-unmanned aerial vehicle coordinated control method based on a dynamic directed graph communication structure
By combining a dynamic directed graph communication structure and a graph collapse network, the communication scalability and decision-making accuracy issues of multi-UAV systems when the number of UAVs changes are solved, achieving efficient information aggregation and global state generation, and improving the robustness and real-time performance of the system.
Patent Information
- Application Number
- CN202510136886.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-07
- Publication Date
- 2025-11-18
- Estimated Expiration
- 2045-02-07
AI Technical Summary
Existing multi-drone reinforcement learning methods lack scalability when the number of drones changes, have high communication costs, information redundancy, and rely on the global state, leading to non-stationarity and decision-making errors.
A multi-UAV coordinated control method based on a dynamic directed graph communication structure is adopted. Through a multi-key gated communication network and a graph collapse network, the communication structure between UAVs is dynamically adjusted. Combined with a Transformer decoder and a QMIX hybrid network, information aggregation and global state generation are achieved, reducing communication dependencies and improving decision-making accuracy.
It improves the scalability and information transmission efficiency of multi-UAV systems, reduces communication overhead, enhances the collaborative perception capability and decision-making accuracy of UAV swarms, solves the communication scalability problem when the number of UAVs changes, reduces dependence on global state, and improves the robustness and real-time performance of the system.
Smart Images
Figure CN120103868B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of multi-UAV cooperative control technology, specifically relating to a multi-UAV cooperative control method based on a dynamic directed graph communication structure. Background Technology
[0002] In recent years, cooperative multi-UAV reinforcement learning (MARL) technology has emerged as an important framework for solving complex cooperative tasks in real-world scenarios, attracting significant attention and demonstrating great potential in practical applications and commercial applications. Centralized cooperative MARL methods treat the multi-UAV system as a whole and then use individual UAV methods to learn the overall policy. However, this approach faces limitations in scalability and the need for a central controller. Distributed cooperative MARL methods treat UAVs as individual entities and use individual UAV methods to learn their individual policies. While this addresses the limitations of a central controller, it leads to non-stationarity and credit assignment problems. To address these issues, existing multi-UAV reinforcement learning algorithms primarily employ a centralized training with decentralized execution (CTDE) paradigm. This paradigm relies on access to the global state, but access to the global state is often an idealized assumption, and a central trainer typically does not exist in the real world. In contrast, researching communication between UAVs is more practically meaningful.
[0003] In multi-UAV systems, communication is particularly crucial for information sharing, learning, and collaborative achievement of common goals, especially in partially observable environments. Effective communication is fundamental to cooperation for complex tasks such as coordinating autonomous vehicles, sensor networks, and multi-robotic systems. Inspired by human collaboration, researchers have integrated communication into multi-UAV reinforcement learning to enhance information sharing among UAVs. Early research primarily relied on broadcasting information, but this approach resulted in high communication costs and information redundancy. Later research aimed to reduce communication overhead by selectively determining communication timing and eliminating redundant information through directed peer-to-peer communication. However, existing methods typically model communication individually for each UAV, lacking scalability as the number of UAVs changes. Currently, a method is still lacking that addresses five key communication challenges: how to identify communication targets, communication timing, communication content, how to integrate received information, and how to leverage communication to avoid dependence on a global state. Summary of the Invention
[0004] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a multi-UAV coordinated control method based on a dynamic directed graph communication structure.
[0005] The objective of this invention can be achieved through the following technical solutions:
[0006] This invention provides a multi-UAV coordinated control method based on a dynamic directed graph communication structure, characterized by comprising the following steps:
[0007] The drone acquires local observations of the environment and obtains historical hidden states from other drones through communication.
[0008] In the distributed execution phase, based on the initial topology of the UAVs, an adjacency trajectory matrix is obtained to represent the communication relationship between UAVs at the current moment. Pre-communication information is calculated based on local observations and historical hidden states. The current adjacency trajectory matrix and pre-communication information are input into a multi-key gated communication network to calculate the adjacency trajectory matrix at the next moment. Each UAV calculates its local action value function based on the pre-communication information and the adjacency trajectory matrix at the next moment through a Transformer-based decoder and updates its hidden state. Based on the local action value function and local observations, the final control decision is generated and executed through a graph collapse network and a hybrid network, and then the next loop begins.
[0009] During the centralized training phase, the adjacency trajectory matrix at each moment of the distributed execution phase is recorded. The local observations and adjacency trajectory matrix at each moment are input into the graph collapse network to generate the global state at each moment. Based on the global state at each moment and the local action value function of each UAV at each moment, the global action value function at each moment is generated. Based on the global action value function, the UAV swarm interacts with the environment and obtains rewards. Based on the rewards, the parameters of the hybrid network are updated through the first loss function.
[0010] After centralized training is completed, the trained graph collapse network and hybrid network are deployed on various drones.
[0011] Furthermore, the adjacency trajectory matrix represents the communication structure of multiple UAVs. The adjacency trajectory matrix is a Boolean matrix, and the values of the matrix elements indicate whether there is communication between the UAVs.
[0012] Furthermore, the calculation of pre-communication information based on local observations and historical hidden states specifically includes:
[0013] Local observations and historical hidden states are converted into pre-communication information C0 through an encoding network MLP:
[0014]
[0015] Wherein, C0 represents pre-communication information. These are local observations. It is in a hidden historical state.
[0016] Furthermore, the step of inputting the current adjacency trajectory matrix and pre-communication information to a multi-key gated communication network, and calculating the adjacency trajectory matrix at the next time step, specifically includes:
[0017] The current adjacency trajectory matrix A t The pre-communication information C0 is input into the multi-key access control communication network for processing. The calculation formula for the multi-key access control communication network is as follows:
[0018]
[0019] Where K0 is the output of the key, representing the updated adjacency trajectory matrix in this communication; Gumbel-softmax is a discrete sampling technique used to sample from discrete communication structures to determine whether each pair of UAVs communicates; w v Linear layer weights for the output dimension, used to generate the final communication decision; tanh is the activation function; C0 is the pre-communication information; W... q For the weight of the query, W k The weight of the key;
[0020] After processing through the multi-key gate communication network, a total of i keys are obtained at time t, denoted as k. t|i The adjacency trajectory matrix at the next time step is represented as:
[0021]
[0022] in, Indicates rounding up, A t+1 This is the adjacent trajectory matrix for the next time step.
[0023] Furthermore, each UAV calculates its local action value function based on pre-communication information and the adjacency trajectory matrix at the next time step using a Transformer-based decoder, and updates its hidden state, specifically including:
[0024] The pre-communication information is repeatedly processed by a multilayer perceptron (MLP) and then input along with the adjacency trajectory matrix for the next time step into a Transformer-based decoder. The Transformer-based decoder includes a self-attention mechanism, a position feedforward network, residual connections, and a normalization layer. The input information undergoes multi-head attention, residual connections and normalization, point-by-point feedforward neural network, and further residual connections and normalization. The processed information is then added to the pre-communication information and passed through a fully connected layer to obtain the updated hidden state of UAV i. With local action value function Q i (s,a).
[0025] Furthermore, the graph collapse network includes a graph convolutional network, a self-attention pooling network, and a readout mechanism.
[0026] Furthermore, the global state acquisition process includes:
[0027] Local observations from each UAV As the feature input for each node in the graph convolutional network, feature extraction and node representation learning are performed, using the following formula:
[0028]
[0029] in, These are the local observations of UAV i at time t. It is a representation of a node. Let be the value of the i-th node in the adjacency trajectory matrix at the next time step. Let σ represent the degree matrix of the i-th UAV at time t+1, and σ be the nonlinear activation function.
[0030] Represent the learned nodes The input is fed into a self-attention pooling network. This network selects important nodes in the graph through feature weighting and node importance evaluation, and then weights the features of these important nodes, outputting a feature-weighted node graph. The formula is as follows:
[0031]
[0032] in, Let i be the weighted feature node representation of node i, concat be the feature fusion method, tanh be the activation function, and ⊙ be the element-wise multiplication.
[0033] The weighted feature node representations of each node are aggregated through a reading mechanism to generate the global state s. t The formula is:
[0034]
[0035] Where N is the number of nodes, max is the aggregation function, and i and n are both positive integers.
[0036] Furthermore, the generation of the global action value function at each moment based on the global state at each moment and the local action value function of each UAV at each moment specifically includes:
[0037] The global state s t With the local motion value function Q of each UAV i (s,a) are input into the hybrid network, and the drone swarm learns from the hybrid network Q. tot(τ,a,Am,s;θ), using the QMIX network, the weights of each local action value function are determined using the global state, and the local action value functions of each UAV are weighted and synthesized into a global action value function Q. tot (s,a).
[0038] Furthermore, the first loss function is:
[0039]
[0040] in, Let b be the first loss function, and b be the batch size, representing the number of samples sampled from the replay buffer. Let Q be the target Q value of drone i. tot (τ t ,α t ,m t ,s t ;θ) is the Q-value of the hybrid network at time t, τ t Let α be the historical observation value at time t. t Let m be the action at time t. t For the information of the drone swarm at time t, s t Let be the global state at time t, θ be the parameters of the hybrid network, r be the reward obtained by the drone swarm through interaction with the environment, and γ be the discount factor. This represents the action a that maximizes the value. t+1 Q tot value.
[0041] Furthermore, the distributed execution phase and the centralized training phase are bridged through an adjacency trajectory matrix.
[0042] Compared with the prior art, the present invention has the following advantages:
[0043] (1) A novel communication and collaboration multi-UAV reinforcement learning paradigm was implemented, which solved the scalability problem when the number of UAVs changes: For multi-UAV systems, a graph collapse network is used to achieve bridging and synchronous communication between the training and execution phases. The multi-UAV system is modeled as a dynamic directed graph, which effectively represents the communication structure of the system at any time, avoiding the problem of traditional methods that require modeling each UAV separately, and improving the scalability of the method.
[0044] (2) A global state information aggregation was achieved using the topological structure of a dynamic directed graph, reducing the system's dependence on the global state: For multi-UAV systems, a multi-key gated communication network was designed. By learning the structure of a dynamic directed graph during the distributed execution phase and combining it with a Transformer-based decoder, communication and feature extraction were realized. The dynamic directed graph structure learned by the multi-key gated communication network was input into the graph collapse network during the centralized training phase for information aggregation, generating approximate global state information.
[0045] (3) This invention dynamically adjusts the adjacency trajectory matrix between UAVs through a multi-key gated communication network, enabling UAV swarms to flexibly adjust their communication structure in complex environments, thereby improving the efficiency and reliability of information transmission. Compared to communication methods with fixed topologies, this method can adapt to different task requirements and achieve more efficient collaborative control.
[0046] (4) The present invention uses a Transformer-based decoder to process the local observations and communication information of the UAV, making full use of the self-attention mechanism to enhance the information interaction capability, improving the decision-making accuracy of each UAV, and avoiding the decision-making error caused by insufficient local information in traditional methods.
[0047] (5) This invention extracts the overall features of a UAV swarm through graph convolutional networks and self-attention pooling networks, and aggregates the information of each UAV using a readout mechanism to generate an accurate global state. This method can effectively reduce noise interference, improve the expressive power of global information, and enhance the collaborative perception capability of the UAV swarm.
[0048] (6) This invention employs a QMIX hybrid network to adaptively adjust the weights of the local action value functions of each UAV based on the global state, thereby achieving the global optimal decision-making for the UAV swarm. Compared to the traditional independent Q-learning method, this method can effectively solve the non-stationarity problem in multi-agent reinforcement learning, improving convergence speed and control accuracy.
[0049] (7) During the training phase, the hybrid network parameters are optimized through centralized training, enabling the UAV swarm to learn the globally optimal strategy. During the execution phase, each UAV can make independent decisions with only local computation and limited communication, achieving efficient distributed execution. This method reduces communication overhead and improves the robustness and real-time performance of the system. Attached Figure Description
[0050] Figure 1 This is a schematic diagram of the overall modules in this invention;
[0051] Figure 2 This is a schematic diagram of a collapsed network module.
[0052] Figure 3 The diagram shows the system relationship between the collapsed network modules and methods.
[0053] Figure 4 This is a flowchart illustrating an example of a multi-UAV reinforcement learning method based on a dynamic directed graph communication structure. Detailed Implementation
[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0055] Example 1:
[0056] One aspect of the present invention provides a multi-UAV reinforcement learning method based on a dynamic directed graph communication structure, applied to multi-UAV collaboration. This method specifically includes the following steps:
[0057] Step S1: The UAV acquires local observations from the environment and obtains historical hidden states from other UAVs via communication. These local observations... This refers to the state information that each UAV obtains from the environment at the current time (time t). This information includes environmental features and its own state perceived by the UAV, such as position, speed, attitude, surrounding obstacles, target position, etc. Local observations are usually obtained directly from sensors (such as vision, radar, lidar, etc.) by each UAV during mission execution and only reflect the local environment around that UAV. The historical hidden state refers to the representation of the internal state of each UAV at a past time (time t-1). It includes the UAV's decision-making process, observed environmental information, actions performed, and communication history with other UAVs over a past period. The historical hidden state is used to save and transmit important memories of the UAV during mission execution for reference in future decisions. The historical hidden state encodes the UAV's observations of the environment (local observations) at past times and the decisions made based on these observations. The historical hidden state records the changes in the UAV's state over time, including dynamic information on how it changes during mission execution. The historical hidden state also includes interaction information between UAVs, communication information from other UAVs, and how to make coordinated decisions based on this information.
[0058] Step S2: In the distributed execution phase, based on the initial topology of the UAVs, the current adjacency trajectory matrix is obtained to represent the communication relationship between UAVs at the current moment. Pre-communication information is calculated based on local observations and historical hidden states. The current adjacency trajectory matrix and pre-communication information are input into the multi-key gated communication network to calculate the adjacency trajectory matrix at the next moment. Each UAV calculates its local action value function based on the pre-communication information and the adjacency trajectory matrix at the next moment through a Transformer-based decoder and updates its hidden state. Based on the local action value function and local observations, the final control decision is generated and executed through a graph collapse network and a hybrid network, and then the next loop begins.
[0059] Step S3: During the centralized training phase, the adjacency trajectory matrix at each moment of the distributed execution phase is recorded. The local observations and adjacency trajectory matrix at each moment are input into the graph collapse network to generate the global state at each moment. Based on the global state at each moment and the local action value function of each UAV at each moment, the global action value function at each moment is generated. Based on the global action value function, the UAV swarm interacts with the environment and obtains rewards. Based on the rewards, the parameters of the hybrid network are updated through the first loss function.
[0060] Step S4: After centralized training is completed, deploy the trained graph collapse network and hybrid network in each UAV.
[0061] As a preferred technical solution, the multi-UAV system is modeled as a dynamic directed graph. Furthermore, each UAV is considered a node in the graph, with an adjacency trajectory matrix A. t This represents the system's communication structure, specifically, A. t This is a Boolean matrix, where the values of the matrix elements indicate whether communication exists between the drones.
[0062] As a preferred technical solution, the Transformer-based graph collapse network consists of a graph collapse network and a Transformer-based multi-key gated communication network. Further, the graph collapse network includes a graph convolutional network, a self-attention pooling network, and a readout mechanism; the Transformer-based multi-key gated communication network includes an encoder based on a dynamic directed graph and a decoder based on Transformer, following an encoder-decoder architecture.
[0063] As a preferred technical solution, the local observations are provided by UAVs. Specifically, at each time step t, each UAV receives a local observation from the observation function.
[0064] As a preferred technical solution, the process of obtaining the adjacency trajectory matrix and fusion information specifically includes initializing and preprocessing the local observations and historical information; performing pre-communication by combining the adjacency trajectory matrix of the previous time step and the hidden information of the UAV in the previous time step and obtaining pre-communication information; and processing the pre-communication information and the initialized adjacency trajectory matrix using a multi-key gating communication network to obtain the adjacency trajectory matrix and fusion information of the next time step.
[0065] As a preferred technical solution, the Transformer-based decoder consists of four parts, specifically including a self-attention mechanism, a position feedforward network, residual connections, and a normalization layer.
[0066] As a preferred technical solution, the process of generating the local action value function and hidden information specifically includes inputting the adjacency trajectory matrix and fusion information output by the multi-key gating communication network into a Transformer-based decoder; and using the output of the Transformer-based decoder to obtain the local action value function and the hidden information of the next time step.
[0067] As a preferred technical solution, the process of obtaining the global action value function specifically includes: inputting the local action value function and the adjacency trajectory matrix into a graph collapse network; normalizing the adjacency trajectory matrix; then using the local observations as feature inputs for each node in the graph convolutional network to perform graph convolution iteration; further, performing self-attention pooling based on the graph convolution iteration results; generating an approximate global state function by performing aggregation operations on all nodes; and finally, outputting the global action value function through a hybrid network based on the approximate global state function.
[0068] As a preferred technical solution, there is a connection between the distributed execution phase and the centralized training phase, specifically, a bridge is formed by a dynamic directed graph.
[0069] Another aspect of the present invention provides a novel reading mechanism, specifically, performing an aggregation operation on all nodes to generate a globally collapsed representation of the graph, approximating the global state.
[0070] This invention designs a method for multi-UAV systems that simultaneously includes communication, information extraction, and decision-making functions, supports multi-UAV systems with dynamic adjustment of the number of UAVs, and provides support for the collaborative control of multi-UAV systems and enhances communication and decision-making among UAVs.
[0071] like Figure 1 As shown, the overall system model of this embodiment includes a multi-key gated communication network, a graph collapse network, and a hybrid network. The multi-key gated communication network and the graph collapse network are connected to the UAV.
[0072] The multi-key gated communication network is used to receive local observations from the UAV via the network's communication section. Historical Information Adjacent trajectory matrix A at the current time t During the distributed execution phase, the adjacency trajectory matrix A for the next time step is generated. t+1 Local action value function Q i and hidden information The local action value function and local observation values are input into the graph collapse network. The multi-key gated communication network is also used to bridge centralized training and distributed execution through the adjacency trajectory matrix of the next time step. The input information of the multi-key gated communication network includes local observation values, historical information and the adjacency trajectory matrix of the current time step. After initialization and preprocessing, the input information is used for pre-communication, and the multi-key gated communication network processes the pre-communication information C0 and the adjacency trajectory matrix A0. The communication frequency between UAVs is determined by the number of keys, and the output of each key represents the updated adjacency trajectory matrix in that communication. The specific calculation formula is shown in formula (1):
[0073]
[0074] Where K0 is the output of the key, A0 is the adjacency trajectory matrix, and Gumbel-softmax sampling is a technique for sampling from discrete distributions. v It is a linear layer used for two output dimensions (communication or no communication), where tanh is the activation function, C0 is the pre-communication information, and W... q For the weight of the query, W k Let k be the weight of the key. After passing through the multi-key gate communication network, a total of i keys are obtained at time t, denoted as k. t|i The adjacent trajectory matrix at the next moment is represented as shown in formula (2):
[0075]
[0076] in, Indicates rounding up, A t+1 The adjacency trajectory matrix at time t+1 is used. Information transmission and reception are accomplished by using the adjacency trajectory matrix updated in the next time step, combined with pre-communication information.
[0077] Graph collapse networks are used to process local observations from drones. Next-moment adjacent trajectory matrix A t+1 During the centralized training phase, all inputs are aggregated to generate a globally collapsed representation of a dynamic directed graph, which is approximately equal to the global state function s. t The input to the graph collapse network includes local observations from each UAV.t And the adjacent trajectory matrix A at the next time step t+1 The input adjacency trajectory matrix is normalized, and the local observations of the UAV are used as the input to the graph convolutional network. The result after three graph convolution iterations is input into the self-attention pooling network. The pooling result is aggregated through a read-out mechanism to generate the global state, and the global state is input into the hybrid network.
[0078] Hybrid networks are used to aggregate the local motion value functions Q of various UAVs. i This forms a global action value function Q. tot This is used to guide the overall strategy optimization of multi-UAV systems. The input to the hybrid network is the local action value function of each UAV and the global state function s output by the graph collapse network. t The output is a global action value function, which evaluates the overall value of a multi-UAV system taking a certain cooperative action in a given state. Hybrid networks can reflect the cooperative effects among UAVs and guide UAV policy updates.
[0079] like Figure 2 As shown, the graph collapse network module of the present invention includes a graph convolutional network, a self-attention pooling network, and a readout mechanism. The readout mechanism is connected to the hybrid network, and the graph convolutional network is connected to the multi-key gating communication network.
[0080] Graph convolutional networks use local observations from drones to... t As the feature input for each node in the graph convolutional network, feature extraction and node representation are performed, and the learned node representations are then input into the self-attention pooling network. The input of the convolutional network is local observation, and the output is the representation of all nodes in the graph. The specific network structure is shown in Equation (3):
[0081]
[0082] in, These are the local observations of UAV i at time t. It is a representation of a node. Let be the value of the i-th node in the adjacency trajectory matrix at the next time step. Let denot be the degree matrix of the i-th UAV at time t+1, and σ be the nonlinear activation function. Graph convolutional networks are mainly responsible for feature extraction and information propagation in the data processing of dynamic directed graphs.
[0083] The function of the self-attention pooling network is to select important nodes in the graph by weighting features and evaluating node importance, and to weight the features of important nodes. The input is the representation of all nodes in the graph, and the output is the feature-weighted nodes of the graph. The structure of the self-attention pooling network in this invention is shown in formula (4):
[0084]
[0085] in, These are the weighted feature nodes of the graph. This represents the nodes; concat is the feature fusion method; and tanh is the activation function. Let be the self-circular adjacency matrix of the i-th node at time t+1. This represents its degree matrix.
[0086] The reading mechanism aggregates all nodes to generate a globally collapsed representation of the graph and an approximate global state function. The input to the reading mechanism is all nodes in the graph, and the output is the globally collapsed representation of the graph. The reading mechanism uses the Mean aggregation function and the Max aggregation function to take the average and maximum values of the nodes, respectively. The Mean aggregation function smoothly integrates the information of all nodes, reducing the influence of noise. The Max aggregation function highlights the significant features of specific nodes, retaining the most important information in the graph. The expression for the reading mechanism is shown in formula (5):
[0087]
[0088] Among them, s t Let t be the global state of the system. is the weighted feature node of the graph, N is the number of nodes, max is the aggregation function, and i and n are both positive integers.
[0089] Specifically, the steps for obtaining fused information and the adjacency trajectory matrix for the next time step through an encoder based on a dynamic directed graph are as follows: Input information is obtained through the communication part of the network module, including local observations of the UAV, historical information, and the current adjacency trajectory matrix A. t The input information is embedded and repeatedly processed to transform it into a continuous vector; the processed data is then input into a multi-key gating unit, which controls the transmission of information through multiple keys; based on the output of the multi-key gating unit, the adjacency matrix A is updated through graph processing. t+1 The graph structure of the next time step is represented; the adjacency trajectory matrix A is... t+1 The information, after repeated processing, is used as fused information and, together with the information after embedding processing, is input into the Transformer-based decoder.
[0090] Specifically, based on the value decomposition algorithm, the relationship between the local action value function and the global action value function is shown in formula (6):
[0091]
[0092] Among them, Q tot(τ,a) is the global action value function, τ is the historical local observation value, a is the action, argmax represents finding the maximum point, and Q is the value of the action value. i Let i be the local motion value function of UAV i.
[0093] Specifically, the steps for generating local action value functions and hidden information through a Transformer-based decoder are as follows: fusion information is obtained from a dynamic directed graph-based encoder; the input information is processed through a multi-head attention mechanism, residual connections and normalization, a pointwise feedforward neural network, and residual connections and normalization; the processed information is added to the embedded information output by the encoder; hidden information is obtained through a fully connected layer; and the UAV learns Q... i A (τ,a,Am,s) network is used to obtain the local motion value function Q of the UAV. i (s,a).
[0094] Specifically, the steps for obtaining the global action value function through graph collapse networks and hybrid networks are as follows: input information is obtained through the communication part of the network module, including local observations and the adjacency matrix A for the next time step. t+1 With the local motion value function of the UAV; for the adjacency matrix A t+1 Normalization is performed, and the communication structure of the current time step node is obtained by combining the adjacency trajectory matrix and local observations. The local observations are input into a graph convolutional network. Three graph convolution iterations are performed. Self-attention pooling is performed based on the information after iteration. The pooled information is aggregated through a reading mechanism to generate an approximate global state of the system. The global state and local action value functions are input into a hybrid network, and the UAV swarm learns the hybrid network Q. tot (τ,a,Am,s;θ), using the QMIX network, the weights of the local action value functions are determined using the global state, and the local action value functions of each UAV are weighted and synthesized into a global action value function Q. tot (s,a).
[0095] Specifically, hybrid network Q tot The θ parameter is updated using the loss function shown in Equation (7):
[0096]
[0097] in, Let b be the first loss function, and b be the batch size sampled from the replay buffer. Let Q be the target Q value of drone i. tot (τ t ,α t ,m t ,s t ;θ) is the Q-value of the hybrid network at time t, τ t Let α be the historical observation value at time t.t Let m be the action at time t. t For the information of the drone swarm at time t, s t Let be the global state at time t, θ be the parameters of the hybrid network, r be the reward obtained by the UAV swarm interacting with the environment, and γ be the discount factor. This represents the action a that maximizes the value. t+1 Q tot value.
[0098] Specifically, in this invention, the information transmitted by the UAV is local observation value, and the relationship between the local observation value and the global state is shown in formula (8):
[0099]
[0100] Among them, s t Let t be the global state of the system. Let Agg represent the local observation of UAV j at time t, v represent the vertex, and N(v) represent the local observation of UAV j at time t. i Let ) represent the set of neighboring nodes of node i, and let σ and W represent the parameters of the feedforward neural network, where i and j are both positive integers.
[0101] Specifically, the global state value function Q tot (s,a) is a function of the state vector s and the joint action a. According to the implicit function theorem, Q tot It can also be viewed as a function of local action value Q. tot (s,Q i The function can be used to obtain the expression of the global action value function as shown in formula (9):
[0102]
[0103] Among them, s t Let t be the global state of the system. Let Agg represent the local observation of UAV j at time t, v represent the vertex, and N(v) represent the local observation of UAV j at time t. i Let σ represent the set of neighbors of node i, and let W and Q represent the parameters of the feedforward neural network. i Let be the local motion value function of the i-th UAV, where i and j are both positive integers.
[0104] Specifically, the parameters of the graph collapse network are updated through the process described in Equation (10):
[0105]
[0106] in, Let W be the first loss function, b be the weight, and α be the learning rate.
[0107] Example 2:
[0108] The parts not mentioned in this embodiment are the same as in Embodiment 1.
[0109] like Figure 3 , Figure 4 As shown, the multi-rotor drone is a highly stable and expandable six-rotor industrial drone with an all-carbon fiber fuselage. This embodiment will use the multi-rotor drone as an intelligent agent and implement it based on the technical solution of this invention.
[0110] The multi-UAV reinforcement learning module based on a dynamic directed graph communication structure distinguishes between two task execution scenarios for multi-rotor UAVs according to different execution strategies, providing support for full-process logical analysis and diagnosis. The specific details are as follows:
[0111] Scenario 1: When a multi-rotor drone needs to perform a task that allows access to the global state.
[0112] Scenario 2: When a multi-rotor drone needs to perform a task in which the global state is inaccessible.
[0113] like Figure 4 The above, Figure 4 'a' is a flowchart illustrating a task that a multi-rotor drone needs to perform, which requires access to the global state. Figure 4 b is a flowchart illustrating a task that a multi-rotor drone needs to perform, where the global state is inaccessible. When the multi-rotor drone faces situation 1, the specific steps for performing the relevant task using the TGCNet module and multi-agent reinforcement learning method include:
[0114] Step S101: Build an integrated equipment platform including the UAV control system, and deploy sensors and a multi-agent reinforcement learning network module based on a dynamic directed graph communication structure.
[0115] Step S102: The UAV control system issues collaborative task instructions for the multi-rotor UAV swarm to the multi-agent system;
[0116] Step S103: Input the mission commands of the UAV control system and the data captured by the sensors into the TGCNet module through the wireless receiving device, and let the TGCNet module analyze the data;
[0117] Step S104: Process all input information through the multi-key gating communication network; input the processed data and the global state of the system into the trained graph collapse network, determine the current action of the multi-rotor UAV cluster based on the generated global action value function, and send the action strategy to the UAV control system.
[0118] Step S105: The UAV control system receives the action strategy issued by the TGCNet module and evaluates and analyzes it. If the current action of the multi-rotor UAV swarm meets the task command requirements issued by the UAV control system, the multi-rotor UAV swarm is allowed to execute the current action by default. If the current action of the multi-rotor UAV swarm does not meet the task command requirements issued by the UAV control system, the UAV control system will send a command to stop the current action of the multi-rotor UAV swarm and require the integrated module to collect all data again and repeat steps S3 to S5 until the action of the multi-rotor UAV swarm meets the task command requirements issued by the UAV swarm control system, and then stop the process.
[0119] When a multi-rotor drone faces situation 2, the specific steps for performing the relevant task using the TGCNet module and multi-agent reinforcement learning method include:
[0120] Step S201: Build an integrated equipment platform that includes an unmanned aerial vehicle (UAV) control system, and deploy sensors and a multi-agent reinforcement learning network module based on a dynamic directed graph communication structure.
[0121] Step S202: The UAV control system issues collaborative task instructions for the multi-rotor UAV swarm to the multi-agent system;
[0122] Step S203: Input the mission commands of the UAV control system and the data captured by the sensors into the TGCNet module through the wireless receiving device, and let the TGCNet module analyze the data;
[0123] Step S204: Process all input information through the multi-key gating communication network, and pass the processed data as input to the graph collapse network; complete the information through the graph collapse network and generate the global state of the current multi-rotor UAV cluster; pass the generated global state and input information as input to the already trained hybrid network to generate a global action value function; determine the action of the current multi-rotor UAV cluster based on the global action value function, and send the action strategy to the UAV control system;
[0124] Step S205: The UAV control system receives the action strategy issued by the TGCNet module and evaluates and analyzes it. If the current action of the multi-rotor UAV swarm meets the task command requirements issued by the UAV control system, the multi-rotor UAV swarm is allowed to execute the current action by default. If the current action of the multi-rotor UAV swarm does not meet the task command requirements issued by the UAV control system, the UAV control system will send a command to stop the current action of the multi-rotor UAV swarm and require the integrated module to collect all data again and repeat steps S3 to S5 until the action of the multi-rotor UAV swarm meets the task command requirements issued by the UAV swarm control system, and then stop the process.
[0125] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0126] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A multi-UAV coordinated control method based on a dynamic directed graph communication structure, characterized in that, Includes the following steps: The drone acquires local observations of the environment and obtains historical hidden states from other drones through communication. In the distributed execution phase, based on the initial topology of the UAVs, the current adjacency trajectory matrix is obtained to represent the communication relationship between UAVs at the current moment. Pre-communication information is calculated based on local observations and historical hidden states. The current adjacency trajectory matrix and pre-communication information are input into a multi-key gated communication network to calculate the adjacency trajectory matrix at the next moment. Each UAV calculates its local action value function based on the pre-communication information and the adjacency trajectory matrix at the next moment through a Transformer-based decoder and updates its hidden state. Based on the local action value function and local observations, the final control decision is generated and executed through a graph collapse network and a hybrid network, and then the next loop begins. During the centralized training phase, the adjacency trajectory matrix at each moment of the distributed execution phase is recorded. The local observations and adjacency trajectory matrix at each moment are input into the graph collapse network to generate the global state at each moment. Based on the global state at each moment and the local action value function of each UAV at each moment, the global action value function at each moment is generated. Based on the global action value function, the UAV swarm interacts with the environment and obtains rewards. Based on the rewards, the parameters of the hybrid network are updated through the first loss function. After centralized training is completed, the trained graph collapse network and hybrid network are deployed on various drones.
2. The multi-UAV coordinated control method based on a dynamic directed graph communication structure according to claim 1, characterized in that, The adjacency trajectory matrix represents the communication structure of multiple UAVs. The adjacency trajectory matrix is a Boolean matrix, and the values of the matrix elements indicate whether there is communication between the UAVs.
3. The multi-UAV coordinated control method based on a dynamic directed graph communication structure according to claim 1, characterized in that, The calculation of pre-communication information based on local observations and historical hidden states specifically includes: Local observations and historical hidden states are converted into pre-communication information C0 through an encoding network MLP: Wherein, C0 represents pre-communication information. These are local observations. It is in a hidden historical state.
4. The multi-UAV coordinated control method based on a dynamic directed graph communication structure according to claim 1, characterized in that, The process of inputting the current adjacency trajectory matrix and pre-communication information to the multi-key gated communication network, and calculating the adjacency trajectory matrix at the next time step, specifically includes: The current adjacency trajectory matrix A t The pre-communication information C0 is input into the multi-key access control communication network for processing. The calculation formula for the multi-key access control communication network is as follows: Where K0 is the output of the key, representing the updated adjacency trajectory matrix in this communication; Gumbel-softmax is a discrete sampling technique used to sample from discrete communication structures to determine whether each pair of UAVs communicates; w v Linear layer weights for the output dimension, used to generate the final communication decision; tanh is the activation function; C0 is the pre-communication information; W... q For the weight of the query, W k The weight of the key; After processing through the multi-key gate communication network, a total of i keys are obtained at time t, denoted as k. t|i The adjacency trajectory matrix at the next time step is represented as: in, Indicates rounding up, A t+1 This is the adjacent trajectory matrix for the next time step.
5. The multi-UAV coordinated control method based on a dynamic directed graph communication structure according to claim 1, characterized in that, Each UAV calculates its local action value function based on pre-communication information and the adjacency trajectory matrix at the next time step using a Transformer-based decoder, and updates its hidden state, specifically including: The pre-communication information is repeatedly processed by a multilayer perceptron (MLP) and then input along with the adjacency trajectory matrix for the next time step into a Transformer-based decoder. The Transformer-based decoder includes a self-attention mechanism, a position feedforward network, residual connections, and a normalization layer. The input information undergoes multi-head attention, residual connections and normalization, point-by-point feedforward neural network, and further residual connections and normalization. The processed information is then added to the pre-communication information and passed through a fully connected layer to obtain the updated hidden state of UAV i. With local action value function Q i (s,a).
6. The multi-UAV coordinated control method based on a dynamic directed graph communication structure according to claim 1, characterized in that, The graph collapse network includes a graph convolutional network, a self-attention pooling network, and a readout mechanism.
7. A multi-UAV coordinated control method based on a dynamic directed graph communication structure according to claim 1 or 6, characterized in that, The global state acquisition process includes: Local observations from each UAV As the feature input for each node in the graph convolutional network, feature extraction and node representation learning are performed, using the following formula: in, These are the local observations of UAV i at time t. It is a representation of a node. Let be the value of the i-th node in the adjacency trajectory matrix at the next time step. Let represent the degree matrix of the i-th UAV at time t+1, and σ be the nonlinear activation function; Represent the learned nodes The input is fed into a self-attention pooling network. This network selects important nodes in the graph through feature weighting and node importance evaluation, and then weights the features of these important nodes, outputting a feature-weighted node graph. The formula is as follows: in, Let i be the weighted feature node representation of node i, concat be the feature fusion method, tanh be the activation function, and ⊙ be the element-wise multiplication. The weighted feature node representations of each node are aggregated through a reading mechanism to generate the global state s. t The formula is: Where N is the number of nodes, max is the aggregation function, and i and n are both positive integers.
8. The multi-UAV coordinated control method based on a dynamic directed graph communication structure according to claim 1, characterized in that, The process of generating the global action value function at each moment based on the global state at each moment and the local action value function of each UAV at each moment specifically includes: The global state s t With the local motion value function Q of each UAV i (s,a) are input into the hybrid network, and the drone swarm learns from the hybrid network Q. tot (τ,a,Am,s;θ), using the QMIX network, the weights of each local action value function are determined using the global state, and the local action value functions of each UAV are weighted and synthesized into a global action value function Q. tot (s,a).
9. A multi-UAV coordinated control method based on a dynamic directed graph communication structure according to claim 1, characterized in that, The first loss function is: in, Let b be the first loss function, and b be the batch size, representing the number of samples sampled from the replay buffer. Let Q be the target Q value of drone i. tot (τ t ,α t ,m t ,s t ;θ) is the Q-value of the hybrid network at time t, τ t Let α be the historical observation value at time t. t Let m be the action at time t. t For the information of the drone swarm at time t, s t Let be the global state at time t, θ be the parameters of the hybrid network, r be the reward obtained by the drone swarm through interaction with the environment, and γ be the discount factor. This represents the action a that maximizes the value. t+1 Q tot value.
10. A multi-UAV coordinated control method based on a dynamic directed graph communication structure according to claim 1, characterized in that, The distributed execution phase and the centralized training phase are bridged by an adjacency trajectory matrix.