Unmanned aerial vehicle sub-cluster multi-controller deployment method based on multi-dimensional graph fusion and deep reinforcement learning

Abstract

The invention discloses an unmanned aerial vehicle sub-cluster multi-controller deployment method based on multi-dimensional graph fusion and deep reinforcement learning, and the method comprises the steps: constructing a multi-dimensional graph model of network topology, network time delay, signal-to-noise ratio, node energy, node load and inter-node task relevance, so as to comprehensively describe the structure and state information of an unmanned aerial vehicle sub-cluster; a graph neural network (GNN) is introduced to carry out feature extraction on a graph structure, high-dimensional embedded representation, namely fusion representation Zi of unmanned aerial vehicle nodes is obtained, then a deep reinforcement learning model is constructed, K controllers are selected to carry out deployment as an action, a multi-target reward function considering indexes such as coverage rate, time delay, energy consumption and load balance is designed, and a multi-target reward function is constructed. And the strategy network is optimized by adopting a PPO algorithm, so that the characteristics of the modeling network are fused by a multi-dimensional graph, and the dynamic learning of the controller deployment strategy is realized by utilizing deep reinforcement learning so as to adapt to a dynamically changing network environment, and the purpose of improving the deployment efficiency and adaptability is realized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of unmanned aerial vehicle sub-cluster multi-controller deployment, and more specifically relates to a multi-dimensional graph fusion and deep reinforcement learning-based unmanned aerial vehicle sub-cluster multi-controller deployment method. BACKGROUND

[0002] In recent years, unmanned aerial vehicles have been rapidly and widely applied and developed due to their highly flexible flight capabilities, autonomously controllable flight control systems, rich and diverse load capabilities, and characteristics of various forms, portability and easy deployment. At present, with the increasing demand for urban management, agricultural monitoring, logistics distribution, environmental monitoring and emergency rescue, unmanned aerial vehicles are evolving towards miniaturization, intelligentization, clustering and informatization.

[0003] In cluster applications, unmanned aerial vehicles usually achieve efficient communication between multiple machines through self-organizing networks, have the ability of dynamic networking, strong anti-interference and multi-hop relay, and are suitable for complex and variable wireless communication environments. However, as unmanned aerial vehicle systems gradually expand towards large-scale and intelligent collaboration, the number of unmanned aerial vehicles increases exponentially, and task collaboration and information sharing become more complex, which poses higher challenges to the communication capabilities, scheduling mechanisms and stability of existing self-organizing networks.

[0004] The prior art uses the idea of software-defined networks to make related improvements to unmanned aerial vehicle self-organizing networks. Software-defined networks logically separate the control plane and data plane of traditional network devices, abstract the control plane of each routing device to form a unified control plane, and the data plane only performs data forwarding. The abstracted control plane is deployed in a general server in the form of software and is called a controller. Network managers write related APPs to issue forwarding rules to the switches of the data plane in the form of flow tables through the controller to achieve efficient and flexible forwarding of business data. The controller has a global network topology, so the strategies deployed on the controller can achieve data packet forwarding in an optimal or near-optimal manner. Software-defined networks are divided into three layers: application layer, control layer and data layer. By using the idea of software definition to improve self-organizing networks, optimal routing strategies for self-organizing networks are developed according to the global network state to achieve efficient and reliable transmission of unmanned aerial vehicle communication data.

[0005] With the wide application of UAV cluster in emergency communication, post-disaster rescue and intelligent inspection, how to realize efficient deployment and collaborative control of multiple controllers becomes a key problem to be solved. The traditional single-controller SDN architecture cannot meet the demand of high performance and high reliability. On the one hand, the single-controller architecture has performance bottleneck and is difficult to handle large-scale data flow control requests; on the other hand, the failure of the controller will lead to paralysis of the whole network control plane. Introducing multiple controllers and reasonable deployment becomes an effective solution to improve the scalability and fault tolerance of SDN.

[0006] The problem of multi-controller deployment of UAV cluster is essentially a combinatorial optimization problem, which requires selecting a number of controller nodes and their controlled switch sets under the given network topology and constraint conditions to minimize network control overhead, minimize link delay, load balance, minimize total energy consumption, meet environmental reachability constraints, etc. Traditional methods rely on heuristic algorithms or static optimization strategies, which are difficult to adapt to dynamic network environment. The efficiency and adaptability of multi-controller deployment in UAV cluster communication and control system are low. SUMMARY

[0007] The purpose of the present application is to overcome the shortcomings of the prior art and provide a multi-dimensional graph fusion and deep reinforcement learning based UAV sub-cluster multi-controller deployment method to model network characteristics by multi-dimensional graph fusion and realize dynamic learning of controller deployment strategy by deep reinforcement learning to adapt to dynamic network environment and improve deployment efficiency and adaptability.

[0008] To achieve the above-mentioned purposes, the multi-dimensional graph fusion and deep reinforcement learning based UAV sub-cluster multi-controller deployment method comprises the following steps:

[0009] (1) UAV sub-cluster network state modeling

[0010] Constructing network topology graph G 1 , network delay graph G 2 , signal-to-noise ratio graph G 3 , node energy graph G 4 , node load graph G 5 and task association graph G 6 between nodes, wherein G k =(V,E k ,W k ), k=1,2,3,4,5,6, V={v1,v2…,v N} represents the set of UAV nodes in the UAV sub-cluster, N is the number of UAV nodes in the UAV sub-cluster, E k is the edge corresponding to G k , W k is the weight of E k , denotes the weight between UAV nodes i and j, respectively related to communication space distance, communication delay, signal-to-noise ratio, node energy, load balancing, and task relevance indicators;

[0011] (2) Neural network construction based on multi-graph fusion

[0012] 2.1) Normalization processing and node feature construction

[0013] For the weight of each graph Normalization is performed using Min-Max:

[0014]

[0015] wherein, are the minimum and maximum values in all respectively;

[0016] For constructing network topology graph G 1 , network delay graph G 2 , signal-to-noise ratio graph G 3 , node load graph G 5 , and task relevance graph G 6 between nodes, node ID is used to construct node features: the embedding vector e i of UAV node v i = Embed(ID i ), Embed represents a learnable Embedding layer, ID i represents the ID information of the i-th UAV node, i.e., a row vector with dimension d, and the node feature

[0017] For the node energy graph G 4 , node energy is used to construct node features: first, the current energy i of UAV node v is normalized:

[0018]

[0019] wherein, are the minimum and maximum values in the current energy respectively;

[0020] Then, a learnable Embedding layer is used to obtain the energy embedding vector i of UAV node v is a row vector with dimension d, the node feature

[0021] 2.2), Multi-graph neural network feature representation and fusion

[0022] First, for each graph, a graph convolutional neural network is used for processing to obtain the output embedding H of all UAV nodes (k) :

[0023]

[0024] where GCN k is the graph convolution operation of the kth graph, T represents the transpose, is the output embedding vector of the ith UAV node of the kth graph, is a row vector with dimension d;

[0025] Then, the attention scoring calculation is performed:

[0026]

[0027] where att k is the attention weight vector, V k is a learnable linear transformation matrix used to map the node representation to the attention space, and both parameters are obtained through the model training process;

[0028] Then, normalization processing is performed:

[0029]

[0030] Finally, the fusion representation Z of the UAV node is obtained i :

[0031]

[0032] For the fusion representation Z of the UAV node at the current time t i is denoted as

[0033] (3), Deep reinforcement learning model construction

[0034] 3.1), Markov decision process construction

[0035] The deployment of the UAV cluster controller is modeled as a Markov decision process, and the current state The current action a t represents selecting K controllers from all UAV nodes for deployment, and R(s t , a t ) represents the reward obtained in the current statet In the case of the current action a t , the reward value is obtained.

[0036] 3.2), the reward function R(s t ,a t ) is calculated.

[0037] R(s t ,a t ) = -λ·D avg (a t ) + ξ·SNR(a t ) + δ·(f v (a t )·d(a t )) + ζ·E r (a t ) + μ·T task (a t )

[0038] wherein D avg (a t ) represents the average network delay of all exchange nodes in the UAV sub-cluster to the corresponding controller in the case of the current action a t , SNR(a t ) represents the signal-to-noise ratio of the UAV sub-cluster link in the case of the current action a t , f v (a t ) represents the link flow between UAV sub-cluster nodes in the case of the current action a t , d(a t ) represents the size of the node degree of the UAV sub-cluster in the case of the current action a t , E r (a t ) represents the size of the residual energy of the UAV sub-cluster node in the case of the current action a t , T task (a t ) represents the time required for task execution in the case of the current action a t , and λ, ξ, δ, ζ, μ are all normalized weight coefficients.

[0039] 3.3), controller deployment and policy network optimization

[0040] The policy network adopts an Actor-Critic architecture based on proximal policy optimization, takes the current state s t as input, selects K UAV nodes for controller deployment according to the reward function R(s t-1 ,a t-1 ), and optimizes the policy network using the PPO algorithm.

[0041] The purpose of the present application is achieved.

[0042] The present application is based on a multi-dimensional graph fusion and deep reinforcement learning method for deploying multiple controllers of a UAV sub-cluster. First, a multi-dimensional graph model of network topology, network delay, signal-to-noise ratio, node energy, node load, and task correlation between nodes is constructed to comprehensively describe the structure and state information of the UAV sub-cluster. Second, a graph neural network (GNN) is introduced to extract features from the graph structure and obtain a high-dimensional embedding representation, i.e., the fusion representation Z of the UAV nodes. i Then, a deep reinforcement learning model is constructed to select K controllers for deployment as actions. A multi-objective reward function is designed to consider coverage, delay, energy consumption, and load balancing, and the PPO algorithm is used to optimize the policy network. In this way, the network characteristics are modeled using multi-dimensional graph fusion, and deep reinforcement learning is used to dynamically learn the controller deployment strategy to adapt to dynamic network environments, achieving the goals of improving deployment efficiency and adaptability.

[0043] In addition, the present application has the following beneficial effects:

[0044] (1) Enhancing the communication delay and task response performance of the UAV sub-cluster

[0045] By introducing a multi-dimensional heterogeneous graph that fuses link topology, communication delay, node load, channel interference strength, and geographic location information, the communication state and scheduling requirements between each UAV in the cluster can be fully reflected, effectively reducing control signaling delay and improving group collaborative response speed.

[0046] (2) Improving the communication quality and anti-interference ability of the UAV cluster control link

[0047] By including the signal-to-noise ratio (SNR) information between nodes in the multi-dimensional heterogeneous graph modeling, a multi-dimensional state space that fuses network topology, link delay, load state, robustness, and channel quality is constructed. The controller deployment strategy can preferentially select areas with high communication link quality and low interference to deploy controllers, thereby improving overall communication reliability and anti-interference performance. By adding a signal-to-noise ratio reward factor in the strategy optimization, the deployment strategy can be guided to avoid high-interference areas and low-signal-to-noise ratio links, thereby reducing retransmission rate and packet loss probability, improving the real-time performance and success rate of critical control instruction transmission, and ensuring the stability of the task control link.

[0048] (3) Highly distributed adaptive control deployment capability

[0049] By using graph neural networks to perform structured modeling of UAV swarm topology, deep reinforcement learning strategies can adjust the controller deployment location online according to the dynamic changes in swarm size, mission area, and channel status, achieving real-time adaptive response to distributed control requirements. The controller deployment strategy has the ability to dynamically perceive changes in the communication environment and can intelligently adjust the controller deployment location according to real-time communication status changes, adapting to complex scenarios such as urban building obstruction, electronic interference, and high-speed flight.

[0050] (4) Reduce the risk of single-point congestion caused by uneven communication load.

[0051] By using controller load balancing as one of the strategy optimization objectives, the complexity of the tasks processed by each control node can be effectively balanced, avoiding communication congestion and control lag caused by centralized control or uneven deployment, and improving the parallel execution efficiency of the unmanned cluster system.

[0052] (5) Possesses cross-scenario generalized deployment capabilities and engineering practical value

[0053] The proposed deployment method forms a deployment model with generalization capabilities through policy learning, which can be quickly migrated to UAV swarm missions under different terrains, mission objectives and networking structures. It supports complex operational requirements such as multi-mission switching, temporary grouping and dynamic addition and removal of nodes, and has strong potential for engineering deployment and practical application. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of a drone swarm communication architecture based on the integration of software-defined networking and ad hoc networking;

[0055] Figure 2 This is a flowchart of a specific implementation of the UAV sub-swarm multi-controller deployment method based on multidimensional graph fusion and deep reinforcement learning of the present invention;

[0056] Figure 3 It is a drone swarm network topology Figure 1 Specific example illustrations;

[0057] Figure 4 The latency of drone swarm networks Figure 1 Specific example illustrations;

[0058] Figure 5 Signal-to-noise ratio of drone swarm Figure 1 Specific example illustrations;

[0059] Figure 6 It is the energy of the drone swarm nodes. Figure 1 Specific example illustrations;

[0060] Figure 7 It is the load of drone cluster nodes Figure 1Specific example schematic diagram

[0061] Figure 8 Task correlation between unmanned aerial vehicle cluster nodes Figure 1 Specific example schematic diagram. DETAILED DESCRIPTION

[0062] The specific embodiments of the application are described below with reference to the accompanying drawings in order to better understand the application by those skilled in the art. It should be particularly noted that in the following description, when the detailed description of the known functions and designs may obscure the main content of the application, these descriptions will be omitted here.

[0063] Figure 1 Unmanned aerial vehicle cluster communication architecture based on software-defined network and ad hoc network fusion.

[0064] As Figure 1 shown, the unmanned aerial vehicle cluster has information interaction between multiple sub-clusters and ground stations in the fields of emergency communication, post-disaster rescue and intelligent inspection, etc. The application is based on software-defined network (Software-Defined Networking, SDN) and ad hoc network, as Figure 2 shown, assuming that the aerial unmanned aerial vehicle cluster coexists in M sub-clusters, each sub-cluster has N unmanned aerial vehicles, and there are NXM unmanned aerial vehicles in total. Each sub-cluster is called a communication domain, such as unmanned aerial vehicle sub-cluster 1 as communication domain 1, and unmanned aerial vehicle cluster m as communication domain m. Each communication domain can use different network protocols, such as communication domain 1 using IPv4, communication domain 2 using IPv6, and communication domain M using an identification network (non-IP network), and the ground station using IPv4 or IPv6, etc. The unmanned aerial vehicle cluster has strong mobility and frequent node joining and exiting, so the unmanned aerial vehicle sub-cluster faces the problem of how to deploy the controller in the software-defined network.

[0065] To solve the above problems, the innovations of the application are as follows: in view of the characteristics of the unmanned aerial vehicle cluster, such as large node scale, strong networking dynamics and strong mobility, combined with software-defined network, an ONOS (Open Network Operating System) distributed multi-controller network architecture based on clustering is proposed. In view of the characteristics of high-dimensional dynamics, heterogeneity and strong coupling of the controller deployment problem in the unmanned aerial vehicle cluster, the application proposes a multi-dimensional graph fusion and deep reinforcement learning based unmanned aerial vehicle sub-cluster multi-controller deployment method to model the network characteristics by multi-dimensional graph fusion, and to realize dynamic learning of the controller deployment strategy by deep reinforcement learning to adapt to the dynamically changing network environment and improve the deployment efficiency and adaptability. As Figure 3 shown, the multi-dimensional graph fusion and deep reinforcement learning based unmanned aerial vehicle sub-cluster multi-controller deployment method of the application includes the following steps:

[0066] Step S1: Modeling of the network state of the UAV sub-cluster

[0067] In order to fully express the network communication state and task constraints of the multi-controller deployment of the UAV sub-cluster, the network state is modeled as a heterogeneous graph structure containing multiple dimensions, which respectively capture the network topology structure, network delay, signal-to-noise ratio, node energy, node load, and inter-node task association, etc. Each dimension constitutes an independent weighted graph, and finally a unified representation is integrated through a graph fusion mechanism, specifically:

[0068] Constructing a network topology graph G 1 , a network delay graph G 2 , a signal-to-noise ratio graph G 3 , a node energy graph G 4 , a node load graph G 5 , and an inter-node task association graph G 6 , where G k =(V, E k , W k ), k = 1, 2, 3, 4, 5, 6, V = {v1, v2, …, v N} represents the set of UAV nodes in the UAV sub-cluster, N is the number of UAV nodes in the UAV sub-cluster, E k is the edge corresponding to G k , and W k is the weight of E k . represents the weight between UAV nodes i and j, which are respectively related to the communication space distance, transmission delay, signal quality, node energy consumption, load balancing, and task association indicators.

[0069] Step S1.1: Construction of the network topology graph

[0070] The network topology graph of the UAV cluster is constructed using an undirected graph G 1 =(V, E 1 , W 1 ), where V = {v1, v2, …, v N} represents the set of UAV nodes in the UAV sub-cluster, E 1 is the set of physical links in the network, and W 1 is the weight of E 1 , which represents the inverse of the distance between two nodes and the link bandwidth, emphasizing the negative impact of spatial distance and low bandwidth on the deployment location of the controller.

[0071] In this embodiment, the network topology graph is as shown in Figure 3 , where <d i,j , B i,j> Represents the weight between drone nodes i and j coefficient and d i,j B i,j The parameters are related, and the specific relationships are as follows:

[0072]

[0073] Where, α topology The coefficient d represents the distance between two drone nodes. i,j β represents the communication spatial distance between drone nodes i and j. topology B is a coefficient representing the link bandwidth. i,j This represents the link bandwidth between drone nodes i and j, where the spatial distance d i,j The calculation method is as follows:

[0074] First, convert latitude and longitude to a three-dimensional rectangular coordinate system, as shown below:

[0075]

[0076] Where x, y, and z represent three-dimensional spatial coordinates, respectively. ω represents longitude and latitude information respectively, and ρ is the Earth's radius;

[0077] Then, the spatial distance d is calculated using the vector dot product. i,j .like Figure 4 As shown, assuming the Earth's center is O, the latitude and longitude of the UAV node A are respectively... ω A The latitude and longitude of UAV node B are respectively ω B The coordinates of UAV nodes A and B in the Cartesian coordinate system are (x, y, y) and (x, y, y) respectively. A ,y A ,z A ),(x B ,y0,z B ),but:

[0078] cos∠AOB=x A x B +y A y B +z A z B / ρ 2

[0079] d can be solved using trigonometric functions. A,B Size:

[0080]

[0081] For drone nodes i and j, the spatial distance d i,j for:

[0082]

[0083] in, ω i These are the latitude and longitude coordinates of UAV node i, respectively. ω j These are the latitude and longitude coordinates of UAV node j.

[0084] Step S1.2: Network Delay Graph Construction

[0085] The communication delay τ of the UAV sub-swarm is mainly composed of the propagation delay τ prop Processing delay τ proc Queuing delay τ queue and transmission delay τ trans Composition, in which the propagation delay τ prop The processing delay τ is the time required for a wireless signal to propagate from the transmitter to the receiver in the channel. proc Queuing delay τ is the time required for a drone node to process data packets. queue The transmission delay τ refers to the time it takes for a data packet to wait in the queue of a drone node for processing or forwarding. trans The time required for the sender to pop a complete data packet from its buffer and transmit it to the channel depends on the packet size and the link's transmission rate. Assuming communication between two UAV nodes i and j requires n hops (n+1 nodes in total), the total delay τ is... total,j> The calculation is as follows:

[0086]

[0087] Where, τ total,j＞ Let l represent the total latency between drone node i and drone node j, and l∈(i→j) represent drone node l in the link from drone node i to drone node j, including drone node i but excluding drone node j. next This indicates the next drone node after drone node l. Represents drone node l and drone node l next The propagation delay Indicates drone section next Processing latency, τ trans denotes the transmission delay of the UAV node i, since the transmission delay of each UAV node is not much different, it can be considered that the transmission delay of each UAV node is the same, and the required transmission delay between the UAV node i and the UAV node j is obtained as [(τ trans +1) - 2] * 2 + 2 = 2n x τ trans .

[0088] so,

[0089] In this embodiment, as Figure 5 As shown, for UAV nodes A, B, C, and D, the total latency for direct communication between each UAV node is τ. total,B> τ total＜A,C＞ τ total<A,D＞ τ total＜B,C> τ total＜B,D＞ τ total,D> .

[0090] Step S1.3: Constructing the signal-to-noise ratio map

[0091] In a wireless environment, although the paths between two communication nodes are the same (e.g., from node A to B and from B to A), differences in antenna transmit power, antenna gain, receiver parameters, multipath effects, link interference, and uplink / downlink frequency bands can cause the wireless channel to be considered a two-way link. That is, the path from node A to B and from B to A involve different links, resulting in different signal-to-noise ratios (SNRs). Assuming a subgroup of UAVs consists of adjacent UAV nodes i and j, the SNR from UAV node i to j is SNR. i→j The signal-to-noise ratio (SNR) from drone node j to i is SNR. j→i SNR i→j The calculation formula is: in, G represents the transmit power of node i. ij Indicates channel gain. This represents the receiver noise power of node j. The signal-to-noise ratio (SNR) graph for UAV sub-swarm communication is constructed based on a directed graph, reflecting the communication quality between two nodes in a real-world environment. It is constructed using a directed weighted graph, where the weights are... This is the sum of the signal-to-noise ratios (SNRs) of all adjacent drone nodes on the link from node i to j. In this embodiment, the SNR of each adjacent drone node is... Figure 6 As shown.

[0092] Step S1.4: Construction of Node Energy Graph

[0093] During mission execution, the UAV's performance is constrained by the energy state of its nodes, and the controller's deployment location is closely related to the node's energy state. (Energy attributes of UAV node i) It changes dynamically over time, and its content includes: current energy. Maximum energy Energy consumption rate Then Based on the above description, an energy graph of drone swarm nodes is constructed using an undirected graph, where the weights of the edges in the graph are... The calculation is as follows:

[0094]

[0095] wherein, and respectively represent the current energy of the unmanned aerial vehicle nodes i and j, and a energy and β energy respectively represent the weight adjustment parameters.

[0096]

[0097] The two terms of the above formula respectively represent the life stress of the unmanned aerial vehicle nodes i and j.

[0098] In this embodiment, the edge weight of the node energy graph of each adjacent unmanned aerial vehicle node is as shown in Figure 7 .

[0099] Step S1.5: Construction of node load graph

[0100] One of the core problems of the controller deployment of the unmanned aerial vehicle cluster is load balancing. The load of the unmanned aerial vehicle cluster node mainly considers the traffic condition of the node and the range of direct communication of the node with other nodes. Therefore, the traffic size f and the node degree de between nodes need to be considered when constructing the node load graph. The node load graph is constructed based on an undirected graph, wherein the weight μ and σ respectively represent the coefficients of traffic and node degree, f i,j is the traffic size between the unmanned aerial vehicle nodes i and j, and de i,j is the sum of the node degrees of the unmanned aerial vehicle nodes i and j.

[0101] In this embodiment, as shown in Figure 8 , wherein represents the weight between the node i and the node j, and the node degree is the number of edges directly connected to a node, such as the degree of 3 of the node A in the figure.

[0102] Step S1.6: Construction of inter-node task correlation graph

[0103] In the UAV cluster system, the tasks between nodes have dependency and cooperation, each UAV node has certain autonomous decision-making ability, and the overall task is realized through local information interaction. The UAV cluster task usually contains multiple sub-tasks, and there are explicit or implicit dependency relationships, such as "area patrol-target identification-precise operation", "communication relay-data backhaul" and other task chains. The controller deployment in the UAV cluster mainly considers the following four points: first, the task association determines the communication demand, which affects the controller deployment structure, such as the task graph being a star-shaped task dependency graph, then considering deploying the controller in the center node; second, the task dependency relationship will affect the cooperation mechanism between controllers, such as strong dependency chain task, which needs to be distributed deployment, close to the task chain node; third, the change of task association graph drives the controller redeployment, such as target change, environmental change or UAV failure, which leads to the change of controller deployment position; fourth, the task-driven controller deployment strategy, including minimizing the task completion time, minimizing the control message overhead, and maximizing the robustness. Based on the cluster task association graph (TDG) of area patrol, target identification, precise operation and other tasks, the task association graph G 6 , E 6 between nodes is constructed, which represents the dependency relationship between tasks, and the weight represents the priority of the task, and the task association graph between UAV cluster nodes is shown in ​ .

[0104] Step S2: neural network construction based on multi-graph fusion

[0105] Step S2.1: normalization processing and node feature construction

[0106] Before using the graph neural network, the above-mentioned 6 graphs need to be normalized:

[0107] For the weight of each graph, Min-Max normalization is adopted:

[0108]

[0109] Among them, are the minimum and maximum values of all . In this way, when , when ,

[0110] For constructing network topology graph G 1 , network delay graph G 2 , signal-to-noise ratio graph G 3 , node load graph G 5 and inter-node task correlation graph G 6 Construct node features with node IDs: UAV node v i Embedding vector e i = Embed(ID i ), Embed represents a learnable Embedding layer, ID i represents the ID information of the i-th UAV node, i.e., a row vector with dimension d, and all embedding vectors e i form a node ID embedding matrix, which belongs to i.e., an N-row d-column matrix, represents a row vector with d columns, and all elements in the matrix are real numbers, and let the node feature

[0111] For the node energy graph, the node energy graph G 4 Construct node features with node energy: first, normalize the current energy i of the UAV node v :

[0112]

[0113] wherein are the minimum and maximum values in the current energy respectively;

[0114] Then, use the learnable Embedding layer to obtain the energy embedding vector i of the UAV node v i.e., a row vector with dimension d, and the node feature

[0115] Step S2.2: Multi-graph neural network feature representation and fusion

[0116] First, use a graph convolutional neural network to process each graph to obtain the output embedding H (k) of all UAV nodes:

[0117]

[0118] wherein GCN k is the graph convolution operation of the k-th graph, T represents the transpose, is the output embedding vector of the i-th UAV node of the k-th graph, i.e., a row vector with dimension d;

[0119] Then, perform attention scoring calculation:

[0120]

[0121] where att k denotes the attention weight vector, V k denotes a learnable linear transformation matrix for mapping node representation to attention space, both of which are obtained through model training process;

[0122] Then, normalization processing is performed:

[0123]

[0124] Finally, the fusion representation Z i of the UAV node is obtained.

[0125]

[0126] The fusion representation Z i of the UAV node at the current time t is denoted as

[0127] Step S3: Deep reinforcement learning model construction

[0128] Step S3.1: Markov decision process construction

[0129] The deployment of the UAV cluster controller is modeled as a Markov decision process, and the current state s The current action a t represents selecting K controllers from all UAV nodes for deployment, and R(s t ,a t ) represents the reward value obtained by selecting the current action a t under the condition that the current state is s t .

[0130] Because the action dimension is large, the invention uses a probability sampling strategy or a step-by-step action construction method to reduce the computational complexity.

[0131] Step S3.2: Calculate the reward function R(s t ,a t )

[0132] R(s t ,a t ) = -λ·D avg (a t ) + ξ·SNR(a t ) + δ·(f v (a t )·d(a t )) + ζ·E r (a t )+ μ · T task (a t )

[0133] wherein, D avg (a t ) represents the average network delay from all switching nodes in the sub-cluster of UAVs to the corresponding controller in the case of taking the current action a t , SNR(a t ) represents the signal-to-noise ratio of the link of the sub-cluster of UAVs in the case of taking the current action a t , f v (a t ) represents the inter-node link traffic of the sub-cluster of UAVs in the case of taking the current action a t , d(a t ) represents the size of the node degree of the sub-cluster of UAVs in the case of taking the current action a t , E r (a t ) represents the size of the residual energy of the node of the sub-cluster of UAVs in the case of taking the current action a t , T task (a t ) represents the time required for task execution in the case of taking the current action a t , and λ, ξ, δ, ζ, μ are all normalized weight coefficients.

[0134] The reward function R(s t ,a t ) is used to guide the optimization of the deployment strategy of the controller of the cluster of UAVs towards the direction of low delay, high load balancing, high signal-to-noise ratio, large node energy, strong robustness, and high task completion quality.

[0135] Step S3.3: Controller deployment and strategy network optimization

[0136] The strategy network adopts an Actor-Critic architecture based on proximal policy optimization, takes the current state s t as input, selects K UAV nodes for controller deployment according to the reward function R(s t-1 ,a t-1 ), and optimizes the strategy network using the PPO algorithm.

[0137] Actor network: maps the current state s t to the probability of each node being selected as a controller A candidate controller set is generated by sampling or greedily selecting K nodes; Critic network: uses the estimated state value function V π (s) to guide the Actor policy update V π (s) = f critic (Z i ). Assuming that the number of sub-clusters of UAVs is K, K nodes need to be selected from all nodes to deploy controllers, and the node probability distribution output by the Actor is used to sample the top-K nodes as the controller deployment nodes.

[0138] The PPO algorithm is used to optimize the policy network, and the stability of the update is ensured, and the policy target is formulated as Wherein, r t (θ) is the policy ratio function, which represents the probability ratio of the new and old policies under the same action; is the advantage value, which represents the degree to which the current action is better than the average level; E s,a [·] represents the state-action pair obtained by sampling using the current policy The expected value is taken. The calculation method of r t (θ) is as follows:

[0139]

[0140] π θ-cur (a t |s t ) represents the probability of the UAV cluster taking the current policy corresponding to the action a t under the state s t ; π θ-las (a t |s t ) represents the probability of the UAV cluster taking the old policy corresponding to the action a t under the state s t . The policy function is used to measure the positive and negative effects of policy update on the overall performance, and is limited by the clip operation to prevent large step updates.

[0141] The Actor-Critic architecture of proximal policy optimization and the PPO algorithm for optimizing the policy network belong to the prior art, and will not be described here.

[0142] The pseudo code of the training process of the UAV sub-cluster multi-controller deployment method based on multi-dimensional graph fusion and deep reinforcement learning is as follows:

[0143]

[0144]

[0145] While the foregoing specific embodiments of the application have been described in some detail to provide a clear understanding thereof, it will be apparent to those of ordinary skill in the art that numerous modifications can be made to the specific embodiments described without departing from the spirit and scope of the application defined by the appended claims.

Claims

1. A method for multi-controller deployment of a sub-swarm of unmanned aerial vehicles based on multi-dimensional graph fusion and deep reinforcement learning, the method comprising: Comprising the following steps: (1) UAV sub-cluster network state modeling constructing a network topology graph G 1 , a network delay graph G 2 , a signal-to-noise ratio graph G 3 , a node energy graph G 4 , a node load graph G 5 , and a task correlation graph G 6 between nodes, wherein G k =(V, E k , W k ), k=1, 2, 3, 4, 5, 6, V={v1, v2, …, v N} represents a set of UAV nodes in a UAV sub-cluster, N is the number of UAV nodes in the UAV sub-cluster, E k is the edge corresponding to G k , W k is the weight of E k , represents the weight between UAV nodes i and j, k=1, 2, 3, 4, 5, 6 respectively relates to communication space distance, communication delay, signal-to-noise ratio, node energy, load balancing, and task correlation indicators. (2) Neural network construction based on multi-graph fusion 2.1) Normalization processing and node feature construction Weight for each image Normalization using Min-Max: wherein are the minimum, maximum values, respectively, in all of the set For constructing network topology graph G 1 , network latency graph G 2 , signal-to-noise ratio graph G 3 , node load graph G 5 , and inter-node task correlation graph G 6 The node features are constructed using node IDs: the embedding vector e i of the unmanned aerial vehicle node v i = Embed(ID i ), where Embed represents a learnable Embedding layer, ID i represents the ID information of the i-th unmanned aerial vehicle node, and e is a row vector with a dimension of d, and the node features are e k=1,2,3,5,6. For node energy graph node energy graph G 4 The node energy is used to construct the node feature: first, the current energy of the unmanned aerial vehicle node v i is normalized: ​ wherein are the minimum, maximum values in the current energy respectively. Then, a learnable Embedding layer is used to obtain the energy embedding vector of the UAV node v i i.e., a row vector of dimension d, the node feature k = 4;​ 2.2) Multi-graph neural network feature representation and fusion First, the graph convolutional neural network is used to process each graph to obtain the output embedding H of all UAV nodes (k) : wherein GCN k is a graph convolution operation for the k-th graph, T denotes transpose, i = 1, 2,.., N is the output embedding vector of the k-th graph drone node i, i.e. a row vector of dimension d; Then, attention scoring calculation is performed: where att k denotes the attention weight vector, V k denotes a learnable linear transformation matrix used to map the node representation to the attention space, both parameters are obtained through the model training process; Then, normalization processing is performed again: Finally, the fusion representation Z of the UAV node is obtained i : for the current time instant t the fused representation Z of the drone node i denoted by (3) Deep reinforcement learning model construction 3.1) Markov decision process construction The deployment of the UAV cluster controller is modeled as a Markov decision process, where the current state The current action a t represents the selection of K controllers for deployment from all UAV nodes, R(s t ,a t represents the reward value obtained by selecting the current action a t in the case where the current state is s t . 3.2), the reward function R(s t ,a t ) is: R(s t ,a t )=-λ·D avg (a t )+ξ·SNR(a t )+δ·(f v (a t )·d(a t ))+ζ·E r (a t )+μ·T task (a t ) wherein D avg (a t ) represents the average network delay from all switching nodes in the sub-cluster of UAVs to the corresponding controller in the case of taking the current action a t , SNR(a t ) represents the signal-to-noise ratio of the link of the sub-cluster of UAVs in the case of taking the current action a t , f v (a t ) represents the inter-node link traffic of the sub-cluster of UAVs in the case of taking the current action a t , d(a t ) represents the size of the node degree of the sub-cluster of UAVs in the case of taking the current action a t , E r (a t ) represents the size of the residual energy of the node of the sub-cluster of UAVs in the case of taking the current action a t , and T task (a t ) represents the time required for task execution in the case of taking the current action a t , and λ, ξ, δ, ζ, μ are all normalized weight coefficients. 3.3) Strategy network optimization The policy network adopts an Actor-Critic architecture based on Proximal Policy Optimization, taking the current state s t as input, according to the reward function R(s t-1 ,a t-1 ) to select K UAV nodes for controller deployment, and uses the PPO algorithm to optimize the policy network.

2. The multi-dimensional graph fusion and deep reinforcement learning based method for deploying multiple controllers for a sub-swarm of UAVs according to claim 1, wherein, In the network topology graph construction, the weight between the unmanned aerial vehicle nodes i and j is: wherein, a topology denotes a coefficient representing the distance between two UAV nodes, d i,j denotes the communication spatial distance between UAV nodes i and j, β topology denotes a coefficient representing the link bandwidth, B i,j denotes the link bandwidth between UAV nodes i and j; wherein the spatial distance d i,j is: wherein p is the radius of the earth, ω i respectively the longitude and latitude of the UAV node i, ω j respectively the longitude and latitude of the UAV node j. 3.The multi-dimensional graph fusion and deep reinforcement learning based method for deploying multiple controllers for a sub-swarm of UAVs according to claim 1, wherein, In network latency graph construction, the weight between the drone nodes i and j is: where τ total<i,j> is the total delay between UAV node i and UAV node j, which is calculated as: wherein l e (i→j) denotes a UAV node l of the UAV node i→j link, including the UAV node i, but excluding the UAV node j, l next denotes a next UAV node of the UAV node l, denotes a propagation delay of the UAV node l to the UAV node l next denotes a processing delay of the UAV node l, τ next trans denotes the transmission delay of UAV node i, n is the number of hops required for communication between two UAV nodes i, j. ​​ 4. The multi-dimensional graph fusion and deep reinforcement learning based method for deploying multiple controllers for a sub-swarm of UAVs according to claim 1, wherein, In the signal-to-noise ratio map construction, the weight between the unmanned aerial vehicle nodes i and j is the sum of the signal-to-noise ratios of all adjacent unmanned aerial vehicle nodes on the unmanned aerial vehicle node i→j link.

5. The multi-dimensional graph fusion and deep reinforcement learning based method for deploying multiple controllers for a sub-swarm of UAVs according to claim 1, wherein, In the node energy graph construction, the weight between the unmanned aerial vehicle nodes i and j is: wherein, and respectively represent the current energy of the UAV nodes i and j, a energy and β energy respectively represent the weight adjustment parameters, and respectively represent the energy consumption rate of the UAV nodes i and j.

6. The multi-dimensional graph fusion and deep reinforcement learning based method for deploying multiple controllers for a sub-swarm of UAVs according to claim 1, wherein, In the node load graph construction, the weight between the unmanned aerial vehicle nodes i and j is: where μ,σ represent the coefficients of flow and node degree, respectively, f i,j is the flow size between UAV nodes i,j, de i,j is the sum of node degrees of UAV nodes i,j.

7. The multi-dimensional graph fusion and deep reinforcement learning based method for deploying multiple controllers for a sub-swarm of UAVs according to claim 1, wherein, In the construction of inter-node task relevance, the weight between the unmanned aerial vehicle nodes i and j is the priority of the task.

Images