Unmanned aerial vehicle cooperative computing method based on graph attention mechanism

Abstract

The present application relates to the field of unmanned aerial vehicle swarm intelligence, and particularly relates to a method for unmanned aerial vehicle cooperative computing based on a graph attention mechanism, a decentralized partially observable Markov decision process model is constructed with the goal of maximizing the expected discount cumulative reward, and a geometric graph is established with unmanned aerial vehicles as nodes; secondly, a cascading feature vector is generated based on real-time observation and action of the nodes, a standard attention coefficient between the nodes is calculated through a graph attention mechanism, a total feature vector of the nodes and a benchmark joint vector are obtained by fusing multi-head features; thirdly, a timing difference target is calculated based on an evaluation network and a target evaluation network, and the evaluation network is updated by back propagation of mean square error; finally, a network update threshold delay update policy network is updated. The present application effectively suppresses overestimation of value and policy shock, and improves convergence stability and global service fairness of cooperative decision making by dynamically masking physically unreachable nodes and combining a double-delay deterministic policy gradient optimization mechanism.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) swarm intelligence, and specifically to a UAV collaborative computing method based on a graph attention mechanism. Background Technology

[0002] With the deep integration of low-altitude economy and edge computing, drones, with their advantages of maneuverability, flexibility, and on-demand deployment, have become an important carrier for building integrated air-space-ground collaborative edge computing networks. By proactively flying close to ground user hotspots, they provide low-latency computing offloading and communication relay services, effectively compensating for the shortcomings of fixed ground infrastructure in terms of coverage and dynamic response capabilities. However, the dynamic resource allocation of drone swarms in three-dimensional space still faces severe technical bottlenecks. Traditional mathematical optimization methods have huge computational overhead and extremely high latency, making it difficult to adapt to rapid changes in network topology and business needs. Standard multi-agent reinforcement learning uses fully connected state splicing, which leads to the curse of dimensionality when the swarm size increases, and redundant noise interferes with credit allocation. Although graph attention networks can capture topological relationships, they lack physical constraints on communication boundaries, continuously aggregating invalid features beyond the range, weakening the physical credibility of decisions. This is because the above solutions generally face the problems of accumulated value overestimation and gradient variance anomalies in continuous action space, making it difficult for collaborative strategies to converge stably, ultimately resulting in significant service coverage gaps.

[0003] Chinese patent CN120499746A discloses a method and system for UAV edge collaborative computing for UAV airport swarms, including the following steps: based on the UAV communication network, analyzing UAV type and real-time network data stream, assessing the current network congestion status, and adjusting the data compression ratio and transmission priority of multiple data packets according to network conditions and data types, optimizing data transmission paths, and generating transmission efficiency optimization records. While this application improves transmission efficiency and network response speed to some extent and optimizes UAV collaborative efficiency, it fails to solve the problem of unstable policy convergence caused by accumulated value overestimation and gradient variance abnormalities in continuous action space, and service coverage gaps still exist. Therefore, there is an urgent need for a UAV swarm collaborative scheduling method to solve the policy convergence problem in continuous action space and eliminate service coverage gaps. Summary of the Invention

[0004] This invention provides a UAV cooperative computing method based on graph attention mechanism to overcome the policy convergence instability caused by the accumulation of value overestimation and gradient variance abnormalities in continuous action space, and to eliminate service coverage gaps.

[0005] This invention provides a UAV cooperative computing method based on a graph attention mechanism, comprising: Taking UAVs and ground equipment as entities, a decentralized partially observable Markov decision process framework is used to transform the UAV collaborative process. The global optimization objective is to maximize the expected discount cumulative reward under the joint strategy. The expected discount cumulative reward is determined based on several expected discount cumulative rewards. A geometric graph is constructed using the aforementioned UAV as nodes, and the set of edges connecting nodes in the geometric graph is periodically updated based on the node spacing. Based on the pre-built policy network, node actions are generated using real-time observation parameters of nodes. Concatenated feature vectors are constructed based on the real-time observation parameters and node actions. Several query vectors, key vectors, and value vectors for each node are determined based on the concatenated feature vectors. Standard attention coefficients for each node to the connected edges are determined based on the query vectors and key vectors. Each node determines several node branch feature vectors based on the standard attention coefficient and the value vector, and determines the total node feature vector based on the node branch feature vectors, and determines the benchmark joint vector based on the total feature vectors of each node; The target joint vector is determined based on the current node actions. The target action is determined based on the target joint vector and the policy network with injected noise. Several evaluation networks are instantiated. The target joint vector and the target action are input into the target evaluation network to determine the target value score. The time-series difference target is determined based on the target value score and the expected discount cumulative reward. The benchmark value score is determined based on the benchmark joint vector and the benchmark node action. The loss scalar is determined based on the temporal difference objective and the benchmark value score. The evaluation network is updated by backpropagation based on the loss scalar. The baseline node action is determined based on the network update threshold. The target value score is determined based on the baseline node action and the baseline joint vector. After taking the partial derivative of the target value score, the chain rule is used to determine the policy network update gradient and update the policy network parameters.

[0006] Furthermore, the expected cumulative discount reward is determined based on Jain's fairness index of service coverage, service latency, and energy consumption; The service coverage rate is determined based on the number of qualified communication time slots and the monitoring duration. The number of qualified communication time slots is determined based on the number of times a user's communication is qualified within the monitoring duration.

[0007] Furthermore, based on the analysis condition that the node spacing is less than or equal to the communication radius, it is determined to construct node pair edges between node pairs and update the node pair edge set.

[0008] Furthermore, a policy network is constructed based on a neural network algorithm, with real-time observation parameters of nodes as input and node actions as output.

[0009] Furthermore, attention coefficients are determined based on the scaled dot product of the query vector and the key vector, and after normalization, the standard attention coefficients for each node pair connection are obtained.

[0010] Furthermore, based on the current node actions, the air-ground collaborative model is used to determine the real-time observation parameters of each node at the next time step, and the reference joint vector at the next time step is determined based on the real-time observation parameters of each node, denoted as the target joint vector.

[0011] Furthermore, the process of determining the target action includes: The policy network generates original node actions based on the target joint vector, applies constraints to the original node actions to explore noise, and obtains the target actions. Wherein, the constrained exploration noise refers to a random vector whose absolute value is less than or equal to the out-of-bounds truncation threshold and which follows a Gaussian distribution with a mean of zero.

[0012] Furthermore, the evaluation network is constructed based on a neural network algorithm, and the value score is used as the output based on the joint vector and the actions of each node as input. The loss scalar is determined based on the mean square error between the temporal difference objective and the benchmark value score; Construct a target evaluation network based on a copy of the evaluation network.

[0013] Furthermore, the benchmark value score is determined by the evaluation network based on the benchmark joint vector and the benchmark node action input; The loss scalar is determined based on the loss scalar of the time-series difference target and the benchmark value score; The reference node action refers to the joint action vector generated by each node based on the real-time observation parameters and using the policy network at the current time step.

[0014] Furthermore, based on the analysis condition that the number of network updates is greater than or equal to the network update threshold, the action to obtain the baseline node is determined.

[0015] Compared with existing technologies, the beneficial effects of this invention are as follows: By constructing an air-ground cooperative model and introducing a decentralized partially observable Markov decision process framework, this invention formalizes the UAV cooperative computing problem into a joint policy optimization problem, with the maximization of the expected discount cumulative reward as the global optimization objective, providing a unified mathematical foundation for subsequent cooperative decision-making. By introducing a graph attention mechanism on the edge set of nodes in the geometric graph, each node can dynamically aggregate the value vectors of its neighbors based on standard attention coefficients and generate a benchmark joint vector, enabling the critic network to obtain a global state representation with physical topology awareness during the centralized training phase. Furthermore, by introducing mean squared error backpropagation to update the evaluation network based on temporal difference objectives and a policy network delayed update mechanism triggered by a network update threshold during the training phase, this effectively suppresses the problems of value overestimation accumulation and gradient variance anomalies in multi-agent reinforcement learning in continuous action spaces, improving the stability of cooperative policy convergence and the final convergence reward.

[0016] Furthermore, this invention introduces Jain's fairness index based on service coverage into the expected discount cumulative reward, so that the reward signal not only reflects the instantaneous performance of task latency and energy consumption, but also covers the long-term balance of historical service coverage of each user. This drives the drone swarm to prioritize service to long-neglected ground users, thereby fundamentally eliminating service coverage gaps from the optimization objective level and achieving long-term fairness guarantees among users.

[0017] Furthermore, this invention uses whether the node spacing is less than or equal to the communication radius as the criterion for constructing and updating node-to-edge connections. By injecting physical communication reachability as a hard constraint into the updating process of the geometric graph topology, this invention ensures that subsequent attention calculations are performed only between physically reachable nodes. This blocks the inflow of information from out-of-distance invalid nodes from the source of the graph structure, reduces computational redundancy, and improves the physical reliability of the topology representation.

[0018] Furthermore, this invention constructs a policy network using a neural network algorithm, taking real-time observation parameters of nodes as input and node actions as output, thereby achieving decentralized distributed execution. This enables each UAV to independently generate actions by relying only on local observations during the execution phase, effectively reducing the system's dependence on global communication and ensuring the real-time performance and robustness of decision-making.

[0019] Furthermore, this invention obtains standard attention coefficients by introducing normalization processing after scaling the dot product calculation, so that the sum of the attention weights of the same node for all neighboring nodes is 1, which ensures the relative comparability and scale consistency of the contributions of each neighbor during feature aggregation, and avoids the training instability problem caused by the fluctuation of the original attention coefficient numerical range.

[0020] Furthermore, this invention obtains the real-time observation parameters of each node in the next time step by inputting the current node's action into the air-ground collaborative model, and determines the target joint vector accordingly. This enables the autonomous generation of the target joint vector in the experience replay pool, ensuring the integrity and automatic closure of the state data of the next time step required for the calculation of the time-series differential target value.

[0021] Furthermore, this invention explores noise by imposing constraints on the original actions of nodes during the target action determination process and restricting the absolute value of the noise components to within the out-of-bounds truncation threshold. This ensures the sufficiency of the policy network's exploration during training and prevents actions from going out of bounds or policies from oscillating violently due to extreme noise, thereby improving the smoothness and safety of the training process.

[0022] Furthermore, this invention constructs an evaluation network using a neural network algorithm and constructs a target evaluation network using a copy of the network. It uses the mean square error between the temporal difference target and the benchmark value score as the loss scalar for backpropagation updates, so that the evaluation network always uses the stable output of the target network as a reference during training. This effectively suppresses the accumulation of bias in the value function prediction and improves the accuracy of the joint strategy evaluation.

[0023] Furthermore, this invention explicitly defines the benchmark node action as a joint action vector generated by the policy networks of each node based on real-time observation parameters at the current time step, and inputs it together with the benchmark joint vector into the evaluation network to determine the benchmark value score. This ensures that the current value score is aligned with the time step used in the temporal difference target calculation during training, and avoids the value function estimation deviation caused by the mismatch between action and state time.

[0024] Furthermore, this invention uses the condition of triggering policy network updates by having the number of evaluation network updates reach a network update threshold, and employs a delayed update mechanism to ensure that the evaluation network is fully optimized before the policy network is updated. This provides the policy network with a more accurate gradient direction, effectively suppressing the value estimation bias and policy oscillation problems caused by the synchronous updates of the evaluation network and the policy network, and further enhancing the convergence stability of the collaborative decision-making algorithm. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating the workflow of the UAV collaborative computing method based on graph attention mechanism in this embodiment of the invention. Figure 2 This is a schematic diagram of the execution architecture and core processing flow of the UAV collaborative computing method based on graph attention mechanism in an embodiment of the present invention; Figure 3 This is a simulation diagram of the initial stage of UAV collaborative computing in Embodiment 1 of the present invention. Figure 4This is a simulation diagram of the collaborative computing effect of UAVs during the learning phase of Embodiment 1 of the present invention; Figure 5 This is a simulation diagram of the UAV collaborative computing effect during the convergence phase of Embodiment 1 of the present invention; Figure 6 The graph shows a comparison of the reward convergence curves and baseline algorithms of Embodiment 1 and Comparative Examples 1-3 during the training process. Detailed Implementation

[0026] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0027] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0028] Please see Figure 2 As shown, this invention provides a UAV cooperative computing method based on a graph attention mechanism, comprising: Taking UAVs and ground equipment as entities, a decentralized partially observable Markov decision process framework is used to transform the UAV collaborative process. The global optimization objective is to maximize the expected discount cumulative reward under the joint strategy. The expected discount cumulative reward is determined based on several expected discount cumulative rewards. A geometric graph is constructed using the aforementioned UAV as nodes, and the set of edges connecting nodes in the geometric graph is periodically updated based on the node spacing. Based on the pre-built policy network, node actions are generated using real-time observation parameters of nodes. Concatenated feature vectors are constructed based on the real-time observation parameters and node actions. Several query vectors, key vectors, and value vectors for each node are determined based on the concatenated feature vectors. Standard attention coefficients for each node to the connected edges are determined based on the query vectors and key vectors. Each node determines several node branch feature vectors based on the standard attention coefficient and the value vector, and determines the total node feature vector based on the node branch feature vectors, and determines the benchmark joint vector based on the total feature vectors of each node; The target joint vector is determined based on the current node actions. The target action is determined based on the target joint vector and the policy network with injected noise. Several evaluation networks are instantiated. The target joint vector and the target action are input into the target evaluation network to determine the target value score. The temporal difference target is determined based on the target value score and the expected discount cumulative reward. The benchmark value score is determined based on the benchmark joint vector and the benchmark node action. The loss scalar is determined based on the temporal difference objective and the benchmark value score. The evaluation network is updated by backpropagation based on the loss scalar. The baseline node action is determined based on the network update threshold. The target value score is determined based on the baseline node action and the baseline joint vector. After taking the partial derivative of the target value score, the chain rule is used to determine the policy network update gradient and update the policy network parameters.

[0029] It is understood that the expected cumulative discount reward is the reward for the current time step and future time steps, and the ground equipment refers to ground user equipment, including but not limited to mobile phones, computers, etc., which will not be elaborated here; Please see Figure 1 As shown, it is a flowchart of the UAV cooperative computing method based on graph attention mechanism in an embodiment of the present invention, including: S1: Taking drones, ground equipment, and edge computing service nodes as entities, a decentralized partially observable Markov decision process framework is used to characterize the drone collaborative computing process. An air-ground collaborative model is constructed with the goal of maximizing the expected discount cumulative reward under the joint strategy. S2: Construct a geometric graph using the UAV as nodes, and periodically update the node pair connection set of the geometric graph based on the node spacing; S3: Based on the initialized policy network, node actions are generated using real-time observation parameters of nodes. Concatenated feature vectors are constructed based on the real-time observation parameters of nodes and node actions. Several query vectors, key vectors, and value vectors of each node are determined based on the concatenated feature vectors. Attention coefficients of candidate node pairs are determined based on the query vectors and key vectors. The standard attention coefficients of the edges connecting the nodes are obtained by combining the dynamic space mask. S4: Each node determines several node branch feature vectors based on the standard attention coefficient and the value vector, and determines the total node feature vector based on the node branch feature vectors, and determines the benchmark joint vector based on the total node feature vectors; S5: Determine the target joint vector based on the current node action, determine the target action based on the target joint vector and the target policy network with injected noise, instantiate at least two target evaluation networks, input the target joint vector and target action into each target evaluation network respectively, select the smaller target value score and combine it with the expected discount to accumulate the reward to determine the temporal difference target; S6: Input the benchmark joint vector and benchmark node actions into at least two evaluation networks to determine the corresponding benchmark value scores, determine the loss scalar based on the temporal difference objective and each benchmark value score, and update the corresponding evaluation network based on the loss scalar through backpropagation. S7: Determine the delayed update timing of the policy network based on the network update threshold, obtain the baseline node action, determine the target value score for policy update based on the baseline node action and the baseline joint vector, and use the chain rule to determine the policy network update gradient and update the policy network parameters after taking the partial derivative of the target value score with respect to the node action.

[0030] Specifically, the formula in this embodiment, which takes maximizing the expected cumulative reward under the joint strategy as the global optimization objective, includes: ; in, This represents the maximum expected cumulative reward under the joint strategy π. This indicates that the expected value of the π-induced trajectory is obtained according to strategy π, where γ is the discount factor. Accumulated rewards for expected discounts at time step t; It is understood that the expected cumulative discount reward is determined based on Jain's Fairness Index of service coverage, service latency, energy consumption, and penalties for constraint violations. Please refer to Expected Cumulative Discount Rewards. Calculation formula: ; in, Jain's fairness index, based on service coverage, is used to characterize the degree of equitable access to qualified services for different ground users. For user k, the service latency under the current service mode, For the flight and calculation of energy consumption of the UAV u; For delayed weights, As energy consumption weight, For fairness weights, all three values ​​are between 0 and 1 and satisfy the following conditions: ε is the logarithmic smoothing constant, with a value range of 1. to ; Penalties for violations of physical constraints such as boundary crossing and collision risks, with values ​​ranging from 0 to... .

[0031] Among them, service coverage The ratio of the number of qualified communication time slots for user k within the monitoring duration t to the monitoring duration t is used to determine the coverage rate. Please refer to Service Coverage. Calculation formula: ;

[0032] in, The number of time slots in which user k successfully communicates within a monitoring duration t is defined as the number of time slots in which user k completes access, transmission, and calculation within the corresponding time slot and meets the delay constraint. This is recorded as a successful communication.

[0033] It is understood that the air-ground cooperative model also includes a communication model, a task calculation model, a kinematic model, an energy consumption model, and a physical constraint model; The communication model configuration method is an existing technology. It can configure the channel power gain between the UAV and the ground equipment based on the free space path loss model, and configure the uplink data wireless transmission rate of the ground equipment based on the Shannon channel capacity formula. This embodiment does not impose any restrictions, as long as it can ensure that the path loss of the wireless link can be determined according to the spatial distance between the UAV and the ground equipment, and the uplink transmission rate can be determined based on the available spectrum resources and channel quality, so as to serve as the basic input for task scheduling delay calculation. The free space path loss model is as follows: ; in, For time step drones With ground users Channel power gain between The fundamental channel gain constant at the reference distance. Let be the three-dimensional spatial coordinate vector of the UAV. For the three-dimensional spatial coordinate vector of the ground user, The extremely small positive number is introduced to prevent the abnormal denominator from being zero; The Shannon channel capacity formula is: ; in, For users To drones The effective uplink transmission rate, The dedicated frequency bandwidth allocated to this associated user for the drone. Transmit power to the user terminal's hardware. The power distribution of ambient additive white Gaussian noise; The task computation model includes a local proximity service strategy, a dynamic cooperative forwarding strategy for adjacent UAVs, and a core network backhaul offload strategy. When a UAV hits the requested content locally, the service latency is determined based on the wireless access transmission latency and the local computation processing latency. The total latency formula for the local service strategy is: ; Total latency for local service strategy The scalar size of the data packets generated for the task. The number of CPU cycles required to complete this task. The computing processing frequency dynamically allocated to the drone;

[0034] When a drone triggers the neighboring drone dynamic cooperative forwarding strategy, the service latency is determined based on the wireless access transmission latency, relay transmission latency, and neighbor calculation and processing latency. The total latency formula for the neighbor cooperative strategy is: ; in, The total delay for the neighbor collaboration strategy, For the initiating drone With collaborative drones The air communication transmission rate between them The processing frequency allocated to the collaborative drone; When a drone triggers the core network backhaul offload policy, the service latency is determined based on the wireless access transmission latency and the wireless backhaul transmission latency. The total backhaul offload latency formula is: ; in, To account for the total unloading delay on the return trip, This refers to the backhaul communication link rate between the UAV and the ground macro base station.

[0035] In the kinematic model, the three-dimensional position update of the UAV is constrained, and the UAV position vector at the next time step... The formula is: = ; in, This represents the actual three-dimensional displacement vector for the current decision-making cycle. The maximum physical flight speed limit for drones, The duration of a single discrete decision time step; The energy consumption model configuration method is an existing technology and can be built based on the aerodynamics and propulsion power model of the rotary-wing UAV. This embodiment does not impose any restrictions, as long as it can ensure that the single-step flight energy consumption can be determined according to the UAV's flight status and cruise speed, and the single-step calculation energy consumption can be determined according to the processor's working frequency, so as to support the calculation of the energy consumption item in the subsequent reward function. The aerodynamic and propulsion power models of the rotary-wing UAV are as follows: ; in, Total energy consumed for single-step flight This represents the actual three-dimensional displacement vector for the current decision-making cycle. This refers to the rated power of the drone while it is maneuvering forward. This refers to the rated power of the drone in a stationary hovering state. This refers to the average cruising speed of the drone; The physical constraint model is configured such that the distance between any two drones must be greater than or equal to the obstacle avoidance Euclidean distance threshold. Specifically, the node spacing is determined based on the physical Euclidean distance between node pairs; Specifically, when periodically updating the set of edges connecting node pairs in the geometry graph based on node spacing: Based on the analysis condition that the node spacing is less than or equal to the communication radius, it is determined to construct node pair edges between node pairs and update the set of node pair edges. Based on the analysis condition that the node spacing is greater than the communication radius, it is determined to continue traversing the node pairs. It is understood that the communication radius is selected from the maximum effective communication sensing radius threshold supported by the airborne radio frequency hardware, and the value range is 100m to 500m; in this embodiment, the communication radius is 300m to be consistent with the reliable communication and image mask calculation range between UAVs.

[0036] like Figure 2 As shown, this scheme adopts a centralized training and distributed execution architecture, including a distributed execution module, an experience replay pool, and a centralized value evaluation and parameter update module. The distributed execution module consists of a policy network deployed by each UAV, and the centralized value evaluation and parameter update module includes a feature encoding layer, a dynamic spatial masking layer, a multi-head graph attention aggregation layer, a dual evaluation network, and a target evaluation network.

[0037] The policy network is constructed based on a neural network algorithm, taking real-time observation parameters of nodes as input and node actions as output; the node actions include the maneuver control quantities of the UAV in the continuous action space, and are used to further determine the results of task unloading and cooperative forwarding.

[0038] It is understood that the policy network does not require pre-training; in the initial stage, after the policy network parameters are randomly initialized, forward inference is directly performed on the real-time observation parameters of the nodes to generate node actions that satisfy the action boundaries. After the node action interacts with the environment, it forms a sample and is stored in the experience replay pool. Subsequently, the policy network parameters are updated in reverse using the evaluation network loss and delayed policy gradient of this scheme.

[0039] It is understandable that when constructing a concatenated feature vector based on the real-time observation parameters of the node and the node actions, the construction method is to concatenate and splice the real-time observation parameters of the node and the node actions into features. When mapping cascaded feature vectors to high-dimensional feature embedding vectors, the high-dimensional feature embedding vector of node i... for: ; in, For nodes The high-dimensional feature embedding vector, For cascaded feature vectors, The learnable weight matrix for the feature encoding network layer of a multilayer perceptron. For the corresponding bias parameter vector, This is the activation function used to introduce nonlinear spatial mapping; The method of determining several query vectors, key vectors and value vectors of each node based on cascaded feature vectors is an existing technology. It can be determined based on linear projection transformation. This embodiment does not impose any restrictions, as long as the query vectors, key vectors and value vectors can reflect the representation of the same node features in different semantic spaces, and the dimensions of the query vectors and key vectors are consistent to support subsequent scaling dot product operations. The formula for determining the query vector, key vector, and value vector based on linear projection transformation is as follows: ; in, Let i be the query vector used to retrieve associated features. Let be the key vector used for matching and searching candidate neighbor node j. is the value vector used for weighted aggregation of candidate neighbor node j; candidate neighbor node refers to other nodes that enter the candidate set when performing attention calculation with node i as the center. Candidate neighbor nodes within the communication radius are determined to be valid neighbor nodes by dynamic spatial masking. These are the independent learnable parameter matrices corresponding to the three linear mapping transformation mechanisms.

[0040] The formula for determining the attention coefficient based on the scaled dot product of the query vector and the key vector is as follows: ; in, The original attention coefficients between node i and node j. The dimensions of the query vector and key vector in a single attention head; when the model's hidden dimension is... The number of attention heads is hour, Desirable In this embodiment, Take positive integers from 1 to 8. Use positive integers from 8 to 128 to scale the dot product results and suppress excessive numerical variance caused by high-dimensional inner products.

[0041] Understandably, the node-to-edge set is used to describe the physically communicable links at the current moment. Since the evaluation network still uses a fixed-length joint vector of all nodes as input during batch training, the candidate node set contains placeholder or distant node information corresponding to physically unreachable nodes. Therefore, it is necessary to apply hard truncation intervention to the physically unreachable connection path at the neural network computation level based on the dynamic spatial mask matrix.

[0042] ; in, Let be the value of the element in the i-th row and j-th column of the dynamic space mask matrix; when the distance between node i and node j does not exceed the communication radius, it is 0 and allowed to participate in attention normalization; when the distance between them exceeds the communication radius, it is negative infinity, so that the corresponding attention weight approaches 0 after Softmax normalization.

[0043] After normalizing the attention coefficients, the standard attention coefficients for each node pair connection are obtained. The normalization formula is as follows: ; in, The standard attention coefficient is a strictly positive real number, and the sum of the weights of the same node with respect to all its neighboring nodes is 1. It is understandable that the specific process by which the node determines several node branch feature vectors based on the standard attention coefficient and the value vector includes: Each node performs a weighted summation of the neighbor node value vectors based on the standard attention coefficients of each node's connections, determining several node branch feature vectors to fuse node physical topology awareness features with effective information filtering features. The formula is: Node branch feature vector ; The process of determining the total eigenvector of a node based on its branch eigenvectors includes: The total node feature vector is obtained by integrating the feature vectors of each node branch based on the projection matrix.

[0044] ; in, For the first Each attention head produces a branch feature vector. This is the linear output projection matrix used for feature dimension reduction and integration; The process of determining the benchmark joint vector based on the total eigenvector of each node includes: The total feature vectors of the nodes are directly concatenated to obtain the benchmark joint vector. ; ; Specifically, the process of determining the target joint vector includes: Based on the current node actions, an air-ground collaborative model is used to determine the real-time observation parameters of each node at the next time step. Based on the real-time observation parameters of each node, the reference joint vector for the next time step is determined, denoted as the target joint vector. .

[0045] Specifically, the process of determining the target action includes: The policy network generates original node actions based on the target joint vector, applies constraints to the original node actions to explore noise, and obtains the target actions. ; ; ; in, For the target action, For having an independent parameter set The policy network, To constrain exploration noise, The mechanism is a normal distribution with a mean of zero. A physical cutoff threshold is set to prevent extreme jitter. It is understood that the evaluation network is constructed based on a neural network algorithm, wherein the joint vector and the actions of each node are used as inputs, and the value score is used as the output. The target evaluation network is constructed based on a copy of the evaluation network; It is understood that the time-series difference objective is determined based on the sum of the target value score and the expected discount cumulative reward. The formula is: ; in, For the first A separate network of value critics. For the target joint vector, The target action has been smoothed. Specifically, the benchmark value score is determined based on the benchmark joint vector and the benchmark node action, which are respectively input into the first evaluation network and the second evaluation network. When updating the j-th evaluation network, the benchmark value score output by the evaluation network and the temporal difference objective are used to calculate the mean square error loss, and the two evaluation networks are updated independently through backpropagation.

[0046] The baseline node action refers to the joint action vector formed by each node generating node actions based on the real-time observation parameters and using the current policy network at the current time step; the target value score in the temporal difference objective selects the smaller value between the output results of the two target evaluation networks to suppress value overestimation.

[0047] It is understandable that this scheme determines the loss scalar based on the mean square error between the time-series difference target and the benchmark value score. The formula is: ; Where B is the total number of batch samples randomly drawn from the experience replay pool. For the j-th evaluation network in the update cycle; a single data sample in the experience replay pool includes at least the current joint observation, joint action, expected discount cumulative reward, next time-instance joint observation, termination flag, and node positions or node pair edge sets used to reconstruct the geometry and dynamic space mask.

[0048] Specifically, the process of determining the action of obtaining the baseline node based on the network update threshold includes: Based on the condition that the number of network updates is less than the network update threshold, it is determined that monitoring will continue. Based on the analysis condition that the number of network updates is greater than or equal to the network update threshold, the action to obtain the baseline node is determined. Understandably, the formula for determining the direction of the gradient vector update of the policy network using the chain rule after taking the partial derivative of the target value score is as follows: ; in, In order to target the The direction of the gradient vector for correcting network weights calculated by each agent. The bounds for the partial derivatives of the actions output by the first set of evaluation networks. Generate the Jacobian partial derivatives of the network's internal parameters with respect to a given local observation.

[0049] Example 1

[0050] This embodiment constructs a 1000m×1000m air-to-ground collaborative service area, with a macro base station at the center of the area serving as a backhaul gateway. Ten UAVs are deployed in the airspace, with a fixed flight altitude of 100m, a maximum flight speed of 15m / s, a minimum safe distance of 100m, and a communication sensing radius of 300m between UAVs. 200 user devices are deployed on the ground, their movement trajectories following a random waypoint model, and task requests following a Zipf distribution with a parameter of 0.6. Task data size ranges from 100KB to 500KB, and UAV computing power ranges from 5GHz to 20GHz. The training phase uses 1000 training rounds, each with 1000 time steps, and the experience replay pool capacity is [not specified]. The batch size is 64 samples, the discount factor is 0.99, and the learning rates for both the policy network and the evaluation network are 0.99. The number of attention heads is 2, the delayed update frequency is 2, and the soft update coefficient is 0.01.

[0051] Comparative Example 1 The simulation experimental environment, UAV and ground user parameters, reward function settings and physical constraints are the same as in Example 1. The only difference is that the critic network algorithm described in this scheme is replaced with the MATD3 algorithm.

[0052] Comparative Example 2 The simulation experimental environment, UAV and ground user parameters, reward function settings and physical constraints are the same as in Example 1. The only difference is that the critic network algorithm described in this scheme is replaced with the MASAC algorithm.

[0053] Comparative Example 3 The simulation experimental environment, UAV and ground user parameters, reward function settings and physical constraints are the same as in Example 1. The only difference is that the critic network algorithm described in this scheme is replaced with the MADDPG algorithm. Please see Figures 3 to 5 , Figure 3 The simulation results of UAV collaborative computing in the initial stage of Example 1 are shown. At this time, the UAVs are relatively dispersed, the service links and topology links have not yet formed a stable collaborative structure, and some ground users are still in an unserviced state. Figure 4 The simulation results during the learning phase are shown. As the reward function gradually takes effect on the joint constraints of service coverage, latency and energy consumption, the drone begins to move towards areas with high user demand and establish more communicable topology links, increasing the number of users served. Figure 5 The simulation results during the convergence phase are shown. The drone swarm forms a relatively stable mesh cooperative topology with a more balanced distribution of service links, enabling it to provide more continuous collaborative computing services to hotspot areas and edge users. This demonstrates that the present invention can drive the drone swarm to gradually evolve from an initial dispersed state to a load-balanced collaborative coverage state.

[0054] Please continue reading. Figure 6 , Figure 6 The average round reward convergence results during training for Example 1 and Comparative Examples 1 to 3 are presented. Figure 6 It can be seen that the reward curve of Example 1 maintains the highest level in the later stages of training and has good convergence stability. Compared with the MATD3 algorithm of Comparative Example 1, the average round reward of Example 1 is improved by about 37.1% in the convergence phase, indicating that after introducing dynamic spatial masking and graph attention value evaluation, the evaluation network can more accurately aggregate effective cooperative information within the reachable neighborhood. The reward curves of Comparative Examples 2 and 3 have lower convergence values ​​and more obvious fluctuations, indicating that without spatial topological constraints or a stable double-delay update mechanism, the algorithm has difficulty forming a stable cooperative computing strategy in dynamic UAV networks. Therefore, Example 1 is superior to all comparative examples in terms of convergence reward, training stability, and cooperative decision-making performance.

[0055] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A UAV cooperative computing method based on graph attention mechanism, characterized in that, include, Using drones, ground equipment, and edge computing service nodes as entities, a decentralized partially observable Markov decision process framework is adopted to characterize the drone collaborative computing process. An air-ground collaborative model is constructed with the goal of maximizing the expected discount cumulative reward under the joint strategy. A geometric graph is constructed using the aforementioned UAV as nodes, and the set of edges connecting nodes in the geometric graph is periodically updated based on the node spacing. Based on the initialized policy network, node actions are generated using real-time observation parameters. Concatenated feature vectors are constructed based on the real-time observation parameters and node actions. Several query vectors, key vectors, and value vectors for each node are determined based on the concatenated feature vectors. Attention coefficients for candidate node pairs are determined based on the query vectors and key vectors. The standard attention coefficients for the edges connecting the nodes are obtained by combining dynamic space masks. Each node determines several node branch feature vectors based on the standard attention coefficient and the value vector, and determines the total node feature vector based on the node branch feature vectors, and determines the benchmark joint vector based on the total node feature vectors; Based on the current node actions, a target joint vector is determined. Based on the target joint vector and the target policy network with injected noise, a target action is determined. At least two target evaluation networks are instantiated. The target joint vector and target action are input into each target evaluation network respectively. The smaller target value score is selected and combined with the expected discount to accumulate the reward to determine the temporal difference target. The benchmark joint vector and benchmark node actions are respectively input into at least two evaluation networks to determine the corresponding benchmark value scores. The loss scalar is determined based on the temporal difference objective and each benchmark value score, and the corresponding evaluation network is updated by backpropagation based on the loss scalar. The delayed update timing of the policy network is determined based on the network update threshold. The baseline node action is obtained. The target value score for policy update is determined based on the baseline node action and the baseline joint vector. The partial derivative of the target value score with respect to the node action is obtained and the chain rule is used to determine the policy network update gradient and update the policy network parameters.

2. The UAV cooperative computing method based on graph attention mechanism according to claim 1, characterized in that, The expected cumulative discount reward is determined based on Jain's Fairness Index of service coverage, service latency, energy consumption, and penalty for violation of constraints. The service coverage rate is determined based on the number of qualified communication time slots and the monitoring duration. The number of qualified communication time slots is determined based on the number of times a user's communication is qualified within the monitoring duration.

3. The UAV cooperative computing method based on graph attention mechanism according to claim 2, characterized in that, Based on the analysis condition that the node spacing is less than or equal to the communication radius, it is determined to construct node pair edges between node pairs and update the node pair edge set.

4. The UAV cooperative computing method based on graph attention mechanism according to claim 3, characterized in that, The policy network is constructed based on a neural network algorithm, and the real-time observation parameters of the nodes are used as inputs, while the node actions are used as outputs.

5. The UAV cooperative computing method based on graph attention mechanism according to claim 4, characterized in that, Attention coefficients are determined based on the scaled dot product of the query vector and the key vector, and then normalized to obtain the standard attention coefficients for each node pair connection.

6. The UAV cooperative computing method based on graph attention mechanism according to claim 5, characterized in that, Based on the current node actions, the air-ground collaborative model is used to determine the real-time observation parameters of each node at the next time step. Based on the real-time observation parameters of each node, the reference joint vector for the next time step is determined, denoted as the target joint vector.

7. The UAV cooperative computing method based on graph attention mechanism according to claim 6, characterized in that, The process of determining the target action includes: The policy network generates the original actions of nodes based on the target joint vector, applies constraints to the original actions of nodes to explore noise, and obtains the target actions; Wherein, the constrained exploration noise refers to a random vector whose absolute value is less than or equal to the out-of-bounds truncation threshold and which follows a Gaussian distribution with a mean of zero.

8. The UAV cooperative computing method based on graph attention mechanism according to claim 7, characterized in that, The evaluation network is constructed based on a neural network algorithm, and the value score is used as the output based on the joint vector and the actions of each node as input. The loss scalar is determined based on the mean square error between the temporal difference objective and the benchmark value score; Construct a target evaluation network based on a copy of the evaluation network.

9. The UAV cooperative computing method based on graph attention mechanism according to claim 8, characterized in that, The benchmark value score is determined by the evaluation network based on the benchmark joint vector and the benchmark node action input; The loss scalar is determined based on the loss scalar of the time-series difference target and the benchmark value score; The reference node action refers to the joint action vector generated by each node based on the real-time observation parameters and using the policy network at the current time step.

10. The UAV cooperative computing method based on graph attention mechanism according to claim 9, characterized in that, Based on the condition that the number of network updates is greater than or equal to the network update threshold, the update strategy network is determined and the actions of the baseline node are obtained.

Images