A multi-unmanned aerial vehicle reinforcement learning cooperative scheduling method based on cross-layer perception and prior action mask

Abstract

This invention relates to the field of UAV networking communication and edge computing technology, specifically a multi-UAV reinforcement learning collaborative scheduling method based on cross-layer perception and prior action masking. This invention, by opening up the interaction interface of the simulation engine, enables the aggregation and flow of high-dimensional state vectors such as high-frequency time-varying physical channel states, three-dimensional topology, and application-layer queue depth without delay. This cross-layer perception capability breaks down the data barriers between lower-level communication and upper-level decision-making, significantly enhancing the system's perception accuracy in complex mountainous or dynamically constrained environments. This invention creatively introduces a prior action masking mechanism and a multi-objective reward function based on ladder shaping. Before action output, physically unreachable nodes are accurately eliminated, completely avoiding network congestion and link breakage caused by blind unloading in traditional algorithms; simultaneously, the multi-objective reward function effectively balances the contradiction between extremely low response time and high resource utilization, significantly improving the robustness of cluster collaboration.

Description

Technical Field

[0001] This invention relates to the field of UAV networking communication and edge computing technology, specifically to a multi-UAV reinforcement learning collaborative scheduling method based on cross-layer perception and prior action masking. Background Technology

[0002] Unmanned aerial vehicle (UAV) swarms, operating in a distributed and collaborative manner, undertake a large number of edge computing tasks and are widely used in various fields such as disaster relief, environmental monitoring, and complex geographical reconnaissance. However, with the exponential growth of data volume from sudden missions, the computing power expansion speed of a single UAV cannot keep up with the growth in computing demands, resulting in resource-constrained and locally overloaded state in the swarm edge computing network. Therefore, utilizing intelligent computing offloading and resource scheduling technologies to achieve elastic allocation of heterogeneous computing power is of significant research importance.

[0003] Existing deep reinforcement learning-based UAV swarm scheduling technologies iteratively aggregate state information from the UAV network, learning the feature embedding representations of nodes and links in the swarm. This allows the agent to learn its local load information and the overall computing power distribution structure of the swarm. Deep reinforcement learning-based methods often employ an architecture similar to "state perception-action decision-making," using a neural network structure as a perception encoder to generate a fused state representation that includes task requirements and global resources. After obtaining the updated global state representation, an evaluation network is used to score candidate scheduling actions, and the temporal difference loss value of the current batch training is calculated. Backpropagation is used to update the training parameters of the neural network layers, and the optimal scheduling strategy is learned through multiple environmental interactions. Methods based on cross-layer optimization further combine application layer requirements and network layer states to selectively retain important communication bottleneck information during the aggregation of state information.

[0004] However, existing methods typically have two problems: the perception module does not take into account the underlying time-varying physical channel fading and heterogeneous communication connections when aggregating key neighborhood state information, and the decision module lacks a priori constraint mechanism based on physical red lines, making it difficult to accurately remove physically unreachable nodes caused by radio reception sensitivity or routing hop count limitations when outputting unloading commands. This can easily lead to blind unloading and network deadlock, so the convergence stability and global scheduling effect of the multi-UAV cooperative scheduling model still need to be improved. Summary of the Invention

[0005] The technical problem to be solved by this invention is to propose a multi-UAV reinforcement learning collaborative scheduling method based on cross-layer perception and prior action masking, which makes up for the shortcomings of existing methods that do not take into account the physical layer constraints of heterogeneous networks and are difficult to simultaneously take into account multiple objectives, so as to better complete the UAV swarm resource scheduling task.

[0006] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0007] A multi-UAV reinforcement learning cooperative scheduling method based on cross-layer perception and prior action masking includes the following steps:

[0008] S1. Through the cross-layer data interaction interface, cross-layer feature extraction and distributed state synchronization are performed on the heterogeneous cluster of multi-UAVs to be scheduled to obtain a global initial composite state representation that characterizes the network load and communication topology.

[0009] S2. Using a priori masking mechanism, based on the underlying radio signal fading model, receiver sensitivity threshold, and routing protocol limitations, the physical connectivity between the source UAV node and other candidate target UAV nodes in the cluster is calculated in real time to generate a dynamic priori action mask matrix.

[0010] S3, the decision filtering module aggregates the original action space of the heterogeneous drone cluster with the prior action mask matrix using the Hadamard product, forces the selection probability of target drone nodes that are physically unreachable or have limited routes to be zero, and generates a safe and feasible subset of candidate decision actions.

[0011] S4. The effective composite observation state after masking and the subset of candidate decision actions are fed into the evaluation network of deep reinforcement learning to extract the multi-dimensional interaction features of the state and action, and output the expected value evaluation of the candidate action. Then the system executes the scheduling action and uses a multi-objective joint reward function that takes into account latency, energy consumption and packet loss penalty to calculate and obtain the multi-objective joint score of the action.

[0012] S5. Calculate the temporal difference loss value of the reinforcement learning model using the loss function, and train the scheduling model based on the loss value using the gradient descent method to optimize the network parameters and obtain the trained scheduling model.

[0013] S6. The trained scheduling model is used for actual edge computing offloading tasks. The concurrent tasks of the source UAV and candidate nodes are combined to form a candidate scheduling scheme. Then, the scores of all candidate schemes are calculated by inference network. Finally, the highest score is selected as the offloading target and the underlying physical network card and protocol stack are driven to complete the routing link establishment.

[0014] Furthermore, the cross-layer features include the UAV's local application layer task features and local hardware attributes; the distributed state synchronization specifically involves fusing and splicing the cross-layer features of the source UAV with the heterogeneous information of its neighbor set in the feature dimension to construct a composite observation state vector representing the heterogeneous connection relationship of the local neighborhood.

[0015] Furthermore, the prior action mask matrix is ​​constructed as follows:

[0016] Based on the currently acquired 3D coordinates, calculate the source drone node. With candidate target drone nodes Euclidean distance between Based on free-space path loss or log-distance path loss models, the received signal strength transmitted from the source UAV node to the candidate target UAV is predicted. ;

[0017] Given the minimum receive sensitivity threshold required for hardware decoding of a low-level network interface card (NIC). and the maximum number of hops allowed by network layer routing protocols ,use Indicates the source drone node To candidate target drone node The actual number of hops between them is used to determine connectivity using a mask indicator function. and Time link connectivity, target drone node Prior action mask value ,otherwise , indicating that it is physically unreachable.

[0018] Furthermore, the specific construction process of the candidate decision action subset is as follows:

[0019] Assume the original scheduling probability distribution of each UAV node output by the deep reinforcement learning network is as follows: Combine it with the generated mask vector Perform Hadamard product aggregation mapping operations and filter out invalid actions;

[0020] When removing unreachable node information, a mask-based safety constraint mechanism is added, and the updated action selection space is represented as follows:

[0021]

[0022] in This indicates that the source drone node is processing this task locally.

[0023] Furthermore, the multi-objective joint score is calculated as follows:

[0024] Calculate the total end-to-end latency cost of executing the action. It covers source node queue waiting latency, wireless transmission latency, and target drone node processing latency;

[0025] Calculate the total energy cost of the system in executing the action. This includes the communication transmission energy consumption of the source node and the processor computing energy consumption of the target node;

[0026] The multi-objective joint score is as follows:

[0027]

[0028] in, , , For learnable normalized weight allocation coefficients, This represents an extreme constant penalty imposed for link disconnection or queue overflow.

[0029] Furthermore, S5 is specifically as follows:

[0030] S51. The multi-agent deep deterministic policy gradient algorithm is adopted to divide the scheduling model into an evaluation network responsible for scoring and a policy network responsible for making decisions. During the training phase, the local state, actions, multi-target joint scores and the state at the next moment of all UAVs are collected. This information is merged into global state and global actions and stored in a unified experience replay pool.

[0031] S52. Training a centralized evaluation network: Extract a batch of data from the experience replay pool and use the Bellman equation to calculate the expected future returns;

[0032] S53. The model is trained using a small-batch experience replay strategy. The error between the current network prediction value and the target value of all candidate actions is calculated. The scheduling model is trained using the mean squared error loss function. The evaluation network and policy network are trained alternately until the model converges, and the optimal local scheduling strategy for each UAV is obtained.

[0033] The beneficial effects of this invention are:

[0034] 1) Addressing the issue that most existing reinforcement learning-based UAV scheduling models neglect cross-layer interaction, this invention enables the seamless aggregation and transfer of high-dimensional state vectors, such as high-frequency time-varying physical channel states, three-dimensional topology, and application-layer queue depth, by establishing an interactive interface with the simulation engine. This cross-layer perception capability breaks down the data barriers between lower-level communication and upper-level decision-making, significantly enhancing the system's perception accuracy in complex mountainous or dynamically constrained environments.

[0035] 2) To address the problem that models are prone to getting trapped in local optima or deadlocks under rigid multi-objective constraints, this invention creatively introduces a prior action masking mechanism and a multi-objective reward function based on ladder shaping. Physically unreachable nodes are precisely eliminated before action output, completely avoiding network congestion and link breakage caused by blind unloading in traditional algorithms. Simultaneously, the multi-objective reward function effectively balances the contradiction between extremely low response time and high resource utilization, significantly improving the robustness of cluster collaboration. Attached Figure Description

[0036] Figure 1This is an overall flowchart of the multi-UAV reinforcement learning collaborative scheduling method in this invention;

[0037] Figure 2 This is a schematic diagram illustrating the architecture principle of unified modeling and multi-objective scheduling of heterogeneous resources in this invention.

[0038] Figure 3 This is a schematic diagram illustrating the scenario of dynamic scheduling of multiple heterogeneous UAV clusters according to the present invention. Detailed Implementation

[0039] This invention aims to propose a multi-UAV reinforcement learning cooperative scheduling method based on cross-layer perception and prior action masks, overcoming the shortcomings of existing methods that do not consider the physical layer constraints of heterogeneous networks and are difficult to simultaneously achieve multi-objective optimization, thus enabling it to better complete UAV swarm resource scheduling tasks. Its overall implementation process includes: S1. For the multi-UAV swarm to be scheduled, extract application layer task features and underlying physical topology through a cross-layer interface to obtain a composite initial state representation of the entities; S2. A connectivity evaluation module calculates the physical connectivity between source entities and candidate target entities based on the underlying radio signal attenuation model and routing hop count limits; S3. A decision filtering module generates a prior action mask based on the decision results obtained from the connectivity calculation, and performs Hadamard product aggregation with the original action space to eliminate the action probabilities of physically unreachable nodes, thereby generating a safe and feasible subset of decision actions, while updating the local topology state representation of the environment; S4. Provide the updated state representation and action subset to... The system features are used to extract interaction features through a multilayer perceptron network. Then, the end-to-end latency, energy consumption, and packet loss rate of the scheduling action are calculated to obtain a multi-objective joint score for the action. S5: The temporal difference loss value of the reinforcement learning model is calculated through the loss function, and the multi-agent scheduling model is trained based on the loss value using gradient descent to optimize the network parameters and obtain a trained scheduling model. S6: The trained UAV scheduling model is used for actual edge computing offloading tasks. The concurrent tasks of the source UAV and candidate nodes are combined to form candidate scheduling schemes. Then, the scores of all candidate schemes are calculated through the inference network. Finally, the highest score is selected as the offloading target and drives the underlying chain construction.

[0040] Example:

[0041] The present invention will now be described in detail. See below. Figure 1 The specific implementation process is as follows:

[0042] S1. Unified modeling of heterogeneous resources, state interaction, and cross-layer composite state extraction: For complex physical environments and high-frequency concurrent task flows, the system extracts state at each decision-making step. Cross-layer feature extraction and distributed state synchronization are performed on a heterogeneous cluster of multi-UAVs to be scheduled through a cross-layer data interaction interface. The cross-layer features include bottom-layer physical network features and upper-layer task queue features. The distributed state synchronization specifically splices and aggregates multi-dimensional heterogeneous features to obtain a global initial composite state representation that characterizes the overall network load and communication topology.

[0043] S11. Extract local application layer task features: Obtain the concurrent task attributes of the source drone node itself, including the size of the task data. Required CPU computation cycles Concurrent queue depth and remaining computing resource capacity The gradients are then normalized to stabilize the network gradients.

[0044] S12. Extracting Local Heterogeneous Hardware and Underlying Topology Features: At the physical and network layers, through a customized cross-layer interface, high-frequency data synchronization is performed with the underlying simulation engine to obtain its absolute coordinates in the 3D topology. And the time-varying channel signal-to-noise ratio between nodes. Simultaneously, the system reads its own hardware capability tags, including maximum transmit power. Current computing power And the corresponding chip energy consumption structure coefficient Because the cluster consists of heterogeneous nodes (high-altitude nodes have strong computing power but high power consumption, while low-altitude nodes have weak computing power but low communication power consumption), the above parameters vary significantly among different nodes.

[0045] S13. Distributed Heterogeneous State Interaction and Combination: Each UAV node acts as an independent intelligent agent, periodically listening to and receiving real-time coordinates, signal-to-noise ratio, load status, and heterogeneous computing power and power consumption parameters broadcast by neighboring UAVs through an underlying self-organizing network. (Source UAV) The obtained neighbor set Heterogeneous information and its own state are fused and spliced ​​together along the feature dimension to construct a time step. The high-dimensional composite observation state vector representing the heterogeneous connectivity of local neighborhoods Its mathematical expression is:

[0046]

[0047] Among them, the source drone node Local state vector Includes application-layer task characteristics and local hardware attributes:

[0048]

[0049] Sensing any neighboring drone node State feature vector Includes spatial topology, channel quality, and heterogeneous attributes:

[0050]

[0051] in Indicates the source drone node with neighboring nodes To improve the signal-to-noise ratio of communication between neighbors, and to adapt to the fixed input dimension requirements of the neural network, the system adopts a zero-padding strategy for the neighbor state feature set, aligning the number of neighbors to the maximum number of neighbors set by the system.

[0052] S2. Connectivity Assessment Module: Due to the highly dynamic heterogeneous physical connections between nodes in a drone swarm, where a source node may experience communication link breaks due to terrain obstruction or excessive distance, or deadlock due to excessive network layer routing hops, a priori masking mechanism is used. Based on the underlying radio signal fading model, receiver sensitivity threshold, and routing protocol limitations, the physical connectivity between the source drone node and other candidate target drone nodes in the swarm is calculated in real time, generating a dynamic priori action mask matrix. In this process, the 3D coordinates of all entities are input, and the output is a Boolean mask score for all target entities, as detailed below:

[0053] S21. First, predict the received signal strength of the target entity based on the free-space large-scale fading model: Calculate the source UAV node's signal strength based on the currently acquired 3D coordinates. With candidate target drone nodes Euclidean distance between The calculation process is as follows:

[0054]

[0055]

[0056] in Indicates free space path loss. Indicates the radio carrier frequency, Indicates the source node The hardware transmission power, where 20 and 147.55 in the formula are physical constants derived based on the speed of light. Candidate target nodes The predicted received signal strength;

[0057] S22. Given the minimum receiver sensitivity threshold required for hardware decoding of a low-level network interface card. And the maximum hop limit allowed by the Ad Hoc On-Demand Distance Vector (AODV) routing protocol in ad hoc networks at the network layer. ,use Indicates the source drone node To candidate target drone node The actual number of hops between them is used to determine connectivity using a mask indicator function. and Time link connectivity, target drone node Prior action mask value ,otherwise It indicates that it is physically unreachable.

[0058] S3, the decision filtering module aggregates the original massive action space of the reinforcement learning agent with the prior action mask matrix using the Hadamard product, forces the selection probability of target drone nodes that are physically unreachable or have limited routes to be zero, eliminates invalid unloading actions from the decision source, and generates a safe and feasible subset of candidate decision actions.

[0059] S31. Assume the original scheduling probability distribution of each UAV node output by the deep reinforcement learning network is as follows: Combine it with the generated mask vector Perform the Hadamard product aggregation mapping operation and filter out invalid actions. The process is as follows:

[0060]

[0061] in It is a subset vector of safe and valid actions after being constrained by the aggregation mask. This operation represents element-wise multiplication. It directly resets the weight of unreachable entities to 0, which is equivalent to forcibly imposing environmental boundary restrictions before the agent tries to make a mistake.

[0062] S32. Residual Connections and State Updates: When removing unreachable node information, a mask-based backup constraint mechanism is added to preserve the local processing options of the source UAV node. The updated action selection space is represented as follows:

[0063]

[0064] in This indicates that the source drone node is processing this task locally.

[0065] The principle of heterogeneous resource unified modeling and action set construction in steps S2-S3 above is as follows: Figure 2 As shown.

[0066] S4. Multi-objective scoring of the evaluator: The effective composite observation state after masking and the subset of candidate decision actions are jointly fed into the evaluation network of deep reinforcement learning. The evaluation network extracts the multi-dimensional interaction features of the state and action through a multi-layer perceptron network and outputs the expected value assessment of the candidate action. Then the system executes the scheduling action and uses a multi-objective joint reward function that takes into account latency, energy consumption and packet loss penalty to calculate the comprehensive environmental feedback reward of the action, thereby guiding the parameter update of the subsequent evaluation network.

[0067] S41. Same-Dimensional Delay Feature Extraction: The total end-to-end delay cost incurred by the system performing this action is as follows:

[0068]

[0069] in This is the total delay feature map. The first term represents the queue waiting delay, the second term represents the wireless transmission delay under Shannon's theorem, and the third term represents the target drone node. Based on its own computing power The computational delay;

[0070] S42. Interactive Extraction of Heterogeneous Energy Consumption Features: The total energy cost incurred by the system in performing this action is as follows:

[0071]

[0072] in This indicates the heterogeneous energy consumption characteristics after fusion. The communication time for the source node to send data. This represents the communication energy consumption of the source node when sending data. The computation time for the target drone node. This indicates the chip processing power consumption of the target node. This represents the energy consumption structure coefficient of the underlying chip. Since the computing power and energy consumption coefficients of different nodes in a heterogeneous cluster are different, this formula guides the reinforcement learning model to find the optimal compromise point between nodes with "sufficient computing power but high energy consumption" and "limited computing power but low energy consumption".

[0073] S43. Multi-target feature fusion scoring: The latency and energy consumption features obtained from S41 and S42 are projected into the joint space and fused with a high penalty term to obtain the output triple (source node, action, target node). The multi-target joint score is as follows:

[0074]

[0075] in, , , For learnable normalized weight allocation coefficients, This represents an extreme constant penalty imposed for link disconnection or queue overflow. It assigns a very high negative penalty to extreme abnormal situations caused by communication deadlock waiting and physical packet loss retransmission, in order to ensure stable convergence of the model under heavy load.

[0076] S5. Training of a Multi-Agent Model Based on "Centralized Training and Distributed Execution": To facilitate subsequent independent deployment on UAVs, this step employs the Multi-Agent Deep Deterministic Policy Gradient Algorithm (MADDPG). The model is divided into an "Evaluation Network (Critic)" responsible for scoring and a "Policy Network (Actor)" responsible for making decisions. A temporal difference algorithm is used to calculate the loss between the predicted value and the target value. Based on this loss value, the deep reinforcement learning scheduling model is trained through backpropagation using a gradient descent optimization algorithm. The neural network parameters are continuously updated until the model converges, resulting in a resource scheduling model with multi-objective optimization capabilities. The specific process is as follows:

[0077] S51. Collect global experience data: During the training phase, the central node collects the local states, actions, rewards, and next-moment states of all drones. This information is then merged into global states and global actions and stored in a unified experience replay pool.

[0078] S52. Training a centralized evaluation network: A batch of data is drawn from the experience pool, and the expected future returns are calculated using the Bellman equation.

[0079]

[0080] in For the target Q value, As a discount factor, This represents the action value function output by the evaluation network, used to evaluate the state at the next time step. Next action The expected cumulative return that can be obtained To evaluate the parameters of the network;

[0081] S53. The model is trained using a mini-batch experience replay strategy, calculating the error between the current network prediction and the target value for all candidate actions. The mean squared error loss function is used to train the multi-UAV cooperative scheduling model, as follows:

[0082]

[0083] in Indicates the batch size of the sample. This represents the loss value for the current batch of training. The parameters of the current evaluation network are defined. By repeatedly training the evaluation network and the policy network alternately until the model stops oscillating and fully converges, the optimal local scheduling policy for each UAV is obtained.

[0084] S6. Lightweight Strategy Model Extraction and Decentralized Closed-Loop Execution of Multiple Drones: To physically implement the "centralized training, distributed execution" architecture, a collaborative scheduling model that completes centralized training is deployed on simulated edge computing drone nodes. During the online operation phase, nodes frequently extract real-time cross-layer states under sudden high-pressure task flows, quickly generate candidate action sets by combining dynamic action masks, and directly calculate the scores of candidate unloading nodes through neural network forward inference. The target node with the highest score is selected to perform computational unloading, and the underlying physical network interface card and protocol stack are simultaneously driven to complete routing and link establishment.

[0085] S61. Lightweight Model Decomposition and Containerized Distributed Deployment: After the scheduling model converges during training at the central node, the system decomposes the centralized evaluation network that relies on the global state during the training phase, extracting only the local policy network corresponding to each UAV node. This local policy network is encapsulated as an independent algorithm node container image, serving as the forward inference model and its dependent environment. This container image is then distributed and deployed on each heterogeneous UAV, enabling each node to possess physically independent decentralized inference capabilities.

[0086] S62, Cross-Layer Feature-Driven and Mask-Safe Reasoning: Under high-frequency concurrent task flow, each UAV node acts as an independent decision-making agent, inputting the real-time local composite observation state extracted in stage S1 into its local container. The network forward propagation outputs the original action distribution. Subsequently, the system forcibly accesses the connectivity prior mask generated in the lower-level stage S2, performing Hadamard product aggregation at the neural network output layer. After filtering out invalid actions from physically unreachable nodes, the agent selects the optimal scheduling action from a safe subset of actions.

[0087]

[0088] S63. Distributed Closed-Loop Control and Adaptive Scheduling in a 3D Heterogeneous Topology: Based on the optimal action inferred locally, each UAV node maps the algorithm results in parallel to the underlying network and physical layer control commands, driving the onboard network card to complete link establishment and task distribution. For example... Figure 3 As shown, in the dynamic 3D topology, each node adaptively completes various scheduling logics based on distributed inference: normal offloading is performed when neighboring nodes have sufficient computing power; computational overflow is triggered to heterogeneous nodes when the source node's local computing power is overloaded; and mask red lines are triggered when a candidate target triggers... At that time, the node autonomously executes the distance penalty mechanism and forces a local rollback. The above process realizes the verification of highly robust full-link closed-loop scheduling without central dependency.

[0089] Although the present invention has been described herein with reference to embodiments thereof, the above embodiments are merely preferred embodiments of the present invention, and the implementation of the present invention is not limited to the above embodiments. It should be understood that those skilled in the art can design many other modifications and implementations, which will fall within the scope and spirit of the principles disclosed in this application.

Claims

1. A multi-UAV reinforcement learning cooperative scheduling method based on cross-layer perception and prior action masking, characterized in that, Includes the following steps: S1. Through the cross-layer data interaction interface, cross-layer feature extraction and distributed state synchronization are performed on the heterogeneous cluster of multi-UAVs to be scheduled to obtain a global initial composite state representation that characterizes the network load and communication topology. S2. Using a priori masking mechanism, based on the underlying radio signal fading model, receiver sensitivity threshold, and routing protocol limitations, the physical connectivity between the source UAV node and other candidate target UAV nodes in the cluster is calculated in real time to generate a dynamic priori action mask matrix. S3, the decision filtering module aggregates the original action space of the heterogeneous drone cluster with the prior action mask matrix using the Hadamard product, forces the selection probability of target drone nodes that are physically unreachable or have limited routes to be zero, and generates a safe and feasible subset of candidate decision actions. S4. The effective composite observation state after masking and the subset of candidate decision actions are jointly fed into the evaluation network of deep reinforcement learning to extract the multi-dimensional interaction features of the state and action, and output the expected value evaluation of the candidate action. The system then executes the scheduling action and uses a multi-objective joint reward function that takes into account latency, energy consumption, and packet loss penalties to calculate and obtain the multi-objective joint score of the action. S5. Calculate the temporal difference loss value of the reinforcement learning model using the loss function, and train the scheduling model based on the loss value using the gradient descent method to optimize the network parameters and obtain the trained scheduling model. S6. The trained scheduling model is used for actual edge computing offloading tasks. The concurrent tasks of the source UAV and candidate nodes are combined to form a candidate scheduling scheme. Then, the scores of all candidate schemes are calculated by inference network. Finally, the highest score is selected as the offloading target and the underlying physical network card and protocol stack are driven to complete the routing link establishment.

2. The multi-UAV reinforcement learning cooperative scheduling method based on cross-layer perception and prior action masking as described in claim 1, characterized in that, The cross-layer features include the UAV's local application layer task features and local hardware attributes; the distributed state synchronization specifically involves fusing and splicing the cross-layer features of the source UAV with the heterogeneous information of its neighbor set in the feature dimension to construct a composite observation state vector representing the heterogeneous connection relationship of the local neighborhood.

3. The multi-UAV reinforcement learning cooperative scheduling method based on cross-layer perception and prior action masking according to claim 2, characterized in that, The prior action mask matrix is ​​constructed as follows: Based on the currently acquired 3D coordinates, calculate the source drone node. With candidate target drone nodes Euclidean distance between Based on free-space path loss or log-distance path loss models, the received signal strength transmitted from the source UAV node to the candidate target UAV is predicted. ; Given the minimum receive sensitivity threshold required for hardware decoding of a low-level network interface card (NIC). and the maximum number of hops allowed by network layer routing protocols ,use Indicates the source drone node To candidate target drone node The actual number of hops between them is used to determine connectivity using a mask indicator function. and Time link connectivity, target drone node Prior action mask value ,otherwise , indicating that it is physically unreachable.

4. The multi-UAV reinforcement learning cooperative scheduling method based on cross-layer perception and prior action masking according to claim 3, characterized in that, The specific construction process of the candidate decision action subset is as follows: Assume the original scheduling probability distribution of each UAV node output by the deep reinforcement learning network is as follows: Combine it with the generated mask vector Perform Hadamard product aggregation mapping operations and filter out invalid actions; When removing unreachable node information, a mask-based safety constraint mechanism is added, and the updated action selection space is represented as follows: in This indicates that the source drone node is processing this task locally.

5. The multi-UAV reinforcement learning cooperative scheduling method based on cross-layer perception and prior action masking according to claim 4, characterized in that, The multi-objective joint score is calculated in the following specific way: Calculate the total end-to-end latency cost of executing the action. It covers source node queue waiting latency, wireless transmission latency, and target drone node processing latency; Calculate the total energy cost of the system in executing the action. This includes the communication transmission energy consumption of the source node and the processor computing energy consumption of the target node; The multi-objective joint score is as follows: in, , , For learnable normalized weight allocation coefficients, This represents an extreme constant penalty imposed for link disconnection or queue overflow.

6. The multi-UAV reinforcement learning cooperative scheduling method based on cross-layer perception and prior action masking according to claim 5, characterized in that, S5 is specifically as follows: S51. The multi-agent deep deterministic policy gradient algorithm is adopted to divide the scheduling model into an evaluation network responsible for scoring and a policy network responsible for making decisions. During the training phase, the local state, actions, multi-target joint scores and the state at the next moment of all UAVs are collected. This information is merged into global state and global actions and stored in a unified experience replay pool. S52. Training a centralized evaluation network: Extract a batch of data from the experience replay pool and use the Bellman equation to calculate the expected future returns; S53. The model is trained using a small-batch experience replay strategy. The error between the current network prediction value and the target value of all candidate actions is calculated. The scheduling model is trained using the mean squared error loss function. The evaluation network and policy network are trained alternately until the model converges, and the optimal local scheduling strategy for each UAV is obtained.

7. A multi-UAV reinforcement learning cooperative scheduling method based on cross-layer perception and prior action masking as described in claim 6, characterized in that, The local application layer task characteristics include the size of the task data. Required CPU computation cycles Concurrent queue depth and remaining computing resource capacity The local hardware attributes include its absolute coordinates in the 3D topology. Maximum transmission power Current computing power And the corresponding chip energy consumption structure coefficient .

8. A multi-UAV reinforcement learning cooperative scheduling method based on cross-layer perception and prior action masking as described in claim 7, characterized in that, The free space path loss is calculated as follows: in Indicates the radio carrier frequency.

9. A multi-UAV reinforcement learning cooperative scheduling method based on cross-layer perception and prior action masking as described in claim 8, characterized in that, The expected future returns are calculated as follows: in For the target Q value, As a discount factor, This represents the action value function output by the evaluation network, used to evaluate the state at the next time step. Next action The expected cumulative return that can be obtained To evaluate the parameters of the network; Then, the scheduling model is trained using the mean squared error loss function, as follows: in Indicates the batch size of the sample. This represents the loss value for the current batch of training. These are the parameters for evaluating the current network.

10. A multi-UAV reinforcement learning cooperative scheduling method based on cross-layer perception and prior action masking according to claim 9, characterized in that, S6 is specifically as follows: S61. After the scheduling model training converges, the system strips away the centralized evaluation network that depends on the global state during the training phase, extracts only the local policy network corresponding to each UAV node, uses the local policy network as the forward inference model, and encapsulates it with its dependent environment as an independent algorithm node container image, which is then distributed and deployed on each heterogeneous UAV. S62. Under concurrent task flow, each UAV node acts as an independent decision-making entity, inputting the real-time local composite observation state extracted in stage S1 into the local container, and the network forward propagation outputs the original action distribution; subsequently, the system forcibly accesses the connectivity prior mask generated in stage S2 at the bottom layer, and performs Hadamard product aggregation at the neural network output layer. After filtering out invalid actions from physically unreachable nodes, the agent selects the optimal scheduling action from a safe subset of actions. S63. Based on the optimal scheduling action inferred locally, each UAV node maps the algorithm results into control commands for the underlying network layer and physical layer in parallel, driving the onboard network card to complete link establishment and task distribution.

Images