A multi-agent joint scheduling method based on communication quality perception

Abstract

The application discloses a kind of multi-agent joint scheduling methods based on communication quality perception, belong to multi-agent cooperative control and intelligent scheduling technical field.Step 1, by obtaining local observation information to construct dynamic communication graph, local communication topology features are adaptively extracted using graph attention mechanism;Step 2, the double-branch network including task-driven branch and communication correction branch is constructed, and continuous action is generated based on the integrated state representation after fusion, and is output to the bottom layer of the agent Execution;Step 3, in the offline training phase, a restricted policy optimization framework based on Lagrange duality is established, the double-branch network is iteratively optimized, and the joint solution of task revenue and communication hard constraint is realized.The application can model the communication requirements as a hard constraint to ensure communication quality in a complex occlusion environment, improve the sensitivity of topology perception, and output continuous and smooth trajectories, with good engineering executability and strong distributed deployment capability.

Description

Technical Field

[0001] This invention belongs to the field of multi-agent cooperative control technology, specifically relating to a multi-agent joint scheduling method based on communication quality awareness. Background Technology

[0002] With the rapid development of artificial intelligence and mobile robotics technologies, multi-agent joint scheduling has become a core supporting technology for complex collaborative tasks such as regional search, target monitoring, and disaster inspection. In multi-agent joint scheduling scenarios, a massive number of agent nodes need to engage in high-frequency information interaction and collaborative control in an open physical space. Due to the high mobility of agent nodes and the limitations of the underlying perception and interaction links, the collaborative topology of multi-agent networks exhibits high-frequency dynamic changes.

[0003] Especially in complex and unknown environments or demanding agent collaborative tasks, the physical interaction obstacles brought about by this dynamic topology severely reduce the reliability of joint scheduling. Agent nodes need to receive and execute joint scheduling instructions from command nodes or core agents at high frequency. To ensure the consistency of cluster actions and the progress of scheduling tasks, a low-level joint scheduling mechanism is urgently needed: that is, implicitly maintaining the connectivity of the collaborative topology links and the efficiency of instruction distribution during physical space movement. However, in high-speed maneuvering agent formations, low-level link disconnection or interaction delays of hundreds of milliseconds can damage the stability of the control system, leading to coordination errors or even system-level stagnation. More seriously, when agents maneuver in complex spaces, their low-level interactions in open environments inevitably suffer from multiple constraints such as multipath interference, physical obstruction by obstacles, and distance loss, which can easily lead to critical scheduling instructions failing to be delivered or experiencing severe delays. Therefore, achieving multi-agent joint scheduling with extremely low latency, high collaborative connectivity, and smooth trajectory even under complex obstacle obstruction and severe low-level constraints is a fundamental challenge to ensure the stable operation of network systems.

[0004] Currently, existing multi-agent joint scheduling mechanisms can be mainly divided into the following categories, but all of them have significant technical limitations:

[0005] 1. Joint Scheduling Mechanisms Based on Traditional Heuristic Rules and Algebraic Graph Theory: These methods typically rely on the Artificial Potential Field (APF) method and control algorithms based on algebraic graph connectivity for scheduling and path planning. However, these rule-based methods heavily depend on globally accurate topological graph information and prior environmental knowledge. The resulting computational overhead and dependence on the global state fundamentally conflict with the stringent requirements of highly dynamic multi-agent networks for purely distributed local observation. Furthermore, heuristic algorithms are prone to getting trapped in local optima when faced with complex physical occlusion or unknown environments, leading to joint scheduling stagnation and an inability to adapt to real-world, ever-changing physical environments.

[0006] 2. Pure Task-Driven Joint Scheduling Methods Based on Deep Reinforcement Learning: With the upgrading of technical requirements in distributed collaborative work scenarios, multi-agent deep reinforcement learning algorithms, represented by Multi-Agent Proximal Policy Optimization (MAPPO), have demonstrated excellent fitting capabilities in continuous action space collaborative scheduling tasks. However, such methods face two major technical bottlenecks in real-world, strongly constrained multi-agent joint scheduling: First, most of them are based on the ideal assumption of "fully connected and barrier-free interaction," failing to fully consider the strict constraints of spatial distance and physical occlusion on entity movement trajectories; second, directly applying conventional reinforcement learning often leads to agents excessively diverging in formation structure in pursuit of single-dimensional task coverage, resulting in severe topological disconnection and loss of key scheduling instructions, limiting their scalability and security in high-reliability joint scheduling tasks.

[0007] 3. Constraint-Aware Scheduling Planning Based on Scalar Reward Shaping: To introduce low-level interaction constraints into joint scheduling, researchers often employ a "reward shaping" mechanism, directly aggregating low-level connectivity as a scalar penalty term into the overall reward function. However, this method, which forcibly compresses a multidimensional optimization objective into a one-dimensional scalar, is highly susceptible to "reward shaping failure." Agents may ignore topological disconnection penalties in pursuit of high task progress rewards, or become overly conservative and lose task exploration capabilities due to excessively high penalties, thus failing to provide strict "cooperative quality" guarantees for the joint scheduling system.

[0008] In summary, existing multi-agent joint scheduling techniques cannot simultaneously meet the comprehensive requirements of highly dynamic agent clusters for strong adaptability to complex and unknown environments, strict guarantee of cooperative connectivity quality, and continuous and smooth control of underlying kinematics. How to effectively extract dynamic topological features under local observation, overcome the communication quality degradation caused by scalar rewards, and decouple and optimize the progress benefits of joint scheduling tasks from stringent underlying interaction constraints, so as to ensure that multi-agents generate highly reliable and easily executable continuous cooperative scheduling trajectories in complex occlusion environments, has become a pressing technical challenge in this field. Summary of the Invention

[0009] To address the technical problems in existing technologies, such as the heavy reliance on global information in multi-agent path planning, the inability of scalar reward shaping to guarantee strict communication quality, and the unsmooth maneuverability and poor engineering feasibility of agent trajectories caused by discrete action spaces, this invention provides a multi-agent joint scheduling method based on communication quality awareness.

[0010] To achieve the above objectives, the specific technical solution adopted by the present invention is as follows: A multi-agent joint scheduling method based on communication quality awareness includes: Step 1: Construct a dynamic communication graph by acquiring local observation information of the agent, and use the graph attention mechanism to adaptively extract local communication topology features to provide effective state input containing dynamic topology and link quality features for subsequent collaborative decision-making; Step 2: Construct a dual-branch Actor network that includes a task-driven branch and a communication correction branch. The communication topology features extracted in Step 1 are fused with the agent's self-state features to form a comprehensive decision representation. Based on this representation, continuous actions that take into account both task advancement and communication connectivity maintenance are generated and output to the agent's underlying execution mechanism for execution. Step 3: During the offline training phase, a constrained policy optimization framework based on Lagrange duality is established. A centralized Critic network is used to evaluate the global value, and the dual-branch network in Step 2 is iteratively optimized to transform the instruction transmission risk into an explicit hard constraint, thereby achieving the joint solution of maximizing multi-agent task benefits and communication constraints.

[0011] Optionally, step 1 specifically includes: Step 1.1 Constructing the local observation space and local communication graph: For any intelligent agent Constructing a complete local observation space, including: observation of its own state information. Characterizes the agent's own position, velocity, remaining energy, and task state, which can be directly obtained; and observes the states of neighboring nodes. This characterizes interaction information from local neighbor nodes, including the number, location, velocity, and remaining energy of neighbor nodes; link state observation. : Represents the communication relationships between nodes, including agents Euclidean distance, signal-to-noise-interference ratio, and link quality metrics with neighbors; constructing an intelligent agent based on local observation information. At any moment Local communication diagram in For intelligent agents Let a local set of nodes consist of itself and its neighbors, and let the Euclidean distance between any two nodes satisfy:

[0012] in, Indicates time The set of locally available edges; Represents intelligent agents At any moment Position vector; Represents intelligent agents At any moment If the position vector is given, then a usable link is considered to exist between the two nodes. The communication radius; an adjacency matrix can be used to define the network communication graph. ,in:

[0013] Constructing degree matrix notation , The Laplace matrix is ​​written as: ; Step 1.2 Linear mapping of original characteristics of nodes and links: For any node in the local communication graph Define the original node feature vector that integrates node motion state and task information. ; for nodes with neighbors The link between them is defined as the link feature vector. Through learnable linear mapping matrices Mapping the original features to a unified latent space:

[0014] Step 1.3 Adaptive Weighted Aggregation of Neighborhood Features Based on Graph Attention Mechanism: Graph attention mechanism is used to calculate node... with neighbors Attention scores between:

[0015] in, These are learnable attention parameters, with the superscript T indicating the vector transpose operation; This represents vector concatenation; by performing Softmax normalization on the scoring results, the attention weights can be obtained.

[0016] Therefore, node The local communication graph aggregation feature can be represented as:

[0017] in, It is a nonlinear ReLU activation function; Step 1.4 Integrated State Representation Fusion: Through a multilayer perceptron encoder Observation of its own state The encoding representation is as follows: After concatenating it with the aggregated features of the local communication graph, it is then processed through a feature fusion network. Output a comprehensive state representation that agrees with the output:

[0018] Optionally, step 2 specifically includes: Step 2.1 Shared Feature Extraction: Based on the comprehensive state representation Extracting public decision features through a shared coding layer :

[0019] in, and These are the weight matrix and bias terms of the shared coding layer, respectively; Step 2.2 Generation of Two-Branch Continuous Actions: Task-driven branches are used to generate basic control actions for task advancement. :

[0020] in, Based on basic control actions, , These are the weight matrix and bias term of the task-driven branch, respectively; The communication preservation branch is used to generate communication correction directions oriented towards maintaining communication connectivity.

[0021] in, Correcting the direction for communication. , These are the weight matrix and bias term of the communication correction branch, respectively; Step 2.3 Adaptive Gated Fusion Output: Introducing a Sigmoid gating mechanism to adaptively adjust the strength of the communication correction term:

[0022] in, The weight matrix represents the adaptive gating mechanism. This represents the bias term of the adaptive gating mechanism; Calculate the communication action correction amount: The basic control actions and communication action corrections are combined to obtain the final continuous action instructions:

[0023] Step 2.4 Gaussian Policy Parameterization: A parameterized Gaussian policy is used to model the distribution of the final continuous actions, enabling the policy to achieve differentiable optimization in the continuous control space.

[0024] in, Determined by the mean motion obtained after the fusion of the two branches. Let be the strategy covariance matrix.

[0025] Optionally, step 3 specifically includes: Step 3.1 Centralized Training and Distributed Execution: A centralized Critic network is introduced, which uses the global node state and the overall communication graph structure to perform joint value evaluation. Each agent outputs actions through a distributed Actor network. Step 3.2 Constructing the composite reward and communication constraint cost function: Setting the global real-time composite reward function It integrates regional coverage benefits, target approximation benefits, effective backhaul benefits, and various physical constraint penalties; and models command transmission risk as an explicit constraint cost function. It comprehensively reflects topology reachability, link quality, and transmission timeliness;

[0026] in, Indicates regional coverage revenue. This indicates the return from target discovery or target approximation. This indicates effective return on investment. Indicates a collision penalty. This indicates a penalty for energy consumption. This indicates a penalty for drastic changes in trajectory; These are the weighting coefficients for each item;

[0027] in, These are the weighting coefficients for various communication risk costs; Step 3.3 Advantage Function Estimation: Constructing a task value network for evaluating task rewards. With constrained value networks used to evaluate communication costs And calculate the task advantage function respectively. With constraint advantage function ; Step 3.4 Dynamic Update of Lagrange Multipliers: Introducing Lagrange Multipliers Construct the Lagrange dual function:

[0028] Dynamically update the multipliers using the nonnegative projection operator. Adaptively adjust the weight of communication cost in policy updates:

[0029] To update the step size for the multiplier, Represents the nonnegative projection operator, threshold ; Step 3.5 Strategy Iteration and Update: Let the old strategy parameters be The probability ratio is defined as:

[0030] in, and Represent the joint action and joint local state characteristics, respectively; calculate the Lagrange-corrected dominance function. Update the Actor network within the near-end policy optimization framework:

[0031] To limit the policy update magnitude and ensure training stability, the pruning parameters are used, while the dual Critic network is updated by minimizing the mean squared error loss until the policy converges.

[0032] Compared with the prior art, the beneficial technical effects of this invention are: 1. Ensures communication quality under harsh environments: Compared with the traditional scalar reward shaping method, this invention models communication requirements as hard constraints through the Lagrange duality mechanism, which significantly reduces the communication violation rate in complex occlusion environments and greatly improves the delivery rate of key control commands and the survivability of the system.

[0033] 2. Improved sensitivity of topology awareness: By introducing the graph attention mechanism (GAT) to extract features of local communication graphs, the agent can capture the evolution trend of local dynamic topology in real time, which can more accurately identify and maintain key relay communication links than traditional distance-based heuristic methods.

[0034] 3. Smooth trajectory and high engineering feasibility: The dual-branch network outputs continuous motion control commands, avoiding execution abrupt changes caused by discrete motion space. The generated physical trajectory is smooth and continuous, which greatly reduces mechanical chattering and tracking loss of the underlying servo and flight control system, and has strong engineering deployability.

[0035] 4. Strong distributed deployment and anti-interference capability: Based on the centralized training and distributed execution (CTDE) paradigm, the model only needs to rely on local perception to make quasi-optimal decisions independently during the online execution phase, which eliminates the cluster's dependence on global real-time high-frequency communication and enhances the robustness of multi-agent networks in strong interference environments. Attached Figure Description

[0036] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram of the overall framework for multi-agent joint scheduling with communication quality awareness according to the present invention. Figure 2This is a schematic diagram of the structure of the local communication topology feature extraction network in an embodiment of the present invention; Figure 3 This is a schematic diagram of the comprehensive state representation output structure in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of the dual-branch continuous action generation module in an embodiment of the present invention; Figure 5 This is a comparison diagram of the local flight trajectories of multiple agents in complex physical occlusion scenarios, based on the embodiments of the present invention and the comparison algorithm. Figure 6 This is a comparison diagram of the communication connectivity status between the embodiments of the present invention and the comparison algorithm; Figure 7 This is a comprehensive performance comparison chart of the embodiments of the present invention and the comparative method; Figure 8 This is a curve comparing the average round reward and communication constraint violation rate during the training process of the embodiment of the present invention and the comparison algorithm; Figure 9 This is a comparison chart of the path smoothing performance of the embodiments of the present invention and the comparison algorithm. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0038] Combination Figure 1-4 The multi-agent joint scheduling method based on communication quality awareness of the present invention includes the following steps: Step 1: A graph attention topology feature extraction module for communication graph awareness is designed to solve the technical problem that dynamic communication graphs are difficult to accurately depict under local observation conditions, and key relay links and transmission risks cannot be adaptively identified, resulting in a lack of effective state support for distributed collaborative decision-making; Specifically, a dynamic communication graph is constructed by acquiring the local observation information of the agents, and local communication topology features are adaptively extracted using the graph attention mechanism to provide effective state input containing dynamic topology and link quality features for subsequent collaborative decision-making; Step 2: Construct a dual-branch Actor network that includes a task-driven branch and a communication correction branch. The communication topology features extracted in Step 1 are fused with the agent's self-state features to form a comprehensive decision representation. Based on this representation, continuous actions that take into account both task advancement and communication connectivity maintenance are generated and output to the agent's underlying execution mechanism for execution. Step 3: During the offline training phase, a constrained policy optimization framework based on Lagrange duality is established. A centralized Critic network is used to evaluate the global value, and the dual-branch network in Step 2 is iteratively optimized to transform the instruction transmission risk into an explicit hard constraint, thereby achieving the joint solution of maximizing multi-agent task benefits and communication constraints.

[0039] Step 1 specifically includes: Step 1.1 Constructing the local observation space and local communication graph: for any intelligent agent Constructing a complete local observation space, including: observation of its own state information. Characterizes the agent's own position, velocity, remaining energy, and task state, which can be directly obtained; and observes the states of neighboring nodes. This characterizes interaction information from local neighbor nodes, including the number, location, velocity, and remaining energy of neighbor nodes; link state observation. : Represents the communication relationships between nodes, including agents The Euclidean distance, signal-to-noise-interference ratio, and link quality metrics with neighbors are analyzed. An intelligent agent is constructed based on local observation information. Local communication graph at time t in For nodes Let a local set of nodes consist of itself and its neighbors, and let the Euclidean distance between any two nodes satisfy:

[0040] This assumes that a usable link exists between the two nodes. The communication radius; an adjacency matrix can be used to define the network communication graph. ,in

[0041] Constructing degree matrix notation , The Laplace matrix is ​​written as: ; Step 1.2 Linear mapping of original characteristics of nodes and links: For any node in the local communication graph Define the original node feature vector that integrates node motion state and task information. ; for nodes with neighbors The link between them is defined as the link feature vector. Through learnable linear mapping matrices Mapping the original features to a unified latent space:

[0042] Step 1.3 Adaptive Weighted Aggregation of Neighborhood Features Based on Graph Attention Mechanism: Graph attention mechanism is used to calculate node... with neighbors Attention scores between:

[0043] in, These are learnable attention parameters. This represents vector concatenation. Softmax normalization of the scoring results yields the attention weights:

[0044] Therefore, node The local communication graph aggregation feature can be represented as:

[0045] in, It is a nonlinear ReLU activation function; Step 1.4 Integrated State Representation Fusion: Through a multilayer perceptron encoder Observation of its own state The encoding representation is as follows: After concatenating it with the aggregated features of the local communication graph, it is then processed through a feature fusion network. Output a comprehensive state representation that agrees with the output:

[0046] Step 2 specifically includes: Step 2.1 Shared feature extraction: based on the comprehensive state representation Extracting public decision features through a shared coding layer :

[0047] in, and These are the weight matrix and bias terms of the shared coding layer, respectively; Step 2.2 Generation of Two-Branch Continuous Actions: Task-driven branches are used to generate basic control actions for task advancement. :

[0048] in, Based on basic control actions, , These are the weight matrix and bias term of the task-driven branch, respectively; The communication preservation branch is used to generate communication correction directions oriented towards maintaining communication connectivity.

[0049] in, Correcting the direction for communication. , These are the weight matrix and bias term of the communication correction branch, respectively; Step 2.3 Adaptive Gated Fusion Output: Introducing a Sigmoid gating mechanism to adaptively adjust the strength of the communication correction term:

[0050] Calculate the communication action correction amount: The basic control actions and communication action corrections are combined to obtain the final continuous action instructions:

[0051] Step 2.4 Gaussian Policy Parameterization: A parameterized Gaussian policy is used to model the distribution of the final continuous actions, enabling the policy to achieve differentiable optimization in the continuous control space.

[0052] in, Determined by the mean motion obtained after the fusion of the two branches. Let be the strategy covariance matrix.

[0053] Step 3 specifically includes: Step 3.1 Centralized training and distributed execution: Introduce a centralized Critic network, use the global node state and the overall communication graph structure to perform joint value evaluation, and each agent outputs actions through a distributed Actor network; Step 3.2 Constructing the composite reward and communication constraint cost function: Setting the global real-time composite reward function It integrates regional coverage benefits, target approximation benefits, effective backhaul benefits, and various physical constraint penalties; and models command transmission risk as an explicit constraint cost function. It comprehensively reflects topology reachability, link quality, and transmission timeliness;

[0054] in, Indicates regional coverage revenue. This indicates the return from target discovery or target approximation. This indicates effective return on investment. Indicates a collision penalty. This indicates a penalty for energy consumption. This indicates a penalty for drastic changes in trajectory; These are the weighting coefficients for each item;

[0055] in, These are the weighting coefficients for various communication risk costs; Step 3.3 Advantage Function Estimation: Constructing a task value network for evaluating task rewards. With constrained value networks used to evaluate communication costs And calculate the task advantage function respectively. With constraint advantage function ; Step 3.4 Dynamic Update of Lagrange Multipliers: Introducing Lagrange Multipliers Construct the Lagrange dual function:

[0056] The multiplier is dynamically updated using a nonnegative projection operator, and the weight of the communication cost in the policy update is adaptively adjusted:

[0057] Step 3.5 Strategy Iteration and Update: Let the old strategy parameters be The probability ratio is defined as:

[0058] in, and Represent the joint action and joint local state characteristics, respectively; calculate the Lagrange-corrected dominance function. Update the Actor network within the near-end policy optimization framework:

[0059] To limit the policy update magnitude and ensure training stability, a pruning parameter (truncation threshold) is used. Simultaneously, the dual Critic network is updated by minimizing the mean squared error loss until the policy converges.

[0060] The multi-agent joint scheduling method of this invention is a universal cooperative scheduling algorithm widely applicable to various mobile intelligent agents such as UAVs, unmanned vehicles, and unmanned surface vessels. For ease of engineering understanding and explanation, this embodiment specifically uses a typical multi-agent joint scheduling scenario—multi-UAV cooperative reconnaissance and target monitoring—as an example to elaborate on the entire process of end-to-end airborne online distributed execution and offline centralized training of this method.

[0061] I. Implementation Example: Environment Setup and Physical Kinematic Model This embodiment simulates a multi-UAV swarm performing a collaborative area search and target monitoring task in a complex and unknown environment. During the task execution, the command node needs to continuously send key control commands carrying authentication information to the swarm members, and the target UAV needs to rely on multi-hop relay paths to achieve continuous distribution and reliable delivery of commands.

[0062] The UAV adopts a three-dimensional continuous kinematics model. Let the UAV... exist The physical state at a given moment is determined by its position vector. and velocity vector Its state update strictly follows the dynamic integral equation:

[0063]

[0064] in, This is the continuous control input output by the policy network. It is defined as a three-dimensional acceleration vector, i.e.:

[0065] To approximate the physical limits of real multi-rotor drones, hard constraints are imposed on their speed and acceleration:

[0066] II. Airborne Online Distributed Decision Execution Process During the online deployment phase, each UAV embeds its trained policy network parameters into its onboard computing platform. Within each decision-making time slot, each UAV can independently output continuous flight control commands based solely on local observations acquired by its onboard sensors and wireless communication modules, without requiring global information exchange. The specific steps are as follows:

[0067] Step 1: Construction of Local Dynamic Communication Graph and Extraction of Topological Features First, drones It acquires its own self-state observation through airborne sensing equipment and communication modules. Observation of neighbor node status Link status observation Based on the Euclidean distance between nodes ( To determine the communication radius and link availability, construct a local communication graph. .

[0068] Subsequently, communication topology features are extracted using a graph attention network (GAT).

[0069] Through learnable linear mapping matrix Let the original node feature vector be denoted as Mapping to the unified latent space Integrating node characteristics and link quality characteristics compute nodes with neighbors Attention scores between: ; The scores of all local nodes are normalized using Softmax to obtain the final attention weights: ; Based on normalized attention weights, the features of all nodes within the local communication graph are weighted and aggregated to obtain the UAV. Local communication graph aggregation features: ; in, This is a nonlinear ReLU activation function. The state observations are encoded using a multilayer perceptron (MLP) encoder. ; The self-state encoding features and topological aggregation features are concatenated into vectors, and then a feature fusion MLP network is used to generate the final comprehensive state representation. ; Step 2: Generation of Dual-Branch Continuous Motions and Output of Flight Control Commands Genetic comprehensive state representation By sharing a fully connected coding layer, common features are extracted from the comprehensive state representation to obtain... .

[0070] A dual-branch structure with functional decoupling is adopted to generate task-driven basic actions and communication-maintaining correction directions, respectively.

[0071] Task-driven branches are used to generate basic control actions for task-oriented progression:

[0072] Communication maintenance branches generate communication correction directions based on the same common feature:

[0073] A sigmoid gating mechanism is introduced to adaptively adjust the strength of the communication correction term based on the current local state of the UAV and the communication topology, thus avoiding excessive intervention of the communication branch on basic actions.

[0074] Therefore, the communication correction branch is used to generate the action correction amount for communication maintenance:

[0075] This branch modifies basic actions based on local topology and link status to suppress maneuvering tendencies that could lead to relay path breaks, link quality degradation, or unreachable target nodes. Therefore, the first... The final continuous actions of an agent are defined as follows:

[0076] The former provides the main direction of trajectory evolution, while the latter applies continuous corrections based on the local communication graph state, thereby enabling the UAV swarm to maintain the communication support capability required for the transmission of coordinated scheduling commands while advancing its mission.

[0077] A parameterized Gaussian strategy is used to model the distribution of the final action, defined as follows:

[0078] in, Determined by the mean motion obtained after the fusion of the two branches. Let be the policy covariance matrix. This parameterization method enables the model to achieve differentiable optimization in a continuous control space, balancing exploration and exploitation.

[0079] III. The Entire Process of Offline Centralized Training and Constrained Policy Optimization The policy network parameters are trained offline using a centralized training and distributed execution (CTDE) framework, combined with multi-agent proximal policy optimization (MAPPO) and Lagrange duality. The derivation and implementation steps of the entire process are as follows.

[0080] Step 1: Constructing the cost function of composite reward and hard communication constraints Construct a global composite reward function that includes coverage, target approximation, effective backpropagation, and collision avoidance penalties. :

[0081] in, Indicates regional coverage revenue. This indicates the return from target discovery or target approximation. This indicates effective return on investment. Indicates a collision penalty. This indicates a penalty for energy consumption. This indicates a penalty for drastic changes in trajectory; These are the weighting coefficients for each item.

[0082] The definitions of each reward item are as follows: (1) Area Coverage Incentive: To encourage UAVs to expand their effective search range and reduce redundant coverage, the increased coverage area or the number of new effective grid cells can be used as a source of incentives:

[0083] in, Represents the task region grid set. Indicates the end time The set of areas that have been effectively covered. The larger this value is, the more significant the contribution of the current joint action to the overall mission advancement.

[0084] (2) Goal-related rewards: If the task scenario includes goal discovery or goal approximation requirements, the following can be defined:

[0085] in, Find indicator functions for the target. For drones The relative distance to the target point. The former encourages target discovery, while the latter encourages continuous approach to high-value areas.

[0086] (3) Data Backhaul Reward. Considering that this paper emphasizes "task execution and reliable backhaul coexisting," a positive reward should be given when key observation data can be successfully backhauled to the ground station via a single-hop or multi-hop path. This can be defined as:

[0087] in, Indicates at time If a valid backhaul path exists from the information source node to the ground station, then 0 is used. To enhance smoothness, this can be generalized to a continuous reward related to end-to-end link quality or packet delivery success rate.

[0088] (4) Collision Penalty. To avoid collisions between drones or conflicts with obstacles, a safe distance constraint penalty is defined:

[0089] in, To ensure the minimum safe distance between drones, This is the obstacle avoidance threshold.

[0090] (5) Energy consumption penalty. To suppress unnecessary large-scale maneuvers and improve the deployability of strategic engineering, the control input module length or flight cost can be included in the penalty:

[0091] (6) Trajectory smoothing penalty. To reduce motion jitter and improve trajectory executability, a motion change rate penalty can be introduced: ; 2. Definition of Explicit Communication Constraint Cost Function The unacceptable risks in the process of coordinating scheduling instruction transmission are modeled separately as an explicit constraint cost function. The instantaneous communication cost at any given moment is defined as: ; in, These are the weighting coefficients for various communication risk costs. This design enables communication constraints to simultaneously reflect topology reachability, link quality, and transmission timeliness, thereby more accurately characterizing the bottom-line requirements for reliable transmission of coordinated scheduling instructions. The mathematical definitions of each cost sub-item are as follows: 1) Cost of unreachable critical nodes. If at time... If there is no valid single-hop or multi-hop path between the command node and the target task node, the coordinated scheduling command cannot be transmitted. The corresponding cost is defined as follows: ; in, This indicates that a valid transmission path exists from the command node to the target node; otherwise, it is discarded. This item is used to characterize the inaccessibility risk caused by topology breaks or missing relay paths.

[0092] 2) Link Quality Degradation Cost. Even if the network topology remains geometrically connected, insufficient link quality can still lead to authentication information detection failures or increased control message error rates. Therefore, this paper introduces the link quality degradation cost. Let... The link quality metric representing the transmission path of coordinated scheduling instructions can be taken as the minimum signal-to-noise-interference ratio, average link quality, or equivalent reception reliability. It is defined as follows:

[0093] in, This is the link quality threshold required to meet the requirements for effective transmission and authentication decisions. It characterizes situations where "the link exists, but the transmission quality is insufficient."

[0094] 3) Cost of Excessive Transmission Delay. In collaborative control scenarios, if the collaborative scheduling command experiences excessive delay, even if it is eventually delivered, control effectiveness may be lost. Let... For end-to-end command transmission delay, As the upper bound of the system's tolerable delay, then we define:

[0095] Define the strategy based on immediate reward and immediate cost. The following are the long-term cumulative task benefits and long-term cumulative communication constraint costs:

[0096] in, As a discount factor, For a moment The real-time communication cost is used to characterize the unacceptable link instability risk during the transmission of coordinated scheduling instructions.

[0097] Step 2: Lagrange multiplier adaptive update Construct the Lagrangian function: During training iterations, the multipliers are dynamically updated based on constraint violations. :

[0098] in, To update the step size for the multiplier, This represents the nonnegative projection operator. When the communication cost exceeds a threshold... hour, Increasing the constraint increases the effectiveness of the strategy in suppressing communication breaches; conversely, when the constraints are well satisfied, The reduction allows the strategy to pursue increased reconnaissance gains while maintaining communication quality.

[0099] Step 3: Lagrange adjustment of the dominance function and network update In the PPO near-end update framework, a Lagrange adjustment advantage function is introduced to achieve stable iterative updates of the policy. The specific process is as follows: Let the old strategy parameters be The probability ratio is defined as: ; in, and These represent the joint action and the joint local state representation, respectively. MAPPO limits the policy update magnitude through a pruning mechanism to ensure training stability.

[0100] During the strategy update process, task advantage functions are constructed respectively. With constraint advantage function The Lagrange corrected advantage is defined as:

[0101] Therefore, the update target of the Actor can be written as:

[0102] in, These are the pruning parameters. Simultaneously, the task value network and the constraint value network are updated by minimizing the mean squared error loss:

[0103]

[0104] The process of cyclically performing trajectory sampling, advantage function estimation, Critic network update, Lagrange multiplier update, and Actor network update continues until the policy converges, resulting in the final multi-UAV cooperative path planning policy.

[0105] IV. Experimental Performance Analysis To verify the technical effectiveness and practical engineering deployment capability of this invention in multi-agent (e.g., multi-UAV) collaborative scenarios, this embodiment constructs a high-fidelity co-simulation environment and conducts systematic performance tests on complex and unknown environments including physical occlusion. Several existing mainstream algorithms are also introduced for comparative analysis. The baselines for comparison include: traditional heuristic algorithms (APF combined with connectivity-maintained potential field), basic unconstrained multi-agent reinforcement learning algorithm (Vanilla MAPPO), graph reinforcement learning algorithm (GA-MATR), and communication-aware algorithm based on scalar reward shaping (Mix-Greedy MAPPO).

[0106] Referring to the accompanying drawings, the comparative experimental performance analysis of the embodiments of the present invention is as follows: (1) Comprehensive performance comparison analysis Combination Figure 7 As shown in the comprehensive performance comparison chart, the communication quality-aware multi-agent joint scheduling method of this invention demonstrates significant advantages in all core communication and task metrics. When facing complex dynamic topologies, Vanilla MAPPO, due to the lack of explicit communication constraints, achieves high task coverage but extremely low delivery rate of critical control commands. GA-MATR and Mix-Greedy MAPPO show some improvement through graph structure modeling or reward mixing, but still cannot guarantee communication quality under strong constraints. In contrast, this invention increases the successful delivery rate of critical control commands to 92.4%, and stably controls the average end-to-end command latency to 118ms. Simultaneously, while ensuring 87.5% task coverage, this invention achieves a 90.8% network connectivity maintenance rate, verifying that this method can achieve an efficient joint solution between task advancement benefits and communication maintenance constraints.

[0107] (2) Analysis of training convergence characteristics and the effectiveness of constraint optimization Combination Figure 8 (The curve comparing the convergence of average round reward and communication constraint violation rate) shows that the convergence performance of each algorithm differs significantly during the offline training phase. Vanilla MAPPO's communication constraint violation rate remains consistently high and fluctuates wildly; methods based on scalar reward shaping, such as Mix-Greedy MAPPO, struggle to stabilize and reduce their violation rates in the later stages of training. The Lagrangian-constrained policy optimization framework adopted in this invention exhibits a clear "constraint convergence—reward recovery" stable characteristic. In the middle of training, the policy can quickly identify and suppress maneuvering behaviors that lead to link degradation, and the communication constraint violation rate (e.g., average round reward versus communication constraint violation rate) decreases significantly. Figure 8 The violation rate curve (in the training data) shows a rapid downward trend; in the later stages of training, the violation rate steadily converges to a certain value. 5% The training stability and constraint protection capabilities of this invention are demonstrated to be below the safety threshold, while the average round task reward steadily increases.

[0108] (3) Balanced task execution efficiency and communication constraints Combination Figure 5 (Comparison of local flight trajectories of multiple agents) and Figure 6 (As shown in the communication connectivity comparison diagram), when encountering physical obstacles such as large buildings that obstruct the line-of-sight link, the comparative algorithm is prone to local nodes blindly moving forward, causing relay network breaks. However, the multi-agent cluster of this invention exhibits adaptive role differentiation and topology reconstruction capabilities: some agents actively hover at the edge of obstruction or in areas with good line-of-sight conditions, acting as relay nodes, while the remaining agents continue to penetrate deeper into the task area while maintaining available multi-hop links. Combined with... Figure 6 As can be seen from the network topology of the connected state, the present invention maintains a highly reliable multi-hop communication path during complex obstacle avoidance processes, effectively overcoming the risk of disconnection caused by non-line-of-sight transmission.

[0109] (4) Good trajectory smoothness and feasibility of underlying engineering Combination Figure 9 (Path smoothing performance comparison chart) As shown, at the microscopic physical execution level, the geometric smoothness of the planned trajectory directly determines the tracking error and airframe load of the underlying flight control system. APF and Vanilla MAPPO are prone to high-frequency oscillations or jagged maneuvering abrupt changes during obstacle avoidance; algorithms based on discrete action spaces produce step-like discontinuous trajectories. This invention relies on a dual-branch continuous action generation mechanism, outputting a smooth and continuous cooperative flight trajectory. For example... Figure 9 As shown, the maximum acceleration fluctuation at the underlying level controlled by this algorithm is smoothly limited to within 2.3 m / s², and the average trajectory smoothness (normalized index) reaches 0.89. This low-jerkiness continuous control command greatly reduces the servo tracking loss of the underlying flight control system, proving that this method has excellent engineering deployment feasibility on real physical hardware platforms.

[0110] (5) Stable policy convergence properties Combination Figure 8 During the offline training phase, the Lagrange constrained optimization mechanism employed in this invention exhibits a stable characteristic of "constraint convergence—reward recovery." In the middle of training, through adaptive adjustment of the Lagrange multipliers, the policy effectively identifies and suppresses maneuvering behaviors that lead to link degradation, causing a rapid decrease in the communication violation rate. In the later stages of training, the violation rate stably converges to below the preset safety threshold, while the overall task reward steadily recovers. This characteristic verifies that the algorithm of this invention possesses good training stability and robustness in complex continuous control tasks.

[0111] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this invention, and these modifications or substitutions should all be covered within the scope of protection of this invention. Therefore, the scope of protection of this invention should be determined by the scope of the claims.

Claims

1. A multi-agent joint scheduling method based on communication quality awareness, characterized in that, include: Step 1: Construct a dynamic communication graph by acquiring local observation information of the agent, and use the graph attention mechanism to adaptively extract local communication topology features to provide effective state input containing dynamic topology and link quality features for subsequent collaborative decision-making; Step 2: Construct a dual-branch Actor network that includes a task-driven branch and a communication correction branch. The communication topology features extracted in Step 1 are fused with the agent's self-state features to form a comprehensive decision representation. Based on this representation, continuous actions that take into account both task advancement and communication connectivity maintenance are generated and output to the agent's underlying execution mechanism for execution. Step 3: During the offline training phase, a constrained policy optimization framework based on Lagrange duality is established. A centralized Critic network is used to evaluate the global value, and the dual-branch network in Step 2 is iteratively optimized to transform the instruction transmission risk into an explicit hard constraint, thereby achieving the joint solution of maximizing multi-agent task benefits and communication constraints.

2. The multi-agent joint scheduling method based on communication quality awareness according to claim 1, characterized in that, Step 1 specifically includes: Step 1.1 Constructing the local observation space and local communication graph: For any intelligent agent Constructing a complete local observation space, including: observation of its own state information. Characterizes the agent's own position, velocity, remaining energy, and task state, which can be directly obtained; and observes the states of neighboring nodes. This characterizes interaction information from local neighbor nodes, including the number, location, velocity, and remaining energy of neighbor nodes; link state observation. : Represents the communication relationships between nodes, including agents Euclidean distance, signal-to-noise-interference ratio, and link quality metrics with neighbors; constructing an intelligent agent based on local observation information. At any moment Local communication diagram in For intelligent agents Let a local set of nodes consist of itself and its neighbors, and let the Euclidean distance between any two nodes satisfy: in, Indicates time The set of locally available edges; Represents intelligent agents At any moment Position vector; Represents intelligent agents At any moment If the position vector is given, then a usable link is considered to exist between the two nodes. The communication radius; an adjacency matrix can be used to define the network communication graph. ,in: Constructing degree matrix notation , The Laplace matrix is ​​written as: ; Step 1.2 Linear mapping of original characteristics of nodes and links: For any node in the local communication graph Define the original node feature vector that integrates node motion state and task information. ; for nodes with neighbors The link between them is defined as the link feature vector. Through learnable linear mapping matrices Mapping the original features to a unified latent space: Step 1.3 Adaptive Weighted Aggregation of Neighborhood Features Based on Graph Attention Mechanism: Graph attention mechanism is used to calculate node... with neighbors Attention scores between: in, These are learnable attention parameters, with the superscript T indicating the vector transpose operation; This represents vector concatenation; by performing Softmax normalization on the scoring results, the attention weights can be obtained. Therefore, node The local communication graph aggregation feature can be represented as: in, It is a nonlinear ReLU activation function; Step 1.4 Integrated State Representation Fusion: Through a multilayer perceptron encoder Observation of its own state The encoding representation is as follows: After concatenating it with the aggregated features of the local communication graph, it is then processed through a feature fusion network. Output a comprehensive state representation that agrees with the output: 。 3. The multi-agent joint scheduling method based on communication quality awareness according to claim 1 or 2, characterized in that, Step 2 specifically includes: Step 2.1 Shared Feature Extraction: Based on the comprehensive state representation Extracting public decision features through a shared coding layer : in, and These are the weight matrix and bias terms of the shared coding layer, respectively; Step 2.2 Generation of Two-Branch Continuous Actions: Task-driven branches are used to generate basic control actions for task advancement. : in, Based on basic control actions, , These are the weight matrix and bias term of the task-driven branch, respectively; The communication preservation branch is used to generate communication correction directions oriented towards maintaining communication connectivity. in, Correcting the direction for communication. , These are the weight matrix and bias term of the communication correction branch, respectively; Step 2.3 Adaptive Gated Fusion Output: Introducing a Sigmoid gating mechanism to adaptively adjust the strength of the communication correction term: in, The weight matrix represents the adaptive gating mechanism. This represents the bias term of the adaptive gating mechanism; Calculate the communication action correction amount: The basic control actions and communication action corrections are combined to obtain the final continuous action instructions: Step 2.4 Gaussian Policy Parameterization: A parameterized Gaussian policy is used to model the distribution of the final continuous actions, enabling the policy to achieve differentiable optimization in the continuous control space. in, Determined by the mean motion obtained after the fusion of the two branches. Let be the strategy covariance matrix.

4. The multi-agent joint scheduling method based on communication quality awareness according to claim 1 or 2, characterized in that, Step 3 specifically includes: Step 3.1 Centralized Training and Distributed Execution: A centralized Critic network is introduced, which uses the global node state and the overall communication graph structure to perform joint value evaluation. Each agent outputs actions through a distributed Actor network. Step 3.2 Constructing the composite reward and communication constraint cost function: Setting the global real-time composite reward function It integrates regional coverage benefits, target approximation benefits, effective backhaul benefits, and various physical constraint penalties; and models command transmission risk as an explicit constraint cost function. It comprehensively reflects topology reachability, link quality, and transmission timeliness; in, Indicates regional coverage revenue. This indicates the return from target discovery or target approximation. This indicates effective return on investment. Indicates a collision penalty. This indicates a penalty for energy consumption. This indicates a penalty for drastic changes in trajectory; These are the weighting coefficients for each item; in, These are the weighting coefficients for various communication risk costs; Step 3.3 Advantage Function Estimation: Constructing a task value network for evaluating task rewards. With constrained value networks used to evaluate communication costs And calculate the task advantage function respectively. With constraint advantage function ; Step 3.4 Dynamic Update of Lagrange Multipliers: Introducing Lagrange Multipliers Construct the Lagrange dual function: Dynamically update the multipliers using the nonnegative projection operator. Adaptively adjust the weight of communication cost in policy updates: To update the step size for the multiplier, Represents the nonnegative projection operator, threshold ; Step 3.5 Strategy Iteration and Update: Let the old strategy parameters be The probability ratio is defined as: in, and Represent the joint action and joint local state characteristics, respectively; calculate the Lagrange-corrected dominance function. Update the Actor network within the near-end policy optimization framework: To limit the policy update magnitude and ensure training stability, the pruning parameters are used, while the dual Critic network is updated by minimizing the mean squared error loss until the policy converges.

Images