Heterogeneous multi-agent cooperative task planning method, system and device based on graph attention network and hybrid path planning, and storage medium

Abstract

The application discloses a heterogeneous multi-agent cooperative task planning method, system, device and storage medium based on a graph attention network and a hybrid path planning. The method constructs multi-type agents, multiple targets and an obstacle environment, generates a dynamic graph according to a communication distance, extracts node embedding and fixed-dimension graph embedding by using GATv2, selects a target by an actor, outputs a target slot Q value by a critic, aggregates team value by a hybrid network and a preference attention, combines Astar and a path planning network during execution, and adopts re-planning caching, staggered updating and collision slip correction. By maximum target slot filling, masking and graph embedding pooling, the network can adapt to changes in the number of agents and targets, improve heterogeneous task allocation efficiency, path safety and training scalability.

Description

Technical Field

[0001] This invention relates to the fields of intelligent agent cooperative control, robot task planning, path planning, and multi-agent reinforcement learning. Specifically, it relates to a heterogeneous multi-agent cooperative task planning method, system, device, and storage medium based on graph attention networks and hybrid path planning, which is applicable to scenarios such as multi-robot inspection, warehouse handling, disaster search, unmanned platform collaborative execution, and complex task allocation. Background Technology

[0002] In multi-robot or multi-agent systems, tasks often simultaneously involve multiple objectives, multiple constraints, and multi-role collaboration. Different agents differ in their perception range, mobility, execution skills, and task preferences, and the target tasks may also have different completion conditions. For example, some objectives must be completed by expert agents, while others require multiple agents of different types to collaborate within the same target neighborhood. Although traditional centralized task allocation methods can obtain relatively clear global planning results, as the number of agents and objectives increases, the state dimension and combinatorial search space expand rapidly, making it difficult to meet real-time and scalability requirements.

[0003] Existing multi-agent reinforcement learning methods typically use a global state vector or a concatenated agent state as input to a value mixing network. When the number of agents changes, the dimension of the directly concatenated global state vector changes accordingly; when the number of targets changes, the input and output dimensions of the task selection network may also change, making network structure difficult to reuse and resulting in low training sample efficiency. On the other hand, learning-based path planning methods are prone to collisions, detours, and local oscillations in the early stages of training; planning methods based solely on search algorithms such as Astar face high computational pressure in large-scale multi-agent execution, and replanning for all agents at each time step incurs significant overhead.

[0004] Therefore, there is an urgent need for a collaborative task planning method that can simultaneously handle heterogeneous role constraints, multi-objective preferences, collaborative goal completion conditions, and obstacle path execution, so that it can still have stable representation dimensions, high training efficiency, and good execution security when the number of agents changes and the actual number of targets does not exceed the preset maximum number of targets. Summary of the Invention

[0005] The purpose of this invention is to provide a heterogeneous multi-agent cooperative task planning method, system, device, and storage medium based on graph attention networks and hybrid path planning, which solves the problems of existing methods in heterogeneous multi-agent cooperative tasks, such as difficulty in expanding the global state dimension, insufficient efficiency in cooperative target allocation, high computational load in path planning, and insufficient security in the execution of learning strategies.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: constructing a heterogeneous multi-agent, multi-objective task environment, and presetting a maximum number of objectives. When the actual target quantity No more than When there are insufficient target slots, zero-fill them and mark them as invalid targets; based on the agent's position, type, and... The target preference vector forms node features, and a dynamic graph is generated based on the communication distance. Node embeddings and graph embeddings are obtained through the GATv2 graph attention encoder. The Actor network outputs the target task selection, and the Critic network outputs the Q-value vector according to the target slot dimension. The graph embedding is used as the state input of the hybrid network, and the Q-value vectors of multiple agents are mixed according to the target slot dimension. The team target preference is then obtained through the preference aggregation network and scalarized training is performed. When executing the target task, the Astar path planning and path planning neural network are combined, supplemented by replanning caching, staggered replanning, collision slip, and perturbation mechanisms to obtain a safe and computationally efficient movement direction.

[0007] Compared with the prior art, the present invention has at least the following beneficial effects: First, it uses a communication graph and a graph attention encoder to express the relationships between agents, and forms a fixed-dimensional graph embedding through global average pooling, so that the state input dimension of the hybrid network does not change with the number of agents; at the same time, it presets a maximum target number. The system employs several mechanisms: First, it fills in invalid target slots and masks them to keep the target dimension of the task selection network and the value network fixed. Second, it uses Q-value vectors based on the target slot dimension and team target preference attention aggregation to balance the trade-offs between different targets and suppress ineffective multi-agent grouping. Third, it sets completion judgment and action masks for expert-type targets and collaborative targets respectively, allowing heterogeneous role constraints to directly enter the strategy selection process. Fourth, the hybrid path planner uses Astar to ensure path reliability in the early stages of training, and uses a path planning neural network to reduce the overhead of repeated planning after the network matures, while improving execution continuity through caching and glide mechanisms. Attached Figure Description

[0008] Figure 1 This is a flowchart illustrating the overall process of the heterogeneous multi-agent cooperative task planning method based on graph attention networks and hybrid path planning of the present invention.

[0009] Figure 2 This is a schematic diagram of the structure of the attention network, actor network, critic network, mixer network, and preference aggregation network of the present invention;

[0010] Figure 3 This is a flowchart of the hybrid path planner of the present invention;

[0011] Figure 4This is a diagram showing the multi-round simulation evaluation results of the trained model in this embodiment of the invention;

[0012] Figure 5 This is a graph showing the task completion progress and communication metrics under a controlled verification scenario in this embodiment of the invention.

[0013] Figure 6 This is a trajectory diagram of multi-agent hybrid path planning in a controlled verification scenario according to an embodiment of the present invention. Detailed Implementation

[0014] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments described herein are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0015] like Figure 1 As shown, the method of the present invention first constructs a heterogeneous multi-agent multi-objective environment and builds a dynamic graph based on the distance threshold between agents; then, the GATv2 graph attention network extracts node embeddings and global graph embeddings, and the Actor, Critic, Mixer and preference aggregation network complete the task selection and team value update; finally, the hybrid path planner is invoked to execute the movement according to the selected objective, and the execution experience is recycled for policy training.

[0016] Example 1: Task Environment and Communication Graph Construction.

[0017] In one embodiment, the task environment is a two-dimensional continuous space, with the spatial scale denoted as . The environment includes Individual agents, One actual target and several obstacles, among which , The preset maximum target quantity. When Less than At that time, for the first To the Each target slot is zero-filled and marked as an invalid target. Agents are classified into several types, such as exploratory, coordinating, and expert; targets are classified into expert targets and collaborative targets. Expert targets are completed only by expert agents, while collaborative targets require at least one basic type agent and at least one other specified type agent to reach the target neighborhood together, and the number of participating agents is not less than a preset minimum.

[0018] For each intelligent agent Construct node features .in, This represents the current position of the i-th agent. Indicates the type of the i-th agent. Let i represent the target preference vector of the i-th agent; the first part of the target preference vector... One element corresponds to the actual target, and the remaining elements are set to zero and correspond to invalid target slots. If the agent and intelligent agents The Euclidean distance between them is no greater than the communication distance threshold. Then, a bidirectional communication edge is established between the two; at the same time, a self-loop edge is established for each node, thus obtaining the communication graph.

[0019] Example 2: Graph Coding and Task Action Selection.

[0020] like Figure 2 As shown, the network structure in this embodiment takes the communication graph as input, the graph attention encoder outputs the node embedding and team graph embedding of each agent; the Actor network outputs the target selection action based on the node embedding and the target preference vector; the Critic network estimates the Q-value vector according to the target slot dimension; the Mixer network performs team value mixing based on the graph embedding; and the preference aggregation network further generates team target preference weights, thereby realizing scalable value estimation in the multi-objective task planning process.

[0021] The communication graph is input into a GATv2 graph attention encoder. The encoder includes a first GATv2 convolutional layer and a second GATv2 convolutional layer. The first GATv2 convolutional layer employs multi-head attention and concatenates the outputs of multiple attention heads. The second GATv2 convolutional layer outputs fixed-dimensional node embeddings. Global average pooling is performed on all node embeddings to obtain the graph embeddings. Image embedding As a representation of the overall state of the team, its dimensions can be set to fixed values ​​such as 64.

[0022] The Actor network receives node embeddings and target preference vectors, concatenates them, inputs the result into a multilayer perceptron, and outputs the agent. correspond The action logic value for each target slot. For invalid target slots and completed targets, the corresponding action logic value is set to unselectable; for non-expert agents, the action logic value corresponding to expert targets is set to unselectable. When all target slots are unselectable, the agent performs a waiting action; otherwise, the agent selects a target task based on the ε-greedy policy or the maximum logic value.

[0023] Example 3: Value mixing and preference aggregation by target slot dimension.

[0024] The Critic network receives node embeddings, one-hot encodings of actions, and target preference vectors, and outputs an agent. correspond The Q-value vectors of each target slot are obtained. The hybrid network uses graph embedding as input to the supernetwork, generates non-negative mixture weights and biases, and monotonically mixes the Q-value vectors of multiple agents according to the target slot dimension, resulting in a dimension of... The team's Q-value vector; the Q-value corresponding to an invalid target slot is masked during value calculation.

[0025] Preference aggregation networks generate query vectors using graph embeddings for each agent. The dimensional objective preference vector generates key and value vectors, which are then aggregated using attention weights to obtain... The team's objective preference vector is defined, and the preference weights corresponding to invalid objective slots are set to zero before renormalization. During training, the team's Q-value vector is weighted and summed with the normalized team objective preference vector to obtain the scalarized team value, which is then used to calculate the temporal difference objective, Critic loss, and Actor loss.

[0026] Example 4: Adaptive Preference Generation.

[0027] During training or execution, a set of all incomplete objectives is acquired and randomly shuffled. The incomplete objectives are then reassigned in a round-robin fashion according to the agent's ID, assigning a primary objective to each agent. A significantly higher preference weight is assigned to the primary objective than to other objectives; for example, the weight of the primary objective is set to 5.0, while the weights of other objectives are set to small random values, and finally, normalization is performed. This strategy can encourage different agents to cover different objectives without explicit prior task assignment, reducing congestion and ineffective waiting caused by multiple agents simultaneously pursuing the same objective. When the actual number of objectives... Less than the preset maximum target number At that time, the first To the The preference weight of an invalid target slot is always set to zero, and it does not participate in the main target allocation and preference normalization.

[0028] Example 5: Hybrid Path Planning.

[0029] like Figure 3 As shown, after receiving the agent's current position, target position, and obstacle information, the hybrid path planner first performs an eight-neighborhood search on the inflated obstacle grid map using the Astar module to obtain look-ahead path points. Simultaneously, the Astar-planned direction serves as a supervisory signal for the path planning network. When the path planning network meets the accuracy and sample size thresholds, it can prioritize outputting the next direction; otherwise, it reverts to the Astar direction. In the current simulation implementation, the path planning network has not been sufficiently trained independently; therefore, the underlying Astar module is used as the path planning execution result.

[0030] For each agent and its selected target, the hybrid path planner first obtains the next look-ahead path point based on the Astar algorithm. The Astar planner discretizes the continuous space into a grid map, expands the grids near the obstacle location according to the obstacle size, and uses eight-neighbor expansion and Euclidean distance heuristics to search for paths. When the starting point or target point falls into an obstacle grid, a breadth-first search is used to find the nearest passable grid as an alternative point.

[0031] Simultaneously, the path planning neural network takes a local map and a position vector as input. The local map includes the current position channel, the target position channel, and the obstacle channel. The position vector includes the normalized current position, the normalized target position, the normalized distance, and the normalized direction angle. The path planning neural network is trained using the direction obtained from ASTAR planning as the supervision label, and the preferred loss function is direction cosine similarity loss. When the network accuracy reaches a preset threshold and the number of training samples reaches a preset number, the network-predicted direction is used preferentially; otherwise, the ASTAR direction is used.

[0032] To reduce the computational burden caused by simultaneous planning by multiple agents, a planning cache is maintained for each agent, containing the previous movement direction and target identifier. The path direction is recalculated only when the target changes, the cache is no longer available, or the staggered replanning cycle is met. If the proposed move is impassable, the agent first attempts to slide along the horizontal axis, then along the vertical axis; if it is still impassable, a small random perturbation is performed and the cache is cleared, forcing a replanning at the next time step.

[0033] Example 6: Target Completion Determination and Reward.

[0034] When an agent enters the target neighborhood, the environment updates the set of neighboring agents corresponding to the target. Expert-type goals require at least one expert agent to be located within the target neighborhood; cooperative goals require the neighborhood to include at least one basic type agent and at least one agent of a specified other type, and satisfy the minimum number of members. Agents that complete the goal are recorded as visiting members for subsequent reward calculations.

[0035] The reward for each agent is based on... Each target slot forms a reward vector, with the reward for an invalid target slot set to zero. The reward includes at least the following components: progress reward based on distance reduction, type matching reward upon approaching the target, collaborative reward for multiple agents appearing together in the neighborhood of a collaborative target, target completion reward, time penalty per step, stall penalty, and congestion penalty near expert targets. By combining the reward vector with the target preference vector, the strategy can dynamically adjust task priorities based on the team's target preferences.

[0036] Example 7: Training and Deployment.

[0037] During training, the current communication graph, actions, rewards, next communication graph, termination flag, and preference matrix are stored in the experience replay buffer. After sampling batch data, the next action is selected based on the target Actor, and the temporal difference objective is calculated using the target Critic and target Mixer. Then, the scalarized value error is calculated using the current Critic, current Mixer, and preference aggregation network, and gradient updates are performed. Actor updates use a differentiable discrete action approximation, such as Gumbel-Softmax, to maximize the team scalarized value. Soft updates are performed on the Critic target network, Mixer target network, and preference aggregation target network at a preset frequency.

[0038] In one specific implementation, the two-dimensional task space is 30×30, the number of obstacles is 6, the maximum number of execution steps is 120; the number of agent types is 3, the number of agents is 8, and the preset maximum number of targets is... =10, the actual target quantity in this embodiment =10, including 2 expert-type objectives and 8 collaborative objectives; the communication distance threshold is 10.0, the GATv2 hidden dimension is 64, the output graph embedding dimension is 64, the number of attention heads is 4, the experience replay buffer capacity is 50,000, the batch size is 32, and the discount factor is 0.99. When the actual number of objectives is less than 10, the remaining objective slots are zero-padded, and the corresponding objective slots are masked in the Actor, Critic, Mixer, and preference aggregation networks. The above parameters are only examples, and those skilled in the art can adjust them according to the task scale and computing resources.

[0039] like Figure 4 As shown, under the above parameter configuration, the trained GATv2 task selection model is loaded for multi-round simulation evaluation. The team reward, task completion rate, number of execution steps, and path planning module activation status under different evaluation rounds can be obtained, which can be used to illustrate the training and execution effect of the present invention in heterogeneous multi-agent task planning.

[0040] like Figure 5 As shown, in the controlled verification scenario, the task completion rate gradually increases with the increase of the number of execution steps. At the same time, the number of communication edges reflects the change in the collaborative relationship between agents. The path planning module remains enabled, indicating that the present invention can drive multiple agents to complete expert-type goals and collaborative goals under dynamic communication graphs and target constraints.

[0041] like Figure 6As shown, in a controlled verification scenario, different types of agents bypass obstacles along the planned path and reach the corresponding target area. The final communication relationship is represented by a dashed line, which reflects the role of the hybrid path planner, replanning cache, staggered update and collision slip correction mechanism in the multi-agent collaborative execution process.

[0042] This invention can also be implemented as a system, an electronic device, and a computer-readable storage medium. The system includes an environment construction module, a communication graph construction module, a graph encoding module, a task action selection module, a path planning module, a target completion determination module, and a strategy training module. The electronic device includes a processor and a memory, wherein a program stored in the memory, when executed by the processor, implements the above-described method. The computer-readable storage medium stores a computer program, which, when executed by the processor, implements the above-described method.

[0043] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A heterogeneous multi-agent cooperative task planning method based on graph attention networks and hybrid path planning, characterized in that, include: A two-dimensional continuous task environment is constructed, including obstacles, multiple heterogeneous agents, and expert and collaborative goals. A maximum number of goals is preset, and insufficient goal slots are filled with zeros and marked as invalid goals. Node features are constructed based on the position, type, and goal preference vectors of each agent, and an agent communication graph is built according to a communication distance threshold. The communication graph is input into a graph attention network encoder to obtain node embeddings and team graph embeddings. The Actor network outputs the task action logic value of the goal slot based on the node embeddings and goal preference vectors, and performs masking based on invalid goals, goal completion status, and agent type constraints to select the target task. A hybrid path planner combining Astar path planning and a path planning neural network is invoked based on the goal task to obtain the next movement direction. The goal completion status is determined based on type matching and collaboration relationships, and a goal slot reward vector is constructed. Reinforcement learning training is performed using experience replay, Critic network, hybrid network, and preference aggregation network, and the target network is soft-updated.

2. The method according to claim 1, characterized in that, The effective elements of the target preference vector correspond to the actual target, and the elements corresponding to the invalid target slots are zero; when the Euclidean distance between two agents is not greater than the communication distance threshold, a bidirectional edge is established, and a self-loop edge is established for each agent node; the graph attention network encoder includes two GATv2 convolutional layers, the first layer uses multi-attention head splicing, the second layer outputs fixed-dimensional node embeddings, and generates team graph embeddings whose dimensions do not change with the number of agents through global average pooling.

3. The method according to claim 1, characterized in that, The Actor network concatenates the node embeddings with the target preference vectors and inputs them into a multilayer perceptron, outputting the action logic values ​​for each target slot. When a target slot is invalid, the target has been completed, or the agent does not meet the type conditions of an expert target, the corresponding action logic value is set to an unselectable state. When all target slots are unselectable, the agent performs a waiting action.

4. The method according to claim 1, characterized in that, The Critic network concatenates node embedding, action one-hot encoding, and target preference vector to output the Q-value vector of each agent arranged according to the target slot; the hybrid network takes team graph embedding as the supernetwork input, generates non-negative hybrid weights and biases, and monotonically mixes the Q-value vectors of multiple agents according to the target slot to obtain the team Q-value vector. The preference aggregation network generates query vectors using team graph embedding, generates key vectors and value vectors using the target preference vectors of each agent, obtains team target preference vectors through attention aggregation, and calculates Critic loss and Actor loss by weighted summation of team Q-value vectors, wherein invalid target slots are masked.

5. The method according to claim 1, characterized in that, The Astar path planning discretizes the continuous task space into a grid map, labels obstacle grids and their neighborhoods, and finds the nearest passable grid through breadth-first search when the start or end point is located in an obstacle area. The path is planned based on the eight-neighborhood expansion and Euclidean distance heuristic function. The path planning neural network takes a local map containing the current position channel, the target position channel, and the obstacle channel, as well as the position vector, as input, and the Astar planning direction as the supervision signal. When the network accuracy and the number of training samples reach the threshold, the network prediction direction is preferred; otherwise, the Astar direction is used.

6. The method according to claim 1 or 5, characterized in that, The hybrid path planner caches the previous movement direction and target task identifier for each agent. It replans when the target changes, the cache is missing, or the staggered replanning cycle is reached; otherwise, it uses the cached direction. If the direct movement position is impassable, it attempts to slide laterally and then longitudinally in turn. If the position is still impassable, it performs a perturbation and clears the cache to trigger the next replanning step.

7. The method according to claim 1, characterized in that, Expert-type objectives are completed only when an expert agent reaches the target neighborhood, while collaborative objectives are completed only when at least one basic type agent and at least one designated other type agent simultaneously reach the target neighborhood and the number of members reaches a threshold. The reward vector includes distance progress reward, type matching reward, collaboration reward, objective completion reward, time penalty, lag penalty, and congestion penalty. The reward for invalid objective slots is zero. The system assigns primary objectives to different agents by polling for uncompleted objectives, sets higher preference weights for primary objectives, and sets zero weights and normalizes invalid objective slots to suppress invalid agent aggregation.

8. A heterogeneous multi-agent cooperative task planning system based on graph attention networks and hybrid path planning, characterized in that, It includes an environment construction module, a communication graph construction module, a graph encoding module, a task action selection module, a path planning module, a target completion determination module, and a strategy training module, each module being configured to perform the method described in any one of claims 1 to 7.

9. An electronic device comprising a processor and a memory, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, it implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 7.

Images