Training method and device for intelligent agent cluster path planning model

Abstract

The invention discloses an agent cluster path planning model training method and device, and the method comprises the steps: constructing a first state matrix of an agent at a current time step, inputting the first state matrix into an initial reinforcement learning model, and predicting the second motion information of each agent at a next time step; based on the second state matrix and the first state matrix, determining a reward score of executing the task by the agent cluster; constructing time sequence data formed by the target tetrad, and inputting the time sequence data into each local Q network to generate a local Q value; combining the local Q values into a global Q value through a mixed Q network; and continuing to train by using the updated local Q network and the mixed Q network to obtain an intelligent agent cluster path planning model. According to the scheme, the hybrid Q network has timing modeling and attention weight distribution capabilities, each agent makes a decision independently, and decentralized control is realized; time sequence dependence is learned by adopting time sequence data, so that path generation is more flexible.

Description

Technical Field

[0001] This solution relates to the field of reinforcement learning technology, and in particular to a training method and apparatus for an agent cluster path planning model. Background Technology

[0002] In recent years, with the rapid development of intelligent agent technology and the increasing demand for intelligent logistics, collaborative path planning for intelligent agent clusters in material delivery tasks has become a research hotspot in the fields of intelligent transportation and emergency response. This problem not only involves the basic theory of multi-agent path planning, but also faces multi-dimensional and complex challenges such as swarm intelligence, dynamic obstacle avoidance, communication coordination, and task allocation.

[0003] Early multi-agent path planning methods mostly adopted a centralized path scheduling framework. These methods schedule the paths of all agents through a unified control platform, achieving efficient path conflict avoidance and global optimal control in small-scale tasks. However, as the system scales up and the complexity of the environment increases, centralized methods face problems such as high computational complexity, large communication overhead, and poor robustness, making it difficult to adapt to real-world application scenarios with dynamic changes and high-frequency disturbances.

[0004] To address this, researchers have proposed various scalable and responsive path planning algorithms, among which Conflict-Based Search (CBS) is widely used due to its hierarchical conflict handling mechanism. This algorithm achieves a balance between computational efficiency and path optimality by resolving conflicts at a high level and independently planning paths for each agent at a low level. Yan Xingyu et al. improved the CBS algorithm based on a conflict avoidance strategy, significantly reducing the solution time in narrow passages.

[0005] Multi-agent path collaborative planning, a key technology for achieving efficient and intelligent agent-based material delivery, is gradually evolving from traditional graph search algorithms to deep reinforcement learning. While centralized scheduling methods still offer advantages in control accuracy and theoretical interpretability, their application is limited in highly dynamic and uncertain environments. Distributed reinforcement learning methods such as VDN and QMIX, as well as fusion methods combining communication mechanisms, environmental modeling, and imitation learning, provide effective pathways for constructing highly adaptable, stable, and scalable path collaborative systems. However, most existing research focuses on theoretical verification or rule-based environments, lacking systematic modeling and testing under real-world terrain, complex constraints, and task uncertainties. Therefore, for real-world applications such as material delivery in mountainous areas with high dynamics and task uncertainties, it is necessary to explore more general, stable, and real-time methodological frameworks. Summary of the Invention

[0006] This solution aims to at least address the technical problems existing in the prior art. To this end, the first aspect of this invention proposes a training method for an intelligent agent cluster path planning model, the method comprising:

[0007] During the process of the intelligent agent cluster executing the material delivery task, a first state matrix is ​​constructed for each intelligent agent at the current time step. The first state matrix includes the first environmental information and the first action information of the intelligent agent. The first environmental information includes the position information of the intelligent agent and the relative distance to the task destination. The first action information includes the yaw angle and pitch angle of the intelligent agent. The task destination of the material delivery task of each intelligent agent is different.

[0008] The first state matrix is ​​input into the initial reinforcement learning model to predict the second action information of each agent in the next time step.

[0009] Each of the intelligent agents is controlled to fly according to the second action information, and at the end of the next time step, the second environmental information set of the intelligent agent cluster is obtained; based on the second environmental information and the second action information, the second state matrix of the next time step is constructed.

[0010] The reward score for the agent cluster to perform this task is determined based on the second state matrix and the first state matrix;

[0011] Based on the reward score and state matrix of the agent at each time step within the target time period, a time series data consisting of target quadruples is constructed. The target quadruples include the environmental information, action information, reward score, and environmental information of each agent at the current time step and the next time step.

[0012] The time series data is stored in a buffer, and the agent is controlled to continue to perform the material delivery task until the amount of data in the buffer reaches the preset batch processing data amount, at which point the model is trained.

[0013] During each training session, the time-series data of each agent is input into the local Q-networks of the initial reinforcement learning model to generate local Q-values; the local Q-values ​​of each agent are combined into a global Q-value through a hybrid Q-network with monotonicity constraints.

[0014] The loss value for this training is calculated based on the global Q-value and the target loss function, and the local Q-network and the hybrid Q-network are updated according to the loss value.

[0015] The updated local Q-network and hybrid Q-network are used to continue training until the preset termination condition is met, at which point the training ends and the agent cluster path planning model is obtained.

[0016] Optionally, the first state matrix includes the agent's first current position, a first altitude relative to the terrain, a first relative distance to the mission destination, a first boundary distance to the center of the nearest danger zone, and a first safe distance to other agents; the first action information includes the agent's first yaw angle and first pitch angle; determining the reward score for the agent cluster performing this task based on the second state matrix and the first state matrix includes:

[0017] The second environmental information and second action information of the agent are obtained from the second state matrix. The second environmental information includes: the second current position, the second altitude relative to the terrain, the second relative distance to the mission destination, the second boundary distance to the center of the nearest danger zone, and the second safe distance to other agents. The second action information includes the agent's second yaw angle and second pitch angle.

[0018] When the second relative distance between the agent and the task destination is 0, the agent is given a sparse reward.

[0019] Determine the magnitude relationship between the second relative distance and the first relative distance; when the second relative distance is smaller than the first relative distance, give the agent a task guidance reward.

[0020] Determine the relationship between the second safe distance and the preset safe threshold; when the second safe distance is less than the safe threshold, impose a distance penalty on the agent.

[0021] Determine the relationship between the second altitude and the preset target flight altitude range; when the second altitude does not belong to the target flight altitude range, impose a flight altitude penalty on the agent.

[0022] Calculate the yaw angle difference between the second yaw angle and the first yaw angle, and the pitch angle difference between the second pitch angle and the first pitch angle, and determine the path stability penalty of the agent based on the yaw angle difference and the pitch angle difference;

[0023] The obstacle avoidance penalty for the agent is determined based on the second boundary distance and the radius of the danger zone;

[0024] The reward score for the agent to perform this task is determined based on the sparse reward, the task guidance reward, the distance penalty, the flight altitude penalty, the path stability penalty, and the obstacle avoidance penalty.

[0025] Optionally, the step of giving the agent a task-guiding reward when the second relative distance is smaller than the first relative distance includes:

[0026] When the second relative distance is smaller than the first relative distance, the difference between the first safety distance and the second safety distance is determined;

[0027] Obtain the agent's fixed flight speed, determine the quotient between the difference and the fixed flight speed, and use the quotient as the agent's task guidance reward value.

[0028] Optionally, the step of imposing a distance penalty on the agent when the second safe distance is less than the safe threshold includes:

[0029] Obtain the second safe distance between the agent and other agents, and determine the distance difference between the preset safe threshold and the second safe distance;

[0030] If the difference is greater than 0, then the agent is determined to be the target agent whose second safe distance is less than the safe threshold;

[0031] For each target agent, determine the total number of target agents and obtain the distance difference corresponding to each target agent;

[0032] Calculate the quotient between the distance difference between each of the target agents and the security threshold to obtain multiple quotients; calculate the sum of the multiple quotients;

[0033] The distance penalty value of the agent is determined based on the sum and the total number.

[0034] Optionally, the step of inputting the time-series data of each of the agents into the local Q-networks of the initial reinforcement learning model to generate local Q-values ​​includes:

[0035] The time series data is encoded into a fixed-dimensional latent space representation through a linear transformation and ReLU activation function.

[0036] In the gated loop unit module, the first hidden state from the previous moment and the hidden space representation of the current encoded input are received, and the second hidden state is output.

[0037] In the multi-head attention mechanism module, the second hidden state of the current agent is used as the query vector, and the concatenated tensor of the second hidden states of all agents is used as the key and value to calculate the weighted representation vector; the weighted representation vector is then used to generate the local Q value of the agent at the current time through a linear mapping layer.

[0038] Optionally, calculating the loss value for this training based on the global Q-value and the target loss function includes:

[0039] Obtain the global reward score of all agents in the previous time step; determine the expected Q reward when all agents choose actions that maximize the global Q value under the environmental and action information conditions in the next time step.

[0040] The target Q value is determined based on the global reward score, the preset discount factor, and the expected Q return.

[0041] The difference between the global Q-value and the target Q-value is determined based on the temporal difference function, and the loss value for this training is obtained.

[0042] Optionally, before the intelligent agent cluster performs the material delivery task, it also includes:

[0043] In the simulation platform, a 2.5-dimensional high-dimensional grid method is used for environmental modeling, where each grid contains planar and height information to simulate the characteristics of mountainous terrain.

[0044] In the environmental modeling process, terrain data is generated using Perlin noise. By adjusting the amplitude and number of noise layers, a realistic mountain environment is simulated.

[0045] A second aspect of this invention provides a training device for an agent swarm path planning model, the device comprising:

[0046] The first state matrix construction module is used to construct the first state matrix of each agent at the current time step during the execution of material delivery tasks by the agent cluster. The first state matrix includes the first environmental information and the first action information of the agent. The first environmental information includes the position information of the agent and the relative distance to the task destination. The first action information includes the yaw angle and pitch angle of the agent. The task destination of the material delivery task of each agent is different.

[0047] The input module is used to input the first state matrix into the initial reinforcement learning model to predict the second action information of each agent in the next time step;

[0048] The second state matrix construction module is used to control each of the intelligent agents to fly according to the second action information, and to obtain the second environmental information set of the intelligent agent cluster at the end of the next time step; and to construct the second state matrix of the next time step based on the second environmental information and the second action information.

[0049] The reward determination module is used to determine the reward score of the agent cluster for performing this task based on the second state matrix and the first state matrix;

[0050] The time series data construction module is used to construct time series data consisting of target quadruples based on the reward score and state matrix of the agent at each time step within the target time period. The target quadruples include the environmental information, action information, reward score of each agent at the current time step, and the environmental information of the next time step.

[0051] The caching module is used to store the time series data into a buffer and control the agent to continue to execute the material delivery task until the amount of data in the buffer reaches the preset batch processing data amount, at which point the model is trained.

[0052] The training module is used to input the time-series data of each agent into the local Q-networks of the initial reinforcement learning model during each training session to generate local Q-values; and to combine the local Q-values ​​of each agent into a global Q-value through a hybrid Q-network with monotonicity constraints.

[0053] The update module is used to calculate the loss value of this training based on the global Q value and the target loss function, and update the local Q network and the hybrid Q network according to the loss value;

[0054] The model determination module is used to continue training using the updated local Q-network and hybrid Q-network until the preset termination condition is met, at which point the training ends and the agent cluster path planning model is obtained.

[0055] A third aspect of the present invention provides an electronic device comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the training method for the intelligent agent cluster path planning model as described in the first aspect.

[0056] A fourth aspect of the present invention provides a computer-readable storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement a training method for an agent cluster path planning model as described in the first aspect.

[0057] The embodiments of the present invention have the following beneficial effects:

[0058] The training method for the intelligent agent cluster path planning model provided in this embodiment of the invention involves constructing a first state matrix for each agent at the current time step during the execution of a material delivery task by the intelligent agent cluster. The first state matrix includes first environmental information and first action information for each agent. The first environmental information includes the agent's position information and relative distance to the task destination. The first action information includes the agent's yaw angle and pitch angle. Each agent has a different task destination for its material delivery task. The first state matrix is ​​input into an initial reinforcement learning model to predict the second action information for each agent at the next time step. Each agent is controlled to fly according to the second action information, and at the end of the next time step, a second environmental information set of the intelligent agent cluster is acquired. A second state matrix for the next time step is constructed based on the second environmental information and the second action information. The reward score for the intelligent agent cluster performing this task is determined based on the second state matrix and the first state matrix. Based on the reward score and state matrix of the agent at each time step within the target time period, a time series data consisting of target quadruples is constructed. Each target quadruple includes the environmental information, action information, reward score, and environmental information for the next time step for each agent at the current time step. The time series data is stored in a buffer, and the agent continues to execute the material delivery task until the data volume in the buffer reaches a preset batch processing data volume, at which point model training begins. During each training iteration, the time series data of each agent is input into the local Q-networks of the initial reinforcement learning model to generate local Q-values. The local Q-values ​​of each agent are combined into a global Q-value using a hybrid Q-network with monotonicity constraints. The loss value for this training iteration is calculated based on the global Q-value and the target loss function, and the local Q-networks and the hybrid Q-networks are updated according to the loss value. Training continues using the updated local Q-networks and the hybrid Q-networks until a preset termination condition is met, at which point training ends, resulting in an agent cluster path planning model. During the training phase, this scheme collects local observation and action information from multiple agents through a centralized structure and combines it with global state information for collaborative optimization, learning a hybrid Q-network with temporal modeling and attention weight allocation capabilities. During the execution phase, each agent makes independent decisions based on local perception information, achieving decentralized control. Furthermore, using time-series data composed of target quadruples helps learn the temporal dependencies in historical trajectories, making path generation more flexible. Attached Figure Description

[0059] Figure 1 This is a flowchart illustrating the steps of a training method for an intelligent agent cluster path planning model provided in an embodiment of the present invention.

[0060] Figure 2This is a structural block diagram of a training device for an intelligent agent cluster path planning model provided in an embodiment of the present invention. Detailed Implementation

[0061] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present solution, and not all embodiments. Based on the embodiments of the present solution, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present solution.

[0062] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more. Furthermore, the use of "based on" or "according to" implies openness and inclusiveness, because processes, steps, calculations, or other actions "based on" or "according to" one or more of the stated conditions or values ​​may in practice be based on additional conditions or beyond the stated values.

[0063] This invention aims to address the core problems of current multi-agent path planning systems, such as insufficient collaborative performance, path instability, and poor task adaptability in complex and dynamic environments. It particularly focuses on efficient collaborative path planning tasks for multi-agent systems in high-risk areas such as mountainous regions. In such scenarios, agents need to complete complex tasks with real-time and security requirements, such as material delivery, emergency rescue, and post-disaster monitoring, under conditions of high terrain undulations, dynamically hazardous areas, and frequently changing environmental constraints. Traditional methods face significant technical bottlenecks.

[0064] In existing technologies, graph search-based path planning algorithms (such as A* and Dijkstra's algorithm) offer advantages such as strong interpretability and shortest planned paths in static environments. However, they cannot respond promptly to environmental changes in dynamic environments, resulting in a high risk of path failure and making them unsuitable for multi-agent collaborative scenarios. Intelligent optimization algorithms (such as particle swarm optimization, genetic algorithms, and ant colony optimization) possess certain global search capabilities, but they suffer from slow convergence speeds, high computational costs, and poor real-time path generation in complex dynamic environments. Furthermore, these methods lack modeling mechanisms for the collaborative behavior among multiple agents, making them prone to path conflicts or resource competition, severely impacting overall system performance.

[0065] The introduction of deep reinforcement learning techniques has alleviated the aforementioned problems to some extent, especially demonstrating stronger expressive and adaptive capabilities when facing high-dimensional state spaces. However, traditional single-agent reinforcement learning methods cannot meet the collaborative needs of multi-agent systems. Even existing multi-agent methods (such as VDN and QMIX) still suffer from problems such as policy instability, inconsistent agent behavior, and insufficient collective effectiveness when dealing with dynamic task changes and real-world tasks with strong environmental constraints, making it difficult to achieve stable convergence to the task objective and globally optimal path planning.

[0066] This invention proposes a multi-agent hybrid value function learning framework—AttentionRNN-QMIX algorithm—that integrates an attention mechanism to overcome the aforementioned technical bottlenecks. This method introduces a local Q-network structure (AttentionRNNAgent) based on an attention mechanism into the original QMIX algorithm. By guiding individual agents to more effectively extract task-related dynamic environmental information and the states of neighboring agents, it enhances the contextual understanding ability in individual decision-making and alleviates the information loss problem caused by observation limitations in multi-agent systems.

[0067] The AttentionRNN-QMIX algorithm employs a centralized training and decentralized execution (CTDE) framework. During training, it leverages global state information through a nonlinear hybrid network to constrain the aggregation of individual Q-values, ensuring the consistency and stability of the learned global policy. During execution, each agent still makes decisions based on its own local observations, thus guaranteeing the system's distributed deployment and loosely coupled communication characteristics. By introducing the attention mechanism, the system can dynamically focus on key task factors in local decision-making and make more temporally relevant path planning decisions based on historical information. This significantly improves the adaptability and collaborative efficiency of multi-agent systems in the face of complex environmental changes such as dynamic targets, moving obstacles, and the spread of dangerous areas.

[0068] Figure 1 This is a flowchart illustrating the steps of a training method for an intelligent agent cluster path planning model provided in an embodiment of the present invention. Figure 1 As shown, the method includes the following steps:

[0069] Step 101: During the process of the intelligent agent cluster executing the material delivery task, construct the first state matrix of each intelligent agent at the current time step. The first state matrix includes the first environmental information and the first action information of the intelligent agent. The first environmental information includes the position information of the intelligent agent and the relative distance to the task destination. The first action information includes the yaw angle and pitch angle of the intelligent agent. The task destination of the material delivery task of each intelligent agent is different.

[0070] In the path planning system of this invention, a multi-agent deep reinforcement learning (MARL) framework is employed. Each agent is treated as an autonomous agent, achieving collaborative control through distributed decision-making and centralized training. In this system, each agent possesses independent local observation information and a local policy network. The local value functions of all individuals are combined into a global value function through an additive hybrid network, thereby achieving the learning of the optimal decision policy for the joint task.

[0071] Environmental information represents the agent's perception of its surroundings, while action information represents the agent's current posture or motion state. Specifically, the first environmental information includes the agent's position, velocity, relative terrain elevation, the position of the task target, its relative distance to the task destination, historical actions, and the boundary distance to the nearest danger zone. This information is uniformly normalized to the [-1,1] interval through a linear transformation to adapt to the input requirements of reinforcement learning.

[0072] Location information consists of coordinates, used to pinpoint the agent's exact location on the map or within the task area. The relative distance to the task destination is the straight-line distance from the agent's current location to its destination. Since each agent has a different task destination, this distance is dynamically changing and directly impacts path planning.

[0073] Each agent's action space is discrete, consisting of two parts: yaw angle (ψ) and pitch angle (φ). The yaw angle is divided into 20 equal increments within the range [0, π / 2), while the pitch angle is divided into 5 equal increments within the range [-14°, +14°]. In this way, the agent can choose different flight directions and altitudes for path planning.

[0074] For example, agent A and agent B are delivering supplies. Agent A's destination is a hospital. Its first state matrix might be [x = 10, y = 20, distance = 5km, yaw angle = 30°, pitch angle = 0°];

[0075] Agent B's destination is the school, and its first state matrix might be [x = 15, y = 5, distance = 3km, yaw angle = -15°, pitch angle = 0°].

[0076] Step 102: Input the first state matrix into the initial reinforcement learning model to predict the second action information of each agent in the next time step.

[0077] The initial reinforcement learning model is an untrained model, such as the QMIX model. The model predicts the second action information, namely the agent's yaw and pitch angles in the next time step, based on the first state matrix input.

[0078] This system employs the QMIX algorithm as the core collaborative policy learning mechanism within a reinforcement learning framework. QMIX is a typical centralized training, distributed execution (CTDE) method that can effectively solve the collaborative decision-making problem among multiple agents. It optimizes the overall policy through a centralized training mechanism while ensuring that each agent makes decisions based only on local observations.

[0079] Step 103: Control each of the intelligent agents to fly according to the second action information, and obtain the second environmental information set of the intelligent agent cluster at the end of the next time step; construct the second state matrix of the next time step based on the second environmental information and the second action information.

[0080] The agent is controlled to fly according to the predicted second action information (e.g., the agent adjusts its course). This is the environment interaction phase in reinforcement learning.

[0081] At the end of the next time step, a second set of environmental information is collected from the agent cluster, and a second state matrix is ​​constructed based on this information. The second state matrix includes second environmental information and second action information.

[0082] Step 104: Determine the reward score for the intelligent agent cluster to perform this task based on the second state matrix and the first state matrix.

[0083] Rewards can be designed based on task objectives, such as positive rewards (e.g., approaching the target point, avoiding collisions, completing the task), and negative penalties (e.g., collisions, deviating from the target).

[0084] The reward function can be a simple linear combination or a complex nonlinear function.

[0085] Step 105: Construct time series data consisting of target quadruples based on the reward score and state matrix of the agent at each time step within the target time period. The target quadruples include the environmental information, action information, reward score, and environmental information of the next time step for each agent at the current time step.

[0086] The system collects experience sequences by observing the interactive behaviors of multiple agents in a simulation environment, and constructs time series data composed of four-tuples. Each sequence records the state transition trajectory, behavior choices, and corresponding reward feedback of all agents over a period of time.

[0087] Specifically, a quadruple can be represented as: in, This indicates the environmental information at the current time step. Indicates action information, Indicates the reward score. This indicates the environmental information for the next time step.

[0088] Step 106: Store the time series data into a buffer and control the agent to continue executing the material delivery task until the amount of data in the buffer reaches the preset batch processing data amount, then train the model.

[0089] Time series data is cached in an experience replay buffer. When the amount of data in the buffer reaches the preset batch size, these samples are fed into the training network for model training and gradient updates.

[0090] Step 107: During each training session, the time-series data of each agent is input into the local Q-networks of the initial reinforcement learning model to generate local Q-values; the local Q-values ​​of each agent are combined into a global Q-value through a hybrid Q-network with monotonicity constraints.

[0091] The core idea of ​​this invention is to construct an independent local Q-network within each agent. This network generates a local action value function, or local Q-value, based on current local observations and historical state information. The local Q-values ​​of all agents are combined into a global action value function, or global Q-value, through an additive hybrid network with monotonicity constraints.

[0092] Specifically, the system uses the sampled experience replay as input, inputs the local observations of each agent into the AttentionRNNAgent network, calculates the local Q-value sequence, and then feeds it into the hybrid network to calculate the global Q-value at the current time step. The QMIX hybrid network has a monotonicity constraint, which requires:

[0093]

[0094] Among them, Q tot Let Q represent the global action value function, and let Q represent the total value of the joint policy of all agents. i Let represent the local action value function of the i-th agent.

[0095] This constraint ensures that each local strategy can select actions through a greedy strategy during the distributed execution phase, thereby maximizing the global Q value without relying on global information.

[0096] Step 108: Calculate the loss value for this training based on the global Q value and the target loss function, and update the local Q network and the hybrid Q network according to the loss value.

[0097] Specifically, the hybrid Q network includes weights and biases, and dynamically adjusts the importance of each local Q value in the joint task at each time step according to the loss value, resulting in an updated local Q network and a hybrid Q network.

[0098] Step 109: Continue training using the updated local Q-network and hybrid Q-network until the preset termination condition is met, and then end the training to obtain the agent cluster path planning model.

[0099] The preset termination conditions may include reaching the maximum number of training steps, the loss value being less than the preset loss value, or reaching the performance threshold. The trained agent cluster path planning model specifically includes a trained local Q network and a hybrid Q network.

[0100] In this embodiment of the invention, the main modules of the QMIX algorithm include a local Q-network (AttentionRNNAgent), a hybrid Q-network (Mixing Network), a state decoupling supernetwork, an experience replay buffer, a TD target calculation module, and a distributed execution interface. Through the collaborative work of these modules, the system can learn stable and efficient multi-agent policies in a three-dimensional space with high dynamic uncertainty, achieving unified modeling and joint optimization of tasks such as path planning, cooperative control, and obstacle avoidance decision-making.

[0101] The QMIX algorithm serves as the training backbone of this system, working in conjunction with an attention-based local Q-network to achieve a complete path from local policy learning to global collaborative optimization. This approach retains the flexibility of autonomous decision-making in distributed agents while ensuring global policy consistency through centralized training, making it a key support for the system's efficient path collaborative planning capabilities.

[0102] This invention introduces an improved algorithm—Attention RecurrentQMIX (ARQMIX)—which incorporates attention mechanisms and recurrent neural structures. ARQMIX demonstrates superior performance and broad adaptability in multi-agent cooperative path planning tasks. It inherits the advantages of the centralized training and decentralized execution (CTDE) framework of the traditional QMIX algorithm, while introducing an attention mechanism during agent policy modeling to achieve dynamic information aggregation. The GRU structure enhances the modeling ability for temporal dependencies, significantly improving policy stability, adaptability, and cooperative capabilities. The algorithm constructs representations of local observations, uses attention modules to extract important information interaction features between the current agent and other agents, and leverages recurrent units to effectively memorize and compress historical observation information. Finally, it achieves nonlinear weighted fusion of individual Q-values ​​in a hybrid Q-network to obtain a global Q-value function, enabling joint policy optimization among multiple agents in complex environments.

[0103] In simulation experiments, ARQMIX demonstrated more stable performance on certain metrics compared to traditional QMIX and VDN algorithms. For example, in mountainous environments with high terrain complexity, ARQMIX achieved slightly higher average performance in task completion rate and trajectory consistency, with fewer cooperative conflicts between agents, more regular overall paths, and better ability to cope with some sudden environmental disturbances. Furthermore, under various task distribution scenarios, the system maintained a relatively balanced completion quality, indicating that it possesses good policy transfer capabilities within a certain range.

[0104] Compared to traditional reinforcement learning methods such as VDN, IPPO, or single-agent PPO algorithms, ARQMIX significantly improves both policy modeling and environmental adaptability. VDN, limited by linear additivity when modeling agent cooperation, struggles to capture complex dependencies between individuals; while IPPO, although supporting parameter sharing and independent training, lacks the ability to model the global state, making it difficult to achieve cooperative optimality. In contrast, ARQMIX, by fusing attention mechanisms and a global hybrid Q-network, achieves joint optimization of individual policies during training while maintaining each agent's independent decision-making ability based on its local observations during execution, thus balancing cooperation and distributed control capabilities.

[0105] During the training phase, the system collects local observations and action information from multiple agents through a centralized structure, and combines this with global state information for collaborative optimization, learning a hybrid Q-network with temporal modeling capabilities and attention weight allocation capabilities. During the execution phase, each agent makes independent decisions based on local perceptual information, achieving decentralized control. This framework not only retains the policy collaboration advantages of QMIX, but also enhances the model's ability to extract key environmental information through an attention mechanism, while the GRU structure helps learn temporal dependencies in historical trajectories, making path generation more flexible.

[0106] This method no longer relies on preset flight formations or control sequences, but instead dynamically generates path strategies through continuous interaction with the environment. ARQMIX's attention mechanism allows the agent to learn the importance of key state features during training, improving its responsiveness to complex local terrain and multi-task interference. The introduction of a cyclic structure enhances the ability to model time dependencies, contributing to improved path continuity and collaborative consistency. Therefore, compared to traditional rule-based methods, this invention is more suitable for unstructured scenarios such as frequently changing targets and dynamically generated obstacles, offering better adaptability and adjustment flexibility.

[0107] Furthermore, the path planning strategy of this invention exhibits higher robustness and generalization ability within a certain range. When facing uncertainties such as dynamic terrain, the ARQMIX-based learning strategy can autonomously adjust local behavior to maintain the overall coordination of the task objective and reduce the risk of path failure. Traditional solutions, on the other hand, rely too heavily on rule definitions and global control instructions, making it difficult to adapt quickly to such changes. Although this method has relatively higher computational complexity and model training requirements, it demonstrates certain technical advantages and application potential in specific scenarios from the perspectives of path flexibility, task execution continuity, and collaborative efficiency.

[0108] In summary, the training method for the intelligent agent cluster path planning model provided in this embodiment of the invention constructs a first state matrix for each intelligent agent at the current time step during the execution of a material delivery task by the intelligent agent cluster. The first state matrix includes the first environmental information and first action information of the intelligent agent. The first environmental information includes the position information of the intelligent agent and its relative distance to the task destination. The first action information includes the yaw angle and pitch angle of the intelligent agent. The task destinations for each intelligent agent's material delivery task are different. The first state matrix is ​​input into an initial reinforcement learning model to predict the second action information of each intelligent agent at the next time step. Each intelligent agent is controlled to fly according to the second action information, and at the end of the next time step, the second environmental information set of the intelligent agent cluster is acquired. A second state matrix for the next time step is constructed based on the second environmental information and the second action information. The reward for the intelligent agent cluster to perform this task is determined based on the second state matrix and the first state matrix. The process involves: constructing time-series data composed of target quadruples based on the reward score and state matrix of each agent at each time step within the target time period. Each target quadruple includes the environmental information, action information, reward score, and environmental information for the next time step for each agent at the current time step. The time-series data is stored in a buffer, and the agent continues to execute the material delivery task until the data volume in the buffer reaches a preset batch processing data volume, at which point model training begins. During each training iteration, the time-series data of each agent is input into the local Q-networks of the initial reinforcement learning model to generate local Q-values. The local Q-values ​​of each agent are combined into a global Q-value using a hybrid Q-network with monotonicity constraints. The loss value for this training iteration is calculated based on the global Q-value and the target loss function, and the local Q-networks and hybrid Q-networks are updated according to the loss value. Training continues using the updated local Q-networks and hybrid Q-networks until a preset termination condition is met, resulting in an agent cluster path planning model. During the training phase, this scheme collects local observation and action information from multiple agents through a centralized structure and combines it with global state information for collaborative optimization, learning a hybrid Q-network with temporal modeling and attention weight allocation capabilities. During the execution phase, each agent makes independent decisions based on local perception information, achieving decentralized control. Furthermore, using time-series data composed of target quadruples helps learn the temporal dependencies in historical trajectories, making path generation more flexible.

[0109] As an optional embodiment, the first state matrix includes the agent's first current position, a first altitude relative to the terrain, a first relative distance to the mission destination, a first boundary distance to the center of the nearest danger zone, and a first safe distance to other agents; the first action information includes the agent's first yaw angle and first pitch angle; step 104 includes:

[0110] Step 1041: Obtain the second environmental information and second action information of the agent from the second state matrix. The second environmental information includes: the second current position, the second altitude relative to the terrain, the second relative distance to the mission destination, the second boundary distance to the center of the nearest danger zone, and the second safe distance to other agents. The second action information includes the second yaw angle and the second pitch angle of the agent.

[0111] The second state matrix contains the same information as the first state matrix, including environmental information and action information.

[0112] In designing the reward function, this invention employs a multi-dimensional composite reward function to guide agents in completing tasks, avoiding collisions, maintaining flight path smoothness, and avoiding danger zones. Specifically, the task completion reward (Rtask) provides a sparse reward when the agent successfully completes the task; the flight path guidance reward (Rguide) provides dense feedback by comparing the target distance between the current step and the previous time step; and the collision risk penalty (Rcoll) penalizes agents based on the minimum safe distance between them to prevent collisions. Other rewards include flight altitude control rewards (Rheight), path smoothness rewards (Rsmooth), and danger zone avoidance rewards (Rdanger). These reward items comprehensively consider factors such as task completion efficiency, path smoothness, and safety, ensuring that the agent swarm can coordinate and execute tasks efficiently in complex environments.

[0113] Step 1042: When the second relative distance between the agent and the task destination is 0, the agent is given a sparse reward.

[0114] When the i-th agent successfully reaches its target point, the system awards a one-time sparse reward to reinforce the agent's learning objective for completing the task. This reward is triggered only at the instant the agent reaches the task destination. The sparse reward R... task，i Specifically, it is calculated using the following method:

[0115]

[0116] Here, UAV represents the intelligent agent. If the agent reaches the task destination, it receives a one-time reward of 500 points; if it fails to reach the destination, it receives no reward. This setting ensures the learning accuracy requirement while retaining a certain degree of robustness.

[0117] Step 1043: Determine the relationship between the second relative distance and the first relative distance. When the second relative distance is smaller than the first relative distance, give the agent a task guidance reward.

[0118] To avoid training stagnation caused by sparse rewards, the system introduces intensive task-guided rewards R. guide，i By comparing the change in the agent's distance from the target point in the current step with that in the previous time step, positive rewards are given.

[0119] As an optional embodiment, the step of giving the agent a task-guiding reward when the second relative distance is smaller than the first relative distance includes:

[0120] Step 10431: When the second relative distance is smaller than the first relative distance, determine the difference between the first safe distance and the second safe distance;

[0121] Step 10432: Obtain the fixed flight speed of the agent, determine the quotient between the difference and the fixed flight speed, and use the quotient as the task guidance reward value of the agent.

[0122] In steps 10431-10432, the task guidance reward value is calculated using the following formula:

[0123]

[0124] Where, d prev and d curr Let d be the Euclidean distance between the agent and the target point in the previous and current steps, respectively. prev Let d be the first relative distance. curr This is the second relative distance.

[0125] v represents the agent's flight speed, for example, 100 m / s. This encourages the agent to continuously move towards the goal, receiving positive feedback even before the task is completed.

[0126] Step 1044: Determine the relationship between the second safe distance and the preset safe threshold. When the second safe distance is less than the safe threshold, impose a distance penalty on the agent.

[0127] To avoid spatial conflicts between agents, a distance-based collision penalty function is designed. If the distance d is any other agent... ijLess than the safety threshold d safe The system will impose a penalty.

[0128] d safe It can be preset according to the actual situation, for example, it can be set to 100 meters.

[0129] As an optional embodiment, the step of imposing a distance penalty on the agent when the second safe distance is less than the safe threshold includes:

[0130] Step 10441: Obtain the second safe distance between the agent and other agents, and determine the distance difference between the preset safe threshold and the second safe distance;

[0131] Step 10442: If the difference is greater than 0, then the agent is determined to be the target agent whose second safe distance is less than the safe threshold;

[0132] Step 10443: For each target agent, determine the total number of target agents and obtain the distance difference corresponding to each target agent;

[0133] Step 10444: Calculate the quotient between the distance difference of each target agent and the security threshold to obtain multiple quotients; calculate the sum of the multiple quotients;

[0134] Step 10445: Determine the distance penalty value of the agent based on the sum and the total number.

[0135] In steps 10441-10445, the distance penalty can be calculated according to the following formula:

[0136]

[0137] Among them, C coll，i d represents the distance penalty. safe This represents the preset safety threshold, N represents the total number of target agents, and d ij This represents the distance between the current agent i and other agents j.

[0138] In the above formula, when the distance difference d between the preset safety threshold and the second safety distance... safe -d ij When the distance difference is less than 0, the distance penalty value is 0. When the distance difference is greater than 0, the distance penalty value is greater than 0.

[0139] Step 1045: Determine the relationship between the second altitude and the preset target flight altitude range. When the second altitude does not belong to the target flight altitude range, impose a flight altitude penalty on the intelligent agent.

[0140] To ensure flight safety and improve energy efficiency, the system sets the desired flight altitude range to 100m-300m. This is based on the agent's altitude relative to the terrain, h. i Penalties are applied in segments. Specifically, the flight altitude penalty R... height,i The calculation formula is:

[0141]

[0142] Specifically, it means that when the agent's second height h i A penalty of 50 points is given when the value is less than 0 or greater than 500; when the second height h... i In the range (0-100), give The penalty points; when the second height h i No penalty is applied within the range (100–300); when the second height h… i When the range is (300-500), give The penalty points.

[0143] This encourages intelligent agents to operate within the optimal flight altitude to avoid risks such as collisions due to excessively low altitudes or loss of contact due to excessively high altitudes.

[0144] Step 1046: Calculate the yaw angle difference between the second yaw angle and the first yaw angle, and the pitch angle difference between the second pitch angle and the first pitch angle, and determine the path stability penalty of the agent based on the yaw angle difference and the pitch angle difference.

[0145] To control the stability and continuity of flight path operations, path stability is determined by the angular change between the current and previous actions. The formula for calculating path stability is as follows:

[0146]

[0147] Where ψ and φ represent the yaw angle and pitch angle of the current action, respectively, and are control variables in the discrete action table. Δψ represents the yaw angle difference, and Δφ represents the pitch angle difference.

[0148] Path stability penalty encourages smooth movements and avoids sharp turns.

[0149] Step 1047: Determine the obstacle avoidance penalty for the intelligent agent based on the second boundary distance and the radius of the danger zone.

[0150] Danger zones are pre-defined dynamic danger zones such as migratory bird flocks, storms, and no-fly zones. To prevent agents from entering danger zones, obstacle avoidance penalties are implemented. These penalties are calculated using the following formula:

[0151]

[0152] Where, d i denoted as , where is the distance from the agent to the center of the danger zone, i.e., the second boundary distance. r is the radius of the danger zone.

[0153] The environment updates the location of the danger zone at each time step, and this penalty is an important factor in enabling the dynamic obstacle avoidance capability of intelligent swarms.

[0154] Step 1048: Determine the reward score for the agent to perform this task based on the sparse reward, the task guidance reward, the distance penalty, the flight altitude penalty, the path stability penalty, and the obstacle avoidance penalty.

[0155] Specifically, the reward score is determined according to the following composite reward function:

[0156] R i =10R guide,i +R task,i -C coll,i +C height,i +C smooth,i +C danger,i

[0157] Among them, R guide,i R represents the task guidance reward. task,i C represents sparse reward. coll,i C represents the distance penalty. height,i Indicates a flight altitude penalty, C smooth,i C represents the path stability penalty. danger,i This indicates a penalty for avoiding obstacles.

[0158] This composite reward function, through a mechanism that combines sparse and dense rewards while balancing safety and efficiency, can comprehensively guide multi-agent systems to learn optimal policy paths with synergy, stability, and feasibility in complex environments.

[0159] As an optional embodiment, step 107 involves inputting the time-series data of each agent into the local Q-networks of the initial reinforcement learning model to generate local Q-values, including:

[0160] Step 1071: Encode the time series data into a fixed-dimensional latent space representation through a linear transformation and ReLU activation function.

[0161] In this system, the AttentionRNNAgent network, serving as a local Q-network for each agent, plays a crucial role in generating action value functions from local observation inputs. Its design integrates the historical memory modeling capabilities of traditional RNN networks with the interactive perception capabilities of multi-head attention mechanisms, aiming to efficiently capture the interaction between individual states and neighboring agents in dynamic and complex multi-agent environments.

[0162] The input to this network is the local observation vector of each agent at the current time step (i.e., time-series data consisting of the target quadruple). i t ∈R d This includes its own environmental perception information, historical behavior, target location relationships, and other elements.

[0163] To enhance time modeling capabilities, time series data is first encoded into a fixed-dimensional latent space representation through a linear transformation and the ReLU activation function. This is represented as:

[0164] x t =ReLU(W x ·o t +b x )

[0165] Where, x t For the latent space representation, W x Let b represent the weight matrix. x Indicates the bias term, o t This represents time series data, and ReLU represents the activation function.

[0166] Step 1072: In the gated loop unit module, the first hidden state of the previous moment and the hidden space representation of the current encoded input are received, and the second hidden state is output.

[0167] To model local temporal correlations and historical information accumulation, a gated cyclic unit structure based on GRUCell was designed for the system. This module receives the hidden state h from the previous time step. t-1 i With the current encoded input x t i Output the new hidden state h t i ,Right now:

[0168]

[0169] This hidden state also serves as a compressed representation of the current agent over time, playing a crucial role in subsequent decision-making.

[0170] Step 1073: In the multi-head attention mechanism module, the second hidden state of the current agent is used as the query vector, and the concatenated tensor of the second hidden states of all agents is used as the key and value to calculate the weighted representation vector; the weighted representation vector is then used to generate the local Q value of the agent at the current time through a linear mapping layer.

[0171] To achieve multi-agent interaction modeling from a local perspective, a multi-head attention mechanism module is further introduced. This module uses the current agent's hidden state h as the basis for its interaction. i t For the query vector q, the set of hidden states of all agents {h} j t}j=1 N For the keys and values, compute the weighted interactive representation to obtain the weighted representation vector:

[0172]

[0173] in, For weighted representation vectors, Let k represent the hidden state concatenation tensor of all agents, where k represents the key and v represents the value.

[0174] This mechanism enables each agent to dynamically focus on the hidden states of agents that are closely related to its behavior when estimating the value of actions, thereby better modeling local influences and conflict relationships in collaborative scenarios.

[0175] Finally, the local Q-values ​​of each agent at the current time are generated through a linear mapping layer:

[0176]

[0177] Among them, Q i ∈R |Ui| This represents the local Q-value of the agent within its set of possible actions. W2 represents the weight matrix, and b2 represents the bias term. This is a weighted representation vector.

[0178] During network forward propagation, the multi-head attention mechanism module can be optionally enabled. By default, this module is enabled during both the training and testing phases to enhance the agent's decision-making adaptability. By introducing this structure, the agent can construct implicit perception channels to other key agents using only local observation information, significantly improving the policy quality and stability of the multi-agent system in high-dimensional dynamic scenarios.

[0179] Furthermore, the module is designed following the end-to-end differentiability principle, enabling seamless integration with QMIX's hybrid network structure. During the centralized training phase, it is simultaneously optimized through backpropagation, thereby achieving collaborative learning throughout the entire process, from local observation encoding, historical modeling, interactive reasoning to action value estimation.

[0180] As an optional embodiment, step 108, which calculates the loss value for this training based on the global Q-value and the target loss function, includes:

[0181] Step 1081: Obtain the global reward score of all agents in the previous time step; determine the expected Q reward of all agents when they choose the action that maximizes the global Q value under the environmental information and action information state in the next time step.

[0182] Step 1082: Determine the target Q value based on the global reward score, the preset discount factor, and the expected Q return;

[0183] Step 1083: Determine the difference between the global Q-value and the target Q-value based on the temporal difference function to obtain the loss value for this training.

[0184] In steps 1081-1083, the global Q function is updated using the TD error minimization method to obtain the loss value for this training, as shown below:

[0185]

[0186] Where y represents the loss value. tot Q represents the target Q value. tot (s, a; θ) represents the global Q-value. s, a, and θ represent the environmental information, action information, and neural network parameters at the current moment, respectively.

[0187] Among them, y tot The method for determining it is as follows:

[0188] y tot =r+γmax a′ (Q tot (s′,a′;θ - )

[0189] Where r represents the global reward score, γ represents the preset discount factor, and max a′ Q tot (s′,a′;θ - Let s', a', and θ represent the expected Q-rewards when all agents choose actions that maximize the global Q-value. - θ represents the environmental information and action information at the next moment, respectively. - It is a copy of the parameters of a historical version of θ.

[0190] A loss function is constructed between the global Q-value and the TD objective using mean squared error, and the backpropagation algorithm is used for end-to-end training of each local Q-network and the hybrid network. The input of the supernetwork in the hybrid network is usually global state information, ensuring that the environmental context features can be fully acquired during intensive training, thereby learning higher-level policy representations.

[0191] During the execution phase, each agent independently selects actions based solely on its own local observations and the forward inference results of AttentionRNNAgent, achieving distributed decision-making. Because an attention mechanism is introduced during training, each agent can not only focus on its own state but also weightedly integrate historical information from key neighboring agents, thereby achieving more refined local policy modeling and ultimately improving the overall collaborative efficiency and robustness of the system.

[0192] As an optional embodiment, before the intelligent agent cluster executes the material delivery task in step 101, the following steps are also included:

[0193] In the simulation platform, a 2.5-dimensional high-dimensional grid method is used for environmental modeling, where each grid contains planar and height information to simulate the characteristics of mountainous terrain.

[0194] In the environmental modeling process, terrain data is generated using Perlin noise. By adjusting the amplitude and number of noise layers, a realistic mountain environment is simulated.

[0195] In this embodiment of the invention, a complex mountainous environment is first simulated on a simulation platform. To accurately represent the characteristics of the mountainous terrain, a 2.5D high-dimensional grid method is used for environmental modeling. In this method, each grid point contains height information, enabling the environment to reflect complex terrain features such as ridges and valleys. The terrain data is generated using Perlin noise, and a more realistic mountainous environment is generated by adjusting the amplitude and number of noise layers.

[0196] For example, the simulation area covers 20km × 20km with a spatial resolution of Δ = 0.01, ensuring the accuracy of terrain details.

[0197] This allows the solution to be applied to complex and dynamically changing three-dimensional environments such as mountainous areas and cities, enabling multi-agent task coordination, collaborative obstacle avoidance, and intelligent decision-making.

[0198] Figure 2 This is a structural block diagram of a training device for an intelligent agent cluster path planning model provided in an embodiment of the present invention. Figure 2 As shown, the device 200 includes:

[0199] The first state matrix construction module 201 is used to construct a first state matrix for each agent at the current time step during the execution of a material delivery task by the agent cluster. The first state matrix includes the first environmental information and the first action information of the agent. The first environmental information includes the position information of the agent and its relative distance to the task destination. The first action information includes the yaw angle and pitch angle of the agent. The task destination of the material delivery task of each agent is different.

[0200] Input module 202 is used to input the first state matrix into the initial reinforcement learning model to predict the second action information of each agent in the next time step;

[0201] The second state matrix construction module 203 is used to control each of the intelligent agents to fly according to the second action information, and to obtain the second environmental information set of the intelligent agent cluster at the end of the next time step; and to construct the second state matrix of the next time step based on the second environmental information and the second action information.

[0202] The reward determination module 204 is used to determine the reward score of the intelligent agent cluster for performing this task based on the second state matrix and the first state matrix;

[0203] The time series data construction module 205 is used to construct time series data composed of target quadruples based on the reward score and state matrix of the agent at each time step within the target time period. The target quadruples include the environmental information, action information, reward score and environmental information of the next time step for each agent at the current time step.

[0204] The caching module 206 is used to store the time series data into a buffer and control the agent to continue to execute the material delivery task until the amount of data in the buffer reaches the preset batch processing data amount, and then train the model.

[0205] The training module 207 is used to input the time series data of each agent into the local Q network of the initial reinforcement learning model during each training session to generate local Q values; and to combine the local Q values ​​of each agent into a global Q value through a hybrid Q network with monotonicity constraints.

[0206] The update module 208 is used to calculate the loss value of this training based on the global Q value and the target loss function, and update the local Q network and the hybrid Q network according to the loss value;

[0207] The model determination module 209 is used to continue training using the updated local Q network and hybrid Q network until the preset termination condition is met, thereby obtaining the agent cluster path planning model.

[0208] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0209] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0210] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that comply with the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0211] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A training method for an agent cluster path planning model, characterized in that, The method includes: During the process of the intelligent agent cluster executing the material delivery task, a first state matrix is ​​constructed for each intelligent agent at the current time step. The first state matrix includes the first environmental information and the first action information of the intelligent agent. The first environmental information includes the position information of the intelligent agent and the relative distance to the task destination. The first action information includes the yaw angle and pitch angle of the intelligent agent. The task destination of the material delivery task of each intelligent agent is different. The first state matrix is ​​input into the initial reinforcement learning model to predict the second action information of each agent in the next time step. Each of the intelligent agents is controlled to fly according to the second action information, and at the end of the next time step, the second environmental information set of the intelligent agent cluster is obtained; based on the second environmental information and the second action information, the second state matrix of the next time step is constructed. The reward score for the agent cluster to perform this task is determined based on the second state matrix and the first state matrix; Based on the reward score and state matrix of the agent at each time step within the target time period, a time series data consisting of target quadruples is constructed. The target quadruples include the environmental information, action information, reward score, and environmental information of each agent at the current time step and the next time step. The time series data is stored in a buffer, and the agent is controlled to continue to perform the material delivery task until the amount of data in the buffer reaches the preset batch processing data amount, at which point the model is trained. During each training session, the time-series data of each agent is input into the local Q-networks of the initial reinforcement learning model to generate local Q-values; the local Q-values ​​of each agent are combined into a global Q-value through a hybrid Q-network with monotonicity constraints. The loss value for this training is calculated based on the global Q-value and the target loss function, and the local Q-network and the hybrid Q-network are updated according to the loss value. The updated local Q-network and hybrid Q-network are used to continue training until the preset termination condition is met, at which point the training ends and the agent cluster path planning model is obtained.

2. The method according to claim 1, characterized in that, The first state matrix includes the agent's first current position, first altitude relative to the terrain, first relative distance to the mission destination, first boundary distance to the center of the nearest danger zone, and first safe distance to other agents; the first action information includes the agent's first yaw angle and first pitch angle. The step of determining the reward score for the agent cluster to perform this task based on the second state matrix and the first state matrix includes: The second environmental information and second action information of the agent are obtained from the second state matrix. The second environmental information includes: the second current position, the second altitude relative to the terrain, the second relative distance to the mission destination, the second boundary distance to the center of the nearest danger zone, and the second safe distance to other agents. The second action information includes the agent's second yaw angle and second pitch angle. When the second relative distance between the agent and the task destination is 0, the agent is given a sparse reward. Determine the magnitude relationship between the second relative distance and the first relative distance; when the second relative distance is smaller than the first relative distance, give the agent a task guidance reward. Determine the relationship between the second safe distance and the preset safe threshold; when the second safe distance is less than the safe threshold, impose a distance penalty on the agent. Determine the relationship between the second altitude and the preset target flight altitude range; when the second altitude does not belong to the target flight altitude range, impose a flight altitude penalty on the agent. Calculate the yaw angle difference between the second yaw angle and the first yaw angle, and the pitch angle difference between the second pitch angle and the first pitch angle, and determine the path stability penalty of the agent based on the yaw angle difference and the pitch angle difference; The obstacle avoidance penalty for the agent is determined based on the second boundary distance and the radius of the danger zone; The reward score for the agent to perform this task is determined based on the sparse reward, the task guidance reward, the distance penalty, the flight altitude penalty, the path stability penalty, and the obstacle avoidance penalty.

3. The method according to claim 2, characterized in that, The step of giving the agent a task guidance reward when the second relative distance is smaller than the first relative distance includes: When the second relative distance is smaller than the first relative distance, the difference between the first safety distance and the second safety distance is determined; Obtain the agent's fixed flight speed, determine the quotient between the difference and the fixed flight speed, and use the quotient as the agent's task guidance reward value.

4. The method according to claim 2, characterized in that, The step of imposing a distance penalty on the agent when the second safe distance is less than the safe threshold includes: Obtain the second safe distance between the agent and other agents, and determine the distance difference between the preset safe threshold and the second safe distance; If the difference is greater than 0, then the agent is determined to be the target agent whose second safe distance is less than the safe threshold; For each target agent, determine the total number of target agents and obtain the distance difference corresponding to each target agent; Calculate the quotient between the distance difference between each of the target agents and the security threshold to obtain multiple quotients; calculate the sum of the multiple quotients; The distance penalty value of the agent is determined based on the sum and the total number.

5. The method according to claim 1, characterized in that, The step of inputting the time-series data of each of the intelligent agents into the local Q-networks of the initial reinforcement learning model to generate local Q-values ​​includes: The time series data is encoded into a fixed-dimensional latent space representation through a linear transformation and ReLU activation function. In the gated loop unit module, the first hidden state from the previous moment and the hidden space representation of the current encoded input are received, and the second hidden state is output. In the multi-head attention mechanism module, the second hidden state of the current agent is used as the query vector, and the concatenated tensor of the second hidden states of all agents is used as the key and value to calculate the weighted representation vector; the weighted representation vector is then used to generate the local Q value of the agent at the current time through a linear mapping layer.

6. The method according to claim 1, characterized in that, The calculation of the loss value for this training based on the global Q-value and the target loss function includes: Obtain the global reward score of all agents in the previous time step; determine the expected Q reward when all agents choose actions that maximize the global Q value under the environmental and action information conditions in the next time step. The target Q value is determined based on the global reward score, the preset discount factor, and the expected Q return. The difference between the global Q-value and the target Q-value is determined based on the temporal difference function, and the loss value for this training is obtained.

7. The method according to claim 1, characterized in that, Before the intelligent agent cluster performs the material delivery task, it also includes: In the simulation platform, a 2.5-dimensional high-dimensional grid method is used for environmental modeling, where each grid contains planar and height information to simulate the characteristics of mountainous terrain. In the environmental modeling process, terrain data is generated using Perlin noise. By adjusting the amplitude and number of noise layers, a realistic mountain environment is simulated.

8. A training device for an intelligent agent cluster path planning model, characterized in that, The device includes: The first state matrix construction module is used to construct the first state matrix of each agent at the current time step during the execution of material delivery tasks by the agent cluster. The first state matrix includes the first environmental information and the first action information of the agent. The first environmental information includes the position information of the agent and the relative distance to the task destination. The first action information includes the yaw angle and pitch angle of the agent. The task destination of the material delivery task of each agent is different. The input module is used to input the first state matrix into the initial reinforcement learning model to predict the second action information of each agent in the next time step; The second state matrix construction module is used to control each of the intelligent agents to fly according to the second action information, and to obtain the second environmental information set of the intelligent agent cluster at the end of the next time step; and to construct the second state matrix of the next time step based on the second environmental information and the second action information. The reward determination module is used to determine the reward score of the agent cluster for performing this task based on the second state matrix and the first state matrix; The time series data construction module is used to construct time series data consisting of target quadruples based on the reward score and state matrix of the agent at each time step within the target time period. The target quadruples include the environmental information, action information, reward score of each agent at the current time step, and the environmental information of the next time step. The caching module is used to store the time series data into a buffer and control the agent to continue to execute the material delivery task until the amount of data in the buffer reaches the preset batch processing data amount, at which point the model is trained. The training module is used to input the time-series data of each agent into the local Q-networks of the initial reinforcement learning model during each training session to generate local Q-values; and to combine the local Q-values ​​of each agent into a global Q-value through a hybrid Q-network with monotonicity constraints. The update module is used to calculate the loss value of this training based on the global Q value and the target loss function, and update the local Q network and the hybrid Q network according to the loss value; The model determination module is used to continue training using the updated local Q-network and hybrid Q-network until the preset termination condition is met, at which point the training ends and the agent cluster path planning model is obtained.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the training method of the intelligent agent cluster path planning model as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or instruction set is loaded and executed by a processor to implement the training method for the intelligent agent cluster path planning model as described in any one of claims 1-7.

Images