Multi-agent intelligent way-finding and real-time decision-making method based on MADDPG

Abstract

The invention discloses a multi-agent intelligent way-finding and real-time decision-making method based on MADDPG, and the method achieves the autonomous collaborative decision-making of multi-target optimization through the technical links of environment perception, dynamic obstacle intention prediction, single-agent pre-training, strategy distillation and multi-agent collaborative training. The method comprises the following core steps: predicting a dynamic obstacle intention based on a long short-term memory network of an attention mechanism; constructing a threat field model to quantify the environmental risk; forming basic obstacle avoidance and path optimization experience through single agent pre-training; empirical migration is realized by utilizing strategy distillation so as to shorten a multi-agent training period; and multi-agent cooperative training is completed based on a centralized training-distributed execution architecture and an MADDPG algorithm. According to the method, the path length, the energy consumption, the safety and the task completion rate can be considered, the survivability, the cooperation efficiency and the task adaptability of the unmanned aerial vehicle formation in the complex dynamic environment are remarkably improved, and the method can be widely applied to different-scale formation and multiple task scenes such as reconnaissance, communication relay and situation awareness.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, unmanned systems and path planning technology, specifically to a multi-agent intelligent pathfinding and real-time decision-making method based on multi-agent deep deterministic policy gradient (MADDPG), which is applicable to autonomous collaborative decision-making and path planning in complex dynamic environments, and is particularly suitable for tasks such as collaborative reconnaissance, communication relay and situational awareness of UAV formations. Background Technology

[0002] With the rapid development of unmanned systems technology, the application of drone formations in military and civilian fields is becoming increasingly widespread, and the environment in which they perform tasks is gradually evolving from static and simple environments to dynamic and complex environments. Traditional path planning methods, such as the A* algorithm and Dijkstra's algorithm, calculate the optimal path based on preset rules and static environmental information. They perform stably in static or semi-static environments, but when faced with complex scenarios involving dynamic obstacles, incomplete information, and limited communication, they suffer from poor real-time performance, insufficient robustness, and an inability to predict obstacle behavior, which can easily lead to an increased risk of collision.

[0003] While single-agent reinforcement learning methods have made some progress in path planning, they suffer from insufficient collaboration, low learning efficiency, and susceptibility to local optima in multi-UAV cooperative tasks. The core challenge of multi-agent systems lies in achieving a balance between individual and global objectives, and in rapidly adapting to and collaboratively responding to external interference in dynamic environments. Existing multi-agent training methods often require significant computational resources and training time, and struggle to simultaneously address multiple objectives such as path optimization, energy consumption control, threat avoidance, and task completion rate, thus limiting their application in real-world scenarios.

[0004] Therefore, there is an urgent need for a collaborative pathfinding and decision-making method that can accurately predict the intention of dynamic obstacles, efficiently realize the transfer of multi-agent experience, and take into account multi-objective optimization, so as to improve the task success rate and robustness of multi-agent systems in complex dynamic environments. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multi-agent intelligent pathfinding and real-time decision-making method based on MADDPG. This method solves the technical problems of lack of intent prediction, low training efficiency, and imbalance of multi-objective optimization in multi-agent formation cooperative pathfinding under complex dynamic environments, and realizes autonomous decision-making with accurate prediction of dynamic obstacle intent, efficient transfer of experience, and multi-objective cooperative optimization.

[0006] To achieve the above objectives, the technical solution of the present invention is implemented as follows: A multi-agent intelligent pathfinding and real-time decision-making method based on MADDPG includes the following steps: S1. Constructing the training environment and data acquisition: Construct a single-agent task scenario in the simulation platform, determine the agent type, dynamic obstacle type and target type, and collect time-series data with a preset time window. The time-series data includes the state feature data of the red and blue sides. S2, Dynamic Obstacle Intent Recognition: Input the time sequence data collected in step S1 into the intent analysis module, and output the short-term intent recognition result of the dynamic obstacle through the long short-term memory network based on the attention mechanism. The short-term intent includes the direction of movement, interception intent, and patrol route. S3. Distance Threat Field Model Calculation: Based on the maximum reconnaissance distance, maximum strike distance, and real-time distance between our agent and the target, the distance threat field value is calculated to quantify environmental risks. S4. Single agent pre-training: Based on the intent recognition results of step S2 and the threat field value of step S3, combined with the agent's heading, speed, altitude, latitude and longitude, damage, state, and mounted state parameters, the single agent is trained through reinforcement learning to update the value network parameters and policy network parameters, forming a transferable state-action experience set. S5. Policy distillation and experience transfer: The first two layers of parameters of the single-agent policy network are copied to the multi-agent policy network to construct a joint optimization objective of distillation loss and critic loss; S6. Multi-agent collaborative training: Based on a centralized training-distributed execution architecture, the MADDPG algorithm is used to jointly train multiple agents. During the training process, the distillation weights are dynamically adjusted to achieve a smooth transition from supervised distillation to autonomous collaboration. S7. Real-time decision deployment: The trained strategy is deployed to each agent. During the execution phase, each agent makes autonomous decisions based on local observations and pre-trained strategies to achieve collaborative pathfinding and dynamic obstacle avoidance.

[0007] Furthermore, the specific implementation process of the intent analysis module in step S2 is as follows: S21. Construct an encoder based on attention mechanism: use temporal sequence data as key, use the hidden layer state and unit state of the encoder as query, obtain attention weight by calculating the correlation between query and key, and extract key temporal features after softmax normalization. S22. Construct an attention-based decoder: Based on the previous hidden layer state and cell state of the decoder, calculate the attention weights of each encoder's hidden layer, and perform a weighted summation of the encoder output features to obtain the tactical intent result. ; S23, Intent Output: The decoder output is regressed through a softmax layer to obtain the intent vector, which is then input into the pathfinding decision module.

[0008] Furthermore, the attention-based encoder specifically extracts feature representations of key moments from the time series, preserves temporal dependencies, and can adaptively select sequences with high relevance for encoding, thus solving the problem of limited capacity of the encoded fixed-length vector; this encoder encodes the entire sequence information of the k-th time series. As the key, the hidden layer and unit state As the query, attention weights are calculated using the query and key, and then normalized using softmax to obtain the attention level of each sequence. :

[0009]

[0010] in, , , For learning parameters, These are attention weights that measure the importance of the k-th input feature at time t. A softmax function is applied to ensure that the sum of all attention weights is 1. These attention weights can be used to adaptively extract the driving sequence.

[0011]

[0012] The hidden state at time t can then be updated as follows:

[0013] in It is an LSTM unit that can be computed based on three sigmoid gates, where replace By using an input attention mechanism, the encoder selectively focuses on important time series, rather than treating all input time series equally; The attention-based decoder specifically involves the attention weights of each encoder hidden layer at time t being based on the previous decoder hidden layers. Cellular state of LSTM unit To calculate;

[0014]

[0015] in, It is a concatenation of the previous hidden layer and the cell layer of the LSTM unit. , and The attention weights are the parameters that need to be learned. This represents the importance of the i-th encoder hidden layer to the prediction, where each encoder hidden layer... This is mapped to a temporal component of the input; therefore, all the aforementioned encoder hidden layers are then processed through an attention mechanism. , ,…, Weighted summation to calculate the tactical intent result. .

[0016] Furthermore, the calculation basis for the distance threat field model in step S3 is as follows: Let our intelligent agent W i The maximum strike distance is r ai The maximum reconnaissance distance is r ri Target T j The maximum strike distance is r taj The maximum reconnaissance distance is r trj Through W i With T j The real-time distance between the two parties is used to quantify the threat intensity generated by their interaction, thus obtaining the distance threat field value T. rij .

[0017] Furthermore, the single-agent pre-training in step S4 specifically involves: The goal is to enable a single UAV to autonomously avoid obstacles, evade surface-to-air missile platoon threats, fly energy-efficiently, and complete attack missions against tank platoons in complex and dynamic threat environments; the UAV's input status... Includes threat field value :

[0018] in, Location of the drone. For drone speed, The relative position of the dynamic obstacle unit; an action is performed and a new state is obtained through a reward function. With instant rewards The samples are stored in the experience pool:

[0019] And the threat field is updated periodically:

[0020] in It comes from the intent analysis module, and then each sampling batch ( Update the critic:

[0021]

[0022] After the Critic update, the value network parameters for a single UAV can be obtained. Then update the actor:

[0023] After the Actor is updated, the strategy parameters for a single drone can be obtained. .

[0024] Furthermore, the reward function for single-agent pre-training in step S4 adopts a multi-objective design, and its expression is: Among them, higher weights are used in single agent pre-training. - Threat penalty is used to enhance the drone's ability to avoid threats; detailed reward definitions are as follows:

[0025] in, This represents the increase in distance to the target in this step, used to encourage faster paths to the target;

[0026] in, As weight, To minimize energy consumption in this step, since drones have an energy limit, low-energy operations are encouraged.

[0027] in, As weight, It is a threat field. When a drone is in a high-threat area, it receives a lot of penalties, which causes the drone to actively avoid the threat area.

[0028] in, This is the collision penalty coefficient. Since experience needs to be transferred to multiple drones, it is necessary to prevent multiple drones from colliding with each other. It is set to 0 when training a single drone.

[0029] Furthermore, in step S5, copying the first two layers of parameters from the single-agent policy network to the multi-agent policy network specifically involves: Two parameters obtained from single UAV pre-training , Process and copy the first two layers. Parameters to multi-drone strategy parameters middle:

[0030] Replicating the first two layers enables multiple drones to have a unified representation of the threat field in the early stages, so that they do not need to relearn basic perception functions during multi-agent training. During the training loop, both the critic and actor updates are performed simultaneously for each agent i. The critic has access to global information. With action From the mixed experience pool Sample small batches according to priority Policy Network With Q network The update is performed according to the following formula:

[0031] Each of these actions , This is the observation of the j-th agent at the next moment.

[0032] For the i-th agent in MADDPG, the Q-network loss is defined as:

[0033] Let i be the centralized critic for the i-th agent. This indicates that smooth copies are used to stabilize training. For the immediate reward of the i-th agent, These are the actions actually performed during training; In MADDPG, the gradient of an actor is:

[0034] in This is the policy network of UAV j. It is a concentrated critic used to calculate the action gradient direction; the sampled 'o' comes from a hybrid empirical pool. ,in Includes features from intent recognition .

[0035] Furthermore, in step S5, the specific implementation of constructing the joint optimization objective of distillation loss and critic loss includes: S51. Sample Selection: Extract high-quality trajectory samples from the single-agent experience pool, and construct sample weights w based on task completion degree, threat field strength, and collision frequency, with the following expression: ,in, =0.4, =0.35, =0.25, =3.0, =0.3, 1success_flag is the task completion flag, threat_level is the threat field level, and collision_count is the number of collisions; S52, Parameter Transfer: The first two layers of parameters of the single agent policy network are copied to the multi-agent policy network, so that the multi-agent initially has threat field representation and basic obstacle avoidance capabilities. S53, Loss Construction: Introducing distillation supervision in multi-agent training, using distillation sets... To guide multi-agent systems in mimicking the experience of a single agent to avoid threats and complete tasks, the distillation loss for the i-th drone is defined as:

[0036] The critical loss and distillation loss are combined into the training objective for each agent:

[0037] Distillation weight The definition is as follows:

[0038] =0.5, =0.05, For the number of preheating steps, This indicates the number of distillation steps that will be completed.

[0039] Furthermore, in step S1, the simulation platform is the Mozi combat simulation platform, the intelligent agent is the MQ-1C "Gray Eagle" UAV, the dynamic obstacle is a platoon of surface-to-air missiles, the target is a platoon of tanks, and the preset time window is 30, that is, the time sequence data from time t-29 to time t is collected.

[0040] Furthermore, the state vector of the single agent in step S4 ,in For location, For speed, For local observation, For dynamic obstacle intention perception, For the task state, the action vector is defined as follows: This indicates continuous speed / heading control.

[0041] Beneficial effects: This invention achieves accurate prediction of dynamic obstacle intentions through Attention-LSTM, and combines it with a threat field model to quantify environmental risks, enabling the agent to avoid dynamic threats in advance, significantly reducing the probability of collisions and improving survivability in complex environments. Experiments show that the threat avoidance rate has increased from 86% to 92%. An experience transfer mechanism of single-agent pre-training + policy distillation is adopted to replicate the parameters of the first two layers of the single-agent policy network to multiple agents, avoiding the need for multiple agents to learn basic perception and obstacle avoidance capabilities from scratch, shortening the training convergence time, and reducing the computational resource requirements of the execution phase. The design incorporates a multi-objective reward function that balances path length, energy consumption, safety, and task completion rate. By using configurable weight coefficients to adapt to different task requirements, it overcomes the limitations of traditional single-objective optimization methods. Based on a centralized training-distributed execution architecture, the system optimizes collaborative strategies using global information during training and relies on local observations for autonomous decision-making during execution. It balances collaborative efficiency and robustness, and has good scalability, making it adaptable to different formation sizes and various mission types such as reconnaissance, communication relay, and attack. Experimental verification shows that the average trajectory deviation of the present invention is only 3.4±0.6m, which is highly consistent with the expert experience path, and the dynamic deviation rate is 6%, demonstrating excellent decision-making accuracy and stability in complex dynamic environments. Attached Figure Description

[0042] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart of the main process of the multi-agent intelligent pathfinding and real-time decision-making method based on MADDPG as described in an embodiment of the present invention. Figure 2 This is a flowchart of the dynamic obstacle intent recognition in the multi-agent intelligent pathfinding and real-time decision-making method based on MADDPG described in an embodiment of the present invention; Figure 3 This is the value network training graph of a single agent in the MADDPG-based multi-agent intelligent pathfinding and real-time decision-making method described in this embodiment of the invention. Figure 4 This is the policy network training graph of a single agent in the MADDPG-based multi-agent intelligent pathfinding and real-time decision-making method described in this embodiment of the invention. Figure 5 This is the value network training graph of the multi-agent intelligent pathfinding and real-time decision-making method based on MADDPG described in this embodiment of the invention. Figure 6 This is the training graph of the multi-agent policy network in the MADDPG-based multi-agent intelligent pathfinding and real-time decision-making method described in this embodiment of the invention. Detailed Implementation

[0043] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0044] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0045] Example 1 See Figure 1-6 A multi-agent intelligent pathfinding and real-time decision-making method based on MADDPG includes the following steps: S1. Constructing the training environment and data acquisition: Construct a single-agent task scenario in the simulation platform, determine the agent type, dynamic obstacle type and target type, and collect time-series data with a preset time window. The time-series data includes the state feature data of the red and blue sides. It should be noted that the simulation platform mentioned in step S1 is the Mozi combat simulation platform, the intelligent agent is the MQ-1C "Gray Eagle" UAV, the dynamic obstacle is a platoon of surface-to-air missiles, the target is a platoon of tanks, and the preset time window is 30, that is, the time sequence data from time t-29 to time t is collected.

[0046] S2, Dynamic Obstacle Intent Recognition: Input the time-series data collected in step S1 into the intent analysis module, and output the short-term intent recognition result of the dynamic obstacle through the attention-based long short-term memory network (Attention-LSTM). The short-term intent includes the direction of movement, interception intent, and patrol route. S3. Distance Threat Field Model Calculation: Based on the maximum reconnaissance distance, maximum strike distance, and real-time distance between our agent and the target, the distance threat field value is calculated to quantify environmental risks. S4. Single agent pre-training: Based on the intent recognition results of step S2 and the threat field value of step S3, combined with the agent's heading, speed, altitude, latitude and longitude, damage, state, and mounted state parameters, the single agent is trained through reinforcement learning to update the value network parameters and policy network parameters, forming a transferable state-action experience set. It should be noted that the state vector of the single agent in step S4... ,in For location, For speed, For local observation (positions of obstacles and friendly forces). For dynamic obstacle intention perception, For the task state (target point, priority), the action vector is defined as follows: This indicates continuous speed / heading control.

[0047] S5. Policy distillation and experience transfer: The first two layers of parameters of the single-agent policy network are copied to the multi-agent policy network to construct a joint optimization objective of distillation loss and critic loss; S6. Multi-agent collaborative training: Based on the centralized training-distributed execution (CTDE) architecture, the MADDPG algorithm is used to jointly train multiple agents. During the training process, the distillation weights are dynamically adjusted to achieve a smooth transition from supervised distillation to autonomous collaboration. S7. Real-time decision deployment: The trained strategy is deployed to each agent. During the execution phase, each agent makes autonomous decisions based on local observations and pre-trained strategies to achieve collaborative pathfinding and dynamic obstacle avoidance.

[0048] This embodiment integrates "intent recognition → threat quantification → single agent pre-training → policy distillation → multi-agent collaboration → real-time deployment" into a closed-loop process for the first time, overcoming the shortcomings of traditional multi-agent systems such as "insufficient collaboration and inefficient training", and achieving full-link optimization of "perception-decision-execution" in complex dynamic environments; This embodiment creatively combines four core technologies: the MADDPG algorithm, attention mechanism, policy distillation, and CTDE architecture. It ensures the consistency of multi-agent collaboration through centralized training and enhances anti-interference ability through distributed execution, thus resolving the contradiction of "difficulty in balancing real-time performance and robustness" in traditional methods.

[0049] In a specific example, the implementation process of the intent analysis module in step S2 is as follows: S21. Construct an encoder based on attention mechanism: use temporal sequence data as key, use the hidden layer state and unit state of the encoder as query, obtain attention weight by calculating the correlation between query and key, and extract key temporal features after softmax normalization. S22. Construct an attention-based decoder: Based on the previous hidden layer state and cell state of the decoder, calculate the attention weights of each encoder's hidden layer, and perform a weighted summation of the encoder output features to obtain the tactical intent result. ; S23, Intent Output: The decoder output is regressed through a softmax layer to obtain the intent vector, which is then input into the pathfinding decision module.

[0050] In the specific implementation, time series data with a time window of 30 is obtained from the Mozi battle dataset. This refers to the state feature data of the red and blue teams from time t-29 to time t, where t is the prediction time. The processed feature data is input into a pre-trained attention-based encoder neural network, then into a pre-trained attention-based decoder, and finally processed through a softmax layer to obtain the target's most likely intention at that time. .

[0051] This embodiment adopts a hierarchical architecture of "encoder → decoder → softmax output" to first extract key temporal features and then focus on core intent information, avoiding the redundant calculations of the traditional LSTM that "treats all temporal data equally" and improving the speed of intent recognition. The encoder adaptively selects highly relevant temporal sequences, and the decoder selectively weights the encoder output, solving the problems of "inaccurate temporal dependency capture and large intent prediction deviation" in traditional models, and providing a reliable basis for subsequent obstacle avoidance decisions.

[0052] In a specific example, the attention-based encoder extracts feature representations of key moments from the time series, preserves temporal dependencies, and adaptively selects sequences with high relevance for encoding, thus addressing the limited capabilities of fixed-length encoded vectors. This encoder encodes the entire sequence information of the k-th time series. As the key, the hidden layer and unit state As the query, attention weights are calculated using the query and key, and then normalized using softmax to obtain the attention level of each sequence. :

[0053]

[0054] in, , , For learning parameters, These are attention weights that measure the importance of the k-th input feature (driving sequence) at time t. A softmax function is applied to ensure that the sum of all attention weights is 1. The driving sequence can be adaptively extracted using these attention weights.

[0055]

[0056] The hidden state at time t can then be updated as follows:

[0057] in It is an LSTM unit that can be based on three sigmoid gates (forget gates). Input gate and output gate To calculate, where replace By using an input attention mechanism, the encoder selectively focuses on important time series, rather than treating all input time series equally; The attention-based decoder specifically involves the attention weights of each encoder hidden layer at time t being based on the previous decoder hidden layers. Cellular state of LSTM unit To calculate;

[0058]

[0059] in, It is a concatenation of the previous hidden layer and the cell layer of the LSTM unit. , and The attention weights are the parameters that need to be learned. This represents the importance of the i-th encoder hidden layer to the prediction, where each encoder hidden layer... This is mapped to a temporal component of the input; therefore, all the aforementioned encoder hidden layers are then processed through an attention mechanism. , ,…, Weighted summation to calculate the tactical intent result. .

[0060] This embodiment clearly defines attention weights. , The computational logic defines the feature extraction and intent fusion process through strict mathematical formulas. Compared with the traditional "fuzzy attention mechanism", the technical solution can be accurately reproduced, reducing the difficulty of implementation.

[0061] Dual protection of temporal dependence and key features: The encoder retains the dynamic correlation of temporal data, while the decoder focuses on the contribution of core features. This dual optimization makes intent recognition both "comprehensive" (covering complete temporal information) and "precise" (highlighting key behavioral features), with a prediction accuracy significantly higher than that of the traditional LSTM model.

[0062] In a specific example, the calculation basis of the distance threat field model in step S3 is as follows: Let our intelligent agent W i The maximum strike distance is r ai The maximum reconnaissance distance is r ri Target T j The maximum strike distance is r taj The maximum reconnaissance distance is r trj Through W i With T j The real-time distance between the two parties is used to quantify the threat intensity generated by their interaction, thus obtaining the distance threat field value T. rij .

[0063] This embodiment calculates the range threat field value by relating the maximum reconnaissance range and maximum strike range of both sides (our drone and the target) to the distance between them. ; , Each of our drones With the goal The maximum firing range of the weapons carried. and Each of our drones With the goal Maximum reconnaissance range.

[0064] In a specific example, the single-agent pre-training in step S4 specifically involves: The goal is to enable a single UAV to autonomously avoid obstacles, evade surface-to-air missile platoon threats, fly energy-efficiently, and complete attack missions against tank platoons in complex and dynamic threat environments; the UAV's input status... Includes threat field value :

[0065] in, Location of the drone. For drone speed, The relative position of the dynamic obstacle unit; an action is performed and a new state is obtained through a reward function. With instant rewards The samples are stored in the experience pool:

[0066] And the threat field is updated periodically:

[0067] in It comes from the intent analysis module, and then each sampling batch ( Update the critic:

[0068]

[0069] After the Critic update, the value network parameters for a single UAV can be obtained. Then update the actor:

[0070] After the Actor is updated, the strategy parameters for a single drone can be obtained. .

[0071] It should be noted that this embodiment allows a single agent to master basic skills such as obstacle avoidance, energy saving, and threat avoidance in advance, forming a transferable experience set. This avoids multiple agents learning from scratch and significantly reduces the sample requirements and convergence time for training multiple agents.

[0072] Furthermore, the threat field (integrating intent recognition results) is periodically updated during training, making the training environment closer to real dynamic scenarios. Compared with traditional "fixed threat field" training, the model has stronger generalization ability and better adaptability in actual deployment. Through alternating updates of the Critic / Actor dual network, it is ensured that the policy learned by a single agent is "locally optimal", providing a high-quality experience foundation for subsequent multi-agent collaboration and avoiding "poor experience transfer leading to a decline in multi-agent performance".

[0073] In a specific example, the reward function for single-agent pre-training in step S4 adopts a multi-objective design, and its expression is: Among them, higher weights are used in single agent pre-training. - Threat penalty is used to enhance the drone's ability to avoid threats; detailed reward definitions are as follows:

[0074] in, This represents the increase in distance to the target in this step, used to encourage faster paths to the target;

[0075] in, As weight, To minimize energy consumption in this step, since drones have an energy limit, low-energy operations are encouraged.

[0076] in, As weight, It is a threat field. When a drone is in a high-threat area, it receives a lot of penalties, which causes the drone to actively avoid the threat area.

[0077] in, This is the collision penalty coefficient. Since experience needs to be transferred to multiple drones, it is necessary to prevent multiple drones from colliding with each other. It is set to 0 when training a single drone.

[0078] This embodiment integrates four objectives: path length, energy consumption, threat avoidance, and collision penalty. Compared with the traditional reward function that "only focuses on the shortest path," it achieves a multi-dimensional balance of "optimal path, low energy consumption, high security, and high task completion rate," which is more in line with actual task requirements.

[0079] Furthermore, by adjusting the weights of each sub-reward (such as strengthening the threat penalty weight during single agent pre-training), different task priorities can be flexibly adapted (such as reconnaissance tasks emphasizing concealment, and strike tasks emphasizing rapid arrival), thus solving the problem of "poor scenario adaptability" of traditional reward functions.

[0080] In a specific example, in step S5, copying the first two layers of parameters from the single-agent policy network to the multi-agent policy network specifically involves: The policy knowledge of a single agent is transferred to multiple agents in a soft-constraint manner. In this system, the policy knowledge of the single agent is the UAV policy network trained by single-agent reinforcement learning, which has mastered the ability to avoid obstacles, threats, and optimize paths to local optima.

[0081] Two parameters obtained from single UAV pre-training , Process and copy the first two layers. Parameters to multi-drone strategy parameters middle:

[0082] Replicating the first two layers enables multiple drones to have a unified representation of the threat field in the early stages, so that they do not need to relearn basic perception functions during multi-agent training. During the training loop, both the critic and actor updates are performed simultaneously for each agent i. The critic has access to global information. With action From the mixed experience pool Sample small batches according to priority Policy Network With Q network The update is performed according to the following formula:

[0083] Each of these actions , This is the observation of the j-th agent at the next moment.

[0084] For the i-th agent in MADDPG, the Q-network loss is defined as:

[0085] For the i-th agent, there is a centralized critic (the input is global observations and all actions, used for centralized evaluation during the training phase). This indicates that smooth copies are used to stabilize training. For the immediate reward of the i-th agent, For actions to be actually performed during training, since the threat field and dynamic obstacles intend to affect the safety and survival status of each agent, the Critic must see global information to evaluate the long-term value of joint actions. The above formula ensures that the objective value of Q comprehensively considers the long-term reward impact of all agent actions on each agent.

[0086] In MADDPG, the gradient of an actor is:

[0087] in This is the policy network of UAV j. It is a concentrated critic used to calculate the action gradient direction; the sampled 'o' comes from a hybrid empirical pool. ,in Includes features from intent recognition The Actor learns to output actions based on local observations containing threat information, while the critic provides a "cooperative-aware" value gradient to ensure that individual actions are advantageous in global collaborative behavior. When the intention of a dynamic obstacle changes abruptly, that is, when the intention recognition module provides a new... Actor can be based on local Quickly adjust your actions to avoid the threat.

[0088] This embodiment directly replicates the parameters of the first two layers of the single-agent policy network, enabling multiple agents to initially possess threat field representation and basic obstacle avoidance capabilities, without needing to relearn the underlying perception skills, thus shortening the "start-up phase" time of multi-agent training.

[0089] The Critic network has access to global information and all actions, providing each agent with a "globally optimal" value assessment, avoiding the problem of "local decision-making conflicts" in traditional multi-agent systems, and ensuring the uniformity of the formation's collaborative behavior.

[0090] In addition, the Actor network is based on local observation and decision-making, without relying on real-time global communication. Even if some agents have limited or failed communication, the remaining nodes can still work normally. Compared with the traditional "centralized decision-making", the system's anti-interference capability is greatly improved.

[0091] In a specific example, step S5, specifically implementing the joint optimization objective of constructing distillation loss and critic loss, includes: S51. Sample Selection: Extract high-quality trajectory samples from the single-agent experience pool, and construct sample weights w based on task completion degree, threat field strength, and collision frequency, with the following expression: ,in, =0.4, =0.35, =0.25, =3.0, =0.3, 1success_flag is the task completion flag, threat_level is the threat field level, and collision_count is the number of collisions; S52, Parameter Transfer: The first two layers of parameters of the single agent policy network are copied to the multi-agent policy network, so that the multi-agent initially has threat field representation and basic obstacle avoidance capabilities. S53, Loss Construction: Introducing distillation supervision in multi-agent training, using distillation sets... To guide multi-agent systems in mimicking the experience of a single agent to avoid threats and complete tasks, the distillation loss for the i-th drone is defined as:

[0092] The critical loss and distillation loss are combined into the training objective for each agent:

[0093] Distillation weight The definition is as follows:

[0094] =0.5, =0.05, For the number of preheating steps, This indicates the number of distillation steps that will be completed.

[0095] This embodiment constructs sample weights based on task completion, threat field strength, and collision frequency, prioritizing the selection of high-quality samples for distillation to avoid "inferior experience misleading multiple agents" and ensure the reliability of transfer experience.

[0096] Joint loss function, balancing imitation and innovation: It combines distillation loss (ensuring that multiple agents inherit the superior experience of a single agent) and Critic loss (ensuring autonomous collaboration among multiple agents). Compared with a single loss function, it avoids both "insufficient collaboration due to complete imitation" and "lack of basic skills due to autonomous learning".

[0097] Dynamic distillation weights enable a smooth transition: the distillation weights gradually decrease as the training progresses, allowing the multi-agent system to gradually transition from "imitating a single agent" to "autonomous collaborative decision-making," thus solving the problem of "imitation and autonomy conflict" in traditional experience transfer, ensuring a stable training process, and ultimately achieving better collaborative performance.

[0098] Case Study: This model constructs a hypothetical scenario based on the Mozi combat simulation platform and extracts the UAV's heading, speed, altitude, latitude and longitude, damage, status, and payload parameters from the scenario. The threat field of dynamic obstacles is calculated using the intent analysis module. The time step is 1 second, and the total simulation time is 3000 seconds. A discount factor is applied during the single-agent reinforcement learning phase. =0.99, maximum step size max_step=2000, experience pool size buffer_size= The Critic learning rate is The Actor learning rate is The batch size is 128. In the policy distillation phase, after single-agent training is completed, high-quality trajectory samples are extracted from their experience pool to construct a distillation set. Then, the first two layers of the single-agent policy network are copied to the multi-agent policy network to achieve initial distillation transfer. The distillation sample ratio is p_distill= Distillation of initial weights during multi-agent training phase batch_size=256, discount factor =0.97, maximum step size max_step=3000, experience pool size buffer_size= Target network soft update .

[0099] On a test set of 50 typical scenarios (100 independent trajectories in total, repeated experiments using 5 random seeds), the average trajectory deviation after distillation was determined by... m drops to m( Threat avoidance rate increased from 86% to 92%. In addition, distillation significantly shortened training convergence time (approximately 1.5 hours) while having a negligible impact on inference latency.

[0100] After the experiment, the reliability of the path prediction was verified by comparing it with the expert experience path. The expert experience path is a reference path formulated by domain experts with rich combat or mission planning experience, based on the mission scenario, threat situation, and UAV characteristics. Comparisons were made from 50 typical scenarios in the test set, with an average trajectory deviation of 3.4 meters, a dynamic offset rate of 6%, and a simulated threat avoidance rate of 92%. The results show that the simulated predicted path is highly consistent with the expert experience path, and the avoidance rate is within the normal error range.

[0101] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A MADDPG-based multi-agent intelligent pathfinding and real-time decision-making method, characterized in that, The method comprises the following steps: S1, constructing a training environment and collecting data: constructing a single-agent task scenario in a simulation platform, determining the type of agent, the type of dynamic obstacle, and the type of target, and collecting time series data with a preset length of time window, wherein the time series data comprises state feature data of red and blue parties; S2, dynamic obstacle intention recognition: inputting the time series data collected in step S1 into an intention analysis module, outputting a short-term intention recognition result of the dynamic obstacle through a long short-term memory network based on an attention mechanism, and the short-term intention comprising a moving direction, an interception intention, and a patrol route; S3, distance threat field model calculation: calculating a distance threat field value based on the maximum reconnaissance distance, the maximum attack distance, and the real-time distance between the agent and the target, and quantifying the environmental risk; S4, single-agent pre-training: based on the intention recognition result of step S2 and the threat field value of step S3, combining the heading, speed, altitude, longitude and latitude, damage, state, and state parameters of the agent, training the single agent through reinforcement learning, updating the value network parameters and the policy network parameters, and forming a transferable state-action experience set; S5, policy distillation and experience transfer: copying the first two layers of parameters of the single-agent policy network to the multi-agent policy network, and constructing a joint optimization target of distillation loss and critic loss; S6, multi-agent collaborative training: based on a centralized training-distributed execution architecture, using a MADDPG algorithm to jointly train the multi-agent, dynamically adjusting the distillation weight during the training process, and realizing a smooth transition from supervised distillation to autonomous cooperation; S7, real-time decision deployment: deploying the trained strategy to each agent, and each agent making autonomous decisions based on local observations and pre-trained strategies in the execution phase to realize cooperative pathfinding and dynamic obstacle avoidance.

2. The MADDPG-based multi-agent intelligent pathfinding and real-time decision-making method according to claim 1, characterized in that, The specific implementation process of the intention analysis module in step S2 is as follows: S21, constructing an encoder based on an attention mechanism: taking the time series data as a key, taking the hidden layer state and unit state of the encoder as a query, calculating the correlation between the query and the key to obtain an attention weight, and extracting key time sequence features after softmax normalization; S22, constructing a decoder based on an attention mechanism: based on the hidden layer state and the cell state of the previous moment of the decoder, the attention weight of each encoder hidden layer is calculated, and the tactical intention result is obtained by weighted sum of the encoder output features ; S23, intention output: inputting the intention vector obtained by the softmax layer regression of the decoder output result into a pathfinding decision module.

3. The MADDPG-based multi-agent intelligent pathfinding and real-time decision-making method according to claim 1, characterized in that, The attention mechanism-based encoder specifically extracts a feature representation of a key moment from a time sequence, retains time sequence dependence, can adaptively select a sequence with high relevance for encoding, and solves the problem of limited capacity of the encoded fixed-length vector; the encoder integrates the entire sequence information of the kth time sequence As the key, the hidden layer And the cell state As the query, the attention weight is calculated by the query and the key, and then the attention degree of each sequence is obtained by normalizing the softmax : wherein, , , are learning parameters, is an attention weight measuring the importance of the kth input feature at time t, a softmax function is applied to ensure that the sum of all attention weights is 1; through these attention weights, the driving sequence can be adaptively extracted. The hidden state at time t can be updated as: wherein is an LSTM unit that can be computed in terms of three sigmoid gates, wherein instead of With the use of the input attention mechanism, the encoder selectively focuses on important time series, rather than treating all input time series equally; The attention mechanism-based decoder is specifically: the attention weight of each encoder hidden layer at time t is calculated based on the previous decoder hidden layer and the cell state of the LSTM unit is calculated; in, It is a concatenation of the previous hidden layer and the cell layer of the LSTM unit. , and The attention weights are the parameters that need to be learned. This represents the importance of the i-th encoder hidden layer to the prediction, where each encoder hidden layer... This is mapped to a temporal component of the input; therefore, all the aforementioned encoder hidden layers are then processed through an attention mechanism. , ,…, Weighted summation to calculate the tactical intent result. .

4. The MADDPG-based multi-agent intelligent pathfinding and real-time decision-making method according to claim 1, characterized in that, The calculation basis of the distance threat field model in the step S3 is that the maximum attack distance of the intelligent agent W i is r ai , the maximum reconnaissance distance is r ri , the maximum attack distance of the target T j is r taj , the maximum reconnaissance distance is r trj , the threat strength generated by the interaction of W i and T j is quantified by the real-time distance between W i and T j , and the distance threat field value T rij is obtained.

5. The MADDPG-based multi-agent intelligent pathfinding and real-time decision-making method according to claim 1, characterized in that, The single-agent pre-training in step S4 is specifically as follows: Let a single unmanned aerial vehicle learn to autonomously avoid obstacles, evade threats from ground-to-air missile batteries, fly energy-efficiently, and complete a mission to attack a tank battery in a complex, dynamic threat environment; the input states of the unmanned aerial vehicle include threat field values : wherein, is the drone position, is the drone velocity, is the dynamic obstacle unit relative position; perform action through reward function and get new state with immediate reward store the sample into the experience pool: And periodically update the threat field: wherein critic updates for each sampled batch (B) from the intent analysis module: ) are performed: The critic updates can obtain the single drone value network parameters The actor updates are performed again: Actor update can get single UAV strategy parameters .

6. The MADDPG-based multi-agent intelligent route searching and real-time decision method according to claim 5, characterized in that, The reward function of the single-agent pre-training in the step S4 is designed with multiple targets, and the expression is as follows: wherein a higher weight is adopted during single-agent pre-training a threat penalty to strengthen the avoidance of the unmanned aerial vehicle to the threat; the detailed reward definition is as follows: wherein, is the amount of increase in the distance to the goal in this step, used to encourage a faster path to the goal; wherein, is the weight, is the consumption of this step, since the drone has an energy upper limit, low energy consuming actions are encouraged; wherein, is a weight, is a threat field, when the unmanned aerial vehicle is in a high-threat area, a large amount of punishment is received, so that the unmanned aerial vehicle actively avoids the threat area; wherein, is a collision penalty coefficient, since experience needs to be transferred to multi-UAVs, it is necessary to prevent multi-UAVs from colliding with each other, and is set to 0 when a single UAV is trained.

7. The MADDPG-based multi-agent intelligent route searching and real-time decision method according to claim 5, characterized in that, In step S5, copying the first two layers of parameters of the single-agent policy network to the multi-agent policy network is specifically as follows: two parameters obtained in the single drone pre-training , processing, copying the parameters of the first two layers to the strategy parameters of the multi-drone ​​ Copying the first two layers enables the multi-UAV to have a unified representation of the threat field in the early stage, so that it does not have to learn the basic perception function again in multi-agent training; The training loop performs critic and actor updates for each agent i simultaneously, with the critic having access to global information With action From the mixed experience pool Priority sampling mini-batches Policy network With Q network Updates are performed as follows: where each action , is the observation of the jth agent at the next time instant. For the i-th agent in MADDPG, the Q network loss is defined as: a centralized critic for the ith agent, denotes a smoothed copy to stabilize training, is the immediate reward for the ith agent, is the action actually performed at training time; In MADDPG, the gradient of the actor is: wherein is a policy network for the jth drone, is a centralized critic for computing the direction of action gradient, o is sampled from the mixed experience pool wherein contains features from intent recognition .

8. The MADDPG-based multi-agent intelligent route searching and real-time decision method according to claim 5, characterized in that, In step S5, the specific implementation of constructing the joint optimization target of distillation loss and critic loss includes: S51, sample screening: extracting high-quality trajectory samples from the single-agent experience pool, constructing sample weights w based on task completion, threat field intensity, and collision frequency, the expression is: wherein, = 0.4, = 0.35, = 0.25, = 3.0, = 0.3, 1success_flag is the task completion identifier, threat_level is the threat field level, and collision_count is the collision count. S52, parameter migration: copying the first two layers of parameters of the single-agent policy network to the multi-agent policy network, so that the multi-agent initially has threat field characterization and basic obstacle avoidance capability; S53, loss construction: introduce distillation supervision in multi-agent training to distill set Guiding multi-agent to imitate the experience of single-agent to evade threats and complete tasks, for the i-th UAV, the distillation loss is defined as: The critic loss and the distillation loss are combined into a training target of each agent: Distillation weight Is defined as follows: = 0.5, = 0.05, is the number of preheating steps, is the number of distillation steps.

9. The MADDPG-based multi-agent intelligent route searching and real-time decision method according to claim 1, characterized in that, In the step S1, the simulation platform is a Mozi confrontation simulation platform, the agent adopts an MQ-1C type "gray eagle" unmanned aerial vehicle, the dynamic obstacle is an air-to-ground missile battery, the target is a tank battery, and the preset length of the time window is 30, that is, the time sequence data from t-29 to t is collected.

10. The MADDPG-based multi-agent intelligent routing and real-time decision-making method according to claim 1, characterized in that, the state vector of the single agent in the step S4 wherein is the position, is the velocity, is the local observation, is the dynamic obstacle intention perception, is the task state, the action vector is defined as denotes the velocity / heading continuous control.

Images