Multi-agent cooperative navigation optimization method

Abstract

The application discloses a multi-agent cooperative navigation optimization method, and the optimization method takes a multi-agent deep deterministic policy gradient algorithm as a basic framework; the following mechanism is introduced in the training stage: in the multi-agent cooperative navigation training stage, a joint priority experience extraction mechanism is introduced, joint experience samples are extracted and spliced according to the time difference error priority through cross-agent synchronization indexing; an information discarding and correction compensation mechanism is introduced at the input end of the centralized Critic network, information from other agents is divided into independent information blocks, the information blocks are randomly discarded with a preset probability, and a correction factor is applied to the retained information blocks to maintain the mathematical expectation of the input data unbiased, and the input of the reduced Critic network is obtained. The application provides a multi-agent cooperative navigation optimization method which can improve the key experience utilization rate, reduce the dependence of the model on high-dimensional global input, and enhance the robustness in a communication limited environment.

Description

Technical Field

[0001] This invention relates to the field of multi-agent cooperative control and autonomous navigation technology, specifically to a multi-agent cooperative navigation optimization method based on deep reinforcement learning, and in particular to a multi-agent deep deterministic policy gradient algorithm that integrates priority experience extraction and information discarding techniques. Background Technology

[0002] With the development of artificial intelligence and robotics, multi-agent cooperative navigation technology has shown great application potential in military and civilian fields such as UAV formations, mobile robot swarms, and warehousing and logistics. Traditional navigation methods mostly rely on accurate prior environmental models, which are difficult to adapt to dynamic, unknown, and complex environments. They are prone to getting stuck in local optima and cannot effectively coordinate the dynamic interactions between multiple agents, resulting in a sharp decline in the overall system performance as the number of agents increases.

[0003] Multi-Agent Deep Deterministic Policy Gradient (MADDPG) is a representative method in the field of multi-agent reinforcement learning. It employs a "centralized training, distributed execution" framework, using a centralized Critic network capable of acquiring global information to guide multiple Actor networks, each possessing only local observations, in policy learning. This partially addresses the problems of environmental non-stationarity and the limitations of local observations. However, existing MADDPG-based cooperative navigation methods still suffer from the following significant drawbacks: First, the uniform random experience replay mechanism is usually adopted, which cannot effectively identify and reuse those rare but key high-value experience samples, resulting in slow algorithm convergence speed and unstable training process.

[0004] Second, the input of a centralized Critic network requires splicing information from all agents, and its dimensionality increases linearly with the number of agents. This not only increases the network size and exacerbates gradient noise, but also easily leads to oscillations during training and getting stuck in local optima.

[0005] Third, over-reliance on complete global information during the training phase may cause the learned strategy to experience a sharp drop in performance during actual execution due to communication limitations and information gaps, resulting in a discrepancy between training and execution. Summary of the Invention

[0006] This invention aims to overcome the aforementioned shortcomings of existing technologies and provide a multi-agent cooperative navigation optimization method based on a joint priority experience extraction mechanism and information discarding technology. This method uses the multi-agent deep deterministic policy gradient algorithm as its baseline framework. By introducing a joint priority experience extraction mechanism during the multi-agent cooperative navigation training phase and introducing an information discarding and correction compensation mechanism at the input of the centralized Critic network, it can improve the utilization rate of key experiences, reduce the model's dependence on high-dimensional global input, and enhance robustness under communication-constrained conditions. The specific scheme is as follows: A multi-agent cooperative navigation optimization method is proposed, which uses the multi-agent deep deterministic policy gradient (MADDPG) algorithm as its basic framework. It employs a centralized training and distributed execution architecture, and integrates the following optimization mechanisms synchronously during the training phase: During the multi-agent cooperative navigation training phase, a joint priority experience extraction mechanism is introduced to assign a priority to each experience sample in the experience replay pool based on its temporal difference error, and non-uniform sampling is performed according to this priority; at the same time, a cross-agent synchronous indexing mechanism is adopted to splice together joint experience samples for centralized training. An information discarding and correction compensation mechanism is introduced at the input end of the centralized Critic network. Information from other agents is divided into independent information blocks, and the information blocks are randomly discarded with a preset probability. A correction factor is applied to the retained information blocks to maintain the mathematical expectation of the input data as unbiased, thus obtaining the input of the dimensionality-reduced Critic network.

[0007] Preferably, in the introduced joint priority experience extraction mechanism, the priority is calculated as follows: ; Where, δ k Let ε be the temporal difference error of the kth empirical sample, ε be the smoothing constant, and α be the priority intensity coefficient. The sampling probability is: ; Importance sampling weights are: ; After normalization, we get: ; : ; Where M is the experience pool capacity and β is the importance sampling correction parameter.

[0008] Preferably, the joint priority experience extraction mechanism uses a prefix sum tree structure to achieve efficient sampling according to priority ratio, and the initial priority of newly stored samples in the experience replay pool is set to the highest priority in the current pool.

[0009] Preferably, the cross-agent synchronization indexing mechanism specifically comprises: In each sampling, all agents extract local experiences at the same time from their respective experience replay pools based on the same set of randomly generated indices, and concatenate these local experiences in the order of the agents to form a joint experience sample for centralized training.

[0010] Preferably, the information discarding and correction compensation mechanism specifically includes: generating a binary mask that follows a Bernoulli distribution for each information block from other agents to perform block-level discarding; retaining information blocks with a retention probability of 1-p, and multiplying the retained information blocks by a correction factor 1 / (1-p), where p is a preset information loss rate; and 0 < p < 1.

[0011] Preferably, the training phase of the multi-agent cooperative navigation includes the following steps: S1, In the current training round, each agent selects actions and interacts with the environment through the Actor network based on its local observations. The joint experience samples generated by the interaction, including global state, joint actions, immediate rewards, global information of the next state, and termination flags, are stored in their respective experience replay pools, and new samples are assigned initial priorities. S2, when the number of samples in the experience replay pool reaches a set threshold, a batch of sample indices is extracted from the pool according to the joint priority experience extraction mechanism, and a batch of joint experience samples is formed by using a cross-agent synchronous indexing mechanism; S3, for each joint experience sample in the batch, the information blocks from other agents in the centralized Critic network input are processed according to the information discarding and correction compensation mechanism to obtain the dimensionality-reduced Critic network input; S4. Based on the input of the dimensionality-reduced Critic network, calculate the target Q value and Critic loss of each agent, and update the Critic network parameters by combining the importance sampling weights; then use the updated Critic network to calculate the policy gradient of each Actor network to update the target network parameters. S5, using the new temporal difference error calculated in S4, update the priority of the sampled samples in the current batch in the priority experience replay pool; S6, repeat S1 to S5 until the termination condition of round training is met.

[0012] Preferably, the information block includes: the agent's own state information block, target information block, obstacle information block, and teammate message information block from other agents; wherein, the information discarding takes priority on the teammate message information block.

[0013] Preferably, different information loss rates are applied to information blocks with different semantic meanings.

[0014] The beneficial effects that this application can produce include: This application uses the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm as its basic framework. During the training phase, it synchronously integrates a Joint Priority Experience Extraction (PER) mechanism and an information discarding and correction technique. The PER mechanism defines sample priority based on temporal difference error, employs non-uniform sampling and importance sampling weights, and focuses on learning experiences at critical decision moments, concentrating on high-value experiences. This avoids the sample waste caused by uniform replay, resulting in a faster rise and stabilization of the average reward curve, reducing the number of training rounds required to achieve the desired performance. This accelerates convergence and improves training efficiency. The Message-Dropout technique randomly discards block-level information from other agents' inputs to the Critic network. This reduces the input complexity and training difficulty of the Critic network, while forcing both the Critic and Actor networks to learn robust policies independent of complete information from any single agent. A correction factor maintains unbiased input expectations, reduces computational dimensionality, and enhances the model's robustness in communication-constrained environments. This allows for a high task success rate even when encountering unforeseen situations such as packet loss or new obstacles during testing. This application integrates the aforementioned priority experience extraction mechanism and message-dropout technique into the centralized training and distributed execution framework of MADDPG through the synergy of Sample Learning Optimization (PER) and Model Generalization Optimization (Message-Dropout). During training, the Critic evaluates using global information and joint actions after message-dropout and correction, and samples from an experience pool using the PER mechanism; Actors update based on local observations. During execution, each agent makes independent decisions using only its own local Actor network based on local observations. The PER mechanism optimizes the quality and efficiency of training data utilization, while the Message-Dropout technique optimizes the stability, efficiency, and adaptability to execution phase conditions (information-constrained) during training. The combination of these two techniques enhances the overall performance of the algorithm from both sample learning and model generalization perspectives, resulting in a higher task success rate under various obstacle environments.

[0015] Meanwhile, the information block discarding mechanism proposed in this application can flexibly adapt to different numbers of agents. By adjusting the discarding rate, a good balance can be achieved between model performance and input dimension, making it suitable for collaborative navigation tasks of large-scale agent clusters.

[0016] This application strictly follows the principle of centralized training and distributed execution. After training, each agent only needs to run a lightweight local Actor network without the need for real-time global communication, which is very suitable for deployment on edge devices with limited computing and communication resources. Attached Figure Description

[0017] Figure 1 This diagram illustrates a comparison of the task success rates of the optimization method (PD-MADDPG) described in this application and the baseline algorithm in a dynamic obstacle environment. Detailed Implementation

[0018] The optimization method uses the multi-agent deep deterministic policy gradient (MADDPG) algorithm as its basic framework; it adopts a centralized training and distributed execution architecture, and integrates the following optimization mechanisms simultaneously during the training phase: During the multi-agent cooperative navigation training phase, a joint priority experience extraction mechanism is introduced to assign a priority to each experience sample in the experience replay pool based on its temporal difference error, and non-uniform sampling is performed according to this priority; at the same time, a cross-agent synchronous indexing mechanism is adopted to splice together joint experience samples for centralized training. An information discarding and correction compensation mechanism is introduced at the input end of the centralized Critic network. Information from other agents is divided into independent information blocks, and the information blocks are randomly discarded with a preset probability. A correction factor is applied to the retained information blocks to maintain the mathematical expectation of the input data as unbiased, thus obtaining the input of the dimensionality-reduced Critic network.

[0019] It should be noted that: In this application, a joint priority experience extraction mechanism is introduced during the training phase of the multi-agent deep deterministic policy gradient algorithm. Within the training framework of the multi-agent deep deterministic policy gradient algorithm, the joint priority experience extraction mechanism and information discarding and correction compensation techniques are introduced simultaneously.

[0020] The joint priority experience extraction mechanism calculates the temporal difference error of experience samples as their importance priority and performs non-uniform sampling based on this priority. This increases the training frequency of high-value experiences, such as collision boundaries and successful collaborations, accelerating model convergence and improving policy quality. The information discarding and correction compensation technique, at the input of the centralized Critic network, randomly discards information from other agents at the block level and performs scaling correction on the retained information to maintain the unbiasedness of value estimation. This effectively reduces the input dimensionality of the Critic network, alleviates the curse of dimensionality, and simulates and adapts to possible communication information loss during the execution phase, thereby enhancing the robustness and generalization ability of the learned policy. By simultaneously integrating the above optimization mechanisms into the multi-agent deep deterministic policy gradient algorithm, the PD-MADDPG algorithm is formed, achieving synergistic optimization of training efficiency and model performance.

[0021] The specific construction process of the PD-MADDPG algorithm is as follows: The MADDPG basic framework is established using the centralized training and distributed execution framework as its foundation: (1) Establish a two-dimensional kinematic model of multiple agents and obstacles, and define the agent state s. i (t), action a i (t) The specific process of establishing a two-dimensional kinematic model of multiple agents and obstacles based on the state update rule is as follows: Let the task space be a two-dimensional plane, with a step length of Δt. The state of the i-th agent at time t is: ; Action is defined as acceleration The status has been updated to:

[0022] Obstacles can be set to static (speed is 0) or dynamic.

[0023] (2) The cooperative navigation task is formalized as a multi-agent Markov game. The specific process of constructing a multi-agent Markov game model (centralized training, distributed execution) is as follows: Model cooperative navigation as a quintuple: ; in A set of intelligent agents; joint state Joint Action ; transfer Joint Awards .

[0024] In distributed execution, the policy of the i-th agent depends only on local observations. i : ; During centralized training, the Critic receives global information x (which can be obtained by stitching together all observations) and joint action a:

[0025] (3) Define a hybrid navigation reward function r for each agent, which includes arrival reward, collision penalty, obstacle proximity penalty and time step penalty. i (t).

[0026] Among them, the navigation reward function r i The definition process of (t) is as follows: For each agent i, let the target point be... Let the set of obstacles be Ω. At time t, construct the reward: ; Among them, the arrival reward is: ; like Termination of positive rewards .

[0027] Collision penalty (with obstacles or other machines): ; Near-impairment penalty / safe boundary: ; Time step penalty:

[0028] (4) The update formulas for the Critic and Actor networks using the MADDPG standard are adopted, including the calculation of the target Q value y. i Critic loss and Actor gradient θ i J.

[0029] Secondly, construct P-MADDPG (integrating priority empirical extraction and parameter noise): Based on the established MADDPG framework, an optimization mechanism is integrated for the first time to construct the P-MADDPG algorithm: Introduce a Priority Experience Extraction (PER) mechanism and construct a global experience concatenation mechanism: Calculate the temporal difference TD error δ for each empirical sample k in the empirical replay pool. k .

[0030]

[0031] Error δ of time-differential TDk Calculate sample priority .

[0032] Non-uniform sampling is performed based on priority, with a sampling probability of... .

[0033] To correct for sampling bias, the importance sampling weights w are calculated and normalized. k : ; Used for weighted Critic loss: .

[0034] Efficient sampling is achieved using a prefix sum tree (Sum-Tree) data structure, and an initial priority is set for new samples.

[0035] A cross-agent synchronous indexing mechanism is adopted to ensure that experiences from the same time point are extracted from the experience pools of each agent to form joint training samples.

[0036] Introducing parametric noise enhancement exploration: During the agent action selection phase, exploration noise is added to the action output by the Actor network. ,Right now To improve the efficiency of strategy exploration.

[0037] Finally, PD-MADDPG (Integrated Information Dropout and Correction Compensation) is constructed.

[0038] Based on the P-MADDPG algorithm described above, a second optimization mechanism is further integrated to finally construct the PD-MADDPG algorithm.

[0039] Introduce an information dropout and correction compensation mechanism at the input end of the centralized Critic network: This invention employs block-wise information discarding: information from other agents is discarded. Observations, actions, etc., can be categorized into message / local observation splicing blocks based on semantics or source, denoted as...

[0040] Generate a binary mask for each block: for each information block, use a preset probability p (information loss rate). Generate a binary mask that follows a Bernoulli distribution. , and perform block-level random discarding.

[0041] For the information blocks that were not discarded, multiply by the correction factor 1 / (1-p) to obtain To maintain the unbiased mathematical expectation of the input data.

[0042] The agent's own observations i This data is concatenated with other processed agent information blocks to form the input to the Critic network, which is dimensionality-reduced and fault-tolerant. .

[0043] In this input Calculating the Q value on the joint action a Achieve an unbiased approximation of "discarding + correcting".

[0044] The PD-MADDPG algorithm was finally constructed. Its training phase incorporates both a joint priority experience extraction mechanism and an information discarding and correction compensation mechanism at the Critic input. During execution, each agent relies solely on its local Actor network and local observations for decision-making, meeting the requirements of distributed execution. This algorithm aims to improve the utilization rate of key experiences, reduce dependence on high-dimensional global inputs, and enhance robustness under non-ideal conditions such as limited communication.

[0045] Furthermore, the training phase of the multi-agent cooperative navigation includes the following steps: Step 1: Environment and Policy Model Initialization: Construct an environmental dynamics model that includes multiple agents and obstacles, and define the local dynamics of each agent. The joint reward function for the observation space, action space, and cooperative navigation task is defined. The Actor network, Critic network, and their corresponding target network parameters for each agent are initialized, and a priority experience replay pool is established.

[0046] Step 2: Initialize the training rounds: Begin a new training round. Reset the environment and set the initial state of the agent, such as intelligence. The body's position, speed, and maximum number of moves T in a round. max The system generates static / dynamic obstacle and target point locations randomly or according to a preset configuration.

[0047] Step 3: Agent-Environment Interaction and Experience Storage: For each time step t in the current training round, perform the following sub-steps: Action selection: Each agent i selects actions based on its current local observation o. i (t), which outputs the basic action μ through its Actor network. i (o i (t)), and add exploration noise. This forms the final execution action. .

[0048] Contextual execution: All agents execute joint action a t =[a1(t),...,aN [t], the environment advances by one time step Δt, and returns the reward r for each agent. i (t), Local observation at the next time step o i (t+1) and the flag indicating whether the round has ended (d) t .

[0049] Experience building and storage: The current global state x t (or a composite of observations from all agents), joint action a t Rewards r t Next global state x t+1 and termination marker d t Construct a joint experience sample (x) t ,a t ,r t ,x t+1 ,d t ).

[0050] Priority initialization: Store the new experience in the priority experience replay pool and set its initial priority to the highest priority of all samples in the current pool to ensure that the new experience can be sampled quickly.

[0051] Step 4: Trigger the neural network update: When the amount of data in the experience replay pool reaches the preset batch update threshold, for example, when the capacity is greater than... When the batch size is B, the current interactive loop is interrupted, and a batch of neural network parameter updates is performed, as follows: Priority Experience Sampling: Calculate the sampling probability P(k) of each sample in the experience pool based on its priority πk, and use a prefix sum tree (Sum-Tree) data structure to efficiently extract a batch (size B) of experience sample indices.

[0052] Semantically, for example, a teammate message block divides information from other agents into blocks. .

[0053] For each information block, generate a Bernoulli mask with a preset loss rate p. A correction factor of 1 / (1-p) is applied to the retained information block, and the input is discarded and corrected to preserve... The input is expected to be unbiased.

[0054] Observing the agent itself i The input to the Critic network is concatenated with other processed agent information blocks to form a dimensionality-reduced network. .

[0055] Network parameter update: Update the Critic network: Use the input processed in step 4. Combined with action a, calculate the target Q value y. i and the current Q value ; For agent i, the objective Q-value is: ; Using importance sampling weights Weighted TD error, calculate Critic loss The Critic network parameters are updated using gradient descent.

[0056] Update the Actor network: Utilize the gradients provided by the updated Critic network. ai Q i Using the policy gradient formula Update the Actor network parameters for each agent: Soft update the target network: Softly update the parameters of the target Actor network and the target Critic network for all agents by a small scale τ. ; Priority Re-update: Use the new TD error δk calculated in this update to update the priority of the sampled samples in the experience replay pool. .

[0057] Multi-agent global experience splicing extraction method: After sampling by one agent, the samples are spliced ​​into a joint sample from the experience pools of the other agents.

[0058] Step 5: Loop Control and Termination: Repeat steps three and four until the current training round results in all agents reaching the goal and issuing [a response / initiative]. Collision or reaching the maximum number of steps T max Then terminate. Return to step two to begin the next training round. Repeat this process until the cooperative navigation policy performance of the agents converges to meet the requirements.

[0059] Furthermore, to ensure the consistency of joint experience, the joint priority experience extraction adopts a cross-agent synchronous indexing mechanism. Specifically, in a certain sampling round, each agent retrieves the local experience corresponding to the time moment from its own experience pool according to the same set of sampling indices, and splices them together in the order of the agents to form a joint experience sample, which is used for updating the Critic and Actor during the centralized training phase.

[0060] Furthermore, the joint priority experience extraction mechanism employs a prefix sum tree structure (Sum-Tree) to achieve efficient sampling according to priority proportions, and the initial priority of newly added samples to the experience replay pool is set to the highest priority in the current pool. Specifically: A binary tree is constructed using the priority of each experience sample as the weight of the leaf nodes, with the parent node's weight being the sum of its child node weights. During sampling, the total priority interval is divided into several sub-intervals, and random numbers are generated within each sub-interval. The corresponding leaf nodes are located by searching from top to bottom using the tree structure, thus achieving efficient sampling according to priority proportions. The initial priority of newly added experience samples can be set to the highest priority in the current experience pool to ensure that new experiences are sampled promptly and participate in network updates. Priority strength parameter. The range of values ​​can be: Importance sampling correction parameters It can be gradually increased to close to 1 as the training progresses, in order to reduce the estimation bias caused by priority sampling; To avoid smoothing terms with a priority of 0, a small positive number can be used. The above parameters can be adjusted according to environmental complexity and training stability requirements.

[0061] Furthermore, the information discarding and correction compensation mechanism specifically includes: generating a binary mask that follows a Bernoulli distribution for each information block from other agents to perform block-level discarding; and multiplying the information blocks that are not discarded by a correction factor 1 / (1-p), where p is the information loss rate.

[0062] In this application, the message information can be divided into multiple information blocks according to semantics or source, including but not limited to: ontology state information block, target information block, obstacle information block, and teammate message information block; wherein information discarding is preferentially applied to teammate message information block, so as to maintain the ability to perceive key environmental elements while reducing the criterion input dimension. Different information blocks can adopt different information loss rates. For example, a higher loss rate can be set for message blocks from distant teammates, while a lower loss rate can be set for message blocks from nearby teammates or teammates with high collaboration needs, in order to achieve a balance between refined dimensionality compression and collaborative performance. To maintain the unbiasedness of the estimation, a correction factor is introduced for the retained information blocks, so that the expected value of the input after discarding is consistent with that of the input before discarding, thereby ensuring that the estimation of the value network is not subject to systematic bias due to random discarding.

[0063] Training phase: Centralized Critic reception Joint actions; experience replay uses PER; Actor distributed input local observation. Execution phase: Each agent only uses its local Actor to output actions, requiring no global information, adapting to communication-constrained scenarios.

[0064] It should be noted that Table 1 defines the input and output variables involved in this invention; Table 1

[0065] The present application is described in detail below with reference to the embodiments, but the present application is not limited to these embodiments.

[0066] Example 1: This embodiment provides a specific example of the optimized collaborative obstacle avoidance navigation described in this application, placed in a static obstacle environment. In this embodiment, taking the example of three agents collaboratively navigating to a designated target point within a two-dimensional plane containing varying numbers of static circular obstacles, the specific implementation process of the optimization method described in this application is explained.

[0067] Step 1: Build the task scenario environment and initialize parameters: 1) Define the task space: The task space is set as a two-dimensional finite region, which can be normalized to [-100, 100] × [-100, 100]. 2) Set the number of agents N=3, each agent corresponds to one target point, the number of target regions is 3, and a discrete-time kinematics model is adopted. The agent state includes position and velocity, and the action is two-dimensional acceleration. 3) Set parameters such as the collision radius between the agent and obstacles, the threshold for determining the target point, and the maximum number of simulation steps; set the arrival criterion: when the distance between the agent and the target center is ≤ If the target is reached, it is considered to have reached the target. The collision criterion is set as follows: if the agent overlaps with the circular area of ​​the obstacle, a collision is determined and the agent's turn is terminated.

[0068] 4) Obstacles are set as static circular obstacle areas; different obstacle density environments are set: no obstacles, 5 obstacles, 10 obstacles, 15 obstacles, and 20 obstacles. Initialize the initial position and initial velocity of all agents; here, the initial velocity is set to 0.

[0069] Step 2, construct the observations (i.e., local observations) and joint inputs (i.e., centralized training): 1) For each agent i, local observation This includes: its own position and velocity, relative target vector, relative position information of the nearest obstacles detected by the sensor, and relative position information of other intelligent agents within the communication range. To reduce the influence of dimensions, the features such as position, velocity, and distance are normalized. The specific method is to divide the position coordinates by the radius of the environment area, the velocity by the maximum velocity, and the distance by the maximum range of the sensor.

[0070] 2) Construct the Actor network and Critic network for each agent, both of which are fully connected neural networks. During centralized Critic training, the observations of each agent are concatenated into global information x. Furthermore, following the centralized Critic input in S6, an information dropout and correction compensation mechanism is introduced to perform message-dropout on information blocks from other agents, resulting in... The Actor network input is The output is a normalized acceleration action. The input to the Critic network is the processed global information and joint action. Simultaneously, a corresponding target network is created for each online network.

[0071] Step 3, Network Structure and Training Hyperparameter Settings: 1) Each agent contains an Actor and a Critic network as well as a corresponding target network; 2) Actor input is The output is an action. (The amplitude can be limited using tanh and mapped to) [ ]). Critic input is Combined with action 'a', the output is the Q value; 3) The target network uses soft updates: ; 4) The experience replay pool capacity is set to M, the batch size for each update is B, and the parallel training step size is at most per round. .

[0072] In one optional embodiment, both the Actor network and the Critic network employ a multi-layer fully connected structure. The number of hidden layers, the number of neurons per layer, and the type of activation function can be configured according to the task scale. Training hyperparameters include, but are not limited to, discount factor, learning rate, soft update coefficient, batch size, experience pool capacity, exploration noise type, and noise intensity. Furthermore, continuous states such as position, velocity, and relative distance can be normalized to improve network training stability. Action outputs can be mapped to the velocity / acceleration range allowed by the device through linear mapping or nonlinear compression.

[0073] Step 4, construct the priority experience replay pool and define priority updates: 1) Define the format of empirical tuples as follows: ; in This is a termination marker; 2) Initialize a priority experience replay pool with a fixed capacity. Use a prefix sum tree structure to manage experience samples and their priorities. The initial priority of newly added experience samples is set to the highest priority in the current pool to ensure that new experiences are fully sampled; after training, update the priorities according to the TD error. 3) To meet the needs of centralized Critic training, when an agent extracts an index set from its own experience pool, it needs to be concatenated with other agents' experience pools according to the same index to obtain joint samples, i.e., a global experience concatenation mechanism.

[0074] Step 5: Execute a closed-loop training cycle of "explore-store-update" within each training round: The overall training process in this embodiment can be summarized as follows: round initialization → loop interaction → batch update after conditions are met → round end judgment → enter the next round.

[0075] Within each training round, the following sub-steps are repeated: (1) Round initialization: Set the round count. =1, sets the maximum number of steps. .

[0076] Generate obstacle locations randomly or at fixed locations; place three agents at their respective starting points with a speed of 0; set the center points of the target area to target1, target2, and target3.

[0077] (2) Action selection and execution: For each agent i, based on the current local observation Actions output by the Actor network: ; in To explore noise.

[0078] The environment performs a joint action, advancing one step according to the action, and obtains the next state and reward. Termination mark .

[0079] (3) Refinement of reward calculation: Based on the reward function defined in the invention, the immediate reward obtained by each agent is calculated, specifically as follows: If the agent reaches the target threshold, issue the achievement reward and set [the appropriate parameters]. =1.

[0080] If a collision occurs with an obstacle, a collision penalty is issued and settings are configured. =1.

[0081] Otherwise, an immediate reward is given based on the negative distance to the target, the near obstacle penalty, and the time penalty.

[0082] (4) Experience storage and priority update: The obtained global experience (including observations, actions, rewards, next observations, etc. of each agent) is stored in the priority experience replay pool, and the initial TD error of the experience is calculated as its priority. Specifically, the joint experience is formed. And write it to the playback pool; Calculate / assign the initial priority of this experience This is used for subsequent priority sampling.

[0083] (5) Trigger network update: When the amount of data in the experience pool is sufficient, perform a batch update. That is, when the number of samples in the experience pool reaches a threshold, such as not less than B, then proceed with the batch update. a. Priority Sampling: Based on sample priority, a batch of experience sample indices is extracted using a prefix sum tree. Local experiences at corresponding times are then extracted from the experience pools of each agent based on the same index, concatenated into joint experience samples, and then... Draw a batch of samples from the experience pool. b. Importance Sampling Weight Calculation: Calculate and normalize the importance sampling IS weight for each selected sample. .

[0084] c. Information Drop-out Critic Forward: When constructing the Critic network input, information blocks from other agents (such as teammate location information blocks) are randomly dropped according to a preset drop-out rate p, and a correction factor 1 / (1-p) is applied to the retained blocks to form the dimensionality-reduced input x. hati .

[0085] d. Network parameter updates: use Weighted Critic loss function: Update the parameters of each Critic network.

[0086] Calculate the target Q value

[0087] Minimize weighted Critic loss update

[0088] Update Actor parameters using policy gradient.

[0089] Synchronize target network parameters using a soft update method.

[0090] (5) Priority Re-update: Use the new TD error calculated in this update to update the priority of the corresponding experience in the replay pool.

[0091] Step 6: Implement Message-Dropout dimensionality reduction and fault tolerance enhancement on the Critic side: 1) For each batch of samples, when constructing the Critic input, divide the information related to other agents' observations / actions into message blocks according to the agent dimension; 2) Generate a binary mask z for each message block according to the information loss rate p, set the discarded block to zero, and use... Perform correction and compensation to form ; 3) This approach can reduce dimensionality and improve training speed / robustness when the centralized Critic input dimension increases with the number of agents.

[0092] Step 7, Static environment testing and result reproduction after training (1000 rounds of testing): After training, test environments were generated either in a fixed or uniformly distributed manner, and 1000 rounds of testing were conducted in environments containing 0, 5, 10, 15, and 20 static obstacles respectively. Exploration noise was disabled during testing, and the agent made decisions solely based on its local Actor network. The task success rates of the three algorithms (IMDDPG, MADDPG, and PD-MADDPG) were statistically analyzed. The results are shown in Table 2. Table 2 Success Rate of Static Obstacle Environment Test Tasks (%)

[0093] As can be seen from Table 2, the optimization method described in the invention, namely PD-MADDPG, achieves a higher success rate at all obstacle densities. Although the success rate decreases with the increase of obstacles, it is still better than the comparison algorithm. It can be seen that the combination of priority experience extraction and Dropout makes the convergence and task completion performance better, thus verifying its superiority.

[0094] Example 2: This embodiment provides a specific example of the PD-MADDPG-based cooperative obstacle avoidance navigation optimization method in a dynamic obstacle environment. Building upon Embodiment 1, this embodiment further verifies the adaptability of the invention in dynamically changing environments.

[0095] Step 1, Dynamic Environment Construction and Barrier Attribute Initialization: The environment setup includes both static and dynamic obstacles. Dynamic obstacles can move along a preset trajectory (such as a uniform straight line or a random walk) or be activated and suddenly appear after a certain number of simulated steps to simulate a sudden threat.

[0096] 1) The number of agents remains 3, the number of targets is 3, the initial velocity is 0, and they are labeled agent1, agent2, and agent3 from left to right according to their initial positions; the target center points are labeled target1, target2, and target3 from left to right. 2) Environment A: 20 obstacle areas in total, including 14 static obstacles and 6 dynamic obstacles; Environment B: 25 obstacle areas in total, including 14 static obstacles and 11 dynamic obstacles. 3) Obstacle areas are described by their center position, radius, and motion attributes; dynamic obstacles can be generated / activated at predetermined times or moved according to a random walk model during the simulation. Examples of dynamic environment obstacle area attributes (Table 5-6), such as obstacle 1 being static, position (80,80), radius 14; obstacle 5 being dynamic, position (-10,60), radius 13, etc.

[0097] Step 2, the mechanism of movement and sudden onset of dynamic obstacles: 1) For dynamic obstacles, assume that the obstacle updates its position at a predetermined speed / acceleration at each step, or that it walks randomly within a limited area; 2) To improve the realism of simulating sudden threats, dynamic obstacles can be set to be activated at certain time steps, for example, a new dynamic obstacle appears at step 50; 3) When dynamic obstacles appear or movement causes changes in nearest neighbor obstacles, the agent's local observations Immediate change; the Message-Dropout and PER mechanisms of this invention do not depend on a fixed number of obstacles, and therefore can naturally adapt to changes in observation.

[0098] Step 3, apply PD-MADDPG for training and testing: The training process is exactly the same as step 4 of Example 1. The training process is still executed according to step 5 of Example 1: interactive sampling—PER pooling—batch update—target network soft update; the difference is that the obstacle state changes over time during environmental interaction, and the agent must learn to reach the target even in non-stationary local observations where new threats appear. Since the priority experience extraction mechanism focuses on the key decision experience generated by the appearance of dynamic obstacles, and the information discarding mechanism enhances the model's robustness to the missing parts of the observation (analogous to the sudden appearance of dynamic obstacles causing abrupt changes in local observations), this method can naturally adapt to dynamic environments.

[0099] Step 4, Task Completion Process and Key Milestones in a Dynamic Environment: Taking a 20-obstacle environment as an example: After starting from the starting point, the agent can avoid static obstacles and avoid newly appearing dynamic obstacles; after 193 steps, all three agents successfully reach their respective task areas.

[0100] When implementing the project, this "milestone step count" can be used as a debugging indicator: if the training is good, the total number of steps in the test rounds should decrease as the training progresses, while the success rate increases.

[0101] Step 5, Dynamic Environment Testing Comparison and Result Reproduction: The same 1000 rounds were tested in a dynamic environment to calculate the success rate of different algorithms. The tests were conducted in environments containing 20 and 25 obstacles (some of which were dynamic), and the results are shown in Table 3.

[0102] Table 3 Success Rate of Dynamic Obstacle Environment Testing Tasks (%)

[0103] As shown in Table 3, when the number of obstacles is 20, PD-MADDPG is 76.0%, MADDPG is 69.4%, and IMDDPG is 62.3%; when the number of obstacles is 25, PD-MADDPG is 69.7%, MADDPG is 62.0%, and IMDDPG is 59.2%. It is evident that PD-MADDPG maintains a significant advantage even with more complex dynamic obstacles, demonstrating the effectiveness of this invention in improving training efficiency and execution robustness through "priority experience extraction + information discard correction." Compared to the baseline method using random experience replay, this invention increases the training frequency of key samples by jointly prioritizing experience extraction, resulting in a faster rise and earlier stabilization of the average reward curve, thereby reducing the number of training rounds required to reach the preset success rate threshold. Furthermore, by reducing the criterion input dimension and enhancing robustness under communication constraints through information discard and correction mechanisms, higher success rates and a more stable training process can be achieved in both static and dynamic obstacle environments.

[0104] Example 3: This embodiment applies the present invention to the scenario of collaborative picking and obstacle avoidance of automated guided vehicles (AGVs) in warehousing and logistics, and provides an example of collaborative picking and obstacle avoidance of AGVs in warehousing. Step 1, Scene Mapping and Parameter Settings: The AGV is mapped as an intelligent agent, and the shelves and walls in the warehouse map are mapped as static obstacles, while moving personnel and other vehicles are mapped as dynamic obstacles. The AGV's starting point and picking / dropping point are mapped as the initial position and target point, respectively. Kinematic model parameters are set according to the AGV's physical characteristics (such as maximum speed and acceleration).

[0105] In a warehouse picking scenario, N Automated Guided Vehicles (AGVs) are deployed within a two-dimensional warehouse area, each starting from its own origin and proceeding to a picking / delivery location. The environment includes static obstacles (shelves, fixed workstations) and dynamic obstacles (personnel, forklifts, etc., temporarily moving through the area). The system consists of a training server (for centralized training) and AGV end-side controllers (for distributed execution). The communication network can be Wi-Fi / 5G / self-organizing network, allowing for obstructions, packet loss, or latency.

[0106] The training phase employs centralized training and distributed execution: the centralized Critic can utilize global information to complete strategy learning; during the execution phase, each AGV only runs a local Actor network, outputting actions based on local observations, without needing to obtain global information in real time, thus adapting to unstable warehouse communication scenarios.

[0107] Step 2, Integrating the model / algorithm with the scenario: The training end uses a centralized Critic to evaluate the impact of multi-AGV joint actions on "task completion / collision risk / cooperative conflict". The experience playback end introduces Joint Priority Experience Extraction (PER) to increase the frequency of key experience samples (passing each other in narrow passages, near collision boundaries, sudden appearance of dynamic obstacles) being extracted. At the same time, the Critic input end introduces Message-Dropout, which randomly discards "teammate message information blocks" and keeps the input expectation unchanged through a correction factor, so as to reduce the input dimension and enhance robustness under the condition of "teammate information missing".

[0108] Centralized training is performed on a central server using historical or simulation data, employing the PD-MADDPG algorithm described in this invention. After training, the local Actor network model for each AGV is distributed to the onboard controller of each AGV. In actual operation, each AGV acquires local observations through its own sensors (such as LiDAR and vision cameras). o i (Including its own pose, relative to the target, nearby obstacles, and the relative position of teammate AGVs), input into the local Actor network, calculate speed or acceleration control commands in real time, drive AGVs to complete collaborative transport and obstacle avoidance, the whole process does not require real-time and complete information interaction between AGVs or with the center.

[0109] Step 3, Input Data Settings: Each control cycle (e.g., 0.05s ~ 0.2s), the i-th AGV constructs a local observation vector. o i And organized by information blocks as follows: Body block: its own position ,speed ; Target block: relative to target vector Distance to target; Obstacle block: The relative position information or distance of the m nearest obstacles within the sensor range (m is fixed; if less than m, it is filled with a zero vector and accompanied by a mask to indicate the missing ones); Teammate message block: Relative position or relative speed information of up to K teammates within the communication radius (K is fixed; if less than K, it is also padded and accompanied by a mask).

[0110] During the training phase, the local observations of all AGVs are concatenated into a global input in numerical order. and combined with joint actions Input is centralized; during the execution phase, each AGV only uses local input. Input Actor, no global concatenation information is required (consistent with the requirement that no global information is required during the execution phase).

[0111] Step 4, Output Data Settings: The Actor network outputs continuous actions. It can be defined as a two-dimensional acceleration command. Or speed / angular velocity command Output After being limited, the amplitude is mapped to the vehicle's actuable range and converted into motor control signals by the onboard controller. The variable table in the manual clearly states that the output is an action command / acceleration.

[0112] Step 5, Relationship between global input X and information blocks during training: Actor observes locally The input consists of a body block and a target block, which determine the basic motion trend toward the target; an obstacle block, which determines the obstacle avoidance direction and safety margin; and a teammate message block, which determines the cooperative avoidance and yielding strategies. The centralized Critic outputs a Q-value using a global input x and a joint action a, which measures the impact of the action combination on task completion and collision risk. Message-Dropout is introduced into the teammate message block, and an unbiased approximation is maintained through a correction factor, enabling the Critic to perform stable evaluations even when teammate information is missing, thereby promoting the Actor's learning of robust cooperative strategies.

[0113] Step 6, Business Criteria and Termination Conditions: Arrival Criterion: When Time determination of arrival ( (To reach the threshold); Collision criterion: When the agent overlaps with the circular area of ​​the obstacle ( ) Determine a collision and terminate the agent's turn; Round ends: All reach the destination / Any collision / Maximum number of moves reached It is still not finished.

[0114] To verify the beneficial effects of this invention, under the same training framework, network structure, and training budget, a comparison group IMDDPG and MADDPG were set up, with the only difference being whether Joint Priority Experience Extraction (PER) and Message-Dropout mechanisms were introduced. The main results are as follows (attached). Figure 1 As shown.

[0115] From the appendix Figure 1 It can be seen that the PD-MADDPG used in this application exhibits the best success rate under all obstacle numbers, indicating that it outperforms other algorithms in complex environments.

Claims

1. A multi-agent cooperative navigation optimization method, characterized in that, The optimization method uses the multi-agent deep deterministic policy gradient (MADDPG) algorithm as its basic framework; it adopts a centralized training and distributed execution architecture, and integrates the following optimization mechanisms simultaneously during the training phase: During the multi-agent cooperative navigation training phase, a joint priority experience extraction mechanism is introduced to assign a priority to each experience sample in the experience replay pool based on its temporal difference error, and non-uniform sampling is performed according to this priority; at the same time, a cross-agent synchronous indexing mechanism is adopted to splice together joint experience samples for centralized training. An information discarding and correction compensation mechanism is introduced at the input end of the centralized Critic network. Information from other agents is divided into independent information blocks, and the information blocks are randomly discarded with a preset probability. A correction factor is applied to the retained information blocks to maintain the mathematical expectation of the input data as unbiased, thus obtaining the input of the dimensionality-reduced Critic network.

2. The multi-agent cooperative navigation optimization method according to claim 1, characterized in that, In the introduced joint priority experience extraction mechanism, the priority is calculated as follows: ； Where, δ k Let ε be the temporal difference error of the kth empirical sample, ε be the smoothing constant, and α be the priority intensity coefficient. The sampling probability is: ； Importance sampling weights are: ； After normalization, we get: ； ： ； Where M is the experience pool capacity and β is the importance sampling correction parameter.

3. The multi-agent cooperative navigation optimization method according to claim 1, characterized in that, The joint priority experience extraction mechanism uses a prefix sum tree structure to achieve efficient sampling according to priority ratio, and the initial priority of newly stored samples in the experience replay pool is set to the highest priority in the current pool.

4. The multi-agent cooperative navigation optimization method according to claim 1, characterized in that, The cross-agent synchronization indexing mechanism is specifically as follows: In each sampling, all agents extract local experiences at the same time from their respective experience replay pools based on the same set of randomly generated indices, and concatenate these local experiences in the order of the agents to form a joint experience sample for centralized training.

5. The multi-agent cooperative navigation optimization method according to claim 1, characterized in that, The information discarding and correction compensation mechanism specifically includes: generating a binary mask that follows a Bernoulli distribution for each information block from other agents to perform block-level discarding; retaining information blocks with a probability of 1-p and multiplying the retained information blocks by a correction factor 1 / (1-p), where p is the preset information loss rate; and 0 < p < 1.

6. The multi-agent cooperative navigation optimization method according to claim 1, characterized in that, The training phase of the multi-agent cooperative navigation includes the following steps: S1, In the current training round, each agent selects actions and interacts with the environment through the Actor network based on its local observations. The joint experience samples generated by the interaction, including global state, joint actions, immediate rewards, global information of the next state, and termination flags, are stored in their respective experience replay pools, and new samples are assigned initial priorities. S2, when the number of samples in the experience replay pool reaches a set threshold, a batch of sample indices is extracted from the pool according to the joint priority experience extraction mechanism, and a batch of joint experience samples is formed by using a cross-agent synchronous indexing mechanism; S3, for each joint experience sample in the batch, the information blocks from other agents in the centralized Critic network input are processed according to the information discarding and correction compensation mechanism to obtain the dimensionality-reduced Critic network input; S4. Based on the input of the dimensionality-reduced Critic network, calculate the target Q value and Critic loss of each agent, and update the Critic network parameters by combining the importance sampling weights; then use the updated Critic network to calculate the policy gradient of each Actor network to update the target network parameters. S5, using the new temporal difference error calculated in S4, update the priority of the sampled samples in the current batch in the priority experience replay pool; S6, repeat S1 to S5 until the termination condition of round training is met.

7. The multi-agent cooperative navigation optimization method according to claim 1, characterized in that, The information block includes: the agent's own state information block, target information block, obstacle information block, and teammate message information block from other agents; wherein, the information discarding takes priority on the teammate message information block.

8. The multi-agent cooperative navigation optimization method according to claim 7, characterized in that, Different information loss rates are applied to information blocks with different semantic meanings.

Images