A MADDPG multi-agent motion control method based on hybrid three types of experience

Abstract

This invention proposes a hybrid three-class experience (MADDPG) multi-agent motion control method, belonging to the field of multi-agent deep reinforcement learning. The MADDPG algorithm introduces both poor and good experiences on top of expert experience; poor experiences are generated by a negative example module, while good experiences are generated by a post-experience replay module. By fusing expert, poor, and good experiences to construct a hybrid three-class experience mechanism, the training efficiency and stability of the MADDPG algorithm are improved; the diversity and quality of training samples are enhanced, strengthening the stability and decision-making ability of the multi-agent system in complex dynamic environments; and by integrating the negative example module to generate poor experiences, efficient and reliable motion control can be performed in uncertain environments, significantly improving the adaptability and robustness of the multi-agent system.

Description

Technical Field

[0001] This invention relates to the field of multi-agent deep reinforcement learning, and specifically to a MADDPG multi-agent motion control method. Background Technology

[0002] With the continuous advancement of technology, cooperative motion control technology for multi-agent systems has been widely applied in numerous fields, especially in emergency rescue and military operations. Intelligent agents and vehicles have successfully executed complex tasks through efficient path planning and collaborative strategies. In complex and ever-changing battlefield environments, autonomous navigation technology for multi-agent systems has become a research hotspot. The core objective of this research is to plan a path for each agent that avoids collisions and can be completed within the task time constraint. However, traditional multi-agent motion planning methods often rely on completely known and relatively fixed environmental conditions, employing search algorithms such as the A* algorithm, Artificial Potential Field (APF), and Vector Field Histogram Plus (VFH+) to calculate paths. While these methods perform well in static and stable environments, they lack adaptability and flexibility when the environment changes dynamically, making it difficult to cope with the rapid changes and uncertainties in the battlefield environment.

[0003] Multi-agent reinforcement learning involves multiple agents interacting and coordinating their behaviors in the same environment to learn optimal policies. For example, the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm uses an Actor-Critic framework to solve motion planning problems in multi-agent systems. The core idea of ​​MADDPG is centralized training followed by distributed execution. Each agent independently generates actions based on its own state information, and the value of the actions is estimated using the joint state information of all agents. Theoretically, this method can achieve efficient decision-making and good coordination. However, MADDPG also faces some challenges. First, in the early stages of training, agents rely on random exploration to learn the environment, which is not only inefficient but also results in slow neural network training speed and poor experience pool quality due to the need to avoid collisions simultaneously. Second, most samples are concentrated in specific states, lacking high-quality positive feedback and containing extremely poor-quality experiences, both of which affect the learning effect of the policy. Furthermore, the Actor-Critic network is updated after each iteration, leading to frequent changes in policy direction, making it difficult for agents to learn and adapt to the environment stably.

[0004] To address these issues, some researchers have proposed methods to optimize the MADDPG algorithm. For example, published patent CN113341958A mentions a method to optimize MADDPG by mixing experience pools, namely the ME-MADDPG algorithm. This method improves training efficiency and stability by introducing high-quality experience. In this patent, in addition to the agent's own exploration experience, expert experience generated by the artificial potential field method and post-processed improved experience are added to increase sample diversity and improve learning efficiency. However, although the method proposed in CN113341958A improves learning efficiency to some extent, it still has some limitations and problems. First, while the method in the patent adds high-quality experience, it does not adequately handle low-quality experience in the experience pool, which may lead to the agent being less sensitive to certain important negative situations during training. Second, the method may rely too heavily on the artificial potential field method when generating expert experience, which may not be flexible or adaptable enough in some highly dynamic and unstructured environments. In addition, the method may be too fixed in handling the mixing ratio of exploration experience and high-quality experience, lacking the ability to dynamically adjust to environmental changes, which may limit the performance of the algorithm in constantly changing environments.

[0005] This invention proposes a three-experience mixed multi-agent deep deterministic policy gravitant (3E-MADDPG) multi-agent motion control method. By improving the MADDPG algorithm, it solves the multi-agent motion control problem and enhances the stability and decision-making ability of multi-agent systems in complex dynamic environments. Summary of the Invention

[0006] To address the aforementioned problems, this invention proposes a MADDPG multi-agent motion control method that integrates three types of experience, belonging to the field of multi-agent deep reinforcement learning. The MADDPG algorithm introduces both poor and good experiences on top of expert experience; poor experiences are generated by a negative example module, while good experiences are generated by a Hindsight ExperienceReplay (HER) module. This invention improves the MADDPG algorithm by fusing these three types of experience and constructing a hybrid mechanism, thereby enhancing the training efficiency and stability of the MADDPG algorithm; increasing the diversity and quality of training samples; and strengthening the stability and decision-making ability of the multi-agent system in complex dynamic environments. By integrating the negative example module to generate poor experiences, it can perform efficient and reliable motion control in uncertain environments, significantly improving the adaptability and robustness of the multi-agent system.

[0007] likeFigure 1 As shown:

[0008] This invention provides a MADDPG multi-agent motion control method that combines three types of experience, comprising the following steps:

[0009] Step 1: Set up the environment model and motion model of the agent;

[0010] Step 2: Construct a multi-agent joint state model;

[0011] Step 3: Construct the Actor-Critic network for the MADDPG algorithm and initialize the parameters and experience pool of the Actor-Critic network;

[0012] Step 4: Determine how the agent chooses its action;

[0013] Step 5: Agent state update;

[0014] Step 6: Use the HER algorithm to generate excellent experiences and use the negative example module to generate poor experiences;

[0015] Step 7: Determine whether the multi-agent system has entered the training phase;

[0016] Step 8: Set up sampling strategy and experience playback.

[0017] like Figure 2 As shown:

[0018] Furthermore, in step 1, the environment model is set as follows:

[0019] In a multi-agent system, each agent is a circular body with radius r. i The external environment of the multi-agent system contains a series of dynamic obstacles, and the agents need to avoid obstacles with a radius of r during flight. obs The agent safely reaches the target location by navigating circular obstacles; the collision distance D between the agent and the obstacle. io =r i +r obs When the Euclidean distance d between the agent and the obstacle satisfies d ≤ D io When this occurs, it indicates that a collision has occurred between the agents.

[0020] When the obstacle is in motion, the obstacle motion model is consistent with the agent motion model.

[0021] The agent's motion model is defined as follows:

[0022] The agent motion model simulates the movement of an agent in a two-dimensional plane, flying at a constant altitude and coordinating turns using inertia. The agent acquires its own data through an inertial measurement unit, and the agent's motion state is updated at each time step t by acceleration. speed Location Heading angle and angular velocity Decide:

[0023] angular velocity The calculation is as follows:

[0024]

[0025] in, Let be the heading angle of the agent at time t. The rate of change of ω, i.e., angular velocity; for Velocity in the x-direction; for Velocity in the y-direction; heading angle The range is [0, 2π].

[0026] The intelligent agent obtains acceleration through an inertial measurement device. and speed

[0027] Obtain the current location based on navigation data.

[0028] acceleration The net force acting on the intelligent agent With agent mass m u Decision, acceleration Represented as:

[0029]

[0030] and They represent Acceleration in the x and y directions; and They represent The resultant force acting on the object in the x and y directions;

[0031] The agent's velocity at time t The velocity at the previous time t-1 and acceleration Decision; similarly, Location information It is also determined by the position information from the previous moment and the current speed, speed and information The formula is as follows:

[0032]

[0033] and They represent The position in the x and y directions; δt represents the time interval from the previous time t-1 to the current time t;

[0034] To ensure the physicality of the agent system, a maximum acceleration 'a' of the agent was set. max and maximum speed v max To prevent the agent's motion parameters from exceeding actual constraints;

[0035] Furthermore, in step 2, the process of constructing the multi-agent joint state model is as follows:

[0036] Step 2.1: Use a Markov decision model for multi-agent path planning:

[0037] Markov decision models use quintuples<N,S,A,C,R> Let S be the set of n agents, where N = {1, 2, ..., n}, representing the set of n agents; n represents the number of agents; and S = s1 × s2 × s3. i ×…×s n S represents the joint state space, s i Let S be the state of agent i. S is composed of the states of all agents, because an agent's current state is not only related to its previous state but also closely linked to the states of other agents; A = a1 × a2 × A i ×…×A n A represents the joint action space, A i Let be the action of agent i; each agent chooses an action based on its current state and information about cooperation or competition from other agents; C is the state transition model, which is: S×A×S→[0,1], representing the probability that multiple agents will transition to the next joint state after taking a joint action in the current joint state; R=r1×r2×r i ×…×r n R is the joint reward function, r i This represents the reward value obtained by agent i under a specific state-action pair;

[0038] In Markov decision models, the agent's optimal policy J(θ) i J(θ) is the maximization of the joint cumulative reward of a multi-agent system. i Achieve this through expected returns:

[0039]

[0040] Where, θ i G is the policy parameter of agent i; i γ is the expected cumulative reward of agent i under the current policy; t It is the discount factor at time t, which represents the agent's emphasis on future rewards; The instantaneous reward for agent i at time t;

[0041] Step 2.2, Construction of the joint state space of multiple agents:

[0042] Each agent's sensors can detect information in seven different directions to obtain the detection distance value; when other agents or obstacles are detected, the sensors return the straight-line distance between the agent and the detected object; if no object is detected, the sensors return the default value of the maximum detection distance; through the sensors, the agent can obtain multi-dimensional information about itself, the environment, and the target point, thereby making decisions in complex task scenarios;

[0043] Agent state information S i Including its own state information s u Sensor detection information s r and environmental information e That is, S i =(s u ,s r ,s e );

[0044] s u =[v i ,ψ i Equation (6)

[0045]

[0046] Among them, the self-state information s u Including the agent's current velocity v i and heading angle ψ i Sensor detection information s r Composed of the detection information returned by the sensors, and These represent the distance values ​​detected by the sensor in seven directions; target position P. ig =(P igx ,P igy ), P igx and P igy Representing P respectively ig Position in the x and y directions; relative position of agent i to the target. The relative distance between agent i and other agents Represents the difference in heading angle between agent i and the target; s represents the difference in heading angle between agent i and other agents. e The range is [0,π].

[0047] State information S of the agent i Perform standardization, mapping to the range [0,1]; S i The standardization process is as follows:

[0048] First, its own state information s u Standardization:

[0049]

[0050] Among them, v i ψ represents the velocity of agent i. i π represents the heading angle of agent i; π represents the set of strategies chosen by all agents throughout the entire motion.

[0051] Secondly, the sensor detects information s r Standardization:

[0052]

[0053] Where, d R This represents the maximum detection range of the sensor of agent i;

[0054] Finally, environmental information s e Standardization:

[0055]

[0056] Among them, L max The boundary length of the environment;

[0057] The final agent state space S i It is a 15-dimensional vector that contains all the information related to multi-agent collaboration and competition tasks, ensuring that agents can effectively learn and adapt to changes in different task scenarios;

[0058] Step 2.3, Construction of the multi-agent joint action space:

[0059] By controlling the acceleration and angular velocity of the intelligent agent, the motion attitude and flight speed of the intelligent agent can be controlled; the intelligent agent's action space A i for:

[0060] A i =[a i ,ω i Equation (12)

[0061] Among them, a i ω is the acceleration of agent i. i Let be the angular velocity of agent i, and a i ∈[-a max ,a max ],ω i ∈[-ω max ,ω max ], a max ω represents the maximum acceleration of agent i. max This represents the maximum angular velocity of agent i;

[0062] To ensure that the agent moves within physical constraints, the velocity v of agent i is... i and heading angle ψ i All must satisfy v i ∈[0,v max ], ψ i ∈[0,ψ max The constraint of v max ψ represents the maximum speed of the agent. max The maximum heading angle of the intelligent agent;

[0063] Action Space A i This describes the ability of an agent to adjust its motion at any given time. Due to the complex cooperative and competitive relationships between agents, the joint action space A = A1 × A2 × A i ×…×A n Motion control of the entire intelligent agent system is crucial;

[0064] Step 2.4, Construction of the multi-agent objective function:

[0065] The goal of path planning is to ensure that agents reach the target location in the shortest time and along the shortest path without collisions, thereby achieving the optimal motion strategy π for all agents to reach the target. * :

[0066]

[0067] P i =P ig Equation (14)

[0068]

[0069] Among them, P i P represents the position of agent i; ig Indicates the target location; T i L represents the time it takes for agent i to reach the target location; i The path length from agent i to the target location represents the total path length; the optimal motion strategy π represents the optimal path length from agent i to the target location.* That is, the strategy that minimizes the total time and path length for all agents to reach the goal; D io This refers to the collision distance. This indicates that at any given time, agent i cannot collide with an obstacle or other agent; j represents an obstacle or other agent, i.e., an adjacent object. This represents the position of agent i at time t; This represents the position of j at time t;

[0070] Step 2.5, Construction of the multi-agent reward function:

[0071] Based on the framework of deep reinforcement learning, the reward function also sets a sparse reward mechanism to guide the agent to avoid obstacles and other agents; the reward function is divided into sparse reward and dense reward.

[0072] When the agent achieves its goal without a collision, that is... Give positive rewards r arrive :

[0073] r arrive =C arrive +σ t Equation (16)

[0074] Among them, C arrive σ is the reward constant for the agent to successfully reach the goal; t Time-dependent dense rewards are used to encourage agents to reach the goal quickly, σ t =-W(ΔT-t) min W is the penalty parameter, ΔT is the time consumed, and t is the time consumed. min =[||P0,P ig ||-d goal ] / v max , t min For the agent at maximum speed v max The shortest time consumed, where P0 is the agent's initial position, P ig For the target location, d goal This represents the distance threshold for the agent to reach the target.

[0075] When the agent collides with another entity, a negative reward r is given. collision :

[0076]

[0077] C collision =-C arrive Equation (18)

[0078] Among them, C collision This represents the penalty for an agent causing a collision; dcollision The distance threshold at which a collision may occur;

[0079] When the agent neither reaches the target nor causes a collision, four non-sparse reward functions are defined:

[0080]

[0081] r4=C t Equation (22)

[0082] r1 represents the distance difference between agent i and the target from time t-1 to t. If r1>0, it means that the agent is moving closer to the target and is given a positive reward; otherwise, it is punished. r2 represents the heading angle difference between agent i and the target from time t-1 to t. If r2>0, it means that the agent is adjusting its angle toward the target and is given a positive reward; otherwise, it is punished.

[0083] in, This represents the relative position of agent i and the target at time t; Let represent the difference in heading angle between agent i and the target at time t;

[0084] r3 represents the collision warning for the agent. When an obstacle is detected within the sensor's detection range, r3 = 0. When an obstacle is detected, a penalty is imposed, warning the agent to stay away from the obstacle.

[0085] C t This represents the time penalty at each moment, in order to encourage the agent to reach the target point as quickly as possible.

[0086] The reward function r of the agent during normal movement normal :

[0087]

[0088] in, This represents the normalization of the relevant values; μ1+μ2+μ3+μ4=μ=1, where μ represents the contribution rate to the reward function.

[0089] like Figure 3 As shown:

[0090] Furthermore, in step 3, the Actor-Critic network construction process is as follows:

[0091] In the MADDPG algorithm, each agent uses an independent Actor policy network and a Critic evaluation network. The Actor network generates the optimal policy for the agent, and the Critic network evaluates the value function of each agent's actions. The Actor network optimizes the policy based on the feedback provided by the Critic network through the policy gradient method to maximize the long-term cumulative reward. When there are n agents in the agent system, there are n pairs of independent Actor-Critic networks.

[0092] Step 3.1, Construct the Actor policy network:

[0093] The Actor network is based on the agent's state s i Output action a i The Actor network is updated using the policy gradient method, aiming to continuously optimize the policy parameters θ. i This allows the agent to be in state s i The following action a is generated i To achieve maximized long-term returns; the gradient of the expected cumulative return of agent i.

[0094]

[0095] in, The Q-value for evaluating the Critic network represents the sum of agent i's actions A1, ..., A1 in the joint state s and all agent actions A1, ..., A2. n The expected long-term return; μ represents the gradient of the Q-value with respect to the action of agent i. i (A i |s i ) represents the policy function μ i Input state s i Output action A i ; Represents the policy function μ i For θ i The gradient of θ; i The policy parameters for agent i are used to generate the policy function μ. i ;

[0096] In the joint state, the multi-agent system learns from the experience pool and continuously optimizes its strategy using the Actor network to improve the coordination ability and task completion efficiency of the entire system.

[0097] Step 3.2, Construct the Critic evaluation network:

[0098] In the MADDPG algorithm, the Critic network Q i (s,A|φi For each agent's action A i Perform an evaluation and output the agent's state s. i The Critic network estimates the long-term reward under the following conditions: it combines the joint state S and joint action A of all agents and calculates the Q-value based on this, which is to evaluate the long-term reward of the current action in the global state; the update of the Critic network is based on the temporal difference error. By minimizing this error, the Critic network can gradually learn the long-term reward of the agent performing different actions in each state.

[0099] Loss function of Critic network The mean squared error between the current predicted Q value and the target Q value is expressed by the following formula:

[0100]

[0101] Among them, y i This represents the target Q value, which combines the immediate reward with the future discounted reward; γ represents the discount factor. It is the immediate reward that agent i receives at time t; This represents the target network's state s' and actions A'1, A' at the next time step. i …,A' n Q-value estimation; A' i =μ′ i (s i ) represents the action of agent i at the next moment, determined by its policy function μ′. i (s i )Decide;

[0102] Furthermore, the parameters and the experience pool initialization method are as follows:

[0103] Step 3-1, set the training steps and iteration cycle:

[0104] The total number of training steps is set to M = 15000. During training, the simulated step size is set to Δt = 0.2s, and the maximum movement time of each agent in a single training round is t. max =300×Δt=60s, to ensure that the agent has sufficient time to explore the environment;

[0105] Step 3-2, set the sampling probability and related hyperparameters:

[0106] Set the sampling quantity η = 64 and the Actor-Critic network update frequency ξ = 100; to ensure that the agent learns from different types of experience, the sampling probability p0 = 0.1 needs to be set, and the sampling probability p0 controls the proportion of sampling from the excellent experience pool D' and the poor experience pool D”; generate a random number p, p∈[0,1]; if p≤p0, then select the next action to be executed according to the artificial potential field method; if p>p0, then select the action according to the policy network;

[0107] Both the Actor network and the Critic network are fully connected neural networks, consisting of an input layer, two hidden layers, and an output layer. The Actor network uses ReLU as the activation function for its hidden layers. The first hidden layer of the Actor network contains 64 nodes, and the second hidden layer contains 32 nodes. The output layer of the Actor network has one node, representing the action selection without human intervention, and uses the tanh activation function. The learning rate of the Actor network is 0.001.

[0108] The Critic network has 64 nodes in each of its two hidden layers and 1 node in its output layer, outputting the Q-value. It uses the linear activation function y = x + b, where b is the bias parameter, and the learning rate of the Critic network is 0.0001.

[0109] Both the Actor network and the Critic network use the Adam optimizer and set a discount factor γ = 0.97 to balance the impact of current rewards and future rewards.

[0110] Step 3-3, Initialize the noise model:

[0111] To enhance the exploration capabilities of the agents, exploration noise needs to be added to each agent. The Ornstein-Uhlenbeck (OU) noise model is used to generate random perturbations for each agent's actions, thereby improving exploration capabilities. OU noise dx t The expression is as follows:

[0112] dx t =θ(μ-x) t Equation (26) is: dt + ρdW

[0113] Where, x t θ represents the current value of the noise OU; μ represents the noise mean regression rate; ρ represents the noise long-term mean; dW represents the noise fluctuation intensity; and dW represents the derivative of the noise Brownian motion.

[0114] Steps 3-4: Initialize the experience pool:

[0115] In the 3E-MADDPG algorithm, the agent's learning relies on three experience pools to store different types of experience. The regular experience pool D stores the regular experience collected by the agent when exploring the environment; the excellent experience pool stores the high-quality experience generated by the HER algorithm and the artificial potential field method; and the poor experience pool stores the failed experience during the training process, as well as the poor experience generated by the counterexample model.

[0116] For the regular experience pool D: it stores the regular experience collected by the agent while exploring the environment; each sample is a quintuple (s t A t ,r t ,s t+1 ,g),s t A t r t s t+1 Let g and g represent the agent's current state, the action generated by the agent in the current state according to the policy network, and the action A executed by the agent, respectively. t Immediate rewards obtained from the environment, and the agent's execution of action A t The next state and task objective;

[0117] For the excellent experience pool D': it stores excellent experiences generated by the HER algorithm and the artificial potential field method; these excellent experiences are used to handle sparse reward problems and goal achievement problems in complex environments; and the policy parameters θ of the Actor network are initialized. i At that time, set the excellent experience pool D' generated by HER;

[0118] For the poor experience pool D”: it stores the failure experiences encountered by the agent during training, including collisions and deviations from the path; the negative example model uses these poor experiences to help the agent recognize and avoid repeating the same mistakes; it initializes the policy parameters θ of the Actor network. i At that time, set the excellent experience pool D generated by HER;

[0119] The capacity of the regular experience replay buffer is D = 1,000,000. Since D' and D” are auxiliary experience pools, their capacities are both set to 50,000. Training begins after the experience pool capacity reaches 5,000.

[0120] Furthermore, in step 4, the process for determining the action mode selected by the intelligent agent is as follows:

[0121] The action mode selected by the agent is determined based on the results of step 3;

[0122] If p≤p0, proceed to step 4.1 and select the next action to be performed according to the artificial potential field method;

[0123] If p > p0, then proceed to step 4.2 to select an action based on the policy network;

[0124] Step 4.1, the agent generates a decision action based on the artificial potential field method: it is necessary to obtain the position P of agent i at time t. i The gravitational force and repulsive force Let P i =[P ix ,P iy ] represents the position of agent i, P ig =[P igx ,P igy If ] is the target position, then the normalized gravitational component acting on the agent is expressed as:

[0125]

[0126] and Represent Gravitational forces in the x and y directions;

[0127] The normalized repulsive force component of the agent i subjected to the neighboring object j is expressed as follows:

[0128]

[0129] The position of the adjacent object j is P j =[P jx ,P jy ]; P jx and P jy Representing P respectively j Position in the x and y directions; and These represent the repulsive forces of adjacent object j in the x and y directions, respectively.

[0130] The resultant force acting on agent i is obtained by combining the attractive and repulsive forces.

[0131]

[0132] In the formula, represents the set of neighboring objects of agent i; σ ij Let σ be the collision function, representing the degree of influence of each member of the neighboring objects on the repulsive force exerted on agent i. ij The expression is as follows:

[0133]

[0134] d ij d is the Euclidean distance between agent i and its neighbor j; md is the minimum collision distance between agent i and its neighboring object j; c d represents the sensor's detection distance. r Let be a constant, and let its value range be (d). m ,d c The calculation formulas and expressions for parameters a, b, c, and d are as follows:

[0135]

[0136] The linear velocity v of agent i under the action of the net force i and angular velocity ω i The calculation is as follows:

[0137]

[0138] Where, k u The speed control gain is set to the maximum speed v. max ;||P i -P ig || represents the Euclidean distance between the position of agent i and the target position; k ω The angular velocity control constant is k. ω =2.5; ψ i This represents the heading angle value of agent i. Represents resultant force The direction angle, with a value of express The derivative with respect to time, The expression is as follows:

[0139]

[0140] Step 4.2, generating decision actions for the agent based on the Actor network: The Actor network receives the state input of the agent, extracts features from the state through a multi-layer neural network, and finally outputs the action that the agent should perform; the Actor network enhances its exploration ability by introducing random noise to ensure that the agent can effectively explore in unknown environments; as training progresses, the Actor network continuously optimizes the policy parameters;

[0141] The input to the Actor network is the agent's current state s. i The input layer extracts features by passing the original state information to the hidden layers. Each hidden layer weights the input and handles non-linear relationships using the ReLU activation function. The Actor network extracts key features from the state layer by layer, ultimately generating the agent's action A. i :

[0142]

[0143] in, These are the trainable parameters of the Actor network; at each time step Δt, the agent will perform an action a based on the output of the Actor network. i It interacts with the environment; based on the feedback from the environment, it updates the parameters of the Actor network. To optimize strategies and improve the efficiency of agent actions.

[0144] Furthermore, in step 5, the state of the agent is updated as follows:

[0145] The agent determines the current state s. i Action A generated by the policy network i When interacting with the environment, the environment will respond with a new state s. t+1 and reward r t ; state s t+1 The update process ensures that the agent can adjust its behavioral strategies based on feedback received from the environment and provides data support for subsequent experience replay.

[0146] Furthermore, in step 6, the HER algorithm is used to generate excellent experiences as follows:

[0147] When the agent fails to achieve the goal, the actual state reached is redefined as the new goal, so that the experience trajectory that was originally considered a failure can still provide experience for policy learning. In each training cycle, the agent starts from the initial state s1 and executes policy π, sampling trajectories according to the current policy. The strategy sampling trajectory at time t is then used as (s t A t ,r t ,s t+1 The state s in the format of g is stored in the experience pool D; when the agent fails to complete the objective g, the HER algorithm randomly selects a state s from the current trajectory. t 'As an alternative objective g', and recalculate the reward r based on g'. t ', forming a new sampling trajectory {s t ',A t ,r t ',s t+1 ,g'}; Store the new sampling trajectory in the excellent experience pool D'.

[0148] The negative example module generates poor-quality experiences in two ways:

[0149] One approach is to directly extract poor-quality experience from the agent's collision or failure behavior: once a collision or failure behavior is detected, the final state of the trajectory is immediately recorded. As inferior experience, it is stored in the inferior experience pool D”;

[0150] Another way is to generate inferior experiences through reverse operations on excellent historical experiences; during the trajectory sampling process, monitor whether the behavior of the intelligent agent shows collision or failure behavior: randomly sample some high-quality experiences at time t from the excellent experience pool D' (s t , A t , r t , s t+1 , g), and generate inferior experiences by reversely adjusting s t and A t ; calculate the reversely generated s new+1 and r new , and store the generated inferior experiences (s new , A new , r new , s new+1 ) into the inferior experience pool D”; the counterexample module provides more diverse training data for the policy learning of the intelligent agent, further improving the robustness and adaptability of the algorithm.

[0151] Furthermore, in step 7, the judgment of whether the multi-agent enters the training stage is as follows:

[0152] The intelligent agent needs to store experiences first when exploring the environment for subsequent training and updating; the number of experiences in the regular experience pool D, the excellent experience pool D', and the inferior experience pool D” are |D|, |D'|, and |D”| respectively; add |D|, |D'|, and |D”| to obtain the total number of experiences stored in the three experience pools; compare the total number of experiences with the sampling number η. If |D| + |D'| + |D| ≥ Q, the condition for entering the training stage is met, and go to step 8; if |D| + |D'| + |D| < Q, go to step 4 to continue generating the number of experiences.

[0153] Furthermore, in step 8, the setting process of the sampling strategy and experience replay is as follows:

[0154] Step 8.1, perform experience sampling:

[0155] The intelligent agent will sample experiences from the regular experience pool D, the excellent experience pool D', and the inferior experience pool D” to update the parameters of the Actor-Critic network; set the sampling probability as p0 = 1 / 8, and the number of samples sampled from D, D', and D” are n0, n1, and n2 respectively:

[0156]

[0157] Randomly sample from the three experience pools and store the sampled experience data in the sample set for calculating the target Q value;

[0158] Step 8.2, calculate the target Q value:

[0159] Based on the empirical samples The objective Q-value is calculated using a target Critic network, and the Q-value is combined with the current immediate reward r. i And the expectation of future rewards help the agent assess the long-term reward of performing a specific action in the current state;

[0160]

[0161] Step 8.3, update the Q-value estimate of the Critic network agent:

[0162] The Critic network calculates the error between the currently estimated Q-value and the target Q-value to adjust its parameters. By minimizing this error, the Critic network learns a more accurate Q-value, providing a precise reference for the Actor network's policy updates. The Critic network's update method is shown in the following equation:

[0163]

[0164] loss function The smaller the value, the more accurate the Critic network's estimate of the Q-value; the Critic network's updates can learn a more accurate Q-value.

[0165] Step 8.4, Update the Actor network:

[0166] The goal of the Actor network is to find the action policy that maximizes the Q-value of the Critic network; the update formula for the Actor network is as follows:

[0167]

[0168] The Actor network minimizes the policy gradient loss. This allows the generated actions to maximize the Q-value estimated by the Critic network.

[0169] Step 8.5, determine whether to update the target network:

[0170] The target network parameters are updated only at fixed time intervals ξ, which can reduce oscillations during the policy update process and make the estimation of the target Q value more stable. When Δt%ξ = 0 is satisfied, proceed to step 8.7; when Δt%ξ = 0 is not satisfied, continue to use the current target network parameters and do not update the Actor-Critic network parameters.

[0171] Step 8.6: Update the target network using a soft update method to maintain the stability of the target network.

[0172] Target network parameters and The update method is as follows:

[0173]

[0174] in, Formula for Critic network parameters; τ represents the parameters of the Actor network; τ is the update coefficient that gradually increases from 0 to 1, causing the Actor and Critic network parameters to update slowly, thus improving the stability of neural network training.

[0175] Step 8.7, iteration terminates:

[0176] The training of the neural network continues until the agent's Actor and Critic networks have reached the expected performance standards using the training data. When training terminates, the agent system saves the parameters of the Actor and Critic networks at the time of termination and uses these parameters in practical applications.

[0177] The beneficial effects of this invention are as follows:

[0178] (1) This invention uses the HER algorithm to take the actual state reached by the agent during playback as the new target, so that the experience trajectory that was originally considered a failure can still provide useful information for policy learning, and the experience that was originally considered a failure is stored in the excellent experience pool to form an excellent experience pool; excellent experience usually contains the optimal decision made by the agent in a specific environment, which helps the agent to quickly learn and imitate these successful strategies.

[0179] (2) This invention extracts negative experience directly from the collision or failure behavior of the agent, and at the same time reverses the historical high-quality experience to generate inferior experience, forming an inferior experience pool; the inferior experience pool samples help to enhance the learning performance of the agent in complex environments, enabling it to better avoid failure behavior and optimize the decision-making process.

[0180] (3) The MADDPG algorithm with three types of experience involved in this invention divides the experience pool into three types: regular experience pool, excellent experience pool and poor experience pool, which effectively improves the utilization rate and diversity of experience samples and avoids the problem of too high proportion of meaningless experience in the traditional MADDPG algorithm.

[0181] (4) The present invention determines the proportion of sampling from different experience pools based on the set sampling probability; samples are sampled from both the excellent experience pool and the poor experience pool, and the remaining samples are sampled from the regular experience pool; by using a sampling strategy that mixes three types of experience, the agent can be ensured to learn from multiple types of experience, so that it can acquire policy information from different experience pools in a balanced manner during the learning process. Attached Figure Description

[0182] Figure 1 A flowchart for a multi-agent motion control method that combines three types of experience;

[0183] Figure 2 This is a schematic diagram of multi-agent motion control in two-dimensional space.

[0184] Figure 3 A framework diagram of the MADDPG algorithm that combines three types of experience;

[0185] Figure 4 The graph shows the arrival rate test data when the number of agents is 5, 6, 7, and 8. Detailed Implementation

[0186] like Figure 2 As shown, agent i is deployed at the initial position P. i (Indicated by the blue circle), its target location is P. ig (Represented by blue-filled circles). During movement, the agent must move within a defined space and traverse various obstacle areas (represented by black circles) without colliding or conflicting with other moving obstacles (such as other agents j and k). The obstacles move irregularly, and each agent can perceive potential obstacle threats ahead through onboard light detection and ranging or other sensors.

[0187] Figure 2 The scene layout of the multi-agent motion planning model is displayed, where circles with colored borders represent radii of r. i Different agents are represented by arrows indicating their direction of movement, solid-color filled circles representing their destinations, and solid-color lines connecting two circles representing their trajectories. Circles with black borders indicate a diameter of r. obs A moving obstacle is represented by a dashed line starting from it, which is considered its random trajectory. The collision distance between the agent and the obstacle is defined as D. io =r i +r obs By continuously sensing, avoiding obstacles, and planning paths in the environment, the intelligent agent can eventually reach the target location safely, thus completing autonomous movement tasks in complex environments.

[0188] In actual training, the 3E-MADDPG and MADDPG algorithms were tested for 5 to 8 agents and different numbers of obstacles. Each experimental scenario underwent 1000 rounds of testing, with the positions of agents, target, and obstacles randomly set in each round. A test round ended under any of the following conditions: all agents reached the target, all agents collided, or the round time limit was reached. The experimental metric was the average arrival rate of the agents; specific data are shown in Table 1, and relevant curves are shown in... Figure 4 As shown.

[0189] Table 1 shows the test data for agents 5, 6, 7, and 8.

[0190]

[0191] Analysis of the charts shows that in a five-agent environment, when the number of obstacles is 5, the improved MADDPG achieves a reach rate exceeding 95%, reaching 97.69%, while the original MADDPG algorithm only achieves 86.80%. When the number of obstacles is 10, the reach rate of MADDPG drops to 71.94%, indicating that the original algorithm struggles to effectively control agents in complex environments. However, the improved MADDPG algorithm still maintains a 79.55% reach rate. With increasing obstacles, the performance degradation of the improved MADDPG is relatively slower, while that of the original MADDPG is faster, especially when the number of obstacles exceeds 8, where the performance of MADDPG drops sharply. This demonstrates that the improved MADDPG is more adaptable to environmental complexity.

[0192] Overall, the improved MADDPG algorithm achieved an average arrival rate of 88.61% across all obstacle counts, approximately 9.93% higher than the average arrival rate of the standard MADDPG algorithm (78.67%). Especially when dealing with a large number of obstacles, such as 10, the improved MADDPG algorithm maintained an arrival rate of around 79.55%, demonstrating good robustness.

[0193] For tests with different numbers of agents, when there were 10 obstacles in the environment, the improved MADDPG with 6 agents maintained an 80.53% arrival rate, while MADDPG only achieved 73.05%. When the number of agents increased to 7, the improved MADDPG achieved a 78.60% arrival rate, maintaining around 80%, while MADDPG's arrival rate had fallen below 70%. When the number of agents increased further to 9, the improved MADDPG achieved a 73.71% arrival rate, still higher than MADDPG. As the number of agents increased from 5 to 8, the arrival rates of both algorithms decreased, but the improved MADDPG's decrease was more gradual, maintaining a higher arrival rate. Even with 8 agents, the improved MADDPG still maintained an arrival rate above 73.71%. This indicates that the improved MADDPG has stronger scalability in multi-agent collaborative tasks. With an increase in the number of agents, the system complexity increases significantly, including factors such as communication between agents and path coordination. The improved MADDPG is better able to handle multi-agent collaborative tasks, while MADDPG is less capable of handling multi-agent tasks.

[0194] In summary, the improved MADDPG algorithm demonstrated stronger adaptability and robustness in test environments with multiple agents and varying numbers of obstacles. Regardless of whether the scenarios had few or many obstacles, the improved MADDPG consistently achieved a significantly higher arrival rate than the standard MADDPG, especially in highly complex environments such as those with 10 obstacles, where the improved MADDPG algorithm showed a clear advantage. This indicates that the improved MADDPG algorithm can better plan paths in complex and variable environments, effectively improving the agent's task completion capabilities.

Claims

1. A mixed three-class experience MADDPG multi-agent motion control method, characterized in that, The technical scheme of the control method comprises the following steps: Step 1: setting an environment model of an intelligent agent and an intelligent agent motion model; The environment model is set as follows: Each agent in the multi-agent system is a circle with radius r i ; the environment of the multi-agent system contains a series of dynamic obstacles, and the agent needs to avoid the circular obstacle with radius r obs to safely reach the target position during flight; the collision distance D io between the agent and the obstacle is r i + r obs ; when the Euclidean distance d between the agent and the obstacle satisfies d ≤ D io , it means that the agent has collided with the obstacle; When the obstacle is in a motion state, the obstacle motion model is consistent with the intelligent agent motion model; Step 2: constructing a multi-agent joint state model; Step 3: constructing an Actor-Critic network of the MADDPG algorithm, and initializing parameters and an experience pool of the Actor-Critic network; Step 4: determining a manner of selecting an action of the intelligent agent; Step 5: intelligent agent state updating; Step 6: generating excellent experience using the HER algorithm, and generating poor experience using an anti-example module; The excellent experience generated using the HER algorithm is as follows: When the agent fails to achieve the goal, the actually reached state is redefined as a new goal, so that the experience trajectory that was originally considered a failure still provides experience for policy learning; in each training cycle, the agent starts from the initial state s1 and executes the policy Π, and samples a trajectory according to the current policy and stores the policy sampled trajectory at time t in the format of (s t , A t , r t , s t+1 , g) in the experience pool D; when the agent fails to achieve the goal g, the HER algorithm randomly selects a state s t ' as an alternative goal g' from the current trajectory, and recalculates the reward r t ' according to g', forming a new sampled trajectory {s t ', A t , r t ', s t+1 , g'}; the new sampled trajectory is stored in the excellent experience pool D'. The anti-example module generates poor experience in two ways: One way is to directly extract bad experiences from the collision or failure behavior of the agent: as soon as a collision or failure behavior is detected, the final state of the trajectory is stored as a bad experience into the bad experience pool D". Another way is to generate poor experience for the reverse operation of historical excellent experience; in the process of trajectory sampling, monitor whether the behavior of the agent appears collision or failure behavior: randomly sample some excellent experience (s t , A t , r t , s t+1 , g) of t time from the excellent experience pool D'; generate poor experience by adjusting s t and A t of the trajectory in reverse; calculate s new+1 and r new generated in reverse, and store the generated poor experience (s new , A new , r new , s new+1 ) in the poor experience pool D''; the counterexample module provides more diverse training data for the policy learning of the agent, and further improves the robustness and adaptability of the algorithm; Step 7: determining whether the multi-agent enters a training stage; Step 8: setting a sampling strategy and experience replay.

2. The multi-agent motion control method of claim 1, wherein, In step 1, The intelligent agent motion model is set as follows: The agent motion model simulates the agent moving on a two-dimensional plane, flying at a constant height, and coordinating turns with inertia; the agent obtains its own data through an inertial measurement device, and the update of the motion state of the agent at each time step t is determined by the acceleration velocity position heading angle and angular velocity ​ Angular velocity is calculated as follows: wherein, is the heading angle of the agent at time t is the rate of change of the heading angle, i.e. the angular velocity; is the velocity in the x-direction; is the velocity in the x-direction; is the velocity in the y-direction; is the velocity in the y-direction; the heading angle ranges from [0, 2π]; The agent obtains acceleration through an inertial measurement device and velocity Acquiring a current position from navigation data acceleration resulting force on the agent with the mass m of the agent u determines the acceleration is expressed as: and respectively represent acceleration in the x direction and the y direction; and respectively represent resultant force in the x direction and the y direction; The speed of the agent at time t is determined by the speed at the previous time t-1 and the acceleration ; similarly, the position information at time t is determined by the position information at the previous time and the current speed and information is given by the following formula: and respectively represent a position in the x direction and the y direction; δt represents a time interval from a previous time t-1 to a current time t; To ensure the physicality of the agent system, the maximum acceleration a max and maximum speed v max of the agent are set to avoid the motion parameters of the agent exceeding the actual constraints.

3. The multi-agent motion control method of claim 1, wherein, In step 2, the multi-agent joint state model is constructed as follows: Step 2.1: using a Markov decision model to plan a path of the multi-agent: The Markov decision model is described by a five-tuple <N, A, S, C, R>, where N = {1, 2, …, n} represents a set of n agents; n represents the number of agents; S = s1x s2x s i ×…×s n , S represents a joint state space, s i is the state of agent i, S is composed of the combination of states of all agents, because the current state of an agent is not only related to the previous state of the agent itself, but also closely related to the states of other agents; A = A1x A2x A i ×…×A n , A represents a joint action space, A i is the action of agent i; each agent selects an action according to the current state and cooperation or competition information from other agents; C is a state transition model, the state transition model is: SxAxS→[0, 1], which represents the probability of transitioning to the next joint state after the multi-agent takes the joint action under the current joint state; R = r1x r2x r i ×…×r n , R is a joint reward function, r i represents the reward value obtained by agent i under a specific state-action pair; In the Markov decision model, the optimal policy J(θ i ) of an agent is to maximize the joint cumulative reward of the multi-agent system, J(θ i ) is achieved through the expected return: wherein θ i is a policy parameter of the agent i; G i is the expected cumulative return of the agent i under the current policy; γ t is a discount factor at time t, which is the degree of importance of the agent to future rewards; is the immediate reward of the agent i at time t; Step 2.2: constructing a multi-agent joint state space: The sensor of each intelligent agent can detect detection information in seven different directions to obtain a detection distance value; when other intelligent agents or obstacles are detected, the sensor returns a straight-line distance between the intelligent agent and the detected object; if no object is detected, the sensor returns a default value of the maximum detection distance; through the sensor, the intelligent agent can obtain multi-dimensional information about itself, the environment and the target point, thereby making decisions in a complex task scenario; Agent state information S i including self state information s u , sensor detection information s r and environment information s e ; that is, S i = (s u , s r , s e ) s u = [v i , ψ i ] Equation (6); where s is the self-state information of the agent i u including the current speed v of the agent i i and the heading angle ψ of the agent i i ; the sensor detection information s r consists of the detection information returned by the sensors; represent the distance detection values of the 7 directions detected by the sensors respectively; the target position P ig = (P igx , P igy ), P igx and P igy represent the positions of P ig in the x direction and the y direction respectively; the relative position of the agent i to the target the relative distance of the agent i to other agents represent the heading angle difference of the agent i to the target; represent the heading angle difference of the agent i to other agents, the range of s e is [0, π] State information S of the agent i is standardized and mapped into the range of [0, 1]; S i The standardization process of S is as follows: First, the own state information s u Standardization: where v i represents the velocity of agent i; ψ i represents the heading angle of agent i; π represents the mathematical constant pi; Secondly, the sensor detection information s r standardization: where d R represents the maximum detection distance of the sensor of the agent i; Finally, the standardization of environmental information s e of the system wherein L max is the length of the boundary of the environment; The final agent state space S i is a 15-dimensional vector containing all the information related to multi-agent cooperation and competition tasks, ensuring that the agent can effectively learn and adapt to changes in different task scenarios; Step 2.3: constructing a multi-agent joint action space: By controlling the acceleration and angular velocity of the agent, the motion posture and flight speed of the agent are controlled; the agent action space A i is: A i = [a i , ω i ] Equation (12); where a i is the acceleration of agent i; ω i is the angular velocity of agent i, and a i ∈ [-a max , a max ], ω i ∈ [-ω max , ω max ], a max represents the maximum acceleration of agent i, and ω max represents the maximum angular velocity of agent i; To ensure the agent moves under physical constraints, the velocity v i and the heading angle ψ i of the agent i must satisfy the constraints v i ∈ [0, v max ] and ψ i ∈ [0, ψ max ], where v max represents the maximum velocity of the agent and ψ max represents the maximum heading angle of the agent. Action space A i The motion adjustment ability of the agent at any time is described, and since there is a complex cooperation and competition relationship between the agents, the joint action space A=A1×A2×A i ×…×A n is crucial to the motion control of the whole agent system; Step 2.4: constructing a multi-agent objective function: The goal of path planning is to ensure that the agent reaches the target position in the shortest time and path without collision, so as to achieve the optimal motion strategy Π of all agents to reach the target * : P i = P ig Equation (14); where P i represents the position of agent i; P ig represents the target position; T i represents the time for agent i to reach the target position; L i represents the total length of the path for agent i to reach the target position; optimal motion strategy Π * , i.e., the strategy with the minimum total value of the time for all agents to reach the target and the total length of the path; D io is the collision distance; represents that at any time, agent i cannot collide with obstacles or other agents; j represents obstacles or other agents, i.e., adjacent objects; represents the position of agent i at time t; represents the position of j at time t; Step 2.5: constructing a multi-agent reward function: Based on the framework of deep reinforcement learning, the reward function also sets a sparse reward mechanism to guide the intelligent agent to avoid obstacles and other intelligent agents; the reward function is divided into sparse rewards and dense rewards; When the agent achieves the goal without a collision, i.e., when a positive reward r is given arrive : r arrive = C arrive +σ t Equation (16); where C arrive is the reward constant for the agent successfully reaching the goal; σ t is the time-dependent dense reward used to encourage the agent to reach the goal quickly, σ t = -W(ΔT - t min ), W is the penalty parameter, ΔT is the consumed time, t min = [||P0, P ig || - d goal ] / v max , t min is the shortest time consumed by the agent at the maximum speed v max , P0 is the initial position of the agent, P ig is the target position, and d goal is the distance threshold for the agent to reach the goal. When the agent collides, give a negative reward r collisoin : where C collisoin represents the penalty of collision of the agent; d collisoin is the distance threshold of collision When the intelligent agent does not reach the target and no collision occurs, four non-sparse reward functions are set: r4 = C t Equation (22); r1 represents a distance difference of the intelligent agent i from time t-1 to t relative to the target, if r1>0 indicates that the intelligent agent approaches the target, a positive reward is given, otherwise a penalty is applied; r2 represents a heading angle difference of the intelligent agent i from time t-1 to t relative to the target, if r2>0 indicates that the intelligent agent adjusts the angle towards the target, a positive reward is given, otherwise a penalty is applied; wherein, represents the relative position of agent i and the target at time t; represents the difference in heading angle of agent i and the target at time t; r3 represents a collision warning of the intelligent agent, when an obstacle is detected within the sensor detection range, r3=0, when the obstacle is detected, a penalty is given to warn the intelligent agent to move away from the obstacle; C t represents the time penalty at each time, hoping that the agent can reach the target point as soon as possible; Reward function r for normal motion of the agent normal : wherein denotes the normalization of the relevant values; μ1+μ2+μ3+μ4=1.

4. The multi-agent motion control method of claim 1, wherein, In step 3, the Actor-Critic network is constructed as follows: In the MADDPG algorithm, an independent Actor policy network and Critic evaluation network are used on each agent; the Actor network generates the optimal policy for the agent, and the Critic network evaluates the value function of each agent's action; the Actor network optimizes the policy through the policy gradient method based on the feedback provided by the Critic network, maximizing the long-term cumulative return; when there are n agents in the agent system, there are n pairs of independent Actor-Critic networks; Step 3.1, build the Actor policy network: The Actor network generates an action a i from the state s i of the agent; the Critic network evaluates the action a i and provides a value v i for the state s i ; the Critic network is updated by minimizing the mean squared error between the predicted value v Where η is the number of samples; This represents agent i in the joint state s and all agent actions A1,…,A n The expected long-term return; μ represents the gradient of the Q-value with respect to the action of agent i. i (A i |s i ) represents the policy function μ i Input state s i Output action A i ; Represents the policy function μ i For θ i The gradient of θ; i The policy parameters for agent i are used to generate the policy function μ. i ; In the joint state, the multi-agent system learns from the experience in the experience pool, and uses the Actor network to continuously optimize its policy to improve the coordination ability and task completion efficiency of the entire system; Step 3.2, build the Critic evaluation network: In the MADDPG algorithm, the Critic network Q i (s,A|φ i ) evaluates the action A i of each agent, and outputs the long-term return estimation of the agent in the state s i ; the Critic network combines the joint state S and joint action A of all agents, and calculates the Q value based on this, that is, the long-term return of the current action in the global state; the update of the Critic network is based on the time difference error, and by minimizing this error, the Critic network can gradually learn the long-term return of the agent performing different actions in each state; Loss function of critic network denotes the mean squared error between the current Q value prediction and the target Q value, and is given by the following equation: wherein, are trainable parameters of the Actor network; y i represents the target Q value, combining the immediate reward and the future discounted reward; γ represents the discount factor; is the immediate reward obtained by the agent i at time t; represents the Q value estimate of the target network for the next time state s' and the next time action A'1, A' i ... n . A' i = μ' i (s i ) represents the action of the agent i at the next time, determined by the policy function μ' i (s i ).

5. The multi-agent motion control method of claim 1, wherein, In step 3, the parameters and experience pool initialization method are as follows: Step 3-1, set the number of training steps and iteration cycles: The total number of training steps M = 15000 is set. During the training process, the simulation step size is set as Δt = 0.2s, and the maximum movement time of each agent in a single training round is t max = 300 * Δt = 60s to ensure that the agent fully explores the environment within a reasonable time range; Step 3-2, set the sampling probability and related hyperparameters: Set the sampling number η = 64 and the Actor-Critic network update frequency ξ = 100; in order to ensure that the agent learns from different types of experience, the sampling probability p0 = 0.1 needs to be set, which controls the proportion of sampling from the excellent experience pool D' and the poor experience pool D"; Generate a random number p, p ∈ [0, 1]; if p ≤ p0, select the action to be executed next according to the artificial potential field method; if p > p0, select the action according to the policy network; Both the Actor network and the Critic network are fully connected neural networks, both including an input layer, two hidden layers, and an output layer; the activation function of the Actor network hidden layer is ReLU; the first hidden layer of the Actor network contains 64 nodes, and the second hidden layer contains 32 nodes; the output layer of the Actor network is 1 node, representing the action selection without human intervention, using the tanh function as the activation function, and the learning rate of the Actor network is 0.001; The Critic network has two hidden layers with 64 nodes each, and an output layer with 1 node that outputs the Q value, using a linear activation function y = x + b, where b is the bias parameter, and the learning rate of the Critic network is 0.0001; Use the Adam optimizer and set the discount factor γ = 0.97 to weigh the influence of current rewards and future rewards; Step 3-3, initialize the noise model: To increase the exploration ability of the agent, an exploration noise needs to be added to each agent; a random disturbance is generated when using the OU noise model for each agent to generate action, thereby enhancing the exploration ability; the OU noise dx t The expression is as follows: dx t = θ(Ξ - x t ) dt + ρ dW Equation (26); where x t represents the current value of the OU noise; represents the noise mean reversion rate; represents the long-term mean of the noise; represents the noise volatility intensity; and dW represents the differential of the noise Brownian motion. Step 3-4, initialize the experience pool: In the 3E-MADDPG algorithm, the learning of the agent relies on three experience pools to store different types of experience; the regular experience pool D stores the regular experience collected by the agent when exploring the environment; the excellent experience pool stores high-quality experience generated by the HER algorithm and the artificial potential field method; the poor experience pool stores failure experience during training, and the poor experience generated by the counterexample model; For regular experience pool D: store the regular experience collected by the agent when exploring the environment; each sample is a five-tuple (s t , A t , r t , s t+1 , g ) t , A t , r t , s t+1 , and g respectively represent the state of the agent at the current time, the action generated by the agent in the current state according to the policy network, the immediate reward obtained by the agent from the environment after executing the action A t , the next state of the agent after executing the action A t , and the task goal; For the excellent experience pool D': store the excellent experience generated by the HER algorithm and the artificial potential field method; the excellent experience is used to deal with the sparsely rewarded problem and the goal achievement problem in a complex environment; initialize the policy parameter θ of the Actor network i At this time, set the excellent experience pool D' generated by the HER For the poor experience pool D": store the failure experience encountered by the agent during training, the failure experience includes the collision and deviation of the agent; the counterexample model helps the agent to recognize and avoid repeating the same mistake through the poor experience; initialize the policy parameter θ of the actor network i , set the excellent experience pool D" generated by HER; The capacity of the regular experience replay buffer is D = 1000000, and since D' and D" are auxiliary experience pools, their capacities are both set to 50000, and training begins after the experience pool capacity reaches 5000.

6. The multi-agent motion control method of claim 5, wherein, In step 4, the agent selects the action mode as follows: According to the results of step 3, the agent selects the action mode; If p≤p0, the action to be executed in the next step is selected according to the artificial potential field method; If [>p0, the action is selected according to the strategy network; Step 4.1, the agent generates a decision action according to the artificial potential field method: the position P of the agent i at time t needs to be obtained i the received attractive force and repulsive force Let P i =[P ix ,P iy ] represent the position of the agent i; P ig =[P igx ,P igy ] is the target position; then the normalized attractive force component received by the agent is represented as: and represent respectively the attractive forces in the x and y directions; The normalized repulsive force component of agent i on adjacent object j is expressed as follows: The position of the adjacent object j is P j = [P jx , P jy ]; P jx and P jy represent the positions of P j in the x direction and the y direction, respectively; and represent the repulsive forces of the adjacent object j in the x direction and the y direction, respectively; combining the attractive and repulsive forces results in a resultant force experienced by the agent i where represents the set of neighbors of agent i; σ ij is the collision function, representing the degree to which each member of the set of neighbors exerts a repulsive force on agent i, σ ij The expression for σ is as follows: d ij is the Euclidean distance between agent i and neighboring agent j;d m is the minimum collision distance between agent i and neighboring agent j;d c is the sensor detection distance;d r is a constant, with a value range of (d m ,d c ); the calculation formula and expression of parameters a, b, c and d are as follows: Linear velocity v of the agent i under the resultant force i and angular velocity ω i is calculated as follows: Where, k u The speed control gain is set to the maximum speed v. max ;||P i -P ig || represents the Euclidean distance between the position of agent i and the target position; k ω The angular velocity control constant is k. ω =2.5; ψ i This represents the heading angle value of agent i. Represents resultant force The direction angle, with a value of express The derivative with respect to time, The expression is as follows: Step 4.2, generating the decision action of the agent based on the Actor network: the Actor network receives the state input of the agent, extracts the features of the state through a multi-layer neural network, and finally outputs the action that the agent should execute; the Actor network introduces random noise to enhance the exploration ability, ensuring that the agent can effectively explore the unknown environment; as the training progresses, the Actor network continuously optimizes the policy parameters; The input of the Actor network is the current state s of the agent i The input layer extracts features by passing the original state information to the hidden layer. Each layer in the hidden layer weights the input and processes the non-linear relationship through the activation function ReLu. The Actor network extracts key features in the state layer by layer, and finally generates the action A of the agent i : wherein, are trainable parameters of the Actor network; the agent interacts with the environment at each time step At according to the action a output by the Actor network i and updates the parameters of the Actor network according to the feedback from the environment to optimize the policy and improve the efficiency of the agent's actions.

7. The multi-agent motion control method of claim 1, wherein, In step 5, the agent state is updated as follows: The agent generates an action A according to the current state s i and the policy network i interacts with the environment, which will feedback a new state s t+1 and a reward r t ; state s t+1 The update process ensures that the agent can adjust the behavior policy through the feedback received from the environment, and provides data support for subsequent experience replay.

8. The multi-agent motion control method of claim 1, wherein, In step 7, the judgment of whether the multi-agent enters the training phase is as follows: The agent needs to store experience when exploring the environment in order to update the training later; the number of experiences in the regular experience pool D, the excellent experience pool D', and the poor experience pool D" is |D|, |D'|, and |D" | respectively; the total number of experiences stored in the three experience pools is obtained by accumulating |D|, |D'|, and |D" |; compare the total number of experiences with the sampling number η, if |D|+|D'|+|D" |≥η, it meets the condition to enter the training phase, and go to step 8; if |D|+|D'|+|D" |<η, go to step 4 to continue generating experience number.

9. The multi-agent motion control method of claim 4, wherein, In step 8, the setting process of the sampling strategy and experience replay is as follows: Step 8.1, experience sampling: The agent will sample experiences from the regular experience pool D, the excellent experience pool D', and the poor experience pool D" to update the parameters of the Actor-Critic network; set the sampling probability as p0=1 / 8, and the number of samples sampled from D, D', and D" is n0, n1, and n2 respectively: Randomly sample from the three experience pools and store the sampled experience data into the sample set for computing target Q-values; Step 8.2, calculate the target Q value: From the sampled experience samples The target Q-values are computed using a target Critic network, which combines the current immediate reward r i and the expected future rewards, to help the agent evaluate the long-term return of performing a particular action in the current state; Step 8.3, update the Critic network to optimize the Q value estimation of the agent: The Critic network calculates the error between the current estimated Q value and the target Q value to adjust the Critic network parameters; by minimizing the error, the Critic network can learn more accurate Q values to provide accurate reference for the policy update of the Actor network; the update method of the Critic network is shown in the following formula: Loss function The smaller, the more accurate the Critic network's estimate of the Q value; the Critic network's update can learn a more accurate Q value; Step 8.4, update the Actor network: The goal of the Actor network is to find the action policy that maximizes the Q value of the Critic network; the update formula of the Actor network is as follows: The Actor network minimizes the policy gradient loss such that the generated action maximizes the Q-value estimated by the Critic network; Step 8.5, judge whether to update the target network: The target network parameter update is performed only once every fixed time interval ξ, which can reduce the oscillation in the policy update process and make the estimation of the target Q value more stable; when Δt%ξ=0 is met, go to step 8.7; if Δt%ξ≠0 is not met, continue to use the current target network parameters without updating the Actor-Critic network parameters; Step 8.6, the target network is updated in a soft update manner to maintain the stability of the target network: Target network parameters And The update mode is as follows: wherein, Critic network parameters; are Actor network parameters; τ is an update coefficient, gradually increasing from 0 to 1, which makes the Actor and Critic network parameters slowly update, improving the stability of neural network training; Step 8.7, the iteration is terminated: The training of the neural network continues until the Actor and Critic networks of the agent have reached the expected performance standard through the training data; when the training is terminated, the agent system saves the parameters of the Actor and Critic networks at the time of termination and uses these parameters in actual applications.

Images