Multi-agent collaborative pursuit method based on intelligent algorithm in dynamic environment

Abstract

The invention discloses a multi-agent collaborative pursuit method based on an intelligent algorithm in a dynamic environment, and the method comprises the steps: firstly obtaining an image containing the state information of an obstacle, processing the image through employing a variational auto-encoder VAE, and obtaining the motion information of an unknown obstacle; secondly, obtaining a complete state vector of the intelligent agent according to unknown obstacle motion information and intelligent state information obtained through observation; and finally, acquiring a pursuit strategy through a pursuit reinforcement learning network, and training through a multi-agent double-delay depth deterministic strategy gradient MATD3 algorithm. And acquiring an escaper strategy through the escaper reinforcement learning network, and performing training through a near-end strategy optimization PPO algorithm to complete multi-agent collaborative pursuit. According to the method, the stability of the control strategy and the network is improved, the possibility that the intelligent agent control strategy falls into local optimum is reduced, and the problem of efficient pursuit of multiple intelligent agents in a complex dynamic environment is solved.

Description

Technical Field

[0001] The present invention belongs to the field of multi-agent game optimization, and specifically relates to a multi-agent collaborative pursuit method based on intelligent algorithms in a dynamic environment. Background Art

[0002] With the rapid development of artificial intelligence technology, traditional decision-making methods based on preset rules have become difficult to cope with increasingly complex, dynamic and unknown environments. Therefore, the collaborative decision-making and / or adversarial decision-making of multi-agent systems are gradually relying on the support of intelligent technologies such as autonomous learning algorithms. The rise of Multi-Agent Reinforcement Learning (MARL) has brought significant breakthroughs in solving complex decision-making problems in multi-agent games. In such games, each agent not only needs to continuously optimize its own strategy to cope with the dynamic changes of the opponent's strategy, but also needs to quickly adapt to external conditions and adjustments to other agents' strategies in a complex dynamic environment.

[0003] In the research of multi-agent game optimization based on reinforcement learning, the framework of centralized training and distributed execution (CTDE) is usually adopted, combined with deep reinforcement learning (DRL) technology, so that the agent can autonomously learn and generate effective individual strategies in complex and dynamic environments. Compared with the traditional game method with preset rules, the reinforcement learning-based method allows the agent to autonomously generate the optimal strategy through the exploration-exploitation balance mechanism in the process of real-time interaction with the environment. This method can not only handle complex and high-dimensional state spaces, but also realize adaptive optimization of strategies in highly dynamic, uncertain and adversarial scenarios.

[0004] Although the technology of multi-agent cooperative pursuit and escape game has made some progress, there are still several problems that need to be solved in existing research: (1) In conventional reinforcement learning simulation environments, the motion information of dynamic obstacles is usually known, but in practical applications, the information of environmental obstacles is often unpredictable or difficult to perceive in advance. How to effectively obtain and process unknown obstacle information in the pursuit and escape game problem becomes a challenge; (2) In the interactive environment of sparse rewards, the overall reward value of experience in the reward pool is low, and the proportion of high-value experience in the experience pool is small, resulting in a lower probability of the agent sampling and learning high-reward experience, which may affect the learning efficiency and convergence speed; (3) In the multi-target pursuit scenario, the pursuer needs to reasonably allocate limited attention and resources among multiple escapees to maximize the capture efficiency. Therefore, how to effectively perceive and process unknown obstacle information, improve the sample efficiency in the reinforcement learning process, and optimize the pursuer's multi-target strategy execution have become key technical issues that need to be overcome in this field. Summary of the invention

[0005] In view of the above problems, the present invention provides a multi-agent multi-hybrid collaborative pursuit method based on reinforcement learning algorithm in a dynamic environment. First, the variational autoencoder (VAE) algorithm is used to obtain the motion information of dynamic obstacles, and provide obstacle information to both parties of the pursuit and escape to achieve obstacle avoidance. In the multi-agent strategy training stage, the pursuer is trained by the Multi-Agent Twin Delayed Deep Deterministic Policy Gradient MATD3 (Multi-Agent Twin Delayed Deep Deterministic PolicyGradient) algorithm, and the escaper adopts the proximal policy optimization PPO (Proximal Policy Optimization) algorithm. Among them, the decoupled reward function and the pursuit priority mechanism are designed in the collaborative pursuit strategy to balance individual and collaborative tasks and achieve optimal decision-making.

[0006] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0007] S1. Sample images containing obstacles, use variational autoencoder VAE to encode this high-dimensional image data into low-dimensional hidden layer variables, and obtain motion information containing the position and speed of the obstacles.

[0008] The training objective function of VAE is selected as follows:

[0009] in, , They represent the input image containing obstacle information and the image reconstructed by the VAE network. represents the expected value, is a hidden variable, , are the network parameters of the decoder and encoder respectively, is a hyperparameter that controls the importance of the KL divergence term, is the output of the decoder, representing a given latent variable Reconstruction under the conditions The probability distribution of is the output of the encoder, indicating that the given input The latent variable under the condition The probability distribution of . is the prior distribution, usually assumed to be a standard normal distribution ,KL divergence is used to measure the output distribution of the encoder With prior distribution to facilitate learning reasonable coding. To reconstruct the loss function, is the KL divergence loss function, is the objective function for the overall training of VAE.

[0010] In the network selection of the encoder and decoder of VAE, the present invention adopts the ResNet18 (ResidualNetwork 18 layers) network structure, which is a deep convolutional neural network widely used in computer vision tasks. It introduces residual blocks to solve the gradient vanishing and gradient explosion problems in deep network training. The basic structure of the ResNet18 network includes an input layer, a convolutional layer, a residual block, a global average pooling layer, and a fully connected layer (output layer). A 3×128×128 image is used as the input of the network, after the initial convolutional layer, batch normalization and pooling are performed, and the convolutional layer uses a 3×3 convolution kernel to extract local features of the image. There are 8 residual blocks in total, each of which consists of two convolutional layers and a jump connection. The global average pooling layer performs global average pooling on the feature map and converts the feature map into a one-dimensional vector. The output layer converts it into a column vector, and then passes through the fully connected layer to output the mean and variance to obtain the latent space. The encoder generates the latent space, and the latent representation containing the obstacle speed information and position information is sampled in the latent space.

[0011] S2. Based on the motion information containing the position and speed of the obstacle and the intelligent state information observed in the simulation environment, the complete state vector of the agent is obtained, the pursuer strategy is obtained through the pursuer reinforcement learning network, and trained through the multi-agent double-delay deep deterministic policy gradient MATD3 algorithm.

[0012] The agent state information includes the agent's position information, speed, the relative position between the current agent and other agents, and the escaper's speed. The pursuer reinforcement learning network consists of two critic networks, two target critic networks, an actor network, and a target actor network. The input is a complete state vector, and the output is a pursuer strategy.

[0013] The specific contents of the MATD3 algorithm used to train the pursuer strategy are as follows:

[0014] S2.1, the parameters of the pursuer reinforcement learning network trained by the MATD3 algorithm, the two critic network parameter sets and the strategy set are expressed as follows:

[0015]

[0016] in, represents the set of parameters of the pursuer's policy network, , Represents the two Critic network parameter sets of the pursuer, represents the set of strategies of all pursuers.

[0017] S2.2. In the dual-delay deep deterministic policy gradient TD3 algorithm, a dual Critic network is introduced, and two independent Critic networks are used to estimate the Q value (state-action value). Since the initial parameters of the two networks are different, the Q values ​​they calculate are also different. The minimum value is taken as the target Q value, which reduces the over-estimation bias and improves the stability of the strategy.

[0018] In the MATD3 algorithm, The target Q value of each pursuer is calculated as follows:

[0019]

[0020] in, Indicates The instantaneous reward of a single step for a pursuer, , Indicates Two target critic networks for the pursuer, , Indicates The parameters of the two target critic networks of the pursuer, represents the pursuer discount factor, which is the same for each pursuer. is the cumulative set of the next state of all pursuers’ observation spaces, For the The target policy network of the pursuer, As its network parameters, Representative The next state of each pursuer based on the local observation The actions taken, , .

[0021] S2.3, Similar to TD3, the MATD3 algorithm also uses the mean square error between the target Q value and the estimated Q value as the loss function. The loss function expression of the critic network parameter update of the pursuer and its gradient descent parameter update are as follows:

[0022]

[0023] in, Indicates The first The loss function of the Critic network is: represents the mathematical expectation, is the global state of all pursuers at the current moment , is the global state after executing the pursuit action, is the combined action of all pursuers at the current moment , The current pursuer's Critic network, represents the Q value estimated by the Critic network, represents the learning rate, represents the descent gradient, For the experience replay pool, The quadruple data stored in the experience pool.

[0024] S2.4 and TD3 introduce target strategy smoothing: when calculating the target Q value, noise is added to the target strategy and a shearing operation is performed to make the estimate more accurate, further reducing the deviation of the Q value estimation and improving the stability of the network.

[0025] No. The target strategy output behavior of each pursuer is as follows:

[0026]

[0027] in, is the next action generated by its target policy network; is the added noise, indicating a mean of 0 and a standard deviation of The normal distribution of is a clipping operation that limits the action to the range (-1,1). The strategy noise distribution of each pursuer is the same. (-1,1).

[0028] S2.5. The goal of the Actor network is to maximize the Q value output by the Critic network. Therefore, the loss function of the pursuer's Actor network can be defined as a negative Q value, and the gradient ascent method is used to update the network parameters. The Actor network parameters of the pursuer are updated as follows:

[0029]

[0030] in, No. The loss function of the Actor network of agents, is the ascending gradient, is the global state of all pursuers at the current moment , is the combined action of all pursuers at the current moment ; When updating the Actor network parameters, only the Q value calculated by a Critic network is needed, so you can choose or , here select ; Indicates Actor network parameters of the pursuer, Indicates Each pursuer observes its own local The action taken.

[0031] S2.6. For both the target Actor network and the target Critic network, a soft update method is used:

[0032] in , Respectively represent The target strategy network parameters of the pursuer, The mth target Critic network parameters of the pursuer, is the soft update parameter, and the soft update parameter of the target Actor network and the target Critic network are the same.

[0033] In addition, TD3 adopts delayed policy updates. After each update of the Critic network, TD3 delays the update of the Actor network, reducing the update frequency of the policy network and thus reducing the noise of the policy update.

[0034] The higher update frequency of the Critic network makes it more stable relative to the Actor, and thus more efficient in guiding the Actor. The above improvements reduce the possibility of the agent's control strategy falling into the local optimum.

[0035] S2.7. In the pursuit-escape confrontation environment, the pursuer needs to capture the escapee as quickly as possible, avoid obstacles in time and reduce collisions with other agents. For the multi-constrained reward function design problem, it is often necessary to use a decoupled reward function to design the pursuer's reward function. The specific process of the pursuer reward function design is as follows:

[0036] S2.7.1. In actual situations, different escapees have different pursuit priorities, so Assign priority weights to escapees . When an escapee is captured, the pursuer should switch to chasing other escapees. Since it is not easy to change the state space during training, and in order to simulate the above process, it is assumed that when an escapee is captured, its priority weight is reduced, so that its pursuit weight can be ignored relative to other escapees that have not been captured. The priority weights of the escapees are as follows:

[0037]

[0038] When an escapee is captured, its priority weight is reduced, making its pursuit weight negligible relative to other escapees that have not been captured. The escapee priority weight acts on the pursuer's reward function as well as its own reward function.

[0039] S2.7.2. Let the decoupled reward function of the pursuer be as follows:

[0040] ;

[0041] in, For pre-capture rewards, For the coordinated siege reward, To capture the reward, is the time step reward; Indicates The pursuer and The distance of the escapee, Indicates the capture distance, Indicates the pre-capture distance (encirclement distance).

[0042] If the distance between the current pursuer and a certain escapee is less than the pre-capture distance and greater than the capture distance, the pre-capture reward will be obtained. ; When there are other pursuers and the distance between them and the escapee is less than the pre-capture distance and greater than the capture distance, the cooperative encirclement reward is obtained. , guide all pursuers to form a siege. If the distance between the current pursuer and a certain escapee is less than the capture distance, the capture reward is obtained. When the distance between all pursuers and escapers is greater than the pre-capture distance, they will receive a step bonus. .

[0043] S2.7.3. Design the pre-capture reward and collaborative encirclement reward as follows:

[0044]

[0045] in, For the The coordinates of the agents, For the The priority weight of each escapee, is the pre-capture reward coefficient, is the encirclement situation reward coefficient, , are the number of pursuers and escapers respectively, It is a constraint reward in the z direction. When capturing and forming a closed situation, it is hoped to remain at the same height. is the height difference reward coefficient, It is the intelligent agent in On the plane The pursuer and The adjacent angles of the pursuers (there are no other pursuers between the two pursuers) relative to the escaper, .

[0046] S2.7.4 Design capture rewards are as follows:

[0047]

[0048] Among them, Gained when an escapee is captured , The current pursuer is chasing The priority weight of each escapee, For the capture reward coefficient, the capture reward coefficient of all pursuers is the same. is the distance reward coefficient, Indicates the shortest distance between the current pursuer and the remaining escapees who have not been captured. It is used to guide the current pursuer to turn to other escapees after capturing a certain escapee.

[0049] S2.7.5 Design step size rewards are as follows:

[0050]

[0051] in,

[0052]

[0053] in, + + =1, They are distance bonus coefficient, obstacle collision bonus coefficient, and similar collision bonus coefficient. It is a distance guidance reward, which guides the pursuer to shorten the distance with the escapee. If the current escapee's priority weight is less than 1, it means that the current escapee has been arrested, and the minimum distance with other escapees needs to be calculated. If the current escapee's priority weight is equal to 1, it means that the current escapee has not been arrested, and the pursuer continues to pursue. Represents the collision reward with the obstacle, which is used to guide the pursuer to learn the obstacle avoidance strategy. Since it is difficult to output the hidden variables from VAE The specific functional relationship with the obstacle is determined in , so the relative distance constraint reward is not used here, but a supervised judgment is made on whether a collision occurs. We assume that the trajectory equation of the obstacle is known, and the real spatial position of the obstacle at each time step is known to judge whether it collides with the agent. However, for all agents themselves, the real spatial position of the obstacle cannot be directly obtained; A collision reward for the agent, so that when the capture is successful, it will avoid colliding with other agents as much as possible. represents the minimum distance between the pursuer and the escaper, is the distance between the current pursuer and the obstacle, , is the shortest distance between collisions between agents ( =2 ).

[0054] S3. The complete state vector of the agent is obtained based on the unknown obstacle motion information obtained by the variational autoencoder (VAE) algorithm and other state information observed in the simulation environment. Based on the complete state vector of the agent, the escaper strategy is obtained through the escaper reinforcement learning network, and the escaper strategy is trained through the proximal strategy optimization PPO algorithm. The escaper reinforcement learning network has the same structure as the pursuer reinforcement learning network. The specific content of the PPO algorithm used to train the escaper strategy is as follows:

[0055] S3.1, PPO belongs to the policy gradient method, which maximizes the expected return by directly optimizing the strategy. PPO improves the traditional policy gradient method, making the policy update more stable and efficient. The core idea of ​​the algorithm is to limit the amplitude of the policy update so that each policy update is carried out within a certain range close to the original policy, thereby maintaining the stability of the policy update, that is, proximal policy optimization.

[0056] Importance sampling is used to estimate the policy gradient ratio between the new and old policies. In PPO, the expected return under the new strategy is estimated by re-weighting the sampled trajectories under the old strategy according to the probability of the new strategy, avoiding the need to re-sample data for each update.

[0057] No. The probability ratio of the new and old strategies for the escapees is as follows:

[0058]

[0059] in, Indicates The escapee is observed locally Under the current strategy Take Action The probability of Indicates The escapee is observed locally Next, according to the previous strategy take The probability of Indicates Batch data.

[0060] S3.2. In reinforcement learning, advantage estimation is used to evaluate the superiority of a specific action over the average behavior. The generalized advantage estimation (GAE) is commonly used in the PPO algorithm. GAE introduces a parameter , is used to smooth the advantage estimate in time, thus achieving a balance between variance and bias.

[0061]

[0062] in, Indicates The timing difference error of the escapees, For the Runaway time steps Instant rewards, is the escapee discount factor, which is the same for all escapees; , Respectively represent Individual fugitive status , The value function of . Symbol Indicates an offset or delay in the time step. Indicates escapee from time The weighted summation of the timing difference errors starts from the beginning to the future. >0, means Agents in state The reward of taking the current action is higher than the average, otherwise Agents in state The reward for taking the current action is below average.

[0063] S3.3, PPO introduces an alternative optimization objective function (Surrogate Objective) to approximate the original objective function of the Actor network, maintaining optimization stability while making it easier to optimize. In PPO, its optimization objective function is:

[0064]

[0065] in Represents the batch data size, is the number of escapees, Representative The first batch of data The advantage of an escapee is estimated, For the The probability ratio of the new and old strategies of the escapees, Represents the clipping range, which is used to limit the size of the policy update to prevent instability caused by excessive policy updates. is the entropy regularization coefficient (the entropy regularization coefficient of each escapee is the same). The larger the value, the stronger the exploration ability. It is The escapees observe locally according to their strategies By introducing the entropy regularization term, we can enhance the exploration of the strategy and slow down the strategy from converging too early into the local optimum. , the Actor can improve in the direction of maximizing the advantage function, thereby improving the quality of the strategy.

[0066] S3.4, Actor network parameter gradient update method is as follows:

[0067]

[0068] in, For the The Actor network parameters of the escapee, is the learning rate of the escapees, and the learning rate of each escapee is the same, is the Actor network loss function with respect to the parameters gradient.

[0069] S3.5. At the same time, for the Critic network in the PPO algorithm, its goal is to minimize the mean square error between the predicted value and the actual return. Its loss function and network parameter update calculation are as follows:

[0070]

[0071] in, For the escapees to the global state The estimated value of For the The critic network parameters of the escapees, is the batch data size, For the The actual reward obtained by the escapee in each round , is the learning rate.

[0072] S3.6. The reward function design for the escaper is similar to that for the pursuer. The collaboration reward and pre-capture reward are cancelled, and the capture reward is At the same time, if the escapee has been captured, according to the actual situation, he should not continue to participate in the escape process, so that the escapee will no longer move after being captured and will always receive the capture reward.

[0073] The relative distance constraint rewards for escapers are as follows:

[0074]

[0075] in, is the escaper distance reward coefficient, Represents the minimum distance between all pursuers and the escaper.

[0076] In general, the decoupled reward function between the pursuer and the escaper improves the team's collaboration while retaining the subjective initiative of the individuals. Designed as a reward for cooperation, It is an individual reward, which acts on a single individual. The joint reward part is the main purpose of the task, and a certain amount of obstacle avoidance errors is allowed.

[0077] Beneficial effects of the present invention: The present invention uses the VAE framework to learn the latent model of obstacle sampling images, reduces learning complexity and provides movement information of obstacles, which helps the intelligent agent to make obstacle avoidance decisions when it cannot directly observe the environmental state. In order to realize the confrontation process of the intelligent agents of both camps, the pursuer adopts the MATD3 algorithm, and the escaper adopts PPO for training. Centralized training and distributed execution are adopted to improve the stability of the control strategy and the network, and reduce the possibility of the intelligent agent control strategy falling into the local optimum. A decoupled reward function mechanism is designed to incentivize the pursuer to capture the escaper and avoid collisions with obstacles and other intelligent agents, while reducing the impact of sparse rewards on learning efficiency and improving the accuracy of the algorithm. A pursuit priority mechanism is designed to enhance the resource utilization efficiency of the pursuer and improve the capture success rate. In summary, the present invention solves the problem of efficient pursuit of multiple agents in complex dynamic environments and provides a strategic method for multi-agent adversarial tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 This is a schematic diagram of the multi-agent pursuit space;

[0079] Figure 2 A diagram of the distributed execution framework for centralized training;

[0080] Figure 3 Schematic diagram of the network model of VAE;

[0081] Figure 4 This is the overall implementation framework diagram of the pursuit based on VAE;

[0082] Figure 5 It is the position information diagram of the obstacle on three axes;

[0083] Figure 6 This is a schematic diagram of simulation rendering;

[0084] Figure 7 This is the VAE test result diagram;

[0085] Figure 8 The average reward curve of the pursuer trained with MATD3-DDPG and MATD3-IPPO;

[0086] Fig. 9 This is the average reward curve for escapee training under MATD3-DDPG and MATD3-IPPO;

[0087] Fig.10 It is the capture success rate curve under the two algorithms;

[0088] Fig.11 It is the simulation trajectory diagram;

[0089] Fig.12 A diagram showing the pursuer's collaborative encirclement capabilities. DETAILED DESCRIPTION

[0090] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0091] A multi-agent collaborative pursuit method based on intelligent algorithm in a dynamic environment includes the following steps:

[0092] S1. Sample images containing obstacles, use variational autoencoder VAE to encode this high-dimensional image data into low-dimensional hidden layer variables, and obtain motion information containing the position and speed of the obstacles.

[0093] S2. Based on the motion information containing the position and speed of the obstacle and the intelligent state information observed in the simulation environment, the complete state vector of the agent is obtained, the pursuer strategy is obtained through the pursuer reinforcement learning network, and trained through the multi-agent double-delay deep deterministic policy gradient MATD3 algorithm.

[0094] S3. Based on the complete state vector of the agent, the escaper strategy is obtained through the escaper reinforcement learning network, and trained through the proximal strategy optimization PPO algorithm.

[0095] Embodiment 1:

[0096] In order to simplify the modeling of the environment while being close to the real environment, the pursuit problem in the embodiment of the present invention is, in a three-dimensional free continuous space, and in the presence of a dynamic obstacle moving along a fixed trajectory, how can three pursuers quickly cooperate to capture an escapee and avoid the obstacle in time. During the pursuit process, if the distance between all escapees and pursuers is less than the pre-set safety distance, the pursuit is considered successful. If the above conditions are not met within the preset pursuit time, it is judged as a failure. Collisions between the pursuers and obstacles are not judged as failures.

[0097] like Figure 1 As shown, each agent and obstacle are set with a safe distance, which is represented by and , when the distance between the pursuer and the obstacle Less than and If the sum of , the obstacle avoidance fails, Indicates The pursuer and The distance between the escapees is If the distance is less than the safe distance between the pursuer and the escapee, the pursuit is successful. The main task of the pursuer is still to get close to the escapee, and the secondary task is to avoid collisions with obstacles and friendly parties; the main task of the escapee is to distance himself from all pursuers, and the secondary task is to avoid collisions with obstacles. In order to improve the capture success rate, consider adding an encirclement situation reward to the design of the pursuer's reward function. The encirclement situation is conducive to the pursuer's more efficient pursuit.

[0098] In a multi-agent training environment, the single-agent reinforcement learning algorithm is limited. This is because a single agent will dynamically adjust its control strategy during training, and this strategy change causes the environment to be unstable, which in turn interferes with the training of the agent. In order to achieve the confrontation process of the agents of both camps, the pursuer uses the MATD3 algorithm and the escaper uses PPO for training. For the training of the pursuer, centralized training distributed execution is used, such as Figure 2 shown.

[0099] Embodiment 2:

[0100] Figure 3 The network model of VAE is given. The block diagram of the entire implementation of VAE-based pursuit is as follows Figure 4 Assume that the trajectory of the obstacle is a spatial spiral line, and the equation is as follows:

[0101]

[0102] Both the pursuit and escape agents use a non-holonomic constraint kinematic model. , represents the position coordinates of the agent on the horizontal plane, Indicates its vertical position coordinates, Indicates the horizontal movement direction and The heading angle of the axis, Represents the agent and , The angle with the horizontal plane is also called the pitch angle. is the rotation angle of the agent around its own front and rear axis, that is, the roll angle, ignoring the effect of the roll angle on its own linear velocity. , , are the heading angular velocity, pitch angular velocity and roll angular velocity respectively, is the linear velocity of the agent in the direction of its motion.

[0103] The state space equation is as follows:

[0104]

[0105] The experiment combines VAE with reinforcement learning (RL) and uses the encoded latent variables as the state representation in the RL algorithm. The specific implementation is as follows: first, an image containing obstacles is collected and preprocessed, and the image collected in three-dimensional space is projected onto , , On the plane, , Plane projection and , The plane projection images are stitched together, and the resulting image contains the position information of the obstacle in three axes, such as Figure 5 shown.

[0106] The image dataset containing obstacle information is input into the VAE network for pre-training.

[0107] The decoder reconstructs the image based on the probability distribution of the obstacle trajectory. The trained encoder is used for low-dimensional mapping of the obstacle image information. Then the strategy training of the RL agent is carried out. When training the RL agent, the obstacle image is retrieved from the environment, and after processing, the encoder is used to generate the latent space. Then, the encoder outputs the probability distribution (latent space) and samples to obtain the potential representation of the obstacle. Other state information observed from the simulation environment and The complete state vector of the agent is recombined and used to train the control strategies of both the pursuit and escape parties.

[0108] The simulation scene is a three-dimensional space cube with an x, y, z range of [-1, 1]. There are three pursuers and two escapers in the area. The safe space of all agents is a sphere with the agent position as the center and a radius of inside the sphere, =0.03. In order to better demonstrate the cooperative ability of the pursuer, it is assumed that the pursuer is more flexible but slower. The maximum speed of the pursuer is set to 1.5, and the maximum acceleration is set to 4.5; the maximum speed of the escaper is set to 2.4, and the maximum acceleration is set to 3.0. The escaper has a larger range of behavior space than the pursuer, that is, the behavior is more flexible. Minimum distance for collision penalty between agents =0.06, safe distance for collision with obstacles =0.06. When the distance is less than this, it means a collision occurs and a collision penalty is imposed. Figure 6 As shown, the red dot represents the obstacle, the three purple dots represent the three pursuers, and the green dot represents the escaper.

[0109] The VAE test results are as follows Figure 7 Assume that one time step represents 1 The first row is the original input image of VAE, which represents the obstacle position in five time steps. , and , The projection of the plane, the second line is the image reconstructed by the VAE decoder. In the simulation environment with less interference and without too many image details, the VAE fitting effect is better, and the input image is basically restored. This shows that the VAE decoder can effectively learn and capture the probability distribution of obstacle trajectories.

[0110] Figure 8 The average reward curve of the pursuer trained by MATD3-DDPG and MATD3-IPPO. In the figure, the horizontal axis Episode represents the number of interaction rounds, the vertical axis represents the average reward of the three pursuers, the red and blue curves represent the average rewards of 100 rounds under the MATD3-DDPG and MATD3-IPPO algorithms, respectively, and the shaded part is the average reward in each round. In the first 500 rounds, it is in the experience collection process. After 500 rounds, the curve shows a rapid upward trend. Between 3000 and 6000 rounds, the escaper reward has converged. At the same time, MADDPG adopts a fixed strategy, and the pursuer is relatively easy to learn, so the pursuer reward under MATD3-DDPG is higher than that under MATD3-IPPO. After 7000 rounds, the pursuer reward tends to converge, and both camps learn the best strategy.

[0111] Average reward curves for escaper training under MATD3-DDPG and MATD3-IPPO Fig. 9 As shown. In the figure, the horizontal axis Episode represents the number of interaction rounds, the vertical axis represents the average reward of the three escapees, the red and blue curves represent the average rewards of 100 rounds under the MATD3-DDPG and MATD3-IPPO algorithms respectively, and the shaded part is the average reward in each round. PPO uses the strategy distribution probability output. In the first 500 rounds, it deviates far from the ideal strategy distribution, so its reward is relatively lower than DDPG. In 500-2500 rounds, the escapee reward curve shows a convergence trend. Since DDPG uses a deterministic strategy, it is more likely to fall into the local optimum (at 1600 rounds); for the PPO adopted by the escapee, its strategy behavior variance is large, but it is not easy to fall into the local optimum. Between 4000 and 6000 rounds, the pursuer's learning ability is stronger under MATD3-DDPG, so the escapee's reward declines. After 6000 rounds, the average rewards of both parties tend to converge.

[0112] In the test, in order to reduce the chance, a total of 10 experiments were conducted, each with 10 rounds. Fig.10 The capture success rate curves of the two algorithms. ET represents the number of experiments, and ST represents the number of successful pursuits by the pursuer (both escapees were captured). It can be seen that the capture success rate of the pursuer under MATD3-DDPG is relatively high, reaching about 92%. This proves that the pursuer can learn the escaper's strategy very well and successfully capture it. PPO is less sensitive to algorithm parameters and outputs a strategy distribution. Its escape strategy is more diverse, and it is more difficult for the pursuer to learn.

[0113] Fig.11 This is a simulation trajectory diagram. The red color is the obstacle trajectory, the blue color is the pursuer trajectory, and the purple color is the escaper trajectory. It can be found that before entering the pre-capture distance, the pursuer first chases the escaper that is relatively close to itself. When one of the pursuers enters the capture distance and successfully captures an escaper, it immediately changes its strategy and chases other escapers. Fig.12 It demonstrates the cooperative encirclement capability of the pursuers during the pursuit process. Within the pre-capture range, the pursuers present the expected encirclement posture, thereby better pursuing the escapees.

[0114] The VAE with ResNet18 network structure successfully fits the probability distribution of the obstacle location, and its latent variables can represent the shallow state information of the obstacle.

[0115] Under the MATD3-DDPG algorithm, the pursuer can effectively capture all escapees, proving the effectiveness of the decoupled reward function design. After successfully capturing one of the escapees, the strategy is changed immediately, indicating that the escapee weight and the transfer target reward successfully guide the pursuer to obtain the collaborative reward while satisfying the individual maximum reward. The encirclement situation displayed during the pursuit process proves that the cooperative encirclement reward of the pursuer is considerable.

[0116] TD3 has a more obvious advantage over DDPG due to its improvement of network structure and introduction of target strategy noise, and the pursuer has stronger learning ability. Although the strategy learned by the pursuer under MATD3-DDPG is very impressive, the probability distribution output and parameter insensitivity of PPO make its learning difficulty significantly increased, and the success rate is only about 52%. In the problem of multi-target pursuit, there is still a lot of room for improvement.

Claims

1. A multi-agent collaborative pursuit method based on intelligent algorithm in a dynamic environment, characterized in that: The following steps are involved: S1. Sample images containing obstacles, use variational autoencoder VAE to encode image data into hidden layer variables, and obtain motion information containing obstacle position and speed; S2. Based on the motion information containing the position and speed of the obstacle, combined with the intelligent state information observed from the simulation environment, the complete state vector of the agent is obtained, the pursuer strategy is obtained through the pursuer reinforcement learning network, and trained through the multi-agent double-delay deep deterministic policy gradient MATD3 algorithm; S3. Based on the complete state vector of the agent, the escaper strategy is obtained through the escaper reinforcement learning network, and trained through the proximal strategy optimization PPO algorithm.

2. The multi-agent collaborative pursuit method based on intelligent algorithm in a dynamic environment according to claim 1 is characterized in that: The specific implementation process of step S1 is as follows: The dataset consisting of images containing obstacle status information is input into VAE for training, and the encoder is used to obtain the probability distribution of obstacle motion information; The training objective function of VAE is selected as follows: ; in, , They are the input image containing obstacle information and the image reconstructed by the VAE network. is the expected value, is a hidden variable, , are the network parameters of the decoder and encoder respectively, is a hyperparameter that controls the importance of the KL divergence term, is the output of the decoder, representing the given latent variable Reconstruction under the conditions The probability distribution of is the output of the encoder, indicating that the given input The latent variable under the condition The probability distribution of is the prior distribution, and the KL divergence is used to measure the output distribution of the encoder With prior distribution The difference between To reconstruct the loss function, is the KL divergence loss function, The objective function for the overall training of VAE; In the network selection of the VAE encoder and decoder, the ResNet18 network structure is used. The residual block is introduced to solve the gradient vanishing and gradient exploding problems in deep network training. The encoder generates a latent space, and the latent space is sampled to obtain a potential representation containing the obstacle speed information and position information.

3. The multi-agent collaborative pursuit method based on intelligent algorithm in a dynamic environment according to claim 2 is characterized in that: In step S2, the agent state information includes the agent's position information, speed, the relative position between the current agent and other agents, and the escaper's speed; The pursuer reinforcement learning network consists of two critic networks, two target critic networks, an actor network and a target actor network, the input is a complete state vector, and the output is a pursuer strategy.

4. The multi-agent collaborative pursuit method based on intelligent algorithm in a dynamic environment according to claim 3 is characterized in that: The specific implementation process of the MATD3 algorithm is as follows: S2.1, the parameters of the pursuer reinforcement learning network trained by the MATD3 algorithm, the two critic network parameter sets and the strategy set are expressed as follows: ； in, represents the set of parameters of the pursuer's policy network, , Represents the two Critic network parameter sets of the pursuer, represents the set of strategies of all pursuers; S2.2, in the MATD3 algorithm, The target Q value of each pursuer is calculated as follows: ； in, Indicates The instantaneous reward of a single step for a pursuer, , Indicates Two target critic networks for the pursuer, , Indicates The parameters of the two target critic networks of the pursuer, represents the pursuer discount factor, and the discount factor of each pursuer is the same; is the cumulative set of the next state of all pursuers’ observation spaces, For the The target policy network of the pursuer, As its network parameters, Representative The next state of each pursuer based on the local observation The actions taken, , ; S2.3, MATD3 algorithm uses the mean square error between the target Q value and the estimated Q value as the loss function. The loss function expression of the critic network parameter update of the pursuer and its gradient descent parameter update are as follows: ； in, Indicates The first The loss function of the Critic network is: represents the mathematical expectation, is the global state of all pursuers at the current moment , is the global state after executing the pursuit action, is the combined action of all pursuers at the current moment , The current pursuer's Critic network, represents the Q value estimated by the Critic network, represents the learning rate, represents the descent gradient, For the experience replay pool, The four-tuple data stored for the experience pool; S2.4, introduce target strategy smoothing: when calculating the target Q value, add noise to the target strategy and perform a shearing operation; The target strategy output behavior of each pursuer is as follows: ； in, is the next action generated by its target policy network; is the added noise, indicating a mean of 0 and a standard deviation of The normal distribution of is a clipping operation that limits the action to the range (-1,1). The strategy noise distribution of each pursuer is the same. (-1,1); S2.5, the loss function of the pursuer's Actor network is defined as the negative Q value, and the gradient ascent method is used to update the network parameters; The Actor network parameters of the pursuer are updated as follows: ； in, No. The loss function of the Actor network of agents, To increase the gradient, only one Q value calculated by the Critic network is used when updating the Actor network parameters, so choose or ; Indicates Actor network parameters of the pursuer, Indicates Each pursuer observes its own local the actions taken; S2.

6. For both the target Actor network and the target Critic network, a soft update method is used: ； in , Respectively represent The target strategy network parameters of the pursuer, The mth target Critic network parameters of the pursuer, is the soft update parameter, and the soft update parameter of the target Actor network and the target Critic network are the same; S2.

7. Use the decoupled reward function to design the pursuer’s reward function.

5. The multi-agent collaborative pursuit method based on intelligent algorithm in a dynamic environment according to claim 4 is characterized in that: The specific process of designing the reward function of the pursuer is as follows: S2.7.1, for Assign priority weights to escapees , depending on whether the escapee is captured or not, the priority weights are as follows: ； S2.7.

2. Let the decoupled reward function of the pursuer be as follows: ； in, For pre-capture rewards, For the coordinated siege reward, To capture the reward, is the time step reward; Indicates The pursuer and The distance of the escapee, Indicates the capture distance, Indicates the pre-capture distance; If the distance between the current pursuer and a certain escapee is less than the pre-capture distance and greater than the capture distance, the pre-capture reward will be obtained. ; When there are other pursuers and the distance between them and the escapee is less than the pre-capture distance and greater than the capture distance, the cooperative encirclement reward is obtained. , guide all pursuers to form a siege; if the distance between the current pursuer and a certain escapee is less than the capture distance, the capture reward is obtained ; When the distance between all pursuers and escapers is greater than the pre-capture distance, they will receive a step bonus ; S2.7.

3. Design the pre-capture reward and collaborative encirclement reward as follows: ； in, For the The coordinates of the agents, is the pre-capture reward coefficient, is the encirclement situation reward coefficient, , are the number of pursuers and escapers respectively, It is a constraint reward in the z direction. When capturing and encircling, it can stay at the same height. is the height difference reward coefficient, It is the intelligent agent in On the plane The pursuer and The adjacent angles of the pursuer, that is, the escaper, ; S2.7.4 Design capture rewards are as follows: ； Among them, Gained when an escapee is captured , The current pursuer is chasing The priority weight of each escapee, For the capture reward coefficient, the capture reward coefficient of all pursuers is the same; is the distance reward coefficient, Indicates the shortest distance between the current pursuer and the remaining escapees that have not been captured. It is used to guide the current pursuer to turn to other escapees after capturing a certain escapee. S2.7.5 Design step size rewards are as follows: ； ； in, + + =1, They are distance bonus coefficient, obstacle collision bonus coefficient, and similar collision bonus coefficient. It is a distance guidance reward, which guides the pursuer to shorten the distance with the escapee. If the current escapee priority weight is less than 1, it means that the current escapee has been arrested, and the minimum distance with other escapees is calculated. If the current escapee priority weight is equal to 1, it means that the current escapee has not been arrested, and the pursuer continues to pursue; Represents the collision reward with obstacles, which is used to guide the pursuer to learn obstacle avoidance strategies; A collision reward for the agent, so that it does not collide with other agents when the capture is successful. represents the minimum distance between the pursuer and the escaper, is the distance between the current pursuer and the obstacle, , is the shortest distance for collision between agents, =2 .

6. The multi-agent collaborative pursuit method based on intelligent algorithm in a dynamic environment according to claim 5 is characterized in that: The escaper reinforcement learning network has the same structure as the pursuer reinforcement learning network.

7. The multi-agent collaborative pursuit method based on intelligent algorithm in a dynamic environment according to claim 6 is characterized in that: The specific implementation process of the PPO algorithm is as follows: S3.

1. Use importance sampling to estimate the policy gradient ratio between the new and old policies ; In PPO, the expected return under the new strategy is estimated by reweighting the sampled trajectories under the old strategy according to the probability of the new strategy; The probability ratio of the new and old strategies for the escapees is as follows: ； in, Indicates The escapee is observed locally Under the current strategy Take Action The probability of Indicates The escapee is observed locally Next, according to the previous strategy take The probability of Indicates Batch data; S3.2, PPO algorithm uses generalized advantage estimation GAE, by introducing a parameter , used to smooth the advantage estimate in time; ； in, Indicates The timing difference error of the escapees, For the Runaway time steps Instant rewards, is the escapee discount factor, which is the same for all escapees; , Respectively represent A fugitive status , The value function of represents the offset or delay of the time step; Indicates escapee from time The weighted summation of the timing difference error from the beginning to the future; when >0, indicating the Agents in state The reward of taking the current action is higher than the average, otherwise Agents in state The reward for taking the current action is below average; S3.3, PPO introduces an alternative optimization objective function to represent the original objective function of the Actor network. Its optimization objective function is: ； in Represents the batch data size, is the number of escapees, Representative The first batch of data The advantage of an escapee is estimated, For the The probability ratio of the new and old strategies of the escapees, Represents the clipping range, is the entropy regularization coefficient, It is The escapees observe locally according to their strategies By maximizing the entropy , the Actor can improve in the direction of maximizing the advantage function; S3.4, Actor network parameter gradient update method is as follows: ； in, For the The Actor network parameters of the escapee, is the learning rate of the escapees, and the learning rate of each escapee is the same, is the Actor network loss function with respect to the parameters The gradient of S3.

5. For the Critic network in the PPO algorithm, its goal is to minimize the mean square error between the predicted value and the actual return. Its loss function and network parameter update are calculated as follows: ； in, For the escapees to the global state The estimated value of For the The critic network parameters of the escapees, is the batch data size, For the The actual reward obtained by the escapee in each round is is the learning rate; S3.

6. The reward function design for the escaper is the same as that for the pursuer. The collaboration reward and pre-capture reward are cancelled, and the capture reward is If the escapee has been captured, according to the actual situation, he should not continue to participate in the escape process, so that the escapee will not move after being captured, and will always receive the capture reward; The relative distance constraint rewards for escapers are as follows: ； in, is the escaper distance reward coefficient, Represents the minimum distance between all pursuers and the escaper.

Images