Multi-agent learning confrontation hunting method based on intrinsic motivation and trajectory prediction

Abstract

The embodiment of the invention provides a multi-agent learning confrontation hunting method based on internal motivation and trajectory prediction, and the method comprises the steps: constructing a multi-agent confrontation hunting environment model, and constructing a square plane region as a multi-agent confrontation region which comprises two types of agents of a pursuit party and an escape party; designing the state and action space of the intelligent agent, wherein the state space of the intelligent agent is a total set of all observable states of the intelligent agent; the action space is a total set of all actions which can be selected by the intelligent agent, is designed to be in a continuous state, and is defined as acceleration values in different directions of static, left movement, right movement, up movement and down movement; designing a reward function based on the internal motivation of knowledge, and obtaining a total reward for confrontation and enrolment of the intelligent agent through comprehensive rewards; and pre-judging an activity range of an enemy agent based on a trajectory prediction method of extended Kalman filtering, training the agent based on trajectory prediction until network parameters converge, and ending the training.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-agent confrontation encirclement, and particularly relates to a multi-agent learning confrontation encirclement method based on intrinsic motivation and trajectory prediction. BACKGROUND

[0002] Multi-agent confrontation encirclement is a classic problem in multi-agent systems, which involves the cooperation and confrontation among multiple unmanned agents. The agent confrontation encirclement strategy has applications in military aspects (tank, warplane cluster encirclement attack, missile defense system defense interception of external threats), civil aspects (most ball games, animal cooperative predation), and is a research problem that is concerned in military and civil fields. The basic algorithms of multi-agent confrontation encirclement strategy include rule-based finite state machine, behavior tree, expert system method, etc., which all need to model the environment to a certain extent, and can only adapt to discrete finite state space. In order to adapt to complex environment and state space, multi-agent reinforcement learning technology has become one of the main development directions of multi-agent confrontation encirclement.

[0003] The traditional multi-agent reinforcement learning technology aims at the encirclement confrontation environment, and has the problems of difficulty in convergence and low learning efficiency caused by continuous state and action space, sparse reward return, etc. Only using expert guide knowledge to assist agent training may cause some invisible restrictions in the early training, so that the exploration space of the agent is narrow, and the decision network model finally converges to a local optimal solution. Therefore, on the basis of expert guide knowledge, integrating multiple intrinsic motivations to further refine the reinforcement learning reward function is an effective solution to solve the reward sparsity problem and break the invisible restrictions. At the same time, the traditional multi-agent reinforcement learning technology only focuses on the current state information of the agent, and only reacts according to the current state, so the training speed and decision correctness are limited. Therefore, combined with the trajectory prediction technology, the future motion trajectory of the enemy is considered in advance, which can upgrade the encirclement method to a “pre-judgment type”, speed up the training speed, and enhance the decision correctness. SUMMARY

[0004] In view of the aforementioned problems in the prior art, this invention provides a multi-agent reinforcement learning adversarial trapping method based on intrinsic motivation and trajectory prediction. The method implements a multi-agent reinforcement learning adversarial trapping model based on intrinsic motivation and trajectory prediction. It reconstructs the reward function using knowledge-based intrinsic motivation (including artificial potential fields, out-of-bounds penalties, and other intrinsic motivations) to obtain the process-time reward at each decision time step, which is used to evaluate the quality of the action output. An appropriate trajectory prediction range is selected based on the distance between agents in the field. By introducing the trajectory prediction process, the input information of the decision network is changed, transforming the agent's action output from "reactive" to "predictive," effectively alleviating the reward sparsity problem in multi-agent reinforcement learning training for adversarial trapping tasks and improving training efficiency and decision accuracy.

[0005] This invention provides a multi-agent learning adversarial trapping method based on intrinsic motivation and trajectory prediction, comprising:

[0006] Step 1: Construct a multi-agent adversarial encirclement environment model. Construct a set area as a multi-agent adversarial area, including two types of agents: pursuers and escapees. Set the pursuers' task as completing the encirclement and capture of all escapees.

[0007] Step 2: Design the agent's state and action space. The agent's state space is the total set of all observable states of the agent, defined as position, velocity, and attitude information. The action space is the total set of all actions that the agent can select. It is designed as a continuous state, and acceleration is selected as the physical meaning. The action space is defined as the acceleration values ​​in different directions such as stationary, leftward, rightward, upward, and downward, to represent the changes in velocity direction and magnitude.

[0008] Step 3: Based on the intrinsic motivation of knowledge, design a reward function. The reward function design is divided into four types: collision reward, expert guidance reward, out-of-bounds reward, and artificial potential field reward. By combining the above four types of rewards, the total reward for the agent's adversarial encirclement is obtained.

[0009] Step 4: Use the trajectory prediction method based on extended Kalman filtering to predict the activity range of the enemy agent and use it as new state information. Then, train the agent based on trajectory prediction until the network parameters converge, and end the training.

[0010] In some embodiments of the present invention, step 1, establishing the multi-agent adversarial trapping environment model includes:

[0011] In an environment Env, there are multiple agents, of which n agents are pursuers and m agents are fleeing agents, represented by a set H = {h1, h2, ..., h...}. n Let P = {p1, p2, ..., pn} represent n pursuing agents. m} represents m fugitive agents;

[0012] The task of the pursuer is to complete the task of capturing the fugitive in the confrontation area;

[0013] The final condition for the completion of the pursuer task is that all agents successfully pursue, that is, the final task of the pursuer.

[0014] In some embodiments of the application, in step 2, the state space is designed, including:

[0015] According to the established multi-agent confrontation and capture environment model, the state space of the agent is defined as:

[0016]

[0017] Where X represents the state information of the agent, p represents the position information of the agent, v represents the speed information of the agent, represents the attitude information of the agent.

[0018] In some embodiments of the application, in step 2, the action space is designed, including:

[0019] The moving agent is regarded as a planar object, and the height size is ignored;

[0020] At each time step, the agent selects actions through its own decision network;

[0021] The action space is the total set of all actions that the agent can select, and in this environment, the action is designed as a continuous state, and all locations in the confrontation area can be reached by the agent. The task of the agent is to pursue the enemy agent. Each action in the environment is determined by the speed direction and size of the agent. Therefore, the acceleration is designed as the physical meaning of the action space, and the size of the acceleration in different directions is output to represent the change of the speed direction and size. The specific form is: the decision network outputs a vector of length 5,

[0022] A={a stay ,a left ,a right ,a up ,a down}

[0023] The above A represents the action space of the agent, where the five elements respectively represent the acceleration size of static, left movement, right movement, up movement and down movement. The size of the element in the vector is limited in the range of 0-1. The final acceleration of the agent is obtained by vector addition of the elements in the vector.

[0024] In some embodiments of the application, in step 3,

[0025] The collision reward is an external motivation, which gives a reward when the agent as a pursuer successfully collides with the enemy agent, or gives a penalty when the agent collides with the ally agent, guides the agent to surround the enemy agent and avoid colliding with the ally agent;

[0026] The expert guidance reward is an internal motivation, which gives a reward when the agent action decision value is close to the expert guidance decision value, and gives a penalty when the agent action decision value is not close to the expert guidance decision value, guides the agent to quickly make correct actions;

[0027] The out-of-bound reward is an internal motivation, which gives a penalty when the agent is about to go out of the bound or has gone out of the bound, guides the agent to move within the confrontation area;

[0028] The artificial potential field reward is an internal motivation, which regards the surrounding environment information of the agent as an energy field, regards the enemy agent as a target point, regards the ally agent as an obstacle, the target point has an attractive force to the agent, and the obstacle has a repulsive force to the agent, the artificial potential field reward is obtained by calculating the attractive and repulsive potential fields and the attractive and repulsive forces generated by the target point and the obstacle on the agent, guides the agent to move towards the enemy agent and away from the ally agent to avoid collision.

[0029] In some embodiments of the present application, in step 3, the four rewards are designed as follows:

[0030] For the collision reward, the collision reward value of the agent with the enemy agent or with the ally agent is designed, if the distance between the agent and the enemy agent is less than d goal ;

[0031] d goal is the sum of the radii of the agent and the enemy agent, that is, a reward value of 10 is obtained when the enemy agent is successfully collided with;

[0032] If the distance between the agent and the ally agent is less than d obs ;

[0033] d obs is the sum of the radii of the agent and the ally agent, that is, a penalty value of -2 is obtained when the agent collides with the ally agent, and the collision penalty is an external motivation feedback value given by the environment.

[0034] For expert guidance reward, introduce expert guidance knowledge dataset, take expert guidance knowledge as one of the intrinsic motivations, give positive reward when the decision of agent and expert knowledge has set degree similarity, reduce invalid search of training network; give a positive reward when the distance angle between decision value output by decision network of agent and expert guidance decision value is similar, give negative reward when the angle exceeds set value, which means the action given by decision network is wrong or even opposite direction, wherein, decision value is action value; expressed as follows:

[0035] r1 = η if |A * -A|<τ

[0036] r1 = -η if |A * -A|≥τ

[0037] Wherein, η represents the reward value given by expert guidance, A * represents the action value output by current decision network of agent, A represents the decision value of expert guidance, |A * -A| represents the angle difference of two action decisions, τ represents the threshold value set according to the situation;

[0038] For out-of-bound reward, design out-of-bound reward as another intrinsic motivation, give appropriate negative reward to agent according to the distance of agent approaching or crossing the boundary, to urge it to move in the expected area; expressed as follows:

[0039]

[0040] Wherein, x is the horizontal coordinate of current position coordinate (x, y) of agent, l is the radius of square confrontation area, assuming that the square confrontation area is the area with horizontal and vertical axis range of ( -l ,+l); similarly, the vertical coordinate y of agent is also given reward according to the above formula; if the agent is outside the boundary, give exponentially increasing negative reward, and set the maximum value of negative reward within 10; if the agent is inside the boundary but the distance from the boundary is within 10% of the boundary value, give negative reward proportional to the distance; if the agent is in other positions, do not give reward;

[0041] For artificial potential field reward, the environmental information around the agent is regarded as an energy field, in the process of agent movement, the target point has attractive force to the agent, and the obstacle has repulsive force to the agent; the target point is the point with lowest potential energy in the environment, the movement of agent is always from the place with high potential energy to the place with low potential energy, finally the agent will reach the target point under the traction of potential field, one of the intrinsic motivations given to the agent is to hope that the agent can approach the enemy agent, i.e. the target point, at the same time, the teammates of the agent are regarded as obstacles in the environment, the reward value obtained based on the above intrinsic motivation is:

[0042]

[0043] wherein, U att (q) is the attractive potential field of agent j to agent i at position q, a is the proportional coefficient of the attractive field function, d ij is the distance between agent i and agent j, when the Euclidean distance is adopted, then m can be 2;

[0044]

[0045] wherein, U rep (q) is the repulsive potential field of agent j to agent i at position q, k is the proportional coefficient of the repulsive field function, d0 is the critical value of the repulsive field function, if the distance between agent i and agent j is less than d0, the repulsive field works, and if the distance is greater than the critical distance, the repulsive field does not work;

[0046]

[0047] wherein, F att (q) is the attractive potential field of agent j to agent i at position q, a is the proportional coefficient of the attractive field function, d

[0048]

[0049] wherein, F rep (q) is the attractive potential field of agent j to agent i at position q, a is the proportional coefficient of the attractive field function, d

[0050] The final reward value of the artificial potential field is:

[0051] r3 = U att (q) + F att (q) + U rep (q) + F rep (q)

[0052] Through the attraction and repulsion, the agent is guided to move in the direction of the enemy agent, and the agent is informed to move away from the friendly agent to avoid unnecessary casualties caused by collision;

[0053] The total reward of the confrontation is as follows:

[0054]

[0055] wherein, r(d) = r1 + r2 + r3, that is, the sum of the expert guidance reward, the out-of-bound reward and the artificial potential field reward, is the reward function based on the intrinsic motivation given to the agent;

[0056] The absolute value of the maximum value of the reward function based on intrinsic motivation is smaller than the order of magnitude of the reward value of the final task. The reward value for the agent based on the above design to successfully collide with the enemy agent in the final task is 10. Therefore, the absolute value of the maximum value of the reward function based on intrinsic motivation is chosen to be 1, that is, |r(d)|<1.

[0057] In some embodiments of the present invention, in step 4,

[0058] Taking unmanned vehicles as an example, we establish the motion equation and observation equation of the enemy intelligent agent, and use the extended Kalman filter algorithm to predict the trajectory range of the enemy intelligent agent in the next time step based on the current observation information of the enemy intelligent agent.

[0059] The design employs a secondary correction method for the predicted trajectory range. This method involves weighting the enemy agent's current actual position with its predicted position. The weights are proportional to the distance between the friendly and enemy agents. This results in a larger predicted trajectory range when the two agents are far apart, guiding the friendly agent to move in the general direction of the enemy agent. Conversely, a smaller predicted trajectory range when the two agents are close together, allowing the friendly agent to accurately move towards and capture the enemy agent's position.

[0060] In some embodiments of the present invention, step 4 of the method further includes:

[0061] Using the MADDPG algorithm as the basic framework for reinforcement learning, and combining it with trajectory prediction methods, the decision network and evaluation network of the friendly agent (as the pursuer) are trained, enabling the friendly agent to effectively cooperate in capturing the enemy agent.

[0062] Among them, the enemy agent's position prediction value obtained through trajectory prediction and secondary correction will replace the friendly agent's current actual observation position information of the enemy agent, and combined with the predicted velocity information, it will be used as part of the input of the friendly agent's decision network to train the decision network.

[0063] For each agent's action decision at each time step, an immediate evaluation is performed using a knowledge-based intrinsic motivation reward function.

[0064] In some embodiments of the present invention, step 4, trajectory prediction based on extended Kalman filtering, includes:

[0065] The trajectory prediction method based on extended Kalman filtering is used to predict the activity range of the enemy agent and use it as new state information to replace the real-time observed current enemy agent state information {ρ,v}. This new state information is then used as the new input value for the agent decision network to train the agent decision network.

[0066] Since the enemy agent's position in the agent's field of vision changes from a precise point to a range at each time step, it will interfere with the agent's decision-making in the later stages of the pursuit; the design performs secondary trajectory correction based on the different distances between the two agents.

[0067] When the agent is an autonomous vehicle, the state information required by the extended Kalman filter algorithm is the already calculated accurate speed and position information, and the observation information is the direct information obtained by the agent through the sensors, namely speed, position and attitude angle.

[0068] Enemy agent state information:

[0069] x t =x(t)=[p x ,p y ,v x ,v y ] T

[0070] Where, p x p y For the location information of the enemy intelligent agent, ρ t =ρ(t)=[p x ,p y ] T v x v y For the enemy agent's speed information, v t =v(t)=[v x ,v y ] T ;

[0071] Establish the motion equations of the enemy intelligent agent:

[0072]

[0073] Convert to matrix form:

[0074]

[0075] Where dt is the sampling period, for example, if dt = 0.1s, a x a y F represents the magnitude of the agent's lateral and longitudinal accelerations. t Let be the Jacobian matrix of the state equation;

[0076] Establish the enemy agent observation equation:

[0077]

[0078] The measurement matrix H is:

[0079]

[0080] After establishing the enemy agent's motion equations, observation equations, and measurement matrix, the extended Kalman filter algorithm is used to predict the enemy agent's state at the next time step; specifically, this includes:

[0081] Initialize enemy agent state information x t Covariance matrix P t Obtain the observed value z t ;

[0082] A prediction: x t Substituting into the equation of motion, we can calculate... Depend on Predict the covariance matrix, where Q is the process noise covariance;

[0083] Secondary prediction: Substituting into the measurement matrix formula, we get H t+1 Substituting these values ​​into the observation equation, the predicted observation values ​​are obtained. Therefore, calculate the Kalman gain. Where R is the measurement noise covariance, and the state prediction value of the enemy agent at the next time step is further calculated. and covariance matrix

[0084] The trajectory is corrected a second time based on the different distances between the two agents, and the range of trajectory prediction is adjusted.

[0085] The weight of the enemy agent's actual position is increased, while the weight of the predicted position is decreased, thus altering the predicted trajectory range. The weight ω is proportional to the distance between the friendly and enemy agents. The position prediction value after this second correction is:

[0086]

[0087] Where ρ(t) is the current actual position of the enemy agent, and ρ′(t+1) is the predicted position of the enemy agent obtained through extended Kalman filtering, which is included in the calculated predicted state value x of the enemy agent. t+1 middle, The second-corrected position prediction value obtained through two weighted operations is used to replace the current actual observed position information of the enemy agent by the friendly agent, combined with x. t+1 The predicted speed information is used as part of the input to the decision network of the agent to train the decision network;

[0088] As the distance between friendly and enemy agents decreases, the weight of the position prediction value gradually decreases, that is, the range of trajectory prediction gradually narrows, so as to move in the general direction when the distance is far and to accurately capture when the distance is close.

[0089] In some embodiments of the present invention, step 4, the agent training based on trajectory prediction includes:

[0090] Step 41: Initialize the decision network π for each friendly agent. i With evaluation network Q i , i∈{1,…,n}, initialize the environment scene;

[0091] Step 42: Obtain the predicted trajectory of the enemy agent through extended Kalman filtering. After secondary correction, replace the observation value of the enemy agent by the friendly agent with the observation value of the friendly agent. Use this as part of the input of the friendly agent's decision network and output the friendly agent's action value for the next time step.

[0092] Step 43: Each friendly agent executes its actions, and the environment progresses;

[0093] Step 44: Calculate and store the immediate reward value of the agent's output action decision at the current time step using a knowledge-based intrinsic motivation reward function;

[0094] Step 45: Store a sufficient number of samples (s) t ,a t ,s t+1 ,r t ), where s t Let a be the environment state vector at time step t. t Let r be the joint action vector of each friendly agent at time step t. t After providing each agent with an instant reward value at time step t, random samples are taken from the sample. Using the sampled batch samples, the gradient descent method is used to update the evaluation network parameters of each agent with the goal of minimizing the time difference error. Then, the decision network parameters are updated with the goal of maximizing the evaluation value.

[0095] Step 46: Repeat steps 42-45 above until the network parameters converge, and end the training.

[0096] Compared with existing technologies, the beneficial effects of the multi-agent reinforcement learning adversarial trapping method based on intrinsic motivation and trajectory prediction provided in this invention are as follows:

[0097] 1) This invention employs a knowledge-based intrinsic motivation reward function, utilizing artificial potential fields, expert guidance, and out-of-bounds penalties with regional restrictions to enhance the agent's intrinsic motivation. This further refines the reward function of reinforcement learning, avoiding the problem of sparse rewards due to poor strategies in the early stages of training. Compared to existing technologies that solely rely on expert guidance as intrinsic motivation rewards, which may lead to some implicit limitations in the early stages of training, narrowing the agent's exploration space and causing the decision training model to eventually converge to a local optimum, this invention uses knowledge-based intrinsic motivation to add multiple rewards, breaking these limitations and enabling the agent to converge to the optimal decision network better and faster.

[0098] 2) This invention employs a combination of trajectory prediction and reinforcement learning. When agents are conducting encirclement operations, the future trajectory of the enemy is considered in advance. Based on the extended Kalman filter algorithm, the predicted state of the enemy agent at the next time step is calculated. Through secondary correction, the predicted value is adjusted according to the distance between the agent and the enemy agent. Finally, the corrected state prediction value replaces the actual observation of the enemy agent and is used as the input to the agent's decision network for training. Compared with traditional multi-agent reinforcement learning methods, this method transforms the encirclement approach from "reactive" to "predictive," accelerating network training, increasing the accuracy of decisions, and making the encirclement strategy more intelligent. Attached Figure Description

[0099] Figure 1 A flowchart of a multi-agent reinforcement learning adversarial trapping method based on intrinsic motivation and trajectory prediction provided in an embodiment of the present invention;

[0100] Figure 2(a) is a schematic diagram of the encirclement process in the multi-agent adversarial encirclement environment model of the multi-agent reinforcement learning adversarial encirclement method based on intrinsic motivation and trajectory prediction provided in the embodiment of the present invention;

[0101] Figure 2(b) is a schematic diagram of the successful encirclement of the multi-agent adversarial encirclement environment model in the multi-agent reinforcement learning adversarial encirclement method based on intrinsic motivation and trajectory prediction provided in the embodiment of the present invention.

[0102] Figure 3 A flowchart of target trajectory prediction based on extended Kalman filtering in a multi-agent reinforcement learning adversarial trapping method based on intrinsic motivation and trajectory prediction provided in an embodiment of the present invention.

[0103] Figure 4(a) is a schematic diagram of the original process of agent training in the multi-agent reinforcement learning adversarial trapping method based on intrinsic motivation and trajectory prediction provided in the embodiment of the present invention.

[0104] Figure 4(b) is a schematic diagram of agent training after adding the trajectory prediction process in the multi-agent reinforcement learning adversarial trapping method based on intrinsic motivation and trajectory prediction provided in the embodiment of the present invention. Detailed Implementation

[0105] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0106] Various embodiments and features of this application are described herein with reference to the accompanying drawings.

[0107] These and other features of this application will become apparent from the following description of preferred forms of embodiments given as non-limiting examples, with reference to the accompanying drawings.

[0108] It should also be understood that although this application has been described with reference to some specific examples, those skilled in the art can certainly implement many other equivalent forms of this application, which have the features described in the claims and are therefore all within the scope of protection defined herein.

[0109] The above and other aspects, features and advantages of this application will become more apparent when taken in conjunction with the accompanying drawings and in view of the following detailed description.

[0110] Specific embodiments of this application are described below with reference to the accompanying drawings; however, it should be understood that the claimed embodiments are merely examples of this application, which can be implemented in various ways. Well-known and / or repeated functions and structures are not described in detail to ascertain the true intent based on the user's historical operations, and to avoid unnecessary or redundant details that would obscure this application. Therefore, the specific structural and functional details claimed herein are not intended to be limiting, but merely serve as the basis and representative basis for the claims to teach those skilled in the art to use this application in various ways with substantially any suitable detailed structure.

[0111] This specification may use the phrases “in one embodiment,” “in another embodiment,” “in yet another embodiment,” or “in other embodiments,” all of which may refer to one or more of the same or different embodiments according to this application.

[0112] This invention provides a multi-agent learning adversarial trapping method based on intrinsic motivation and trajectory prediction, such as... Figure 1 Figure 4 includes:

[0113] Step 1: Construct a multi-agent adversarial encirclement environment model. Construct a set area as a multi-agent adversarial area, including two types of agents: pursuers and escapees. Set the pursuers' task to complete the encirclement and capture of all escapees. The set area can be a circular area, a rectangular area, or a fan-shaped area, etc., based on the set conditions.

[0114] Step 2: Design the agent's state and action space. The agent's state space is the total set of all observable states of the agent, defined as position, velocity, and attitude information. The action space is the total set of all actions that the agent can select. It is designed as a continuous state, and acceleration is selected as the physical meaning. The action space is defined as the acceleration values ​​in different directions such as stationary, leftward, rightward, upward, and downward, to represent the changes in velocity direction and magnitude.

[0115] Step 3: Based on the intrinsic motivation of knowledge, design a reward function. The reward function design is divided into four types: collision reward, expert guidance reward, out-of-bounds reward, and artificial potential field reward. By combining the above four types of rewards, the total reward for the agent's adversarial encirclement is obtained.

[0116] Step 4: Use the trajectory prediction method based on extended Kalman filtering to predict the activity range of the enemy agent and use it as new state information. Then, train the agent based on trajectory prediction until the network parameters converge, and end the training.

[0117] To facilitate understanding of the above technical solutions, a detailed explanation is provided below with reference to specific examples and accompanying drawings:

[0118] The overall flowchart of the multi-agent reinforcement learning adversarial trapping method based on intrinsic motivation and trajectory prediction can be found in the attached diagram. Figure 1 The specific steps can be described as follows:

[0119] 1) Construct a multi-agent adversarial encirclement environment model.

[0120] The multi-agent adversarial encirclement environment model is established as follows: In the environment Env, there are multiple agents, where n agents are the pursuers and m agents are the fleeing agents. The set H = {h1, h2, ..., h...} is used. n Let P = {p1, p2, ..., pn} represent n pursuing agents. m Let} represent m fleeing agents. A schematic diagram of the multi-agent adversarial encirclement environment model is shown in Figure 2(a). The diagram includes 3 pursuers (square icons) and 2 fleeing agents (circular icons). The task of the 3 pursuers is to encircle and capture the 2 fleeing agents. The square area represents the multi-agent adversarial area. The final condition for the pursuers to complete their task is that all agents successfully pursue the fleeing agents; this is the final task of the pursuers, as shown in Figure 2(b).

[0121] 2) Design of agent state and action space.

[0122] (1) State-space design

[0123] Based on the established multi-agent adversarial trapping environment model, the state space of the agents can be defined as:

[0124]

[0125] Where X represents the agent's state information, ρ represents the agent's position information, and v represents the agent's velocity information. This represents the pose information of the intelligent agent.

[0126] (2) Action space design

[0127] The moving agent is treated as a planar object, ignoring height and size. At each time step, the agent selects actions through its own decision network (Actor). The action space is the total set of all actions the agent can choose. The action space design must be complete, ensuring the agent can achieve its intended goal and avoiding inaccessible blind spots. In this environment, actions are designed as continuous, and the agent can reach all locations in the adversarial region. The agent's task is to pursue the enemy agent. Every action in the environment is determined by the agent's velocity direction and magnitude. Therefore, acceleration is chosen as the physical meaning of the action space, and the magnitude of acceleration in different directions is output to represent changes in velocity direction and magnitude. Specifically, the decision network outputs a vector of length 5.

[0128] A={a stay ,a left ,a right ,a up ,a down}

[0129] In the above, A represents the agent's action space, where the five elements represent the magnitudes of acceleration for stationary, leftward, rightward, upward, and downward movements, respectively. The magnitudes of the elements in the vector are limited to the range of 0-1, and the agent's final acceleration is obtained by vectorially summing the elements in the vector.

[0130] 3) Design of knowledge-based intrinsic motivation reward function.

[0131] In the initial stage of agent training, although the agent can extract environmental information, the actual task rewards are relatively sparse, making it impossible to judge the quality of the current environmental information and thus unable to improve the current training network (which includes decision network and evaluation network) based on the rewards. Therefore, the design introduces the concept of artificial potential field in the early stage of training, using the potential field function as the intrinsic motivation to transform sparse rewards into dense rewards, and simultaneously using expert knowledge to guide the agent's actions.

[0132] (1) Collision Rewards

[0133] The design determines the collision reward value for an agent when it collides with an enemy agent or with an allied agent. If the distance between the agent and the enemy agent is less than d... goal(The sum of the radii of the agent and the enemy agent) equals a successful collision with the enemy agent, granting a reward of 10. If the distance between the agent and the friendly agent is less than d... obs The sum of the radii of the agent and the friendly agent (the sum of the radii) results in a -2 reward / penalty value upon collision with a friendly agent. This collision reward / penalty is an external motivational feedback value provided by the environment.

[0134] (2) Expert guidance reward

[0135] An expert-guided knowledge dataset is introduced, using expert-guided knowledge as one of the intrinsic motivations. Essentially, expert strategies and optimal decision-making strategies are consistent; therefore, when the agent's decisions and expert knowledge have a certain degree of similarity, positive rewards can be given to reduce ineffective searches during network training. A positive reward is given when the distance angle between the agent's decision network output decision value (action value) and the expert-guided decision value is close. When the angle exceeds a certain value, indicating that the decision network's action is incorrect or even in the opposite direction, a negative reward is given.

[0136] r1=η if|A * -A|<τ

[0137] r1=-η if|A * -A|≥τ

[0138] Where η represents the reward value given by the expert guidance, and A * Let A represent the action value output by the decision network of the agent, and let A represent the expert-guided decision value. * -A| represents the angle difference between the two action decisions, and τ represents the threshold set according to the situation.

[0139] (3) Out-of-bounds reward

[0140] Designing an out-of-bounds reward as another intrinsic motivation aims to ensure that the agent only moves within the adversarial region and does not deviate from it. The out-of-bounds reward involves providing the agent with an appropriate negative reward based on the distance it approaches or crosses the boundary, thus encouraging it to operate within the desired region.

[0141]

[0142] Where x is the abscissa of the agent's current position (x, y), and l is the radius of the square adversarial region, assuming that the square adversarial region is within the range of both the abscissa and ordinate (x, y). -lThe region is defined as , +l). Similarly, the agent's vertical coordinate y is also rewarded according to the above formula. If the agent is outside the boundary, an exponentially increasing negative reward is given, with the maximum negative reward set to within 10; if the agent is inside the boundary but within 10% of the boundary value, a negative reward proportional to the distance is given; no reward is given when the agent is in other positions.

[0143] (4) Artificial potential field reward

[0144] By introducing the concept of an artificial potential field, we can endow the agent with another intrinsic motivation. The basic idea of ​​an artificial potential field is to treat the environmental information surrounding the agent as an energy field. During the agent's movement, the target point attracts the agent, while obstacles repel it. The target point is the point with the lowest potential energy in the environment. The agent's movement always proceeds from areas of high potential energy to areas of low potential energy, ultimately reaching the target point under the pull of the potential field. Borrowing from the basic idea of ​​an artificial potential field, we can endow the agent with an intrinsic motivation to approach the enemy agent (i.e., the target point), while simultaneously treating friendly agents as obstacles in the environment.

[0145] The reward value obtained based on the above intrinsic motivation is:

[0146]

[0147] Among them, U att (q) is the attractive potential field of agent j towards agent i at position q, α is the scaling factor of the gravitational field function, and d ij Let m be the distance between agent i and agent j. When using Euclidean distance, m can be 2.

[0148]

[0149] Among them, U rep (q) is the repulsive potential field of agent i to agent j when i is at position q, k is the proportionality coefficient of the repulsive field function, and d0 is the critical value at which the repulsive field function takes effect. If the distance between agent i and agent j is less than d0, the repulsive field takes effect; if the distance is greater than the critical distance, it does not take effect.

[0150]

[0151] Among them, F att (q) is obtained by the negative gradient of the attractive potential field, which is the attraction of agent j to agent i when agent i is at position q.

[0152]

[0153] Among them, F rep(q) is obtained by the negative gradient of the repulsive potential field, which is the repulsive force of agent j on agent i when agent i is at position q.

[0154] The final reward value of the artificial potential field is:

[0155] r3=U att (q)+F att (q)+U rep (q)+F rep (q)

[0156] Using attraction and repulsion, the agent is guided to move towards the enemy agent, and informed to stay away from friendly agents to avoid collisions and unnecessary casualties.

[0157] (5) Total Rewards for Countering Encirclement and Capture

[0158] Based on the above-mentioned collision reward, expert guidance reward, out-of-bounds reward, and artificial potential field reward, the total reward function of the agent is obtained as follows:

[0159]

[0160] Where r(d) = r1 + r2 + r3, which is the sum of expert-guided reward, out-of-bounds reward, and artificial potential field reward, is the intrinsic motivation-based reward function given to the agent, focusing on the agent's execution process in the encirclement task. Furthermore, to avoid local optimization during the encirclement task leading to excessively large cumulative reward values ​​and misjudgments by the agent that prevent the formation of an optimal strategy, the absolute value of the maximum value of the intrinsic motivation-based reward function should be smaller than the order of magnitude of the final task reward value. Since the reward value for the agent in the above design to successfully collide with the enemy agent in the final task is 10, the absolute value of the maximum value of the intrinsic motivation-based reward function is chosen to be 1, i.e., |r(d)| < 1.

[0161] 4) Training of agents based on trajectory prediction.

[0162] (1) Trajectory prediction based on extended Kalman filter

[0163] When agents conduct encirclement operations, considering the future movement trajectory of the enemy agent in advance can transform the encirclement method from "reactive" to "predictive." A trajectory prediction method based on extended Kalman filtering is used to predict the enemy agent's activity range and use this as new state information, replacing the real-time observed current enemy agent state information {ρ,v}, as the new input value for the agent's decision network, which is then used to train the agent's decision network. However, due to trajectory prediction, the enemy agent's position in the agent's field of vision at each time step changes from a precise point to a range, which can interfere with the agent's decision-making in the later stages of the pursuit. To enable the agent to capture the enemy agent more quickly and accurately in the later stages of the pursuit, a secondary trajectory correction is designed based on the different distances between the two agents. The encirclement target trajectory prediction process based on extended Kalman filtering is shown in the appendix. Figure 3 .

[0164] Taking an autonomous vehicle as an example, the state information required by the extended Kalman filter algorithm is selected as the accurate speed and position information that has already been solved, and the observation information is the direct information obtained by the intelligent agent through the sensor, namely speed, position and attitude angle.

[0165] Enemy agent state information:

[0166] x t =x(t)=[p x ,p y ,v x ,v y ] T

[0167] Where, p x p y For the location information of the enemy intelligent agent, ρ t =ρ(t)=[p x ,p y ] T v x v y For the enemy agent's speed information, v t =v(t)=[v x ,v y ] T .

[0168] Establish the motion equations of the enemy intelligent agent:

[0169]

[0170] Convert to matrix form:

[0171]

[0172] Where dt is the sampling period, for example, if dt = 0.1s, ax a y F represents the magnitude of the agent's lateral and longitudinal accelerations. t Let be the Jacobian matrix of the state equation.

[0173] Establish the enemy agent observation equation:

[0174]

[0175] The measurement matrix H is:

[0176]

[0177] After establishing the motion equations, observation equations, and measurement matrix of the enemy agent, the extended Kalman filter algorithm can be used to predict the state prediction value of the enemy agent at the next time step.

[0178] ① Initialize enemy agent state information x t Covariance matrix P t Obtain the observed value z t ;

[0179] ② First prediction: x t Substituting into the equation of motion, we can calculate... Depend on Predict the covariance matrix, where Q is the process noise covariance;

[0180] ③ Secondary prediction: Substituting into the measurement matrix formula, we get H t+1 Substituting these values ​​into the observation equation, the predicted observation values ​​are obtained. Therefore, the Kalman gain can be calculated. Where R is the measurement noise covariance, and the state prediction value of the enemy agent at the next time step is further calculated. and covariance matrix

[0181] In the process of combating encirclement and capture, the intelligent agent has two main objectives: First, when the enemy intelligent agent is far away, it hopes to move in the general direction of the enemy intelligent agent to lay the groundwork for capturing the enemy intelligent agent later; second, when the enemy intelligent agent is close or even within the capture range, it needs to accurately move towards the location of the enemy intelligent agent and capture it. Therefore, it is necessary to perform secondary trajectory correction based on the different distances between the two intelligent agents to adjust the range of trajectory prediction.

[0182] The predicted trajectory range is altered by increasing the weight of the enemy agent's actual position and decreasing the weight of the predicted position. The weight ω is proportional to the distance between the friendly and enemy agents. The predicted position value after secondary correction is:

[0183]

[0184] Where ρ(t) is the current actual position of the enemy agent, and ρ′(t+1) is the predicted position of the enemy agent obtained through extended Kalman filtering, which is included in the calculated predicted state value x of the enemy agent. t+1 middle, The second-corrected position prediction value obtained through two weighted operations is used to replace the current actual observed position information of the enemy agent by the friendly agent, combined with x. t+1 The predicted velocity information is used as part of the input to the decision network of the friendly agent to train the network. As the distance between the friendly agent and the enemy agent decreases, the weight of the position prediction value gradually decreases, that is, the range of trajectory prediction gradually narrows, so as to move in the general direction when the distance is far and to accurately capture when the distance is close.

[0185] (2) Training of agents based on trajectory prediction

[0186] The basic framework for agent training combines the MADDPG algorithm with trajectory prediction. The training process is illustrated in Figures 4(a) and 4(b). The training steps are as follows:

[0187] ① Initialize the decision network π for each friendly agent. i With evaluation network Q i , i∈{1,…,n}, initialize the environment scene;

[0188] ② Obtain the predicted trajectory of the enemy agent through extended Kalman filtering. After secondary correction, replace the observation value of the enemy agent by the friendly agent with the observation value of the friendly agent. Use it as part of the input of the friendly agent's decision network and output the friendly agent's action value for the next time step.

[0189] ③ Each agent executes its own actions, and the environment progresses;

[0190] ④ Calculate and store the immediate reward value of the agent's output action decision at the current time step using a knowledge-based intrinsic motivation reward function;

[0191] ⑤ Store a sufficient number of samples (s) t ,a t ,s t+1 ,r t (where s) t Let a be the environment state vector at time step t. t Let r be the joint action vector of each friendly agent at time step t. tAfter providing the instantaneous reward value for each agent at time step t, random samples are taken from the sample. Using the sampled batch of samples, the gradient descent method is used to update the evaluation network parameters of each agent with the goal of minimizing the time difference error, and then the decision network parameters are updated with the goal of maximizing the evaluation value.

[0192] ⑥ Repeat steps (2-5) above until the network parameters converge, then end the training.

[0193] As can be seen from the above technical solutions, the beneficial effects of the multi-agent reinforcement learning adversarial trapping method based on intrinsic motivation and trajectory prediction provided by the above embodiments of the present invention are as follows:

[0194] 1) This invention employs a knowledge-based intrinsic motivation reward function, utilizing artificial potential fields, expert guidance, and out-of-bounds penalties with regional restrictions to enhance the agent's intrinsic motivation. This further refines the reward function of reinforcement learning, avoiding the problem of sparse rewards due to poor strategies in the early stages of training. Compared to existing technologies that solely rely on expert guidance as intrinsic motivation rewards, which may lead to some implicit limitations in the early stages of training, narrowing the agent's exploration space and causing the decision training model to eventually converge to a local optimum, this invention uses knowledge-based intrinsic motivation to add multiple rewards, breaking these limitations and enabling the agent to converge to the optimal decision network better and faster.

[0195] 2) This invention employs a combination of trajectory prediction and reinforcement learning. When agents are conducting encirclement operations, the future trajectory of the enemy is considered in advance. Based on the extended Kalman filter algorithm, the predicted state of the enemy agent at the next time step is calculated. Through secondary correction, the predicted value is adjusted according to the distance between the agent and the enemy agent. Finally, the corrected state prediction value replaces the actual observation of the enemy agent and is used as the input to the agent's decision network for training. Compared with traditional multi-agent reinforcement learning methods, this method transforms the encirclement approach from "reactive" to "predictive," accelerating network training, increasing the accuracy of decisions, and making the encirclement strategy more intelligent.

[0196] The above embodiments are merely exemplary embodiments of the present invention and are not intended to limit the present invention. The scope of protection of the present invention is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to the present invention within its spirit and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of the present invention.

Claims

1. A multi-agent learning pursuit-evasion method based on intrinsic motivation and trajectory prediction, characterized in that, Comprise: Step 1: Constructing multi-agent confrontation hunting environment model, constructing the multi-agent confrontation area as a fixed area, including: two types of agents, the pursuit side and the escape side, and setting the task of the pursuit side as completing the hunting of all the escape sides; Step 2: Designing the state and action space of the agent, wherein the state space of the agent is the total set of all states that the agent can observe, which is defined as position, velocity and attitude information; the action space is the total set of all actions that the agent can select, which is designed as a continuous state, and the acceleration is selected as the physical meaning, and the action space is defined as the acceleration values of different directions of stillness, left movement, right movement, upward movement and downward movement, so as to represent the change of velocity direction and size; Step 3: Based on the internal motivation of knowledge, designing the reward function, which is designed into four kinds of collision reward, expert guidance reward, out-of-bound reward and artificial potential field reward, and the total reward of the agent confrontation hunting is obtained by combining the above four kinds of rewards; Step 4: Based on the trajectory prediction method of extended Kalman filter, the activity range of the enemy agent is predicted, and it is used as new state information, then the agent training is carried out based on trajectory prediction until the network parameters converge, and the training is ended.

2. The intrinsic motivation and trajectory prediction based multi-agent learning pursuit and evasion method of claim 1, wherein, In step 1, the multi-agent confrontation hunting environment model is established, including: In an environment Env, there are multiple agents, of which n agents are pursuers and m agents are fleeing agents, represented by a set H = {h1, h2, ..., h...}. n Let P = {p1, p2, ..., pn} represent n pursuing agents. m } represents m fleeing agents; The task of the pursuit side is to complete the hunting of the escape side in the confrontation area; The final condition for completing the task of the pursuit side is that all agents are successfully pursued, that is, the final task of the pursuit side.

3. The intrinsic motivation and trajectory prediction based multi-agent learning pursuit and evasion method of claim 2, wherein, In step 2, the state space is designed, including: According to the established multi-agent confrontation hunting environment model, the state space of the agent is defined as: Wherein, X represents the state information of the agent, p represents the position information of the agent, v represents the speed information of the agent, represents the posture information of the agent.

4. The multi-agent learning pursuit and evasion method based on intrinsic motivation and trajectory prediction of claim 3, wherein, In step 2, the action space is designed, including: The moving agent is regarded as a planar object, and the height is ignored; At each time step, the agent selects actions through its own decision network; The action space is the total set of all actions that the agent can select, and in this environment, the action is designed as a continuous state, and all places in the confrontation area can be reached by the agent, and the task of the agent is to pursue the enemy agent, and each action in the environment is determined by the direction and size of the agent's velocity, therefore, the acceleration is selected as the physical meaning of the action space, and the size of the acceleration in different directions is output to represent the change of velocity direction and size, the specific form is: the decision network outputs a vector with a length of 5, A = {a stay ,a left ,a right ,a up ,a down} The above A represents the action space of the agent, wherein the five elements respectively represent the acceleration size of stillness, left movement, right movement, upward movement and downward movement, and the size of the elements in the vector is limited in the range of 0-1, and the final acceleration of the agent is obtained by adding the elements in the vector.

5. The intrinsic motivation and trajectory prediction based multi-agent learning pursuit method of claim 4, wherein, In step 3, The collision reward is an external motivation, which gives a reward when the agent as the pursuit side successfully collides with the enemy agent, or gives a punishment when the agent collides with the agent, guiding the agent to hunt the enemy agent and avoid collision with the agent; The expert guidance reward is an internal motivation, which gives a reward when the action decision value of the agent is similar to the expert guidance decision value, and gives a punishment when it is not similar, guiding the agent to make correct actions quickly; The out-of-bound reward is an intrinsic motivation, which gives a punishment when the agent is about to go out of the bound or has gone out of the bound, and guides the agent to move in the confrontation area; The artificial potential field reward is an intrinsic motivation, which regards the environment information around the agent as an energy field, regards the enemy agent as a target point, regards the teammates as obstacles, and regards the target point as attractive to the agent and the obstacles as repulsive to the agent. The artificial potential field reward is obtained by calculating the attractive and repulsive potential fields of the target point and the obstacles, and guides the agent to move towards the enemy agent and away from the teammates.

6. The intrinsic motivation and trajectory prediction based multi-agent learning pursuit method of claim 5, wherein, In step 3, the four rewards are designed as follows: For collision reward, the collision reward value of the agent with the enemy agent or with the friendly agent is designed, if the distance between the agent and the enemy agent is less than d goal ; d goal For the sum of the radii of the agent and the enemy agent, i.e. a successful hit on the enemy agent, a reward value of 10 is obtained; If the distance between the agent and the agent of the same side is less than d obs ; d obs For the agent to collide with the self-agent, the sum of the radii of the two agents, i.e., the self-agent, is obtained -2 reward value, and the collision reward is the external motivation feedback value given by the environment. For the expert guidance reward, an expert guidance knowledge dataset is introduced, and the expert guidance knowledge is taken as one of the intrinsic motivations. When the decision of the agent is similar to the expert guidance knowledge to a certain extent, a positive reward is given to reduce the invalid search of the training network. When the distance angle between the decision value output by the decision network of the agent and the expert guidance decision value is close, a positive reward is given. When the angle exceeds a certain value, it means that the action given by the decision network is wrong or even in the opposite direction, and a negative reward is given. The decision value is the action value, and is represented as follows: r1 = η if |A * -A|<τ r1 = -η if |A * -A|≥τ Wherein, η represents the reward value given by the expert guidance, A * represents the action value output by the current decision network of the agent, A represents the decision value guided by the expert, |A * represents the angle difference value of the two action decisions, τ represents the threshold value set according to the situation; For the out-of-bound reward, the out-of-bound reward is designed as another intrinsic motivation. According to the distance of the agent from the boundary, a proper negative reward is given to the agent to urge it to move in the expected area. The out-of-bound reward is represented as follows: Where x is the horizontal coordinate of the current position coordinate (x, y) of the agent, and l is the radius of the square confrontation area. It is assumed that the square confrontation area is an area with a horizontal and vertical axis range of (-l, +l). Similarly, the reward given to the agent according to the vertical coordinate y of the agent is also according to the above formula. If the agent is outside the boundary, an exponentially increasing negative reward is given, and the maximum negative reward is set to be within 10. If the agent is inside the boundary but within 10% of the boundary value, a negative reward proportional to the distance is given. If the agent is in other positions, no reward is given. For the artificial potential field reward, the environment information around the agent is regarded as an energy field. During the movement of the agent, the target point is attractive to the agent, and the obstacles are repulsive to the agent. The target point is the point with the lowest potential energy in the environment. The movement of the agent is always from the place with high potential energy to the place with low potential energy. Finally, the agent will reach the target point under the traction of the potential field. An intrinsic motivation given to the agent is to hope that the agent can approach the enemy agent, i.e. the target point. Meanwhile, the teammates are regarded as obstacles in the environment. The reward value obtained based on the above intrinsic motivation is as follows: where U att (q) is the attractive potential field of agent j to agent i at position q, a is the proportional coefficient of the attractive potential field function, d ij is the distance between agent i and agent j, when the Euclidean distance is adopted, then m can be 2; wherein U rep (q) is the repulsive potential field of agent j to agent i at position q, k is the proportionality coefficient of the repulsive field function, d0 is the critical value of the repulsive field function, the repulsive field acts when the distance between agent i and agent j is less than d0, and does not act when the distance is greater than the critical distance; where F att (q) is the negative gradient of the attractive potential field, which is the attractive force of agent j on agent i at position q; where F rep (q) is the negative gradient of the repulsive potential field, which is the repulsive force of agent j on agent i at position q; The final reward value of the artificial potential field is as follows: r3 = U att (q) + F att (q) + U rep (q) + F rep (q) Through the gravity and repulsion, the agent is guided to move towards the enemy agent, and is informed to move away from the teammates to avoid unnecessary casualties; The total reward of the confrontation and hunting is as follows: Where r(d) = r1 + r2 + r3, i.e. the sum of the expert guidance reward, the out-of-bound reward and the artificial potential field reward, is the reward function given to the agent based on the intrinsic motivation. The absolute value of the maximum value of the intrinsic motivation-based reward function is designed to be smaller than the order of magnitude of the reward value of the final task, and the reward value of the final task of the agent based on the above design is 10, so the absolute value of the maximum value of the intrinsic motivation-based reward function is selected as 1, that is, |r(d)|<1.

7. The intrinsic motivation and trajectory prediction based multi-agent learning pursuit and evasion method of claim 6, wherein, In step 4, Taking the agent as an unmanned vehicle as an example, the motion equation and the observation equation of the enemy agent are established, and the extended Kalman filter algorithm is used to predict the trajectory range of the enemy agent at the next time step according to the current observation information of the enemy agent; A quadratic correction method for the predicted trajectory range is designed, which controls the weight value to be proportional to the distance between the two agents by calculating the weighted sum of the current actual position and the predicted position of the enemy agent, so that when the distance between the two agents is far, the predicted trajectory range is large, and the agent moves in the general direction of the enemy agent, and when the distance between the two agents is close, the predicted trajectory range is reduced, and the agent accurately moves towards the enemy agent and captures it.

8. The multi-agent learning pursuit and evasion method based on intrinsic motivation and trajectory prediction according to claim 7, characterized in that, In step 4, the method further comprises: Taking the MADDPG algorithm as the basic framework of reinforcement learning, the decision network and the evaluation network of the agent as the pursuer are trained in combination with the trajectory prediction method, so that the agent has the ability to effectively cooperate to capture the enemy agent; Wherein, the predicted position value of the enemy agent obtained by trajectory prediction and quadratic correction replaces the current actual observation position information of the enemy agent by the agent, and combines the predicted speed information as part of the input of the decision network of the agent, and trains the decision network; The action decision of each agent at each time step is evaluated in real time using the knowledge-based intrinsic motivation reward function.

9. The intrinsic motivation and trajectory prediction based multi-agent learning pursuit method of claim 8, wherein, In step 4, the trajectory prediction based on the extended Kalman filter comprises: The trajectory prediction method based on the extended Kalman filter is used to predict the activity range of the enemy agent, and the activity range is used as new state information to replace the current state information {p, v} of the enemy agent observed in real time as a new input value of the agent decision network, and the agent decision network is trained; Because the position of the enemy agent in the field of view of the agent at each time step changes from a precise point to a range, which will interfere with the agent's decision in the later stage of pursuit, a trajectory quadratic correction is designed according to the distance between the two agents; When the agent is an unmanned vehicle, the state information required by the extended Kalman filter algorithm is the accurate speed and position information which has been calculated, and the observation information is the direct information obtained by the agent through the sensor, i.e. speed, position and attitude angle; The state information of the enemy agent: x t = x(t) = [p x , p y , v x , v y ] T wherein p x y is the position information of the enemy agent, p t = p(t) = [p x , p y ] T , v x y is the speed information of the enemy agent, v t = v(t) = [v x , v y ] T ;​​ The motion equation of the enemy agent is established: Converted into matrix form: Wherein, dt is the sampling period, such as dt = 0.1s, a x , a y The lateral and longitudinal acceleration of the agent, F t The Jacobian matrix of the state equation; The observation equation of the enemy agent is established: Wherein, the measurement matrix H is: After the motion equation, the observation equation and the measurement matrix of the enemy agent are established, the extended Kalman filter algorithm is used to predict the state prediction value of the enemy agent at the next time step; specifically including: Initialize the state information x of the enemy agent t , covariance matrix P t , obtain the observation value z t ; One-step prediction: x t into the motion equation to calculate By The prediction covariance matrix, where Q is the process noise covariance; Secondary prediction: Substituting into the measurement matrix formula, we get H t+1 Substituting these values ​​into the observation equation, the predicted observation values ​​are obtained. Therefore, calculate the Kalman gain. Where R is the measurement noise covariance, and the state prediction value of the enemy agent at the next time step is further calculated. and covariance matrix According to the distance between the two agents, the trajectory quadratic correction is performed to adjust the range of the trajectory prediction; The weight of the actual position of the current enemy agent is increased, and the weight of the predicted position is reduced to change the size of the predicted trajectory range, wherein the size of the weight ω is proportional to the distance between the friendly agent and the enemy agent. The position prediction value after the second correction is: wherein p(t) is the current actual position of the enemy agent, p'(t+1) is the predicted position of the enemy agent obtained by the extended Kalman filter, which is included in the calculated state prediction value x t+1 of the enemy agent, is the quadratic correction position prediction value obtained by two-weighting, which is used to replace the current actual observation position information of the enemy agent by the friendly agent, combined with the predicted speed information in x t+1 , as part of the input of the decision-making network of the friendly agent, to train the decision-making network. As the distance between the friendly agent and the enemy agent continues to decrease, the weight of the position prediction value gradually decreases, that is, the range of trajectory prediction is gradually reduced, so that the friendly agent can move in the general direction when the distance is far, and can be accurately captured when the distance is close.

10. The intrinsic motivation and trajectory prediction based multi-agent learning pursuit method of claim 9, wherein, In step 4, the agent training based on trajectory prediction includes: Step 41 : initialize the decision network p of each own agent i with the evaluation network Q i , i e {1,..., n}, initialize the environment scenario; Step 42: obtain the predicted trajectory of the enemy agent through extended Kalman filtering, and after the second correction, replace the observation value of the friendly agent to the enemy agent as the input of the decision network of the friendly agent, and output the action value of the friendly agent at the next time step; Step 43: each friendly agent action is executed, and the environment is stepped; Step 44: calculate the immediate reward value of the output action decision of the friendly agent at the current time step through the knowledge-based intrinsic motivation reward function and store it; Step 45: store a sufficient number of samples (s t , a t , r t+1 ) of the environment state vector s t at time step t, the joint action vector a t of the ego agents at time step t, and the immediate reward value r t of the ego agents at time step t t , randomly sample in the samples, use the sampled batch of samples, and update the evaluation network parameters of the ego agents by using the gradient descent method to minimize the time difference error, and then update the decision network parameters to maximize the evaluation value Step 46: repeat the above steps 42-45 until the network parameters converge, and end the training.

Images