A multi-agent cooperative dynamic target interception decision-making method based on reinforcement learning

Abstract

The application discloses a kind of multi-agent collaborative dynamic target interception decision-making methods based on reinforcement learning, comprising: initializing environment and agent;The state of the interception target is predicted;Based on the predictive interception cost of multi-agent, the global optimal task is distributed;An autonomous control model based on proximal policy optimization algorithm PPO is constructed for each agent, and a state representation vector for future collision perception is designed;The state representation vector is input into the neural network of the agent, and the action instruction is output;Design a multi-object reward function to guide the agent to learn behavior during the training process;The agent of the application has a higher strategy. More sophisticated state representation and more complex reward function design enable the agent trained by PPO not only to accurately perform the pursuit task on the target, but also to learn advanced strategies such as autonomous obstacle avoidance, energy consumption reduction and efficiency improvement.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of artificial intelligence and autonomous control technology, and particularly relates to a multi-agent collaborative dynamic target interception decision-making method based on reinforcement learning. BACKGROUND

[0002] With the rapid development of agent technology, agent clusters are increasingly widely used in fields such as forest exploration, battlefield attack, and logistics transportation. Therefore, in the scenario of multi-agent collaborative interception of high-speed maneuvering targets, how to efficiently allocate tasks for each agent in the cluster and generate the optimal interception strategy is a core technical problem in this field. In traditional schemes, a pre-set task allocation and centralized path planning are usually relied on to solve the problem. In recent years, deep reinforcement learning (DRL) provides a new way to solve such complex decision-making problems, allowing agents to learn optimal strategies through interaction with the environment. The common implementation scheme in this field is as follows: target state perception: each agent independently detects the target through its own sensor (such as radar) to obtain noisy position information. These raw observation data can be directly used for subsequent decision-making. Task allocation: at the beginning of the task, a static allocation strategy (such as the "nearest principle") is adopted to independently allocate a fixed target to each agent, and the target will not change during the task execution. Trajectory planning and control: after task allocation, each agent simplifies the target interception problem to a single-agent tracking problem. Traditional guidance algorithms such as proportional navigation guidance (PNG) or independent reinforcement learning agents are used to control the agent to fly to the specified target. In the reinforcement learning method, the state input of the agent is usually the relative position and velocity of the agent and the target (which can be obtained through target state perception), and the reward function is simply designed to be negatively correlated with the distance between the target. However, the above existing technical solutions have the following four significant shortcomings in practical application:

[0003] 1: The agent perception and decision-making system lacks robustness. Because the agent directly uses noisy detection data when making decisions, the agent's flight trajectory will therefore frequently fluctuate, and accordingly, the tracking efficiency is low and the decision quality is poor. When the detection noise is large or there is a short-term data loss, the decision system may make incorrect judgments, and even further cause the task to fail.

[0004] 2: The task allocation of the swarm of agents in the target pursuit is too rigid, resulting in low coordination efficiency. The static task allocation strategy cannot adapt to the dynamically changing battlefield situation. For example, when an agent deviates from its target due to obstacle avoidance or energy depletion, the system cannot dynamically assign another agent with a more optimal position to replace its task, resulting in prolonged overall interception time and low coordination efficiency.

[0005] 3: Limitations of single-agent decision-making. Independent single-agent decision-making ignores the global information provided by the team. For example, an agent may choose a path to pursue its assigned target, but it may inadvertently block the interception path of another agent in the swarm. In addition, the reward function is too simple to guide the agent to learn complex strategies such as balancing between obstacle avoidance, energy consumption, and interception efficiency.

[0006] 4: Separation of perception and decision-making. In traditional methods, the perception process of the target state is separated from the decision-making process of the agent. The perception module only provides observation data at the current time, and the decision-making module cannot effectively predict the future state of the target using information such as time, resulting in a lack of foresight in decision-making. SUMMARY

[0007] Invention purpose: To overcome the problems of low interception efficiency, poor coordination, and insufficient robustness caused by detection noise interference, rigid task allocation, limitations of single-agent decision-making, and lack of foresight in decision-making, this invention provides a multi-agent collaborative dynamic target interception decision-making method based on reinforcement learning, to improve the autonomous collaborative combat capability of multi-agent in complex dynamic environments.

[0008] Technical solution: A multi-agent collaborative dynamic target interception decision-making method based on reinforcement learning, comprising the following steps:

[0009] Step 1, initialize the environment and agents;

[0010] Step 2, predict the state of the interception target;

[0011] Step 3, allocate the globally optimal task based on the predictive interception cost of multi-agent;

[0012] Step 4, construct an autonomous control model for each agent based on the Proximal Policy Optimization algorithm (PPO), and design a state representation vector for future collision perception; input the state representation vector into the neural network of the agent, and output the action instruction;

[0013] Step 5, design a multi-target reward function to guide the agent to learn behaviorally during the training process;

[0014] Step 6, repeat steps 2 to 5 until all interception targets are intercepted or the entire round reaches the maximum simulation step, complete the multi-agent cooperative dynamic target interception decision.

[0015] Further, step 1 is specifically: initialize the environment including the initial positions and velocities of N agents and the initial states of M targets; initialize M independent Kalman filters for M targets respectively, and initialize N independent PPO agents with the same network structure for N agents respectively; the Actor-Critic network structure of the PPO agent is a multi-layer perception containing two hidden layers.

[0016] Further, step 2 is specifically: at each time frame dt, for each target j that has not been intercepted, call the predict() method in the target's Kalman filter to calculate the prior estimate and prior covariance at time t based on the optimal estimate at time t-1; determine the agent i closest to target j as the observer, simulate a noisy observation process based on the true positions of agent i and target j to generate an observation value; wherein the standard deviation of the observation noise is proportional to the distance d between the agent and the target; calculate the observation noise covariance matrix based on this; call the update() method in the KF to correct the prior estimate combined with the observation value to obtain the posterior and optimal estimated state at time t.

[0017] Further, step 3 is specifically: construct a global optimal task allocation model based on the estimated interception time as the cost, representing the estimated cost of agent i intercepting target j, the cost is the dynamically calculated estimated interception time, and its calculation method is as follows:

[0018] (1)

[0019] where, , are the current position vectors of agent i and target j respectively; ‖ ‖ represents the Euclidean norm; is the current position and velocity of target j; is the maximum speed of the agent; is an optional task switching penalty term to improve the stability of the allocation scheme; the numerator term represents the straight-line distance between agent i and target j; the denominator term considers the projection of the target velocity in the direction of the line connecting the two, dynamically evaluating the interception difficulty of the target: the smaller the value, the closer the target velocity is to the maximum speed of the agent at this time, the more difficult it is for the agent to catch up with the target.

[0020] Further, step 4 is specifically: constructing an autonomous control model based on the proximal policy optimization algorithm PPO for each agent; and ensuring the stability of the training process by clipping the objective function, the PPO clipping objective function:

[0021] (2)

[0022] where, is the parameter of the policy network; represents the expectation for time step t; is the probability ratio of the new and old policies; is the advantage function estimate; clip() is a clipping function that limits the value of to the interval ; is a hyperparameter that defines the clipping range; by using the min() function to take the smaller value of the unclipped term and the clipped term , the stability of policy update is ensured;

[0023] Design the state representation vector S i for future collision perception:

[0024] (3)

[0025] where, indicates the collision risk flag, which is predicted by the agent through ray casting whether there will be a collision with the obstacle in the current speed direction, if yes, is 1, otherwise 0; t tti is the normalized estimated collision time, v esc is the recommended avoidance vector;

[0026] Input the state vector S i into the agent's Actor-Critic network, the Actor network outputs the mean value μ of the action instruction, combined with the learnable logarithmic standard deviation, to construct a normal distribution Normal(μ, ), std represents the standard deviation; a action is sampled from the normal distribution; the Critic network outputs the value estimate of the current state; the sampled action is mapped to the interval [-1, 1] through the Tanh function to get the final normalized action .

[0027] Further, step 5 is specifically: all agents execute the normalized action Multiplying the maximum acceleration by the value and updating its state in the environment through the physics engine, we obtain its position and velocity at time t+1. The transfer information is stored in the experience replay buffer of the corresponding agent. When the amount of data accumulated in the buffer of any agent reaches a preset threshold, in order to train an efficient, safe, and physically realistic control strategy, the agent performs a learning update:

[0028] (4)

[0029] in, This represents the progress reward, which is proportional to the reduction in distance between the agent and the target, and is the main driving force of the entire training process. This indicates a speed alignment reward, which encourages agents to move their speed and direction toward the target, thus improving the efficiency of pursuing the target. This refers to the end-potential field reward, specifically a non-linearly increasing reward provided when approaching the target, encouraging the agent to move more precisely toward the target rather than simply "running alongside" the target. This represents an obstacle avoidance penalty. When an agent enters the "buffer zone" around an obstacle (without actually entering the obstacle), a penalty related to the depth of intrusion is applied. This is used to smoothly guide the agent to learn to weigh the pros and cons of approaching the obstacle without entering versus approaching the target. and It represents time and energy consumption penalties, encouraging agents to complete tasks in less time and with smoother control, achieving high efficiency and low energy consumption. This represents the weight of each item, k∈[1,6]; by adjusting the weights of each item... It can balance the urgency, safety and economy of the task, and shape the most ideal and desired intelligent agent behavior.

[0030] Furthermore, the transition information includes a state vector, normalized action, state value estimate, and collision risk flag.

[0031] The present invention also discloses a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method of the present invention.

[0032] The present invention also discloses a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the method of the present invention.

[0033] The present invention also discloses a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method of the present invention.

[0034] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0035] 1: Decision robustness and precision are significantly improved. The Kalman filter is introduced in the target state perception stage to predict and estimate the target state, effectively filtering out most of the detection noise, and providing an accurate and smooth data basis for the decision system.

[0036] 2: The efficiency of cooperative interception is greatly improved. The dynamic task allocation mechanism ensures the optimal use of agent resources in the global range. In complex scenarios such as multiple target trajectories crossing or agent motion trajectories being blocked, this mechanism can adjust the task allocation for the agent group in real time, effectively avoiding resource waste.

[0037] 3: The agent has higher-level strategies. The more detailed state representation and more complex reward function design of the present application enable the agent trained by PPO not only to accurately perform the pursuit task on the target, but also to learn advanced strategies such as autonomous obstacle avoidance, energy consumption reduction, and efficiency improvement. The agent can actively slow down or detour when approaching obstacles, rather than reacting at the edge of collision, demonstrating its stronger survival ability and task adaptability. By including these forward-looking state vectors into the input of the neural network, the agent can "anticipate" danger, thereby further learning to slow down and turn in advance when approaching obstacles, rather than reacting passively at the last moment.

[0038] 4: The "perception-planning-decision" integration is realized. The state perception and estimation, task planning and allocation, and agent control and decision are integrated in a cycle, realizing the closed loop of information. This integrated design makes the system highly adaptive and forward-looking, capable of coping with most dynamic and uncertain complex environments. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is a method flowchart of the present application;

[0040] Figure 2 is a 3D simulation diagram of the best round;

[0041] Figure 3 is a 2D simulation diagram of the best round;

[0042] Figure 4 is a multi-agent cooperative task allocation history diagram; (a) is a task allocation history diagram for agent 1; (b) is a task allocation history diagram for agent 2; (c) is a task allocation history diagram for agent 3;

[0043] Figure 5 is a multi-agent decision analysis simulation diagram; (a) is a cost simulation diagram for agent 1; (b) is a cost simulation diagram for agent 2; (c) is a cost simulation diagram for agent 3;

[0044] Figure 6 It is a distance map between the agent and the center of the obstacle;

[0045] Figure 7 This is a KF observation noise standard deviation plot;

[0046] Figure 8 (a) is a trend diagram of the agent's maneuver parameters; (b) is a diagram of the agent's velocity; (c) is a diagram of the agent's azimuth angle; and (d) is a diagram of the agent's pitch angle.

[0047] Figure 9 This is a graph showing the total reward during the training process. Detailed Implementation

[0048] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0049] like Figure 1 As shown, the complete closed-loop process of this invention includes steps such as environment and agent initialization, main loop of the algorithm, selection of agent actions, dynamic task allocation, agent update, and drawing of simulation graph corresponding to the best round result, which clearly demonstrates the core innovation of this invention and their interrelationships.

[0050] The method of this invention is executed in a loop of a discrete time frame dt (e.g., 0.1 seconds), and can be specifically divided into the following steps:

[0051] Step 1: System Initialization

[0052] At the start of the task, the environment and PPO agents are initialized. Environment initialization includes the initial positions of N agents (N=3 in this invention). and speed (Initial velocity set to zero), the initial states of M targets (M=2 in this invention). M independent Kalman filter instances are initialized for each of the M targets, and N independent PPO agents with the same network structure are initialized for each of the N agents. The Actor-Critic network structure of the PPO agents is a multilayer perception system containing two hidden layers (256 neurons per layer, activation function Tanh).

[0053] Step 2: Target State Estimation and Prediction

[0054] At each time frame dt, for each unintercepted target j, the algorithm will call the predict() method in its corresponding Kalman filter to calculate the priori estimation and priori covariance at time t according to the optimal estimation at time t-1. Then, determine the agent i closest to target j as the observer. According to the real positions of agent i and j, simulate a noisy observation process to generate an observation value. The standard deviation of the observation noise is proportional to the distance d between the agent and the target. According to this, the observation noise covariance matrix is calculated. Call the update() method in KF to correct the priori estimation combined with the observation value to obtain the posteriori (and optimal) estimation state at time t.

[0055] Step three: dynamic task allocation

[0056] This step is the core step for the algorithm to realize multi-agent cooperation. The algorithm will first construct an N x M dimensional cost matrix, where N is the current number of agents and M is the number of unintercepted targets. For each agent i and target j combination, according to step two, the target estimation state and the current state of the agent are obtained, and then the estimated interception time is calculated as the basic cost. In addition, in order to avoid the possible switching of the target by the agent at a special position (such as the intersection of the target trajectory, etc.), which may lead to unstable training process, a fixed switching penalty term is added to the agent cost calculation . Then, use the Hungarian algorithm to solve the cost matrix to obtain a assignment scheme that minimizes the total cost. This scheme clearly shows which target each agent should intercept at the current time frame dt. And the allocation can ensure that each target has at least one agent tracking it.

[0057] Step four: distributed generation of control actions

[0058] For each agent i, a 19-dimensional state vector S will be constructed according to the target assigned to it in step three. Its components are strictly concatenated in the following order: the normalized vector of the difference between the target predicted pursuit point and the agent position (3 dimensions); the normalized vector of the agent's own speed (3 dimensions); the normalized vector of the target estimated speed (3 dimensions), the flag bit of whether the target is inside the obstacle (1 dimension), the flag bit of whether the agent is about to collide with the obstacle (1 dimension), the normalized predicted collision time (1 dimension), the recommended collision avoidance direction vector (3 dimensions), the normalized vector of the difference between the obstacle center and the agent position (3 dimensions), and the normalized vector of the obstacle radius (1 dimension).

[0059] Then, the state vector S iThe input is fed into the Actor-Critic network of the PPO agent of the agent i. The Actor network outputs a mean value μ of an action (three-dimensional acceleration instruction), which, combined with a learnable logarithmic standard deviation, constructs a normal distribution Normal(μ, ). A sampled action is sampled from the distribution. The Critic network outputs a value estimate V( ,i) of the current state.

[0060] Finally, the sampled action is mapped to the interval [-1, 1] through a Tanh function to obtain the final normalized action .

[0061] Step five: interaction with the environment and model learning

[0062] In this step, all agents will execute the normalized action generated in step four, multiply it by the maximum acceleration, and update their state in the environment through the physics engine to obtain the position and velocity at time t+1. In this process, the code also calculates the reward obtained by each agent i at this time frame. The complete transition information (including state vector, normalized action, state value estimate, collision risk flag, etc.) is stored in the experience replay buffer of the corresponding agent. When the amount of data accumulated in the buffer of any agent reaches the preset threshold (set to 4096 steps in this invention), the agent will perform a learning update: use the general advantage estimation method to calculate the advantage function, and according to the clipping surrogate objective function and the loss of the value function, finally update the parameters of the Actor-Critic network through the Adam optimizer.

[0063] Step six: loop execution

[0064] Repeat steps two to five until all targets are intercepted or the maximum simulation step is reached, thus completing a round of training.

[0065] In this way, through a large number of rounds of training, the PPO network of each agent will gradually converge, thus learning to execute an efficient cooperative interception strategy.

[0066] For example, Figure 2 , Figure 3As shown, three agents and their trajectories are represented by red, blue and green colors respectively, two targets and their trajectories are represented by red and blue colors respectively, their starting points and ending points are represented by dot and cross points respectively, and the obstacles are represented by gray hemispheres in space. As can be seen from the figure, after the cooperative combat, target tracking and obstacle avoidance of the three agents, the agent 1 successfully captures the target 1, and the agent 3 successfully captures the target 2, achieving the purpose of the application.

[0067] As shown in Figure 4 , the initial allocation relationship of the three agents is agent 1->target 2, agent 2->target 1, and agent 3->target 1, because the agent is closest to the corresponding target at the beginning, and the estimated interception time is also the shortest, so the cost matrix is also the smallest, and the best allocation result is obtained after the cooperation of the agents. However, as time goes on, the targets 1 and 2 enter the obstacles in turn, at this time the estimated interception time calculated in the cost matrix will get a larger value because the agent avoids obstacles, and after continuous re-coordination and allocation, the allocation strategy of the agent 1 and the agent 3 changes, and they turn to chase new targets, i.e. agent 1->target 1 and agent 3->target 2. And the agent 2 is special in position (located directly above the obstacle), its role is more of an "auxiliary tracking". Therefore, there are simulation results like Figure 2 , Figure 3 .

[0068] As shown in Figure 5 , the changes of the cost parameters of the two targets and the changes of the cooperative allocation relationship of the three agents at different times in the best round Figure 4The solid line represents the pursuit cost, which means the cost evaluation of the current target being pursued by the agent. This line represents the current selected scheme by the agent, and its value is the cost of the current decision, which is the best choice in the view of the task assignment algorithm. The dashed line represents the alternative cost, which means the cost evaluation of another target (not being pursued). This line represents the cost of the "alternative scheme", which is being monitored by the agent, and once it becomes lower than the pursuit cost, the decision switching can be triggered. The dotted line represents the estimated time to intercept (ETI), which means the theoretical time to intercept calculated by considering the distance and relative velocity. It is the most important input parameter in the total cost calculation, and it can be seen that the trend of the dotted line is highly consistent with the total cost line (solid / dashed), because ETI is the main factor determining the cost. The dashed-dotted line represents the physical distance between the agent and the target, which means the most intuitive physical distance between the agent and the target. This line can be used to verify the rationality of the cost calculation, and generally, the closer the distance, the lower the cost. But sometimes, even if the distance is a little farther, if the direction of the agent's speed is consistent with the target, its ETI and total cost may be lower. The circle points represent the decision switching points, which means marking the exact time point when the agent changes its pursuit target. When a blue circle point appears, it means that the agent switches from pursuing target 1 to target 2 at this moment; when a red circle point appears, it means that it switches from pursuing target 2 to target 1. The appearance of this point is inevitably accompanied by the switching of the solid line and the dashed line. Of course, this switching is completely corresponding to the switching in Figure 4

[0069] So, the whole pursuit target switching process can be explained as follows: assume that at the beginning of the task, the red line is the solid line and the blue line is the dashed line, as in (c) of Figure 5 This indicates that at the beginning of the task, the Hungarian algorithm assigns target 1 to agent 3. As time goes on, agent 3 flies towards target 1, so we can see that the red solid line (pursuit cost) and its corresponding red distance dashed line are steadily decreasing. At the same time, it may be farther away from target 2, so the blue dashed line (alternative cost) may be rising or slowly falling (such as the alignment of the speed of the agent and target 2). Suddenly, due to the maneuvering of target 2 or the coordinated position change of other agents, the cost of the agent pursuing target 2 (blue dashed line) drops sharply and becomes lower than the cost of pursuing target 1 (red solid line). The Hungarian algorithm finds that when this agent is assigned to pursue target 2 while other agents are assigned to pursue target 1, the total cost of the whole agent cluster is the lowest. Therefore, the algorithm issues the switching target instruction. At this moment, Figure 5 ​A blue dot appears on (c) in the figure. At the same time, the red line changes from solid to dashed, and the blue line changes from dashed to solid. This marks the beginning of a new task for agent 3. After that, the agent begins to fly towards target 2, and at this time the blue solid line representing the pursuit cost and the blue dashed line representing the distance both begin to decrease.

[0070] As shown in Figure 6 , the agent not only accurately avoids obstacles to pursue targets through obstacles, but also flies close to (without touching or entering) obstacles to efficiently and quickly catch up with its corresponding target. This intuitively demonstrates the invention's future collision perception part. The controller based on reinforcement learning fuses the state representation of future collision perception (collision risk , estimated collision time , avoidance vector ) through learning, and obtains forward-looking decision-making capability. Instead of passively avoiding collisions, it actively plans a safe path.

[0071] As shown in Figure 7 , the KF observation noise standard deviation curve shows a trend of first decreasing, then increasing, and then decreasing over time. This is because the observation noise of KF in the invention is affected by the distance between the agent and the target. The closer the distance, the smaller the observation noise. Therefore, this figure can also reflect the distance between the agent and the target. At the same time, this setting also conforms to the actual physical meaning: the smaller the distance between the agent and the target, the smaller the noise standard deviation of KF, and the higher the prediction accuracy of KF. At this time, the distance between the agent and the target is also closer.

[0072] Figure 8 is a trend graph of the agent's maneuvering parameters. In the invention, the physical quantity directly controlled and optimized by the PPO agent is the three-dimensional acceleration vector of the agent, and the speed and angle of the agent are directly affected by its acceleration. Therefore, these three figures can intuitively reflect the changes of some key parameters of the agent during the tracking process, and also prove the physical realizability of the invention.

[0073] Figure 9 can well reflect the effect of the algorithm in training the agent to track the target. As can be seen from the figure, the total reward is very low at the beginning of the training round, and increases with the increase in the number of training rounds, and finally stabilizes at a relatively high level with only small fluctuations. At this time, the agent has been able to achieve high-quality cooperative allocation of targets, tracking and obstacle avoidance, and almost achieved the purpose of the invention. In addition, the smoothness of the reward curve also represents that the agent has learned a robust strategy, so it can successfully complete the task with a high probability and obtain stable and high rewards in the face of different random disturbances under similar initial conditions (such as changes in the initial position of the target).

Claims

1. A multi-agent cooperative dynamic target interception decision-making method based on reinforcement learning, characterized in that, Includes the following steps: Step 1: Initialize the environment and the agent; Step 2: Predict the status of the target to be intercepted; Step 3: Allocate the globally optimal task based on the predictive interception cost of multi-agent systems; Step 3 is specifically: constructing a global optimal task allocation model based on the estimated interception time as the cost, The estimated cost of the agent i intercepting the target j is the estimated interception time calculated dynamically, and the calculation method is as follows: （1） where, , are the current position vectors of agent i and target j, respectively; ‖·‖ denotes the Euclidean norm; is the current position velocity of target j; is the maximum velocity of the agent; is an optional task-switching penalty term to improve the stability of the assignment scheme; the numerator term represents the straight-line distance between agent i and target j; the denominator term considers the projection of the target velocity in the direction of the line connecting the two, dynamically evaluating the interception difficulty of the target: the smaller the value, the closer the target velocity is to the maximum velocity of the agent at this time, and the more difficult it is for the agent to catch up with the target; Step 4: Construct an autonomous control model based on the Proximal Policy Optimization (PPO) algorithm for each agent, and design a state representation vector for future collision perception; input the state representation vector into the agent's neural network to output action commands; Step 5: Design a multi-objective reward function to guide the agent in behavioral learning during training; Step 6: Repeat steps 2 to 5 until all intercepted targets are intercepted or the entire round reaches the maximum simulation step size, thus completing the multi-agent collaborative dynamic target interception decision.

2. The multi-agent collaborative dynamic target interception decision-making method based on reinforcement learning according to claim 1, characterized in that, Step 1 specifically involves: initializing the environment, including the initial positions and velocities of N intelligent agents and the initial states of M targets; initializing M independent Kalman filters for each of the M targets, and initializing N independent PPO agents with the same network structure for each of the N intelligent agents; the Actor-Critic network structure of the PPO agents is a multi-layer perception system containing two hidden layers.

3. The multi-agent cooperative dynamic target interception decision-making method based on reinforcement learning according to claim 1, characterized in that, Step 2 specifically involves the following steps: In each time frame dt, for each unintercepted target j, the predict() method in the Kalman filter corresponding to the target is called. Based on the optimal estimate at time t-1, the prior estimate and prior covariance at time t are calculated. The agent i closest to target j is determined as the observer. Based on the true positions of agent i and target j, a noisy observation process is simulated to generate observation values. The standard deviation of the observation noise is proportional to the distance d between the agent and the target. The observation noise covariance matrix is ​​then calculated accordingly. Call the update() method in KF, combine the observed values ​​to correct the prior estimate, and obtain the posterior and optimal estimated state at time t.

4. The multi-agent cooperative dynamic target interception decision-making method based on reinforcement learning according to claim 1, characterized in that, Step 4 specifically involves: constructing an autonomous control model for each agent based on the Proximal Policy Optimization (PPO) algorithm; and ensuring the stability of the training process by pruning the objective function. The PPO pruning objective function is as follows: （2） in, These are the parameters of the policy network; This represents the expectation for time step t; It is the probability ratio of the old and new strategies; It is the dominance function estimate; clip() is the clipping function, which will... The value is limited to the interval Inside; These are hyperparameters that define the range of clipping; the min() function is used to retrieve the undone items. and trimming items The smaller value in the range ensures the stability of the strategy update; Designing state representation vectors S for future collision awareness i : （3） in, This indicates a collision risk flag. The agent predicts whether it will collide with an obstacle in the current velocity direction using ray projection. If so, then... t is 1 if it is 1, otherwise it is 0; tti For the normalized estimated collision time, v esc Recommended avoidance vector; The state vector S i The input is fed into the Actor-Critic network of the agent. The Actor network outputs the mean μ of the action commands. Combined with the learnable log-standard deviation, a normal distribution Normal(μ, ), std Represents the standard deviation; samples an action from a normal distribution. The Critic network outputs a value estimate of the current state; the sampled actions are then processed. By mapping to the [-1, 1] interval using the Tanh function, the final normalized action is obtained. .

5. The multi-agent cooperative dynamic target interception decision-making method based on reinforcement learning according to claim 1, characterized in that, Step 5 specifically involves all agents performing a normalization action. Multiply by the maximum acceleration and update its state in the environment through the physics engine to obtain its position and velocity at time t+1; store the transition information in the experience replay buffer of the corresponding agent; when the amount of data accumulated in the buffer of any agent reaches a preset threshold, the agent performs a learning update: （4） in, Indicates process reward, Indicates speed alignment bonus, This represents the end-potential field reward. This indicates an obstacle avoidance penalty. and Indicates time and energy consumption penalties. Let k represent the weight of each item, k∈[1,6].

6. The multi-agent cooperative dynamic target interception decision-making method based on reinforcement learning according to claim 5, characterized in that, The transition information includes state vectors, normalized actions, state value estimates, and collision risk indicators.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method of claim 1.

8. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method of claim 1.

9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method of claim 1.

Images