Multi-agent collaborative dynamic target interception decision-making method based on reinforcement learning

Abstract

The invention discloses a multi-agent collaborative dynamic target interception decision-making method based on reinforcement learning. The method comprises the following steps: initializing an environment and agents; predicting the state of the intercepted target; distributing a global optimal task based on the predictive interception cost of the multi-agent; constructing an autonomous control model based on a near-end strategy optimization algorithm PPO for each agent, and designing a state representation vector of future collision perception; inputting the state representation vector into a neural network of the intelligent agent, and outputting an action instruction; designing a multi-objective reward function for guiding the intelligent agent to perform behavior learning in a training process; the intelligent agent provided by the invention has a higher-level strategy. Through finer state representation and more complex reward function design, the intelligent agent trained through PPO not only can accurately execute a pursuit task on a target, but also can learn advanced strategies such as autonomous obstacle avoidance, energy consumption reduction, efficiency improvement and the like.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of artificial intelligence and autonomous control technology, and particularly relates to a multi-agent collaborative dynamic target interception decision-making method based on reinforcement learning. BACKGROUND

[0002] With the rapid development of agent technology, agent clusters are increasingly widely used in fields such as forest exploration, battlefield attack, and logistics transportation. Therefore, in the scenario of multi-agent collaborative interception of high-speed maneuvering targets, how to efficiently allocate tasks for each agent in the cluster and generate the optimal interception strategy is a core technical problem in this field. In traditional schemes, a pre-set task allocation and centralized path planning are usually relied on to solve the problem. In recent years, deep reinforcement learning (DRL) provides a new way to solve such complex decision-making problems, allowing agents to learn optimal strategies through interaction with the environment. The common implementation scheme in this field is as follows: target state perception: each agent independently detects the target through its own sensor (such as radar) to obtain noisy position information. These raw observation data can be directly used for subsequent decision-making. Task allocation: at the beginning of the task, a static allocation strategy (such as the "nearest principle") is adopted to independently allocate a fixed target to each agent, and the target will not change during the task execution. Trajectory planning and control: after task allocation, each agent simplifies the target interception problem to a single-agent tracking problem. Traditional guidance algorithms such as proportional navigation guidance (PNG) or independent reinforcement learning agents are used to control the agent to fly to the specified target. In the reinforcement learning method, the state input of the agent is usually the relative position and velocity of the agent and the target (which can be obtained through target state perception), and the reward function is simply designed to be negatively correlated with the distance between the target. However, the above existing technical solutions have the following four significant shortcomings in practical application:

[0003] 1: The agent perception and decision-making system lacks robustness. Because the agent directly uses noisy detection data when making decisions, the agent's flight trajectory will therefore frequently fluctuate, and accordingly, the tracking efficiency is low and the decision quality is poor. When the detection noise is large or there is a short-term data loss, the decision system may make incorrect judgments, and even further cause the task to fail.

[0004] 2: The task allocation of the swarm of agents in the target pursuit is too rigid, resulting in low coordination efficiency. The static task allocation strategy cannot adapt to the dynamically changing battlefield situation. For example, when an agent deviates from its target due to obstacle avoidance or energy depletion, the system cannot dynamically assign another agent with a more optimal position to replace its task, resulting in prolonged overall interception time and low coordination efficiency.

[0005] 3: Limitations of single-agent decision-making. Independent single-agent decision-making ignores the global information provided by the team. For example, an agent may choose a path to pursue its assigned target, but it may inadvertently block the interception path of another agent in the swarm. In addition, the reward function is too simple to guide the agent to learn complex strategies such as balancing between obstacle avoidance, energy consumption, and interception efficiency.

[0006] 4: Separation of perception and decision-making. In traditional methods, the perception process of the target state is separated from the decision-making process of the agent. The perception module only provides observation data at the current time, and the decision-making module cannot effectively predict the future state of the target using information such as time, resulting in a lack of foresight in decision-making. SUMMARY

[0007] Invention purpose: To overcome the problems of low interception efficiency, poor coordination, and insufficient robustness caused by detection noise interference, rigid task allocation, limitations of single-agent decision-making, and lack of foresight in decision-making, this invention provides a multi-agent collaborative dynamic target interception decision-making method based on reinforcement learning, to improve the autonomous collaborative combat capability of multi-agent in complex dynamic environments.

[0008] Technical solution: A multi-agent collaborative dynamic target interception decision-making method based on reinforcement learning, comprising the following steps:

[0009] Step 1, initialize the environment and agents;

[0010] Step 2, predict the state of the interception target;

[0011] Step 3, allocate the globally optimal task based on the predictive interception cost of multi-agent;

[0012] Step 4, construct an autonomous control model for each agent based on the Proximal Policy Optimization algorithm (PPO), and design a state representation vector for future collision perception; input the state representation vector into the neural network of the agent, and output the action instruction;

[0013] Step 5, design a multi-target reward function to guide the agent to learn behaviorally during the training process;

[0014] Step 6, repeat steps 2 to 5 until all interception targets are intercepted or the entire round reaches the maximum simulation step, complete the multi-agent cooperative dynamic target interception decision.

[0015] Further, step 1 is specifically: initialize the initial position and velocity of N agents and the initial state of M targets; initialize M independent Kalman filters for M targets, and initialize N independent PPO agents with the same network structure for N agents; the Actor-Critic network structure of PPO agent is a multi-layer perception containing two hidden layers.

[0016] Further, step 2 is specifically: at each time frame dt, for each target j that has not been intercepted, call the predict() method in the target's Kalman filter to calculate the prior estimate and prior covariance at time t based on the optimal estimate at time t-1; determine the agent i closest to target j as the observer, simulate a noisy observation process based on the true positions of agent i and target j to generate an observation value; the standard deviation of the observation noise is proportional to the distance d between the agent and the target; calculate the observation noise covariance matrix based on this; call the update() method in KF to correct the prior estimate based on the observation value to obtain the posterior and optimal estimated state at time t.

[0017] Further, step 3 is specifically: construct a global optimal task allocation model based on the estimated interception time as the cost, representing the estimated cost of agent i intercepting target j, the cost is the dynamically calculated estimated interception time, and its calculation method is as follows: (1) where, , are the current position vectors of agent i and target j respectively; ‖ ‖ represents the Euclidean norm; is the current position and velocity of target j; is the maximum speed of the agent; is an optional task switching penalty term to improve the stability of the allocation scheme; the numerator term represents the straight-line distance between agent i and target j; the denominator term considers the projection of the target speed in the direction of the line connecting the two, dynamically evaluating the interception difficulty of the target: the smaller the value, the closer the target speed is to the maximum speed of the agent at this time, the more difficult it is for the agent to catch up with the target.

[0018] Further, step 4 is specifically: for each agent, construct an autonomous control model based on the Proximal Policy Optimization algorithm PPO; and ensure the stability of the training process by clipping the objective function, the PPO clipping objective function: (2) where, are parameters of the policy network; represents the expectation at time step t; is the probability ratio of new and old policy; is the advantage function estimate; clip() is a clipping function that limits the value of to the interval ; is a hyper-parameter that defines the range of clipping; by taking the smaller value of the unclipped term and the clipped term using the min() function, the stability of policy update is guaranteed; The state representation vector S i for future collision awareness is designed as: (3) where, is a collision risk flag, which is predicted by the agent through ray casting whether there will be a collision with the obstacle in the current velocity direction, if yes, is 1, otherwise 0; t tti is the normalized estimated collision time, v esc is the recommended evasion vector; The state vector S i is input into the agent's Actor-Critic network, the Actor network outputs the mean value μ of the action instruction, combined with the learnable logarithmic standard deviation, a normal distribution Normal(μ, ) is constructed, std where, represents the standard deviation; a action is sampled from the normal distribution; the Critic network outputs the value estimate of the current state; the sampled action is mapped to the interval [-1, 1] through the Tanh function, and the final normalized action is obtained.

[0019] Further, step 5 is specifically: all agents execute the normalized action , multiply by the maximum acceleration, and update their state in the environment through the physics engine to obtain the position and velocity at time t+1; the transition information is stored in the experience replay buffer of the corresponding agent, when the amount of data accumulated in the buffer of any agent reaches a preset threshold, in order to train an efficient, safe and physically realistic control policy, the agent performs a learning update:

[0020] (4)

[0021] where, represents the process reward, which is proportional to the reduction of the distance between the agent and the target, and is the main driving force of the entire training process. represents the speed alignment reward, which encourages the agent to align the speed direction to the target, and is conducive to improving the efficiency of chasing the target. represents the end potential field reward, which specifically provides a non-linearly increasing reward when approaching the target, and encourages the agent to more accurately approach the target, rather than always "running around" the target. represents the obstacle avoidance penalty, which applies a penalty related to the intrusion depth when the agent enters the "buffer zone" around the obstacle (at this time, it has not entered the inside of the obstacle), and is used to smoothly guide the agent to learn to weigh the pros and cons between approaching the obstacle without entering and approaching the target. and represents the time and energy consumption penalty, which encourages the agent to complete the task in a shorter time and smoother control, and achieves high efficiency and low energy consumption. represents the weight of each item, k∈[1, 6]; by adjusting the weight of each item , the urgency, safety and economy of the task can be balanced, and the most ideal and expected agent behavior can be shaped.

[0022] Further, the transfer information includes a state vector, a normalized action, a state value estimate, and a collision risk flag.

[0023] The application also discloses a computer device, which comprises a memory, a processor and a computer program stored in the memory, and the processor executes the computer program to realize the steps of the method of the application.

[0024] The application also discloses a computer readable storage medium, which stores a computer program / instruction, and the computer program / instruction is executed by a processor to realize the steps of the method of the application.

[0025] The application also discloses a computer program product, which comprises a computer program / instruction, and the computer program / instruction is executed by a processor to realize the steps of the method of the application.

[0026] Advantages: Compared with the prior art, the application has the following obvious advantages:

[0027] 1: The decision robustness and precision are significantly improved. The Kalman filter is introduced in the target state perception stage to predict and estimate the target state, effectively filters out most of the detection noise, and provides an accurate and smooth data basis for the decision system.

[0028] 2: Synergistic interception efficiency is greatly improved. The dynamic task allocation mechanism ensures the optimal use of agent resources in the global range. In complex scenarios such as the intersection of multiple target trajectories or the obstruction of the agent's motion trajectory, this mechanism can adjust the task allocation for the agent group in real time, effectively avoiding resource waste.

[0029] 3: The agent has a higher-level strategy. The more detailed state representation and more complex reward function design of the application enable the agent trained by PPO to not only accurately perform the pursuit task of the target, but also learn advanced strategies such as autonomous obstacle avoidance, energy consumption reduction, and efficiency improvement. The agent can actively slow down or detour when approaching an obstacle, rather than reacting at the edge of a collision, demonstrating its stronger survival ability and task adaptability. By including these forward-looking state vectors into the input of the neural network, the agent can "anticipate" danger, thereby further learning to slow down and turn in advance when approaching an obstacle, rather than reacting passively at the last moment.

[0030] 4: The "perception-planning-decision" integration is realized. The application integrates state perception and estimation, task planning and allocation, agent control and decision-making in a cycle, realizing the closed loop of information. This integrated design makes the system highly adaptive and forward-looking, capable of coping with most dynamic and uncertain complex environments. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 is a method flowchart of the application;

[0032] Figure 2 is a 3D simulation diagram of the best round;

[0033] Figure 3 is a 2D simulation diagram of the best round;

[0034] Figure 4 is a multi-agent cooperative task allocation history diagram; (a) is a task allocation history diagram for agent 1; (b) is a task allocation history diagram for agent 2; (c) is a task allocation history diagram for agent 3;

[0035] Figure 5 is a multi-agent decision analysis simulation diagram; (a) is a cost simulation diagram for agent 1; (b) is a cost simulation diagram for agent 2; (c) is a cost simulation diagram for agent 3;

[0036] Figure 6 is a diagram of the distance between the agent and the center of the obstacle;

[0037] Figure 7 is a KF observation noise standard deviation diagram;

[0038] Figure 8(a) is a trend diagram of the agent's maneuver parameters; (b) is a diagram of the agent's velocity; (c) is a diagram of the agent's azimuth angle; (d) is a diagram of the agent's pitch angle.

[0039] Figure 9 This is a graph showing the total reward during the training process. Detailed Implementation

[0040] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0041] like Figure 1 As shown, the complete closed-loop process of this invention includes steps such as environment and agent initialization, main loop of the algorithm, selection of agent actions, dynamic task allocation, agent update, and drawing of simulation graph corresponding to the best round result, which clearly demonstrates the core innovation of this invention and their interrelationships.

[0042] The method of this invention is executed in a loop of a discrete time frame dt (e.g., 0.1 seconds), and can be specifically divided into the following steps:

[0043] Step 1: System Initialization

[0044] At the start of the task, the environment and PPO agents are initialized. Environment initialization includes the initial positions of N agents (N=3 in this invention). and speed (Initial velocity set to zero), the initial states of M targets (M=2 in this invention). M independent Kalman filter instances are initialized for each of the M targets, and N independent PPO agents with the same network structure are initialized for each of the N agents. The Actor-Critic network structure of the PPO agents is a multilayer perception system containing two hidden layers (256 neurons per layer, activation function Tanh).

[0045] Step 2: Target State Estimation and Prediction

[0046] At each time frame dt, for each unintercepted target j, the algorithm calls the predict() method in its corresponding Kalman filter to calculate the prior estimate and prior covariance at time t based on the optimal estimate at time t-1. Then, the agent i closest to target j is identified as the observer. Based on the true positions of agents i and j, a noisy observation process is simulated to generate observation values. The standard deviation of the observation noise is proportional to the distance d between the agent and the target. The observation noise covariance matrix is ​​calculated accordingly. Call the update() method in KF, combine the observed values ​​to correct the prior estimate, and obtain the posterior (and optimal) estimated state at time t.

[0047] Step 3: Dynamic Task Allocation

[0048] This step is the core step for the algorithm to realize multi-agent cooperation. The algorithm will first construct an N x M dimensional cost matrix, where N is the current number of agents, and M is the number of unintercepted targets. For each combination of agent i and target j, according to step two, the estimated state of the target and the current state of the agent are obtained, and then the estimated interception time is calculated as the basic cost. In addition, in order to avoid the instability of the training process caused by the back and forth switching of the target by the agent at a special position (such as the intersection of the target trajectory, etc.), a fixed switching penalty term is added to the agent cost calculation . Then, the Hungarian algorithm is used to solve the cost matrix to obtain an assignment scheme that minimizes the total cost. This scheme clearly shows which target each agent should intercept at the current time frame dt. And the assignment can ensure that each target has at least one agent tracking it.

[0049] Step four: Distributed generation of control actions

[0050] For each agent i, a 19-dimensional state vector S is constructed according to the target assigned to it in step three. Its components are strictly concatenated in the following order: the normalized vector of the difference between the target's predicted pursuit point and the agent's position (3 dimensions); the normalized vector of the agent's own speed (3 dimensions); the normalized vector of the target's estimated speed (3 dimensions), the flag indicating whether the target is inside the obstacle (1 dimension), the flag indicating whether the agent is about to collide with the obstacle (1 dimension), the normalized predicted collision time (1 dimension), the recommended collision avoidance direction vector (3 dimensions), the normalized vector of the difference between the obstacle center and the agent's position (3 dimensions), and the normalized vector of the obstacle radius (1 dimension).

[0051] Then, the state vector S i is input into the PPO agent's Actor-Critic network of agent i. The Actor network outputs a mean value μ of an action (a three-dimensional acceleration instruction), which, combined with a learnable logarithmic standard deviation, constructs a normal distribution Normal(μ, ). A action is sampled from this distribution. The Critic network outputs the value estimate V( ,i) of the current state.

[0052] Finally, the sampled action is mapped to the [-1, 1] interval through the Tanh function to obtain the final normalized action .

[0053] Step five: Interaction with the environment and model learning

[0054] In this step, all agents will execute the normalized action they generated in step four , multiply it by the maximum acceleration, and update their state in the environment through the physics engine to get the position and velocity at time t+1. In this process, the code will also calculate the reward each agent i gets in this time frame. And the complete transition information (including state vector, normalized action, state value estimate, collision risk flag, etc.) will be stored in the experience replay buffer of the corresponding agent. When the amount of data accumulated in the buffer of any agent reaches the preset threshold (set to 4096 steps in this invention), the agent will perform a learning update: calculate the advantage function using the general advantage estimation method, and according to the clipping surrogate objective of PPO, replace the loss of the target function and the value function, and finally update the Actor-Critic network parameters through the Adam optimizer.

[0055] Step six: loop execution

[0056] Repeat steps two to five until all targets are intercepted or the maximum simulation step is reached, thus completing a round of training.

[0057] In this way, through a large number of rounds of training, the PPO network of each agent will gradually converge, thus learning to execute an efficient cooperative interception strategy.

[0058] As shown in Figure 2 , Figure 3 , the three agents and their trajectories are represented by red, blue, and green colors respectively, and the two targets moving towards each other and their trajectories are represented by red and blue colors respectively. Their starting points and ending points are represented by round dots and cross-shaped points respectively, and the obstacles are represented by gray hemispherical bodies in space. As can be seen from the figure, after the cooperative combat, target tracking and obstacle avoidance of the three agents, agent 1 successfully captures target 1 and agent 3 successfully captures target 2, achieving the purpose of the invention.

[0059] As shown in Figure 4As shown, the initial assignment of the three agents is agent 1 -> target 2, agent 2 -> target 1, and agent 3 -> target 1. This is because the agents are initially closest to the corresponding targets, and the estimated interception time is the shortest, and thus the cost matrix is the smallest. After the agents cooperate, the optimal assignment is obtained. However, as time goes on, target 1 and target 2 enter the obstacle one after another. At this time, the estimated interception time calculated in the cost matrix is a larger value because the agents avoid the obstacle. After continuous re-assignment and cooperation, the assignment strategy of agent 1 and agent 3 changes, and they switch to chasing new targets, i.e., agent 1 -> target 1 and agent 3 -> target 2. Agent 2, due to its special position (directly above the obstacle), mainly plays a role of "auxiliary tracking". Therefore, there are simulation results such as Figure 2 、 Figure 3 .

[0060] As shown in Figure 5 , the changes in the cost parameters of the two targets and the cooperative assignment relationship of the three agents at different times in the current best round are consistent with Figure 4 . In the figure, the thick solid line represents the pursuit cost, which means the cost evaluation of the agent to the target being currently pursued. This line represents the current selected scheme, and its value is the cost of the current decision, which is the best choice according to the task assignment algorithm. The long dashed line represents the alternative cost, which means the cost evaluation of the agent to the other target (not selected as the pursuit target). This line represents the cost of the "alternative scheme", and the agent continuously monitors this cost. Once this cost becomes lower than the pursuit cost, the decision switching may be triggered. The dotted line is the estimated interception time (ETI), which means the theoretical interception time calculated considering the distance and relative speed. It is the most core input parameter in the total cost calculation. It can be seen that the trend of the dotted line and the total cost line (solid / dashed line) is highly consistent, because ETI is the main factor determining the cost. The dotted dashed line is the physical distance between the agent and the target, which means the most intuitive physical distance between the agent and the target. This line can be used to verify the rationality of the cost calculation. Generally, the closer the distance, the lower the cost. However, sometimes even if the distance is slightly far, if the speed direction of the agent is consistent with that of the target, the ETI and the total cost may be lower. The circle point is the decision switching point, which means marking the exact time point when the agent changes its pursuit target. When a blue circle point appears, it means that the agent switches from pursuing target 1 to target 2 at this moment; when a red circle point appears, it means that it switches from pursuing target 2 to target 1. The appearance of this point is necessarily accompanied by the switching of the solid line and the dashed line. Of course, this switching is completely corresponding to the switching in Figure 4 .

[0061] Therefore, the entire target switching process can be explained as follows: Assuming that at the start of the mission, the red line is solid and the blue line is dashed, such as... Figure 5 (c) This indicates that at the start of the task, the Hungarian algorithm assigned target 1 to agent 3. As time progresses, agent 3 flies towards target 1, so the solid red line (pursuit cost) and its corresponding dashed red distance line can be seen steadily decreasing. Meanwhile, it may be further away from target 2, so the dashed blue line (alternative cost) may be increasing or slowly decreasing (e.g., due to factors like agent-target 2 speed alignment). Suddenly, due to target 2's maneuvering or other agents' coordinated positional changes, the cost of this agent pursuing target 2 (blue dashed line) drops sharply and becomes lower than the current cost of pursuing target 1 (solid red line). During global optimization, the Hungarian algorithm finds that having this agent pursue target 2 while simultaneously coordinating with other agents to pursue target 1 results in the lowest total cost for the entire agent cluster. Therefore, the algorithm issues a target switching command. At this moment, Figure 5 A blue dot appears on (c). Simultaneously, the red line changes from solid to dashed, and the blue line changes from dashed to solid. This signifies the start of a new task for agent 3. Afterward, the agent begins flying towards target 2, at which point both the solid blue line representing pursuit cost and the dashed blue line representing distance begin to decrease.

[0062] Depend on Figure 6 As shown, the intelligent agent can not only accurately avoid obstacles to pursue targets that have passed through them, but also fly close to (without contacting or entering) obstacles, thus efficiently and quickly catching up with its corresponding target. This intuitively demonstrates the future collision perception aspect of this invention. The reinforcement learning-based controller learns and integrates future collision perception (collision risk). Estimated collision time , avoidance vector The state representation of the system enables proactive decision-making. It does not passively avoid collisions, but actively plans safe paths.

[0063] Depend on Figure 7 As shown, the standard deviation curve of the observation noise in KF exhibits a trend of first decreasing, then increasing, and then decreasing again over time. This is because the observation noise in KF, as defined in the invention, is affected by the distance between the agent and the target; the closer the distance, the lower the observation noise. Therefore, this figure also reflects the distance between the agent and the target. Furthermore, this setting aligns with practical physical meaning: the smaller the distance between the agent and the target, the smaller the standard deviation of KF's noise, and the higher the prediction accuracy of KF; in this case, the closer the agent and the target are.

[0064] Figure 8For the agent motor parameter trend chart, the physical quantity directly controlled and optimized by the PPO agent in the application is the three-dimensional acceleration vector of the agent, and the speed and angle of the agent are directly affected by the acceleration. Therefore, the three charts can intuitively reflect the changes of some key parameters of the agent in the process of tracking the target, and also prove the realizability of the application at the physical level.

[0065] Figure 9 The effect of the algorithm in training the agent to track the target can be well embodied. As can be seen from the figure, the total reward is very low at the beginning of the training round, increases with the increase of the number of training rounds, and finally tends to be stable at a relatively high level with only small fluctuations. At this time, the agent has been able to achieve high-quality collaborative allocation of targets, tracking and obstacle avoidance, and almost achieves the purpose of the application. In addition, the smoothness of the reward curve also represents that the agent has learned a robust strategy, so it can successfully complete the task with a high probability and obtain stable and high rewards in the face of different random disturbances under similar initial conditions (such as changes in the initial position of the target).

Claims

1. A multi-agent cooperative dynamic target interception decision-making method based on reinforcement learning, characterized in that, Comprising the following steps: Step 1, initializing the environment and agents; Step 2, predicting the state of the intercepted target; Step 3, assigning the globally optimal task based on the multi-agent predictive interception cost; Step 4, constructing an autonomous control model for each agent based on the Proximal Policy Optimization algorithm (PPO), and designing a state representation vector for future collision perception; inputting the state representation vector into the neural network of the agent, and outputting the action instruction; Step 5, designing a multi-objective reward function to guide the agent's behavior learning during the training process; Step 6, repeating steps 2 to 5 until all intercepted targets are intercepted or the maximum simulation step is reached, completing the multi-agent collaborative dynamic target interception decision.

2. The multi-agent cooperative dynamic target interception decision-making method based on reinforcement learning according to claim 1, characterized in that, Step 1 specifically includes initializing the initial positions and velocities of N agents and the initial states of M targets; initializing M independent Kalman filters for M targets, and initializing N independent PPO agents with the same network structure for N agents; the Actor-Critic network structure of the PPO agent is a multi-layer perception containing two hidden layers.

3. The multi-agent collaborative dynamic target interception decision-making method based on reinforcement learning according to claim 1, characterized in that, The step 2 is specifically: in each time frame dt, for each unintercepted target j, a predict() method in a Kalman filter corresponding to the target is called to calculate a priori estimation and a priori covariance at time t according to optimal estimation at time t-1; a smart body i closest to the target j is determined as an observer, and an observation value is generated by simulating a noisy observation process according to the real positions of the smart body i and the target j; wherein the standard deviation of the observation noise is proportional to the distance d between the smart body and the target; and an observation noise covariance matrix is calculated according to the observation noise ; an update() method in the KF is called to correct the priori estimation by combining the observation value to obtain a posteriori and optimal estimation state at time t.

4. The multi-agent collaborative dynamic target interception decision-making method based on reinforcement learning according to claim 1, characterized in that, Step 3 is specifically: constructing a global optimal task allocation model based on the estimated interception time as the cost, The estimated cost of the agent i intercepting the target j is the estimated interception time calculated dynamically, and the calculation method is as follows: （1） where, , are the current position vectors of agent i and target j, respectively; ‖·‖ denotes the Euclidean norm; is the current position velocity of target j; is the maximum velocity of the agent; is an optional task-switching penalty term to improve the stability of the assignment scheme; the numerator term represents the straight-line distance between agent i and target j; the denominator term considers the projection of the target velocity in the direction of the line connecting the two, dynamically evaluating the interception difficulty of the target: the smaller the value, the closer the target velocity is to the maximum velocity of the agent at this time, and the more difficult it is for the agent to catch up with the target.

5. The multi-agent collaborative dynamic target interception decision-making method based on reinforcement learning according to claim 1, characterized in that, Step 4 specifically includes constructing an autonomous control model for each agent based on the Proximal Policy Optimization algorithm (PPO); and ensuring the stability of the training process by clipping the objective function, the PPO clipping objective function is: （2） wherein, are parameters of the policy network; represents the expectation at time step t; is the probability ratio of the new and old policy; is the advantage function estimate; clip() is a clipping function that limits the value of to the interval ; is a hyperparameter that defines the range of clipping; by taking the smaller value of the unclipped term and the clipped term using the min() function, the stability of the policy update is guaranteed; Designing state representation vectors S for future collision awareness i : （3） wherein, represents a collision risk flag bit, the agent predicts whether a collision with an obstacle will occur in the current speed direction by means of ray casting, and if so, is 1, otherwise 0; t tti is a normalized estimated collision time, v esc is a recommended evasion vector; The state vector S i is input into the Actor-Critic network of the agent, the Actor network outputs the mean value μ of the action instruction, and the learnable logarithmic standard deviation is combined to construct a normal distribution Normal(μ, ), std , where σ represents the standard deviation; a sampled action from the normal distribution; the Critic network outputs the value estimation of the current state; and the sampled action is mapped to the interval [-1, 1] through a Tanh function to obtain the final normalized action .

6. The multi-agent collaborative dynamic target interception decision-making method based on reinforcement learning according to claim 1, characterized in that, Step 5 specifically is: all agents perform normalization action , multiply by the maximum acceleration, and update its state in the environment through the physics engine to obtain the position and velocity at time t+1; store the transition information into the experience replay buffer of the corresponding agent, and when the amount of data accumulated in the buffer of any agent reaches a preset threshold, the agent performs a learning update: （4） where, represents a progress reward, represents a velocity alignment reward, represents an end-effector potential field reward, represents an obstacle avoidance penalty, and represents a time and energy consumption penalty, represents the weight of each term, k ∈ [1, 6].

7. The multi-agent collaborative dynamic target interception decision-making method based on reinforcement learning according to claim 6, characterized in that, The transition information includes state vector, normalized action, state value estimation and collision risk flag.

8. A computer apparatus comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program, when executed by the processor, causes the processor to perform the method of any one of claims 1 to 7. The processor executes the computer program to realize the steps of the method of claim 1.

9. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, The computer program / instructions are executed by the processor to realize the steps of the method of claim 1.

10. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions are executed by the processor to realize the steps of the method of claim 1. The computer program / instructions are executed by the processor to realize the steps of the method of claim 1.

Images