Multi-agent cluster cooperative attack and defense game method based on near-end strategy optimization

Abstract

The invention discloses a multi-agent cluster cooperative attack and defense game method based on near-end strategy optimization, and relates to the technical field of automatic control, and the method comprises the steps: constructing a partial considerable Markov decision process, and generating a multi-agent decision model; establishing a cooperative attack and defense game simulation framework including an attacker agent, a defender agent, a target area and irregular obstacles in the preset task area; parallel trajectory sampling is executed, an observation-action-reward sequence of each agent is collected, and a training data set is generated; a centralized state is constructed by cascading and splicing local observation of each agent, a time sequence difference error and a dominant function are calculated, network parameter updating is carried out by adopting cutting strategy gradient and value function regression, and a trained cooperative attack and defense game strategy network is generated when a reward curve converges. And the multi-agent cluster realizes intelligent cooperative attack and defense game and autonomous obstacle avoidance in a task environment with irregular obstacles.

Description

Technical Field

[0001] This invention relates to the field of automatic control technology, and in particular to a multi-agent cluster collaborative attack and defense game method based on near-end strategy optimization. Background Technology

[0002] Currently, multi-agent cooperative attack-defense game tasks have been widely applied in fields such as military defense and public safety, demonstrating significant application value. However, existing methods often suffer from insufficient cooperative efficiency, difficulty in balancing obstacle avoidance and attack / defense objectives, and unstable strategies when dealing with complex factors such as environmental obstacles, communication delays, and local observations. Therefore, there is an urgent need to research an efficient, stable, and environmentally adaptable multi-agent cluster cooperative attack-defense game method to improve the reliability and intelligence level of task execution.

[0003] Multi-agent proximal policy optimization (MAP) is a deep reinforcement learning method applicable to multi-agent systems, belonging to the online policy category, and employing the classic Actor-Critic architecture in reinforcement learning. The algorithm aims to find the optimal policy to generate the optimal actions of the agents. Compared to proximal policy optimization methods in single-agent scenarios, MAP introduces a centralized global value function to fuse global information, adopting a "centralized training, distributed execution" framework. During training, a centralized critic is used to obtain global information, improving the overall decision-making quality; during execution, each agent makes independent decisions based on local observations, achieving distributed action execution and supporting collaborative cooperation within the cluster. Furthermore, the algorithm introduces a pruning mechanism in the objective function, effectively limiting the gradient update magnitude, improving instability during training, and making the policy more likely to converge and maintain stability.

[0004] Therefore, there is an urgent need for a multi-agent cluster collaborative attack and defense game method based on near-end strategy optimization. Summary of the Invention

[0005] This invention provides a multi-agent cluster collaborative attack and defense game method based on near-end strategy optimization to solve the above-mentioned problems in the prior art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A multi-agent cluster cooperative attack and defense game method based on proximal policy optimization includes: S1: Construct a partially observable Markov decision process that includes an action space, an observation space, and a reward function to generate a multi-agent decision model; S2: Based on a multi-agent decision-making model, a collaborative attack and defense game simulation framework is established within a preset task area, including attacking agents, defending agents, target areas, and irregular obstacles. S3: Perform parallel trajectory sampling based on the simulation framework, collect the observation-action-reward sequences of each agent, and generate a training dataset; S4: Based on the training dataset, a centralized state is constructed by cascading and splicing the local observations of each agent. The temporal difference error and the advantage function are calculated. The network parameters are updated by using the pruning policy gradient and the value function regression. When the reward curve converges, the trained collaborative attack and defense game strategy network is generated.

[0007] Furthermore, step S1 includes: S11: Set the range of velocity vector values ​​for the attacking agent. Construct a two-dimensional continuous motion space containing lateral velocity components and longitudinal velocity components; S12: To meet the task requirements, construct a six-dimensional observation vector as the observation space, which includes the relative position of the target area, the relative position of the nearest defender, and the relative projection position of the nearest obstacle edge. S13: Based on the task objective, design a comprehensive reward function that includes distance change reward, speed reward, collision penalty, success reward and capture penalty, and complete the construction of a multi-agent decision model.

[0008] Furthermore, in step S12: The six-dimensional observation vector includes the lateral and longitudinal distances relative to the center of the target area, the lateral and longitudinal distances relative to the nearest defending agent, and the lateral and longitudinal distances relative to the nearest obstacle edge projection point.

[0009] Furthermore, step S2 includes: S21: Set the mission area size, target area size, number of attacking agents, number of defending agents, and the defending capture radius; S22: Arrange irregular polygonal obstacles within the task area, randomly and evenly distribute the initial positions of the attacking and defending agents, and generate a collaborative attack and defense game simulation framework.

[0010] Furthermore, step S3 includes: S31: Within the simulation framework, the network performs action selection according to the current policy, records the local observations, actions performed, and immediate rewards of each agent at each time step, and generates an interaction trajectory of a preset number of steps T. S32: Based on the interaction trajectory, extract the observation-action-reward sequence of each agent and the cascaded centralized state sequence to generate a training dataset.

[0011] Furthermore, step S4 includes: S41: Cascade and stitch together the local observation vectors of all agents at the same time step to generate a centralized state vector; S42: Based on the centralized state vector, the value estimate of the current state and the value estimate of the next state are calculated through the centralized value function. The time difference error is calculated by combining the immediate reward and the discount factor. The advantage function is generated by the time difference error sequence of the accumulated discount. S43: Based on the advantage function, calculate the action probability ratio between the current policy and the old policy, constrain the range of the probability ratio through the pruning mechanism and construct the policy loss, calculate the mean square error of value prediction and reward target to construct the value loss, and then perform gradient descent to update the network parameters after weighted combination of policy loss and value loss. S44: Repeat steps S41 to S43 until the global reward curve converges, then terminate the training and generate a collaborative attack and defense game strategy network.

[0012] Furthermore, in step S42: Timing Differential Error The calculation formula is:

[0013] in For the instant reward of time step t, As a discount factor, The centralized state of the centralized value function at time step t Valuation of the location Value estimation for time step t+1; Advantage function The calculation formula is:

[0014] Where λ is the generalized dominance estimation coefficient, T is the trajectory length, and the summation range is... From 0 to .

[0015] Furthermore, in step S43: Action probability ratio The calculation formula is:

[0016] in For the current policy network, For old policy networks, For the action of agent i at time step t, For the local observation of agent i at time step t; The formula for calculating strategy loss is:

[0017] in The cropping threshold, The entropy regularity coefficient is... For policy entropy; The formula for calculating value loss is:

[0018] in The return objective equals the advantage function. With value estimation The sum of.

[0019] Furthermore, step S13 includes: The comprehensive reward function includes a reward for changes in distance to the target area. ,in This represents the distance from the agent to the center of the target region. To adjust the hyperparameters for the weights; Speed ​​increase bonus ,in This represents the agent's current velocity scalar value. To adjust the hyperparameters for the weights; Penalty for colliding with obstacles When there is no collision, the value is 0, where, To adjust the hyperparameters for the weights; Rewards for successfully entering the target area When not entered, the value is 0, where, To adjust the hyperparameters for the weights; Penalty when captured by the defending side When not captured, the value is 0, where, To adjust the hyperparameters for the weights; The overall reward function is: .

[0020] Compared with the prior art, the present invention has the following advantages: The purpose of this invention is to provide a multi-agent cluster cooperative attack and defense game method based on proximal policy optimization. Control engineers can implement multi-agent cluster cooperative attack and defense game tasks by following this method and combining it with task scenario parameters. This invention, based on a proximal policy optimization multi-agent cluster cooperative attack and defense game algorithm, achieves effective learning of the attacker's intelligent strategy in attack and defense game tasks. A partially observable Markov decision process is established, including: designing the agent action space, observation space, and reward function; establishing a cooperative attack and defense game task simulation framework, conducting policy training, and obtaining the multi-agent cooperative attack and defense game strategy. Applying the training results to actual pursuit tasks enables autonomous cooperation and decision-making among multiple agents. Attached Figure Description

[0021] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a multi-agent cluster collaborative attack and defense game method based on near-end strategy optimization in an embodiment of the present invention; Figure 2 This is a simulation framework diagram of the collaborative attack and defense game task in an embodiment of the present invention; Figure 3 This is a typical training process reward function curve diagram in an embodiment of the present invention; Figure 4 This is a flowchart illustrating the typical training process of the reward function in an embodiment of the present invention. Detailed Implementation

[0022] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0023] The embodiments of the present invention provide, as follows Figure 1 As shown, a multi-agent cluster cooperative attack and defense game method based on near-end policy optimization includes: S1: Construct a partially observable Markov decision process that includes an action space, an observation space, and a reward function to generate a multi-agent decision model; S2: Based on a multi-agent decision-making model, a collaborative attack and defense game simulation framework is established within a preset task area, including attacking agents, defending agents, target areas, and irregular obstacles. S3: Perform parallel trajectory sampling based on the simulation framework, collect the observation-action-reward sequences of each agent, and generate a training dataset; S4: Based on the training dataset, a centralized state is constructed by cascading and splicing the local observations of each agent. The temporal difference error and the advantage function are calculated. The network parameters are updated by using the pruning policy gradient and the value function regression. When the reward curve converges, the trained collaborative attack and defense game strategy network is generated.

[0024] The following is a detailed description with reference to specific embodiments.

[0025] This embodiment provides a multi-agent cluster cooperative attack and defense game method based on near-end strategy optimization, including the following steps: S1: Construct a partially observable Markov decision process that includes an action space, an observation space, and a reward function to generate a multi-agent decision model.

[0026] Specifically, it includes the following sub-steps: S11: Set the range of velocity vector values ​​for the attacking agent. Construct a two-dimensional continuous motion space containing lateral velocity components and longitudinal velocity components.

[0027] In multi-agent pursuit scenarios, the action space of the attacking agent is defined as a two-dimensional velocity vector, representing the direction and magnitude of its movement within the task area. Specifically: Dimensions: Two-dimensional, representing the velocity in the x-direction and the velocity in the y-direction, respectively; Value range: Each velocity component is within It takes continuous values ​​within a range.

[0028] By adjusting the velocity vector, the attacking agent can implement strategies such as moving towards the target area, evading the defending agent, and avoiding obstacles.

[0029] S12: To meet the mission requirements, construct a six-dimensional observation vector as the observation space, which includes the relative position of the target area, the relative position of the nearest defender, and the relative projection position of the nearest obstacle edge.

[0030] The observer space of the attacking agent is defined as the local environmental information it can acquire, used as input for policy decisions. According to the task requirements, the six-dimensional observation vector consists of the following three parts: ① Position relative to the target area : This represents the relative coordinates between the attacking agent's current position and the center point of the target area. ,in , , The x and y coordinates of the center point of the target area are respectively. These are the current horizontal and vertical coordinates of the agent.

[0031] ② Relative to the position of the nearest defending team : This represents the relative distance and direction between the attacking agent and the nearest defending agent. It is calculated using Euclidean distance. ,in These are the current horizontal and vertical coordinates of the defending agent.

[0032] ③ Projected position relative to the nearest obstacle edge : This represents the shortest relative coordinates from the attacking agent to the projection point of the nearest obstacle edge. The choice is made by calculating the distances between the agent and the projection points of all edges of the obstacle. The minimum value is taken as the observed value.

[0033] The observation space is ultimately represented in vector form, as a fixed-length feature vector with a dimension of 6, containing the three parts of information mentioned above: .

[0034] S13: Based on the task objective, design a comprehensive reward function that includes distance change reward, speed reward, collision penalty, success reward and capture penalty, and complete the construction of a multi-agent decision model.

[0035] The reward function directly influences the direction of policy optimization. For the attacking agent's mission objective, the comprehensive reward function consists of the following parts: (1) Reward for changes in distance to the target area: The reward increases if the attacking agent approaches the target area. This reward incentivizes the agent to move closer to the target area. Reward formula: ,in It is the distance from the current agent to the center of the target region. It is a hyperparameter for adjusting weights.

[0036] (2) Rewards for increased speed: The reward increases if the attacking agent's speed increases. This reward incentivizes the agent to increase its speed. Reward formula: ,in It is the current scalar value of the agent's combined velocity. It is a hyperparameter for adjusting weights.

[0037] (3) Penalty for colliding with the boundary of an obstacle: When the attacking agent does not collide with an obstacle, the reward remains unchanged; when a collision occurs, a larger penalty score is applied. This item is used to penalize the agent's behavior of approaching the obstacle. The penalty values ​​are as follows: .

[0038] (4) Rewards for completing the task A significant one-time reward is given when the attacking agent successfully enters the target area. The reward value is shown below: .

[0039] (5) Punishment for arrest When an attacking agent is captured by the defending agent, a significant negative reward is immediately imposed. The penalty values ​​are shown below: .

[0040] (6) Total Reward Formula The total reward function combines all the above parts:

[0041] This completes the construction of the multi-agent decision-making model, including the design of the action space, observation space, and reward function.

[0042] S2: Based on a multi-agent decision-making model, a collaborative attack and defense game simulation framework is established within a preset task area, including attacking agents, defending agents, target areas, and irregular obstacles.

[0043] Specifically, it includes the following sub-steps: S21: Set the mission area size, target area size, number of attacking agents, number of defending agents, and defender capture radius.

[0044] In this embodiment, typical parameter settings are as follows: The task area is a square area of ​​700m × 700m; The target area is a square area of ​​120m × 120m, located at a predetermined position within the mission area; The attacking agent has 10 agents. The number of defending agents is 10; The capture radius of the defending agent is 5m, meaning that when the distance between the attacking agent and the defending agent is less than 5m, the attacking agent is considered to be captured.

[0045] S22: Arrange irregular polygonal obstacles within the task area, randomly and evenly distribute the initial positions of the attacking and defending agents, and generate a collaborative attack and defense game simulation framework.

[0046] Several irregular polygonal obstacle areas are arranged within the task area to simulate obstacles in a real-world task environment. The initial positions of the attacking and defending agents are randomly and uniformly distributed within the task area to ensure the diversity of each simulation scenario.

[0047] like Figure 2 The diagram shows a typical scenario of a collaborative attack-defense game simulation framework. Large red dots represent defending agents, blue dots represent attacking agents, yellow dots represent attacking agents that have successfully entered the target area, small red dots represent captured attacking agents, green areas represent the target area, and gray areas represent irregular polygonal obstacle areas.

[0048] The mission objectives are defined as follows: The attacking agent's objective is to evade the defending agent's pursuit and enter the target area. If the agent enters the target area, the mission is considered successful. The defending agent's objective is to pursue and capture the attacking agent. When the distance between the attacking and defending agents is less than the defending agent's capture radius, the attacking agent is considered captured.

[0049] At this point, the collaborative attack and defense game simulation framework has been established.

[0050] S3: Based on the simulation framework, parallel trajectory sampling is performed to collect the observation-action-reward sequences of each agent and generate a training dataset.

[0051] Specifically, it includes the following sub-steps: S31: Within the simulation framework, the network selects actions according to the current policy, records the local observations, actions performed, and immediate rewards of each agent at each time step, and generates an interaction trajectory of a preset number of steps T.

[0052] In the simulation environment, each attacking agent operates according to its current policy network. Based on their respective local observations Select Action And execute. The environment transitions states based on the actions of all agents, returning to the new observation. and instant rewards .

[0053] The maximum number of steps in a single scenario is set to 200 steps, the total number of training steps is 100,000, and the learning rate is 0.0001. Training is considered effective when the reward curve converges. A typical training curve is shown in Figure 2. After training, a multi-agent cooperative attack and defense game strategy is obtained. At each time step t (t = 0, 1, …, T-1), the following information is recorded: Each attacking agent ( Local observations ; Actions performed by each attacking agent i ; Global instant rewards returned by the environment ; Next-moment observations of each attacking agent i ; Parallel sampling allows multiple simulation environment instances to be run simultaneously, improving data collection efficiency.

[0054] S32: Based on the interaction trajectory, extract the observation-action-reward sequence of each agent and the cascaded centralized state sequence to generate a training dataset.

[0055] Extract and construct a training dataset from the sampled interaction trajectories:

[0056] in, For intelligent agents Local observations at time step t, For intelligent agents The action performed at time step t The global instant reward for time step t. For the local observation of agent i at time step t+1.

[0057] To support centralized training, a centralized state vector is constructed by cascading and concatenating the local observation vectors of all agents at the same time step:

[0058] Here, concat represents the vector concatenation operation. Let t be the centralized state vector at time step t, which contains the local observation information of all agents.

[0059] This generates a complete training dataset containing observation-action-reward sequences and centralized state sequences.

[0060] S4: Based on the training dataset, a centralized state is constructed by cascading and splicing the local observations of each agent. The temporal difference error and the advantage function are calculated. The network parameters are updated by using the pruning policy gradient and the value function regression. When the reward curve converges, the trained collaborative attack and defense game strategy network is generated.

[0061] Specifically, it includes the following sub-steps: S41: Cascade and concatenate the local observation vectors of all agents at the same time step to generate a centralized state vector.

[0062] As described in S32, a centralized state vector is generated through a cascading concatenation operation. This provides global information input for the centralized value function.

[0063] S42: Based on the centralized state vector, the value estimate of the current state and the value estimate of the next state are calculated through the centralized value function. The time difference error is calculated by combining the immediate reward and the discount factor. The advantage function is generated by the time difference error sequence of the accumulated discount.

[0064] (1) Calculation of timing difference error Centralized value function Used to estimate the value of a centralized state x. Based on the value function, the temporal difference (TD) error at time step t is calculated. :

[0065] in, For the instant reward of time step t, As a discount factor, The centralized state of the centralized value function at time step t Valuation of the location The value estimate for time step t+1.

[0066] (2) Dominance function estimation The generalized advantage estimation (GAE) method is used to generate the advantage function from the cumulatively discounted time-series difference error sequence. :

[0067] in, Here are the generalized dominance estimation coefficients, T is the trajectory length, and the summation range is... For the range from 0 to Tt-1, the advantage function represents the state... The degree of advantage of taking action relative to the average level.

[0068] (3) Calculation of return target Calculate the return target based on the advantage function and value estimation. :

[0069] Before training, the advantage function was evaluated. Normalization with zero mean and unit variance is performed to stabilize the optimization process.

[0070] S43: Based on the advantage function, calculate the action probability ratio between the current policy and the old policy, constrain the range of the probability ratio through the pruning mechanism and construct the policy loss, calculate the mean square error of value prediction and reward target to construct the value loss, and then perform gradient descent to update the network parameters after weighted combination of policy loss and value loss.

[0071] (1) Calculation of action distribution and probability ratio The policy for each agent is distributed using a Gaussian distribution and compressed within the bounded action space.

[0072]

[0073]

[0074] in, The raw actions output by the policy network are compressed to the range of (-1, 1) using the tanh function, and then mapped to the actual velocity range using the scale function. For the tanh transform, the logarithmic probability needs to be compensated according to the rules of invertible transform.

[0075] Define the current policy Compared to the old strategy The probability of action is higher than : right The logarithmic probability of the transformation is compensated according to the rules of invertible transformation. Define the policy probability ratio:

[0076] in, For the current policy network, For old policy networks, For the action of agent i at time step t, For the local observation of agent i at time step t.

[0077] (2) Construction of strategy loss A pruning mechanism is used to constrain the policy update magnitude, and a pruning policy objective function is constructed. The expectation is calculated for all agents and time steps, maximizing the following expression:

[0078] in, This is the pruning threshold, used to limit the probability ratio. The range of variation is within [1-ε, 1+ε]; This is the entropy regularization coefficient, used to encourage strategy exploration; Let be the policy entropy, and calculate the entropy value of the policy distribution.

[0079] (3) Value loss construction Value loss is calculated using the mean squared error method to determine the difference between the predicted value and the target return.

[0080] in, In order to achieve the goal, .

[0081] (4) Total loss and network parameter update We construct the total loss function by weighting the strategy loss and value loss of the combined strategy:

[0082] in, This is the value loss weighting coefficient. The weighting coefficients for the KL divergence penalty are: KL divergence is used to measure the difference between the current policy and the old policy.

[0083] The Adam optimizer is used to optimize the total loss function using gradient descent, updating the policy network parameters θ and the value network parameters φ. During training, a mini-batch stochastic gradient descent approach is employed, dividing the dataset into multiple mini-batches for K epoch updates.

[0084] To prevent performance degradation due to excessively large policy updates, an early stopping or adaptive step size strategy is adopted: when If necessary, terminate the strategy update for the current round or reduce the learning rate accordingly.

[0085] S44: Repeat steps S41 to S43 until the global reward curve converges, then terminate the training and generate a collaborative attack and defense game strategy network.

[0086] The training process consists of the following four steps: Step 1: Parallel environmental sampling and data collection ,Record ; Step 2: Centralized input Evaluate , ,calculate ; Step 3: Divide the data into small batches and perform... Epoch: Maximize , minimize and monitor ; Step 4: Synchronize and Update , Then proceed to the next round of sampling.

[0087] In this embodiment, the total number of training steps is set to 100,000, and the learning rate is set to 0.0001. As training progresses, the global reward gradually increases and then stabilizes. Figure 3 The figure shows the reward function curve of a typical training process. When the reward curve converges, the training is considered effective, the training process is terminated, and the trained collaborative attack and defense game strategy network is obtained.

[0088] After training, the multi-agent cluster can achieve autonomous collaborative decision-making and control in actual attack and defense scenarios based on the trained policy network. For example... Figure 4 As shown, local observation Input their respective policy networks The desired two-dimensional velocity action is generated and executed by the actuator to achieve autonomous maneuvering of the intelligent agent; the centralized commentator observes through global stitching. Estimating the value function This allows the collaborative relationships and game dynamics among multiple agents to be reflected in the advantage assessment. The design of the reward function directly participates in the calculation of the advantage function and loss, driving continuous strategy optimization, enabling the cluster to achieve autonomous decision-making and control in attack and defense tasks, including "efficient penetration, proactive avoidance, and safe obstacle avoidance".

[0089] In application, control engineers can freely set parameters such as the number of units in the intelligent agent cluster, the size of the task area, and the layout of obstacles according to the actual task scenario. The policy network trained by this method is used to calculate the expected speed of each intelligent agent and solve the control quantity, which is then transmitted to the actuator, thereby realizing the automatic control of multi-agent cluster collaborative attack and defense game tasks.

[0090] The advantages of this invention are: (1) This method effectively improves the autonomous decision-making and collaborative ability of the agent under the conditions of irregular obstacles and defensive interference by establishing a partially observable Markov decision process and combining it with the near-end policy optimization algorithm, and significantly improves the robustness and stability of task execution.

[0091] (2) This method adopts a framework of centralized training and distributed execution, which can make full use of global information for policy learning, while ensuring the independence and flexibility of each agent in the execution process, thereby achieving more efficient multi-agent collaborative attack and defense game in complex environments.

[0092] (3) This method takes into account multiple factors such as target approach, speed improvement, obstacle avoidance penalty, task completion reward and capture penalty in the design of reward function, so that the direction of agent strategy optimization is clear, and it can take into account both penetration efficiency and obstacle avoidance safety, thereby improving the overall task completion rate.

[0093] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from the spirit and scope of this invention.

Claims

1. A multi-agent swarm cooperative attack-defense game method based on proximal policy optimization, characterized in that, The method comprises the following steps: S1: constructing a partially observable Markov decision process comprising an action space, an observation space and a reward function, and generating a multi-agent decision model; S2: based on the multi-agent decision model, establishing a cooperative attack and defense game simulation framework comprising attacking agents, defending agents, target areas and irregular obstacles in a preset task area; S3: based on the simulation framework, performing parallel trajectory sampling, collecting observation-action-reward sequences of each agent, and generating a training data set; S4: based on the training data set, constructing a centralized state by concatenating the local observations of each agent, calculating the temporal difference error and advantage function, and updating the network parameters by using the clipped policy gradient and value function regression, and generating the trained cooperative attack and defense game strategy network when the reward curve converges.

2. The method of claim 1, wherein, The step S1 comprises: S11: set the value range of the speed vector of the attacking agent , a two-dimensional continuous action space containing a lateral speed component and a longitudinal speed component is constructed; S12: for the task requirement, a six-dimensional observation vector comprising the relative target area position, the relative nearest defender position and the relative nearest obstacle edge projection position is constructed as the observation space; S13: based on the task target, a comprehensive reward function comprising distance change reward, speed reward, collision penalty, success reward and capture penalty is designed, and the multi-agent decision model construction is completed.

3. The method of claim 2, wherein, In the step S12: The six-dimensional observation vector comprises the lateral distance and longitudinal distance of the relative target area center, the lateral distance and longitudinal distance of the relative nearest defender agent, and the lateral distance and longitudinal distance of the relative nearest obstacle edge projection point.

4. The method of claim 1, wherein, The step S2 comprises: S21: setting the task area size, target area size, number of attacking agents, number of defending agents and defense capture radius; S22: irregular polygon obstacles are arranged in the task area, and the initial positions of the attacking agents and the defending agents are randomly and uniformly distributed to generate the cooperative attack and defense game simulation framework.

5. The method of claim 1, wherein, The step S3 comprises: S31: in the simulation framework, the action selection is performed according to the current strategy network, the local observation, the executed action and the immediate reward of each agent at each time step are recorded, and the interaction trajectory with a length of preset steps T is generated; S32: based on the interaction trajectory, the observation-action-reward sequence and the concatenated centralized state sequence of each agent are extracted to generate the training data set.

6. The method of claim 1, wherein, The step S4 comprises: S41: concatenating the local observation vectors of all agents at the same time step to generate a centralized state vector; S42: based on the centralized state vector, the value estimate of the current state and the value estimate of the next state are calculated through the centralized value function, the temporal difference error is calculated by combining the immediate reward and the discount factor, and the advantage function is generated by accumulating the discounted temporal difference error sequence; S43: based on the advantage function, the action probability ratio of the current strategy and the old strategy is calculated, the probability ratio range is constrained through the clipping mechanism, the strategy loss is constructed, the mean square error of the value prediction and the return target is calculated to construct the value loss, and the network parameters are updated by gradient descent after weighted combination of the strategy loss and the value loss; S44: the steps S41 to S43 are repeatedly executed, and the training is terminated when the global reward curve converges to generate the cooperative attack and defense game strategy network.

7. The method of claim 6, wherein the method further comprises: In the step S42: Timing difference error The calculation formula is: ； wherein is the immediate reward for time step t, is a discount factor, is the value estimate of the centralized value function at the centralized state at time step t, is the value estimate for time step t+1; Advantage function The formula for calculating the advantage function is: ； where λ is the generalized dominance coefficient, T is the length of the trajectory, and the summation ranges from from 0 to .

8. The method of claim 6, wherein the method is characterized by, In the step S43: Action probability ratio The formula for calculating the action probability ratio is: ； wherein is the current policy network, is the old policy network, is the action of agent i at time step t, is the local observation of agent i at time step t; The calculation formula of the strategy loss is: ； wherein is a clipping threshold, is an entropy regularization coefficient, is a policy entropy; The formula for calculating the value loss is: ； where is the return target, equal to the advantage function and the value estimate sum.

9. The method of claim 2, wherein, The step S13 comprises: The composite reward function includes a distance to the target region variation reward wherein is the distance of the agent to the center of the target region, is a hyperparameter that adjusts the weight; speed increase reward wherein is a current speed scalar value for the agent, is a hyperparameter for the adjustment weight; Penalty for collision with obstacle 0 if no collision, where is a hyperparameter that adjusts the weight; a reward for successfully entering the target area , 0 otherwise, is a hyperparameter that adjusts the weight; Penalty for being captured by the defending side , 0 otherwise, is a hyperparameter that adjusts the weight; The comprehensive reward function is: .

Images