Multi-agent cooperative hunting escape method based on adversarial strategy attack

Abstract

The application discloses a kind of multi-agent cooperative hunting escape method based on confrontation strategy attack, it is related to artificial intelligence security and multi-water unmanned ship cooperative control technical field.The method of the present application first constructs the simulation environment of cooperative hunting-confrontation escape for the current multi-water unmanned ship cooperative hunting scene, deploys multiple pursuer ships with pre-trained cooperative strategy, obtains the complete knowledge of cooperative strategy reward function, constructs the confrontation reward function of escapee ship, then runs multiple instances in parallel in simulation environment, collects the experience data of pursuer and escapee interaction, trains Actor network based on proximal policy optimization algorithm to output escape decision, optimizes network parameters using the generalized advantage estimation value of experience data, and finally the obtained Actor network can be used to test and optimize the existing cooperative strategy.The method of the present application significantly improves the attack efficiency and training stability, and provides a solid foundation for accurately quantifying the security risk of hunting system.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence security, multi-agent reinforcement learning, and multi-surface unmanned vessel cooperative control, specifically to a multi-agent cooperative encirclement and escape method based on adversarial strategy attacks. Background Technology

[0002] Multi-Agent Cooperative Pursuit is a key research scenario at the intersection of artificial intelligence, control theory, and multi-agent systems (MAS), with extremely high application value. Typical applications include security patrols using drone swarms, collaborative law enforcement using unmanned surface vessels (USVs), and distributed security network management in smart cities. In dynamic target cooperative pursuit missions using USVs, such as intercepting escaped speedboats suspected of illegal activities or carrying out maritime security blockades, each USV is considered an agent. In this scenario, multiple pursuing agents need to use highly strategic coordination and collaborative movement based on environmental information to surround and capture an escaped speedboat within a specified area and time. Achieving continuous tracking, effective surveillance, and containment of dynamic targets through multi-vessel cooperation is a crucial capability of cooperative pursuit systems. In recent years, with the breakthroughs in Deep Reinforcement Learning (DRL) technology, training hunter strategies for cooperative encirclement systems using advanced algorithms such as Multi-agent Proximal Policy Optimization (MAPPO) has become a mainstream technical approach. These agents, trained on a large scale, often exhibit sophisticated cooperative encirclement strategies, such as flanking maneuvers and encirclement, demonstrating performance far exceeding that of traditional rule-based methods.

[0003] However, the inherent complexity and vulnerability of deep neural network models often result in serious security risks for collaborative capture systems when facing malicious adversarial interference. To accurately assess the robustness of these systems and uncover potential vulnerabilities, existing technologies have undergone a process of development from simple to complex. Early evaluation methods primarily relied on rule-based or heuristic algorithms such as the Artificial Potential Field (APF) and Min-Max algorithms to control escapees. While computationally simple, these methods exhibited rigid and monotonous behavioral patterns, lacking game theory thinking and failing to simulate highly intelligent escape targets with strong counter-surveillance capabilities. Subsequent research drew inspiration from attack methods in computer vision, proposing observational perturbation-based approaches. This involves superimposing minute noise, imperceptible to the human eye, onto the sensor input of the pursuer to mislead the neural network. However, this primarily targets the perception layer and is extremely difficult to implement in real-world physical environments. Currently, the most threatening cutting-edge technology is adversarial strategy training based on the black-box testing concept. In this approach, the attacker exists as a legitimate agent in the environment, engaging in numerous self-games with the victim in a simulated environment through reinforcement learning. In the field of adversarial attacks, so-called black-box testing typically refers to attack training conducted by an attacker who has no knowledge of the target model's internal structure, parameter weights, and gradient information, relying solely on input observations and win / loss reward feedback. While this method can induce the victim to fail through physically legitimate bizarre actions, it still has significant limitations when dealing with high-dimensional, dynamic, and asymmetric cooperative trapping environments.

[0004] The limitations of existing adversarial strategy research in cooperative encirclement scenarios are mainly reflected in the following three aspects:

[0005] (1) Existing attack methods are inefficient and lack specificity. Most studies are based on the black-box attack assumption, which cannot utilize the internal information of the victim's strategy and can only explore its weaknesses through a lot of trial and error. At the same time, the reward design of adversarial strategies is usually based on simple task win or loss or distance indicators, lacking a deep understanding of the victim's decision-making mechanism, making it difficult to generate effective targeted attack behaviors.

[0006] (2) Narrow applicability of scenarios and lack of targeted attacks on cooperative mechanisms: Existing research focuses on symmetrical competitive environments such as robot soccer and sumo wrestling, with insufficient research on asymmetrical cooperative encirclement scenarios. In particular, in cooperative encirclement, the pursuer achieves effective encirclement through complex cooperative mechanisms, but existing methods lack targeted exploitation of the weaknesses of such cooperative mechanisms.

[0007] (3) The existing evaluation system is imperfect. The evaluation mainly relies on surface indicators such as win rate, and lacks in-depth analysis of the internal state of the strategy and the attack mechanism. For example, whether the attack takes advantage of speed or successfully destroys the cooperative structure of the pursuer, existing methods often cannot provide a quantitative explanation.

[0008] Therefore, it is necessary to study a more powerful escape strategy for high-dimensional, dynamic, and asymmetric cooperative encirclement environments of multi-surface unmanned surface vessels (MSVs). This escape strategy should enable the user to escape more quickly and effectively in such environments. Furthermore, it should be used to evaluate and test existing multi-surface unmanned surface vessel cooperative encirclement systems in these environments, thereby exposing their vulnerabilities. This is a problem that needs to be solved. Summary of the Invention

[0009] To address the high-dimensional, dynamic, and asymmetric cooperative encirclement environment of multiple unmanned surface vessels (USVs), and considering the problems of existing adversarial strategies in cooperative encirclement scenarios, this invention proposes a multi-agent cooperative encirclement escape method based on adversarial strategy attacks. This method combines white-box testing with adversarial strategy optimization to implement an escape strategy that enables faster and more effective escape from multi-agent cooperative encirclement systems. This allows for the evaluation and testing of multi-USV cooperative encirclement systems. In this invention's application scenario, white-box testing specifically refers to granting the escapee agent access to critical information within the pursuer system during the simulation training phase, particularly the complete structure of the pursuer's reward function, its weight coefficients, and penalty logic.

[0010] This invention provides a multi-agent cooperative encirclement and escape method based on adversarial strategy attacks, applied to a multi-surface unmanned surface vessel cooperative encirclement scenario involving N pursuer agents and 1 escaper agent. The escaper agent possesses complete knowledge of the pursuer's reward function. This invention utilizes this complete knowledge to design a highly targeted adversarial reward mechanism and trains an efficient escape strategy through reinforcement learning. The method includes the following:

[0011] Step 1: Construct a collaborative encirclement and counter-escape simulation environment. This includes setting the activity area of ​​the current multi-surface unmanned surface vessel collaborative encirclement scenario. The scenario includes: N pursuing agents representing unmanned patrol boats used by the coast guard for collaborative encirclement, each with its own observation space, whose action space typically consists of discrete changes in heading or speed; and one escape agent representing an illegal speedboat suspected of violating the law, also including its observation and action spaces. Establish dynamic models for each agent and pre-train the pursuing agent collaborative strategy, deploying it to the N pursuing agents. The task of the current scenario is that, within a specified time and area, the pursuing agents, through collaborative movement, must ensure that all pursuing agents enter a "capture zone" with the escapee as the center and a certain radius. The escapee's task is to avoid entering the capture zone, preventing this from happening.

[0012] Step 2: Obtain the reward components of the hunter's cooperative strategy and construct the escapee's adversarial reward function. The core innovation of this invention lies in designing a highly targeted adversarial reward using white-box knowledge of the hunter's reward function. The adversarial reward function consists of multiple carefully designed components, each optimized for a different part of the hunter's reward function. The specific form is:

[0013] ;

[0014] in This indicates that the illegal speedboat was successfully surrounded by the coast guard patrol boat; This indicates that the speedboat that violated the rules successfully escaped the encirclement area; This indicates a collision between the ships; This indicates that the illegal speedboat has left the designated activity area; This represents the number of task steps, with a small negative reward applied at each step to encourage the agent to escape as quickly as possible; It is the reward function value of the pursuing party (coast guard). , The value can be 0 or 1, where 1 indicates that the corresponding event has occurred and 0 indicates that the corresponding event has not occurred.

[0015] Step 3: Train the escape policy based on the Proximal Policy Optimization (PPO) algorithm.

[0016] First, experience collection is performed: multiple instances are run in parallel in a simulation environment. The escapee interacts with the pursuer based on its current escape strategy, collecting experience trajectory data at time steps. Empirical data are represented as , where state In time step The observed vector of the escapee, For the escapee in state The action to choose from, Performing actions for escapees The immediate reward obtained afterward is calculated based on the aforementioned adversarial reward function. Performing actions for escapees The next state observed after the event. A flag indicating whether the round has ended.

[0017] Secondly, perform advantage estimation: use generalized advantage estimation (GAE) to calculate the advantage function of the empirical data at each time step.

[0018] Then, escape strategy optimization is performed: the Actor network of the near-end policy optimization algorithm outputs the escape decision, and the Critic network estimates the state value; the Actor network parameters are optimized based on the generalized advantage estimate.

[0019] In step 3, PPO maximizes the following pruning objective function. To update the Actor network parameters :

[0020] ;

[0021] in, It is the probability ratio of the new strategy to the old strategy. It's a clipping hyperparameter. The function represents the expectation, the min function finds the minimum value, and the clip function calculates the expected value. Limited to [ Within the range, It is a time step The generalized advantage estimate is obtained. This pruning mechanism prevents the policy update step from being too large and ensures training stability.

[0022] In step 3, during the escape strategy optimization training, the Actor network parameters are first randomly initialized, and empirical data is obtained through the interaction between the escapee and the pursuer. During the strategy optimization phase, the Critic network is trained to estimate the value of the state. The Critic network is updated by minimizing the difference between the predicted value and the actual reward. The Actor network parameters are optimized by calculating the generalized advantage estimate of the selected action at each time step. This iterative update continues until the escape success rate reaches a convergent state, at which point the iteration stops, and the final Actor network is output. The trained and optimized Actor network is deployed on the escape speedboat to test and optimize the cooperative strategy deployed by the coast guard patrol boat.

[0023] The advantages and positive effects of this invention are as follows:

[0024] (1) The method of this invention adopts an adversarial strategy generation method based on white-box knowledge, which significantly improves attack efficiency and training stability. This method differs from traditional black-box attacks in that it directly utilizes the reward function of the coast guard patrol boat as an important component of the target reward function of the escaped speedboat. This method allows the escapee to deeply utilize the policy gradient information and reward structure of the pursuer, thereby accurately identifying key vulnerabilities in the collaborative encirclement system. Compared to other methods, the escape strategy optimized by this invention significantly improves the escape success rate of the escapee, and the training process exhibits better convergence speed and stability.

[0025] (2) The method of this invention establishes an efficient attack strategy training framework that combines a comprehensive and multi-dimensional evaluation system with a reward function design. The method of this invention has a high degree of comprehensiveness in reward design, not only introducing escape rewards and capture penalties, but also designing specific rewards and penalties for cooperative mechanisms, such as boundary constraints and collision penalties. Combined with an attack effect evaluation system that includes multiple dimensions such as task success rate, cumulative reward function, and decision steps, the method of this invention can effectively guide the agent to explore advanced strategies such as disrupting the pursuit formation and inducing collisions, thereby successfully training an effective adversarial attack strategy with high targeting and strong destructive power, providing a solid foundation for accurately quantifying the security risks of the encirclement system.

[0026] (3) The adversarial strategies trained by the method of this invention have good generalization ability, especially strong adaptability to the inherent asymmetry of cooperative encirclement environments. Thanks to parallel data acquisition and diversified interactive training mechanisms, the escape strategies generated by this invention can not only adapt to different initial situations and dynamic environmental changes, but also perform well in dealing with the asymmetric game feature of multiple pursuers against a single escapee. This generalization ability effectively overcomes the limitation of poor adaptability of existing technologies in asymmetric scenarios, ensuring that the adversarial strategies can maintain stable escape performance when facing diversified and difficult cooperative encirclement tasks, and providing a reliable benchmark for evaluating the robustness of the system under complex real-world conditions. Attached Figure Description

[0027] Figure 1 This is a flowchart of the overall process of the adversarial strategy attack method in the collaborative encirclement scenario in this invention embodiment;

[0028] Figure 2 This is an example diagram illustrating the countermeasure effect of the present invention. Detailed Implementation

[0029] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. The embodiments of the present invention aim to illustrate the implementation of a multi-agent cooperative encirclement and escape method based on adversarial strategy attacks. A complete technical solution for training adversarial agents and implementing strategy attacks in a scenario of cooperative encirclement and capture of dynamic targets by unmanned surface vessels is provided below.

[0030] In a multi-surface unmanned surface vessel (USV) cooperative encirclement scenario, the escape agent can acquire complete knowledge of the pursuer's reward function. The method of this invention fully utilizes this white-box knowledge to construct an adversarial reward function containing negative feedback, and trains the escape agent using the PPO algorithm. This method fully leverages the internal information of the pursuer's reward function to train the escape agent to generate targeted adversarial behaviors, enabling the escapee not only to achieve physical escape but also to proactively break through the pursuer's cooperative formation at the decision-making logic level. In the cooperative encirclement task, by utilizing the white-box knowledge of the pursuer's reward function, reinforcement learning is used to train adversarial escape strategies to test and improve the robustness of the cooperative encirclement system.

[0031] The implementation process of this invention embodiment is as follows: Figure 1 As shown, a collaborative encirclement scenario is as follows: Figure 2 As shown, it includes N hunter agents and 1 escape agent. Figure 2 The blue boats cooperate to surround and capture the red boat. The blue boats are the pursuers and are marked as the coast guard law enforcement force, while the red boats are the escapees and are marked as the suspected violators. The following five steps will explain this method.

[0032] Step 1: System initialization and environment setup.

[0033] Step 1.1: First, establish a simulation training environment for collaborative encirclement. This environment needs to accurately simulate the physical characteristics of a real encirclement scenario, including: Activity area: a rectangular sea area of ​​111km × 111km; Agent dynamics model: including state variables such as position, velocity, and heading angle; Collision detection mechanism: real-time detection based on the agent's safety radius; Boundary constraints: rules for handling the boundary of the activity area, with the maximum speed of the coast guard pursuit patrol boat set at 54km / h, the maximum speed of the escape speedboat at 24km / h, and the target area radius at 2.5km.

[0034] Step 1.2: The pursuers are pre-trained using the MAPPO method with a strategy that has been trained for 1 million time steps and demonstrates good cooperative pursuit capabilities in routine tests. The pursuer's observation space includes multiple dimensions of features, such as its own state, relative position with other pursuers, relative position with the escapee, and target point information. The network structure includes:

[0035] The observation space includes: the distance from the vessel to its assigned target point (1D); the angle between the vessel's course and the target point's direction (2D); the distance between all pairs of coast guard patrol vessels (N×(N-1) / 2D); the distance from the vessel to the escape speedboat (1D); the minimum distance from the vessel to the boundary (1D); the speed of the vessel (1D); and the speed of the escape speedboat (1D). The dimension of the observation space is 7+N×(N-1) / 2, where N is the number of coast guard patrol vessels.

[0036] The state space includes: the distances and angles from all pursuers to their respective target points (3N dimensions); the distances between any two pursuers (N×(N-1) / 2 dimensions); the distances from all pursuers to the escapee (N dimensions); the minimum distance from all pursuers to the boundary (N dimensions); the velocities of all pursuers (N dimensions) and the velocity of the escapee (1 dimension). The dimensions of the state space are N×6 + N×(N-1) / 2 + 1.

[0037] The motion space includes heading angles evenly spaced between (-30° and +30°). The motion space has 9 dimensions.

[0038] The escape agent strategy is trained using the PPO algorithm. The escape agent's observation space is meticulously designed, incorporating multiple dimensions of features, including normalized self-state, relative information with all pursuers, boundary distance information, and detailed information about the nearest pursuer. The escape agent's action space is designed with nine fixed movement directions, covering all possible movement directions. The network structure includes:

[0039] The observation space includes: the escapee's normalized longitude and latitude, normalized velocity and heading angle (5 dimensions); distance to each pursuer (N dimensions) and relative angle (2N dimensions); the pursuer's velocity (N dimensions); the minimum distance to the boundary (1 dimension); and the distance and angle to the nearest pursuer (3 dimensions). The dimensions of the observation space are 5 + N × 4 + 1 + 3.

[0040] The motion space includes heading angles evenly spaced between (-30° and +30°). The motion space has 9 dimensions.

[0041] Step 2: White-box information acquisition and adversarial reward design.

[0042] Step 2.1: First, by analyzing the reward function structure of the pursuer, we obtain its complete reward composition information. The pursuer's reward consists of six parts: proximity reward, dwelling reward, task completion reward, collision penalty, out-of-bounds penalty, and timeout penalty. Each part has a clear weight coefficient.

[0043] ;

[0044] in, To get closer to the reward; As a reward for staying; The reward for completing the task has a coefficient of 120.0. Collision penalty coefficient: -40.0; Penalty for going out of bounds: coefficient -40.0; Penalty for timeout: coefficient -30.0.

[0045] Step 2.2: Based on this white-box information, construct the escapee's adversarial reward function. This function not only includes negative feedback on the pursuer's reward but also designs several auxiliary reward components. The escape success reward encourages avoiding capture before the mission time expires; the survival reward incentivizes extending survival time; and the boundary penalty prevents the escapee from using illegal areas to escape. The white-box adversarial reward function designed in this embodiment of the invention. as follows:

[0046] ;

[0047] in, This indicates that the illegal speedboat was successfully surrounded by the coast guard patrol boat; This indicates that the speedboat that violated the rules successfully escaped the encirclement area; This indicates a collision between the ships; This indicates that the illegal speedboat has left the designated activity area; This represents the number of task steps, with a small negative reward applied at each step to encourage the agent to escape as quickly as possible; This is the reward function value for the pursuing party (Coast Guard). , , The value can be 0 or 1, where 1 indicates that the event has occurred and 0 indicates that the event has not occurred.

[0048] Step 3: Train the escape policy of the escapee using the PPO algorithm. The PPO algorithm consists of an Actor network and a Critic network. The Actor network outputs the escape decision of the escapee, and the parameters of the Actor network correspond to the escape policy parameters. The Critic network outputs the state value. The Critic network is updated by minimizing the value function loss, enabling it to more accurately predict the cumulative reward.

[0049] Step 3.1: The experience collection phase for breaking the cooperative encirclement involves using multiple parallel environments to collect data simultaneously. This includes collecting state, action, reward, and next state. This includes empirical data, which focuses on reflecting the dynamic changes in the coordinated formation of the pursuing forces and the interference from the escapees. Each environment runs for 800 time steps; this parallel acquisition method greatly improves the diversity and efficiency of the training data.

[0050] in: Indicates at time step The environmental state observation includes quantitative characterization of the pursuer team's formation, such as the integrity of the encirclement and the direction of coordinated advance; specifically, in this embodiment of the invention... This refers to the local observation vector of the escapee, which is determined based on the escapee observation space set in step 1.2. Indicates the escapee's state The action of making a selection; Indicates the escapee's actions The immediate reward obtained afterward, calculated by the white-box adversarial reward function of this invention, directly reflects the contribution of the action to the dismantling of the collaborative effectiveness of the pursuit; Indicates the escapee's actions The next state observation is used to assess the changes in the pursuer's cooperative state caused by the escapee's actions. A flag indicating whether a round has ended. This empirical data is stored in an experience replay buffer, providing training samples for subsequent policy updates. Through a large amount of such interaction data, the algorithm is able to learn the complex relationships between states, actions, and rewards.

[0051] Step 3.2: Advantage Estimation. After obtaining sufficient empirical data, to evaluate the effectiveness of the escapee's actions in disrupting the pursuer's coordination, this invention employs a generalized advantage estimation (GAE) method to assess the advantage of actions at each time step. GAE balances the estimation bias and variance, enabling a more accurate assessment of the effectiveness of actions. At each time step... Generalized advantage estimation as follows:

[0052] ;

[0053] in, Indicates timing difference error; This is a discount factor with a value of 0.99, used to balance the importance of immediate rewards and future rewards; value function. The core task of learning from the Critic network is to accurately predict the cumulative damage that the pursuing forces can cause to the coordinated encirclement and capture in the future under the current state. Representing states respectively , Value estimation; It is the total length of the trajectory, i.e., the maximum time step of the empirical data; This is the GAE parameter, with a value of 0.95, used to balance the bias and variance of the generalized dominance estimate. Quantified in state Execute action The advantage of the pursuit and capture formation in terms of confusion and decision-making failure relative to the average level is indicated by a positive value, which means that the action is better than the average level, and a negative value means that it is worse than the average level.

[0054] Step 3.3: Perform escape policy optimization. The policy optimization phase updates the Actor network parameters, i.e., the escape policy parameters, by maximizing the objective function. The core of the PPO algorithm is the pruning of the objective function. The PPO objective function... It is expressed as follows:

[0055] ;

[0056] in: The policy parameters represent the escape strategy; The function represents the expectation over time steps. The average of the data above is calculated; the min function finds the minimum value. It is the probability ratio of the new and old strategies; the clip function represents... Limited to [ Within the range, This is the pruning threshold, with a value of 0.2, used to limit the magnitude of each policy update and prevent excessive policy updates.

[0057] The core of this step is to enable the escapee's counter-strategy to learn to generate a series of action sequences through iterative optimization. These actions can most effectively exploit the blind spots of the pursuer's strategy, causing individual behavior to conflict with team objectives, thereby systematically undermining the escapee's ability to coordinate a capture.

[0058] Step 4: Conduct adversarial strategy training, as shown in steps 4.1-4.3 below.

[0059] Step 4.1: Training Environment Setup. During training, the hunter team employs a single, fixed cooperative strategy. This strategy is trained using the MAPPO algorithm and its parameters remain constant throughout the escapee training process. This setup simulates a real-world security testing scenario for a specific deployment system, aiming to simulate a realistic capture system that relies on a fixed cooperative algorithm. The training objective for the escapees is to overcome this specific cooperative mechanism, rather than to develop a general, strong strategy.

[0060] Step 4.2: Training process execution. At the start of training, the escapee's policy is randomly initialized, i.e., the Actor network parameters are randomly initialized. Initial experience data is collected through interaction with the fixed pursuer's policy. In each training step, the escapee selects an action based on the current policy, and the environment returns the corresponding reward and new state observations.

[0061] Experience data is stored in an experience replay buffer, including state observations, actions performed, rewards obtained, the next state observation, and a round-ending flag. Once a sufficient amount of experience data has been collected, the algorithm begins policy optimization.

[0062] During the policy optimization phase, the generalized advantage estimation method is first used to calculate the advantage function value at each time step, evaluating the relative merits of each action. Then, the policy network parameters are updated by maximizing the pruning objective function, ensuring the stability of the policy update. Simultaneously, the value function network is updated by minimizing the difference between the predicted value and the actual return.

[0063] Step 4.3: Settings to ensure training stability, including:

[0064] Learning rate scheduling: Gradually reduce the learning rate as training progresses to improve convergence stability and ensure that training can stably converge to a strategy that can continuously hunt down the cooperative weaknesses of the parties.

[0065] Gradient clipping: Limits the maximum magnitude of each parameter update to prevent gradient explosion.

[0066] Entropy regularization: Encourages sufficient exploration in the early stages of training to discover unconventional escape paths and strategies that can unexpectedly disrupt pursuit coordination.

[0067] The training process continues until the strategy performance converges, meaning the escape success rate no longer shows significant improvement over multiple consecutive evaluation periods. Through this end-to-end training method, escapees can autonomously learn effective countermeasures against specific pursuer strategies.

[0068] Step 5: Experimental Setup and Result Analysis. To verify the effectiveness of the white-box adversarial strategy optimization proposed in this invention, the embodiments of this invention underwent rigorous testing in a simulation environment.

[0069] Step 5.1: Experimental Environment and Parameter Settings. To ensure the reproducibility of the experimental results, this embodiment adopts the following specific experimental parameter settings. The experiment is conducted on a simulation platform equipped with a high-performance computing unit, and the simulation environment interacts with the control algorithm via the TCP / IP protocol.

[0070] (1) Simulation scene and physical environment parameters:

[0071] Simulation time step: 90 seconds / step, meaning that for every decision made by the agent, the physical world advances by 90 seconds.

[0072] Maximum task duration: 7200 seconds, corresponding to the maximum number of decision steps.

[0073] Activity area: A rectangular sea area of ​​111km × 111km.

[0074] Target capture determination radius: 2.5km, meaning that the pursuer is considered to have entered the target area when they enter this radius centered on the escapee.

[0075] Boundary safety distance: The generation point must be at least 1.5 km away from the boundary.

[0076] Maneuvering constraints: The turning radius of the vessel is set to 200 meters.

[0077] Speed ​​limits: Coast Guard pursuit patrol boats have a maximum speed of 54 km / h, while speedboats used for illegal escape have a maximum speed of 24 km / h.

[0078] (2) Counterattack settings:

[0079] The escape strategy of the escapee is trained using the PPO algorithm based on white-box reward proposed in this invention, with the following parameters:

[0080] Training scale: Total training time steps 1,000,000, maximum number of steps per round 800.

[0081] Network parameters: Actor learning rate is 3e-4, Critic learning rate is 1e-3, batch size is 64, and update iterations are 10.

[0082] Algorithm hyperparameters: clipping threshold is 0.2, maximum gradient norm is 0.5, discount factor is 0.99, and GAE parameter is 0.95.

[0083] (3) Observation space dimension and action space dimension:

[0084] In this experiment, the number of coast guard patrol boats is N=4, and the number of escape speedboats is 1;

[0085] The spatial dimension observed by the coast guard patrol boat is: 7 + 4 × (4 - 1) / 2 = 13;

[0086] Action space dimensions: 9;

[0087] The observation space of a coast guard patrol boat includes: the distance from the boat to its assigned target point (1D); the angle between the boat's course and the direction of its target point (2D); the distance between all pairs of coast guard patrol boats (4×(4-1) / 2=6D); the distance from the boat to the escape speedboat (1D); the minimum distance from the boat to the boundary (1D); the speed of the boat (1D); and the speed of the escape speedboat (1D).

[0088] Action space dimension: heading angles with uniform intervals of (-30°, +30°).

[0089] The spatial dimension of the escape speedboat is: 5 + 4 × 4 + 1 + 3 = 25;

[0090] Action space dimensions: 9;

[0091] The observation space of the escape speedboat consists of: the escape direction's normalized longitude and latitude, normalized speed and heading angle (5-dimensional); the distance to each pursuer (4-dimensional) and relative angle (8-dimensional); the pursuer's speed (4-dimensional); the minimum distance to the boundary (1-dimensional); and the distance and angle to the nearest pursuer (3-dimensional).

[0092] Action space dimension: Action space dimension: heading angles with uniform intervals of (-30°, +30°).

[0093] Step 5.2: Performance Dimension Evaluation. The escape strategy obtained from training can be used to test and optimize existing cooperative strategies. Key performance indicators are calculated, including the cumulative round reward, average round reward, average number of decision steps, and pursuit success rate of the coast guard patrol boat. These indicators are used to test the robustness of the pursuer's cooperative strategy. A vulnerable pursuit system will exhibit a sharp decline in mission rewards and a high escape rate under adversarial strategy attacks.

[0094] In a rigorous evaluation comprising 10 test rounds, the adversarial strategy trained by the method of this invention achieved significant attack effectiveness. Specific data are shown in Table 1 below:

[0095] Table 1 Evaluation results under coordinated encirclement

[0096]

[0097] Based on the data analysis in the table above, we can conclude that:

[0098] 1) The success rate of the mission drops drastically: Under normal conditions without attacks, the success rate of the Coast Guard pursuers' coordinated encirclement is as high as 100%, indicating that the original MAPPO strategy has extremely strong cooperative capabilities. However, after introducing the white-box adversarial strategy generated by this invention, the success rate of the pursuit drops sharply to 10%, proving that the adversarial strategy trained by the method of this invention can effectively escape the coordinated encirclement of the pursuers.

[0099] 2) Significant Decrease in Cumulative Rewards: The cumulative round reward for coast guard patrol boats decreased dramatically from 1394.60 to 37.41, and the average round reward dropped from 139.46 to 3.74, a decrease of over 97%. This indicates that under the interference of adversarial strategies, pursuers not only find it difficult to complete the capture mission, but also struggle to maintain normal approach and lingering behavior, verifying the effectiveness of the negative feedback design for pursuer rewards in the adversarial reward function.

[0100] 3) The time required for the task increased significantly: the average number of decision steps increased from 56.8 to 75.8, indicating that even in the few cases where success was possible, the pursuers needed to spend more time and resources to deal with the escapee's maneuverability. The successful counter-strategy increased the time cost and complexity of the encirclement mission.

[0101] Step 5.3: To gain a deeper understanding of the mechanism by which the attack takes effect, this invention conducts a more in-depth analysis: by observing the synchronicity of the pursuers' formation movements in the simulation, the degree of disruption to the coordination mechanism by the countermeasure strategy is quantified. Observations reveal that the escapee does not simply rely on speed advantage to escape, but rather employs a deception strategy: when the pursuers are about to form an encirclement, the escapee deliberately maneuvers to the side of a particular pursuer, inducing that pursuer to change course, thereby disrupting the overall compactness of the formation, and even inducing adjacent pursuers to experience path conflicts or collision risks, thus creating a gap in the encirclement. For example... Figure 2As shown, the illegal speedboat pretended to break through by slightly adjusting its angle to the left, forcing the blue patrol boat above to surround and capture it. Then, the illegal speedboat took the opportunity to adjust its course and break through to the right front.

[0102] The above evaluation results fully demonstrate that the method of the present invention can generate efficient, intelligent attack strategies that can fundamentally disrupt the target system, providing a high-intensity benchmark testing method for the robustness evaluation of collaborative encirclement systems.

[0103] Except for the technical features described in the specification, all other technologies are known to those skilled in the art. Descriptions of well-known components and technologies are omitted in this invention to avoid redundancy and unnecessary limitation. The embodiments described above do not represent all embodiments consistent with this application. Various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this invention are still within the protection scope of this invention.

Claims

1. A multi-agent cooperative encirclement and escape method based on adversarial attack strategy, characterized in that, include: Step 1: Construct a collaborative encirclement-counter-escape simulation environment, including: setting the activity area of ​​the current multi-surface unmanned surface vessel collaborative encirclement scenario, assuming that there are N pursuer vessels and 1 escapee vessel, with each vessel being an intelligent agent; establishing the dynamic model of each intelligent agent; determining the observation space, state space, and action space of each intelligent agent; and pre-training the collaborative strategy of the pursuers and deploying it to the N pursuers; the task of the current scenario is: within a specified time and area, the pursuers, through collaborative movement, cause all pursuers to enter the capture zone with the escapee as the center and a set radius; the escapee's task is to avoid entering the capture zone. Step 2: Obtain the reward components of the hunter's cooperative strategy and construct the escapee's adversarial reward function. ; ;in This indicates that the illegal vessel was successfully surrounded by the pursuing vessels. This indicates that the illegal vessel successfully escaped the encirclement area; This indicates a collision between the ships; This indicates that the illegal vessels have left the designated activity area; Indicates the number of task steps; It is the reward function value for the pursuing party. , The value can be 0 or 1, where 1 indicates that the corresponding event has occurred and 0 indicates that the corresponding event has not occurred. Step 3: Optimize the escape strategy of the escapee using the near-end policy optimization algorithm; Running the instance in the simulation environment, the escapee interacts with the pursuer based on the current escape strategy, collecting experiential data on the interaction at time steps. Empirical data are represented as , where state In time step The observed vector of the escapee, For the escapee in state The action to choose from, Performing actions for escapees The immediate reward obtained afterward is calculated based on the aforementioned adversarial reward function. Performing actions for escapees The state after observation, A flag indicating whether the round has ended; The near-end policy optimization algorithm outputs escape decisions from the Actor network and estimates the value of the state from the Critic network; it calculates a generalized advantage estimate for the action at each time step of the empirical data and optimizes the Actor network parameters based on the estimate. The trained and optimized Actor network is deployed on the escapee ship to test and optimize the hunter strategy deployed on the hunter ship.

2. The method according to claim 1, characterized in that, In step 1, the observation space set for each pursuer includes: the distance from the vessel to the assigned target point; the angle between the vessel's course and the direction of the target point; the distance between all pairs of pursuer vessels; the distance from the vessel to the escapee vessel; the minimum distance from the vessel to the boundary; the speed of the vessel; and the speed of the escapee vessel. The state space for all pursuers includes: the distance and angle between each pursuer and its target point; the distance between any two pursuers; the distance between each pursuer and the escapee; the minimum distance of each pursuer to the boundary; the speed of each pursuer and the speed of the escapee. The action space of each pursuer includes heading angles evenly spaced at (-30°, +30°).

3. The method according to claim 1 or 2, characterized in that, In step 1, the observation space for the escapee includes: the escapee's normalized longitude and latitude, normalized speed and heading angle; the escapee's distance and relative angle to each pursuer; the pursuer's speed; the escapee's minimum distance to the boundary; and the escapee's distance and angle to the nearest pursuer. The escapee's action space includes heading angles evenly spaced between (-30°, +30°).

4. The method according to claim 1, characterized in that, In step 3, the near-end policy optimization algorithm maximizes the objective function. Update Actor network parameters objective function as follows: ； in, The function represents the expectation; the min function finds the minimum value. It is the probability ratio of the new and old strategies; the clip function will... Limited to [ Within the range, It is the clipping threshold; It is a time step The generalized advantage estimate.

5. The method according to claim 1, characterized in that, In step 3, when training to optimize the escape strategy, the Actor network parameters are first randomly initialized, and empirical data is obtained through the interaction between the escapee and the pursuer. During the policy optimization phase, the Critic network is trained to estimate the value of the state. The Critic network is updated by minimizing the difference between the predicted value and the actual reward. The Actor network parameters are optimized by calculating the generalized advantage estimate of the selected action at each time step. The process is iteratively updated until the escape success rate reaches a convergent state, at which point the iteration stops and the final Actor network is output.

Images