A multi-unmanned ship cooperative hunting training system and a training method

Abstract

The application belongs to the technical field of intelligent control of multiple unmanned ships, and provides a multiple unmanned ship cooperative hunting training system and a training method. The training system comprises: a plurality of first-type agents for simulating behaviors of the unmanned ships for hunting, at least one second-type agent for simulating behaviors of an escape target, a simulation environment unit, an experience pool and an optimization unit. The first-type agent carries a main strategy network, a main Q network, a target strategy network and a target Q network. The main Q network and the target Q network each comprise an attention module. The attention module is used for performing information dimension compression based on attention feature extraction on global state observation information and action observation information received by the first-type agent. The training system and the training method provided by the application can effectively improve the efficiency of training of multiple unmanned ship cooperative hunting.

Description

Technical Field

[0001] This application belongs to the field of intelligent control technology for multiple unmanned vessels, specifically relating to a multi-unmanned vessel collaborative encirclement training system and training method. Background Technology

[0002] The MADDPG (Multi-Agent Deep Deterministic Policy Gradient) algorithm is a deep reinforcement learning algorithm for training cooperative control policies among multiple agents. It is derived by extending the single-agent DDPG (Deep Deterministic Policy Gradient) algorithm. Unlike DDPG, which only optimizes the policy of a single agent, MADDPG introduces a cooperative mechanism among multiple agents. During the centralized training phase, information is shared, allowing each agent to access the observations and actions of other agents. This design not only enhances the cooperation ability between agents but also makes it easier to achieve successful coordinated actions in complex environments. Therefore, MADDPG is particularly suitable for application scenarios that require the cooperation of multiple agents, such as multi-unmanned vessel formations performing tasks.

[0003] Several intelligent control schemes for multi-unmanned vessels based on the MADDPG algorithm have been disclosed. For example, Chinese invention patent CN120124675A provides a decision-making method for marine unmanned swarm attack and defense game based on an improved MADDPG algorithm. This method modifies the structure of the policy network and Q network in the conventional MAPPDG framework into a Long Short-Term Memory (LSTM) network, utilizing its forward prediction characteristics to predict the future state of the vessel and the target. This enables the vessel to learn from environmental changes and predict the target's position and state information in advance. Thus, when the attack and defense game decision-making environment changes, it generates execution actions that match the changed attack and defense game decision-making environment, improving the vessel's win rate. Another example is Chinese invention patent CN117131770A, which provides a multi-agent cooperative pursuit and adversarial method based on the P3C-MADDPG algorithm. This method adopts a priority experience replay mechanism based on tree structure storage and designs a 3-thread parallel Critic network model, enabling the generated multi-agent cooperative pursuit and adversarial strategy to effectively avoid obstacles and more intelligently pursue agents with unknown strategies who are escaping.

[0004] In the process of multiple unmanned vessels collaborating to complete real-world marine environmental tasks, especially when multiple unmanned vessels are encircling and capturing multiple intelligent targets, they often need to face complex obstacles such as islands and reefs, and encounter communication shielded areas with perception interference characteristics. At the same time, these tasks usually involve multi-stage and highly dynamic decision-making processes, which place higher demands on the collaborative game capabilities of the agents. Therefore, directly using the existing MADDPG algorithm framework for training is prone to reduced training efficiency due to factors such as the large dimensionality and scale of input information, as well as the presence of invalid and redundant data in the experience pool. Summary of the Invention

[0005] To address the problems existing in the prior art, the first aspect of this application provides a multi-unmanned vessel collaborative encirclement training system, which can train multiple encirclement unmanned vessels based on an attention optimization mechanism to meet the highly dynamic decision-making needs of multiple encirclement unmanned vessels in complex maritime environments, requiring multi-source perception and local response. This enables the trained encirclement unmanned vessels to still make accurate and efficient strategy responses in complex dynamic environments.

[0006] This multi-unmanned vessel collaborative encirclement training system includes:

[0007] Multiple first-class intelligent agents are used to simulate the behavior of encircling and capturing unmanned ships. Each first-class intelligent agent carries a main policy network, a main Q network, a target policy network, and a target Q network. The main Q network and the target Q network both include attention modules. The attention modules are used to compress the information dimensionality of the global state information and action information received by the first-class intelligent agents based on attention feature extraction.

[0008] At least one second-type agent is used to simulate the behavior of an escaping target, and the second-type agent carries an escape policy network;

[0009] The simulation environment unit is used to provide a training environment in which multiple first-type intelligent agents cooperate to surround and capture multiple second-type intelligent agents, and to enable each intelligent agent to interact with the environment and reward each first-type intelligent agent according to the interaction results. The training environment includes at least one obstacle and at least one communication shielding area.

[0010] An experience pool is used to store training samples for multi-unmanned vessel collaborative encirclement and capture. The training samples are generated based on the interaction and reward of each first-class intelligent agent with the environment.

[0011] The optimization unit uses the training samples to optimize the policy networks and Q-networks of each first-class agent using the MADDPG algorithm.

[0012] Furthermore, for any first-type agent i, the global observation information it receives at the current time t... and global actions for:

[0013]

[0014] Where N is the number of first-class agents, and j is the ID of any first-class agent other than i. These are the observation information received by agent i and agent j of the first type at the current time t, respectively. These represent the actions of agent i and agent j at the current time t, respectively, and concat() is the element concatenation function.

[0015] Preferably, for any first-class agent i, the attention modules in its main Q-network and target Q-network include:

[0016] N observation-action encoders are used to encode the global observation information and global actions received by the first type of agent into embedding vectors corresponding to each first type of agent.

[0017] The first attention compression layer is used to compress the embedding vectors corresponding to each first-class agent other than the first-class agent i into a global attention embedding vector.

[0018] The first attention fully connected layer is used to connect the embedding vector corresponding to the first type of agent i and the global attention embedding vector and output it as an attention feature vector.

[0019] Preferably, for any first-type agent i, the first attention layer generates the global attention embedding vector at the current time t based on the weighted summation model shown below.

[0020]

[0021] Among them, f o-a () represents the encoding function of the observation-action encoder. Let α be the embedding vector corresponding to the j-th agent; i,j for The weighting coefficient, β i,j The inner product of the parameterized matrices of the embedding vectors corresponding to the j-th agent, W ψ W φ Let T be the parameter matrix, and T be the transpose.

[0022] Preferably, the observation-action encoder includes:

[0023] A first entity encoder is used to encode the observation information of the first type of intelligent agent corresponding to the observation-action encoder into an encoding vector;

[0024] M-1 second entity encoders are used to encode the observation information of the first type of agent corresponding to the observation-action encoder to other entities into an encoding vector, wherein the entities include the first type of agent, the second type of agent, obstacles in the training environment and communication shielding areas, and M is the total number of entities;

[0025] An action encoder is used to encode the actions of the first type of agent corresponding to the observation-action encoder into an encoding vector;

[0026] At least one second attention compression layer is used to perform global attention compression on the encoded vectors output by M-1 second entity encoders;

[0027] The second attention fully connected layer is used to connect the inputs of the first entity encoder, the action encoder and the second attention compression layer and output them as an embedding vector.

[0028] Preferably, the multi-unmanned vessel cooperative encirclement training system further includes a high-value path unit, used to determine and store the high-value path sample combinations formed by each first-type intelligent agent in the process of executing the current action strategy and several historical action strategies;

[0029] The optimization unit uses samples from the experience pool and combinations of high-value path samples to optimize the main policy network, target policy network, main Q network, and target Q network of each first-class agent.

[0030] Preferably, the high-value path unit includes:

[0031] The evaluation module is used to evaluate the overall trajectory importance of each path generated by the first type of intelligent agent executing the current action strategy and several historical action strategies, and to determine the high-value path.

[0032] A storage module is used to store the high-value path sample combination, wherein the high-value path sample combination is the collection of training samples corresponding to all times of the high-value path.

[0033] Preferably, the evaluation module evaluates the overall trajectory importance of any path τ generated by the first type of agent based on the following formula:

[0034]

[0035] in, Let be the dominance function at time t along the path, where 0 to T represents the time span experienced along the path. to Let λ be the TD error from time t to time T-1. τHere, λ1, λ2, and λ3 are the weight parameters, std() is the standard deviation function, mean() is the mean function, and λ1, λ2, and λ3 are the weight coefficients.

[0036] Preferably, the total number of paths identified as high-value paths at any given time does not exceed a preset upper limit; and the storage module updates its stored high-value path sample combination based on a first-in-first-out (FIFO) approach.

[0037] Preferably, the rewards for each first-class intelligent agent include target task rewards, collision avoidance rewards, communication maintenance rewards, and boundary rewards.

[0038] The second aspect of this application provides a multi-unmanned vessel cooperative encirclement training method. This training method uses the aforementioned multi-unmanned vessel cooperative encirclement training system to train multiple first-type intelligent agents. The training is carried out through multiple rounds, and each round includes a training sample generation stage and a cooperative encirclement strategy optimization stage based on MADDPG.

[0039] The multi-unmanned vessel collaborative encirclement training system and method provided in this application have at least the following advantages over existing technologies:

[0040] First, this application improves the architecture of the main Q-network and target Q-network in the conventional MAPPDG architecture. By introducing an attention module, the global state information and action information received by the evaluation network of the first type of agent are compressed based on attention feature extraction. This enables the first type of agent to adaptively focus on the state information most critical to decision-making. At the same time, after compression, only small-scale vectors are needed to represent the features closely related to each first type of agent, which can effectively improve training efficiency and avoid the information explosion and dimensionality curse problems that occur when the entity scale is large in the training environment.

[0041] Secondly, by setting up high-value path units in the training system to extract high-value path sample combinations, since the experience carried by high-value paths represents the continuous changes in the decision-action of the first type of agent when attempting to take newer actions, introducing them into the centralized training process can effectively improve learning efficiency. In addition, since high-value path sample combinations are extracted only from paths generated by the current policy and a few historical policies, with a preset upper limit on the number and updated in a first-in-first-out manner, while making full use of the effective information they carry for training, it can also effectively avoid the problem of invalid / low-value samples accumulating over time that exists in the sample storage mechanism of the conventional MADDPG. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of a mission scenario in some embodiments where multiple unmanned vessels surround and capture multiple intelligent targets;

[0043] Figure 2 This is a schematic diagram of the architecture of a multi-unmanned vessel collaborative encirclement and capture training system provided according to an embodiment of this application;

[0044] Figure 3 This is a schematic diagram of the architecture of the attention module included in the first type of intelligent agent Q network provided according to an embodiment of this application;

[0045] Figure 4 This is a schematic diagram of the architecture of an observation-action encoder provided according to an embodiment of this application;

[0046] Figure 5 This is a schematic diagram of the architecture of a multi-unmanned vessel collaborative encirclement and capture training system provided according to an embodiment of this application;

[0047] Figure 6 This is a flowchart illustrating the participation of the evaluation module and storage module in the training process according to an embodiment of this application;

[0048] Figure 7 This is a schematic diagram of a simulation training environment provided according to a specific embodiment of this application;

[0049] Figure 8 The average reward curve for the training process of a first-class intelligent agent according to a specific embodiment of this application;

[0050] Figure 9 The average reward curve for the training process of another first-class intelligent agent according to a specific embodiment of this application;

[0051] Figure 10 This is the average reward curve for the training process of yet another first-class intelligent agent provided according to a specific embodiment of this application;

[0052] Figure 11 This is a schematic diagram of the trajectory of three first-class intelligent agents cooperating in a trap, according to a specific embodiment of this application. Detailed Implementation

[0053] The present application will now be further described based on preferred embodiments and with reference to the accompanying drawings.

[0054] In the description of the embodiments of this application, it should be noted that if terms such as "upper," "lower," "inner," or "outer" are used to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this application is in use, they are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this application. In addition, in the description of this application, in order to distinguish different units, the terms "first," "second," etc. are used in this specification, but these are not limited by the manufacturing order, nor should they be construed as indicating or implying relative importance. Their names may differ in the detailed description and claims of this application.

[0055] The vocabulary used in this specification is for illustrative purposes and is not intended to limit the scope of this application. It should also be noted that, unless otherwise expressly specified and limited, the terms "set," "connected," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection, a direct connection, or an indirect connection via an intermediate medium; or they can refer to the internal communication between two components. Those skilled in the art will understand the specific meaning of these terms in this application.

[0056] Figure 1 This is a schematic diagram illustrating a mission in a complex maritime environment where multiple unmanned surface vessels (USVs) collaboratively apprehend multiple targets (such as escape USVs or other maneuverable intelligent agents). Figure 1 As shown, multiple unmanned vessels (UAVs) work together to encircle and capture multiple targets. However, in the limited marine environment, there are various types of obstacles and communication shielding zones that may affect the communication of the UAVs. Therefore, in the process of encircling and capturing targets, each UAV needs to not only consider pursuing the target, but also avoid collisions with each other and with obstacles, and also minimize the time spent in the communication shielding zone.

[0057] It is evident that training a network of strategies for capturing unmanned vessels is necessary to enable it to adapt to various situations. Figure 1 When multiple unmanned vessels cooperate to capture targets in complex sea conditions, the global state information includes a large number of entities of different types and characteristics, such as agents, obstacles, and special areas, and their states and / or actions. The increase in the scale and dimensionality of the information not only exacerbates the perception uncertainty and decision-making pressure of unmanned vessels in complex mission scenarios, but also directly inputting the original global state information into the value network not only faces the curse of dimensionality, but also fails to highlight the key local information closely related to the current unmanned vessel strategy, resulting in inaccurate strategy evaluation.

[0058] To address the aforementioned issues, this application improves upon the existing MADDPG framework to provide a multi-unmanned vessel collaborative encirclement training system. This training system can effectively train multiple unmanned vessels to meet the highly dynamic decision-making requirements of multi-source perception and local response in complex maritime environments, enabling the trained encirclement unmanned vessels to still make accurate and efficient strategy responses in complex dynamic environments.

[0059] Figure 2 A schematic diagram of the framework of a multi-unmanned vessel cooperative encirclement training system according to some embodiments of this application is shown, such as... Figure 2 As shown, the training system includes multiple first-class agents, at least one second-class agent, a simulation environment unit, an experience pool, and an optimization unit.

[0060] The first type of intelligent agent is used to simulate the behavior of hunting down unmanned ships. As shown in the figure, each first type of intelligent agent carries a master policy network, a master Q network, a target policy network, and a target Q network.

[0061] The main policy network is used to control the first type of agent to continuously make actions in the training environment. Its input is the global observation information and global actions acquired by the first type of agent. Its output is the action performed by the first type of agent based on the global observation information. This process from input observation information to output action is called action policy. The training of the first type of agent for cooperative encirclement is a process of continuous optimization of the action policy reflected by its main policy network.

[0062] The target policy network has the same network structure as the main policy network and is used to provide stable and consistent target values, thereby supporting the learning of the main policy network (when the action estimated by the main policy network is used to calculate the Q value, the target policy network is used to generate the proposed action for the next state). During the intensive training phase, the parameters of the target network are usually updated using soft updates or by periodically synchronizing with the main policy network.

[0063] To clearly distinguish the outputs of the primary policy network and the target policy network, we generally use 'a' to represent the output of the primary policy network and the corresponding action. target This represents the action output by the target policy network.

[0064] The input to the main Q-network is the experience (i.e., training samples) collected from the states and actions of the first type of agent. Its output is the Q-value for a given state and action pair. The Q-value is used to represent the expected reward of taking the action in that state. During the intensive training phase (i.e., the phase of optimizing the network parameters), the Q-value output by the main Q-network is used to continuously optimize the agent's action policy through the policy gradient optimization algorithm, thereby increasing the probability of it choosing a high-reward action in the actual state.

[0065] The target Q network has the same network structure as the main Q network. In the loss function calculation of the main Q network, the target Q network provides the target Q value for training, making the Q value update process smoother and reducing overfitting and instability.

[0066] Similarly, to clearly distinguish the outputs of the main Q-network and the target Q-network, we can use Q() and Q() respectively. target () represents the outputs of the master Q-network and the target Q-network.

[0067] Within the MADDPG framework, main and target networks are set up for the policy network and Q network respectively, which can effectively reduce the fluctuation of the main network during training, improve the convergence speed, and reduce the risk of overfitting. The interaction of the four networks and the policy gradient optimization process are known to those skilled in the art.

[0068] The second type of intelligent agent is used to simulate the behavior of the escape target. In order to improve the training effect of the first type of intelligent agent, in the embodiments of this application, the second type of intelligent agent carries an escape policy network. The input of the network is the global observation information acquired by the second type of intelligent agent, and its output is the escape action that the second type of intelligent agent needs to take (that is, the escape policy network reflects the escape policy of the second type of intelligent agent).

[0069] The escape strategy network can adopt the same or similar network structure as the main strategy network of the first type of intelligent agent. By pre-training it, the escape strategy executed by the first type of intelligent agent can be made to have a high level of intelligence. Using the second type of intelligent agent with high escape ability to train the first type of intelligent agent can effectively improve the training effect of the cooperative encirclement ability of the first type of intelligent agent. For the specific bidirectional training process, please refer to the multi-unmanned vessel cooperative encirclement training method based on bidirectional deep reinforcement learning disclosed in Chinese invention patent CN118626867A, which will not be elaborated here.

[0070] In the embodiments of this application, the simulation environment unit is used to provide a training environment in which multiple first-type intelligent agents cooperate to capture multiple second-type intelligent agents.

[0071] The training environment provided by the simulation environment unit can be referenced. Figure 1 The illustrated embodiments, such as Figure 1 As shown, the training environment is generally a sea area model with a limited area, which contains at least one obstacle and at least one communication shielding zone.

[0072] The obstacles are used to simulate static entities such as islands, reefs, and rocks, as well as dynamic entities such as ships or other unmanned vessels. During the coordinated encirclement and capture process, the encirclement unmanned vessels (i.e., the first type of intelligent agents) need to avoid collisions with the obstacles. The communication shielding zone is used to simulate various communication interference situations that may be encountered in a complex marine environment. After the encirclement unmanned vessel enters the communication shielding zone, its communication function will be interfered with. In extreme cases, it will be unable to communicate effectively with other encirclement unmanned vessels. Therefore, during the coordinated encirclement and capture process, the encirclement unmanned vessels should try to avoid entering the communication shielding zone or minimize the time spent in the communication shielding zone.

[0073] In addition to providing a training environment, the simulation environment unit is also used to enable the interaction between each agent (including first-class agents and second-class agents) and the environment, and to reward each first-class agent based on the interaction results.

[0074] Specifically, the simulation environment unit can determine the state of each agent at the next time t+1 based on the state of each agent at the current time t and the actions to be performed by each agent according to their respective strategies, and provide real-time rewards to each agent of the first type according to a pre-determined reward function.

[0075] Through the aforementioned interactive and reward operations, the simulation unit can generate global observation information and global action information received by each first-type agent at each time step, as well as training samples stored in the experience pool (described later). The following example, a training system containing N first-type agents, several second-type agents, obstacles, and communication shielding zones, illustrates the structure of global observation information, global action information, and training samples.

[0076] Assume that in addition to N first-class intelligent entities, the training environment also includes at least one second-class intelligent entity, at least one obstacle, and at least one communication shield. For ease of representation, we can assume that the total number of entities in the training environment is M, that is, N entities are first-class intelligent entities (i.e., unmanned boats used for coordinated capture), and MN entities are second-class intelligent entities (i.e., escape targets), obstacles, or communication shields, and MN≥3.

[0077] At any given time t, the observations of each Type I agent regarding the states of all entities in the training environment include its own position, velocity, and relative position to other Type I agents, Type II agents, obstacles, and communication barriers. Considering the actual encirclement process, the information observed by any Type I agent, such as Type I agent i, can be represented as... Clearly, this observational information can be further represented as That is, the observation results of agent i of the first type on all M entities in the training environment (it should be noted that in the simulation training environment, the observation information is...). In reality, this reflects the state of each entity. However, in actual coordinated encirclement and capture processes, the observation information obtained by each unmanned vessel through its measuring devices generally contains observation errors compared to the state of each entity.

[0078] Since observation information can be shared through two-way communication during cooperative encirclement, the simulation unit can construct global observation information from the observation information of each Type I agent. And this information is provided to each Type I intelligent agent. In addition to the aforementioned global observation information O... t In addition, the actions of each Type I agent at the current time t This constitutes a global action. Accordingly, the simulation environment unit rewards each of the first-type agents according to the actions they perform, based on a preset reward function, thereby determining the global reward. Then, by using the preset unmanned vessel dynamics model and constraints, the actions of each Type I intelligent agent are determined, and the global observation information for the next time step t+1 is obtained.

[0079] It should be noted that, in the embodiments of this application, since the Q-network adopts an attention-based architecture, for any first-type agent i, the observation information it obtains through its own observations... and its own actions The observation information and actions of the first-type intelligent agent i, which it obtains through bidirectional communication, have different status and processing methods. Therefore, the global observation information received by the first-type intelligent agent i from the simulation environment unit at the current time t can be considered as such. and global actions Recorded as:

[0080]

[0081] Where j is the ID of any first-class intelligent agent other than i among the N first-class intelligent agents. These are the observation information received by agent i and agent j of the first type at the current time t, respectively. These represent the actions of agent i and agent j of the first type at the current time t, respectively, with `concat()` being the element concatenation function. For example, for agent i of the first type numbered 2, the global observation information received from the simulation environment unit... and global actions It can be written as:

[0082]

[0083] Setting reward functions based on the actions and states of each agent is a crucial step in reinforcement learning algorithms. Appropriately designed reward functions help improve the learning efficiency and final performance of the agents. When designing rewards, it is necessary to consider the task objectives, environmental characteristics, and potential agent behaviors to ensure the learning process is effective and stable. For example, in some optional embodiments, the following reward functions can be set:

[0084] A. Target task reward r1,

[0085] The target mission reward r1 is used to reward the cooperative encirclement and capture of the target by each unmanned vessel around the core. In some specific embodiments, r1 can take the form of the following formula:

[0086] r1=ρ1[-min(Distance)]+ρ2[boolean(is_done)],

[0087] Where ρ1 and ρ2 are adjustment coefficients, Distance = (d1, ... d N Let be the set of distances between each Type I agent and the escape target, i.e., Type II agents. min() is the minimum value function, and the larger this part is, the greater the reward. is_done is the task completion flag, which can be set when the distance between the Type I agent and the Type II agent is less than a preset capture success threshold. boolean() is a Boolean function, which is 1 for task completion and 0 for task failure. In addition, when there are multiple Type II agents, multiple task completion flags can be set, and a tiered task completion reward can be formed according to the number of successful captures.

[0088] B. Collision avoidance bonuses r2 and r3

[0089] The collision avoidance reward for the first type of intelligent agent may include two parts: a reward r2 for avoiding collisions with obstacles and a reward r3 for avoiding collisions with other first type of intelligent agents. In some embodiments, r2 and r3 may take the following forms respectively.

[0090]

[0091] In the formula, ρ3 is the adjustment coefficient, and d ob Let d be the distance between the first-type agent and the obstacle, and D be the actual collision distance between the first-type agent and the obstacle. safe,ob θ represents the safe distance threshold between the first type of agent and the obstacle. ob C1 is the angle between the direction of motion of the first type of intelligent agent and the line connecting the first type of intelligent agent and the obstacle. C1 is a constant, indicating that when the first type of intelligent agent has a real collision with the obstacle, it receives a constant penalty.

[0092] r3=-ρ4[max(0,D safe -Δd)],

[0093] In the formula, ρ4 is the adjustment coefficient, Δd is the distance between two first-type agents, and D safe The set safe distance threshold is max[], which is a function for maximizing the value. This expression indicates that when the distance between two Type I agents is less than D... safe When this happens, a linearly increasing penalty will be imposed.

[0094] C. Maintain communication reward r4,

[0095] Because communication-blocked areas exist in the training environment, unmanned surface vessels should avoid prolonged stays in areas where communication is impossible. In some embodiments, r4 can take the following form:

[0096] r4=-ρ5Δt durance ,

[0097] In the formula, ρ5 is the adjustment coefficient, and Δt durance The duration of the first type of intelligent agent in the communication shielded zone is accumulated from the moment the first type of intelligent agent enters the communication shielded zone and is reset to zero after the agent leaves the communication shielded zone.

[0098] D. Boundary reward r5,

[0099] The boundary reward function r5 guides the unmanned surface vessel (USV) to operate within a defined range. Approaching the environmental boundary incurs a guidance penalty, while exceeding the boundary results in a larger penalty. In some embodiments, r5 can take the following form:

[0100]

[0101] In the formula, d1 and d2 are two set distance thresholds, ρ6 is the adjustment coefficient, and d b C1 represents the distance between the first type of agent and any boundary of the training environment, and C2 is a set constant value.

[0102] The adjustment coefficients and constants in the above formulas can be set by comprehensively considering factors such as the specific training task, agent size, and training environment size. Adding these terms together yields the complete form r for rewarding each type I agent. total :

[0103] r total = r1+r2+r3+r4+r5.

[0104] In the embodiments of this application, the experience pool is used to store training samples for multi-unmanned vessel cooperative encirclement. Specifically, as described above, the simulation environment unit continuously interacts between each agent and the training environment and rewards the first type of agent, thereby obtaining the experience of each agent at each time step. As the cooperative trapping process of the various Type I intelligent agents continues, the simulation environment unit continuously generates experience... In deep reinforcement learning algorithms, this process of storing training samples into the experience pool is called the training sample generation stage.

[0105] The duration of the training sample generation phase can be determined based on the task completion status or the number of steps of the preset execution action strategy. For example, in some specific embodiments, the training sample generation phase can end and enter the centralized training phase after multiple first-type agents have cooperated to complete the encirclement of all second-type agents (generally, the distance between at least one first-type agent and a certain second-type agent is less than a preset threshold is used as the mark of successful encirclement of the second-type agent). In other specific embodiments, the training sample generation phase can also end and enter the centralized training phase when the preset strategy execution steps reach the preset step limit (e.g., 50 steps or 100 steps).

[0106] During the intensive training phase, the optimization unit uses training samples from the experience pool to optimize the main policy network, target policy network, main Q network, and target Q network of the first type of agent using the MADDPG algorithm.

[0107] The process of optimizing each network using the MADDPG algorithm is known to those skilled in the art. For example, in some optional embodiments, firstly, several experiences of each first-type agent are randomly extracted from the experience pool according to a preset priority (e.g., extracting K experiences corresponding to each first-type agent). Then, for the policy network and Q network of each first-type agent, the following optimization steps are performed sequentially:

[0108] 1) Calculate the target Q-value using the target Q-network:

[0109]

[0110] in, Let Q be the target Q value of the i-th type 1 agent at time k. Let γ be the reward value of the i-th agent of the first type at time k, and γ be the discount factor. This represents the global observation information of the i-th type 1 agent at time k+1. ω represents the global action at time k+1 output by the target policy network of the i-th first-class agent. target,i Let these be the network parameters of the target policy network for the i-th first-class agent. It represents the target Q-network of the i-th class 1 agent's estimation of long-term reward given the state (i.e. global observation information) and action.

[0111] 2) Determine the loss function L of each master Q network based on the following formula. i The main Q-network parameters are updated with the goal of minimizing the loss function.

[0112]

[0113] Where, ω i For the i-th first-class intelligent agent Let Q be the Q-value output by the main Q-network of the i-th first-class agent under a given state and action.

[0114] 3) Update the parameters of the main policy network based on the following formula:

[0115]

[0116] Where, π i Let θ be the current policy of the main policy network for the i-th agent of the first class. i J is the policy parameter. i The expected total return under the current strategy, These are the gradient operators for the policy parameters and the action, respectively.

[0117] 4) Update the target policy network and target Q network periodically using soft updates:

[0118]

[0119] Where, θ target,i ζ represents the policy parameters of the target policy network, and ζ is a hyperparameter that controls the update rate, typically ranging from 0.001 to 0.01.

[0120] The above describes the overall architecture of the multi-unmanned vessel cooperative encirclement training system and the implementation methods of distributed generation of training samples and centralized training. As can be seen, under the MADDP framework, when training each first-type agent in a centralized manner, global observation information and global actions need to be input into the Q-network of each first-type agent. Since the global observation information includes the state information of all entities in the training environment, the dimensionality and scale of information will increase sharply when the number of entities is large. This not only easily leads to slow convergence speed during training, but also makes it difficult to distinguish between important local information closely related to each first-type agent and redundant information that is not closely related, resulting in insufficient accuracy of the Q-value network output and thus poor optimization effect on the policy network.

[0121] To address this, the multi-unmanned vessel cooperative encirclement training system provided in this application improves the architecture of the main Q-network and the target Q-network. Specifically, both the main Q-network and the target Q-network include an attention module. This attention module is used to compress the information dimensionality of the global state information and action information received by the first type of intelligent agent based on attention feature extraction. After compression, only a small vector is needed to represent the features closely related to each first type of intelligent agent, effectively improving training efficiency and avoiding the information explosion and dimensionality curse problems that occur when the entity size is large.

[0122] Figure 3 The diagram illustrates the structure of the attention module included in the Q-network of any first-type agent i in some embodiments (assuming the training system includes N first-type agents). It should be understood that since the main Q-network and the target Q-network have the same network architecture, therefore... Figure 3 It can represent either the attention module contained in the main Q network or the attention module contained in the target Q network.

[0123] like Figure 3 As shown, this attention module consists of N observation-action encoders f o-a It consists of a first attention compression layer (attention1) and a first attention fully connected layer (FC1).

[0124] Observation-action encoder f o-a The input is the observation information and actions of the first type of agent, and the output is the encoded, dimension-reduced embedding vector. Since the global observation information and global actions received by the first type of agent contain the observations and actions of N first type agents, it is necessary to use N observation-action encoders f. o-a They are encoded into embedding vectors corresponding to each of the first-class agents.

[0125] Specifically, the form of the global observation information and global action received by the first type of intelligent agent i is shown in equation (1):

[0126]

[0127] The above information can be combined in pairs to form N pairs of observation information-action combinations:

[0128]

[0129] These N pairs of observation-action combinations correspond to the observations of each Type I agent on the system at time t and the actions taken, and are respectively input into N observation-action encoders f. o-a Then, we can obtain the embedding vectors that correspond one-to-one with the N first-class agents:

[0130]

[0131] like Figure 3 As shown, the input to the first attention compression layer, attention1, is the embedding vectors corresponding to each of the first-class agents other than agent i. After compression, the output is a global attention embedding vector.

[0132] In some preferred embodiments, the first attention layer attention1 generates the global attention embedding vector at the current time t based on the weighted summation model shown in equation (9).

[0133]

[0134] Among them, f o-a () represents the encoding function of the observation-action encoder. Let α be the embedding vector corresponding to the j-th agent; i,j for The weighting coefficient, β i,j The inner product of the parameterized matrices of the embedding vectors corresponding to the j-th agent, W ψ W φ Let T be the parameter matrix, and T be the transpose.

[0135] (9) W ψ W φTo continuously optimize the parameters for improving attention during training, in some specific embodiments, neural network architectures known to those skilled in the art can be used. For example, attention1 can be constructed using a multilayer perceptron, which may include an input layer for inputting the embedding vectors corresponding to each first agent j other than first agent i, and several hidden layers whose connection weights are W to be optimized. ψ W φ A weighting layer is used to generate the weights for each first-class agent j, and an output layer is used to output the weighted global attention embedding vector.

[0136] The first attention-based fully connected layer FC1 is used to embed the vector corresponding to the first type of agent i, i.e. and global attention embedding vector The concatenation and output are attention feature vectors. Obviously, due to and Since the data dimensions and structures are the same, they can be easily concatenated and output as vectors of the same dimension.

[0137] Preferably, the attention module can be set at the beginning of the Q-network architecture. As can be seen from the process of generating the attention feature vector through the attention module, the attention feature vector output by the attention module includes not only the feature encoding results of the first agent i's own observations and actions, but also the high-value information extracted from the encoding information of other first agents. Furthermore, by using a weighted summation method, the huge global observation information and global actions are compressed to the same dimension as the encoded vector of a single first agent. Therefore, while improving the estimation accuracy of the Q-network, it can avoid the interference of various redundant and invalid information, effectively improving the convergence speed of training.

[0138] It is important to note that although the observation-action encoders have the same structure, because they are located in the Q-networks of different first-type agents, their output embedding vectors will be transmitted in different directions, and their parameters will continuously change as the Q-network in which they reside undergoes training. For example, in the Q-network of first-type agent i, As the embedding vector of "other" first-class agents, it is transmitted to the first attention compression layer (attention1) in the Q-network of first-class agent i, while in the Q-network of first-class agent j, the observation-action encoder outputs... Then, as the encoding result of its own observation information - action, it is input into the first attention fully connected layer FC1.

[0139] Figure 4This illustrates, in some preferred embodiments, an observation-action encoder that takes the observation information-action combination of a first agent j as input. The specific structure, such as Figure 5 As shown, the observation-action encoder consists of a first entity encoder f e,1 M-1 second entity encoders (assuming the training environment contains M entities) f e,2 A motion encoder f a It includes at least one second attention compression layer (attention2) and one second attention fully connected layer (FC2).

[0140] Figure 4 The observation-action encoder shown includes M observation encoders, each corresponding to one of the M entities in the training system. These entities include N first-type agents, several second-type agents, obstacles, and communication shielding zones. Each first-type entity encodes f. e,1 The device is used to store the observation information of the first agent j corresponding to the observation-action encoder regarding itself. Encoded as an encoding vector; M-1 second entity encoders f e,2 Then, the observation information of the first agent j corresponding to the observation-action encoder on other entities is recorded one-to-one. Encode it as an encoded vector. Clearly, just like f o-a In that case, f e,1 f e,2 The network results can be exactly the same, but the network parameters change continuously as training progresses.

[0141] motion encoder f a This is used to encode the action of the first type of agent j corresponding to the observation-action encoder into an encoding vector.

[0142] The structure of the second attention compression layer (attention2) can be constructed with reference to the first attention compression layer (attention1). It can have one layer, meaning the encoded vectors obtained by encoding the observation results of j for all entities other than j can be input into the same second attention compression layer (attention2) for global attention compression. Alternatively, it can have multiple layers; for example, four second attention compression layers (attention2) can be set up, each inputting the encoded vectors of j's observation results for all first-type agents other than j, the encoded vectors of its observation results for second-type agents, the encoded vectors of its observation results for obstacles, and the encoded vectors of its observation results for communication shielding areas into their respective second attention compression layers (attention2) for attention compression. Finally, the outputs of the first entity encoder, each of the second attention compression layers (attention2), and the action encoding layer are connected through a second fully connected attention layer (FC2) to obtain the final result. The form output is the embedding vector of the first type of agent j at time t.

[0143] Figure 5 Another preferred embodiment of this multi-unmanned vessel cooperative encirclement training system is provided. Figure 5 The illustrated embodiments and Figure 2 The difference in the illustrated embodiment is that a high-value path unit is added. The high-value path unit is used to determine and store the high-value path sample combinations formed by each first-type agent in the process of executing the current action policy and several historical action policies. During the centralized training phase, the optimization unit will jointly use the samples in the experience pool and the high-value path sample combinations to optimize the main policy network, target policy network, main Q network and target Q network of each first-type agent.

[0144] In the embodiments of this application, a high-value path refers to a complete path with high training value obtained by a first-type agent through executing the current action policy of its main policy network, or through executing historical action policies in several historical training rounds. The high-value path sample combination is the combination of all training samples experienced at each time step on the complete path. For example, when the complete path τ experienced by a first-type agent i in a certain round is determined to be a high-value path, assuming that the path τ starts executing actions from time 0 to time T and completes the pursuit of the escaped target, forming a total of T training samples (i.e., single-step experience) from time 0 to time T-1, then the high-value path sample combination corresponding to the high-value path is the set of all training samples experienced by the first-type agent i at each time step on the complete path τ.

[0145]

[0146] The reason for including high-value path sample combinations during the intensive training phase is that, compared to the single-step samples stored in the experience pool of the conventional PPDG algorithm, the process of the first-type agent encircling the second-type agent through a complete encirclement path can reflect its complete process from discovering each second-type agent to integrating the states and actions of each second-type agent, other first-type agents, and the location distribution of obstacles and communication shielding areas, and then making a judgment and implementing an updated action. When the first-type agent successfully completes the encirclement task through a less frequently experienced path, the series of experiences generated likely characterize the continuous changes in the decision-making and action of the first-type agent in trying to take newer actions, which can often provide high value for the optimization of action strategies. In addition, high-value paths are only extracted from the paths generated by the current strategy and a few historical strategies (rather than extracting high-value paths generated by all historical strategies), which can effectively avoid the problem of invalid / low-value samples accumulating over time due to the conventional MADDPG experience pool storing all historical experiences.

[0147] In some embodiments of this application, the high-value path unit further includes an evaluation module and a storage module. The evaluation module is used to evaluate the overall trajectory importance of each path generated by the first type of intelligent agent executing the current action strategy and several historical action strategies and to determine the high-value path. The storage module is used to store the set of training samples corresponding to all moments of the high-value path, i.e., the high-value path sample combination.

[0148] Figure 6 The diagram illustrates the flowchart of the training process in some embodiments, where the evaluation module and storage module participate. Figure 6 As shown, during the policy execution phase of each round, the simulation environment unit stores the training samples of each type 1 agent into the experience pool, while also evaluating the overall trajectory importance of each complete path and determining the critical path, and then storing it into the critical path storage module according to priority.

[0149] During the training phase, the combination of high-value path samples in the storage module is sampled by the optimization unit according to a certain probability, and then combined with the training samples sampled from the experience pool to optimize the policy network and Q network based on the MADDPG algorithm.

[0150] In some preferred embodiments, the evaluation module can select high-value paths from the complete paths through the following steps:

[0151] The first step is to calculate the dominance function at each time step along any path τ using the following formula.

[0152]

[0153] in, to Let λ be the TD error from time t to time T-1 (in deep reinforcement learning algorithms, the TD error measures the difference between the predicted value of the current state and the actual reward; for the MADDPG algorithm, the value prediction result can be given by the main Q network). τ Let λ be the weighting parameter, preferably λ. τ It can be obtained by multiplying γ and λ, where γ is the discount factor and λ is the trade-off parameter.

[0154] The second step is to determine the dominance function at each time point (from 0 to T-1). Then, the overall trajectory importance P of the path τ is determined by the following formula. τ :

[0155]

[0156] In equation (12), std() is the standard deviation function and mean() is the mean function. Let λ1, λ2, and λ3 be the Q-value at time t on this path, output by the main Q-value network, and let λ1, λ2, and λ3 be the weight coefficients.

[0157] As can be seen from equation (12), in the embodiments of the present invention, for any path, its "value" in centralized training is evaluated from three aspects. The first term is used to measure the degree of advantage fluctuation of the first type of agent in different states-actions during the cooperative encirclement process. The larger the fluctuation value, the more special the area where the advantage function changes drastically during the encirclement process. This drastic change may be caused by the unmanned ship choosing a new encirclement route when encountering obstacles or communication shielding areas during navigation, and the policy network has not yet formed a stable and effective response strategy. The second term is the average difference between the predicted value and the actual return of the Q network. This term indicates the degree of "deviation" in the Q network's "understanding" of the overall trajectory of the path. The larger the value, the more the Q network has not yet learned to effectively evaluate the experience. The third term gives the overall evaluation of the encirclement path by the Q network. The higher the value, the more reasonable the encirclement path chosen by the first agent is in terms of the overall effect of the cooperative encirclement.

[0158] At the same time, it should also be pointed out that, such as Figure 6As shown in the embodiments of this application, the training samples stored in the experience pool are continuously accumulated as the training rounds increase. As the training process progresses, the value of some earlier training samples will become lower and lower. Therefore, extracting training samples from all historical training samples will significantly affect training efficiency. To this end, in some preferred embodiments of this application, the storage module presets a capacity threshold (for example, in some embodiments, the capacity threshold is 100, that is, the number of critical paths does not exceed 100) and stores the high-value path sample combination generated by the current action strategy or several historical action strategies in a first-in-first-out manner. When the capacity reaches the upper limit, that is, while storing the high-value path sample combination generated by the current action strategy, the high-value path sample combination generated by the earliest historical action strategy is discarded. By "storing a limited number of complete training sample combinations of high-value paths and updating them in real time", the problem of not being able to obtain enough high-value samples by relying solely on the current action strategy can be solved, and the problem of invalid / low-value samples accumulating over time, which is easy to generate by the conventional MADDPG experience storage mechanism, can be avoided.

[0159] Back Figure 6 During the intensive training phase, the optimization unit will simultaneously extract several single-step training samples from the experience pool and several combinations of high-value paths from the critical path storage unit. Then, by combining the above samples, the main policy network, target policy network, main Q network, and target Q network of each first-class agent will be optimized using the MADDPG algorithm described above.

[0160] Training can be conducted in multiple batches, with training samples drawn from the experience pool each time in a sequential or random manner. In some preferred embodiments, for any critical path τ... n The samples will be extracted by the optimization unit according to the following sampling probabilities for use in optimizing the policy network and the value network:

[0161]

[0162] Among them, S τ Let P be the set of critical paths. τ,n For the critical path τ n Trajectory priority, P τ,m For S τ The trajectory priority of the m-th critical path in the middle.

[0163] Some embodiments of this application also provide a multi-unmanned vessel cooperative encirclement training method. This training scheme uses the aforementioned multi-unmanned vessel cooperative encirclement training system to train multiple first-type intelligent agents. The training can be carried out in multiple rounds, and each round includes a training sample generation stage and a cooperative encirclement strategy optimization stage based on MADDPG.

[0164] After training multiple Type I intelligent agents using the aforementioned multi-unmanned vessel cooperative encirclement training system and method, the main strategy network of each Type I intelligent agent can be installed in the host computer of each real encirclement unmanned vessel. The encirclement unmanned vessel is equipped with various information acquisition devices that can monitor the actual encirclement environment, the encirclement unmanned vessel, the encirclement target, obstacles, and communication interference areas in real time, and conduct two-way communication and information sharing between the encirclement unmanned vessels through communication devices. After receiving data sent by the information acquisition devices and communication devices, the main strategy network installed on the encirclement unmanned vessel executes the trained action strategy based on the input data to output the control quantity of the encirclement unmanned vessel, which is executed by the power unit of the encirclement unmanned vessel, thereby jointly performing the cooperative encirclement task. Specific Implementation Example 1

[0166] This embodiment verifies the training effect of the multi-unmanned vessel cooperative encirclement training system proposed in this application through simulation experiments, such as... Figure 7 As shown, the simulation training environment is a two-dimensional simulated sea area of ​​300m×300m. The number of first-class intelligent agents used to simulate the capture of unmanned ships is 3, the number of second-class intelligent agents used to simulate the escape target is 1, the number of obstacles is 5, and the number of communication shielding zones is 1.

[0167] During the training process, at the beginning of each round, the positions and speeds of the first and second types of intelligent agents are randomly set, as are the positions of obstacles and communication shielding areas. To improve training efficiency and avoid getting bogged down in meaningless exploration during training, the maximum step size per round is set to 60 in this embodiment to prevent long-term ineffective searches during training. The entire training process consists of 50,000 rounds, and the decision cycle of both the first and second types of intelligent agents is set to 1 second.

[0168] Table 1 lists some parameters of the training scenario.

[0169] Table 1 Specific Implementation Example 1 Training Scenario Parameters

[0170] parameter numerical values <![CDATA[The maximum speed v of the first type of agent amax (m / s)]]> 10 <![CDATA[The maximum acceleration a of the first type of agent amax (m / s 2 )]]> 3 Maximum angular velocity (rad / s) of a Type I intelligent agent π / 4 Size of the first type of intelligent agent (m) 2 <![CDATA[The initial position x of the first type of agent a , y a (m)]]> [-140,140],[-140,140] <![CDATA[The maximum speed v of the second type of agent dmax (m / s)]]> 7 <![CDATA[The maximum acceleration a of the second type of agent dmax (m / s 2 )]]> 2 Maximum angular velocity (rad / s) of the second type of intelligent agent π / 4 Size of the second type of intelligent agent (m) 3 Initial position (m) of the second type of agent [-140,140],[-140,140] <![CDATA[Initial position of obstacle x o , y o (m)]]> [-140,140],[-140,140] Obstacle dimensions (m) 15 Communication shielding zone dimensions (m) 25 Initial position of the communication shielding zone (m) [-140,140],[-140,140]

[0171] Figures 8 to 10 The average reward curves for training three Class I agents using the training system provided in this application are shown for comparison. Figures 8 to 10The average reward curves using the conventional MADDPG algorithm are also shown. By comparison, it is clear that for each type I agent, the convergence speed of training using the technical solution of this application is faster than that of the conventional MADDPG algorithm.

[0172] To verify the training effect of the main policy network, three first-class agents were trained to cooperate in surrounding and capturing second-class agents under the control of the main policy network. Figure 11 The escape trajectories of the second type of agents and the capture trajectories of the various first type of agents are shown. Figure 11 It can be seen that the three first-class intelligent agents can cooperate with each other, effectively avoid obstacles and communication shielding areas, and track and intercept the second-class intelligent agents from different directions, and successfully achieve encirclement and capture.

[0173] The specific embodiments of this application have been described in detail above. For those skilled in the art, several improvements and modifications can be made to this application without departing from the principle of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A multi-unmanned vessel collaborative encirclement and capture training system, comprising: Multiple first-class intelligent agents are used to simulate the behavior of encircling and capturing unmanned ships. The first-class intelligent agents carry a master policy network, a master Q network, a target policy network, and a target Q network. At least one second-type agent is used to simulate the behavior of an escaping target, and the second-type agent carries an escape policy network; The simulation environment unit is used to provide a training environment in which multiple first-type intelligent agents cooperate to surround and capture multiple second-type intelligent agents, and to enable each intelligent agent to interact with the environment and reward each first-type intelligent agent according to the interaction results. The training environment includes at least one obstacle and at least one communication shielding area. An experience pool is used to store training samples for multi-unmanned vessel collaborative encirclement and capture. The training samples are generated based on the interaction and reward of each first-class intelligent agent with the environment. The optimization unit uses the training samples to optimize the policy networks and Q networks of each first-class agent using the MADDPG algorithm; Its features are: Both the main Q-network and the target Q-network include an attention module, which is used to compress the information dimensionality of the global observation information and action information received by the first type of agent based on attention feature extraction. For any first-type intelligent agent At the current moment Received global observation information and global actions for: ， in, The number of first-class intelligent agents. For the first type of intelligent agent The ID of any first-class intelligent agent other than , They are respectively the first type of intelligent agents and the first type of intelligent agent At the present moment Received observation information, , They are respectively the first type of intelligent agents and the first type of intelligent agent At the present moment The action, This is a function for concatenating elements. For any first-type intelligent agent The attention modules in its main Q-network and target Q-network include: An observation-action encoder is used to encode the global observation information and global actions received by the first type of intelligent agent into an embedding vector corresponding to each first type of intelligent agent; The first attention compression layer is used to compress the first type of intelligent agent. The embedding vectors corresponding to each of the first-class intelligent agents other than those are compressed into global attention embedding vectors; The first attention-based fully connected layer is used to connect the first type of intelligent agents. The corresponding embedding vector and the global attention embedding vector are concatenated and output as an attention feature vector; For any first-type intelligent agent The first attention layer generates the current time step based on the weighted summation model shown in the following formula. Global attention embedding vector : ， in, For the encoding function of the observation-action encoder, For the first Embedding vectors corresponding to each agent; for The weighting coefficients, No. The inner product of parameterized matrices of the embedding vectors corresponding to each agent. , For parameter matrices, This is a transpose.

2. The multi-unmanned vessel collaborative encirclement training system according to claim 1, characterized in that, The observation-action encoder includes: A first entity encoder is used to encode the observation information of the first type of intelligent agent corresponding to the observation-action encoder into an encoding vector; A second entity encoder is used to encode the observation information of the first type of agent corresponding to the observation-action encoder on other entities into an encoding vector, wherein the entities include the first type of agent, the second type of agent, obstacles in the training environment, and communication shielding areas. The total number of entities; An action encoder is used to encode the actions of the first type of agent corresponding to the observation-action encoder into an encoding vector; At least one second attention compression layer is used to... -1 The encoded vector output by the second entity encoder is subjected to global attention compression; The second attention fully connected layer is used to connect the inputs of the first entity encoder, the action encoder and the second attention compression layer and output them as an embedding vector.

3. The multi-unmanned vessel collaborative encirclement training system according to claim 1, characterized in that, It also includes high-value path units, which are used to identify and store combinations of high-value path samples formed by each Type I agent in the process of executing the current action strategy and several historical action strategies; The optimization unit uses samples from the experience pool and combinations of high-value path samples to optimize the main policy network, target policy network, main Q network, and target Q network of each first-class agent.

4. The multi-unmanned vessel cooperative encirclement training system according to claim 3, characterized in that, The high-value path unit includes: The evaluation module is used to evaluate the overall trajectory importance of each path generated by the first type of intelligent agent executing the current action strategy and several historical action strategies, and to determine the high-value path. A storage module is used to store the high-value path sample combination, wherein the high-value path sample combination is the collection of training samples corresponding to all times of the high-value path.

5. The multi-unmanned vessel cooperative encirclement training system according to claim 4, characterized in that, The evaluation module evaluates any path generated by the first type of intelligent agent based on the following formula. Assess the overall trajectory importance: in, For the time on this path The dominant function, 0 to The time elapsed along this path to For a moment At that time The corresponding TD error, For the time on this path of value, For weight parameters, It is a function of standard deviation. It is a mean function. , , These are the weighting coefficients.

6. The multi-unmanned vessel cooperative encirclement training system according to claim 4, characterized in that, The total number of paths identified as high-value paths at any given time shall not exceed a preset upper limit; Furthermore, the storage module updates its stored high-value path sample combinations based on a first-in-first-out (FIFO) approach.

7. The multi-unmanned vessel cooperative encirclement training system according to claim 1, characterized in that, The rewards for each Type I agent include target task reward, collision avoidance reward, communication maintenance reward, and boundary reward.

8. A training method for coordinated encirclement and capture of multiple unmanned vessels, characterized in that, The multi-unmanned vessel cooperative encirclement training system described in claim 1 is used to train multiple first-type intelligent agents; The training is carried out in multiple rounds, and each round includes a training sample generation phase and a MADDPG-based collaborative encirclement strategy optimization phase.

Images