Method for unmanned surface vehicle swarm hunting by using feature embedding and curriculum learning thought

Abstract

The application discloses a method for hunting of unmanned ship cluster by using feature embedding and course learning thought, and simulates environment construction, adopts PettingZoo open source framework to establish a simulation environment, determines the number of unmanned ships in the system through the framework, and is used for hunting and chasing tasks; behavior rules of escape unmanned ships are written, and a deep reinforcement learning model is built; the unmanned ship cluster is trained by using deep reinforcement learning; in order to improve the robustness of the model, the failure of the unmanned ship in the process of executing the task influences the whole task, and the feature embedding technology is used for the state space to realize the extension invariance. The escape radius is set by using the course learning thought, the course learning thought is applied in the training stage, the escape unmanned ship is initially set to be static, the escape speed is zero, the hunting radius condition is set to be large, and then the escape speed is gradually increased and the hunting radius is gradually reduced, so that the unmanned ship cluster can better adapt to the real task scene.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a method for swarming and capturing unmanned surface vessels using feature embedding and curriculum learning concepts. Background Technology

[0002] Unmanned surface vessels (USVs), as intelligent, multi-purpose unmanned transport platforms that operate remotely or autonomously, are a significant indicator of a nation's marine technological strength. USVs possess the capability to perform dangerous, time-consuming, and labor-intensive tasks in harsh environments, offering a wide range of applications in both military and civilian sectors. However, the capabilities of a single USV are limited. Faced with complex environments and diverse tasks, USV swarming has become a development trend and an inevitable result of informatization and intelligentization. Multiple USVs are not merely an expansion of quantity or a simple addition of functions, but rather an organic integration into a system, highly fused and working in unison. Currently, USV swarming for maneuvering is receiving increasing attention, and how to achieve USV swarming for maneuvering has practical value.

[0003] Current research on multi-agent reinforcement learning largely focuses on terrestrial or aerial environments, while the characteristics of unmanned surface vessels (USVs) differ significantly from those of land robots and aerial drones. Traditional deep reinforcement learning models for multi-agent systems typically utilize MPE (multi-agent particle environment) for training. However, this model has several drawbacks: 1) it cannot customize the magnitude of action values ​​according to its own needs; 2) the reward function is also provided by the MPE and cannot be adjusted to meet specific requirements.

[0004] MAPPO is a deep reinforcement learning algorithm, a variant of PPO applied to multi-agent reinforcement learning. PPO is a policy gradient optimization algorithm based on the Actor-Critic framework, addressing the problem of excessively large update discrepancies through gradient truncation. As a multi-agent variant of PPO, MAPPO is also an Actor-Critic algorithm, differing in that it uses centralized training with decentralized execution (CTDE). Specifically, the Critic network learns a central value function. All agents share this network, and after training, each agent, based on its own observation state, takes the optimal action based on the action function generated by its own Actor network. Summary of the Invention

[0005] To address the shortcomings of traditional multi-agent deep reinforcement models and achieve unmanned surface vessel (USV) swarm capture missions in complex environments, this invention proposes a method for USV swarm capture using feature embedding and curriculum learning. A simulation environment is constructed using the PettingZoo framework, and then the USV swarm model is trained using the MAPPO algorithm.

[0006] This invention is specifically implemented through the following technical solutions:

[0007] A method for swarming and hunting unmanned surface vessels using feature embedding and curriculum learning concepts, including...

[0008] S1) Simulation Environment Construction: The PettingZoo open-source framework is used to build a simulation environment. The number of unmanned surface vessels in the system is determined by this framework and used for pursuit and capture missions.

[0009] S2) Deep reinforcement learning training: Use deep reinforcement learning to train the unmanned surface vessel swarm, write behavioral rules for the escaped unmanned surface vessels, and enable them to adapt to the encirclement mission in different situations;

[0010] S3) Feature embedding idea: Directly concatenating the observation information of multiple unmanned surface vessels into a one-dimensional vector cannot flexibly handle scenarios with dynamically changing input dimensions. In order to improve the robustness of the model, feature embedding technology is used to achieve scalability invariance in the state space for individual unmanned surface vessels that may fail when performing tasks.

[0011] S4) Course learning concept: Apply the course learning concept during the training phase. Initially, the escape unmanned surface vessel is set to be stationary with an escape speed of zero and a relatively large capture radius. Then, the escape speed is gradually increased and the capture radius is reduced, so that the unmanned surface vessel swarm can better adapt to real mission scenarios.

[0012] This invention first utilizes the PettingZoo open-source framework to construct a simulation environment that mimics a real-world ocean scenario. Within this simulation environment, considering potential obstacles in actual missions, obstacles are added. By setting a negative reward for unmanned surface vessels (USVs) approaching obstacles, this invention simulates dangerous situations that need to be avoided in real-world missions. At the start of each training round, the initial position of the USV is set, providing different starting conditions for the model and increasing the diversity of training.

[0013] For unmanned surface vessel (USV) swarm encirclement missions, this invention employs deep reinforcement learning for training. During the training phase, the USV swarm is trained using the MAPPO algorithm. The escaping USVs act according to given rules, dynamically adjusting their escape speed and direction based on factors such as the position and orientation of other USVs and the encirclement radius.

[0014] To flexibly handle potential dynamic changes in the input dimension, this invention introduces the concept of feature embedding. Using feature embedding, features are embedded into the input state space to achieve scalability invariance. Specifically, the observations of each unmanned surface vessel (USV) are first fed into a feedforward network with consistent weights to obtain the observation vectors of each USV. Then, the average of the observation vectors is calculated, and local environmental features are concatenated to obtain state input features independent of the number of USVs.

[0015] The observation inputs of the unmanned surface vessel are o1, o2, ..., o n The feedforward network f is

[0016] f = h(wo) i +b)

[0017] Where w and b are weights and biases, and h is the activation function.

[0018] Then, the mean of all eigenvectors f is calculated to obtain the feature dimension vector f, which is independent of the number of unmanned surface vessels. mean

[0019]

[0020] Finally, by concatenating the local features of the unmanned surface vessel, we obtain the final state input f required by the network. state :

[0021] f state =f mean +o env

[0022] Among them, o env The local characteristics of the unmanned surface vessel.

[0023] To improve the robustness and adaptability of the model, this invention introduces the concept of curriculum learning. In the early stages of training, the escaping unmanned surface vessels (USVs) are set to be stationary in one place with zero escape velocity and a relatively large capture radius, allowing the USV swarm to complete the task during the exploration phase. As training progresses, the speed of the escaping USVs is gradually increased while the capture radius is decreased, enabling the USV swarm to better adapt to real-world mission scenarios.

[0024] During the course, this invention employs a linear difficulty parameter modifier. The capture radius and escape UAV speed are dynamically adjusted to modify the training difficulty based on the current number of training steps and dynamic environmental conditions. This dynamic adjustment mechanism helps the model adapt more flexibly to different task scenarios.

[0025] Among them, the encirclement radius d of the unmanned surface vessel arr for:

[0026]

[0027] The speed v of the escape unmanned surface vessel esc The expression is:

[0028]

[0029] Where, α i S is the proportionality coefficient. t S represents the current training steps. total C is the maximum number of time steps defined in an episode. p For training ratio, This represents the average distance between the escaping unmanned surface vessel (USV) and the pursuing USV.

[0030] This invention designs a sophisticated reward function to guide unmanned surface vessel (USV) swarms in learning effective capture strategies. The reward function considers the distance and direction between USVs and escaped USVs, their relationships with other USVs, and their interactions with obstacles, thereby enabling the USV swarm to better execute capture missions.

[0031] Wherein, the reward value r i for:

[0032]

[0033] r st =r disForEsc +r disang F orPur +r disForObs +r step

[0034] r disForEsc =-k0d0

[0035]

[0036]

[0037] r step =-k4

[0038] Where, k i d is the proportionality coefficient. i Let δ represent the distance of the corresponding unmanned surface vessel, and δ be the standard deviation of the distance.

[0039] In deep reinforcement learning training, this invention employs the MAPPO algorithm, which updates parameters through an experience replay pool to improve the training efficiency and stability of the model. Gradient parameter pruning addresses the difficulties in policy gradient updates and the challenge of determining the step size. The MAPPO algorithm effectively handles the problem of multi-agent collaborative training, ensuring better performance for unmanned surface vessel swarms during training.

[0040] Compared with other technologies, the beneficial effects achieved by this invention include the following:

[0041] 1. This invention uses the PettingZoo open-source framework to build a simulation environment, which allows for customization of the action space and value function, enabling unmanned surface vessels to simulate real ocean conditions and obtain more realistic training results.

[0042] 2. This invention uses the MAPPO algorithm to train unmanned surface vessel (USV) swarms for encirclement tasks. Through a framework structure of centralized training and distributed execution, the USVs can output control information based on their own observation information, avoiding the problems of high dimensionality and difficulty in expansion caused by single-agent reinforcement learning algorithms when dealing with multi-agent problems.

[0043] 3. This invention adopts the idea of ​​feature embedding to embed the state space. By averaging the mapping states of each unmanned surface vessel, scalability invariance is achieved, which allows for flexible handling of individual unmanned surface vessel failures.

[0044] 4. This invention adopts the concept of course learning. By setting the speed and the encirclement radius of the escape unmanned surface vessel, the time for the unmanned surface vessel to pursue and encircle is simplified, enabling the unmanned surface vessel to complete the task faster. Finally, by setting the escape unmanned surface vessel to a normal escape speed and the encirclement radius to a normal radius, the model trained by the algorithm is made to better match the real task scenario. Attached Figure Description

[0045] Figure 1 Overall flowchart for the encirclement and capture of unmanned surface vessels (USVs)

[0046] Figure 2 A schematic diagram of the unmanned surface vessel swarm encirclement method provided as an example of the present invention.

[0047] Figure 3 A schematic diagram of the capture distance provided as an example of the present invention.

[0048] Figure 4 A schematic diagram of a policy network provided as an example of the present invention.

[0049] Figure 5 A schematic diagram of a value network provided as an example of the present invention. Detailed Implementation

[0050] The technical solutions in the embodiments of the present invention will now be described in a complete and clear manner with reference to the accompanying drawings and examples.

[0051] like Figure 1 As shown, the present invention provides a method for swarming and capturing unmanned surface vessels using feature embedding and curriculum learning concepts, comprising:

[0052] S1) Simulation Environment Construction: The PettingZoo open-source framework is used to build a simulation environment. The number of unmanned surface vessels in the system is determined by this framework and used for pursuit and capture missions.

[0053] S2) Deep reinforcement learning training: Use deep reinforcement learning to train the unmanned surface vessel swarm, write behavioral rules for the escaped unmanned surface vessels, and enable them to adapt to the encirclement mission in different situations;

[0054] S3) Feature embedding idea: Directly concatenating the observation information of multiple unmanned surface vessels into a one-dimensional vector cannot flexibly handle scenarios with dynamically changing input dimensions. In order to improve the robustness of the model, feature embedding technology is used to achieve scalability invariance in the state space for individual unmanned surface vessels that may fail when performing tasks.

[0055] S4) Course learning concept: Apply the course learning concept during the training phase. Initially, the escape unmanned surface vessel is set to be stationary with an escape speed of zero and a relatively large capture radius. Then, the escape speed is gradually increased and the capture radius is reduced, so that the unmanned surface vessel swarm can better adapt to real mission scenarios.

[0056] In step S1), a simulation environment is built under the PettingZoo open-source framework. This environment contains obstacles. Unmanned surface vessels (USVs) that approach obstacles will receive negative rewards. At the start of each training round, the initial position of the USV is set in the simulation environment. After each action is performed, a corresponding reward function is set to guide the USV to learn.

[0057] In step S2), training the unmanned surface vessel (USV) swarm using deep reinforcement learning includes using the MAPPO algorithm to train the USV swarm using deep reinforcement learning. The escaping USV performs a weighted sum based on the distance and direction of other USVs and escapes in the opposite direction at a certain speed to simulate a real-world scenario.

[0058] In step S3), the concept of feature embedding is used to realize state input features independent of the number of unmanned surface vessels (USVs). Specifically, the observation input of the USVs is o1, o2, ..., o n The feedforward network f is:

[0059] f = h(wo) i +b)

[0060] Where w and b are weights and biases, and h is the activation function.

[0061] Then, the mean value of the feature vector f for all unmanned surface vessels is calculated to obtain the feature dimension vector f that is related to the number of unmanned surface vessels. mean

[0062]

[0063] Finally, by concatenating the local features of the unmanned surface vessel, we obtain the final state input f required by the network. state :

[0064] f state =f mean +o env

[0065] Among them, o env The local characteristics of the unmanned surface vessel.

[0066] In step S4), training the unmanned surface vessel (USV) swarm model using the concept of course learning includes introducing a parameter modifier with linear difficulty and dynamically adjusting the capture radius and the speed of the escaping USV based on the current number of training steps. The actual capture radius and the speed of the escaping USV output by the parameter modifier are adjusted according to exponential decay within the training ratio threshold to improve the robustness of the training.

[0067] Specifically, the encirclement radius and the speed of the escaping unmanned surface vessel are dynamically adjusted, assuming the current training steps are S. t The actual capture radius d output by the parameter modifier arr The expression is:

[0068]

[0069] Speed ​​of the escape unmanned boat The expression is:

[0070]

[0071] Where, α i S is the proportionality coefficient. total C is the maximum number of time steps defined in an episode. p d represents the training ratio, and d is the average distance between the escaping unmanned surface vessel (USV) and the pursuing USV.

[0072] The reward function is defined as follows: the reward is determined by the distance between the tracking UWD and the escaping UWD; the reward is determined by the distance and direction between the tracking UWDs; the reward is determined by the tracking UWD and the recorded distance between the tracking UWD and the obstacle; and the reward is determined by the time steps taken. The reward is r.

[0073]

[0074] r st =r disForEsc +r disangForPur +r disForObs +r step

[0075] r disForEsc =-k0d0

[0076]

[0077]

[0078] r step =-k4

[0079] Where, k i d is the proportionality coefficient. i Let δ represent the distance of the corresponding unmanned surface vessel, and δ be the standard deviation of the distance.

[0080] Among them, the MAPPO algorithm is used to train the unmanned surface vessel swarm. During the training phase, the environment outputs the global state s. t The reward function R and the environment Env give the reward r obtained by each unmanned surface vessel. t and observation o t Each unmanned surface vessel (USV) gives action a based on the strategy network. t , obtain the updated global state s t+1 Based on this, one iteration of the time step is completed, and the intermediate state information [s] is recorded. t o t a t r t s t+1 The data is stored in the experience replay pool. When the episodic condition is met, the parameters of the policy network and the value network are updated.

[0081] Figure 2 As shown, this method takes into account the actual environment of the unmanned surface vessel in the ocean, such as obstacles. Guided by the reward function of deep reinforcement learning, the unmanned surface vessel can make appropriate decisions during the trial and error process, avoid obstacles, and complete the encirclement and capture mission.

[0082] Specifically, the entire model includes the following steps: the escaping unmanned surface vessel (USV) escapes according to fixed rules, its escape speed is inversely proportional to the speed of the pursuing USV, and the encirclement radius is also dynamically adjusted, such as... Figure 3 As shown, in order to accelerate the success rate of unmanned surface vessel (USV) swarm encirclement missions in the early stages, a larger radius was set for successful USV encirclement. Under certain conditions, namely S... t / S total ≥C p At that time, the encirclement radius was set to a fixed value. The pursuit unmanned surface vessels (USVs) were trained using the MAPPO algorithm. The parameters of the USV network were initialized. Specifically, a centralized training and distributed execution approach was adopted to train the USV swarm on the environment in which it operated. Figure 4 This is a schematic diagram of a policy network. Figure 5 This is a schematic diagram of a value network. Figure 4In this process, the policy network receives parameters from the observations of a single unmanned surface vessel (USV), including its own position and environmental information. These parameters are passed through two fully connected layers, with the ReLU activation function used. Finally, the network outputs the USV's acceleration and angular acceleration for the next time step. Figure 5 The value network takes as input the sum of observation and action information from each unmanned surface vessel (USV) globally. This sum is then passed through two fully connected layers using the ReLU activation function, and finally through a linear connected layer to output the overall value for that time step. The training data for this iteration is then added to a buffer. After this epoisode, the network parameters are updated. This process is repeated until the target point is reached, at which point iteration stops, and the network parameters are obtained.

[0083] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for swarming and capturing unmanned surface vessels using feature embedding and curriculum learning concepts, characterized in that: include S1) Simulation Environment Construction: The PettingZoo open-source framework is used to build the simulation environment. The number of unmanned surface vessels in the system is determined through this framework for pursuit and capture missions. S2) Deep reinforcement learning training: Use deep reinforcement learning to train the unmanned surface vessel swarm, write behavioral rules for the escaped unmanned surface vessels, and enable them to adapt to the encirclement mission in different situations; Step S2) involves training the unmanned surface vessel (USV) swarm using deep reinforcement learning, which includes using the MAPPO algorithm to train the USV swarm. The escaping USV performs a weighted summation based on the distance and direction of other USVs and escapes in the opposite direction at a certain speed to simulate a real-world scenario. S3) Feature embedding idea: Directly concatenating the observation information of multiple unmanned surface vessels into a one-dimensional vector cannot flexibly handle scenarios with dynamically changing input dimensions. In order to improve the robustness of the model, feature embedding technology is used to achieve scalability invariance in the state space for individual unmanned surface vessels that may fail when performing tasks. In step S3), the feature embedding concept is used to embed features into the input state space to achieve scalability invariance. Specifically, firstly, a feedforward network with consistent observation input weights for each unmanned surface vessel (USV) is used to obtain the observation vectors of each USV. Then, the mean of the observation vectors is calculated, and the local features of the environment are spliced ​​together to obtain state input features that are independent of the number of USVs. S4) Course Learning Approach: The course learning approach is applied during the training phase. Initially, the escaping unmanned surface vessels (USVs) are set to be stationary with zero escape speed and a relatively large capture radius. Subsequently, the escape speed is gradually increased and the capture radius is decreased, allowing the USV swarm to better adapt to real-world mission scenarios. The step S4) of training the unmanned surface vessel swarm model using the concept of course learning includes introducing a parameter modifier with linear difficulty and dynamically adjusting the capture radius and the speed of the escaping unmanned surface vessel according to the current number of training steps. The actual capture radius and the speed of the escaping unmanned surface vessel output by the parameter modifier are adjusted according to exponential decay within the training ratio threshold to improve the robustness of training. Specifically, the encirclement radius and the speed of the escaping unmanned surface vessel are dynamically adjusted, assuming the current training steps are... The actual capture radius output by the parameter modifier. The expression is: , Speed ​​of the escape unmanned boat The expression is: in, This is the proportionality coefficient. This is the maximum number of time steps defined in an episode. For training ratio, This represents the average distance between the escaping unmanned surface vessel (USV) and the pursuing USV.

2. The method for swarming and hunting unmanned surface vessels using feature embedding and curriculum learning concepts as described in claim 1, characterized in that: In step S1), a simulation environment is built under the PettingZoo open-source framework. This environment contains obstacles. Unmanned surface vessels (USVs) that approach obstacles will receive negative rewards. At the start of each training round, the initial position of the USV is set in the simulation environment. After each action is performed, a corresponding reward function is set to guide the USV to learn.

3. The method for swarming and hunting unmanned surface vessels using feature embedding and curriculum learning concepts as described in claim 2, characterized in that: The reward function is set as follows: the reward is determined by the distance between the tracking UAV and the escaping UAV; the reward is determined by the distance and direction between the tracking UAVs; the reward is determined by the tracking UAV's records of its interaction with obstacles; and the reward is determined by the time steps taken. The reward is... , in, This is the proportionality coefficient. For the distance of the corresponding unmanned surface vessel, denoted as the standard deviation of the distance.

4. The method for swarming and hunting unmanned surface vessels using feature embedding and curriculum learning concepts as described in claim 1, characterized in that: The MAPPO algorithm is used to train the unmanned surface vessel swarm. Specifically, during the training phase, the environment outputs the global state. Reward function The Environment Env provides the rewards obtained by each unmanned surface vessel. and observation Each unmanned surface vessel (USV) gives actions based on the strategy network. Get the updated global state Based on this, an iterative loop of one time step is completed, and the intermediate state information is recorded. The data is stored in the experience replay pool. When the episode reaches the termination condition, the parameters of the policy network and value network are updated.

5. The method for swarming and capturing unmanned surface vessels using feature embedding and curriculum learning concepts as described in claim 1, characterized in that: The unmanned surface vessel observation input is Feedforward network for in, and For weights and biases, For activation functions; Then for all eigenvectors Calculate the mean to obtain the feature dimension vector related to the number of unmanned surface vessels. Finally, by concatenating the local characteristics of the unmanned surface vessel, the final state input required by the network is obtained. : in, The local characteristics of the unmanned surface vessel.

Images