An unmanned boat intelligent capture control method, device and equipment

Abstract

The present invention provides an unmanned boat intelligent capture control method, device, and equipment, relating to the field of unmanned boat intelligent control technology. The method comprises obtaining the current state information of the unmanned boat and the local environment data perceived by the current unmanned boat to obtain the perception state information corresponding to the current unmanned boat; constructing a structured state input vector based on the perception state information using a distance sorting and fill-and-cut mechanism; inputting the structured state input vector into a hierarchical strategy network, wherein the high-level layer generates task roles and behavior decisions, and the low-level layer combines the high-level output and the structured state input vector to generate navigation speed and direction control instructions; wherein the hierarchical strategy network optimizes a preset strategy network using reinforcement learning through a structured reward mechanism; and controlling the current unmanned boat to execute a capture mission on a target boat according to the navigation speed and direction control instructions. The solution of the present invention effectively improves the stability and flexibility of the multi-boat collaborative strategy and the success rate of the capture mission.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned boat intelligent control, and in particular to an unmanned boat intelligent capture control method, device and equipment. Background Art

[0002] In recent years, research on unmanned surface vehicle (USV) technology for dynamic target monitoring and intelligent interception in the ocean has deepened, particularly in the field of multi-vessel capture and control. Through the division of labor and cooperation among multiple vessels, USVs can effectively encircle and suppress dynamic targets. However, existing USV intelligent capture and control technologies still face the following challenges: First, the coordination mechanism is still mainly based on static formations or centralized control rules, lacking a flexible role division mechanism for dynamic target tasks; second, the input state design is generally simplified, failing to effectively integrate target behavior, environmental disturbances, and multi-vessel situation information, resulting in poor strategy adaptability to environmental changes; third, the functional differences between different roles in the capture process are insufficiently modeled, resulting in weak strategy generalization and difficulty in handling targets with different numbers and behavior patterns; fourth, during training, the reward function often suffers from sparseness and insufficient collaborative guidance, which restricts the formation of high-quality strategies and affects the success rate of capture missions. Summary of the Invention

[0003] The present invention provides an unmanned boat intelligent capture control method, device and equipment, which solve the problems existing in existing unmanned boat intelligent capture control methods, such as single capture strategy role allocation, inability of state input to adapt to dynamic entity changes, and low capture success rate.

[0004] In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0005] An embodiment of the present invention provides an intelligent capture control method for an unmanned boat, comprising:

[0006] Obtain the status information of the current unmanned boat and the local environment data perceived by the current unmanned boat, and construct a local coordinate system based on the target boat to represent the status information, and obtain the perception status information corresponding to the current unmanned boat;

[0007] Based on the perception state information, a distance sorting and padding and clipping mechanism is used to construct a structured state input vector;

[0008] The structured state input vector is input into a hierarchical policy network, wherein the upper layers of the hierarchical policy network generate task roles and behavioral decisions, and the lower layers combine the upper layer outputs with the environmental state in the structured state input vector to generate navigation speed and direction control instructions; wherein the hierarchical policy network is obtained by optimizing a preset policy network using reinforcement learning through a structured reward mechanism;

[0009] The current unmanned boat is controlled to execute the encirclement and capture mission on the target boat according to the navigation speed and direction control instructions.

[0010] Optionally, the state information of the current unmanned boat and the local environment data perceived by the current unmanned boat are obtained, and a local coordinate system is constructed based on the target boat to represent the state information, thereby obtaining the perception state information corresponding to the current unmanned boat, including:

[0011] Obtain the current status information of the unmanned boat and the local environment data perceived by the unmanned boat;

[0012] A local coordinate system is established with the position of the target boat as the origin. The status information of the current unmanned boat and the local environmental data perceived by the current unmanned boat are uniformly represented in the local coordinate system to obtain the perception status information corresponding to the current unmanned boat; wherein, the local environmental data includes: the status of the target boat, the status of the cooperative unmanned boat, the status of the obstacle and the status of the ocean environment.

[0013] Optionally, through a reward mechanism, reinforcement learning is used to optimize the preset policy network to obtain a hierarchical policy network including:

[0014] Construct typical mission scenarios in the unmanned boat capture training environment and generate training status data;

[0015] Based on the training state data, and in response to the dynamic changes in the number of cooperative unmanned boats and obstacles, a distance sorting, filling, and clipping mechanism is used to organize the state information perceived by the current unmanned boat into an input vector in a unified format;

[0016] Input the input vector to the input layer of the preset strategy network, perform feature fusion on the input vector through the multi-head attention layer, and generate a fusion representation of the multi-source state;

[0017] The decision layer of the preset policy network generates task roles and behavior decisions based on the output of the multi-head attention layer, and outputs high-level discrete action instructions;

[0018] The base layer of the policy network combines the output of the decision layer with the output of the multi-head attention layer to generate continuous control instructions;

[0019] Execute continuous control commands and interact with the unmanned boat capture training environment to update the status of the unmanned boat;

[0020] Based on the structured reward function, the immediate reward is calculated according to the updated state of the unmanned boat and the continuous control instructions. The preset policy network is optimized by the multi-agent proximal policy optimization algorithm to obtain a hierarchical policy network.

[0021] Optionally, construct a typical mission scenario in the unmanned boat capture training environment and generate training status data, including:

[0022] In the unmanned boat roundup training environment, a target boat, a current unmanned boat, and multiple coordinated unmanned boats are set up to simulate the interaction between them and the environment, and construct a typical roundup mission scenario;

[0023] The training status data is generated through a typical capture mission scenario, and the training status data includes the status data of the target boat, the current unmanned boat, the coordinated unmanned boat and the obstacles.

[0024] Optionally, based on the training state data, and in response to the dynamic changes in the number of cooperative unmanned boats and obstacles, a distance sorting, filling, and clipping mechanism is used to organize the state information perceived by the current unmanned boat into an input vector in a unified format, including:

[0025] Based on the training state data, state objects perceived by the current unmanned boat in the training state data are sorted according to their relative distances from the current unmanned boat, the state objects including a variable number of cooperative unmanned boat state objects and obstacle state objects perceived by the current unmanned boat;

[0026] When the number of state objects exceeds the preset maximum value, they are pruned. If it does not reach the maximum value, they are padded with zero vectors. The sorted, pruned and padded state data are constructed into an input vector of a unified format.

[0027] Optionally, a multi-head attention layer is used to perform feature fusion processing on various state information to generate a fused representation of multiple source states, including:

[0028] Embed all kinds of state information in the input vector of unified format into the feature space of the same dimension to obtain the input matrix;

[0029] Based on the input matrix, the outputs of multiple attention heads are calculated, and the results of each attention head are concatenated and linearly mapped to generate a fusion representation of the multi-source state.

[0030] Optionally, the structured reward function includes:

[0031] The forward role reward function is used for the forward unmanned boat to quickly approach the target boat and block it head-on; the flanking role reward function is used for the flanking unmanned boat to assist in encircling and enhancing deterrence coordination from both sides; and the rear guard role reward function is used for the rear guard unmanned boat to suppress the target boat from behind and deter intervention in high-threat situations.

[0032] An embodiment of the present invention further provides an unmanned boat intelligent capture control device, comprising:

[0033] The acquisition module is used to obtain the status information of the current unmanned boat and the local environment data perceived by the current unmanned boat, and to construct a local coordinate system based on the target boat to represent the status information, thereby obtaining the perception status information corresponding to the current unmanned boat;

[0034] A processing module, configured to construct a structured state input vector based on the perception state information by adopting a distance sorting and padding clipping mechanism;

[0035] a control module configured to input the structured state input vector into a hierarchical policy network, wherein the upper layers of the hierarchical policy network generate task roles and behavioral decisions, and the lower layers combine the upper layer outputs with the environmental state in the structured state input vector to generate navigation speed and direction control instructions; wherein the hierarchical policy network is obtained by optimizing a preset policy network using reinforcement learning through a structured reward mechanism;

[0036] The execution module is used to control the current unmanned boat to execute the capture mission on the target boat according to the navigation speed and direction control instructions.

[0037] An embodiment of the present invention further provides a computing device, comprising: a processor and a memory, wherein the memory stores a computer program, and when the program runs on the processor, the computer executes the above method.

[0038] An embodiment of the present invention further provides a computer-readable storage medium storing instructions, which, when executed on a computer, enable the computer to execute the above method.

[0039] The above solution of the present invention includes at least the following beneficial effects:

[0040] The unmanned boat intelligent capture control method of the present invention obtains the state information of the current unmanned boat and the local environment data perceived by the current unmanned boat, and constructs a local coordinate system based on the target boat to represent the state information, thereby obtaining the perception state information corresponding to the current unmanned boat; based on the perception state information, a distance sorting and filling and clipping mechanism is used to construct a structured state input vector; the structured state input vector is input into a hierarchical strategy network, the high-level layer of the hierarchical strategy network generates task roles and behavior decisions, and the low-level layer combines the high-level output and the environmental state in the structured state input vector to generate navigation speed and direction control instructions; wherein, the hierarchical strategy network uses a structured reward mechanism to optimize the preset strategy network using reinforcement learning; the current unmanned boat is controlled to perform the capture task on the target boat according to the navigation speed and direction control instructions. The present invention effectively improves the stability, flexibility and generalization ability of the multi-boat collaborative strategy, especially in a complex marine dynamic target environment, and significantly improves the success rate of the capture task. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 11 is a flow chart of the unmanned boat intelligent capture control method of the present invention;

[0042] Figure 2 It is a structural diagram of the hierarchical strategy network in the unmanned boat intelligent capture control method of the present invention;

[0043] Figure 3 Schematic diagram of the training process of the strategy network in the unmanned boat intelligent capture control method of the present invention;

[0044] Figure 4 Schematic diagram of role distribution and coordination strategy in the unmanned boat capture task in the unmanned boat intelligent capture control method of the present invention;

[0045] Figure 5 It is a schematic diagram of the module structure of the unmanned boat intelligent capture control device of the present invention. DETAILED DESCRIPTION

[0046] Exemplary embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in a variety of ways and is not limited to the embodiments described herein. These embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the technical scope of the present invention to those skilled in the art.

[0047] like Figures 1 to 4 As shown, an embodiment of the present invention provides an unmanned boat intelligent capture control method, comprising:

[0048] Step 11: Obtain the status information of the current unmanned boat and the local environment data perceived by the current unmanned boat, and construct a local coordinate system based on the target boat to represent the status information, thereby obtaining the perception status information corresponding to the current unmanned boat;

[0049] Step 12: Based on the perception state information, a distance sorting and padding and clipping mechanism is used to construct a structured state input vector;

[0050] Step 13: Input the structured state input vector into a hierarchical policy network. The high-level layers of the hierarchical policy network generate task roles and behavioral decisions, and the low-level layers combine the high-level outputs with the environmental state in the structured state input vector to generate navigation speed and direction control instructions. The hierarchical policy network is obtained by optimizing a preset policy network using reinforcement learning through a structured reward mechanism. Specifically, the preset policy network is obtained by offline training using reinforcement learning based on a reward mechanism designed based on role division and environmental state.

[0051] Step 14: Control the current unmanned boat to execute the capture mission on the target boat according to the navigation speed and direction control instructions.

[0052] In this embodiment, each unmanned boat completes decision-making and control based on the perceived local environmental information. All state information is uniformly represented in a local coordinate system with the target boat position as the origin, and is organized into a structured state input vector, which is input into a strategy network with high-level and low-level decoupling; the strategy network consists of a decision layer and a basic layer: wherein the decision layer generates high-level discrete action instructions according to the input state vector, including task role allocation (forward, flank, defender) and behavior selection (no deterrence, non-lethal deterrence, lethal deterrence), which are used to guide the tactical mode; the task roles have different divisions of labor: the forward role is responsible for quickly approaching the target and blocking its escape path head-on; the flank role is responsible for interfering with and controlling the target's maneuvering space from both sides; the defender role is responsible for suppressing from the rear of the target boat and assuming deterrence and defense control tasks; the basic layer combines the environmental state and the output results of the decision layer to generate control instructions including target speed and heading angle to guide the actual movement of the unmanned boat; the target speed is a normalized control value with a range of , corresponding to the maximum reverse to maximum forward speed; the target heading angle is The angle between them represents the target direction in the local coordinate system. Through the design of "strategy layering + action decoupling", the strategy network is supported to dynamically generate individual behaviors according to the local situation during task execution, thereby improving the multi-boat coordination efficiency, structural flexibility and control robustness during the capture process. In this embodiment, the unmanned boat intelligent capture control method is suitable for mission scenarios such as offshore monitoring, rapid interception, and formation control for dynamic targets, and can significantly improve the level of intelligent collaborative control of surface unmanned boat clusters in complex sea conditions.

[0053] In this embodiment, the structured state input vector is composed of the following five sub-vectors: current UAV state, target UAV state, cooperative UAV state, obstacle state, and ocean environment state. To maintain consistent input format and adapt the policy network, each state is first serialized into a unified vector, defined as: ;in, is the structured state input vector, is the current state of the unmanned boat, is the target boat status, ( ) is the cooperative unmanned boat state; ( ) is the obstacle state, is the environmental state; and Represents the number of observed cooperative unmanned boats and obstacles respectively. The above two types of states are sorted from near to far according to the relative distance to the current unmanned boat to facilitate subsequent cutting and filling operations;

[0054] In an optional embodiment of the present invention, step 11 may include:

[0055] Step 111, obtaining the current state information of the unmanned boat and the local environment data perceived by it;

[0056] Step 112: Establish a local coordinate system with the position of the target boat as the origin (east is Axis, North direction Axis), in which the state information of the current unmanned boat and its perceived local environment data are uniformly represented to obtain the perception state information corresponding to the current unmanned boat, wherein the local environment data includes: the state of the target boat, the state of the cooperative unmanned boat, the state of the obstacle and the state of the marine environment (environmental state).

[0057] In this embodiment, by constructing a local coordinate system with the target boat position as the origin to represent multi-source state information, the spatial consistency of the input data can be enhanced; the various state information specifically includes: the state vector of the current unmanned boat ,in, is the current position of the unmanned boat in the local coordinate system, is the current speed of the unmanned boat, is the current heading angle of the unmanned boat, The mission role of the current unmanned boat, 0 means forward, 1 means flank, 2 means defender is the current behavior state of the unmanned boat, 3 means no deterrence, 4 means non-lethal deterrence, and 5 means lethal deterrence; the state vector of the target boat is ,in, is the speed of the target boat, is the heading angle of the target boat, is the behavior state of the target boat, 6 means stopping, 7 means escaping, and 8 means attacking; the state vector of the cooperative unmanned boat is ,in, For the The location of the coordinated unmanned boat, For the The speed of a coordinated unmanned boat, For the The heading angle of a cooperative unmanned boat, For the A collaborative unmanned boat mission role, For the The behavior state of the cooperative unmanned boat; the state vector of the obstacle ,in, is the position of the obstacle center, is the speed of the obstacle, is the direction of movement of the obstacle, is the radius of the obstacle equivalent circle; the state vector of the ocean environment ,in, is the wind speed, For wind direction, For the high waves, For the wave direction.

[0058] In an optional embodiment of the present invention, the hierarchical strategy network is optimized by using reinforcement learning through a structured reward mechanism, including:

[0059] Step 131: Construct a typical mission scenario in the unmanned boat roundup training environment and generate training state data; the training state data includes: state data of the target boat, the current unmanned boat, the coordinated unmanned boat, and obstacles, as well as environmental disturbance data; the typical mission scenario includes a target boat, a current unmanned boat, and multiple coordinated unmanned boats; the scenario includes a target boat, a current unmanned boat, and multiple coordinated unmanned boats, and sets obstacle and environmental disturbance factors;

[0060] Step 132: Based on the training state data, and in response to the dynamic changes in the number of cooperative unmanned vehicles and obstacles, a distance sorting, filling, and clipping mechanism is used to organize the state information sensed by the current unmanned vehicle into a unified format input vector for subsequent network processing;

[0061] Step 133: Input the input vector to the input layer of the preset strategy network, perform feature fusion processing on various state information through the multi-head attention layer, and generate a fusion representation of multi-source states;

[0062] In step 134, the decision layer of the preset policy network generates task roles and behavior decisions based on the fusion representation and outputs high-level discrete action instructions. The high-level discrete action instructions include role assignment (e.g., forward, wing, defender) and behavior selection (e.g., no deterrence, non-lethal deterrence, lethal deterrence).

[0063] Step 135: The base layer of the policy network combines the fusion representation with the output of the decision layer to generate continuous control instructions, including the normalized target speed and heading angle, to guide the actual movement of the unmanned boat.

[0064] Step 136 , executing the control command and interacting with the training environment (unmanned boat capture training environment) to update the current unmanned boat status;

[0065] In step 137, according to the structured reward function, the immediate reward is calculated based on the updated state and action (continuous control instructions), and the policy network is optimized and trained using the multi-agent proximal policy optimization algorithm (MAPPO), ultimately obtaining a control strategy with good task adaptability.

[0066] In this embodiment, the unmanned boat capture training environment refers to a comprehensive simulation platform for developing and evaluating capture control algorithms. This environment integrates a hydrodynamic physics engine, a sensor and sea state disturbance model, an action-state interaction interface, a reward evaluation interface, and a log visualization tool. It is capable of simulating the real-world dynamic responses of target boats, unmanned boats, and obstacles, as well as their interactions with environmental factors such as wind and waves, with high fidelity on a computer. The training environment provides a large, configurable, repeatable, and operational "virtual testing ground" that doesn't require a real boat. It provides a closed-loop data flow for reinforcement learning algorithms: the policy network outputs actions, and the environment returns the next state and immediate rewards, thereby supporting iterative optimization of the policy and performance verification. A typical mission scenario is a specific mission configuration loaded once in the training environment to define the participating entities and initial conditions for a capture round. The scenario consists of a fixed target vessel, a current unmanned vessel (the training subject), and multiple coordinated unmanned vessels. Static or dynamic obstacles can be deployed as needed, and sea conditions such as wind speed, wave height, and wave direction can be set. Each scenario also specifies the starting position, speed, role label, and the rules for determining the end of the roundup, enabling the training environment to generate continuous time-series state data and reward signals during runtime. By batch-generating different typical mission scenarios, it can cover a variety of target maneuvers, the number of coordinated vessels, and complex sea conditions, thereby improving the generalization and robustness of the policy network.

[0067] like Figure 2 As shown, in this embodiment, the preset policy network adopts a multi-layered structure with decoupled high and low layers, responsible for high-level policy generation and low-level control instruction output, respectively. The high-level policy network (decision layer) and the low-level control network (base layer) share a set of multi-head attention modules to extract multi-source fusion feature representations from the structured state input vector, unify the state information related to the modeling task, and improve feature expression efficiency and overall network stability.

[0068] The decision layer is used to generate high-level discrete action instructions, including task role assignment and behavior selection; role types include forward (0), wing (1), and defender (2); behavior types include no deterrence (3), non-lethal deterrence (4), and lethal deterrence (5). The output discrete action vector is recorded as: ,in, Indicates the role mission currently performed by the unmanned boat. Indicates the current behavior command of the unmanned boat; the decision layer network includes two fully connected hidden layers, containing 256 and 128 neurons respectively, with the activation function being ReLU. The output layer uses a softmax structure to normalize the probability of the role and the behavior action respectively, and selects the action corresponding to the maximum probability as the execution command;

[0069] The base layer (low-level control network) generates continuous control action vectors based on the received fusion features and high-level outputs to adjust the navigation behavior of the unmanned boat. The output action is defined as: ,in, represents the normalized target speed, Represents maximum reverse, Represents the greatest advancement, Represents the target heading angle in the local coordinate system; the input vector of the base layer is the fusion state feature Output action with decision layer The splicing result is expressed as: ,in, , is the attention output dimension; the base layer network also includes two fully connected hidden layers, containing 256 and 128 neurons respectively, and the activation function is ReLU, and the final output control action , guiding the unmanned boat to execute the target navigation speed and course;

[0070] like Figure 3 As shown, in this embodiment, offline reinforcement learning of the policy network is completed in an iterative manner in the unmanned boat capture training environment. The specific process is as follows:

[0071] Initialize the capture scene, set up a target boat, a current unmanned boat and several cooperative unmanned boats; establish a local coordinate system with the target boat as the origin, synchronously obtain the current unmanned boat, cooperative unmanned boat, obstacles and sea conditions and express them in a unified way; on this basis, generate a unified format input vector, sort the cooperative unmanned boats and obstacle states according to the distance from the current unmanned boat, if the number exceeds the limit, trim them, if it is insufficient, fill them with zero vectors to maintain a fixed dimension; the input vector is fused with multi-source information through the multi-head attention module (the current unmanned boat state is used as the query vector, and the other states are used as keys / values), and the output is hidden The features are used by subsequent networks. The high-level policy network probabilistically assigns discrete actions such as role assignment and deterrence based on the fused features. The low-level control network combines high-level actions with environmental features to generate normalized velocity and heading angles. Each unmanned vehicle navigates according to its own instructions, while the target vehicle and the environment evolve synchronously. Immediate returns are calculated based on a structured reward function (role reward + environmental safety reward + task-level reward), and network parameters are adjusted using the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm. If the continuous capture score reaches a threshold, a termination reward is assigned and the current round ends; otherwise, the next cycle begins. Through this "high- and low-level decoupling + action decoupling" architecture, collaborative unmanned vehicles with different roles can form differentiated and transferable capture strategies during training, significantly improving the capture success rate in complex sea conditions.

[0072] In an optional embodiment of the present invention, step 131 may include:

[0073] Step 1311: Construct a capture scenario in the unmanned boat capture training environment, including a target boat, a current unmanned boat, and multiple coordinated unmanned boats. Simultaneously generate obstacles and ocean environment disturbances to simulate their interactions with each other and the environment, thereby constructing a typical capture mission scenario.

[0074] Step 1312: Generate training status data through a typical capture mission scenario. The training status data includes the status data of the target boat, the current unmanned boat, the coordinated unmanned boat, and the obstacles; that is, record the initial status of the target boat, the current unmanned boat, the coordinated unmanned boat, the obstacles and the ocean environment, and start the capture training round.

[0075] In a preferred embodiment, in order to improve the realism of the encirclement and capture training environment, the following key scenario elements can be further introduced, including: obstacle modeling: setting static obstacles (such as fixed facilities, navigation marks) and dynamic obstacles (such as moving ships, floating objects); environmental disturbance modeling: parameterizing sea disturbances such as wind speed, wind direction, wave height, and wave direction, and inputting them into the strategy network as state variables; game confrontation simulation: the target boat performs evasive and breakthrough maneuvers based on the anti-encirclement and capture strategy to test the robustness of the strategy.

[0076] In an optional embodiment of the present invention, step 132 may include:

[0077] Step 1321, based on the training state data, sort the state objects perceived by the current unmanned boat in the training state data according to the relative distance from the current unmanned boat. The state objects include a variable number of cooperative unmanned boat state objects and obstacle state objects perceived by the current unmanned boat; these two types of objects are heterogeneous targets with dynamically changing numbers in the capture mission, and need to be sorted separately according to the relative distance from the current unmanned boat in order to unify the input format, including the cooperative unmanned boat state (such as position, speed, heading, role, etc.); obstacle state (such as position, speed, direction, radius, etc.).

[0078] Step 1322: When the number of state objects exceeds the preset maximum value, they are pruned; if the number is less than the preset maximum value, they are filled with zero vectors, and the sorted, pruned, and filled state data are constructed into an input vector in a unified format.

[0079] In this embodiment, the cooperative unmanned boats and obstacles are sorted from near to far according to their distance from the current unmanned boat: if the number exceeds the upper limit 、 , only the most recent Cooperative unmanned boats and The state information of obstacles; if it is insufficient, it will be padded with a zero vector at the end; the processing results are spliced ​​to obtain an input vector of fixed dimension ,in, is the current state of the unmanned boat, is the target state, and They are the states of the cooperative unmanned boat and obstacles after sorting / cropping / filling, This preprocessing can keep the input dimension consistent when the number of entities changes dynamically, ensuring stable network operation and improving system scalability.

[0080] In an optional embodiment of the present invention, step 133 may include:

[0081] Step 1331 , embedding various types of state information in the unified format input vector into a feature space of the same dimension to obtain an input matrix;

[0082] Step 1332, based on the input matrix, uses a multi-head attention mechanism to perform feature fusion, specifically including: calculating the output of multiple attention heads (using the current UAV state information as the query vector and the remaining states as keys and values, calculating the attention weights through multiple independent attention heads and obtaining the output), and concatenating the results of each attention head and performing linear mapping to generate a fused feature representation as the input of subsequent modules of the policy network.

[0083] In this embodiment, to enhance the policy network's ability to model multi-source state information and improve its perception and decision-making capabilities in complex roundup tasks, the present invention introduces a customized multi-head attention mechanism into the network structure. The multi-head attention mechanism is embedded in the policy network as a state aggregation module, and uses structured input vectors to fuse multi-source information, improving the ability to express heterogeneous situations. The specific operations are as follows:

[0084] The input vector ,in, 、 、 、 、 Represent the state vectors of the current unmanned boat, target boat, cooperative unmanned boat, obstacle and ocean environment respectively, 、 In order to coordinate the maximum number of unmanned boats and obstacles, the input dimension consistency is maintained by cropping and filling; the above five types of state data are embedded into the unified dimension The feature space of constructing the input matrix: , ,in Represents the total number of state sources, corresponding to the current UAV, target boat, cooperative UAV, obstacles, and ocean environment in sequence. Based on the scaled dot product attention mechanism, the embedded representation of the current UAV state is used as the query vector, and the remaining states are used as context information (keys and values). The attention calculation process is constructed as follows: , , ,in, Represents the embedded features of the current unmanned boat; Indicates the status inputs other than the current unmanned boat. 、 、 is a trainable linear transformation matrix corresponding to the transformation parameters of query, key and value vectors respectively, and are the dimensions of the query / key vector and the value vector respectively; the above vectors pass through the attention function: Calculate the attention weight and aggregate feature representation of the current unmanned boat for each state source; In order to enhance the expression ability of different feature subspaces, introduce Parallel attention heads, each head independently computes the attention output: ; The outputs of each attention head are concatenated and linearly mapped to obtain the final fusion feature output: ,in, is the output mapping matrix, is the dimension of the final fusion feature; this fusion feature serves as a shared input for the decision layer and the basic layer of the strategy network, which enables the unmanned boat to dynamically pay attention to the target position, collaborative formation, obstacle threats and sea disturbances according to the real-time situation, thereby improving the perception accuracy and collaborative control performance in the encirclement and capture scenario.

[0085] In an optional embodiment of the present invention, the structured reward function includes role-specific reward items designed based on task division, including:

[0086] The forward role reward function is used for the forward unmanned boat to quickly approach the target boat and block it head-on; the flanking role reward function is used for the flanking unmanned boat to assist in encircling and enhancing deterrence coordination from both sides; and the rear guard role reward function is used for the rear guard unmanned boat to suppress the target boat from behind and deter intervention in high-threat situations.

[0087] like Figure 4 As shown, in this embodiment, for the behavioral goals of the three types of task roles, forward, wing, and defender, role-specific reward functions are designed respectively, including:

[0088] Forward role reward function ;in, Indicates the change in the relative distance between the forward and target boats, used to evaluate the immediate approach efficiency; Indicates the change in the relative distance between the forward and target boats in the current frame relative to the initial moment, which is used to measure the overall approach trend; represents the positive blockade effect bonus, defined as: , where the cutoff angle Reflects the interception capability of the striker in the escape direction of the target boat (e.g. Figure 4 in ), defined as: , and are the two-dimensional position vectors of the target boat and the forward boat, is the velocity vector of the target boat; is the weighting coefficient of the forward role bonus;

[0089] Flanker role reward function ;in, Indicates the degree of coordination with the forward in forming an encirclement angle. The angle between the target boat and the front and flank is Figure 4 in and ), is the ideal angle (e.g. ), Control angle sensitivity; Measure the impact of the distance between the flank and the target boat on the deterrent effect, is the distance between the flank and the target boat, The maximum distance at which deterrence takes effect; The weight of the flanking role bonus item;

[0090] Defender role reward function ; Among them, the suppression situation reward , To quantify whether the defender is behind the target boat in the escape direction, the suppression angle (e.g. Figure 4 in ), is the two-dimensional position vector of the defender; deterrence response matching reward , Indicates whether the target boat is in a threatening state (e.g. escape / attack). is the real-time distance between the defender and the target boat, To determine the optimal distance for deterrence, Controlling the distance sensitivity of deterrent responses; is the current defender's deterrent behavior level, The definition is as follows: , 0 means no deterrence, Represents the reward intensity of non-lethal and lethal deterrence respectively; dynamic distance maintenance reward , and are the minimum and maximum acceptable tracking distances, is the recommended tracking distance; are the weighting coefficients of suppression, deterrence response and distance keeping reward respectively.

[0091] In a preferred embodiment, the structured reward function further sets an environment interaction related reward item to enhance the obstacle avoidance capability and path stability of the unmanned vehicle under complex sea conditions, mainly including two submodules: collision avoidance reward and collision penalty;

[0092] The total environmental safety reward is defined as: ;

[0093] Among them, the obstacle avoidance reward is used to encourage the unmanned boat to actively avoid surrounding obstacles and maintain a safe distance. It is expressed by an exponential function: ,in, Indicates the Euclidean distance between the current unmanned boat and the nearest obstacle. is the perceptual range scale parameter, To prevent a small constant with a zero denominator, is the weight coefficient of the collision avoidance reward item;

[0094] The collision penalty term is used to prevent the unmanned boat from entering the high-risk area and is expressed by a continuous risk function based on the Sigmoid function: ,in, is the maximum penalty intensity, is the distance threshold for collision risk determination, is the response slope coefficient.

[0095] In a preferred embodiment, the structured reward function further sets a task-level reward item, and guides the strategy to stably converge to an effective roundup solution through a task termination reward mechanism based on continuous situation scoring, specifically including:

[0096] Comprehensively measure the distance between the individual and the target and the direction suppression, and construct a capture success scoring function ,in, is the number of effective unmanned boats currently involved in the roundup, For the The distance between the unmanned boat and the target boat, To determine the distance threshold for effective capture, Used to truncate negative values, is the weight of the distance term and the direction term, Indicates the target boat's direction of motion and the The azimuth angle of the unmanned boat; the role direction suppression function Differentiated rewards are given according to role type: in, Indicates that the current character is a forward. Indicates that the current character is a wing. Indicates that the current character is a defender. and It is an angle-sensitive parameter used to adjust the accuracy of wing double-teaming and guard suppression.

[0097] In this embodiment, in order to improve the stability of the roundup success judgment, the sliding window method is used to smooth the score fluctuation; the window length is set to , at the moment Calculate the average rating within the window ,in For the Instant scoring of frames; when the conditions are met , that is, the sliding average capture score is not lower than the set capture success threshold , the round is deemed successful and ends;

[0098] This embodiment further sets up task-level rewards to provide global feedback based on the capture results at the end of the round. Its core logic includes: when the sliding average capture score meets the requirements, , that is, when the roundup is successful, the termination reward is issued in a linear proportion of the roundup score ,in, The maximum reward value that can be obtained when the task is successful; if the current round reaches the maximum number of steps , but the roundup score is still below the mission success threshold , the task is considered failed and a fixed penalty is imposed ,in, The penalty value for mission failure; the mission termination reward is settled uniformly at the end of the round: ; If the roundup is successful, Indicates if the roundup fails, and 0 indicates other situations. In this embodiment, the total instant reward for each time step is ,in, For role-specific bonuses (forward, wing, defender), is the environment interaction reward (collision avoidance reward and collision penalty), Termination rewards for the above tasks; through the three-level reward structure of "role-environment-task", the algorithm can maintain individual division of labor and collaborative suppression during training, while taking into account safe tracks and task orientation, thereby significantly improving the success rate of encirclement and capture and the robustness of the strategy under complex sea conditions.

[0099] like Figure 5 As shown, an embodiment of the present invention further provides an unmanned boat intelligent capture control device 50, comprising:

[0100] An acquisition module 51 is used to obtain the status information of the current unmanned boat and the local environment data perceived by the current unmanned boat, and to construct a local coordinate system based on the target boat to represent the status information, thereby obtaining the perception status information corresponding to the current unmanned boat;

[0101] A processing module 52 is configured to construct a structured state input vector based on the perception state information by using a distance sorting and padding and clipping mechanism;

[0102] A control module 53 is configured to input the structured state input vector into a hierarchical policy network, wherein the upper layers of the hierarchical policy network generate task roles and behavioral decisions, and the lower layers combine the upper layer outputs with the environmental state in the structured state input vector to generate navigation speed and direction control instructions; wherein the hierarchical policy network is obtained by optimizing a preset policy network using reinforcement learning through a structured reward mechanism;

[0103] The execution module 54 is used to control the current unmanned boat to execute the capture mission on the target boat according to the navigation speed and direction control instructions.

[0104] Optionally, the state information of the current unmanned boat and the local environment data perceived by the current unmanned boat are obtained, and a local coordinate system is constructed based on the target boat to represent the state information, thereby obtaining the perception state information corresponding to the current unmanned boat, including:

[0105] Obtain the current status information of the unmanned boat and the local environment data perceived by the unmanned boat;

[0106] A local coordinate system is established with the position of the target boat as the origin. The status information of the current unmanned boat and the local environmental data perceived by the current unmanned boat are uniformly represented in the local coordinate system to obtain the perception status information corresponding to the current unmanned boat; wherein, the local environmental data includes: the status of the target boat, the status of the cooperative unmanned boat, the status of the obstacle and the status of the ocean environment.

[0107] Optionally, through a reward mechanism, reinforcement learning is used to optimize the preset policy network to obtain a hierarchical policy network including:

[0108] Construct typical mission scenarios in the unmanned boat capture training environment and generate training status data;

[0109] Based on the training state data, and in response to the dynamic changes in the number of cooperative unmanned boats and obstacles, a distance sorting, filling, and clipping mechanism is used to organize the state information perceived by the current unmanned boat into an input vector in a unified format;

[0110] Input the input vector to the input layer of the preset strategy network, perform feature fusion on the input vector through the multi-head attention layer, and generate a fusion representation of the multi-source state;

[0111] The decision layer of the preset policy network generates task roles and behavior decisions based on the output of the multi-head attention layer, and outputs high-level discrete action instructions;

[0112] The base layer of the policy network combines the output of the decision layer with the output of the multi-head attention layer to generate continuous control instructions;

[0113] Execute continuous control commands and interact with the unmanned boat capture training environment to update the status of the unmanned boat;

[0114] Based on the structured reward function, the immediate reward is calculated according to the updated state of the unmanned boat and the continuous control instructions. The preset policy network is optimized by the multi-agent proximal policy optimization algorithm to obtain a hierarchical policy network.

[0115] Optionally, construct a typical mission scenario in the unmanned boat capture training environment and generate training status data, including:

[0116] In the unmanned boat roundup training environment, a target boat, a current unmanned boat, and multiple coordinated unmanned boats are set up to simulate the interaction between them and the environment, and construct a typical roundup mission scenario;

[0117] The training status data is generated through a typical capture mission scenario, and the training status data includes the status data of the target boat, the current unmanned boat, the coordinated unmanned boat and the obstacles.

[0118] Optionally, based on the training state data, and in response to the dynamic changes in the number of cooperative unmanned boats and obstacles, a distance sorting, filling, and clipping mechanism is used to organize the state information perceived by the current unmanned boat into an input vector in a unified format, including:

[0119] Based on the training state data, state objects perceived by the current unmanned boat in the training state data are sorted according to their relative distances from the current unmanned boat, the state objects including a variable number of cooperative unmanned boat state objects and obstacle state objects perceived by the current unmanned boat;

[0120] When the number of state objects exceeds the preset maximum value, they are pruned. If it does not reach the maximum value, they are padded with zero vectors. The sorted, pruned and padded state data are constructed into an input vector of a unified format.

[0121] Optionally, a multi-head attention layer is used to perform feature fusion processing on various state information to generate a fused representation of multiple source states, including:

[0122] Embed all kinds of state information in the input vector of unified format into the feature space of the same dimension to obtain the input matrix;

[0123] Based on the input matrix, the outputs of multiple attention heads are calculated, and the results of each attention head are concatenated and linearly mapped to generate a fusion representation of the multi-source state.

[0124] Optionally, the structured reward function includes:

[0125] The forward role reward function is used for the forward unmanned boat to quickly approach the target boat and block it head-on; the flanking role reward function is used for the flanking unmanned boat to assist in encircling and enhancing deterrence coordination from both sides; and the rear guard role reward function is used for the rear guard unmanned boat to suppress the target boat from behind and deter intervention in high-threat situations.

[0126] It should be noted that the device is a device corresponding to the above method, and all implementation methods in the above method are applicable to the embodiments of the device and can achieve the same technical effects.

[0127] An embodiment of the present invention further provides a computing device comprising: a processor and a memory, wherein the memory stores a computer program that, when executed on the processor, causes the computer to execute the above-described method. All implementations in the above-described method embodiments are applicable to this embodiment and can achieve the same technical effects.

[0128] The embodiment of the present invention further provides a computer-readable storage medium storing instructions, which, when executed on a computer, causes the computer to execute the above method. All implementations in the above method embodiments are applicable to this embodiment and can achieve the same technical effects.

[0129] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. An intelligent capture control method for unmanned boats, characterized in that: include: Obtain the status information of the current unmanned boat and the local environment data perceived by the current unmanned boat, and construct a local coordinate system based on the target boat to represent the status information, and obtain the perception status information corresponding to the current unmanned boat; Based on the perception state information, a distance sorting and padding and clipping mechanism is used to construct a structured state input vector; The structured state input vector is input into a hierarchical policy network. The high-level layer of the hierarchical policy network generates task roles and behavioral decisions. The task roles include forward, wing, and defender. The behavioral decisions include no deterrence, non-lethal deterrence, and lethal deterrence. The low-level layer combines the high-level output with the environmental state in the structured state input vector to generate navigation speed and direction control instructions. The hierarchical policy network is obtained by optimizing the preset policy network using reinforcement learning through a structured reward mechanism, including: Construct typical mission scenarios in the unmanned boat capture training environment and generate training status data; Based on the training state data, and in response to the dynamic changes in the number of cooperative unmanned boats and obstacles, a distance sorting, filling, and clipping mechanism is used to organize the state information perceived by the current unmanned boat into an input vector in a unified format; Input the input vector to the input layer of the preset strategy network, perform feature fusion on the input vector through the multi-head attention layer, and generate a fusion representation of the multi-source state; The decision layer of the preset policy network generates task roles and behavior decisions based on the output of the multi-head attention layer, and outputs high-level discrete action instructions; The base layer of the policy network combines the output of the decision layer with the output of the multi-head attention layer to generate continuous control instructions; Execute continuous control commands and interact with the unmanned boat capture training environment to update the status of the unmanned boat; Based on the structured reward function, the immediate reward is calculated according to the updated state of the unmanned boat and the continuous control instructions. The preset policy network is optimized by the multi-agent proximal policy optimization algorithm to obtain a hierarchical policy network. The structured reward function includes: The forward role reward function is used for the forward unmanned boat to quickly approach the target boat and blockade it head-on; the flanking role reward function is used for the flanking unmanned boat to assist in encircling and enhancing deterrence coordination from both sides; the rear guard role reward function is used for the rear guard unmanned boat to suppress the target boat from behind and deter intervention in high-threat situations; The current unmanned boat is controlled to execute the encirclement and capture mission on the target boat according to the navigation speed and direction control instructions.

2. The intelligent capture control method of unmanned boat according to claim 1 is characterized in that: Obtain the current state information of the unmanned boat and the local environment data perceived by the current unmanned boat, and construct a local coordinate system based on the target boat to represent the state information, and obtain the perception state information corresponding to the current unmanned boat, including: Obtain the current status information of the unmanned boat and the local environment data perceived by the unmanned boat; A local coordinate system is established with the position of the target boat as the origin. The status information of the current unmanned boat and the local environmental data perceived by the current unmanned boat are uniformly represented in the local coordinate system to obtain the perception status information corresponding to the current unmanned boat; wherein, the local environmental data includes: the status of the target boat, the status of the cooperative unmanned boat, the status of the obstacle and the status of the ocean environment.

3. The intelligent capture control method of unmanned boat according to claim 1, characterized in that: Construct typical mission scenarios in the unmanned boat capture training environment and generate training status data, including: In the unmanned boat roundup training environment, a target boat, a current unmanned boat, and multiple coordinated unmanned boats are set up to simulate the interaction between them and the environment, and construct a typical roundup mission scenario; The training status data is generated through a typical capture mission scenario, and the training status data includes the status data of the target boat, the current unmanned boat, the coordinated unmanned boat and the obstacles.

4. The intelligent capture control method of unmanned boat according to claim 1, characterized in that: Based on the training state data, and in response to the dynamic changes in the number of cooperative unmanned boats and obstacles, a distance sorting, filling, and clipping mechanism is used to organize the state information perceived by the current unmanned boat into an input vector in a unified format, including: Based on the training state data, state objects perceived by the current unmanned boat in the training state data are sorted according to their relative distances from the current unmanned boat, the state objects including a variable number of cooperative unmanned boat state objects and obstacle state objects perceived by the current unmanned boat; When the number of state objects exceeds the preset maximum value, they are pruned. If it does not reach the maximum value, they are padded with zero vectors. The sorted, pruned and padded state data are constructed into an input vector of a unified format.

5. The intelligent capture control method of unmanned boat according to claim 1, characterized in that: The multi-head attention layer is used to perform feature fusion processing on various state information to generate a fusion representation of multi-source states, including: Embed all kinds of state information in the input vector of unified format into the feature space of the same dimension to obtain the input matrix; Based on the input matrix, the outputs of multiple attention heads are calculated, and the results of each attention head are concatenated and linearly mapped to generate a fusion representation of the multi-source state.

6. An intelligent capture control device for an unmanned boat, characterized in that: include: The acquisition module is used to obtain the status information of the current unmanned boat and the local environment data perceived by the current unmanned boat, and to construct a local coordinate system based on the target boat to represent the status information, thereby obtaining the perception status information corresponding to the current unmanned boat; A processing module, configured to construct a structured state input vector based on the perception state information by adopting a distance sorting and padding clipping mechanism; The control module is configured to input the structured state input vector into a hierarchical policy network. The high-level layer of the hierarchical policy network generates task roles and behavioral decisions, wherein the task roles include forward, wing, and defender; the behavioral decisions include no deterrence, non-lethal deterrence, and lethal deterrence; and the low-level layer combines the high-level output with the environmental state in the structured state input vector to generate navigation speed and direction control instructions. The hierarchical policy network is obtained by optimizing a preset policy network using reinforcement learning through a structured reward mechanism, including: Construct typical mission scenarios in the unmanned boat capture training environment and generate training status data; Based on the training state data, and in response to the dynamic changes in the number of cooperative unmanned boats and obstacles, a distance sorting, filling, and clipping mechanism is used to organize the state information perceived by the current unmanned boat into an input vector in a unified format; Input the input vector to the input layer of the preset strategy network, perform feature fusion on the input vector through the multi-head attention layer, and generate a fusion representation of the multi-source state; The decision layer of the preset policy network generates task roles and behavior decisions based on the output of the multi-head attention layer, and outputs high-level discrete action instructions; The base layer of the policy network combines the output of the decision layer with the output of the multi-head attention layer to generate continuous control instructions; Execute continuous control commands and interact with the unmanned boat capture training environment to update the status of the unmanned boat; Based on the structured reward function, the immediate reward is calculated according to the updated state of the unmanned boat and the continuous control instructions. The preset policy network is optimized by the multi-agent proximal policy optimization algorithm to obtain a hierarchical policy network. The structured reward function includes: The forward role reward function is used for the forward unmanned boat to quickly approach the target boat and blockade it head-on; the flanking role reward function is used for the flanking unmanned boat to assist in encircling and enhancing deterrence coordination from both sides; the rear guard role reward function is used for the rear guard unmanned boat to suppress the target boat from behind and deter intervention in high-threat situations; The execution module is used to control the current unmanned boat to execute the capture mission on the target boat according to the navigation speed and direction control instructions.

7. A computing device, characterized in that include: A processor and a memory, wherein the memory stores a computer program, and when the program is run on the processor, the method according to any one of claims 1 to 5 is executed.

8. A computer-readable storage medium, characterized in that The device stores instructions, which, when executed on a computer, cause the computer to perform the method according to any one of claims 1 to 5.

Images