Multi-ship cooperative control method and system based on hierarchical reward mechanism

Abstract

The invention relates to a multi-ship cooperative control method and system based on a hierarchical reward mechanism, and belongs to the field of intelligent ocean and multi-agent, and the method comprises the steps: S1, initializing a multi-ship system, each ship having an individual strategy network and a shared strategy network; s2, at each control time step, each ship obtains the current observation state, and a control action is generated and executed through the individual strategy network based on the current observation state; calculating a corresponding reward sequence based on the current observation state and the control action, and storing the current observation state, the control action, the reward sequence and the next observation state as shared sample data into an experience buffer pool; s3, after num control time steps are preset, obtaining shared sample data corresponding to the num control time steps from the experience buffer pool to train a shared strategy network, and obtaining trained shared strategy network parameters to update individual strategy network parameters of each ship; and returning to the step S2. And multi-ship behavior cooperation and stable control in a complex marine environment is realized.

Description

Technical Field

[0001] The present invention relates to the field of intelligent ocean and multi-agent technologies, and in particular to a multi-vessel collaborative control method and system based on a hierarchical reward mechanism. Background Art

[0002] As the core supporting technology in intelligent ocean systems, multi-vessel collaborative control has broad application prospects in scenarios such as formation cruising, target capture, collaborative search, disaster monitoring, and surface operations. With the scale and intelligent development of unmanned systems, higher requirements for stability, adaptability, and generalization capabilities are placed on their collaborative control systems. However, the current mainstream multi-vessel control methods still have significant shortcomings when facing complex environments and multi-task collaboration. When faced with complex ocean environments, dynamic task requirements, and multi-scale, multi-target collaborative control challenges, existing multi-vessel collaborative control methods generally expose the following technical bottlenecks, which seriously restrict the stability of system performance and the generalization ability of strategies:

[0003] It's difficult to unify local decision-making and global coordination. In a multi-vessel system, each vessel typically only plans its path and makes action decisions based on its own and neighboring sensory information, lacking effective understanding and modeling of the overall formation state, the overall mission objectives, and the intentions of other members. This information-island-like control structure easily leads to individual individuals acting selfishly in their local environment, which in turn reduces the overall system's collaborative effectiveness and can even cause formation disintegration or mission conflicts, making it difficult to achieve optimal global collaboration.

[0004] Traditional reward mechanisms are limited in design and lack dynamic adaptability. Traditional reinforcement learning methods typically rely on a single, static reward function for policy optimization. This design pattern cannot simultaneously meet the dual goals of optimizing local navigation behavior and maintaining a global collaborative structure. Furthermore, static incentives struggle to reflect dynamic characteristics such as environmental disturbances and changes in mission phases. This results in policies that may converge quickly in the early stages of training but lack generalization capabilities. In actual deployment, they are slow to respond to changing sea conditions or task switching, and are prone to overfitting or non-steady-state behavior.

[0005] In uncertain and dynamic ocean environments, system stability and robustness are difficult to ensure. Multi-vessel collaborative control systems typically operate in highly dynamic and uncertain environments, requiring real-time response to complex disturbances such as obstacle interference, unstructured water flow, communication delays and packet loss, and navigation errors. Existing methods generally lack mechanisms to adjust collaborative structures or incentive strategies based on environmental conditions. This results in the inability to promptly reconfigure system control logic in the face of emergencies or interference shocks, leading to interruptions in collaborative tasks or severe performance degradation. Summary of the Invention

[0006] In view of the above analysis, the embodiments of the present invention aim to provide a multi-vessel collaborative control method and system based on a hierarchical reward mechanism to solve the technical problems of traditional multi-vessel collaborative control, such as a single reward mechanism, poor dynamic adaptability, and weak collaborative modeling.

[0007] The purpose of the present invention is mainly achieved through the following technical solutions:

[0008] The present invention provides a multi-vessel collaborative control method based on a hierarchical reward mechanism, comprising the following steps:

[0009] Step S1: Initialize the multi-ship system, where each ship holds an individual policy network and a shared policy network;

[0010] Step S2: At each control time step, each ship obtains the current observation state, generates and executes a control action based on the current observation state through the individual policy network; calculates a corresponding reward tuple based on the current observation state and the control action, and stores the current observation state, control action, reward tuple, and next observation state as shared sample data in the experience buffer pool;

[0011] Step S3: After num control time steps are preset, shared sample data corresponding to num control time steps are obtained from the experience buffer pool to train the shared policy network, and the trained shared policy network parameters are obtained to update the individual policy network parameters of each ship; return to step S2.

[0012] Furthermore, the reward tuple includes a basic reward, a collaborative reward, and an adaptive reward, as well as a comprehensive reward calculated based on the basic reward, the collaborative reward, and the adaptive reward;

[0013] in,

[0014] The basic reward is used to guide each ship to approach the target point and sail safely;

[0015] The cooperation reward is used for cooperation among multiple ships and ensuring the formation of multiple ships;

[0016] The adaptive reward is used to dynamically adapt according to task progress and situational awareness.

[0017] Furthermore, the comprehensive rewards are as follows:

[0018]

[0019] Among them, r i (t), and are the comprehensive reward, basic reward, collaborative reward and adaptive reward of ship i at time step t; w b 、wc 、w a are the weights of basic reward, collaborative reward, and adaptive reward respectively.

[0020] Furthermore, the shared policy network is trained through the following process:

[0021] Using the shared sample data, the shared policy network is trained using a proximal policy optimization (PPO) algorithm;

[0022] The AdamW optimizer is used to update the shared policy network parameters until the clipping objective function converges, obtaining the trained shared policy network and saving the network parameters.

[0023] Furthermore, the basic reward as follows:

[0024]

[0025] in, are the target approach reward, obstacle avoidance reward and boundary penalty for ship i at time step t, respectively;

[0026] The target is close to the reward as follows:

[0027]

[0028] in, Indicates the maximum approach reward value, d i (t) is the Euclidean distance from ship i to the nearest target point, D tar is the reward decay distance scale;

[0029] The obstacle avoidance reward as follows:

[0030]

[0031] Where O is the number of obstacles in the obstacle set; P obs is the obstacle avoidance reward intensity, d ij (t) is the Euclidean distance from ship i to target point j, D obs Control the scope of obstacle avoidance rewards;

[0032] The boundary penalty as follows:

[0033]

[0034] Among them, P bnd is the penalty constant for exceeding the boundary; m is the edge safety width, B is the navigation boundary allowed, [m, Bm] 2 It is the safety interval after reducing the edge safety width.

[0035] Furthermore, the collaboration reward as follows:

[0036]

[0037] Among them, Q form (t), Q enc (t), They are formation maintenance reward, encirclement structure reward and leader mechanism reward; form 、w enc 、w lead The weights of formation retention rewards, encirclement structure rewards, and leader-follower mechanism rewards, respectively;

[0038] The formation maintains the reward Q from (t), as follows:

[0039]

[0040] Among them, D form (t) is the sum of the total deviations of all slave ships from their ideal positions at the current time step; is the number of slave ships in a multi-ship system, R f is the ideal formation deviation tolerance;

[0041] The enclosing structure reward Q enc (t), as follows:

[0042]

[0043] in, is the average encirclement radius of all ships in the multi-ship system to the target; R e To preset the ideal enclosing radius; is the maximum angular distance between all ships in the multi-ship system and the target; M is the number of ships participating in surrounding the target;

[0044] The following mechanism rewards as follows:

[0045]

[0046] Among them, Π i (t) Rate the leader ship’s current mission progress; The degree to which the subordinate ship maintains the formation structure of the leading ship.

[0047] Furthermore, the adaptive reward as follows:

[0048]

[0049] in, They are progress feedback reward, situational awareness reward and potential field guidance reward respectively; w pong 、w sit 、w pot are the weights of progress feedback rewards, context awareness rewards, and potential field guided rewards respectively;

[0050] The progress feedback rewards are as follows:

[0051]

[0052] Among them, D p is the progress reward scale factor; N is the total number of ships in the multi-ship system;

[0053] The context-aware rewards are as follows:

[0054]

[0055] Among them, D a 、D e They are the distance adjustment factor and the attenuation amplitude of the control reward in the target approach and obstacle avoidance scenarios respectively;

[0056] The potential field guide reward is as follows:

[0057]

[0058] Among them, d iL (t) is the distance between ship i and its leading ship; D pt ,D po ,D pf are the distance adjustment factors of the attraction field, repulsion field, and structural potential field respectively; δ i is a binary indicator variable. When ship i is a subordinate ship, δ i =1, otherwise 0.

[0059] Furthermore, the ideal position of the slave vessel is obtained as follows:

[0060] Get the leader ship position at the current control time step;

[0061] Based on the preset formation of the multi-ship system, the relative positions of the subordinate ships in the preset formation are obtained;

[0062] Based on the position of the leader ship at the current control time step and the relative position of the subordinate ships in the preset formation, the ideal position of the subordinate ships is generated in real time.

[0063] Furthermore, initializing the multi-vessel system includes setting a mission target area, a target point set, an obstacle set, a preset formation, and an allowed navigation boundary.

[0064] The present invention provides a multi-vessel cooperative control system based on a hierarchical reward mechanism, comprising:

[0065] System initialization module, used to initialize the multi-ship system, each ship holds an individual strategy network and a shared strategy network;

[0066] A data collection and storage module is configured to obtain the current observation state of each ship at each control time step, generate and execute control actions based on the current observation state through the individual policy network, calculate a corresponding reward sequence based on the current observation state and control action, and store the current observation state, control action, reward sequence, and next observation state as shared sample data in an experience buffer pool;

[0067] The policy network training and updating module is used to obtain shared sample data corresponding to num control time steps from the experience buffer pool after a preset num control time steps, to train the shared policy network, obtain the trained shared policy network parameters, and update the individual policy network parameters of each ship; then return to the data collection and storage module to continue execution.

[0068] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0069] 1. By constructing a hierarchical reward mechanism consisting of basic rewards, collaborative rewards, and adaptive rewards, this invention effectively guides individual ships to approach the target point and navigate safely (basic rewards). It also establishes a coordination mechanism among multiple ships to ensure formation (collaborative rewards), and dynamically adjusts rewards based on mission progress and environmental changes (adaptive rewards). Compared to the existing technology, which struggles to unify local decision-making and global coordination, this method organically combines local behavior optimization with global collaborative goals, improving the overall collaborative efficiency of the multi-ship system and avoiding problems such as formation breakdown or mission conflicts caused by individual selfish behavior.

[0070] 2. Compared with traditional reinforcement learning methods, which usually rely on a single, static reward function and have insufficient generalization capabilities, the present invention adopts the proximal policy optimization (PPO) algorithm to train the shared policy network and utilizes the shared sample data in the experience buffer pool during the training process, so that the strategy can adapt to the dynamic changes in the complex marine environment. By dynamically adjusting the reward mechanism, the strategy has stronger adaptability to environmental disturbances, changes in task stages, etc., avoiding the overfitting or non-steady-state behavior of the strategy caused by static incentives in the existing technology, improving the stability and generalization ability of the strategy in actual deployment, and enhancing the adaptability of the strategy.

[0071] 3. To address the inadequate stability and robustness of existing technologies in uncertain and dynamic ocean environments, this invention updates shared policy network parameters in real time and applies them to each ship's individual policy network. This ensures that the entire multi-vessel system can promptly adjust its control logic in the face of complex disturbances such as obstacle interference, communication delays, and packet loss. This continuous parameter update and optimization mechanism effectively enhances the system's robustness to emergencies and interference shocks, reduces the risk of collaborative mission interruption or severe performance degradation, and ensures the stable operation of the multi-vessel collaborative control system.

[0072] In the present invention, the above-mentioned technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of the present invention will be described in the following description, and some advantages will become apparent from the description or be learned through practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] The accompanying drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like parts throughout the drawings.

[0074] Figure 1 This is a flow chart of a multi-vessel collaborative control method based on a hierarchical reward mechanism in an embodiment of the present invention;

[0075] Figure 2 This is a schematic diagram of hierarchical rewards in an embodiment of the present invention;

[0076] Figure 3 Schematic diagram of a multi-vessel collaborative control system module based on a hierarchical reward mechanism in an embodiment of the present invention. DETAILED DESCRIPTION

[0077] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0078] To address these challenges, this paper proposes a multi-vessel collaborative control method and system based on a hierarchical reward mechanism. By constructing a three-tiered incentive structure consisting of basic rewards (for individual navigation and safety control), collaborative rewards (for maintaining formation and collaborative relationships), and adaptive rewards (for responding to mission changes and environmental disturbances), this method achieves spatial hierarchical guidance and temporal control of vessel behavior. This reward mechanism, deeply integrated with a multi-agent reinforcement learning algorithm, enables strategic self-organization, behavioral coordination, and structural stability control of multi-vessel systems in dynamic ocean environments, effectively improving their collaborative efficiency, learning robustness, and deployment adaptability.

[0079] Multi-Ship Control refers to the design and implementation of a set of control strategies for a collaborative system composed of multiple autonomous ships (usually unmanned surface vehicles, USVs) in marine missions, so that each ship can achieve the goals of overall formation, collaborative mission execution, obstacle avoidance navigation and dynamic adjustment while maintaining its individual autonomy.

[0080] This invention takes hierarchical structure and dynamic mechanism as its core, and by constructing a more expressive and adaptable three-layer incentive, it provides theoretical support and engineering implementation path for the collaborative control of multi-vessel systems under complex tasks.

[0081] The policy network in this invention is seamlessly integrated with mainstream multi-agent reinforcement learning algorithms (such as Proximal Policy Optimization, PPO), taking into account both learning stability and policy generalization performance, and improving the actual deployment capability and cross-task migration capability in complex marine environments.

[0082] Example 1

[0083] A specific embodiment of the present invention discloses a multi-vessel cooperative control method based on a hierarchical reward mechanism, such as Figure 1 As shown, the following steps are included:

[0084] Step S1: Initialize the multi-ship system, where each ship holds an individual policy network and a shared policy network;

[0085] Step S2: At each control time step, each ship obtains the current observation state, generates and executes a control action based on the current observation state through the individual policy network; calculates a corresponding reward tuple based on the current observation state and the control action, and stores the current observation state, control action, reward tuple, and next observation state as shared sample data in the experience buffer pool;

[0086] Step S3: After num control time steps are preset, shared sample data corresponding to num control time steps are obtained from the experience buffer pool to train the shared policy network, and the trained shared policy network parameters are obtained to update the individual policy network parameters of each ship; return to step S2.

[0087] Step S1, specifically.

[0088] Within the reinforcement learning framework, the multi-ship system is modeled as a multi-agent system (MAS). Each ship is considered an agent, making independent decisions based on local observations within limited sensing range and communication conditions. Its control objectives include not only its own safety and mission accomplishment, but also coordination with other agents in time and space. Each ship maintains both an individual policy network and a shared policy network.

[0089] The individual policy network and the shared policy network have the same structure. Each ship has an independent individual policy network, and all ships in the multi-ship system have a shared policy network. The training of the shared policy network depends on the data obtained by the individual policy network on each ship. The parameters of the individual policy network are optimized to update and optimize the parameters of the trained shared policy network.

[0090] Key features of multi-vessel control include:

[0091] Local perception, distributed decision-making: Each ship takes action based on its own and its neighbors' information, embodying decentralized control;

[0092] Cooperative constraints and structure maintenance: maintaining the formation structure and executing multi-objective division of labor tasks;

[0093] Dynamic environmental adaptation: Robustness and adaptability in the face of environmental disturbances and task switching;

[0094] Three-tier reward-driven optimization: Optimize control strategies through multi-level reward mechanisms to achieve global collaborative goals.

[0095] The control objectives of the multi-ship control system are decomposed into three levels: basic navigation control, collaborative structure maintenance, and task adaptive adjustment. End-to-end strategy optimization is achieved through a hierarchical reward mechanism combined with reinforcement learning methods, thereby improving the control performance and deployment reliability of the multi-ship system under complex sea conditions and multiple tasks.

[0096] In this method, the multi-ship system contains a total of N autonomous ships, denoted as the agent set A = {1, 2, …, N}, and each ship is numbered i∈A.

[0097] The initialization of the multi-vessel system includes setting a mission target area, a target point set, an obstacle set, a preset formation, and an allowed navigation boundary.

[0098] Set the mission target area Obstacle Collection Initial formation structure And the allowed navigation boundary Ω. All ships are based on their initial state s i (0) Initialization, each ship has its own individual strategy network and shared policy network π θθ i , θ is the parameter set of the individual policy network and the shared policy network, and is the weight and bias of the feedforward deep neural network.

[0099] In this invention, the individual policy network (PolicyNetwork) and the shared policy network (SharedPolicyNetwork) have the same structure and adopt a feedforward neural network (FFNN). The input of the policy network is the observed state vector of the ship at the current time step, and the output is the action probability distribution or the specific action vector. The individual / shared policy network adopts the following structure:

[0100] (1) Input layer: Receives the current observation state vector, including its own state, target information, obstacle information, and neighbor ship information;

[0101] (2) Hidden layer: It consists of several fully connected layers and uses activation functions (such as ReLU and Tanh) to improve nonlinear modeling capabilities. For example, two hidden layers are used; ranging from 64 to 128 neuron units;

[0102] (3) Output layer: For deterministic strategies, the output action mean and variance are used to construct a Gaussian distribution for sampling; for discrete actions, the probability distribution of each action is output and normalized using the softmax function.

[0103] The output of the individual policy network on each ship is the action (heading adjustment angle, speed change) selected by each ship in the current control time step.

[0104] A shared policy network refers to a neural network structure in which all agents (ships) share the same set of policy network parameters, and the parameters θ are shared among multiple agents.

[0105] In a multi-ship system, designing individual strategy networks and shared strategy networks has the following advantages: reducing the number of model parameters and training complexity; achieving cross-agent transfer of experience and improving sample efficiency; and facilitating collaborative learning of symmetric structures, which is particularly suitable for homogeneous fleets (multi-ship systems with similar structures and functions).

[0106] The main function of the policy network is to output the control action of the current agent given the current observation state.

[0107] In the present invention, the input of the policy network is the current observation state, and the output is the control action; the individual policy network is used to implement the autonomous strategy of multi-ship collaborative tasks; the shared policy network is used to achieve strategy optimization in multi-ship collaborative tasks.

[0108] The function of step S1 is to initialize the multi-ship system, configure the individual policy network and shared policy network for each ship, and set the initial state and parameters, laying the foundation for subsequent control and training steps.

[0109] Step S2 includes steps S21-S23.

[0110] Step S21: At each control time step, each ship obtains the current observation state, generates and executes a control action based on the current observation state through the individual strategy network.

[0111] Each ship generates a state-action-reward tuple at each control time step;

[0112] Continuously iteratively update the parameters of the individual policy network to make it more inclined to output better control actions under the current reward structure.

[0113] At each control time step t, ship i obtains the current observation state s i (t), including:

[0114] Own state: own position p i (t), speed v i (t), heading ψ i (t);

[0115] Target information: relative position, speed, radius, distance to the nearest target point p target distance

[0116] Obstacle information: relative position, speed, radius, type, and the closest distance to the nearest obstacle

[0117] Neighbor ship information: relative position, relative speed, heading, role;

[0118] Environmental complexity indicator: communication signal quality.

[0119] Neighboring ship information can only be sensed when the neighboring ship is within communication range and the current ship's communication quality is greater than 0.2. 1.0 represents the best communication quality, and 0 represents no communication capability. A communication quality greater than 0.2 means the communication link between the two ships is available.

[0120] The relative position, relative speed, direction, and role of the perceived neighboring ships are as follows:

[0121] Relative position (rel x ,rel y ), in order to ensure that the relative position of the neighboring ship is within the communication range, the position is normalized, where rel x rel yare the normalized longitude and latitude of the neighbor ship;

[0122] Relative speed (rel vx ,rel vy ), in order to ensure that the relative speed is within the communication range, the speed is normalized, where rel vx rel vy are the ship's longitude and latitude speeds respectively.

[0123] The heading (sin, cos) describes the direction the neighboring ship is facing. The heading is calculated by angle, where sin and cos represent the angles in the longitude and latitude of the neighboring ship, respectively.

[0124] Role: A value of 1.0 indicates a leader ship, and 0.0 indicates a follower ship. This role assignment is mandatory for transmission to ensure precedence in the event of information conflicts. For example, a leader ship might be responsible for navigation or decision-making, while a follower ship might be responsible for following or performing specific tasks. Only by knowing the roles of neighboring ships can effective division of labor and collaboration be achieved.

[0125] When making decisions, ships need to adjust their behavior based on the roles of their neighbors. For example, a follower ship may prioritize following a leader ship, or give priority to the leader ship when encountering conflicting information.

[0126] If the neighbor ship is out of communication range or the communication quality is too low, the observation value of the neighbor ship is 0.

[0127] There is no explicit message passing or active communication between the current ship and its neighboring ships. Whether to add the neighboring ship's information to the current ship's observation state vector is determined based on the physical distance and communication quality.

[0128] For example, assume that there are 4 ships in the multi-ship system, and the second neighbor ship sensed by the current ship is:

[0129]

[0130] If the neighbor ship is out of communication range, the above data values ​​will all be 0.0.

[0131] Neighbor ships are defined as other ships within the current ship's communication range and with a communication quality greater than 0.2. For each ship in the multi-ship system, the distance to all other ships is calculated. If the distance is less than or equal to the communication range and the communication quality is greater than 0.2, the ship is considered a neighbor and its information is detected. If the distance is outside the communication range or the communication quality is too low, the ship is considered a non-neighbor and its observation is set to 0.

[0132] A follower is a ship that is not a leader. In the observed state vector, a role field of 1.0 indicates a leader, and a role field of 0.0 indicates a follower.

[0133] The specific allocation of roles will be assigned when the multi-vessel system is initialized in step S1, and the leader ship will be configured and set iteratively during the process.

[0134] At each control time step, the individual policy network of each ship generates a control action based on the current observed state and executes the control action.

[0135] The i-th ship inputs the current state s i (t) to the individual policy network, output action:

[0136] a i (t)=π θi (s i (t)) Formula (1)

[0137] The ship adjusts its course and speed according to the action and enters the next state s i (t+1).

[0138] The intelligent agent of each ship outputs corresponding actions based on the current observation information through the individual strategy network.

[0139] Actions are usually in the form of vectors, e.g. Represent the adjustment amount of heading angle and velocity in the current control time step. After the control action is generated by the agent, it is passed to the environment for state transfer.

[0140] In action Indicates the change in heading angle in the current time step (usually in degrees). The environment updates the current heading of the ship based on this value:

[0141]

[0142] The result is normalized to the range of [-π,π] or [0,2π] to avoid heading angle overflow.

[0143] The Δv in the action represents the change in velocity. The environment adds this value to the current velocity to get the new velocity:

[0144] v {new} =clip(v {current} +Δv,v {min} ,v {max} ) Formula (3)

[0145] Among them, clip(·) means clipping within the maximum and minimum speed range.

[0146] Based on the updated heading and speed, the environment advances the ship's position according to the following formula:

[0147]

[0148] Among them, Δt is the control time step, x t ,y t is the current location coordinate.

[0149] Step S22: Calculate a corresponding reward tuple based on the current observation state and control action.

[0150] At each time control step, the individual policy network of each ship calculates the corresponding reward tuple based on the current state and control action;

[0151] The behavior effect of each ship at the current time step is evaluated, and the reward value is calculated according to the following three-layer structure:

[0152] The structure is as Figure 2 As shown in the figure, the core idea is to use basic rewards to guide ships to complete basic tasks such as target approach and safe navigation; to build a coordination mechanism between ships through collaborative rewards to ensure the stability of the formation and cooperation relationship; and to dynamically adjust the relative weights of various rewards through adaptive rewards, so that the strategy can cope with different sea conditions and mission scenarios, and realize multi-scale adaptive control of rewards.

[0153] The reward sequence includes a basic reward, a collaborative reward, and an adaptive reward, as well as a comprehensive reward calculated based on the basic reward, the collaborative reward, and the adaptive reward;

[0154] in,

[0155] The basic reward is used to guide each ship to approach the target point and sail safely;

[0156] The cooperation reward is used for cooperation among multiple ships and ensuring the formation of multiple ships;

[0157] The adaptive reward is used to dynamically adapt according to task progress and situational awareness.

[0158] The comprehensive rewards are as follows:

[0159]

[0160] Among them, r i (t), and are the comprehensive reward, basic reward, collaborative reward and adaptive reward of ship i at time step t; w b 、w c 、w aare the weights of basic reward, collaborative reward, and adaptive reward respectively.

[0161] Satisfaction: w b +w c +w a =1, for example, w b =0.4,w c =0.3,w a =0.3, and the comprehensive reward is used as the main optimization target for updating the ship individual strategy network.

[0162] Basic reward design ensures the navigation safety of each ship. To ensure that each ship in the multi-ship system has independent navigation capabilities and basic safety control behaviors, the present invention first constructs a basic reward function.

[0163] The design goals of basic rewards include: guiding individual ships to approach the mission target point, actively avoiding obstacles, and avoiding crossing the boundary (allowed navigation boundary), thereby providing a stable individual foundation for the upper-level collaborative rewards.

[0164] The basic reward as follows:

[0165]

[0166] in, are the target approach reward, obstacle avoidance reward and boundary penalty for ship i at time step t, respectively;

[0167] represents the basic reward of vessel i at time t. The basic reward term focuses on the effectiveness and local safety of individual navigation behavior, providing behavioral priors for the subsequent establishment of collaborative structures.

[0168] The position of ship i at the control time step t is p i (t), the target point position is q(t) (the target point position refers to the position of one or more target ships, or it can also refer to the position of one or more target sets), then the Euclidean distance between ship i and the nearest target point is as follows:

[0169] d i (t)=||p i (t)-q(t)|| Formula (8)

[0170] The target is close to the reward as follows:

[0171]

[0172] in, Indicates the maximum approach reward value, d i(t) is the Euclidean distance from ship i to the nearest target point, D tar is the reward decay distance scale;

[0173] When d i (t) = 0 Take the maximum, D tar Used to control the rate at which distance affects rewards.

[0174] Obstacles are environmental elements with fixed or dynamic positions, radius, speed, and type. Ships observe them as elements to be avoided and use them to calculate obstacle avoidance penalties and path adjustment behaviors. The total number of obstacles is O, and the position of the jth obstacle is o j , the Euclidean distance between ship i and the nearest j-th obstacle at time step t is controlled as follows:

[0175] d ij (t)=||p i (t)-o j || Formula (10)

[0176] The obstacle avoidance reward as follows:

[0177]

[0178] Where O is the number of obstacles in the obstacle set; P obs is the obstacle avoidance reward intensity, d ij (t) is the Euclidean distance from ship i to target point j, D obs Control the scope of obstacle avoidance rewards;

[0179] P obs >0, if the distance is close (within the warning range D obs If the obstacle avoidance penalty is imposed,

[0180] The boundary area where ships are allowed to navigate is defined as a two-dimensional map [0, B] 2 In order to avoid potential risks caused by ships approaching the edge, the edge safety width m>0 is set to limit ships from approaching the allowed navigation boundary. The effective navigation area is the safety interval after the boundary is reduced:

[0181] [m,Bm] 2 Formula (12)

[0182] If the ship's position p i (t), exceeds the region at a certain time step t, that is:

[0183]

[0184] A fixed boundary penalty is immediately imposed.

[0185] The boundary penalty as follows:

[0186]

[0187] Among them, P bnd is the penalty constant for exceeding the boundary; m is the edge safety width, B is the navigation boundary allowed, [m, Bm] 2 It is the safety interval after reducing the edge safety width.

[0188] P bnd >0 is the out-of-bounds penalty constant, which controls the intensity of negative feedback on the strategy caused by out-of-bounds behavior;

[0189] m is used to limit the ship's proximity to the edge of the map, and is usually a certain ratio of the map size B (such as B / 20); P bnd >0, for example, for a small-scale simulation scenario, P bnd The value is 1.0; for medium-scale simulation scenes, P bnd The value is 5.0; for large-scale simulation scenarios, P bnd The value is 20.0.

[0190] Collaboration Reward Design: Maintaining Multi-Ship Coordinated Formation. Collaboration rewards aim to guide individual ships in completing their mission while maintaining reasonable spatial relationships and formation structure, thereby achieving system-level coordinated control. Collaboration reward indicators are designed from three dimensions: formation maintenance, target encirclement, and leader-follower coordination mechanisms.

[0191] The collaboration reward as follows:

[0192]

[0193] Among them, Q form (t), Q enc (t), They are formation maintenance reward, encirclement structure reward and leader mechanism reward; form 、w enc 、w lead The weights of formation retention rewards, encirclement structure rewards, and leader-follower mechanism rewards, respectively;

[0194] Satisfy w form +w enc +w lead = 1. For example, w form =0.4,w enc =0.3,w lead =0.3.

[0195] Leader ships are directly rewarded for their mission progress, while subordinate ships are encouraged to behave in a manner consistent with and close to the leader, helping to form an effective organizational structure and maintain an orderly formation. This leader-follower mechanism encourages multi-ship systems to maintain a stable and orderly formation, achieving structural awareness and functional synergy.

[0196] Collaboration rewards enhance the structural perception and functional coordination capabilities of the multi-ship system during the collaborative execution of tasks by guiding all ships in the multi-ship system to maintain structural stability (formation maintenance), spatial coverage (target encirclement) and organizational responsiveness (leader-follower coordination).

[0197] In order to evaluate the degree of deviation of subordinate ships from the ideal formation structure in a multi-ship system, the present invention defines the following formation keeping reward index.

[0198] Assume that the actual position of the i-th slave ship at the control time step t is p i (t), and its corresponding ideal formation position is Then its offset is defined as:

[0199]

[0200] Sum up the offsets of all slave ships and define the total offset of the system at the current moment as:

[0201]

[0202] in, Represents a set of subordinate ships.

[0203] The formation maintains the reward Q from (t), as follows:

[0204]

[0205] Among them, D form (t) is the sum of the total deviations of all slave ships from their ideal positions at the current time step; is the number of slave ships in a multi-ship system, R f is the ideal formation deviation tolerance;

[0206] R f >0 indicates the maximum distance threshold that the slave ship is allowed to deviate from the ideal position, which is used to control the decay rate of the formation quality bonus.

[0207] For example, if the average deviation allowed for each ship does not exceed 3 meters, R can be set as follows: f =3.0.

[0208] The ideal position of the slave vessel is obtained as follows:

[0209] Get the leader ship position at the current control time step;

[0210] Based on the preset formation of the multi-ship system, the relative positions of the subordinate ships in the preset formation are obtained;

[0211] Based on the position of the leader ship at the current control time step and the relative position of the subordinate ships in the preset formation, the ideal position of the subordinate ships is generated in real time.

[0212] Ideal location for each subordinate vessel The topology of the formation is generated in real time based on the current position of the leader ship and the preset formation, as follows:

[0213]

[0214] Among them, x leader (t) is the position of the leader ship in the current control time step; is the relative reference position of the i-th subordinate ship in the preset formation. The preset formation can be linear, wedge-shaped, circular, or other structures (circular is selected in this invention). It is set during the initialization of the multi-vessel system and dynamically adjusted during operation based on the mission phase, target distribution, and sea conditions.

[0215] In order to measure the spatial encirclement degree of the multi-vessel system on the target, the present invention introduces the encirclement structure reward index, which combines the radius balance and angle uniformity for evaluation.

[0216] Assume that the polar coordinates of ship i and the nearest target are r i (t) represents the Euclidean distance between ship i and the target, i.e., the current encirclement radius; represents the azimuth of ship i relative to the target (angle range [0,2π)).

[0217] The average encirclement radius of all ships in a multi-ship system is:

[0218]

[0219] Where N is the number of ships participating in the encirclement of the target; To reflect the average distance from the multi-ship system to the target, the "encirclement tightness" is measured.

[0220] In order to measure the uniformity of the distribution of ships in the angular direction, the maximum angular spacing is defined as follows:

[0221]

[0222] in, represents the polar angle of the i-th ship after sorting; Indicates the polar angle of its neighboring ship; Indicates the spacing at the sparsest angles, reflecting the "envelope uniformity".

[0223] The enclosing structure reward Q enc (t), as follows:

[0224]

[0225] in, is the average encirclement radius of all ships in the multi-ship system to the target; R e To preset the ideal enclosing radius; is the maximum angular distance between all ships in the multi-ship system and the target; M is the number of ships participating in surrounding the target;

[0226] The exponential function form can ensure differentiability and adapt to the strategy optimization process. Approaching the ideal enclosing radius R e Indicates that the encirclement distance is moderate; The ideal angular distance 2π / N approaching uniform distribution indicates that the surrounding structure is densely balanced.

[0227] In order to enhance the coordination of the leadership-follower structure in a multi-ship formation, a leader-follower mechanism reward is designed to strengthen the cooperation of the subordinate ships with the leader ship in executing tasks.

[0228] Assume that the leader ship is numbered L, and its mission progress (such as approaching the target, mission completion) is the best in the team. The coordination index with the leader ship is

[0229] The following mechanism rewards as follows:

[0230]

[0231] Among them, Π i (t) Rate the leader ship’s current mission progress; The degree to which the subordinate ship maintains the formation structure of the leading ship.

[0232] Measures the degree to which the subordinate ship follows the leader ship's structure, which is composed of factors such as distance deviation and synchronization with the leader ship; It represents the degree to which the formation structure of the i-th slave ship is maintained relative to its ideal formation position at the control time step t. It is a normalized formation coordination index with a value range of [0,1].

[0233] when Approaching 1, it means that the subordinate ship i basically coincides with its ideal position, that is, its structural maintenance effect is good and consistent with the formation target; when If it is close to 0, it means the degree of deviation is large, and there may be phenomena such as formation destruction and structural incoordination. It is an important indicator to measure the degree of fit between individual ships and the ideal formation, and can explicitly guide ships to maintain local structural integrity.

[0234] Adaptive Rewards: In the complex and constantly changing maritime environment, relying solely on static rewards (basic rewards and collaborative rewards) is insufficient to ensure effective strategy generalization. The starting point for adaptive reward design is to transform task status and environmental characteristics into reward control factors, enabling flexible adjustment and real-time adaptation of strategies.

[0235] The adaptive reward as follows:

[0236]

[0237] in, They are progress feedback reward, situational awareness reward and potential field guidance reward respectively; w pong 、w sit 、w pot are the weights of progress feedback rewards, context awareness rewards, and potential field guided rewards respectively;

[0238] Satisfy w pong +w sit +w pot = 1. For example, w pong =0.4,w sit =0.3,w pot =0.3.

[0239] The progress feedback rewards are as follows:

[0240]

[0241] Among them, D p is the progress reward scale factor; N is the total number of ships in the multi-ship system;

[0242] D p Used to control the impact of distance on progress feedback rewards;

[0243] For example, if the mission target area is 100×100 meters and the target average radius is 5 meters, then D p =(100×100) / 5=20.0. The closer all ships in a multi-ship system get to the target point, the greater the progress feedback reward, strengthening the overall coordination of mission advancement.

[0244] Context-aware reward: It is used to dynamically identify the mission context of the ship and select the corresponding reward function accordingly, giving it environmentally adaptable behavioral preferences.

[0245] The context-aware rewards are as follows:

[0246]

[0247] Among them, D a 、D e They are the distance adjustment factor and the attenuation amplitude of the control reward in the target approach and obstacle avoidance scenarios respectively;

[0248] Context-aware rewards dynamically select an appropriate incentive model by distinguishing the current task stage:

[0249] If a ship is approaching a target, situational awareness encourages it to close quickly;

[0250] If a ship faces an obstacle threat, situational awareness encourages it to stay away from the danger zone;

[0251] If the mission currently emphasizes formation stability, provide positive feedback based on how well it matches the ideal position;

[0252] If in a neutral state or transition phase, situational awareness encouragement is set to 0 to avoid misleading training.

[0253] This design improves the strategy's ability to perceive situational changes and adaptively adjust execution priorities.

[0254] Potential field guidance reward: To further integrate multiple driving factors such as target attraction, obstacle avoidance and structural coordination, potential field guidance reward is designed.

[0255] The potential field guide reward is as follows:

[0256]

[0257] Among them, d iL (t) is the distance between ship i and its leading ship; D pt ,D po ,D pf are the distance adjustment factors of the attraction field, repulsion field, and structural potential field respectively; δ i is a binary indicator variable. When ship i is a subordinate ship, δ i =1, otherwise 0.

[0258] Potential field guided rewards integrate three key navigation behaviors:

[0259] Target attraction: encourages ships to approach mission targets and achieve overall advancement;

[0260] Obstacle repulsion: quickly imposes penalties when approaching obstacles to ensure obstacle avoidance safety;

[0261] Structural coordination: Follower vessels maintain an ideal spacing around the leader vessel, maintaining formation integrity and response consistency.

[0262] By modeling in exponential form, a smooth and differentiable reward gradient can be achieved, which helps to stabilize the training of the policy network.

[0263] Step S23: storing the current observation state, control action, reward tuple and next observation state as shared sample data into the experience buffer pool.

[0264] The state-action-reward sequence (s i (t),a i (t),R i (t),s i (t+1)) is stored in the experience buffer pool

[0265] R i (t) represents the three-layer structured reward set obtained by the i-th ship at the control time step t, which specifically includes three components: basic reward, collaborative reward and adaptive reward:

[0266]

[0267] r i (t) is a single scalar value used by the reinforcement learning algorithm to update the policy:

[0268] s i (t+1) is the next state of ship i; s i (t+1) represents the observed state of the i-th ship at the control time step t+1, and is obtained from s i (t) and control action a i (t) It can be deduced that:

[0269]

[0270] The specific update process includes:

[0271] (1) Heading update:

[0272]

[0273] (2) Speed ​​update:

[0274] v i (t+1)=clip(v i (t)+Δv i (t),v {min} ,v {max} ) Formula (30)

[0275] (3) Location update (2D longitude and latitude):

[0276]

[0277] (4) Environmental status observation update: update target distance, neighbor information, obstacle perception, etc.

[0278] Get s i (t+1) is obtained by simulating the state transfer function of the environment based on the action results and the dynamic model, and is used for state evolution in reinforcement learning.

[0279] The function of step S2 is that in each control time step, each ship obtains the current observation state and generates a control action, calculates the corresponding reward tuple based on the current observation state and control action, and then stores the relevant data in the experience buffer pool to provide sample data for subsequent shared policy network training.

[0280] Step S3, specifically.

[0281] The shared policy network is trained as follows:

[0282] Using the shared sample data, the shared policy network is trained using a proximal policy optimization (PPO) algorithm;

[0283] The AdamW optimizer is used to update the shared policy network parameters until the clipping objective function converges, obtaining the trained shared policy network and saving the network parameters.

[0284] All ships share a global policy network π θ , unified strategic behavior of multiple ships is achieved by sharing neural network parameters.

[0285] The clipping objective function is as follows:

[0286]

[0287] Among them, θ is the shared strategy network parameter; s t ,a t ,r t ,s t+1 are the observation state, control action, comprehensive reward and observation state of control time step t+1 respectively; E is the expectation operation, which expects to average the shared sample data corresponding to num time control steps taken out from the experience buffer pool D; L CLIP (θ), L VF (θ), L ENT (θ) are the policy loss, value function loss term and entropy regularization term respectively; λ1 is the value function loss weight, and λ2 is the entropy regularization coefficient.

[0288] For example, λ1 takes a value of 0.5 and λ2 takes a value of 0.01. In specific applications, these values ​​can be changed according to specific requirements.

[0289] The policy loss function is as follows:

[0290]

[0291] in, The strategy update ratio is the ratio of the current new strategy to the old strategy; is the advantage function, which measures the degree to which the current action is better than the average strategy; Update limit range, for example 0.1 to 0.3; clip(x,min,max) is used to limit the amplitude of policy updates to ensure training stability.

[0292] Among them, the advantage function is as follows:

[0293]

[0294] Among them, r t is the comprehensive reward, γ and λ are the discount factor and smoothing coefficient respectively; for example, γ is set to 0.99 to reduce the weight of future rewards; λ is set to 0.95; is the advantage estimate at the control time step t+1.

[0295] The value function loss term is as follows:

[0296]

[0297] in, is the value network parameter, is the value function network for state s t The predicted value, R t is the actual observed return value.

[0298] The entropy regularization term is as follows:

[0299]

[0300] Where σ is the standard deviation of the action distribution output by the shared policy network; e is a natural constant (approximately 2.71828); +1 is an adjustment term to ensure that the entropy value is non-negative;

[0301] To calculate all possible states s under strategy π t The average value of the entropy.

[0302] The core role of the shared strategy network is to improve the overall learning efficiency and collaborative consistency of the system. Its significance includes:

[0303] Parameter sharing: By unifying the policy function, all agents can perceive the environment and make decisions using the same mechanism, facilitating collective learning.

[0304] Sample sharing: The state-action-reward sequences of all ships can be used to train the same policy network, enhancing the diversity and coverage of training data;

[0305] Improved generalization: Since different ships are in different positions or situations, their shared policy network can obtain stronger state-action generalization ability during the learning process.

[0306] In this step, a trained shared policy network is obtained, and the trained shared policy network parameters are used to update the individual policy network parameters of each ship. The process returns to step S2 to continue execution. By continuously optimizing the individual policy network parameters of each ship, the optimal action for executing each ship is obtained, achieving the global optimality of the multi-ship system.

[0307] The purpose of step S3 is to use the shared sample data to train the shared policy network through the PPO algorithm and AdamW optimizer until the clipping objective function converges, so as to update and optimize the individual policy network parameters of each ship, realize the collaborative optimization and global consistency improvement of multi-ship policy parameters, and ultimately enhance the overall collaborative efficiency and generalization ability of the multi-ship system.

[0308] This paper proposes a collaborative control method based on a hierarchical reward mechanism for multi-ship systems. By constructing three types of progressive reward structures, namely basic, collaborative and adaptive, it achieves efficient learning and stable collaboration of multi-agent systems in a dynamic ocean environment.

[0309] Basic rewards are used to evaluate the basic navigation behaviors of individual ships, such as target approach, obstacle avoidance, and boundary maintenance, to ensure the system has the lowest level of feasibility and safety;

[0310] Collaboration rewards improve the structural stability and efficiency of multi-ship systems in collaborative missions by introducing indicators such as formation structure quality, target encirclement, and leader-follower relationships.

[0311] On this basis, adaptive rewards further integrate progress evolution, situational awareness and potential field guidance mechanisms to enhance the generalization and robustness of control strategies in complex environments.

[0312] Finally, by weighted fusion of rewards at each layer, this method constructs a scalable and controllable reinforcement learning incentive structure, providing systematic support for multi-vessel collaborative control.

[0313] Example 2:

[0314] Another embodiment of the present invention discloses a multi-vessel cooperative control system based on a hierarchical reward mechanism, thereby implementing the multi-vessel cooperative control method based on a hierarchical reward mechanism in Example 1. The specific implementation of each module refers to the corresponding description in Example 1.

[0315] like Figure 3 As shown, the system includes a system initialization module M1, a data collection and storage module M2 and a strategy network training and updating module M3.

[0316] System initialization module M1 is used to initialize the multi-ship system, where each ship holds an individual strategy network and a shared strategy network;

[0317] The data collection and storage module M2 is used to obtain the current observation state of each ship at each control time step, generate and execute control actions based on the current observation state through the individual policy network; calculate the corresponding reward sequence based on the current observation state and control action, and store the current observation state, control action, reward sequence and next observation state as shared sample data in the experience buffer pool;

[0318] The policy network training and updating module M3 is used to obtain shared sample data corresponding to num control time steps from the experience buffer pool after a preset num control time steps, to train the shared policy network, obtain the trained shared policy network parameters, and update the individual policy network parameters of each ship; then return to the data collection and storage module to continue execution.

[0319] In summary, the multi-vessel cooperative control method and system based on the hierarchical reward mechanism according to the embodiments of the present invention have the following beneficial effects:

[0320] 1. By constructing a hierarchical reward mechanism consisting of basic rewards, collaborative rewards, and adaptive rewards, this invention effectively guides individual ships to approach the target point and navigate safely (basic rewards). It also establishes a coordination mechanism among multiple ships to ensure formation (collaborative rewards), and dynamically adjusts rewards based on mission progress and environmental changes (adaptive rewards). Compared to the existing technology, which struggles to unify local decision-making and global coordination, this method organically combines local behavior optimization with global collaborative goals, improving the overall collaborative efficiency of the multi-ship system and avoiding problems such as formation breakdown or mission conflicts caused by individual selfish behavior.

[0321] 2. Compared with traditional reinforcement learning methods, which usually rely on a single, static reward function and have insufficient generalization capabilities, the present invention adopts the proximal policy optimization (PPO) algorithm to train the shared policy network and utilizes the shared sample data in the experience buffer pool during the training process, so that the strategy can adapt to the dynamic changes in the complex marine environment. By dynamically adjusting the reward mechanism, the strategy has stronger adaptability to environmental disturbances, changes in task stages, etc., avoiding the overfitting or non-steady-state behavior of the strategy caused by static incentives in the existing technology, improving the stability and generalization ability of the strategy in actual deployment, and enhancing the adaptability of the strategy.

[0322] 3. To address the inadequate stability and robustness of existing technologies in uncertain and dynamic ocean environments, this invention updates shared policy network parameters in real time and applies them to each ship's individual policy network. This ensures that the entire multi-vessel system can promptly adjust its control logic in the face of complex disturbances such as obstacle interference, communication delays, and packet loss. This continuous parameter update and optimization mechanism effectively enhances the system's robustness to emergencies and interference shocks, reduces the risk of collaborative mission interruption or severe performance degradation, and ensures the stable operation of the multi-vessel collaborative control system.

[0323] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0324] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed in the present invention should be covered by the scope of protection of the present invention.

Claims

1. A multi-vessel cooperative control method based on a hierarchical reward mechanism, characterized in that: The steps include: Step S1: Initialize the multi-ship system, where each ship holds an individual policy network and a shared policy network; Step S2: At each control time step, each ship obtains the current observation state, generates and executes a control action based on the current observation state through the individual policy network; calculates a corresponding reward tuple based on the current observation state and the control action, and stores the current observation state, control action, reward tuple, and next observation state as shared sample data in the experience buffer pool; Step S3: After num control time steps are preset, shared sample data corresponding to num control time steps are obtained from the experience buffer pool to train the shared policy network, and the trained shared policy network parameters are obtained to update the individual policy network parameters of each ship; Return to step S2.

2. The method according to claim 1, characterized in that The reward tuple includes a basic reward, a collaborative reward, and an adaptive reward, as well as a comprehensive reward calculated based on the basic reward, the collaborative reward, and the adaptive reward; in, The basic reward is used to guide each ship to approach the target point and sail safely; The cooperation reward is used for cooperation among multiple ships and ensuring the formation of multiple ships; The adaptive reward is used to dynamically adapt according to task progress and situational awareness.

3. The method according to claim 2, characterized in that The comprehensive rewards are as follows: Among them, r i (t), and are the comprehensive reward, basic reward, collaborative reward and adaptive reward of ship i at time step t; w b 、w c 、w a are the weights of basic reward, collaborative reward, and adaptive reward respectively.

4. The method according to claim 1, characterized in that The shared policy network is trained as follows: Using the shared sample data, the shared policy network is trained using a proximal policy optimization (PPO) algorithm; The AdamW optimizer is used to update the shared policy network parameters until the clipping objective function converges, obtaining the trained shared policy network and saving the network parameters.

5. The method according to claim 3, characterized in that: The basic reward as follows: in, are the target approach reward, obstacle avoidance reward and boundary penalty for ship i at time step t, respectively; The target is close to the reward as follows: in, Indicates the maximum approach reward value, d i (t) is the Euclidean distance from ship i to the nearest target point, D tar is the reward decay distance scale; The obstacle avoidance reward as follows: Where O is the number of obstacles in the obstacle set; P obs is the obstacle avoidance reward intensity, d ij (t) is the Euclidean distance from ship i to target point j, D obs Control the scope of obstacle avoidance rewards; The boundary penalty as follows: Among them, P bnd is the penalty constant for exceeding the boundary; m is the edge safety width, B is the navigation boundary allowed, [m, Bm] 2 It is the safety interval after reducing the edge safety width.

6. The method according to claim 3, characterized in that: The collaboration reward as follows: Among them, Q form (t), Q enc (t), They are formation maintenance reward, encirclement structure reward and leader mechanism reward; form 、w enc 、w lead The weights of formation retention rewards, encirclement structure rewards, and leader-follower mechanism rewards, respectively; The formation maintains the reward Q from (t), as follows: Among them, D form (t) is the sum of the total deviations of all slave ships from their ideal positions at the current time step; is the number of slave ships in a multi-ship system, R f is the ideal formation deviation tolerance; The enclosing structure reward Q enc (t), as follows: in, is the average encirclement radius of all ships in the multi-ship system to the target; R e To preset the ideal enclosing radius; is the maximum angular distance between all ships in the multi-ship system and the target; M is the number of ships participating in surrounding the target; The following mechanism rewards as follows: Among them, Π i (t) Rate the leader ship’s current mission progress; The degree to which the subordinate ship maintains the formation structure of the leading ship.

7. The method according to claim 3, characterized in that The adaptive reward as follows: in, They are progress feedback reward, situational awareness reward and potential field guidance reward respectively; w pong 、w sit 、w pot are the weights of progress feedback rewards, context awareness rewards, and potential field guided rewards respectively; The progress feedback rewards are as follows: Among them, D p is the progress reward scale factor; N is the total number of ships in the multi-ship system; The context-aware rewards are as follows: Among them, D a 、D e They are the distance adjustment factor and the attenuation amplitude of the control reward in the target approach and obstacle avoidance scenarios respectively; The potential field guide reward is as follows: Among them, d iL (t) is the distance between ship i and its leading ship; D pt ,D po ,D pf are the distance adjustment factors of the attraction field, repulsion field, and structural potential field respectively; δ i is a binary indicator variable. When ship i is a subordinate ship, δ i =1, otherwise 0.

8. The method according to claim 6, characterized in that The ideal position of the slave vessel is obtained as follows: Get the leader ship position at the current control time step; Based on the preset formation of the multi-ship system, the relative positions of the subordinate ships in the preset formation are obtained; Based on the position of the leader ship at the current control time step and the relative position of the subordinate ships in the preset formation, the ideal position of the subordinate ships is generated in real time.

9. The method according to any one of claims 1 to 8, characterized in that The initialization of the multi-vessel system includes setting the mission target area, target point set, obstacle set, preset formation and allowed navigation boundary.

10. A multi-vessel cooperative control system based on a hierarchical reward mechanism, characterized in that: include: System initialization module, used to initialize the multi-ship system, each ship holds an individual strategy network and a shared strategy network; A data collection and storage module is configured to obtain the current observation state of each ship at each control time step, generate and execute control actions based on the current observation state through the individual policy network, calculate a corresponding reward sequence based on the current observation state and control action, and store the current observation state, control action, reward sequence, and next observation state as shared sample data in an experience buffer pool; The policy network training and updating module is used to obtain shared sample data corresponding to num control time steps from the experience buffer pool after a preset num control time steps, to train the shared policy network, obtain the trained shared policy network parameters, and update the individual policy network parameters of each ship; then return to the data collection and storage module to continue execution.

Images