An unmanned surface vehicle cooperative planning method based on hierarchical planning and reinforcement learning
By decoupling the multi-agent collaborative planning problem into high-level task planning and low-level path planning, and combining genetic algorithms and reinforcement learning, the contradiction between computational speed and path quality in traditional methods is resolved, enabling unmanned surface vessels to operate efficiently and stably in complex environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SOUTHEAST UNIV
- Filing Date
- 2026-03-02
- Publication Date
- 2026-06-05
AI Technical Summary
Existing multi-agent cooperative planning methods have an inherent contradiction between computational speed and path quality, making it difficult to achieve efficient and stable cooperative operations in complex environments.
The multi-agent collaborative planning problem is decoupled into high-level global task planning and low-level local path planning. A genetic algorithm is used to solve the multi-traveling salesman problem for task allocation, and a collision-free and smooth path is generated through a reinforcement learning model. The hierarchical planning architecture is used to reduce the dimensionality of the state space and the training difficulty.
It achieves efficient and stable path planning in complex environments, reduces planning time from seconds to milliseconds, and improves the system's real-time response capability and scenario generalization capability.
Smart Images
Figure CN122149462A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and robotics, and in particular to a collaborative planning method for unmanned surface vessels based on hierarchical planning and reinforcement learning. Background Technology
[0002] With the rapid development of Industry 4.0 and IoT technologies, application scenarios such as automated warehousing, smart factories, unmanned surface vessel (USV) swarms, and automated ports are becoming increasingly widespread. In these scenarios, multi-agent systems (MAS) composed of numerous autonomous mobile robots or unmanned devices have become the core force for performing large-scale, repetitive tasks. Therefore, the efficient collaborative operation capability of MAS is a key factor determining the throughput, response speed, and operating costs of the entire production or logistics system. In multi-agent collaborative planning tasks, task planning and path navigation are the two most important parts, constituting the most fundamental technical challenges. This challenge typically includes two closely coupled sub-problems: first, the task planning problem, which determines which USV performs which task and the order of task execution to achieve optimality, such as minimizing total time or energy consumption; second, the path planning problem, which plans a collision-free and high-quality path for the USV from its current position to its target position. Therefore, it is necessary to design reasonable task planning and path planning algorithms to improve the operational efficiency, stability, and safety of the entire swarm system.
[0003] Among existing multi-agent cooperative planning methods, the paper [Sharon G, Stern R, Felner A, et al. "Conflict-based search for optimal multi-agent pathfinding. Artificial Intelligence." 2015, 219:40-66] proposes a conflict-based search (CBS) algorithm, which solves the curse of dimensionality problem of the traditional coupled A* algorithm by using the idea of two-layer decoupling, thereby finding a set of collision-free paths for multiple agents and minimizing the total cost. The paper [Camponogara E, Jia D, Krogh B, et al. "Distributed model predictive control." IEEE control systemsmagazine 2002, 22.1: 44-52.] proposes a distributed MPC method based on online optimization and a solid theoretical foundation, providing a concrete method for dynamic and constrained cooperative planning. The paper [Rashid T, Samvelyan M, Witt C, et al. "QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning." arXiv preprint arXiv:1803.11485, 2018.] provides a model-free and implicit algorithm for multi-agent cooperative planning, employing a centralized training, decentralized execution (CTDE) approach, and performing multi-agent cooperative planning through value function decomposition and monotonicity constraints. The paper [Yu C, Velu A, Vinitsky E, et al. "The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games." arXiv preprint arXiv: 2103.01955, 2021.] systematically evaluates the performance of the Proximal Policy Optimization (PPO) algorithm in cooperative multi-agent environments, challenging the common perception that "policy methods are inefficient in sample processing." Research shows that by adopting a centralized training and decentralized execution architecture, combined with meticulous implementation tuning, the simple MAPPO achieves final performance and sample efficiency comparable to or even surpasses more complex off-policy algorithms in several challenging benchmark tests.The paper [Sartoretti G, Kerr J, Shi Y, et al. "PRIMAL: Pathfinding via Reinforcement and Imitation Multi-Agent Learning." arXiv preprint arXiv:1809.03531, 2018.] proposes a two-stage training framework that integrates imitation learning and reinforcement learning, specifically designed to solve large-scale multi-agent path planning problems. In the first stage, the agent quickly learns basic pathfinding capabilities from a traditional suboptimal planning expert (such as priority planning) through imitation learning. In the second stage, reinforcement learning is used to fine-tune the policy, addressing inherent flaws in expert policies (such as deadlock) through direct interaction with the environment, and further optimizing the global performance of the path. This framework significantly improves the sample efficiency and final performance of pure reinforcement learning methods on planning problems. Summary of the Invention
[0004] Purpose of the invention: This invention provides a collaborative planning method for unmanned surface vessels based on hierarchical planning and reinforcement learning. By combining the deterministic optimization of high-level global task planning with the learning optimization capability of local path generation, it provides a new technical paradigm for the efficient, stable and safe operation of multi-agent systems in complex static environments.
[0005] Technical solution: The unmanned surface vessel cooperative planning method based on hierarchical planning and reinforcement learning described in this invention includes the following steps:
[0006] Step 1: Formulate a multi-unmanned surface vessel (USV) collaborative planning problem and decouple the collaborative planning problem into high-level global task planning and low-level local path planning;
[0007] Step 2: Model the high-level global mission planning problem as a multi-traveling salesman problem and solve it to obtain the near-optimal mission execution sequence for each unmanned surface vessel;
[0008] Step 3: Model the underlying local path planning problem as a reinforcement learning problem, and design the state space, action space and reward function to guide the unmanned surface vessel to learn the optimal path;
[0009] Step 4: Train the reinforcement learning path planning model offline to enable it to generate the optimal path in complex obstacle environments.
[0010] Step 5: Online path generation and navigation. The unmanned surface vessel loads a pre-trained reinforcement learning model to generate and execute the optimal path to the task point in real time.
[0011] Furthermore, in step 1, the multi-unmanned surface vessel system is composed of... It consists of 10 unmanned surface vessels, numbered as follows: The unmanned surface vessel's (USV) movement area is designed with sides of length [missing information]. A square region; the unmanned surface vessel at the initial moment Located at the starting point Location; target point set Depend on It consists of 1 point, denoted as . .
[0012] Furthermore, in step 1, the multi-unmanned surface vessel (USV) collaborative planning problem is decoupled into high-level global task planning and low-level local path planning. The goal is to generate a safe path for each USV, starting from its origin, visiting a series of assigned target points, and finally returning to the origin, while satisfying the constraint that all target points are visited, so as to minimize the sum of the total path lengths traversed by all USVs to complete the task.
[0013] Furthermore, the high-level global task planning is responsible for solving the decision-making problem of task allocation and execution order. Its goal is to assign one or more target points that each UAV needs to visit in this collaborative task based on global target point information and the initial state of each UAV, and to determine a unique task execution sequence for the target points assigned to the UAV. The low-level local path planning takes the task execution sequence output by the high-level global task planning as input instructions and is responsible for solving the path generation problem of a single UAV. Its goal is to independently plan and generate a collision-free safe path for each UAV in a preset operating environment, starting from its current position or starting point, which can sequentially visit all the specified target points in its task execution sequence, and avoid all known obstacles in the process.
[0014] Furthermore, in step 2, the high-level global mission planning problem is modeled as a multi-traveling salesman problem, and the near-optimal mission execution sequence for each unmanned surface vessel is obtained by solving it. Specifically, this includes the following steps:
[0015] Step 21: The multi-unmanned surface vessel (USV) high-level global mission planning problem is modeled as an integer programming paradigm of the multi-traveling salesman problem, specifically defined as:
[0016]
[0017] Among them, decision variables This is a binary variable; a value of 1 indicates an unmanned surface vessel. From the goal Head directly to the target point ; Indicates from the target To the target The transfer cost is a pre-calculated core scalar quantity; transfer cost Defined as an unmanned surface vessel (USV) overcoming water resistance and wind / wave interference on the water surface, from the work point sail to the work site The required estimated range; the objective function of the model aims to minimize the total cost of all paths taken by the unmanned surface vessels; to achieve this objective, the model includes four core constraints: the first constraint is the target point access constraint, which ensures that every target in the system... The first constraint ensures that both the starting point and the task point are entered exactly once, thus guaranteeing the completion of all tasks and the closure of the path; the second constraint is a flow conservation constraint, ensuring that any unmanned surface vessel... Entering and leaving any target The number of times each step is taken is equal, thus ensuring the continuity of the path; the third constraint is the starting point and return constraint, ensuring that the unmanned surface vessel... It must and can only be from its starting point (goal) The first constraint is the sub-loop elimination constraint. By prohibiting the formation of any sub-loop that does not intersect with the set of starting nodes, it forces all assigned tasks to be connected on a single and complete path starting from the starting point. Physically, this constraint forces each unmanned surface vessel to complete all its assigned water monitoring tasks and then return to the only shore-based recovery point, preventing unmanned surface vessels from forming isolated navigation paths that cannot be returned in open water.
[0018] Step 22: Solve the Multiple Traveling Salesman Problem constructed in Step 21 using a genetic algorithm. The execution process begins with encoding the solution space. First, each potential task allocation and execution sequence scheme is represented as an independent chromosome, and a random population containing multiple chromosomes is initialized based on this encoding method. Then, through iterative execution of fitness evaluation and core genetic operators such as selection, crossover, and mutation, the population is continuously evolved to generate a population of offspring with better fitness. Finally, when the algorithm reaches the preset termination condition (e.g., reaching the maximum number of iterations or the quality of the solution meets the convergence threshold), the iteration process terminates. The algorithm selects the individual with the highest fitness in the current population for decoding, and the decoding result is the final determined task allocation scheme and execution sequence for each unmanned surface vessel.
[0019] Step 23: In the chromosome encoding and decoding stage, an encoding scheme consisting of a task chromosome and a separator chromosome is adopted; the task chromosome is... A complete permutation of the task point objectives; the dividing chromosome is a series of... The first is an array; the second is the separator chromosome, which is a group of... An array of ascending integers is used, where each integer represents a cut point. During decoding, the integers on the separator chromosome are used as cut positions on the task chromosome to determine the task point number and order performed by each unmanned surface vessel. In the initial population generation stage, a batch of initial chromosomes is constructed by randomly generating all permutations of the task chromosomes and the cut point positions in the separator chromosomes.
[0020] The fitness function is calculated through the following steps: First, the chromosome is decoded to restore it to a specific task planning scheme; then, its total cost is calculated based on the scheme; finally, the reciprocal of the cost value is taken as the fitness of the chromosome. The specific calculation formula is as follows:
[0021]
[0022] In the design of genetic operators, the selection operator adopts the roulette wheel selection method, specifically: first, the total fitness of all individuals in the population is calculated, and then the proportion of each individual's fitness to the total is used as its probability of being selected; the crossover operator is executed independently for the two chromosomes: for the task chromosome, the sequential crossover operator (OX) is used, that is, the offspring inherits the continuous gene segments of the first parent and fills the remaining genes in the order of the second parent; for the split chromosome, the single-point or multi-point crossover operator is used, exchanging some genes of the parent after the crossover point, and after the exchange, the offspring sequence is reordered to ensure that it is in ascending order; the mutation operator is also executed independently for the two chromosomes and is applied to the newly generated offspring with a preset low probability: for the task chromosome, one of the three strategies of exchange mutation, insertion mutation and inversion mutation is selected; for the split chromosome, random reset mutation is used, that is, a valid split position sequence is completely regenerated with a certain probability.
[0023] Furthermore, in step 3, the underlying local path planning problem is modeled as a reinforcement learning problem, and the state space, action space, and reward function are designed to guide the unmanned surface vessel to learn the optimal path. Specifically, this includes the following steps:
[0024] Step 31: The state input of the unmanned surface vessel (USV) is designed as a vector, which includes its own motion information, mission target information, and environmental perception information. The own motion information includes the USV's current dynamic state and key physical quantities, specifically the USV's surge velocity, sway velocity, and yaw rate in the hull coordinate system, to ensure that the planned trajectory generated by the model conforms to physical constraints and has smoothness. The mission target information includes the relative position and attitude of the next navigation target point in the USV's own coordinate system. The environmental perception information is a low-dimensional vector, generated by a preprocessing module after feature extraction and dimensionality reduction of the raw environmental data acquired by the sensors, and includes information on the distance and distribution of obstacles in different directions.
[0025] Step 32: To generate a smooth path that conforms to dynamic characteristics, the action space is defined as the coordinates of control points of a set of Bézier curves. The path curve generated by these control points must satisfy the minimum turning radius constraint of the unmanned surface vessel (USV). The generated trajectory will be used as a reference input, converted into the speed command of the USV's propeller and the rudder angle command of the servo motor. Specifically, the action is defined as follows:
[0026] ,
[0027] in For the first The coordinates of the control points , The number of control points in a single planning iteration;
[0028] Step 33: To guide the reinforcement learning model to autonomously learn a path planning strategy that satisfies multiple optimization objectives, a composite reward function was designed. This function is calculated at each time step of the unmanned surface vessel's interaction with the environment. By providing positive incentives (rewards) or negative feedback (penalties), the "behavioral preferences" of the unmanned surface vessel are shaped. The definition is as follows:
[0029] ,
[0030] in For the corresponding reward weight, This is a safety reward function used to ensure a collision-free path. It calculates the shortest distance from an obstacle to the path curve and compares it with a safety distance threshold. If the distance is less than the threshold, a penalty is applied; otherwise, the value is zero. As a reward for path smoothness, the curvature of the path curve at that point is calculated by taking points along the path and compared with a tiered threshold to impose a penalty, thereby suppressing the frequent large rudder angle switching of the unmanned surface vessel in high sea states, preventing the hull from rolling violently or increasing unnecessary mechanical wear of the rudder. For the velocity stability function, large-amplitude acceleration is suppressed by penalizing the norm of the rate of change of velocity; The endpoint function determines the reward based on whether the unmanned surface vessel (USV) reaches the endpoint.
[0031] Furthermore, in step 4, offline training of the reinforcement learning path planning model to enable it to generate optimal paths in complex obstacle environments specifically includes the following steps:
[0032] Step 41: Initialize network parameters and algorithm hyperparameters, and initialize the policy network (Actor). parameters Initialize two independent Q-value networks (Critic). and parameters , Initialize two sets of corresponding target Q-value networks. and So that its parameters are respectively with , Same, that is , Initialize an empty experience replay pool. Initialize learnable temperature parameters ;
[0033] Step 42: The unmanned surface vessel interacts with the environment and stores its experience. Within each time step, the unmanned surface vessel (USV) at each time step Based on the current state Through policy network Sampling to obtain action Rewards are given after interacting with the environment. and the next state Each state transition sample Store in the experience replay pool middle;
[0034] Step 43, from the experience replay pool A batch of random samples For each sample, Through the current policy network According to the next state Sampling to obtain the next action And calculate its log probability. Subsequently, the minimum Q-value of the next state is calculated using two sets of target Q-value networks, and the policy log probability weighted by the entropy term is subtracted to obtain the softened state value. Finally, the target Q-value is calculated. as follows:
[0035]
[0036] in, Discount factor;
[0037] Step 44: Update the parameters of the two sets of Critic networks by minimizing the Q-value network output relative to the target Q-value. The mean squared error (MSE) between the two sets of Critic networks is used to update the parameters. and Its loss function Defined as:
[0038] .
[0039] By minimizing this loss function, the value network can accurately predict the success rate and expected energy consumption of unmanned surface vessels under the current water flow interference and obstacle distribution.
[0040] Step 45: Update the Actor network and temperature parameters by maximizing the expected reward and entropy of the policy. Its loss function Defined as:
[0041] .
[0042] Simultaneously update temperature parameters It enables it to automatically balance reward and entropy, and its loss function Defined as:
[0043] ,
[0044] in, The physical meaning of introducing the entropy term as the preset target entropy is to encourage unmanned surface vessels to maintain a certain degree of exploratoryness in unknown water environments, and to avoid the strategy from converging too early in local water areas, which would result in the inability to bypass large-scale obstacle areas.
[0045] Step 46: Soft update the target Q-value network at a small scale. The parameters of the two target Q-value networks are softly updated using the Polyak averaging method to stabilize the training process.
[0046] .
[0047] Step 47: Repeat steps 42 to 46 until training is complete and a strategy that can be used for collaborative path planning and task execution is obtained.
[0048] Furthermore, in step 5, online path generation and navigation are performed. The unmanned surface vessel (USV) loads a pre-trained reinforcement learning model to generate and execute the optimal path to the task point in real time. Based on the task sequence generated in step 2, the USV sets the first task point in the sequence as the current navigation target. Subsequently, the USV perceives its local environmental information and its own motion state in real time during operation, and combines this information with the current navigation target to form a state vector, which is then input into the loaded reinforcement learning model. The model outputs the optimal curve trajectory in real time based on the current state, and the USV performs trajectory tracking to drive its own motion. This closed-loop process of "perception-decision-execution" is repeated at a high frequency until the USV reaches the current task point. After reaching the target, the USV updates the next task point in the task sequence as the new navigation target and repeats the above navigation process until all its assigned tasks are completed.
[0049] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: (1) The present invention decouples the multi-agent collaborative planning problem into high-level global task planning and low-level local path planning. By combining global combinatorial optimization with local learning optimization, it effectively solves the inherent contradiction between computational speed and path quality in traditional planning algorithms, and provides a new technical paradigm for the efficient and stable operation of multi-unmanned surface vessel systems in complex environments; (2) The present invention introduces a pre-trained reinforcement learning model in the low-level path planning. This model transforms the complex path search problem into a fast neural network forward inference through offline learning. Under the premise of ensuring that the path is collision-free, smooth and safe, the planning time is reduced from seconds to milliseconds, which greatly improves the real-time response capability of the system; (3) The present invention trains the low-level reinforcement learning model in a diverse procedural generation environment, which enables the model to have a strong scene generalization capability. The model can adapt to complex static obstacle layouts that have never been seen before without reprogramming or parameter adjustment for new environments. This significantly reduces the deployment cost and maintenance difficulty of the system and improves its robustness and applicability in variable scenarios. Attached Figure Description
[0050] Figure 1 This is a schematic diagram of the method flow of the present invention.
[0051] Figure 2 This is a schematic diagram illustrating the specific process of the method of the present invention.
[0052] Figure 3 This is a schematic diagram of the collaborative path planning of multiple unmanned surface vessels in a complex static obstacle environment according to the present invention. Detailed Implementation
[0053] like Figure 1 and Figure 2As shown, a collaborative planning method for unmanned surface vessels based on hierarchical planning and reinforcement learning includes the following steps:
[0054] Step 1: Formulate a multi-unmanned surface vessel (USV) collaborative planning problem and decouple the collaborative planning problem into high-level global task planning and low-level local path planning;
[0055] Step 2: Model the high-level global mission planning problem as a multi-traveling salesman problem and solve it to obtain the near-optimal mission execution sequence for each unmanned surface vessel;
[0056] Step 3: Model the underlying local path planning problem as a reinforcement learning problem, and design the state space, action space and reward function to guide the unmanned surface vessel to learn the optimal path;
[0057] Step 4: Train the reinforcement learning path planning model offline to enable it to generate the optimal path in complex obstacle environments.
[0058] Step 5: Online path generation and navigation. The unmanned surface vessel loads a pre-trained reinforcement learning model to generate and execute the optimal path to the task point in real time.
[0059] In step 1, the multi-unmanned surface vessel system is composed of It consists of 10 unmanned surface vessels, numbered as follows: The unmanned surface vessel's (USV) movement area is designed with sides of length [missing information]. A square region; the unmanned surface vessel at the initial moment Located at the starting point Location; target point set Depend on It consists of 1 point, denoted as . .
[0060] In step 1, the multi-unmanned surface vessel (USV) collaborative planning problem is decoupled into high-level global task planning and low-level local path planning. The goal is to generate a safe path for each USV that starts from its starting point, visits a series of assigned target points, and finally returns to the starting point, while satisfying the constraint that all target points are visited, so as to minimize the sum of the total path lengths traversed by all USVs to complete the task.
[0061] The decoupling operation in step 1 is a direct solution to the challenges faced by pure reinforcement learning methods. The challenge lies in the fact that when global task information and local path planning are coupled to a single reinforcement learning model, a high-dimensional state space is formed, leading to difficulty in training convergence and weak generalization ability. To overcome this problem, this invention introduces a hierarchical planning architecture. Its core is that the higher level handles the variable global task planning, while the lower level focuses on learning task-independent local navigation capabilities. The technical effects of this approach are twofold: firstly, it significantly reduces the dimensionality of the state space of the lower-level reinforcement learning, simplifying training; secondly, it creates a universal lower-level navigation module, eliminating the need for repeated training when the higher-level task changes, fundamentally ensuring the reusability and adaptability of the system.
[0062] In step 2, the high-level global mission planning problem is modeled as a multi-traveling salesman problem, and the near-optimal mission execution sequence for each unmanned surface vessel is obtained by solving it. Specifically, the steps include:
[0063] Step 21: The multi-unmanned surface vessel (USV) high-level global mission planning problem is modeled as an integer programming paradigm of the multi-traveling salesman problem, specifically defined as:
[0064]
[0065] Among them, decision variables This is a binary variable; a value of 1 indicates an unmanned surface vessel. From the goal Head directly to the target point ; Indicates from the target To the target The transfer cost is a pre-calculated core scalar; specifically, the transfer cost... Defined as an unmanned surface vessel (USV) overcoming water resistance and wind / wave interference on the water surface, from the work point sail to the work site The required estimated range; the objective function of the model aims to minimize the total cost of all paths taken by the unmanned surface vessels; to achieve this objective, the model includes four core constraints: the first constraint is the target point access constraint, which ensures that every target in the system... The first constraint (including the starting point and the mission point) ensures that each unmanned surface vessel (USV) is entered exactly once, guaranteeing the completion of all tasks and the closure of the path. Physically, this constraint forces each USV to complete all its assigned water monitoring tasks and ultimately return to the unique shore-based recovery point, preventing USVs from forming isolated, unreturnable navigation paths in open water. The second constraint is a flow conservation constraint, ensuring that any USV... Entering and leaving any target The number of times is equal, thus ensuring the continuity of the path. The third constraint is the starting point and return constraint, ensuring the unmanned surface vessel... It must and can only be from its starting point (goal) The first constraint is the sub-loop elimination constraint, which forces all assigned tasks to be connected to a single and complete path starting from the starting point by prohibiting the formation of any sub-loop that does not intersect with the set of starting nodes.
[0066] The multi-unmanned surface vessel (USV) task allocation model constructed in steps 22 and 21 is essentially a Multi-Traveling Salesman Problem (Multi-TSP), which is mathematically an NP-hard combinatorial optimization problem. The technical challenge lies in the fact that the computational complexity of solving this problem increases exponentially with the number of task points and USVs. Therefore, for large-scale real-world applications requiring rapid response, precise algorithms (such as branch and bound methods) that aim to obtain the global optimum in polynomial time are infeasible due to their excessive computational time and typically cannot meet the real-time requirements of the system.
[0067] To address the aforementioned technical problems, this invention preferably employs a Genetic Algorithm (GA) as the solution method for high-level global task planning in step 2. As a mature metaheuristic algorithm, the Genetic Algorithm is particularly suitable for solving large-scale combinatorial optimization problems, and its technical advantages are reflected in the following aspects:
[0068] First, it meets real-time requirements: Genetic algorithms are anytime algorithms, meaning they can continuously iterate and optimize within a preset, fixed computation time, and output the best solution found at any time. This characteristic ensures that even if the global optimum is not reached, a high-quality approximate solution can be provided within the specified response time, thus meeting the stringent timeliness requirements of multi-unmanned surface vessel systems for planning.
[0069] Second, global search capability and robustness: By simulating genetic mechanisms such as natural selection, crossover, and mutation, genetic algorithms can perform global searches in a vast solution space, effectively avoiding getting trapped in local optima and exhibiting strong robustness. Through careful design of core operators such as crossover and mutation, genetic algorithms can achieve a good balance between computational efficiency and solution quality, quickly converging to a satisfactory solution that meets the needs of most industrial applications.
[0070] Step 23: The genetic algorithm design for high-level global task planning is shown below:
[0071] Chromosome Encoding and Decoding: This invention employs an encoding method that separates task arrangement from separation position. Specifically, a solution consists of two independent chromosomes: the first is the task chromosome, which... The number of each task point target Perform a full permutation to form a sequence of length... The first is an array; the second is the separator chromosome, which is a group of... An array of ascending integers, where each integer represents a cut-off point position, forms the complete chromosome of the problem. During decoding, the integers on the separating chromosome are used as cut-off point positions on the task chromosome, thereby determining the task point numbers and order to be executed by the unmanned surface vessel.
[0072] Initial population generation: The starting point of this algorithm is to randomly generate a large number of such chromosome pairs. Both the full permutation of the task chromosomes and the cutting point positions in the dividing chromosomes are completely random. This ensures that the search space has an extremely broad and unbiased range in both the global order of tasks and the task division among unmanned surface vessels.
[0073] Fitness function: To evaluate the performance of each chromosome in the population using a quantitative metric, the chromosome is first decoded to reconstruct a task planning scheme. Then, the cost is calculated according to this scheme, and its reciprocal is taken to obtain the fitness function.
[0074]
[0075] Selection, crossover, and mutation operators: The selection operator employs a roulette wheel selection method. Specifically, this method involves first calculating the sum of the fitness of all individuals in the population, and then setting the probability of each individual being selected as the proportion of its fitness value within that sum. This selection mechanism ensures, on the one hand, that individuals with higher fitness have a higher probability of being selected, thus guiding the population towards a better solution; on the other hand, its inherent probabilistic nature also provides survival opportunities for individuals with lower fitness, helping to maintain the genetic diversity of the population and effectively preventing premature convergence due to excessive selection pressure.
[0076] The crossover operators are performed independently for the task chromosome and the delimiter chromosome, matching the split encoding scheme adopted in this invention. For the task chromosome, a sequential crossover operator specifically designed for sequence problems is used. This operator ensures the effectiveness of the task arrangement of the offspring by inheriting continuous gene segments from one parent and filling the remaining positions according to the gene sequence of another parent, while preserving the excellent local sequence features of the parents. For the delimiter chromosome, a single-point or multi-point crossover operator is used. Offspring are generated by exchanging gene segments between the crossover points of the parents, and the new sequences are reordered after the operation to ensure the validity of their ascending numerical order. Through this split crossover strategy, the algorithm can explore the solution space in parallel and efficiently on two dimensions: "task execution order" and "task grouping scheme".
[0077] To prevent premature convergence due to loss of genetic diversity, a mutation operation is applied to offspring individuals with a predetermined low probability after crossover. This operation is performed independently on both chromosomes. For the task chromosome, one or more strategies can be selected from exchange mutation (randomly swapping the positions of two task points), insertion mutation (randomly selecting a task point and inserting it into a new position), or inversion mutation (randomly reversing the order of a subsequence) to introduce new combinations of task orders. For the partition chromosome, a random reset mutation strategy is used, which replaces the entire chromosome with a completely new, randomly generated, valid sequence with a certain probability. This aims to explore new possibilities for task partitioning by introducing significant perturbation, thereby enhancing the algorithm's ability to escape local optima.
[0078] In step 3, the underlying local path planning problem is modeled as a reinforcement learning problem, and the state space, action space, and reward function are designed to guide the unmanned surface vessel to learn the optimal path. Specifically, this includes the following steps:
[0079] In steps 31 and 32, to endow the unmanned surface vessel (USV) with sufficient perception of the local environment and mission objectives, its state input is designed as a state vector containing its own motion information, mission objective information, and environmental perception information. The self-motion information includes the USV's current dynamic state and key physical quantities, specifically the USV's surge, sway, and yaw rate in the hull coordinate system, ensuring that the planned trajectory generated by the model conforms to physical constraints and possesses smoothness. The mission objective information is defined in the USV's own coordinate system, specifically including the relative position and attitude of the next navigation point. This design aims to ensure that the learned local navigation strategy does not depend on the USV's global absolute position, thereby enhancing the model's generalization ability. The environmental perception information is a low-dimensional feature vector, generated by preprocessing the sensor raw data through feature extraction and dimensionality reduction. This vector compactly represents the key distribution characteristics of surrounding obstacles, thus providing the model with sufficient and efficient safety obstacle avoidance decision-making basis without significantly increasing the computational burden.
[0080] In steps 32 and 33, to generate a smooth path that conforms to dynamic characteristics, this paper defines the action space as the coordinates of a set of control points of a Bézier curve. The path curve generated by these control points must satisfy the minimum turning radius constraint of the unmanned surface vessel (USV). The generated trajectory will be used as a reference input, converted into the speed command of the USV's propeller and the rudder angle command of the servo motor. Specifically, the action definition is as follows:
[0081] ,
[0082] in For the first The coordinates of the control points , This represents the number of control points in a single planning iteration.
[0083] In steps 33 and 34, to guide the reinforcement learning model to autonomously learn a path planning strategy that satisfies multiple optimization objectives, this invention designs a composite reward function. This function is calculated at each time step of the unmanned surface vessel's interaction with the environment. By providing positive incentives (rewards) or negative feedback (penalties), it shapes the unmanned surface vessel's "behavioral preferences." The definition is as follows:
[0084] ,
[0085] in To determine the weight of the corresponding reward, specifically, This is a safety reward function used to ensure a collision-free path. It calculates the shortest distance from an obstacle to the path curve and compares it with a safety distance threshold. If the distance is less than the threshold, a penalty is applied; otherwise, the value is zero. As a reward for path smoothness, the curvature of the path curve at that point is calculated by taking points along the path and compared with a tiered threshold to impose a penalty, thereby suppressing the frequent large rudder angle switching of the unmanned surface vessel in high sea states, preventing the hull from rolling violently or increasing unnecessary mechanical wear of the rudder. For the velocity stability function, large-amplitude acceleration is suppressed by penalizing the norm of the rate of change of velocity; The endpoint function determines the reward based on whether the unmanned surface vessel (USV) reaches the endpoint.
[0086] Step 4 employs the Soft Actor-Critic (SAC) algorithm as the core training framework for the path planning model. The SAC algorithm specifically includes the following steps:
[0087] Step 41: Initialize network parameters and algorithm hyperparameters, and initialize the policy network (Actor). parameters Initialize two independent Q-value networks (Critic). and parameters , Initialize two sets of corresponding target Q-value networks. and So that its parameters are respectively with , Same, that is , Initialize an empty experience replay pool. Initialize learnable temperature parameters .
[0088] Step 42: The unmanned surface vessel interacts with the environment and stores its experience. Within each time step, the unmanned surface vessel (USV) at each time step Based on the current state Through policy network Sampling to obtain action Rewards are given after interacting with the environment. and the next state Each state transition sample Store in the experience replay pool middle.
[0089] Step 43, from the experience replay pool A batch of random samples For each sample, Through the current policy network According to the next state Sampling to obtain the next action And calculate its log probability. Subsequently, the minimum Q-value of the next state is calculated using two sets of target Q-value networks, and the policy log probability weighted by the entropy term is subtracted to obtain the softened state value. Finally, the target Q-value is calculated. as follows:
[0090]
[0091] in, This is the discount factor.
[0092] Step 44: Update the parameters of the two sets of Critic networks by minimizing the Q-value network output relative to the target Q-value. The mean squared error (MSE) between the two sets of Critic networks is used to update the parameters. and Its loss function Defined as:
[0093] .
[0094] By minimizing this loss function, the value network can accurately predict the success rate and expected energy consumption of unmanned surface vessels under current water flow interference and obstacle distribution.
[0095] Step 45: Update the Actor network and temperature parameters by maximizing the expected reward and entropy of the policy. Its loss function Defined as:
[0096] .
[0097] Simultaneously update temperature parameters It enables it to automatically balance reward and entropy, and its loss function Defined as:
[0098] ,
[0099] in, The target entropy is the preset value. The physical significance of introducing the entropy term is to encourage unmanned surface vessels to maintain a certain degree of exploratory activity in unknown water environments, and to avoid the strategy from converging too early in local water areas, which would result in the inability to bypass large-scale obstacle areas.
[0100] Step 46: Soft update the target Q-value network at a small scale. The parameters of the two target Q-value networks are softly updated using the Polyak averaging method to stabilize the training process.
[0101] .
[0102] Step 47: Repeat steps 42 to 46 until training is complete and a strategy that can be used for collaborative path planning and task execution is obtained.
[0103] In step 4, for training, the training environment is constructed using procedural content generation (PCG) technology. This technology can automatically and in batches generate diverse virtual scenes with random obstacle layouts and random task start and end points. Within this generated environment, the unmanned surface vessel (USV) performs interactions over millions of time steps, and uses the interaction data to continuously update the parameters of the policy network and Q-value network using the SAC algorithm until the policy converges. The final output of this offline training process is a converged path planning model.
[0104] In step 5, each unmanned surface vessel (USV) loads the pre-trained reinforcement learning model from step 4. Specifically, based on the task sequence generated in step 2, the USV sets the first task point in the sequence as its current navigation target. Subsequently, during operation, the USV perceives its local environmental information and its own motion state in real time, and combines this information with the current navigation target to form a state vector, which is then input into the loaded reinforcement learning model. The model outputs the optimal curve trajectory in real time based on the current state, and the USV performs trajectory tracking to drive its own motion. This closed-loop process of "perception-decision-execution" is repeated at a high frequency until the USV reaches the current task point. Upon arrival, the USV updates the next task point in the task sequence as the new navigation target and repeats the above navigation process until all its assigned tasks are completed.
[0105] Example 1:
[0106] The unmanned surface vessel (USV) is designed to move within a square area with sides of 200m. There are 3 USVs, and their maximum speed is 5m / s. The maximum step size per training round is 800. The underlying reinforcement learning path planning model is trained based on the SAC algorithm, and its specific network structure and training hyperparameters are shown in Table 1.
[0107] Table 1. Parameters of the Reinforcement Learning Training Model
[0108]
[0109] The reward parameters were set in the experiment as shown in Table 2.
[0110] Table 2 Reinforcement Learning Reward Parameters
[0111]
[0112] Depend on Figure 1As shown in the figure, this invention discloses a multi-agent cooperative planning method based on hybrid hierarchical planning and reinforcement learning path planning. First, a hierarchical cooperative planning model is constructed in step 1, decoupling the complex cooperative planning problem into high-level global task planning and low-level local path planning. Then, as shown in step 2, the high-level global task planner is activated to model the high-level global task planning problem as a multi-traveling salesman problem, and a genetic algorithm is used to solve it, thereby calculating a near-optimal task execution sequence for the unmanned surface vessel (USV). Next, the core of this method lies in the low-level local path planning part. As shown in step 3, this problem is modeled as a reinforcement learning problem, and a sophisticated state space, action space, and reward function are designed to guide the model to learn the optimal local path strategy. To obtain this capability, as shown in step 4, a reinforcement learning path planning model based on the SAC algorithm is fully trained offline in diverse virtual environments. Finally, in the online execution phase of step 5, the USV loads this pre-trained model and, based on the task objective issued by the high-level layer and real-time environmental perception, instantly generates and executes the optimal path to the target point.
[0113] The specific execution flow of this method can be determined by Figure 2 The flowchart further clarifies this. The system begins with "environment initialization," setting up a 200m × 200m virtual scene containing random circular static obstacles and determining the initial states of the three unmanned surface vessels (USVs). Subsequently, the "high-level planner" phase assigns a trajectory order to the USVs, i.e., solves the MTSP (Mean Transmission Principle Scheme). Afterward, the system enters an online execution loop based on "each time step." In this loop, the USV first "obtains environmental perception and its own motion information," constructing a state vector from velocity, target relative position, and local obstacle information, and then "submits it into the policy network." This network outputs a set of "action control points" defining a fourth-order Bézier curve. Based on this, after "constructing the Bézier curve path for the current time step," the USV performs a small displacement along this path. Finally, the system determines whether to proceed to the "next time step" to continue the loop or ultimately "end" the entire task by judging whether to "end" the loop.
[0114] Figure 3 This vividly demonstrates the final execution effect of the method of the present invention in a specific collaborative planning task. In the figure, three unmanned surface vessels (Agent 1, 2, 3) start from their respective starting points and successfully plan and execute paths to their assigned target areas (light green areas and "△" marks). As can be seen from the trajectories (solid lines of different colors), all paths are highly smooth curves, maintaining safe gaps between densely distributed obstacles (dark gray circles), ultimately reaching their respective target points accurately. This fully verifies the effectiveness of the present invention through... Figure 1 The layered architecture and appendix shown Figure 2The execution process shown can effectively solve the collaborative planning problem of multiple unmanned surface vessels in complex environments. Its ability to directly generate high-quality smooth trajectories shows significant benefits in terms of real-time performance, path quality, and engineering practicality compared to traditional planning methods.
Claims
1. A collaborative planning method for unmanned surface vessels based on hierarchical planning and reinforcement learning, characterized in that, Includes the following steps: Step 1: Formulate a multi-unmanned surface vessel (USV) collaborative planning problem and decouple the collaborative planning problem into high-level global task planning and low-level local path planning; Step 2: Model the high-level global mission planning problem as a multi-traveling salesman problem and solve it to obtain the near-optimal mission execution sequence for each unmanned surface vessel; Step 3: Model the underlying local path planning problem as a reinforcement learning problem, and design the state space, action space and reward function to guide the unmanned surface vessel to learn the optimal path; Step 4: Train the reinforcement learning path planning model offline to enable it to generate the optimal path in complex obstacle environments. Step 5: Online path generation and navigation. The unmanned surface vessel loads a pre-trained reinforcement learning model to generate and execute the optimal path to the task point in real time.
2. The unmanned surface vessel cooperative planning method based on hierarchical planning and reinforcement learning as described in claim 1, characterized in that, In step 1, the multi-unmanned surface vessel system is composed of It consists of 10 unmanned surface vessels, numbered as follows: The unmanned surface vessel's (USV) movement area is designed with sides of length [missing information]. A square region; the unmanned surface vessel at the initial moment Located at the starting point Location; target point set Depend on It consists of 1 point, denoted as . .
3. The unmanned surface vessel cooperative planning method based on hierarchical planning and reinforcement learning as described in claim 1, characterized in that, In step 1, the multi-unmanned surface vessel (USV) collaborative planning problem is decoupled into high-level global task planning and low-level local path planning. The goal is to generate a safe path for each USV, starting from its starting point, visiting a series of assigned target points, and finally returning to the starting point, while satisfying the constraint that all target points are visited, so as to minimize the sum of the total path lengths traversed by all USVs to complete the task.
4. The unmanned surface vessel cooperative planning method based on hierarchical planning and reinforcement learning as described in claim 3, characterized in that, The high-level global task planning is responsible for solving the decision-making problem of task allocation and execution order. Its goal is to assign one or more target points that each UAV needs to visit in this collaborative task based on global target point information and the initial state of each UAV, and to determine a unique task execution sequence for the target points assigned to the UAV. The low-level local path planning takes the task execution sequence output by the high-level global task planning as input instructions and is responsible for solving the path generation problem of a single UAV. Its goal is to independently plan and generate a collision-free safe path for each UAV in a preset operating environment, starting from its current position or starting point, which can sequentially visit all the specified target points in its task execution sequence, and avoid all known obstacles in the process.
5. The unmanned surface vessel cooperative planning method based on hierarchical planning and reinforcement learning as described in claim 1, characterized in that, In step 2, the high-level global mission planning problem is modeled as a multi-traveling salesman problem, and the near-optimal mission execution sequence for each unmanned surface vessel is obtained by solving the problem. Specifically, the steps include: Step 21: The multi-unmanned surface vessel (USV) high-level global mission planning problem is modeled as an integer programming paradigm of the multi-traveling salesman problem, specifically defined as: Among them, decision variables This is a binary variable; a value of 1 indicates an unmanned surface vessel. From the goal Head directly to the target point ; Indicates from the target To the target The transfer cost is a pre-calculated core scalar quantity; transfer cost Defined as an unmanned surface vessel (USV) overcoming water resistance and wind / wave interference on the water surface, from the work point sail to the work site The required estimated range, the objective function of the model aims to minimize the total cost of all paths taken by the unmanned surface vessel; Step 22: Solve the Multiple Traveling Salesman Problem constructed in Step 21 using a genetic algorithm. The execution process begins with encoding the solution space. First, each potential task allocation and execution sequence scheme is represented as an independent chromosome, and a random population containing multiple chromosomes is initialized based on this encoding method. Then, by iteratively executing fitness evaluation and the core genetic operators of selection, crossover, and mutation, the population is continuously evolved to generate a population of offspring with better fitness. Finally, when the algorithm reaches the preset termination condition, the iteration process ends, and the algorithm selects the individual with the highest fitness in the current population for decoding. The decoding result is the final determined task allocation scheme and execution sequence for each unmanned surface vessel. Step 23: In the chromosome encoding and decoding stage, an encoding scheme consisting of a task chromosome and a separator chromosome is adopted; the task chromosome is... A complete permutation of the task point objectives; the dividing chromosome is a series of... The first is an array; the second is the separator chromosome, which is a group of... An array of ascending integers is used, where each integer represents a cut point. During decoding, the integers on the separator chromosome are used as cut positions on the task chromosome to determine the task point number and order performed by each unmanned surface vessel. In the initial population generation stage, a batch of initial chromosomes is constructed by randomly generating all permutations of the task chromosomes and the cut point positions in the separator chromosomes. In the design of genetic operators, the selection operator adopts the roulette wheel selection method, specifically: first, the total fitness of all individuals in the population is calculated, and then the proportion of each individual's fitness to the total is used as its probability of being selected; the crossover operator is executed independently for the two chromosomes: for the task chromosome, the sequential crossover operator is used, that is, the offspring inherits the continuous gene segments of the first parent and fills the remaining genes in the order of the second parent; for the split chromosome, the single-point or multi-point crossover operator is used, exchanging some genes of the parent after the crossover point, and after the exchange, the offspring sequence is reordered to ensure that it is in ascending order; the mutation operator is also executed independently for the two chromosomes and is applied to the newly generated offspring with a preset low probability: for the task chromosome, one of the three strategies of exchange mutation, insertion mutation and inversion mutation is selected; for the split chromosome, the random reset mutation is used, that is, a valid split position sequence is completely regenerated with a certain probability.
6. The unmanned surface vessel cooperative planning method based on hierarchical planning and reinforcement learning as described in claim 5, characterized in that, In step 21, to achieve the objective function of minimizing the total cost of all paths taken by the unmanned surface vessels, the model includes four core constraints: the first constraint is the target point access constraint, which ensures that every target in the system... The first constraint ensures that both the starting point and the task point are entered exactly once, thus guaranteeing the completion of all tasks and the closure of the path; the second constraint is a flow conservation constraint, ensuring that any unmanned surface vessel... Entering and leaving any target The number of times is equal, thus ensuring the continuity of the path; the third constraint is the starting point and return constraint, ensuring that the unmanned surface vessel... It must and can only be based on its starting point and goal. The first constraint is the sub-loop elimination constraint. By prohibiting the formation of any sub-loop that does not intersect with the set of starting nodes, it forces all assigned tasks to be connected to a single and complete path starting from the starting point. This constraint is used to force each unmanned surface vessel to complete all its assigned water monitoring tasks and then return to the only shore-based recovery point, preventing unmanned surface vessels from forming isolated navigation paths that cannot be returned in open water.
7. The unmanned surface vessel cooperative planning method based on hierarchical planning and reinforcement learning as described in claim 5, characterized in that, In step 23, the fitness function is calculated through the following steps: First, the chromosome is decoded to restore it to a specific task planning scheme; then, its total cost is calculated based on the scheme; finally, the reciprocal of the cost value is taken as the fitness of the chromosome. The specific calculation formula is as follows: 。 8. The unmanned surface vessel cooperative planning method based on hierarchical planning and reinforcement learning as described in claim 1, characterized in that, In step 3, the underlying local path planning problem is modeled as a reinforcement learning problem, and the state space, action space, and reward function are designed to guide the unmanned surface vessel to learn the optimal path. Specifically, this includes the following steps: Step 31: The state input of the unmanned surface vessel (USV) is designed as a vector, which includes its own motion information, mission target information, and environmental perception information. The own motion information includes the USV's current dynamic state and key physical quantities, including the USV's surge velocity, sway velocity, and bow rate in the hull coordinate system, to ensure that the planned trajectory generated by the model conforms to physical constraints and has smoothness. The mission target information includes the relative position and attitude of the next navigation target point in the USV's own coordinate system. The environmental perception information is a low-dimensional vector, generated by a preprocessing module after feature extraction and dimensionality reduction of the raw environmental data acquired by the sensors, and includes information on the distance and distribution of obstacles in different directions. Step 32: To generate a smooth path that conforms to dynamic characteristics, the action space is defined as the coordinates of control points of a set of Bézier curves. The path curve generated by these control points must satisfy the minimum turning radius constraint of the unmanned surface vessel (USV). The generated trajectory will be used as a reference input, converted into the speed command of the USV's propeller and the rudder angle command of the servo motor. The action definition is as follows: , in For the first The coordinates of the control points , The number of control points in a single planning iteration; Step 33: To guide the reinforcement learning model to autonomously learn a path planning strategy that satisfies multiple optimization objectives, a composite reward function was designed. This function is calculated at each time step of the unmanned surface vessel's interaction with the environment. By providing positive incentives or negative feedback, the behavioral preferences of the unmanned surface vessel are shaped. The definition is as follows: , in For the corresponding reward weight, This is a safety reward function used to ensure a collision-free path. It calculates the shortest distance from an obstacle to the path curve and compares it with a safety distance threshold. If the distance is less than the threshold, a penalty is applied; otherwise, the value is zero. As a reward for path smoothness, the curvature of the path curve at that point is calculated by taking points along the path and compared with a tiered threshold to impose a penalty, thereby suppressing the frequent large rudder angle switching of the unmanned surface vessel in high sea states, preventing the hull from rolling violently or increasing unnecessary mechanical wear of the rudder. For the velocity stability function, large-amplitude acceleration is suppressed by penalizing the norm of the rate of change of velocity; The endpoint function determines the reward based on whether the unmanned surface vessel (USV) reaches the endpoint.
9. The unmanned surface vessel cooperative planning method based on hierarchical planning and reinforcement learning as described in claim 1, characterized in that, Step 4 involves offline training of the reinforcement learning path planning model to enable it to generate optimal paths in complex obstacle environments. This includes the following steps: Step 41: Initialize network parameters and algorithm hyperparameters, and initialize the policy network. parameters Initialize two independent Q-value networks. and parameters , Initialize two sets of corresponding target Q-value networks. and So that its parameters are respectively with , Same, that is , Initialize an empty experience replay pool. Initialize learnable temperature parameters ; Step 42: The unmanned surface vessel interacts with the environment and stores its experience. Within each time step, the unmanned surface vessel (USV) at each time step Based on the current state Through policy network Sampling to obtain action Rewards are given after interacting with the environment. and the next state Each state transition sample Store in the experience replay pool middle; Step 43, from the experience replay pool A batch of random samples For each sample, Through the current policy network According to the next state Sampling to obtain the next action And calculate its log probability. Subsequently, the minimum Q-value of the next state is calculated using two sets of target Q-value networks, and the policy log probability weighted by the entropy term is subtracted to obtain the softened state value. Finally, the target Q-value is calculated. as follows: in, Discount factor; Step 44: Update the parameters of the two sets of Critic networks by minimizing the Q-value network output relative to the target Q-value. The mean squared error (MSE) between the two Critic networks is used to update the parameters of the two Critic networks. and Its loss function Defined as: . By minimizing this loss function, the value network can accurately predict the success rate and expected energy consumption of unmanned surface vessels under the current water flow interference and obstacle distribution. Step 45: Update the Actor network and temperature parameters by maximizing the expected reward and entropy of the policy. Its loss function Defined as: . Simultaneously update temperature parameters It enables it to automatically balance reward and entropy, and its loss function Defined as: , in, The target entropy is preset; Step 46: Soft update the target Q-value network at a small scale. The parameters of the two target Q-value networks are softly updated using the Polyak averaging method to stabilize the training process. . Step 47: Repeat steps 42 to 46 until training is complete and a strategy that can be used for collaborative path planning and task execution is obtained.
10. The unmanned surface vessel cooperative planning method based on hierarchical planning and reinforcement learning as described in claim 1, characterized in that, In step 5, online path generation and navigation are performed. The unmanned surface vessel loads a pre-trained reinforcement learning model, generates and executes the optimal path to the task point in real time, and sets the first task point in the sequence as the current navigation target based on the task sequence generated in step 2. Subsequently, the unmanned surface vessel perceives its local environmental information and its own motion state in real time during operation, and combines this information with the current navigation target to form a state vector, which is then input into the loaded reinforcement learning model. The model outputs the optimal curve trajectory in real time based on the current state. The unmanned surface vessel (USV) performs trajectory tracking to drive its own movement. The closed-loop process is repeated at a high frequency until the USV reaches the current task point. Upon arrival, the unmanned surface vessel updates the next task point in the mission sequence to a new navigation target and repeats the navigation process described above until all its assigned tasks are completed.