An unmanned ship regional coverage path optimization method based on constraint reinforcement learning

Abstract

This invention provides a method for optimizing unmanned surface vessel (USV) area coverage paths based on constrained reinforcement learning, belonging to the technical field of autonomous navigation and path planning for marine unmanned systems. Each USV acts as an intelligent agent, and the USV area coverage path optimization problem is modeled as a constrained Markov decision process. The constraint condition is that the difference between the optimized path and the planned path is within a preset area. The reinforcement learning framework consists of: a policy network outputting an action probability distribution based on the current environmental state; a reward evaluation network evaluating the expected rewards of states and actions; and a cost evaluation network evaluating the expected cumulative cost of state-action pairs. With the goal of maximizing policy entropy and expected rewards, the penalty strength of the cost constraint is dynamically adjusted using Lagrange multipliers, jointly optimizing the policy network, reward evaluation network, and cost evaluation network. The minimum coverage requirement is modeled as a hard constraint condition for policy optimization, ensuring navigation safety and the integrity of the mission area coverage.

Description

Technical Field

[0001] This invention relates to the field of autonomous navigation and path planning technology for marine unmanned systems, and in particular to a method for optimizing the regional coverage path of unmanned vessels based on constrained reinforcement learning. Background Technology

[0002] In recent years, with the increasing complexity of tasks such as marine monitoring, environmental surveys, and maritime security, the demand for unmanned surface vessel (USV) systems to perform periodic patrols and full-coverage reconnaissance in designated sea areas has been growing. To achieve efficient perception and information collection of target areas, USVs typically conduct systematic patrols according to pre-planned paths to ensure high coverage and low redundancy. Such path planning methods perform well in static environments and can effectively complete area coverage tasks. However, in actual marine operating environments, sudden obstacles frequently appear, forcing USVs to deviate from their original routes for real-time obstacle avoidance. Traditional obstacle avoidance strategies often employ local replanning or reactive control methods, which, while ensuring navigation safety, often neglect the maintenance of the original coverage trajectory, resulting in some areas not being traversed or being repeatedly covered, severely damaging the overall coverage and affecting mission integrity. Furthermore, existing reinforcement learning-based path planning methods typically optimize coverage as a reward, lacking a rigid guarantee mechanism for coverage performance, making it difficult to maintain stable coverage quality in the face of frequent interference.

[0003] Therefore, a method for optimizing the coverage path of unmanned surface vessels based on constrained reinforcement learning is needed. Summary of the Invention

[0004] In view of this, the present invention provides an unmanned surface vessel area coverage path optimization method based on constraint reinforcement learning. By constructing a reinforcement learning framework that integrates constraint mechanisms, the minimum coverage requirement of the mission area can be guaranteed while dealing with sudden obstacles.

[0005] Therefore, the present invention provides the following technical solution: A method for optimizing unmanned surface vessel (USV) area coverage paths based on constrained reinforcement learning includes: Each unmanned surface vessel (USV) is treated as an intelligent agent. The USV area coverage path optimization problem is modeled as a constrained Markov decision process. The constraint condition is that the difference between the USV's optimized path and the planned path is within a preset area. The reinforcement learning framework includes a policy network, a reward evaluation network, and a cost evaluation network; the constraints are introduced into the optimization objective through Lagrange multipliers. The reward evaluation network evaluates the expected returns of states and actions; the cost evaluation network evaluates the expected cumulative cost of state-action pairs; with the goal of maximizing policy entropy and expected returns, the policy network, evaluation network, and cost network are jointly optimized by dynamically adjusting the penalty intensity of cost constraints through Lagrange multipliers.

[0006] Furthermore, the constrained Markov decision process includes:

[0007] Where S is the state space, O is the agent's observation, A is the agent's action space, and P:S a S When state s executes action A, it transitions to state s. The transition probability; R=S a It is a reward function; b is a set of constraint functions, representing functions that indicate the occurrence of risk; b is a set of safety thresholds. λ is the discount factor; λ is the Lagrange multiplier.

[0008] Furthermore, the state space:

[0009] in, This represents the state information of the simulated radar at time t. Indicates its own navigation status information, Indicates the target is The shortest distance between the ship and the path at any given time, and the relative direction between the ship and the obstacle avoidance endpoint; This refers to the action performed in the previous time step.

[0010] Furthermore, the action space includes: The speed and angle output by the agent at time t.

[0011] Furthermore, the reward function includes: =

[0012]

[0013] =-

[0014] =

[0015] in, Rewards for getting closer to the target point. As a time step penalty, A reward for reaching the target point; The time step penalty coefficient; This is the distance reward coefficient. The distance at the current time step. This is the distance from the previous time step; This is the distance reward coefficient. The distance at the current time step. This is the distance from the previous time step.

[0016] Furthermore, the constraints are as follows:

[0017]

[0018] in, The safety threshold represents the upper limit of permissible long-term average constraint violations; For Lagrange multipliers; To cover the boundary constraint function, It is the straight-line distance between the unmanned vessel and the planned path at time t; This represents the maximum obstacle distance violation. When any unmanned vessel enters a distance of m meters from the planned route, it means that the area cannot be completely covered within the specified detection range, and this value begins to incur costs.

[0019] Furthermore, the balancing objective of maximizing policy entropy and expected return includes:

[0020] in, The value function for the rewarded action; and The action value function for cost is the cost action value function for the cost network and the Target_Cost network, respectively, which measures the long-term cumulative cost of performing action a in state s. To maximize policy entropy to facilitate exploration; Estimate the long-term coverage cost of performing actions under the current strategy; The network uses a conservative estimation strategy to prevent cost overestimation.

[0021] Advantages and positive effects of the present invention: This method models the minimum coverage requirement as a hard constraint for strategy optimization, and combines the soft Actor-Critic (SAC) framework with the Lagrange multiplier mechanism to ensure coverage integrity in the mission area while guaranteeing navigation safety. It addresses the problem of coverage degradation caused by obstacle avoidance processes in uncertain environments.

[0022] This method introduces a coverage constraint mechanism to guide the agent to make limited course adjustments within the vicinity, avoiding large-scale path deviations. It naturally achieves "obstacle avoidance while covering," effectively suppressing coverage performance degradation and exhibiting stronger task robustness and behavioral controllability. This addresses the problem of traditional methods often deviating significantly from the original path or failing to actively compensate for missed areas during obstacle avoidance, resulting in incomplete coverage. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a diagram of the reinforcement learning framework in the constrained reinforcement learning-based unmanned surface vessel area coverage path optimization method. Figure 2 This is a schematic diagram of the bow-shaped path in the embodiment; Figure 3 The examples show obstacle avoidance trajectories in four different scenarios. Figure 4 This is a schematic diagram of the cumulative reward during training in the example; Figure 5 This is a schematic diagram illustrating the cumulative training cost in the example. Figure 6 This is a reward curve graph of the comparative experiment in the embodiment; Figure 7 This is a cost curve graph for the comparative experiments in the examples; Figure 8 This is a reward curve diagram of the ablation experiment in the example; Figure 9 This is a cost curve of the ablation experiment in the example. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] This invention provides a method for optimizing unmanned surface vessel (USV) coverage paths based on constrained reinforcement learning. The USV is modeled as an intelligent agent whose decision-making process comprehensively considers environmental state, current position, target position, and obstacle information. Coverage constraints are explicitly introduced into the Markov decision-making process to ensure that the policy output always meets a preset coverage integrity threshold. This ensures reduced coverage gaps during obstacle avoidance and guarantees coverage efficiency.

[0028] 1. The unmanned vessel path optimization problem is modeled as a constrained Markov decision process, expressed as:

[0029] Where S is the state space, O is the agent's observation, A is the agent's action space, and P:S a S When state s executes action A, it transitions to state s. The transition probability; R=S a It is a reward function; b is a set of constraint functions, representing functions that indicate the occurrence of risk; b is a set of safety thresholds. λ is the discount factor; λ is the Lagrange multiplier.

[0030] The definition is as follows: 1) The state space of an unmanned vessel:

[0031] in, This represents the state information of the simulated radar at time t. Indicates its own navigation status information, Indicates the target is The shortest distance between the ship and the path at any given time, and the relative direction between the ship and the obstacle avoidance endpoint; This refers to the action performed in the previous time step.

[0032] Simulated radar observation A virtual lidar consisting of 16 rays is used to cover [0 360 Omnidirectional field of view. The angle between adjacent rays is 22.5 degrees. Arranged clockwise starting from true north (0°). Each ray returns the distance to the nearest obstacle surface in the current direction, denoted as . , where k is the maximum detection range. If there are no obstacles in that direction, then the value is k. ,in Representing the The length of the laser. When a target is detected approaching the ship, the radar length in the corresponding direction will shorten, because the laser also changes with the ship's course. Therefore, the unmanned ship senses the direction and distance of the target's approach.

[0033] It is information about its own navigation status. Indicates that its own ships are in Position information, heading, and speed information at any time.

[0034] Indicates the target is The shortest distance between the ship and the path at any given time, and the relative direction between the ship and the obstacle avoidance endpoint.

[0035] Action as Representing the The boat The heading and speed at any given time. The speed value is... .

[0036] 2) Action space: the velocity output by the agent at time t and angle composition:

[0037] The speed of the unmanned vessel: The speed ranges from 0 to 35. The unmanned vessel's navigation direction: The direction ranges from 0° to 360°.

[0038] 3) Reward function: The reward at time t includes: the reward for getting closer to the target point, the time step penalty, and the reward for reaching the target point. The formula is expressed as: =

[0039] in, Rewards for getting closer to the target point. As a time step penalty, A reward for reaching the target point; The reward for getting closer to the target point is set as the difference between the distance at the current time step and the distance at the previous time step, and the reward and penalty are balanced. Within certain limits, to prevent excessive changes in rewards and punishments that would hinder training. The formula is expressed as:

[0040] in, This is the distance reward coefficient. The distance at the current time step. This is the distance from the previous time step.

[0041] The time step penalty formula is expressed as: =-

[0042] in, This represents the time step penalty coefficient.

[0043] The reward formula for reaching the target point is expressed as: =

[0044] in, The reward coefficient for reaching the target point. The range for determining the destination point. This is an indicator function indicating the proximity to the target point. If it is 1, then it is 1; otherwise, it is 0.

[0045] 4) The constraint conditions are expressed as:

[0046]

[0047] in, The safety threshold represents the upper limit of permissible long-term average constraint violations; For Lagrange multipliers; To cover the boundary constraint function, It is the straight-line distance between the unmanned vessel and the planned path at time t; This represents the maximum obstacle distance violation. When any unmanned vessel enters a distance of m meters from the planned route, it means that the area cannot be completely covered within the specified detection range, and this value begins to incur costs.

[0048] 5) The optimization objective is expressed as:

[0049] 6) To incorporate constraints into the optimization objective, Lagrange multipliers are introduced to construct the final optimization objective. The final optimization objective formula is expressed as:

[0050] in, It's a strategy. It is a discount factor. It is the time step reward, where λ is the Lagrange multiplier; The safety threshold represents the upper limit of allowed long-term average constraint violations.

[0051] The optimal strategy is obtained by solving the problem with the goal of maximizing the Lagrange function while satisfying the constraints. The formula is as follows:

[0052] This method generates optimal obstacle avoidance decisions based on the unmanned surface vessel's (USV) status, target point information, and global environmental information. While allowing the USV to avoid obstacles, it approaches the original planned path as closely as possible to cover its boundary area, ensuring coverage efficiency.

[0053] like Figure 1 As shown, the reinforcement learning framework is based on SAC (Soft Actor-Critic). The Lagrange multiplier method is introduced to embed the long-term coverage cost constraint into the optimization objective, thus constructing a constrained maximum entropy reinforcement learning framework. A dual-cost structure is introduced to achieve explicit modeling and adaptive control of the cost constraint.

[0054] Policy Network: Outputs actions based on feature encoding, with a Gaussian distribution policy that supports exploration and learning in a continuous action space. Action updates depend on samples taken from the experience replay buffer, and the policy is optimized by maximizing a Lagrangian objective function.

[0055] The optimization objective of the strategy is to maximize the following Lagrange function:

[0056] in, The action value function is used to measure the long-term cumulative reward of performing action a in state s; and The action value function for cost is the cost action value function for the cost network and the Target_Cost network, respectively, which measures the long-term cumulative cost of performing action a in state s. To maximize policy entropy to facilitate exploration; Critic network: via Critic network ( The network estimates the long-term cumulative reward for performing actions under the current policy. This is achieved through Target_Critic ( The network uses a conservative estimation strategy to prevent overestimation of cumulative rewards. As a reward value estimate .

[0057] Cost Network: Through the Cost Network ( The network estimates the long-term coverage cost of performing actions under the current policy. This is achieved through Target_Cost (...). The network uses a conservative estimation strategy to prevent cost overestimation. This cost value is used to calculate the Lagrange term. This embeds constraints into the strategy optimization process.

[0058] As a learnable parameter, the cost penalty strength is dynamically adjusted, and the update method is gradient ascent. When the actual expected cost exceeds budget b Increase; otherwise decrease, to ensure that the constraint is eventually satisfied.

[0059] Example like Figure 2 As shown, to verify the effectiveness of this method, an unmanned surface vessel obstacle avoidance simulation environment is used to simulate the unmanned surface vessel encountering obstacles while performing area coverage operations along a preset "bow" shaped path, thereby triggering local obstacle avoidance behavior. The unmanned surface vessel needs to control its lateral deviation from the original path within a preset range while bypassing the obstacle, so as to ensure the coverage efficiency of the target area.

[0060] When an unmanned surface vessel (USV) encounters an obstacle blocking its preset path, this method can dynamically select a new local path while meeting the following requirements: the USV must avoid collisions with obstacles during its movement; and even during obstacle avoidance, the USV should move within a specific distance range to ensure effective coverage of the target area.

[0061] First, by demonstrating the training curves and performance tests of this method, the effectiveness of the coverage constraint in obstacle avoidance is verified. Second, by comparing it with advanced constraint reinforcement learning algorithms and reinforcement learning algorithms that only rely on rewards for obstacle avoidance, the characteristics of this method in achieving cost optimization while effectively controlling risk are verified. Finally, by conducting ablation experiments on the risk constraint mechanism, the effectiveness of each constraint reinforcement learning obstacle avoidance module is verified.

[0062] Experiment 1: Effect Experiment The cumulative reward and cumulative cost graphs for training the obstacle avoidance path coverage optimization strategy based on constraint reinforcement learning are shown below. Figure 4 , Figure 5 As shown, the cost fluctuates greatly in the initial stage, indicating that the agent frequently triggers constraint penalties during the exploration phase. As training progresses, both the cumulative reward and cumulative cost converge at 250 steps, the cost decreases rapidly and tends to stabilize, and finally converges to a level close to zero. The smooth curve (red) shows that the overall learning process is stable, indicating that the strategy gradually learns to avoid violating safety constraints.

[0063] The experimental results trajectory graph is shown as follows: Figure 3 As shown, the experimental results demonstrate the path planning and obstacle avoidance performance of the agent in four different scenarios (obstacle avoidance on inclined paths, horizontal paths, and vertical paths). The trajectory graphs show that the agent successfully reaches the target point from the starting point, with a smooth and continuous path. In all scenarios, the agent effectively avoids obstacles (red squares) and stays outside the threat area (red circles), with the path always remaining within the constrained bandwidth (blue lines), without any out-of-bounds behavior, indicating good security and robustness of the strategy. In summary, this algorithm can achieve safe, stable, and efficient path planning in complex environments, meeting navigation requirements under obstacle avoidance and coverage constraints, improving path coverage efficiency, and reducing coverage gaps, demonstrating the algorithm's adaptability and good generalization in different scenarios.

[0064] Experiment 2: Comparative Experiment This method (denoted as LSAC) is compared with the advanced reinforcement learning method SAC, traditional reinforcement learning methods, path constraint-based reinforcement learning methods using rewards and penalties, and the constraint-based reinforcement learning method PPO_Lagrangian (PPO based on the Lagrangian framework). All algorithms are trained under the same environmental settings and hyperparameters. The test data results are shown in Table 1. Figure 6 and Figure 7 As shown.

[0065] Table 1

[0066] Performance is evaluated from two dimensions: Total Cost measures the degree to which the strategy adheres to the safe distance constraint, while Total Reward reflects task completion efficiency and path optimization capability. For example... Figure 7As shown in the total cost curves, the red curve represents our method, the blue curve represents PPO_Lagrangian, and the purple curve represents SAC. It can be seen that the total cost of our method decreases rapidly in the early stages of training and converges stably to near zero after about 100 rounds, indicating that it can efficiently learn and strictly satisfy the preset safety boundary constraints, avoiding path deviations or out-of-bounds behavior caused by obstacle avoidance. In contrast, although the cost of PPO_Lagrangian also shows a decreasing trend, it fluctuates more significantly, and the final value is slightly higher than our method, indicating that it has insufficient stability in the constraint optimization process. The cost of the SAC method remains consistently high, reflecting that simply relying on negative reward penalties is insufficient to effectively guide the agent to learn precise boundary behavior, easily leading to frequent out-of-bounds behavior. The table above shows the test results of the policies obtained after training 500 episodes of each algorithm in the same random environment (the four environments mentioned above). It can be seen that our method has a cost of 0 under this policy and can safely reach the target point in all cases, significantly outperforming PPO_Lagrangian and the SAC reinforcement learning algorithm with reward design.

[0067] In summary, this method demonstrates stronger robustness and convergence in balancing constraint satisfaction and task performance. Its advantages stem from two main points: firstly, the Critic network structure is better suited for constraint modeling in continuous action spaces, enabling more precise estimation of constraint violation risks; secondly, the Lagrangian framework achieves dynamic coordination between constraints and objectives, avoiding hyperparameter sensitivity and incentive imbalance issues inherent in traditional reward-penalty mechanisms. Therefore, this method has higher practical value and wider application potential in unmanned surface vessel (USV) area coverage tasks. These results validate the effectiveness and robustness of this method in guaranteeing coverage in complex obstacle avoidance scenarios.

[0068] Experiment 3: Ablation Experiment To verify the effectiveness of the constraint Critic mechanism in this method, the performance of three different strategies in terms of total cost is compared. The red curve in the figure represents the present method, which introduces a constraint Critic (cost-critic) to explicitly model the safe distance constraint; the blue curve is LSAC_cost_critic, indicating that after removing the constraint Critic, the constraint is handled only by the main Critic network; the purple curve is the SAC method, which uses traditional reward and penalty methods (such as out-of-bounds penalties) to limit path deviation.

[0069] The results are as shown in 8 and Figure 9As shown, our proposed method converges rapidly in the early stages of training, and the total cost eventually stabilizes at the lowest level, indicating that it can effectively learn a safe and efficient obstacle avoidance strategy and consistently maintain it within the allowable coverage bandwidth. In contrast, the blue curve (unconstrained Critic), while showing acceptable performance initially, exhibits significant fluctuations in the later stages, with the cost consistently exceeding the optimal value. This suggests that the lack of explicit constraint modeling leads to frequent violations of safety boundaries during exploration, making stable convergence difficult. While the purple curve (reward-penalty based) shows a decreasing cost trend, it remains significantly higher than our proposed method overall and exhibits considerable oscillations, reflecting that relying solely on negative rewards and penalties can easily lead to learning instability and makes it difficult to precisely control constraint boundaries. In conclusion, the design of the constrained Critic effectively enhances the algorithm's ability to synergistically optimize safety and coverage efficiency in complex obstacle avoidance tasks, validating the innovation and practicality of our proposed method.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for optimizing the coverage path of unmanned surface vessels (USVs) based on constrained reinforcement learning, characterized in that, include: Each unmanned vessel is treated as an intelligent agent, and the unmanned vessel area coverage path optimization problem is modeled as a constrained Markov decision process. The constraint condition is that the difference between the optimized path and the planned path of the unmanned vessel is within a preset area; The reinforcement learning framework includes: policy network, reward evaluation network, and cost evaluation network; The constraints are introduced into the optimization objective using Lagrange multipliers; The reward evaluation network evaluates the expected returns of states and actions; the cost evaluation network evaluates the expected cumulative cost of state-action pairs; with the goal of maximizing policy entropy and expected returns, the policy network, evaluation network, and cost network are jointly optimized by dynamically adjusting the penalty intensity of cost constraints through Lagrange multipliers.

2. The method according to claim 1, characterized in that, The constrained Markov decision process includes: Where S is the state space, O is the agent's observation, A is the agent's action space, and P:S a S When state s executes action A, it transitions to state s. The transition probability; R=S a It is a reward function; b is a set of constraint functions, representing functions that indicate the occurrence of risk; b is a set of safety thresholds. λ is the discount factor; λ is the Lagrange multiplier.

3. The method according to claim 1, characterized in that, The state space: in, This represents the state information of the simulated radar at time t. Indicates its own navigation status information, Indicates the target is The shortest distance between the ship and the path at any given time, and the relative direction between the ship and the obstacle avoidance endpoint; This refers to the action performed in the previous time step.

4. The method according to claim 1, characterized in that, The action space includes: The speed and angle output by the agent at time t.

5. The method according to claim 1, characterized in that, The reward function includes: = =- = in, Rewards for getting closer to the target point. As a time step penalty, A reward for reaching the target point; The time step penalty coefficient; This is the distance reward coefficient. The distance at the current time step. This is the distance from the previous time step; This is the distance reward coefficient. The distance at the current time step. This is the distance from the previous time step.

6. The method according to claim 1, characterized in that, The constraints are as follows: in, The safety threshold represents the upper limit of permissible long-term average constraint violations; For Lagrange multipliers; To cover the boundary constraint function, It is the straight-line distance between the unmanned vessel and the planned path at time t; This represents the maximum obstacle distance violation. When any unmanned vessel enters a distance of m meters from the planned route, it means that the area cannot be completely covered within the specified detection range, and this value begins to incur costs.

7. The method according to claim 1, characterized in that, The goal of balancing maximizing strategy entropy and expected return includes: in, The value function for the rewarded action; and The action value function for cost is the cost action value function for the cost network and the Target_Cost network, respectively, which measures the long-term cumulative cost of performing action a in state s. To maximize policy entropy to facilitate exploration; Estimate the long-term coverage cost of performing actions under the current strategy; The network uses a conservative estimation strategy to prevent cost overestimation.

Images