Unmanned ship trajectory planning method and system

Abstract

The invention discloses an unmanned ship trajectory planning method and system. The method comprises the following steps: S1, obtaining a historical state data sequence of the unmanned ship under the influence of environmental disturbance; s2, on the basis of the historical state data sequence, training is carried out by adopting a neural network model based on Transform to obtain a coupled hydrodynamic model; s3, taking the coupled hydrodynamic model as a state transition model in a reinforcement learning training environment, and training a reinforcement learning agent to obtain an optimal trajectory planning strategy in the reinforcement learning training environment based on a preset state space, an action space and a reward function; and S4, generating a navigation track by using the optimal track planning strategy according to the current state and the target information of the unmanned ship. According to the method, complex hydrodynamic characteristics are learned in a data driving mode and fused into the reinforcement learning planner, a safe, dynamic and feasible track can be generated, and the accessibility of the track, the target point precision and the overall navigation safety are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of unmanned surface vessel (USV) technology, specifically to a USV trajectory planning method and system. Background Technology

[0002] Unmanned surface vessels (USVs), as intelligent platforms capable of autonomous navigation on water, have broad application prospects in fields such as marine surveying, maritime patrol, environmental monitoring, and search and rescue. Trajectory planning is one of the core technologies for USVs to achieve autonomous navigation. Its goal is to generate an optimal or suboptimal path from the starting point to the target point for the USV while satisfying various constraints (such as obstacle avoidance and dynamic constraints).

[0003] Traditional unmanned surface vessel (USV) trajectory planning algorithms mainly include search-based algorithms (such as the A* algorithm) and sampling-based algorithms (such as the Rapid Expanding Random Tree (RRT) algorithm). These algorithms can effectively generate obstacle avoidance paths in specific scenarios, but they often suffer from the following drawbacks:

[0004] 1) The generated path may not be smooth, requiring additional smoothing processing, which increases computational complexity;

[0005] 2) The computational cost is high, especially in complex or large-scale environments;

[0006] 3) It has poor adaptability to environmental disturbances (such as wind, waves, and water flow). The planned ideal trajectory is prone to large deviations in actual execution because these algorithms usually ignore or simplify the complex hydrodynamic characteristics of unmanned surface vessels in real hydrological environments.

[0007] In recent years, reinforcement learning (RL) has been introduced into the field of unmanned surface vessel (USV) trajectory planning due to its self-learning and optimization capabilities in complex decision-making problems. RL-based methods learn the optimal policy through continuous interaction and trial and error between the agent and the environment, without requiring a precise system model. However, most existing RL-based USV trajectory planning methods are still primarily validated in simulation environments, and their main challenges include:

[0008] 1) The difference between the simulation environment and the real physical environment leads to a decrease in the performance of the strategy trained in the simulation in the real environment;

[0009] 2) The complex and time-varying hydrodynamic disturbances were not adequately considered or accurately modeled, which made it difficult for the planned trajectory to be accurately tracked by the underlying controller during actual navigation, affecting navigation safety and efficiency.

[0010] Traditional planning algorithms ignore environmental disturbances, which can cause the USV to deviate from the desired path. Existing reinforcement learning methods are also limited by the differences between the simulation and the real physical environment and the problem of unmodeled hydrodynamic disturbances.

[0011] Therefore, how to accurately model the hydrodynamic characteristics of unmanned surface vessels (USVs) under environmental disturbances such as wind, waves, and currents, and effectively integrate them into the trajectory planning process to improve the actual feasibility, navigation robustness, and safety of the planned trajectory, is a technical problem that urgently needs to be solved in the field of USV trajectory planning. Summary of the Invention

[0012] The main objective of this invention is to overcome the shortcomings of the prior art and provide an unmanned surface vessel trajectory planning method and system that can adapt to complex hydrological environments, improve trajectory tracking accuracy and navigation safety.

[0013] To achieve the above objectives, the present invention provides an unmanned surface vessel trajectory planning method, comprising the following steps:

[0014] Includes the following steps:

[0015] S1. Obtain the historical state data sequence of the unmanned surface vessel under the influence of environmental disturbances;

[0016] S2. Based on the historical state data sequence, a coupled hydrodynamic model is obtained by training a Transformer-based neural network model.

[0017] S3. The coupled hydrodynamic model is used as the state transition model in the reinforcement learning training environment. In the reinforcement learning training environment, based on the preset state space, action space, and reward function, the reinforcement learning agent is trained to obtain the optimal trajectory planning strategy. The state space contains the information required to describe the current situation and mission objective of the unmanned surface vessel. The action space defines the control commands that the unmanned surface vessel can execute. The reward function guides the agent to learn the desired behavior, such as efficiently reaching the target, smooth navigation, and obstacle avoidance.

[0018] S4. Using the optimal trajectory planning strategy, generate a navigation trajectory based on the current state of the unmanned surface vessel and the target information.

[0019] Preferably, the historical state data sequence includes the unmanned surface vessel's position, attitude, speed, acceleration, control input, and environmental parameters such as wind speed, wave height, and current velocity sensed by sensors. This data can be derived from the unmanned surface vessel's navigation records in real waters or high-fidelity simulation environments.

[0020] Preferably, the Transformer-based neural network model employs an encoder structure, including a multi-head self-attention mechanism layer and a position encoding layer. This model is specifically designed to learn and characterize the nonlinear, time-varying hydrodynamic characteristics of unmanned surface vessels under complex environmental disturbances. The Transformer model is chosen due to its advantages in processing time-series data (such as capturing long-term dependencies).

[0021] Preferably, the state space of the reinforcement learning training environment includes the motion state information of the unmanned surface vessel (USV) at the current and several past moments, the relative distance and relative azimuth angle between the USV and the preset waypoints, the relative distance and relative azimuth angle between the USV and the final target point, and the relative distance and relative azimuth angle between the USV and perceived obstacles.

[0022] Preferably, the action space of the reinforcement learning agent includes the desired longitudinal velocity and desired bow roll rate of the unmanned surface vessel.

[0023] Preferably, the reward function of the reinforcement learning agent includes a speed-maintaining reward for encouraging smooth speed maintenance, a line-of-sight angle reward for guiding the direction toward the target point, a speed range reward for encouraging efficient navigation within a safe range, a distance reward for encouraging approach to the target point, an arrival reward for encouraging reaching intermediate waypoints and the final target point, and a collision penalty for penalizing collisions with or getting too close to obstacles.

[0024] Preferably, the reinforcement learning agent is trained using the proximal policy optimization (PPO) algorithm or a variant thereof to balance exploration and exploitation and to ensure the stability of the training process.

[0025] The present invention also provides an unmanned surface vessel trajectory planning system, which corresponds to the above method and includes:

[0026] The data acquisition module is used to acquire historical state data sequences of unmanned surface vessels under the influence of environmental disturbances;

[0027] The hydrodynamic modeling module is used to train a coupled hydrodynamic model using a Transformer-based neural network model based on the historical state data sequence.

[0028] The reinforcement learning training module is used to use the coupled hydrodynamic model as a state transition model in the reinforcement learning training environment, and in the reinforcement learning training environment, based on the preset state space, action space and reward function, to train the reinforcement learning agent to obtain the optimal trajectory planning strategy.

[0029] The trajectory generation module is used to generate a navigation trajectory based on the current state and target information of the unmanned surface vessel using the optimal trajectory planning strategy.

[0030] Preferably, the hydrodynamic modeling module uses a Transformer-based neural network model with an encoder structure, including a multi-head self-attention mechanism layer and a position encoding layer.

[0031] Preferably, the reinforcement learning training module uses the proximal policy optimization (PPO) algorithm or a variant thereof to train the reinforcement learning agent.

[0032] Compared with the prior art, the beneficial effects of the present invention are:

[0033] 1. Improved accuracy of hydrodynamic model: By adopting a Transformer-based data-driven approach, coupled hydrodynamic characteristics are learned from real or high-fidelity USV state data, which can more accurately capture the impact of complex environmental disturbances such as wind, waves, and currents on USV motion, outperforming traditional simplified models or models based purely on physical parameter identification.

[0034] 2. Enhanced practical feasibility and robustness of trajectory planning: By embedding the learned high-precision hydrodynamic model (NEMO) into the training environment of the reinforcement learning planner, the RL agent can consider real dynamic responses and environmental disturbances during the learning process. Therefore, the planned trajectory is more consistent with the actual motion capabilities of the USV, exhibiting better accessibility and robustness to disturbances in real-world environments.

[0035] 3. Improved navigation safety and efficiency: Because the planned trajectory fully considers dynamic constraints and environmental influences, the USV can track the trajectory more accurately, reduce unnecessary maneuvers and energy consumption, and effectively avoid obstacles through the design of the reward function, thereby improving the overall safety and efficiency of navigation.

[0036] 4. Reduces the burden on the underlying control system: Traditional planning methods often produce trajectories that are difficult for the controller to track accurately due to neglecting dynamics or disturbances, leading to frequent and significant adjustments in the control system. The trajectory generated by this invention itself has good dynamic feasibility, thus reducing the requirements on the underlying controller, resulting in smoother control output and improved overall system stability.

[0037] 5. Improved Sim-to-Real performance: By using hydrodynamic models learned from real data or high-fidelity simulation data, the environment model on which reinforcement learning agents rely for training is closer to the real world, which helps to narrow the gap between simulation and reality and improve the performance after policy transfer to a real USV platform. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the overall framework of the unmanned surface vessel trajectory planning method in this invention;

[0039] Figure 2 This is a flowchart of the unmanned surface vessel trajectory planning method in this invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] like Figure 1 As shown, the specific implementation process of the unmanned surface vessel trajectory planning method proposed in this invention can be summarized into two main stages: the first stage is the offline training of the coupled hydrodynamic model (NEMO) and the training of the reinforcement learning (RL) strategy; the second stage is the online trajectory planning and execution.

[0042] Phase 1: Offline Training

[0043] Step 1: Data Collection and Preprocessing

[0044] First, collect a large amount of unmanned surface vessel (USV) status data. This data can be recorded from real-world USV navigation under different sea conditions (wind, waves, currents) or obtained from high-fidelity simulation environments (such as VRX-based simulation environments). The collected data should include timestamps, USV position (x, y), heading angle ψ, velocities (u, v, r) in the appendage coordinate system (longitudinal velocity, lateral velocity, and bow roll rate), and actual control commands τ = [τ...]. u ,0,τ r ] T And possible environmental parameters (such as wind speed, wind direction, wave height, current velocity, etc., if measurable).

[0045] The collected data is preprocessed, such as data cleaning, format standardization, time series alignment, and feature engineering (e.g., calculating the distance and angle relative to the target point).

[0046] Step 2: Training the Coupled Hydrodynamic Model (NEMO)

[0047] The preprocessed historical state data sequence from step 1 is used to train the coupled hydrodynamic model NEMO. The goal of NEMO is to learn the dynamic response of the USV in complex hydrodynamic environments.

[0048] The simplified three-degree-of-freedom dynamic model of the USV can be expressed as:

[0049]

[0050] Where η = [x, y, ψ] T v = [u, v, r] T ,τ ctrl It is the control input, τ envIt represents environmental disturbance. M, C(v), and D(v) are the inertia matrix (including the added mass), the Coriolis force and centripetal force matrix, and the damping matrix, respectively.

[0051] Traditional methods struggle to accurately model M, C(v), and D(v) with respect to τ. env .

[0052] This invention employs a Transformer-based neural network model to directly learn these complex, coupled dynamic effects from data, or to directly learn the state transition function s. t+1 =f(s) t ,a t ), where s t This is the current state, a t It is the current action (control input), s t+1 It is the state at the next moment.

[0053] Specifically, the encoder structure of the Transformer model is used for this task. The Transformer-based neural network model employs its core encoder structure, which contains one or more encoder layers, each containing a multi-head self-attention sublayer and a feed-forward neural network sublayer. Furthermore, the model includes a positional encoding layer to enable it to understand the temporal order and relative positional relationships of the states in the input sequence. This model uses the state sequence of the unmanned surface vessel over a past period as input to predict its state or the hydrodynamic forces it experiences at a future moment. The input is the state-action sequence (s) of the past k time steps. t-k+1 ,a t-k+1 ,...,s t ,a t The output is the state s at the next time step. t+1 The model predicts or estimates hydrodynamic parameters. It uses a multi-head self-attention mechanism to capture the dependencies between states at different time points in the sequence and interprets temporal information through positional encoding.

[0054] The training objective is to minimize the difference between the state predicted by the model and the state actually observed (e.g., using a mean squared error loss function). The Transformer-based NEMO model achieves higher accuracy than LSTM and MLP in predicting the longitudinal velocity u, lateral velocity v, and yaw rate r of the USV.

[0055] Step 3: Strengthen training of learning trajectory planning strategies

[0056] The state space of the reinforcement learning training environment is designed to include: the motion state of the unmanned surface vessel (USV) at the current moment and several previous historical moments (such as forward speed in the attached coordinate system); the distance and line-of-sight (LOS) angle between the current position of the USV and the preset waypoint; the distance and LOS angle between the current position of the USV and the final target point; and the distance and corresponding LOS angle between the USV and the center of dynamic or static obstacles sensed by sensors (such as lidar and cameras).

[0057] The trained NEMO model is used as a dynamic simulator in a reinforcement learning environment. When the RL agent performs an action a in this environment... t (For example, the expected (u) des ,r des When )), the NEMO model is used to predict the next state s of the USV. t+1 .

[0058] State Space S: States s t This can include the current forward speed u of the USV. t Distance to the nearest waypoint and LOS corner Distance to the final target point and LOS corner To include historical information, the state {s} of the current and the three past time steps can be included. t-3 ,s t-2 ,s t-1 ,s t} is used as input. Additionally, the distance to obstacles is also included. and LOS corner

[0059] Action Space A: The action a = {u} output by the agent des ,r des This refers to the desired forward speed and bow roll rate of the USV. These desired values ​​are subsequently translated into actual commands for the thrusters and servos by the underlying controller (such as a PID controller).

[0060] The reward function of a reinforcement learning agent consists of a weighted sum of multiple sub-items. The reward function R is a comprehensive reward function r. all To guide the intelligent agent's learning. It includes:

[0061] (Speed ​​Maintenance Reward): Encourages steady speed and penalizes drastic changes in speed between the current and previous moments, thus promoting smooth navigation.

[0062] rsight (Line of sight reward): Encourages and rewards the unmanned surface vessel for the correct orientation toward the target (waypoint or final destination).

[0063] (Speed ​​Range Reward): Encourages navigation within a suitable speed range, and rewards or penalizes the unmanned surface vessel based on whether its current speed is within a preset safe or efficient range.

[0064] r dist (Distance Reward): Encourages approaching the target by giving a positive reward when the unmanned vessel gets closer to the target point.

[0065] r waypoint ,r goal (Arrival Reward): A large reward is given for reaching a waypoint / destination. A significant positive reward is given when the unmanned surface vessel successfully reaches an intermediate waypoint or final destination.

[0066] r collision (Collision Penalty): A large penalty is imposed for colliding with or getting too close to an obstacle. A large negative penalty is imposed when the unmanned surface vessel collides with an obstacle or enters the safe radius of the obstacle.

[0067] The Proximal Policy Optimization (PPO) algorithm or its variants (such as PPO-Clip) are used to train the policy network (Actor) and value network (Critic) of the RL agent. The policy network outputs actions based on the current state, and the value network evaluates the value of the current state. The training objective is to maximize the cumulative expected reward. Using the Transformer-based NEMO as the environment model for RL training yields higher total rewards, waypoint arrival rewards, and a more reasonable speed maintenance reward curve compared to using MLP or LSTM models, indicating better policy learning performance.

[0068] Phase Two: Online Trajectory Planning and Execution

[0069] When unmanned surface vessels (USVs) perform tasks in real-world environments:

[0070] 1. The sensors of the unmanned surface vessel (such as GPS, IMU, LiDAR, and cameras) perceive its own status (position, attitude, and speed) and surrounding environmental information (obstacles) in real time.

[0071] 2. This real-time information is processed and then input into the pre-trained RL policy network.

[0072] 3. The policy network outputs the desired action 'a' based on the current state. t ={u des ,r des}

[0073] 4. This desired action can be used as a short-term local objective or directly as the setpoint for the underlying controller. The underlying controller (e.g., a cascaded PID controller) will set the desired action (u) to... des ,r des This is converted into specific control commands for the USV's propulsion and steering systems.

[0074] 5. The USV executes these instructions, changes its motion state, and enters a new cycle.

[0075] In this way, unmanned surface vessels can dynamically plan their trajectory (or behavior) based on real-time conditions.

[0076] The proposed method was verified through simulation and physical experiments.

[0077] Simulation Experiments: In the VRX simulation environment, compared with traditional methods such as A*+DWA and USV-Tracker, the trajectory generated by the method of this invention performs better in terms of reachability, target point accuracy, navigation safety (such as minimum distance to obstacles), and deviation from the desired trajectory. This indicates that an RL planner that considers hydrodynamic effects can generate more realistic trajectories.

[0078] Ablation experiments: demonstrated that the Transformer-based NEMO model outperforms MLP and LSTM in hydrodynamic modeling accuracy and assisted RL training performance.

[0079] Physical experiment: Tests were conducted in a real aquatic environment (wind speeds up to 32 km / h) using a custom-designed small USV platform. The results show that the NEMO model can accurately predict the motion state of a real USV.

[0080] The trajectory generated by the method of this invention is superior to the comparative methods, including RL methods without NEMO integration, in terms of actual trajectory length, deviation from the planned trajectory, position and velocity deviation, and minimum distance to obstacles.

[0081] The output of the underlying PID controller is more stable and the oscillation is significantly reduced, indicating that the method of the present invention reduces the burden on the control system.

[0082] In complex obstacle scenarios, the method of this invention can guide the USV to the target point safely and accurately, while the comparison method may result in large drift or collision.

[0083] In summary, this invention effectively solves the problem of unmanned surface vessel trajectory planning in complex hydrological environments by combining a data-driven Transformer hydrodynamic model with reinforcement learning, thereby improving the accuracy, robustness, and safety of the planning.

[0084] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A method for unmanned surface vessel trajectory planning, characterized in that, Includes the following steps: S1. Obtain the historical state data sequence of the unmanned surface vessel under the influence of environmental disturbances; S2. Based on the historical state data sequence, a coupled hydrodynamic model is obtained by training a Transformer-based neural network model. S3. Use the coupled hydrodynamic model as the state transition model in the reinforcement learning training environment, and in the reinforcement learning training environment, train the reinforcement learning agent to obtain the optimal trajectory planning strategy based on the preset state space, action space and reward function. S4. Using the optimal trajectory planning strategy, generate a navigation trajectory based on the current state of the unmanned surface vessel and the target information.

2. The unmanned surface vessel trajectory planning method according to claim 1, characterized in that, The historical state data sequence includes the unmanned surface vessel's position, attitude, speed, acceleration, control input, and environmental parameters such as wind speed, wave height, and current velocity sensed by sensors.

3. The unmanned surface vessel trajectory planning method according to claim 2, characterized in that, The Transformer-based neural network model employs an encoder structure, including a multi-head self-attention mechanism layer and a positional encoding layer.

4. The unmanned surface vessel trajectory planning method according to claim 3, characterized in that: The state space of the reinforcement learning training environment includes the motion state information of the unmanned surface vessel (USV) at the current and several past moments, the relative distance and relative azimuth between the USV and the preset waypoints, the relative distance and relative azimuth between the USV and the final target point, and the relative distance and relative azimuth between the USV and perceived obstacles.

5. The unmanned surface vessel trajectory planning method according to claim 4, characterized in that, The action space of the reinforcement learning agent includes the desired longitudinal velocity and desired bow angular velocity of the unmanned surface vessel.

6. The unmanned surface vessel trajectory planning method according to claim 5, characterized in that, The reward function of the reinforcement learning agent includes a speed-maintaining reward for encouraging smooth speed maintenance, a line-of-sight reward for guiding the direction toward the target point, a speed range reward for encouraging efficient navigation within a safe range, a distance reward for encouraging approach to the target point, an arrival reward for encouraging reaching intermediate waypoints and the final target point, and a collision penalty for penalizing collisions with or getting too close to obstacles.

7. The unmanned surface vessel trajectory planning method according to claim 6, characterized in that, The reinforcement learning agent is trained using the Proximal Policy Optimization (PPO) algorithm or a variant thereof.

8. An unmanned surface vessel (USV) trajectory planning system, applied in the USV trajectory planning method as described in any one of claims 1-7, characterized in that, include: The data acquisition module is used to acquire historical state data sequences of unmanned surface vessels under the influence of environmental disturbances; The hydrodynamic modeling module is used to train a coupled hydrodynamic model using a Transformer-based neural network model based on the historical state data sequence. The reinforcement learning training module is used to use the coupled hydrodynamic model as a state transition model in the reinforcement learning training environment, and in the reinforcement learning training environment, based on the preset state space, action space and reward function, to train the reinforcement learning agent to obtain the optimal trajectory planning strategy. The trajectory generation module is used to generate a navigation trajectory based on the current state and target information of the unmanned surface vessel using the optimal trajectory planning strategy.

9. The unmanned surface vessel trajectory planning system according to claim 8, characterized in that, The hydrodynamic modeling module uses a Transformer-based neural network model with an encoder structure, including a multi-head self-attention mechanism layer and a position encoding layer.

10. The unmanned surface vessel trajectory planning system according to claim 8, characterized in that, The reinforcement learning training module uses the Proximal Policy Optimization (PPO) algorithm or a variant thereof to train the reinforcement learning agent.

Images