End-to-end ship path following control method based on EMPPO reinforcement learning

Abstract

The invention discloses an end-to-end ship path following control method based on EMPPO reinforcement learning, and the method comprises the steps: constructing an ABCFit adaptive channel path smooth fitting algorithm model, generating channel path information, constructing an NIShip ship motion simulation platform, generating ship state information through the simulation platform according to the channel path information, and carrying out the ship state information through the NIShip ship motion simulation platform. And an EMPPO ship path following control model is constructed and trained, the EMPPO ship path following control model is deployed in target equipment after training is completed, the EMPPO ship path following control model makes an action decision based on a strategy network according to the channel path information and the ship state information, and the ship is controlled to steer in real time so as to reach an expected reference path. Through the process, the ship path following control method which is more efficient, more stable and higher in robustness is realized.

Description

Technical Field

[0001] The present invention relates to a ship path following control method, and in particular to an end-to-end ship path following control method based on EMPPO reinforcement learning. Background Art

[0002] Ship path following is a key issue in ship automatic control. Its goal is to enable the ship to accurately navigate along a predetermined path while minimizing lateral and heading errors. However, due to the underactuated nature of ships, their nonlinear dynamic models, and the influence of external disturbances (such as wind, waves, and currents), traditional path following methods have many limitations. First, they rely on the dynamic model. Most control methods require the accuracy of the ship dynamics model, but ship motion is highly nonlinear and is affected by uncertainties such as wind, waves, and currents, making it difficult to accurately model the model. Second, they have poor anti-disturbance capabilities. External disturbances such as currents and wind and waves can degrade the performance of traditional controllers, making it difficult to achieve stable path following. Third, the control process is unstable. The rudder angle commands output by traditional methods change frequently, which can easily cause servo vibration, affecting navigation stability and servo life. Fourth, path generation is discontinuous. Traditional path generation methods (such as splicing straight lines and circular arcs) suffer from sudden changes in curvature, resulting in uneven ship maneuvering during path following.

[0003] With the great potential shown by deep reinforcement learning in the field of ship control, especially the proximal policy optimization algorithm (PPO), which learns the optimal control strategy through the interaction between the intelligent agent and the environment, there is no need to build a ship dynamics model, thus solving the model dependence problem. However, the existing path following control methods based on deep reinforcement learning still have problems such as low training efficiency, unstable rudder angle control and poor adaptability to path curvature. Specifically, the reinforcement learning training process requires a large amount of data and time, and the convergence speed is slow; most reinforcement learning methods do not constrain the amplitude of the rudder angle change, and the rudder angle vibration problem is not solved; the impact of path curvature on controller input and decision-making is not fully considered, which leads to the problem of decreased following accuracy when the curvature path is complex. Therefore, it has become an industry need to provide a more efficient, more stable and more robust ship path following control method. Summary of the Invention

[0004] In view of the above-mentioned deficiencies in current ship navigation and obstacle avoidance, the present invention provides a method for autonomous ship navigation and obstacle avoidance that integrates visual perception and nonlinear modeling, so as to solve the problems in the existing technology of insufficient adaptability to complex waterway environments, reliance on human intervention, and low algorithm stability and efficiency.

[0005] To achieve the above objectives, a first aspect of the present invention provides an end-to-end ship path following control method based on EMPPO reinforcement learning, comprising: constructing an ABCFit adaptive channel path smooth fitting algorithm model and generating channel path information;

[0006] Constructing a NIShip ship motion simulation platform, wherein the NIShip ship motion simulation platform generates ship status information according to the waterway path information;

[0007] Build and train the EMPPO ship path following control model, and deploy it to the target device after training.

[0008] The EMPPO ship path following control model makes action decisions based on the policy network according to the channel path information and the ship status information, and controls the ship to steer in real time to achieve the expected reference path.

[0009] In some embodiments according to the first aspect of the present invention, the path fitting algorithm of the ABCFit adaptive channel path smooth fitting algorithm model includes:

[0010] Import discrete points of waterway paths, and import the discrete point data containing waterway path information into the algorithm system as the basic data for subsequent processing;

[0011] Determine the control points of the cubic Bezier curve, select the starting point of the original path as the first control point of the cubic Bezier curve, and select the ending point of the original path as the last control point of the cubic Bezier curve;

[0012] Solve the intermediate control points and use the least squares method to calculate the intermediate control points of the cubic Bezier curve based on the determined first and last control points and discrete point data;

[0013] Compare the fitting residual with the threshold, calculate the residual of the current fitting curve, and compare it with the manually set threshold;

[0014] Path segmentation processing: if the fitting residual is greater than the set threshold, the reference path is segmented at the path point corresponding to the maximum fitting residual point;

[0015] The fitting operation is repeated from step 2 to step 5 until the fitting residuals at all curve fitting points are less than the threshold value;

[0016] Control point adjustment, taking curvature and residual factors into consideration, adjusts the control points of several segmented Bezier curves to obtain the final curve.

[0017] In some embodiments of the first aspect of the present invention, the NIShip ship motion simulation platform includes a ship motion model and a first-order Nomoto model. The ship motion model uses the first-order Nomoto model to linearly model inland ships to obtain the motion and state information of the ship, and the information includes coordinates, heading and steering commands.

[0018] In some embodiments according to the first aspect of the present invention, the ship motion model is constructed based on the maneuvering characteristics of inland waterway ships and the maneuverability of the model is verified through Z-shaped experiments and full-turn experiments.

[0019] In some embodiments according to the first aspect of the present invention, the EMPPO ship path following control model is constructed by a deep reinforcement learning network;

[0020] The states of the model’s state space include path information and ship state information;

[0021] The model’s neural network includes an action network and a value network;

[0022] The reward and penalty items of the model's reward function include lateral displacement error, heading angle error and rudder angle change amplitude.

[0023] In some embodiments according to the first aspect of the present invention, mean square error and policy entropy are added to the state function during the training process of the EMPPO ship path following control model to improve the policy performance and exploration ability of the model.

[0024] In some embodiments according to the first aspect of the present invention, the input layers of the action network and the value network each have five neurons;

[0025] The five input characteristic data are lateral displacement error, heading angle error, curvature, heading angular velocity and actual rudder angle of the ship;

[0026] The action network output layer has two neurons that respectively output the mean μ and standard deviation σ of the normal distribution and select actions by sampling;

[0027] The value network output layer outputs the value of state s.

[0028] In some embodiments according to the first aspect of the present invention, the deep reinforcement learning reward function formula of the EMPPO ship path following control model is:

[0029]

[0030] Among them, k1, k2, k3 are the adjustment coefficients of rewards, r d is the lateral error reward, r ψ is the heading error reward, r δ Bonus for rudder angle stability.

[0031] In some embodiments of the first aspect of the present invention, the training process of the EMPPO ship path following control model uses the CL algorithm to decompose the training process to accelerate model training, uses PCLIP to limit the update strategy amplitude to maintain the stability of the strategy update, and uses the PPO algorithm to train model parameters.

[0032] In some embodiments according to the first aspect of the present invention, the CL algorithm establishes a three-level progressive training course by introducing a lateral error threshold, a heading error threshold, and a control point random sampling threshold as a basis for difficulty division.

[0033] In some embodiments according to the first aspect of the present invention, the calculation formula of the PCLIP algorithm clipping function is:

[0034]

[0035] Among them, if the first item is less than the second item, the second item is output; if the first item is greater than the third item, the third item is output; otherwise, the first item is output. ε is a hyperparameter with a value range of 0-1. p θ (a t |s t ) is the action probability distribution when the model parameter is θ, The model parameter is θ k The action probability distribution when .

[0036] The end-to-end ship path following control method based on EMPPO reinforcement learning of the present invention has the following beneficial effects: First, it breaks through the reliance on ship motion modeling in traditional path following methods, merges guidance and control into an end-to-end control process, significantly reduces the complexity of control system design, avoids problems such as target point jumping, and makes path following more continuous, smooth and efficient. Secondly, by adding lateral error rewards and heading angle error rewards, the following accuracy is effectively improved. Thirdly, by introducing rudder angle stability constraints in the reward function, the instantaneous change amplitude of the rudder angle is effectively controlled, the vibration phenomenon caused by frequent rudder turns is significantly reduced, and the rudder angle output is smoother. Finally, there is no need for pre-modeling, and real data is used for training. It is applicable to different ship types and complex paths, so that the model has good generalization performance. At the same time, by introducing path curvature and steering gear hysteresis response modeling, the adaptability to high curvature paths and steering gear hysteresis is further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 This is a schematic flow chart of the end-to-end ship path following control method based on the EMPPO reinforcement learning algorithm of the present invention;

[0039] Figure 2This is a schematic diagram of the structure of the ABCFit path generation module of the present invention;

[0040] Figure 3 This is a schematic diagram of the ABCFit algorithm fitting flow chart of the present invention;

[0041] Figure 4 Schematic diagram of the NIShip environmental following error calculation flow chart of the present invention;

[0042] Figure 5 This is a schematic diagram of the EMPPO ship path following control model structure of the present invention;

[0043] Figure 6 Schematic diagram of the EMPPO reinforcement learning action network structure of the present invention;

[0044] Figure 7 This is a schematic diagram of the EMPPO reinforcement learning value network structure of the present invention;

[0045] Figure 8 This is a flow chart of the EMPPO ship path following control model training process described in the present invention. DETAILED DESCRIPTION

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0047] Before further explaining the present invention in detail, the nouns and terms involved in the embodiments of the present invention are explained. The nouns and terms involved in the embodiments of the present invention are subject to the following interpretations:

[0048] PPO (Proximal Policy Optimization) is a popular reinforcement learning algorithm and an improvement on the policy gradient method. It aims to address the drastic policy updates that can occur in traditional policy gradient methods. By introducing a "truncation" mechanism to limit the difference between the old and new policies, it makes the learning process more stable and efficient.

[0049] Actor-Critic is a reinforcement learning algorithm that combines the Policy Gradient method with the Value Function method. It improves learning efficiency and stability by simultaneously training a policy network (the actor) and a value network (the critic). The Actor-Critic algorithm has been widely used in fields such as robotic control, game intelligence, autonomous driving, and autonomous ship navigation.

[0050] The CL (Curriculum Learning) algorithm is a deep learning model training strategy. It aims to mimic the human learning process by setting a sequence of tasks of increasing difficulty, allowing the model to start learning from simple tasks and gradually transition to complex ones. The CL method can significantly improve model training efficiency, preventing the agent from facing overly complex tasks from the outset, and effectively alleviating the difficulty of training reinforcement learning for complex tasks.

[0051] The PCLIP algorithm is an improved version of the CLIP truncation algorithm, used to limit the magnitude of policy updates. It aims to address the problem of the original CLIP truncation algorithm, which restricts the direction of policy optimization by using the same clipping range for both positive and negative advantages. By relaxing clipping for positive advantages, it encourages the use of high-reward actions, while strictly clipping for negative advantages to prevent sudden policy degradation. The PCLIP algorithm combines minimization of the loss function with the advantage function to guide updates while suppressing over-optimization. This approach improves data utilization efficiency while ensuring training stability, balancing reinforcement learning performance with computational cost.

[0052] Figure 1 A flow chart of an end-to-end ship path following control method based on EMPPO reinforcement learning in an embodiment of the present invention is shown, including the following steps:

[0053] Step S1: Construct an ABCFit adaptive channel path smooth fitting algorithm model to fit and generate smooth channel information and simplify the channel path representation parameters.

[0054] Depend on Figure 2 As shown in the figure, the algorithm model used to generate the waterway roadbed includes four parts: waterway data collection, Bezier curve fitting, path segmentation and splicing, and data visualization and export. The specific functions of each part are as follows:

[0055] Channel data collection: Utilize GPS devices, electronic nautical chart systems, and telemetry to capture key location information of the target channel and convert it into precise coordinate data.

[0056] Bezier curve fitting: Using a series of carefully selected control points as input, a continuous and smooth target navigation trajectory is generated using the Bezier curve formula. To ensure a smooth and safe path, the generated path undergoes curvature evaluation and optimization, adjusting the layout of the control points to achieve a smooth transition in path curvature. During this process, areas of high curvature are automatically adjusted to ensure smoothness throughout the path, ensuring the vessel can navigate the path safely and efficiently.

[0057] Path segmentation and splicing: Divide a complex waterway into multiple smaller segments, and apply Bezier curve fitting to each segment. Then, through splicing technology, these segmented curves are combined into a complete navigation path to ensure the continuity and feasibility of the path.

[0058] Data visualization and export: The generated target path data is displayed through professional visualization tools, making it more intuitive and easy to understand. At the same time, this data can also be exported to a standard format for subsequent use in the path tracking control system.

[0059] The Bézier curve is a parametric curve commonly used in computer graphics to generate smooth curves. Developed by French engineer Pierre Bézier in the 1960s for automobile body design, it is widely used in path planning, animation timing functions, font design (such as TrueType and PostScript font formats), and vector drawing software.

[0060] By constructing an inland waterway path dataset, the fitting effect of the path fitting algorithm is verified and tested based on this dataset until a path fitting algorithm that is suitable for the waterway is obtained. Figure 3 As shown in Figure 2, the process of the path fitting algorithm includes the following steps:

[0061] Step S1a: Importing discrete points of the waterway path, importing the discrete point data containing the waterway path information into the algorithm system as basic data for subsequent processing.

[0062] Step S1b: Determine the control points of the cubic Bezier curve, select the starting point of the original path as the first control point of the cubic Bezier curve, and select the ending point of the original path as the last control point of the cubic Bezier curve.

[0063] Step S1c: solving the intermediate control points, using the least squares method, based on the determined first and last control points and discrete point data, calculate and solve the intermediate control points of the cubic Bezier curve.

[0064] Step S1d: Compare the fitting residual with the threshold, calculate the residual of the current fitting curve, and compare it with the manually set threshold.

[0065] Step S1e: Path segmentation processing: if the fitting residual is greater than the set threshold, the reference path is segmented at the path point corresponding to the point with the maximum fitting residual.

[0066] Step S1f: loop the fitting operation, repeating steps S1b to S1e until the fitting residuals at all curve fitting points are less than the threshold.

[0067] Step S1g: Control point adjustment: taking curvature and residual factors into consideration, the control points of the segmented Bezier curves are adjusted to obtain the final curve.

[0068] Through the above steps, a smooth and continuous piecewise Bezier curve is obtained, and the channel path fitting is completed. At this point, the channel path information is input into the adaptive channel path fitting algorithm to obtain the channel path smoothing result and simplified representation parameters;

[0069] Step S2: Construct the NIShip ship motion simulation platform, which can generate ship status information based on the channel path information.

[0070] Real-ship experiments are associated with high risks (such as the possibility of collisions, severe weather, and other dangerous situations), long lead times (the time from experiment preparation to data collection can be long, affected by various factors such as seasonal and hydrological conditions), and high costs (involving high costs such as ship leasing, fuel consumption, equipment maintenance, and staffing). Simulation platforms can circumvent these issues by studying the movement of inland vessels through virtual simulation.

[0071] The construction process of the NIShip ship motion simulation platform includes constructing a first-order Nomoto model and constructing a ship motion model.

[0072] The first-order Nomoto model, proposed by Japanese scholar Nomoto, has been widely used in the study of inland vessel motion. It dynamically describes changes in system state and is used here to characterize the motion characteristics of inland vessels, taking into account the influence of the previous state on the current state during vessel motion, such as the influence of the vessel's position, speed, and heading at the previous moment on the current state. This model can effectively reflect the response characteristics of a vessel at low and medium speeds and is an important tool for studying ship maneuverability and designing control systems. The specific construction process of this model is not particularly limited in this embodiment of the present invention.

[0073] Understandably, inland waterway vessel maneuvering presents unique challenges. For example, the narrow waterways require frequent turns and evasive maneuvers, placing high demands on maneuverability. A ship motion model comprehensively considers factors related to maneuverability, such as the ship's propulsion system, hull hydrodynamic characteristics, and steering gear performance. It uses mathematical equations to describe the ship's motion under the influence of various forces and moments.

[0074] In the embodiment of the present invention, the construction of the ship motion model needs to be based on the maneuvering characteristics of inland ships. To ensure the maneuverability of the model, the construction process needs to conduct Z-shaped experiments and full-turn experiments to continuously verify the model performance.

[0075] Z-shaped and full-turn tests are important components of ship maneuverability testing. The Z-shaped test involves the ship performing a series of turning maneuvers, recording changes in parameters such as heading and speed, to assess the ship's maneuverability and directional stability. The full-turn test involves the ship performing a 360-degree turn, measuring parameters such as turning radius and turning time to verify the ship's performance under complex maneuvers.

[0076] The ship motion model uses the first-order Nomoto model to linearly model inland ships. By inputting the ship motion model and channel path information into the simulation platform, the precise motion information and status information of the ship can be obtained. This information includes the coordinates of the ship, heading and rudder command, etc.

[0077] like Figure 4 As shown in Figure 2, the error calculation module of the NIShip ship motion simulation platform calculates the reference point closest to the ship's current position based on the ship's current position coordinates and the target path model. It then defines and quantifies the heading angle error by analyzing the angle between the tangent direction of this reference point and the ship's heading. Simultaneously, the Euclidean distance between the ship's current position and this reference point is calculated as a metric for the lateral position error. The curvature of the Bezier curve reference point is used as the curvature input for the model.

[0078] At the same time, in order to make full use of the computing power resources of the simulation platform to facilitate model training, in an embodiment of the present invention, the NIShip ship motion simulation platform also includes a reinforcement learning training interface to provide a training environment interface for model training.

[0079] Step S3: Build and train the EMPPO ship path following control model, and deploy it to the target device after training.

[0080] The EMPPO ship path following control model is constructed through a deep reinforcement learning network. In the embodiment of the present invention, the reinforcement learning model is constructed based on the Actor-Critic reinforcement learning algorithm. Figure 5 As shown in Figure 2, the design of the reinforcement learning model includes state space design, action space design, reward function design, and neural network design.

[0081] The state space is the set of environmental information that the agent needs to perceive at each time step. For a ship path following task, the state space may include information such as the current position, target position, heading angle, velocity vector, and environmental parameters. In this embodiment of the present invention, this information is obtained from the ABCFit adaptive channel path smoothing fitting algorithm model and the NIShip ship motion simulation platform. Therefore, the state space must be designed based on the path fitting algorithm and the NIShip simulation platform, where the state includes path state information and ship state information.

[0082] The action space is the set of all possible actions that the agent can perform. For a ship steering agent model, an action could be adjusting the rudder angle, setting the thruster power, etc.

[0083] The design of reward function is the key to improving model performance. The present invention uses the lateral error, heading angle error and steering command change amplitude of the error calculation module as reward and penalty items to improve the model's following control accuracy and reduce the number of steering strokes to reduce the damage of buffeting.

[0084] The reward function formula is designed as follows:

[0085]

[0086] Where k1, k2, and k3 are the adjustment coefficients for each sub-reward, rd is the lateral error reward, rψ is the heading error reward, and rδ is the rudder stability reward. Each reward component is described by a negative exponential function, and the range of each sub-reward is (0, 1).

[0087] Neural networks include action networks and value networks, such as Figure 6 and Figure 7 As shown, the input layers of both the action network and the value network are configured with five neurons, corresponding to the five input feature data: lateral displacement error, heading angle error, curvature, yaw angular velocity, and the ship's actual rudder angle. In this embodiment of the present invention, both the action network and the value network have two fully connected hidden layers, each with 256 neuron parameters, and the activation function uses the Tanh function. The two neurons in the action network's output layer correspond to the mean μ and standard deviation σ of the normal distribution, and actions are then selected through sampling. The output layer of the value network is the value of state s.

[0088] It is understandable that, due to the continuous action space involved, such as adjusting the rudder angle and propeller power, in this embodiment of the present invention, the policy network outputs a probability density function of the action, such as a Gaussian distribution, rather than a specific action value. In this case, the specific action value must be obtained by sampling this distribution. This action value is then input into the ship motion model of the NIShip ship motion simulation platform, which then controls the ship's motion.

[0089] At the same time, the model training process uses the CL algorithm to accelerate model training and uses PCLIP to limit the update strategy amplitude to maintain the stability of the strategy update.

[0090] The CL algorithm introduces three key parameters as the basis for difficulty classification: lateral error threshold, heading error threshold, and control point random sampling threshold. This establishes a three-level progressive training course. Smaller lateral and heading error thresholds increase the course difficulty. A smaller control point random sampling threshold indicates a more stable control point distribution and a lower course difficulty. Specifically, three training courses are set: Course 1, Course 2, and Course 3. For the easy course, the lateral error threshold is set to 30m, the heading error threshold to 30°, and the random threshold to 200m. For the medium course, the lateral error threshold is set to 10m, the heading error threshold to 10°, and the random threshold to 500m. For the difficult course, the lateral error threshold is set to 1m, the heading error threshold to 5°, and the random threshold to 2000m.

[0091] The calculation formula of PCLIP clipping function is as follows:

[0092]

[0093] If the first item is less than the second item, the second item is output; if the first item is greater than the third item, the third item is output; otherwise, the first item is output. ε is a hyperparameter with a value range of 0-1. θ (a t |s t ) is the action probability distribution when the model parameter is θ, The model parameter is θ k The action probability distribution when .

[0094] In an embodiment of the present invention, a proximal policy optimization (PPO) algorithm is used to train model parameters to improve the efficiency of using sample data and obtain a high-precision model.

[0095] like Figure 8As shown in the figure, the EMPPO model training process consists of two major modules: environment interaction and policy update. Any actual navigation path can be decomposed into the end-to-end connection of arcs of varying curvature. Therefore, during training, a sufficient number of arcs of varying curvature are used as the learning environment. The interaction data obtained by interacting with the control model is stored in the experience pool. The advantage function is calculated based on the sampled data in the experience pool. The clipping objective function is defined by combining the probability ratio of the old and current policies to limit the update range and ensure training stability. Furthermore, the mean squared error of the state value function and the policy entropy are added as regularization terms to form a complete optimization objective, which is used to simultaneously improve policy performance and exploration capabilities. In each iteration, multiple rounds of gradient updates are performed using the collected data to optimize the parameters of the policy network and value network. This entire process is repeated, and through continuous policy updates and environment interaction, the policy performance is continuously improved until convergence or the set termination condition is met. Ultimately, the optimized policy model is obtained, completing model training.

[0096] After completing the model training, the trained ship path following control agent model is deployed to the target device.

[0097] Step S4: The EMPPO ship path following control model makes action decisions based on the policy network according to the channel path information and ship status information, and controls the ship to steer in real time to achieve the expected reference path.

[0098] The EMPPO vessel path-following control model relies on its internally constructed policy network to perform complex calculations and decision-making. Using deep reinforcement learning and other techniques, the policy network continuously learns and optimizes the optimal action strategy for various combinations of channel paths and vessel states.

[0099] During actual navigation, the policy network analyzes input channel path information and vessel status information in real time, comprehensively considering various factors and making rapid and precise action decisions. This decision is directly linked to the ship motion model of the NIShip ship motion simulation platform, which enables steering control, specifically by controlling the ship's rudder angle to adjust in real time. Through precise control of the rudder angle, the ship can change its direction promptly and smoothly, continuing to navigate the desired reference path. This achieves high-precision ship path following control, ensuring the ship completes its navigation mission safely and efficiently.

[0100] In summary, firstly, the traditional path following method breaks through the reliance on ship motion modeling, and combines guidance and control into an end-to-end control process, which significantly reduces the complexity of the control system design, avoids problems such as target point jumping, and makes path following more continuous, smooth and efficient. Secondly, by adding lateral error rewards and heading angle error rewards, the following accuracy is effectively improved. Thirdly, by introducing rudder angle stability constraints in the reward function, the instantaneous change amplitude of the rudder angle is effectively controlled, the vibration phenomenon caused by frequent rudder turns is significantly reduced, and the rudder angle output is smoother. Finally, there is no need for pre-modeling, and real data is used for training. It is applicable to different ship types and complex paths, so that the model has good generalization performance. At the same time, by introducing path curvature and servo hysteresis response modeling, the adaptability to high curvature paths and servo hysteresis is further improved. In short, the technical solution of the present invention significantly improves the ship path following control capability in a complex waterway environment, and has important application value and broad promotion prospects.

[0101] The above is a detailed introduction to the method for autonomous navigation and obstacle avoidance of ships that integrates visual perception and nonlinear modeling provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. An end-to-end ship path following control method based on EMPPO reinforcement learning, characterized in that: include: Construct the ABCFit adaptive channel path smooth fitting algorithm model and generate channel path information; Constructing a NIShip ship motion simulation platform, wherein the NIShip ship motion simulation platform generates ship status information according to the waterway path information; Build and train the EMPPO ship path following control model, and deploy it to the target device after training. The EMPPO ship path following control model makes action decisions based on the policy network according to the channel path information and the ship status information, and controls the ship to steer in real time to achieve the expected reference path.

2. The end-to-end ship path following control method based on EMPPO reinforcement learning according to claim 1 is characterized in that: The path fitting algorithm of the ABCFit adaptive channel path smooth fitting algorithm model includes: Import discrete points of waterway paths, and import the discrete point data containing waterway path information into the algorithm system as the basic data for subsequent processing; Determine the control points of the cubic Bezier curve, select the starting point of the original path as the first control point of the cubic Bezier curve, and select the ending point of the original path as the last control point of the cubic Bezier curve; Solve the intermediate control points and use the least squares method to calculate the intermediate control points of the cubic Bezier curve based on the determined first and last control points and discrete point data; Compare the fitting residual with the threshold, calculate the residual of the current fitting curve, and compare it with the manually set threshold; Path segmentation processing: if the fitting residual is greater than the set threshold, the reference path is segmented at the path point corresponding to the maximum fitting residual point; The fitting operation is repeated from step 2 to step 5 until the fitting residuals at all curve fitting points are less than the threshold value; Control point adjustment, taking curvature and residual factors into consideration, adjusts the control points of several segmented Bezier curves to obtain the final curve.

3. The end-to-end ship path following control method based on EMPPO reinforcement learning according to claim 1 is characterized in that: The NIShip ship motion simulation platform includes a ship motion model and a first-order Nomoto model. The ship motion model uses the first-order Nomoto model to linearly model inland ships to obtain the ship's motion and state information, which includes coordinates, heading and steering commands.

4. The end-to-end ship path following control method based on EMPPO reinforcement learning according to claim 3 is characterized in that: The ship motion model is constructed based on the maneuvering characteristics of inland ships, and the maneuverability of the model is verified through Z-shaped experiments and full-turn experiments.

5. The end-to-end ship path following control method based on EMPPO reinforcement learning according to any one of claims 1 to 4, characterized in that: The EMPPO ship path following control model is constructed through a deep reinforcement learning network; The states of the model’s state space include path information and ship state information; The model’s neural network includes an action network and a value network; The reward and penalty items of the model's reward function include lateral displacement error, heading angle error and rudder angle change amplitude.

6. The end-to-end ship path following control method based on EMPPO reinforcement learning according to claim 5 is characterized in that: During the training process of the EMPPO ship path following control model, mean square error and policy entropy are added to the state function to improve the policy performance and exploration ability of the model.

7. The end-to-end ship path following control method based on EMPPO reinforcement learning according to claim 5 is characterized in that: The input layers of the action network and the value network each have five neurons; The five input characteristic data are lateral displacement error, heading angle error, curvature, heading angular velocity and actual rudder angle of the ship; The action network output layer has two neurons that respectively output the mean μ and standard deviation σ of the normal distribution and select actions by sampling; The value network output layer outputs the value of state s.

8. The end-to-end ship path following control method based on EMPPO reinforcement learning according to claim 5 is characterized in that: The deep reinforcement learning reward function formula of the EMPPO ship path following control model is: Among them, k1, k2, k3 are the adjustment coefficients of rewards, r d is the lateral error reward, r ψ is the heading error reward, r δ Bonus for rudder angle stability.

9. The end-to-end ship path following control method based on EMPPO reinforcement learning according to claim 5 is characterized in that: The training process of the EMPPO ship path following control model uses the CL algorithm to decompose the training process to accelerate model training, uses PCLIP to limit the update strategy amplitude to maintain the stability of the strategy update, and uses the PPO algorithm to train model parameters.

10. The end-to-end ship path following control method based on EMPPO reinforcement learning according to claim 9, characterized in that: The CL algorithm introduces lateral error threshold, heading error threshold and control point random sampling threshold as the basis for difficulty division, and establishes a three-level progressive training course.

11. The end-to-end ship path following control method based on EMPPO reinforcement learning according to claim 9, characterized in that: The calculation formula of the PCLIP algorithm clipping function is: Among them, if the first item is less than the second item, the second item is output; if the first item is greater than the third item, the third item is output; otherwise, the first item is output. ε is a hyperparameter with a value range of 0-1. p θ (a t |s t ) is the action probability distribution when the model parameter is θ, The model parameter is θ k The action probability distribution when .

Images