An unmanned ship automatic berthing control method based on reinforcement learning

Abstract

This invention proposes an automatic berthing control method for unmanned surface vessels (USVs) based on reinforcement learning. A Markov decision process model is constructed based on the direct berthing method to establish the USV's state and action spaces. The method also establishes the action-to-state transition process and task reward function based on the ship's mathematical model. The number of training rounds and the time step per round are set according to the berthing environment, generating the USV's berthing pose point in each round. The SAC algorithm is used to train the neural network model to obtain the optimal parameters, i.e., the dynamic positioning controller. The desired berthing pose point is set, and the method is tested to see if the dynamic positioning controller can output the ship's thrust magnitude based on the input state information until the berthing pose point is reached under interference conditions. This invention simplifies the design process of the automatic berthing control method for USVs and has advantages such as fast response speed, strong robustness, and high positioning accuracy.

Description

Technical Field

[0001] This invention belongs to the field of unmanned vessel motion control technology, specifically relating to an automatic berthing control method for unmanned vessels based on reinforcement learning. Background Technology

[0002] Unmanned surface vehicles (USVs) are intelligent platforms that navigate autonomously or semi-autonomously on the water surface using onboard sensors. They can replace humans in performing dangerous and arduous tasks on the water and play an important role in both military and civilian fields. Automated berthing is one of the key aspects of ensuring the safe and efficient navigation of ships and is also an important component of realizing fully automated USVs.

[0003] During berthing, ships navigate at low speeds and are affected by factors such as the shore wall effect, wind, waves, and currents. Furthermore, the uncertainties inherent in low-speed models pose significant challenges to berthing control. Unmanned surface vessel (USV) automated berthing systems are crucial for achieving remote, unmanned operation.

[0004] Unmanned surface vessel (USV) automated berthing primarily employs two methods depending on the propulsion configuration: external berth stabilization and direct berthing. External berth stabilization requires setting a desired course, enabling the vessel to stabilize at a safe distance of 1.5 times its length from the dock along this virtual course. Direct berthing simulates tugboat berthing, first guiding the vessel to the berthing area and then using its dynamic positioning system to move the vessel and achieve berthing. Since the desired course in external berth stabilization is not universally applicable and difficult to implement in engineering, this patent chooses the direct berthing method.

[0005] Patent CN113110468A proposes a control method for automatic berthing of underactuated twin-propeller, twin-rudder ships. The method includes: a ship motion model parameter identification step based on the extended Kalman filter method, which incorporates frequent misalignment and reversing during the actual berthing process into the identified motion model structure, achieving adaptive correction of the ship motion model parameters during berthing and navigation; a model predictive control step and a PID control step, utilizing nonlinear model predictive control and PID control techniques to achieve path planning and tracking control during the berthing process. Specifically, the route planned using nonlinear model predictive control considers the ship's nonlinear motion characteristics and the influence of actual environmental and obstacle factors. PID control overcomes the limitations of slow solution speed and long solution cycle of nonlinear model predictive control, and overcomes the shortcomings of nonlinear model predictive control methods in generating control commands due to changes in model parameters and environmental interference within the nonlinear model prediction cycle. However, the computational complexity is high, making real-time performance difficult to guarantee.

[0006] Patent CN108267955A proposes an automatic berthing control method for underactuated unmanned surface vessels (USVs). The steps are as follows: determine the current control system mode based on the USV's current berthing state; obtain the USV's actual speed or heading, and acquire the control deviation and rate of change of the heading or speed; use this as input to a fuzzy controller, select appropriate control parameter changes based on the current control system mode, and update the control parameters; use the error as controller input, and the controller outputs the desired control command to the actuator. This method improves upon the traditional PID controller by dividing it into two modes—a remote mode and a near-shore mode—and incorporating adaptive fuzzy control rules to dynamically and adaptively adjust control parameters according to changes in berthing behavior. This addresses the motion control challenges of underactuated USVs autonomously berthing under conditions of weak maneuverability, large disturbances, and strong shoreline effects. However, the design lacks systematicity; even with adaptive parameter tuning, it is limited by the expressive power of manually established mathematical and environmental models, making it difficult to guarantee control accuracy.

[0007] The aforementioned motion control methods face numerous challenges in complex marine environments, including extensive parameter tuning, computational complexity, real-time requirements, and a lack of self-learning capabilities. Consequently, they struggle to guarantee control accuracy and response speed in such environments. Summary of the Invention

[0008] Purpose of the invention: This invention proposes an automatic berthing control method for unmanned vessels based on reinforcement learning. It does not rely on prior knowledge or precise mathematical models, does not require complex calculations, is suitable for applications in uncertain environments, and has self-learning capabilities. It also has advantages such as fast response speed, strong robustness, and high positioning accuracy.

[0009] Technical solution: The present invention provides an automatic berthing control method for unmanned vessels based on reinforcement learning, comprising the following steps:

[0010] (1) A Markov decision process model is constructed based on the direct berthing method to establish the state space and action space of the unmanned vessel; and the change process from action to state and the task reward function are established based on the ship's mathematical model.

[0011] (2) Based on the berthing environment, set the number of training rounds and the time step per round, and generate the berthing pose point of the unmanned vessel in each round.

[0012] (3) The SAC algorithm is used to train the neural network model to obtain the optimal parameters of the neural network model, i.e., the dynamic positioning controller;

[0013] (4) Set the desired berthing position and test whether the dynamic positioning controller can output the ship's thrust through input status information until the berthing position is reached under interference conditions.

[0014] Further, step (1) includes the following steps:

[0015] (11) Construct a Markov decision process model <S,A,P,R>;

[0016] (12) Set up the state space of the unmanned vessel as the input to the controller:

[0017] S = <N,E,ψ,u,v,r,N> e E e ,ψ e > (1)

[0018] Wherein, N,E,ψ,u,v,r,N e E e ,ψ e These represent the unmanned vessel's northeast position, heading, pitch speed, sway speed, bow roll speed, northeast position error, and heading error, respectively.

[0019] (12) Set the motion space of the unmanned vessel as the output of the controller:

[0020] A = <τ X ,τ Y ,τ N > (2)

[0021] Where, τ X ,τ Y ,τ N These represent the magnitudes of the unmanned vessel's pitch, sway, and bow thrust, respectively.

[0022] (13) Set up the process of unmanned vessel's actions to state changes for interaction with the environment:

[0023]

[0024] Where P represents the probability that the unmanned vessel will transition to the next state after taking an action in this state;

[0025] (14) Set up rewards for unmanned vessels to guide them in completing control tasks:

[0026]

[0027] Where R represents the total reward obtained by the unmanned vessel per second, which includes error reward, speed reward, and position limitation reward.

[0028] Furthermore, the implementation process of step (3) is as follows:

[0029] (31) Initialize two Critic networks and Actor Network π θ Initialize two Critic target networks The experience pool is initialized to empty. Where φ1 and φ2 are two Critic networks. Network parameters, θ is the Actor network π θ parameter, For Critic target network Parameter, D is the experience pool, Indicates the empty set;

[0030] (32) Initialize [τ] X ,τ Y ,τ N Both [u,v,r] are [0,0,0] T And based on the set number of rounds, the desired berthing pose point η is randomly generated each round. d ; where τ X ,τ Y ,τ N These are the three control forces of a ship: pitch, sway, and yaw; u, v, and r are the ship's pitch speed, sway speed, and yaw speed, respectively.

[0031] (33) In the current state, select an action based on the current strategy, and store the executed action, the reward obtained, and the next state information of the unmanned vessel into the experience pool. t ,a t ,r(s t ,a t ),s t+1};

[0032] (34) Take N tuples from the experience pool, and calculate the objective function for each tuple using the Critic target network according to the following formula:

[0033]

[0034] Where, r i γ represents the reward received by the unmanned vessel at each moment, and γ is the discount rate. The action value function is calculated for the target network, where α is the regularization coefficient used to control the degree of exploration by the algorithm, and logπ is the value of the action value. θ (a i+1 |s i+1 ) represents the degree of randomness of policy π in state s;

[0035] (35) Update the two Critic networks by minimizing the loss function:

[0036]

[0037] in, The action value function calculated for the Critic network is used to update the Critic network parameters via the policy gradient method; the action value is calculated using the following formula:

[0038]

[0039] Wherein, V(s) t+1 () is the state value function, used to evaluate the quality of a state under a given policy; the state value calculation formula is:

[0040]

[0041] Where, -αlogπ(a t |s t ) and αH(π(·|s t The fact that they are equal indicates the degree of randomness of policy π in state s;

[0042] (36) Sample actions using the reparameterization technique and update the current Actor network using the Actor loss function; for environments with continuous action spaces, the output policy is the mean and standard deviation of a Gaussian distribution. Using the reparameterization technique, the policy function is expressed as:

[0043]

[0044] in, The adoption action is defined using the reparameterization technique, where ξ is a noisy random variable. It follows a unit Gaussian distribution;

[0045] (37) Update the entropy regularization coefficient according to the entropy loss function:

[0046]

[0047] Among them, when the entropy of the policy is lower than the target value H O This will increase the value of α; higher than the target H O When this happens, the value of α decreases, and the trained strategy becomes more focused on improving value.

[0048] (38) Update target network parameters:

[0049]

[0050] Where τ represents the soft update coefficient, For the Critic target network parameters, φ i＝1,2 These are the parameters for the Critic network.

[0051] (39) By repeating (32)-(38) in the set number of rounds, the optimal parameters φ1,φ2,θ of the model are obtained, which is the dynamic positioning controller.

[0052] Furthermore, the desired berthing pose points described in step (4) are four consecutive points.

[0053] Furthermore, the Actor network loss function described in step (36) is:

[0054]

[0055] The policy function π was obtained using KL divergence. θ The loss function is used to update the parameters θ through the policy gradient method.

[0056] Beneficial effects: Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention obtains a method for automatic berthing by training a dynamic positioning controller through reinforcement learning. It does not rely on prior knowledge and precise mathematical models, does not require complex calculations, is suitable for applications in uncertain environments, and has self-learning capabilities. For automatic berthing in unknown environments, the present invention uses the SAC algorithm as the implementation of the dynamic positioning controller, which improves the robustness to unknown interference and enhances the ability of unmanned vessels to accurately position themselves. Attached Figure Description

[0057] Figure 1 This is a flowchart of the present invention;

[0058] Figure 2 This is a block diagram of an automatic berthing control system for unmanned vessels based on reinforcement learning.

[0059] Figure 3 This is the training convergence graph based on the SAC algorithm in this invention;

[0060] Figure 4 This is a diagram showing the test results of the controller in this invention. Detailed Implementation

[0061] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings.

[0062] This invention provides an automatic berthing control method for unmanned surface vessels (USVs) based on reinforcement learning. This method achieves automatic berthing by training a dynamic positioning controller using reinforcement learning. It does not rely on prior knowledge or precise mathematical models, requires no complex calculations, is suitable for applications in uncertain environments, and possesses self-learning capabilities. For automatic berthing in unknown environments, the SAC algorithm is used as the implementation of the dynamic positioning controller, improving robustness to unknown disturbances and enhancing the USV's precise point-to-point positioning capabilities. Figure 1 As shown, the specific steps include the following:

[0063] Step 1: Construct a Markov Decision Process (MDP) model based on the direct berthing method. This model can be represented by a simple quadruple <S,A,P,R> to establish the state space and action space of the unmanned vessel; and establish the change process from action to state and the task reward function based on the ship's mathematical model.

[0064] Step 1.1: Set up the state space of the unmanned vessel as the input to the controller:

[0065] S = <N,E,ψ,u,v,r,N> e E e ,ψ e > (1)

[0066] In equation (1), N,E,ψ,u,v,r,N e E e ,ψ e These represent the unmanned vessel's northeast position, heading, pitch speed, sway speed, bow roll speed, northeast position error, and heading error, respectively. For ease of training and subsequent parameter tuning, the states are set between [-1] and 1.

[0067] Step 1.2: Set the motion space of the unmanned vessel as the output of the controller:

[0068] A = <τ X ,τ Y ,τ N > (2)

[0069] In equation (2), A represents the magnitude of the sway, roll, and bow thrust of the unmanned vessel, respectively. The action is also set between [-1 to 1], and the actual thrust magnitude is amplified on the right side of the dynamic equation.

[0070] Step 1.3 Set up the process of the unmanned vessel's actions to states for interaction with the environment:

[0071]

[0072] In equation (3), P represents the probability that the unmanned vessel will transition to the next state after taking an action in this state. However, since it is model-free reinforcement learning, the probability distribution of this part is unknown. It is only necessary to describe the change process from action to state based on the kinematic and dynamic models of the vessel.

[0073] Mathematical equations for a ship's three degrees of freedom:

[0074]

[0075] In equation (4), η = [x, y, ψ] T Let ν represent the position and attitude in the northeast coordinate system, R(ψ) be the coordinate transformation matrix, and ν = [u, v, r].T Let M ∈ R be the ship's speed. 3×3 and D∈R 3×3 These are the inertia matrix and damping matrix, respectively, and τ represents the ship's control force and torque. env Environmental interference force.

[0076] Step 1.4 Set up rewards for the unmanned vessel to guide it in completing control tasks;

[0077]

[0078] In equation (5), R represents the total reward R obtained by the unmanned vessel per second, which includes error reward, speed reward, and position limitation reward.

[0079] Error reward is:

[0080]

[0081] In Equation (6), based on the method of direct approach of the vessel, negative rewards are given for heading error and near north or east error, and positive rewards are given for north or east direction at a greater distance as active guidance to ensure that the unmanned vessel can directly adjust its attitude to berth.

[0082] At the same time, speed was limited during the movement, with the ship's speed not exceeding 0.5 m / s.

[0083]

[0084] In equation (7), V represents the ship's combined speed. Since berthing is done at low speed, the speed is limited to meet the actual situation.

[0085] The platform map serves as the boundary for the sea area; exceeding this boundary incurs a heavier penalty.

[0086]

[0087] Formula (8) specifies the sea area in which the unmanned vessel can move. When it touches the sea area boundary, the mission is considered to have failed, and the next round will be carried out directly with a heavier penalty value.

[0088] Step 2: Based on the berthing environment, set the number of training rounds and the time step per round, and generate the berthing pose point of the unmanned vessel in each round.

[0089] The environment was set with 4000 rounds, each round corresponding to 1200 time steps. In each round, the desired berthing pose point η was generated within the map area. d The unmanned boat conducts exploration training within a few steps.

[0090] Step 3: Use the SAC algorithm to train the neural network model and obtain the optimal parameters of the neural network model, i.e., the dynamic positioning controller.

[0091] The SAC algorithm is used to train the dynamic positioning controller. Training requires setting the neural network and algorithm hyperparameters. The Actor and Critic in the SAC algorithm adopt an MLP network structure, both containing two hidden layers with 128 neurons each. Hyperparameter settings: discount rate 0.99, batch size 256, experience pool size 1e6, learning rate 2e-4, initial regularization coefficient 0.2.

[0092] The specific training steps are as follows:

[0093] Step 3.1: Initialize two Critic networks and Actor Network π θ Initialize two Critic target networks The experience pool is initialized to empty. Where φ1 and φ2 are two Critic networks. Network parameters, θ is the Actor network π θ parameter, For Critic target network Parameter, D is the experience pool, This represents the empty set.

[0094] Step 3.2: Initialize [τ] X ,τ Y ,τ N Both [u,v,r] are [0,0,0] T And based on the set number of rounds, the desired berthing pose point η is randomly generated each round. d ; where τ X ,τ Y ,τ N These are the three control forces of a ship: pitch, sway, and yaw; u, v, and r are the ship's pitch speed, sway speed, and yaw speed, respectively.

[0095] Step 3.3 In the current state, select an action based on the current strategy, and store the executed action, the reward obtained, and the next state information of the unmanned vessel into the experience pool. t ,a t ,r(s t ,a t ),s t+1}

[0096] Step 3.4: Take N tuples from the experience pool, and for each tuple, use the Critic target network to calculate the objective function according to the following formula:

[0097]

[0098] In equation (9), r i The γ discount rate represents the reward earned by the unmanned vessel at each moment. The action value function is calculated for the target network. Since there are two objective functions to calculate, the smaller one will be chosen to avoid overestimation. α is a regularization coefficient used to control and adjust the degree of algorithm exploration. logπ θ (a i+1 |s i+1 ) represents the degree of randomness of policy π in state s.

[0099] Step 3.5: Update the two Critic networks using the minimum loss function, as shown in the following formula:

[0100]

[0101] In equation (10), The action value function calculated for the Critic network is used to update the Critic network parameters using the policy gradient method.

[0102] The value of an action is calculated using the following formula:

[0103]

[0104] In equation (11), V(s) t+1 Let be the state value function, used to evaluate the quality of a state under a given policy. The formula for calculating the state value is:

[0105]

[0106] In equation (12), -αlogπ(a t |s t ) and αH(π(·|s t The two values ​​are equal, both indicating the degree of randomness of policy π in state s.

[0107] Step 3.6: Sample actions using reparameterization techniques and update the current Actor network using the Actor loss function. However, for environments with continuous action spaces, the output policy is the mean and standard deviation of a Gaussian distribution, requiring the use of reparameterization techniques. The policy function can be expressed as:

[0108]

[0109] In equation (13), The adoption action is defined using the reparameterization technique, where ξ is a noisy random variable. It follows a unit Gaussian distribution.

[0110] Actor network loss function:

[0111]

[0112] In equation (14), the strategy function π is obtained according to the MSBE formula. θ The loss function is updated using the policy gradient method to update the parameters θ.

[0113] Step 3.7: Update the entropy regularization coefficients according to the entropy loss function.

[0114] The SAC algorithm training requires different amounts of entropy in different states, and the corresponding loss function is as follows:

[0115]

[0116] In equation (15), when the entropy of the strategy is lower than the target value H O This will increase the value of α; higher than the target H O When this happens, the value of α decreases, and the trained strategy becomes more focused on improving value.

[0117] Step 3.8: Update the target network parameters using the following formula:

[0118]

[0119] In equation (16), τ represents the soft update coefficient. For the Critic target network parameters, φ i＝1,2 These are the parameters for the Critic network.

[0120] By repeating steps 3.2-3.8 above for a set number of rounds, the optimal parameters φ1, φ2, θ of the model are obtained, which is the dynamic positioning controller.

[0121] Step 4: Set four consecutive desired berthing positions and test whether the dynamic positioning controller can output the ship's thrust magnitude based on the input status information in an interference environment until the berthing position is reached.

[0122] Figure 2 The diagram shows the block diagram of an unmanned surface vessel (USV) automatic berthing control system based on reinforcement learning, where the DRL is a trained dynamic positioning controller. By setting the desired berthing pose, the motion network in the dynamic positioning controller, based on the state information input, outputs the magnitude of the control force to resist interference and reach the desired berthing pose.

[0123] Figure 3The diagram shown is a convergence plot of the training process. To facilitate observation, the plot has been processed using a moving average. In a training run of 4000 rounds, the algorithm began to converge after the 1000th round, indicating that the unmanned vessel had begun to learn to reach the desired pose point and maintained a high reward in the following 3000 rounds without significant fluctuations, indicating relatively stable training.

[0124] Figure 4 The image shows the test results. During the test, wind, waves, and current interference were added, and four desired pose points were set in sequence. The unmanned vessel was able to resist the interference and reach the four points in sequence, which shows that the controller has strong robustness and high positioning accuracy.

Claims

1. A method for automatic berthing control of unmanned vessels based on reinforcement learning, characterized in that, Includes the following steps: (1) Construct a Markov decision process model based on the direct berthing method to establish the state space and action space of the unmanned vessel; and establish the change process from action to state and the task reward function based on the ship's mathematical model; (2) Based on the berthing environment, set the number of training rounds and the time step per round, and generate the berthing pose point of the unmanned vessel in each round; (3) The SAC algorithm is used to train the neural network model to obtain the optimal parameters of the neural network model, i.e., the dynamic positioning controller; (4) Set the desired berthing position and test whether the dynamic positioning controller can output the ship's thrust magnitude by inputting state information in an interference environment until the berthing position is reached. Step (1) includes the following steps: (11) Construct a Markov decision process model ; (12) Set up the state space of the unmanned vessel as the input to the controller: (1) in, These represent the unmanned vessel's northeast position, heading, pitch speed, sway speed, bow roll speed, northeast position error, and heading error, respectively. (12) Set the motion space of the unmanned vessel as the output of the controller: (2) in, These represent the magnitudes of the unmanned vessel's pitch, sway, and bow thrust, respectively. (13) Set up the process of the unmanned vessel's actions to state changes for interaction with the environment: (3) in, This represents the probability that the unmanned vessel will transition to the next state after taking an action in this state. (14) Set up rewards for unmanned vessels to guide them in completing control tasks: (4) in, This represents the total reward obtained by the unmanned surface vessel per second, including error reward, speed reward, and position limitation reward; the error reward is: (6) Based on the method of direct approach of the vessel, negative rewards are given for heading errors and relatively close north or east errors, while positive rewards are given for more distant north or east directions as active guidance to ensure that the unmanned vessel can directly adjust its attitude to berth.

2. The automatic berthing control method for unmanned vessels based on reinforcement learning as described in claim 1, characterized in that, The implementation process of step (3) is as follows: (31) Initialize two Critic networks , and Actor Network Initialize two Critic target networks , The experience pool is initialized to empty. ;in and For two Critic networks , Network parameters, For Actor Network parameter, For Critic target network , parameter, For experience pool, Indicates the empty set; (32) Initialization and All And based on the set number of rounds, the desired berthing pose is randomly generated each round. ;in, These are the three control forces of a ship: pitching, swaying, and bow rolling. These are the ship's pitch speed, sway speed, and bow roll speed; (33) In the current state, select an action according to the current strategy, and store the executed action, the reward obtained, and the next state information of the unmanned vessel into the experience pool. ; (34) Take N tuples from the experience pool, and calculate the objective function for each tuple using the Critic target network according to the following formula: (9) in, The reward that the unmanned ship receives at each moment, For discount rate, The action value function calculated for the target network. The regularization coefficient is used to control the degree of exploration by the adjustment algorithm. Indicating in strategy In state The degree of randomness; (35) Update the two Critic networks by minimizing the loss function: (10) in, The action value function calculated for the Critic network is used to update the Critic network parameters via the policy gradient method; the action value is calculated using the following formula: (11) in, This is the state value function, used to evaluate the quality of a state under this policy; State value calculation formula: (12) in, and Equal, both indicate that in strategy In state The degree of randomness; (36) Sample actions using the reparameterization technique and update the current Actor network using the Actor loss function; for environments with continuous action spaces, the output policy is the mean and standard deviation of a Gaussian distribution. Using the reparameterization technique, the policy function is expressed as: (13) in, For the adopted action after using the reparameterization technique, Let it be a noisy random variable. It follows a unit Gaussian distribution; (37) Update the entropy regularization coefficients according to the entropy loss function: (15) Among them, when the entropy of the strategy is lower than the target value It will increase The value is higher than the target. When, it will decrease The value, and thus the training strategy focuses more on improving value; (38) Update the target network parameters: (16) in, Indicates the soft update coefficient. For Critic target network parameters, These are the parameters for the Critic network. (39) By repeating (32)-(38) in the set number of rounds, the optimal parameters of the model are obtained. This yields the dynamic positioning controller.

3. The automatic berthing control method for unmanned vessels based on reinforcement learning as described in claim 1, characterized in that, The desired berthing pose points described in step (4) are four consecutive points.

4. The automatic berthing control method for unmanned vessels based on reinforcement learning as described in claim 2, characterized in that, The loss function of the Actor network mentioned in step (36) is: (14) The policy function was obtained using KL divergence. The loss function is updated using the policy gradient method. .

Images