Intelligent guidance method suitable for water jet propulsion ship

Abstract

The invention relates to an intelligent guidance method suitable for a water jet propulsion ship, and the method comprises the following steps: 1, determining a tracking path trajectory of the water jet propulsion ship, and setting path points; and step 2, designing intelligent guidance of tracking self-adaptive control of the water jet propulsion ship. And step 3, calculating the next spray pump rotation angle by using the expected first direction angle solved in the step 2 and combining a PID control method to reach the next path point position, and performing the next path point circulation in the step 1 until the whole expected trajectory is completed. The method solves the problem that a ship path tracking control method is difficult to effectively and intelligently guide and control in a complex environment, can be applied to the field of ship motion control, and can design a self-adaptive course control law, resist environmental interference, eliminate ship transverse steady-state deviation and accurately track a target trajectory.

Description

Technical Field

[0001] The present invention relates to the technical field of motion control of water jet propulsion ships, and specifically to an intelligent guidance and path tracking adaptive control method of water jet propulsion ships in waves based on deep reinforcement learning. Background Art

[0002] Ship path tracking control is to design a path tracking controller for ships under different control requirements, so that the ship can track and finally stabilize on the predetermined reference path under the action of the controller. The ship has the characteristics of strong nonlinearity, large time lag, large inertia, and the influence of external environmental interference, which makes it more difficult to design a path tracking controller. The most commonly used method for ship path tracking control is PID heading control combined with guidance method. The guidance method is to convert the path into the desired heading, and the purpose of path tracking can be achieved by adding PID heading control. Although the conventional guidance combined with PID control method has a simple structure, the control accuracy depends on the initial set parameters, especially when the ship encounters different wave conditions, the parameters need to be manually adjusted, which cannot meet the requirements of ship intelligent navigation control. Therefore, the combination of intelligent guidance and control of ships in complex environments is a key issue that urgently needs to be solved in improving the level of ship intelligence.

[0003] Combining deep learning and reinforcement learning, Silver et al. proposed the deep reinforcement learning (DRL) technology. Because the DRL algorithm can adaptively change strategies without human intervention, and does not require a clear model, it can learn directly from the environment, etc., and is currently widely used in robot control, drone flight control and other aspects. Summary of the invention

[0004] Aiming at the problem that the ship path tracking control method is difficult to effectively guide and control in complex environments, an intelligent guidance and path tracking method suitable for water jet propulsion ships is proposed.

[0005] The purpose of the present invention is to provide an intelligent guidance method for a waterjet ship based on deep reinforcement learning, which can realize the intelligent determination of the desired heading angle of the waterjet ship in waves. The obtained desired heading angle will be used for adaptive control of the ship's path tracking, so that the ship can track the desired path.

[0006] The technical solution of the present invention is:

[0007] An intelligent guidance method applicable to a water jet propulsion ship comprises the following steps:

[0008] Step 1: Determine the tracking path trajectory of the water jet propulsion vessel and set the path points;

[0009] Step 2: Design intelligent guidance for waterjet propulsion ship tracking adaptive control;

[0010] Step 2.1: Using the current path point and the next path point, the desired angle of the first shot is preliminarily determined by using the LOS method based on the line of sight guidance method;

[0011] Step 2.2: Use the DRL-based intelligent guidance method to calculate the desired prime angle, resist different environmental interferences, and make the ship sail accurately at the current desired prime angle;

[0012] Step 3: Use the desired heading angle solved by step 2, combined with the PID control method, to calculate the next jet angle, reach the next path point position, and perform the next path point cycle of step 1 until the entire desired trajectory is completed.

[0013] Further, the specific steps are as follows:

[0014] Step 1: Determine the tracking path trajectory of the water jet propulsion vessel and set the path points;

[0015] P0=[x,y] Τ is the current position of the ship, P k-1 =[x k-1 ,y k-1 ] Τ is the route point at the previous moment, P k =[x k ,y k ] Τ is the route point at the current moment;

[0016] Step 2: Design intelligent guidance for waterjet propulsion ship tracking adaptive control;

[0017] Step 2.1: Preliminarily determine the expected prime minister angle based on the current path point and the next path point based on the LOS method:

[0018] Δ is the foresight distance, P los =[x los ,y los ] Τ is the LOS position at the current moment, y e P0 is the distance from the expected trajectory P k-1 P k The vertical distance, i.e. the current lateral deviation of the ship, γ p is the angle between the path line and the coordinate axis; for straight line trajectory tracking, the expected heading angle based on the LOS method

[0019]

[0020] Step 2.2: Use the intelligent guidance method based on deep reinforcement learning to calculate the expected prime angle, resist different environmental interferences, and make the ship sail accurately at the current expected prime angle;

[0021] First, establish a water jet propulsion ship motion simulation model:

[0022]

[0023] Where m is the mass of the ship, m x 、m y is the longitudinal and transverse additional mass of the hull, I z is the moment of inertia about the z-axis, J z is the additional inertia moment about the z axis, u, v, r are the longitudinal, transverse and bow angular velocities of the ship, Corresponding to the response acceleration, T is the thrust of the water jet pump, δ is the rotation angle of the water jet pump, R is the navigation resistance of the hull, τ Xw , τ Yw , τ Nw is the corresponding environmental wave force, X vv , X vr , X rr , Y v , Y r , Y v|v| , Y vvr , Y vrr , Y r|r| 、N v 、N r 、N v|v| 、N vvr 、N vrr 、N r|r| are the corresponding hydrodynamic derivatives respectively;

[0024] A DRL agent is established based on the waterjet ship motion simulation model, that is, an intelligent system that learns autonomously and takes actions in an environment to maximize its cumulative reward; it includes the environment: the external environment with which the agent interacts; the environment is defined as the waterjet ship motion simulation model; the environment receives the action calculated by the agent at each time step and returns a new state and reward; the state: the description of the motion state of the waterjet ship at a specific time point; the action: the behavior selected by the agent at each time step; the action space defines all possible action sets of the agent; the reward: the feedback signal given by the environment after the agent performs a certain action; the policy: the rule or function that the agent decides which action to choose in each state; the PPO algorithm is used to update the neural network; the specific definition of the DRL agent is as follows:

[0025] Action: Prime Minister Angle Change Angle is the output action, and the range is [-π,π]; since the prime angle calculated by LOS cannot adapt to the influence of different wave environments, the expected prime angle is calculated in LOS guidance On this basis, an intelligent variation is superimposed

[0026] State: 10-dimensional variable The heading angle error at time t is e(t): is the current heading angle, which is one of the states because the purpose is heading angle control. Velocity and acceleration are one of the states because they can help the DRL agent reflect the wave force.

[0027] Reward R:

[0028]

[0029] n represents the total number of steps in a trajectory, e min Defined as:

[0030]

[0031] First, based on the ship's current state heading angle ψ and the desired heading angle ψ d The difference plus the prime angle deviation of the DRL agent action output The sum of the state is used as feedback information. The state state is obtained through the environment and passed to the DRL agent. The reward value is calculated by the state based on formula (3) and is also passed to the DRL agent to judge whether the current state is feasible. After that, the DRL agent will issue an action instruction, which is The action given by the DRL agent is received, and the current heading angle error e(t) obtained in the environment is obtained; based on the PID control as shown in formula (5), the current jet angle rudder variable Δδ(t) is calculated, and the jet angle instruction is issued to the agent environment, after which the ship enters the next state state; (state, reward, action) These three items form a transition; All transitions are stored in the DRL agent and used by the PPO algorithm to update network parameters; After the network training is completed, a set of intelligent guidance methods based on deep reinforcement learning are formed to calculate the expected heading angle method PPO-Guidance;

[0032] The PID control calculation formula is:

[0033]

[0034] In the formula, K p , K i and Kd is the PID control parameter;

[0035] Step 3: Use the desired heading angle solved by step 2, combined with the PID control method, to calculate the next jet angle, reach the next path point position, and perform the next path point cycle of step 1 until the entire desired trajectory is completed.

[0036] Further, generally Δ = 3 ~ 5L pp , L pp It's the captain.

[0037] Furthermore, in step 2.1, the desired heading angle is also the sight angle.

[0038] Furthermore, in step 2.2, the reward is a scalar that represents the immediate benefit of the action.

[0039] Furthermore, in step 2.2, the agent’s goal is to maximize its cumulative reward by selecting actions.

[0040] Furthermore, in step 2.2, in deep reinforcement learning, the policy is usually represented by a deep neural network.

[0041] Furthermore, in step 1, e min The initial value is set to 0.

[0042] Furthermore, this method can be used for automatic berthing of water jet propulsion ships, water quality detection of unmanned boats, and target tracking scenarios.

[0043] Furthermore, this method can design an adaptive heading control law to resist the interference of different wave conditions; eliminate the lateral steady-state deviation of the ship and accurately track the target trajectory.

[0044] The beneficial effects of the present invention are:

[0045] 1. The water jet propulsion ship intelligent guidance and track tracking adaptive control method based on deep reinforcement learning described in this patent can be applied to the field of ship motion control. It can design adaptive heading control laws to resist the interference of different wave conditions; eliminate the lateral steady-state deviation of the ship and accurately track the target trajectory.

[0046] 2. The intelligent guidance and path tracking adaptive control method of waterjet ships in waves based on deep reinforcement learning can enable the waterjet ships to accurately track the target trajectory. This method can be used in scenarios such as automatic berthing of waterjet ships, water quality detection of unmanned boats, and target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a framework diagram of the intelligent guidance of a water jet propulsion ship under changing wind and wave environment based on deep reinforcement learning of the present invention. DETAILED DESCRIPTION

[0048] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0049] The present invention is a method for intelligent guidance and path tracking adaptive control of a water jet propulsion ship in waves based on deep reinforcement learning, and the steps include:

[0050] Step 1: Determine the tracking path trajectory of the water jet propulsion vessel and set the path points.

[0051] P0=[x,y] Τ is the current position of the ship, P k-1 =[x k-1 ,y k-1 ] Τ is the route point at the previous moment, P k =[x k ,y k ] Τ The route point at the current time.

[0052] Step 2: Design intelligent guidance for waterjet propulsion ship tracking adaptive control;

[0053] Step 2.1: Preliminarily determine the expected prime minister angle based on the current path point and the next path point based on the LOS method:

[0054] Δ is the forward sight distance (generally Δ=3~5L pp , L pp is the captain), P los =[x los ,y los ] Τ is the LOS position at the current moment, y e P0 is the distance from the expected trajectory P k-1 P k The vertical distance, i.e. the current lateral deviation of the ship, γ p For linear trajectory tracking, the expected heading angle (i.e., line of sight angle) based on the LOS method is

[0055]

[0056] Step 2.2: Use the intelligent guidance method based on deep reinforcement learning to calculate the expected prime angle, resist different environmental interferences, and make the ship sail accurately at the current expected prime angle.

[0057] First, establish a water jet propulsion ship motion simulation model:

[0058]

[0059] Where m is the mass of the ship, m x 、m y is the longitudinal and transverse additional mass of the hull, I z is the moment of inertia about the z-axis, J zz is the additional inertia moment about the z axis, u, v, r are the longitudinal, transverse and bow angular velocities of the ship, Corresponding to the response acceleration, T is the thrust of the water jet pump, δ is the rotation angle of the water jet pump, R is the navigation resistance of the hull, τ Xw , τ Yw , τ Nw is the corresponding environmental wave force, X vv , X vr , X rr , Y v , Y r , Y v|v| , Y vvr , Y vrr , Y r|r| 、N v 、N r 、N v|v| 、N vvr 、N vrr 、N r|r| are the corresponding hydrodynamic derivatives respectively.

[0060] A DRL agent is established based on the waterjet ship motion simulation model, that is, an intelligent system that learns autonomously and takes actions in an environment to maximize its cumulative rewards. It includes an environment: the external environment with which the agent interacts. In the present invention, the environment is defined as a waterjet ship motion simulation model. The environment receives the action calculated by the agent at each time step and returns a new state and reward. State: a description of the motion state of the waterjet ship at a specific point in time. Action: the behavior selected by the agent at each time step. The action space defines the set of all possible actions of the agent. Reward: the feedback signal given by the environment after the agent performs an action. The reward is a scalar that represents the immediate benefit of the action. The goal of the agent is to maximize its cumulative reward by selecting actions. Policy: the rule or function that the agent decides which action to choose in each state. In deep reinforcement learning, the policy is usually represented by a deep neural network, and the PPO algorithm is used in the present invention to update the neural network. The specific DRL agent is defined as follows:

[0061] Action: Prime Minister Angle Change Angle is the output action, and the range is [-π,π]. Since the prime angle calculated by LOS cannot adapt to the influence of different wave environments, the expected prime angle is calculated in LOS guidance. On this basis, an intelligent variation is superimposed

[0062] State: 10-dimensional variable The heading angle error at time t is e(t): is the current heading angle, as one of the states because the purpose is heading angle control, speed and acceleration are as one of the states because they can help the DRL agent reflect the wave force.

[0063] Reward R:

[0064]

[0065] n represents the total number of steps in a trajectory, e min The initial value is set to 0 and is defined as:

[0066]

[0067] like Figure 1 As shown, first, based on the ship's current state heading angle ψ and the desired heading angle ψ d The difference plus the prime angle deviation of the DRL agent action output The sum of the state is used as feedback information. The state state is obtained through the environment and passed to the DRL agent. The reward value is calculated based on the state based on formula (3) and is also passed to the DRL agent to judge whether the current state is feasible. After that, the DRL agent will issue an action instruction, which is The action given by the DRL agent is received, and based on the current heading angle error e(t) obtained in the environment. Based on PID control (Formula (5)), the current jet angle rudder variable Δδ(t) is calculated, and the jet angle instruction is issued to the agent environment, after which the ship enters the next state state. The three items (state, reward, action) form a transition. All transitions are stored in the DRL agent and used by the PPO algorithm to update network parameters. After the network training is completed, a set of intelligent guidance methods based on deep reinforcement learning is formed to calculate the expected prime angle method (PPO-Guidance).

[0068] The PID control calculation formula is:

[0069]

[0070] In the formula, Kp , K i and K d is the PID control parameter.

[0071] Step 3: Use the desired heading angle solved by step 2, combined with the PID control method, to calculate the next jet angle, reach the next path point position, and perform the next path point cycle of step 1 until the entire desired trajectory is completed.

[0072] The contents not described in detail in the specification belong to the prior art known to ordinary technicians in this field.

[0073] The above-mentioned embodiment only expresses one implementation mode of the present invention, and its description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

Claims

1. An intelligent guidance method suitable for a water jet propulsion ship, characterized in that: The following steps are involved: Step 1: Determine the tracking path trajectory of the water jet propulsion vessel and set the path points; Step 2: Design intelligent guidance for waterjet propulsion ship tracking adaptive control; Step 2.1: Preliminarily determine the desired prime minister angle based on the current path point and the next path point, based on the LOS method, i.e., the line of sight guidance method; Step 2.2: Use the DRL-based intelligent guidance method to calculate the desired prime angle, resist different environmental interferences, and make the ship sail accurately at the current desired prime angle; Step 3: Use the desired heading angle solved by step 2, combined with the PID control method, to calculate the next jet angle, reach the next path point position, and perform the next path point cycle of step 1 until the entire desired trajectory is completed.

2. The intelligent guidance method for a water jet propulsion vessel according to claim 1, characterized in that: The specific steps are as follows: Step 1: Determine the tracking path trajectory of the water jet propulsion vessel and set the path points; P0=[x,y] Τ is the current position of the ship, P k-1 =[x k-1 ,y k-1 ] Τ is the route point at the previous moment, P k =[x k ,y k ] Τ is the route point at the current moment; Step 2: Design intelligent guidance for waterjet propulsion ship tracking adaptive control; Step 2.1: Preliminarily determine the expected prime minister angle based on the current path point and the next path point based on the LOS method: Δ is the foresight distance, P los =[x los ,y los ] Τ is the LOS position at the current moment, y e P0 is the distance from the expected trajectory P k-1 P k The vertical distance, i.e. the current lateral deviation of the ship, γ p is the angle between the path line and the coordinate axis; for straight line trajectory tracking, the expected heading angle based on the LOS method Step 2.2: Use the intelligent guidance method based on deep reinforcement learning to calculate the expected prime angle, resist different environmental interferences, and make the ship sail accurately at the current expected prime angle; First, establish a water jet propulsion ship motion simulation model: Where m is the mass of the ship, m x 、m y is the longitudinal and transverse additional mass of the hull, I z is the moment of inertia about the z-axis, J z is the additional inertia moment about the z axis, u, v, r are the longitudinal, transverse and bow angular velocities of the ship, Corresponding to the response acceleration, T is the thrust of the water jet pump, δ is the rotation angle of the water jet pump, R is the navigation resistance of the hull, τ Xw , τ Yw , τ Nw is the corresponding environmental wave force, X vv , X vr , X rr , Y v , Y r , Y vv , Y vvr , Y vrr , Y rr 、N v 、N r 、N vv 、N vvr 、N vrr 、N rr are the corresponding hydrodynamic derivatives respectively; A DRL agent is established based on the waterjet ship motion simulation model, that is, an intelligent system that autonomously learns and takes actions in an environment to maximize its cumulative rewards; it includes the environment: the external environment that the agent interacts with; the environment is defined as the waterjet ship motion simulation model; the environment receives the action calculated by the agent at each time step and returns a new state and reward; the state: the description of the motion state of the waterjet ship at a specific time point; the action: the behavior selected by the agent at each time step; The action space defines the set of all possible actions of the agent; reward: the feedback signal given by the environment after the agent performs an action; policy: the rule or function that the agent uses to decide which action to choose in each state; the PPO algorithm is used to update the neural network; the specific DRL agent is defined as follows: Action: Prime Minister Angle Change Angle is the output action, and the range is [-π,π]; since the prime angle calculated by LOS cannot adapt to the influence of different wave environments, the expected prime angle is calculated in LOS guidance On this basis, an intelligent variation is superimposed State: 10-dimensional variable The heading angle error at time t is the current heading angle, which is one of the states because the purpose is heading angle control. Velocity and acceleration are one of the states because they can help the DRL agent reflect the wave force. Reward R: n represents the total number of steps in a trajectory, e min Defined as: First, based on the ship's current state heading angle ψ and the desired heading angle ψ d The difference plus the prime angle deviation of the DRL agent action output The sum of the state is used as feedback information. The state state is obtained through the environment and passed to the DRL agent. The reward value is calculated by the state based on formula (3) and is also passed to the DRL agent to judge whether the current state is feasible. After that, the DRL agent will issue an action instruction, which is The action given by the DRL agent is received, and the current heading angle error e(t) obtained in the environment is obtained; based on the PID control as shown in formula (5), the current jet angle rudder variable Δδ(t) is calculated, and the jet angle instruction is issued to the agent environment, after which the ship enters the next state state; (state, reward, action) These three items form a transition; All transitions are stored in the DRL agent and used by the PPO algorithm to update network parameters; After the network training is completed, a set of intelligent guidance methods based on deep reinforcement learning are formed to calculate the expected heading angle method PPO-Guidance; The PID control calculation formula is: In the formula, K p , K i and K d is the PID control parameter; Step 3: Use the desired heading angle solved by step 2, combined with the PID control method, to calculate the next jet angle, reach the next path point position, and perform the next path point cycle of step 1 until the entire desired trajectory is completed.

3. The intelligent guidance method for a water jet propulsion vessel according to claim 1, characterized in that: In step 1, generally L pp It's the captain.

4. The intelligent guidance method for a water jet propulsion vessel according to claim 1, characterized in that: In step 2.1, the desired heading angle is also the line of sight angle.

5. The intelligent guidance method for a water jet propulsion vessel according to claim 1, characterized in that: In step 2.2, reward is a scalar that represents the immediate benefit of the action.

6. The intelligent guidance method for a water jet propulsion vessel according to claim 1, characterized in that: In step 2.2, the agent’s goal is to maximize its cumulative reward by selecting actions.

7. The intelligent guidance method for a water jet propulsion vessel according to claim 1, characterized in that: In step 2.2, in deep reinforcement learning, the policy is usually represented by a deep neural network.

8. The intelligent guidance method for a water jet propulsion vessel according to claim 1, characterized in that: In step 1, e min The initial value is set to 0.

9. The intelligent guidance method for a water jet propulsion vessel according to claim 1, characterized in that: This method can be used for automatic berthing of water jet propulsion ships, water quality detection of unmanned boats, and target tracking scenarios.

10. The intelligent guidance method for a water jet propulsion vessel according to claim 1, characterized in that: This method can design an adaptive heading control law to resist the interference of different wave conditions; eliminate the lateral steady-state deviation of the ship and accurately track the target trajectory.

Images