Reinforcement learning ship automatic driving tracking control method based on data driving

Abstract

The invention discloses a reinforcement learning ship automatic driving tracking control method based on data driving. The method comprises the following steps: acquiring a three-degree-of-freedom ship physical model with unknown parameters and uncertain disturbance terms influenced by sea conditions; collecting ship motion data through a Doppler log, and reconstructing a ship data driving model by using a recurrent neural network (RNN) method; on the basis, designing a feedforward controller of the ship by adopting a backstepping method; designing a ship optimal feedback controller by adopting self-adaptive dynamic programming based on a radial basis function neural network RBFNN (Radial Basis Function Neural Network); and through combination of the two controllers, ship automatic driving tracking control based on data driving is realized. The problems that interpretability and stability of a physical model are reserved, ship model parameters are difficult to accurately obtain, the obtaining speed is low and cost is high are solved, innovation of a ship navigation system is promoted, reinforcement learning ship track tracking automatic driving based on data driving is achieved by means of feedforward and optimal feedback control design, and the performance of the ship navigation system is improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent ship technology, and in particular to a data-driven reinforcement learning-based method for autonomous ship tracking and control. Background Technology

[0002] The widespread application of electronic information technology in shipbuilding has provided a technological foundation for the development of autonomous ship navigation. In particular, the continuous advancements in sensor technology and the Internet of Things (IoT) enable ships to automatically perceive and acquire information and data about themselves and the marine environment. Therefore, integrating this acquired information and data into existing technological systems will undoubtedly drive the innovation and upgrading of ship navigation systems.

[0003] In existing technologies, most rely on physical models of ships to achieve autonomous driving. However, the parameters of physical models are difficult to obtain accurately, and the acquisition process is lengthy and costly, which greatly restricts further research on ship navigation systems. Although a few studies have achieved autonomous driving solely through data-driven methods, these methods suffer from high dependence on data quality, poor interpretability, and limited generalization ability. Therefore, combining physical models of ships with data-driven methods to achieve precise and efficient autonomous driving has significant research value.

[0004] Meanwhile, complex sea conditions such as wind, waves, and currents pose severe challenges to ship navigation. To ensure safe and efficient navigation, ships must possess robust autopilot capabilities. Optimal control theory, as an advanced control concept dedicated to maximizing rewards at minimal cost, plays a crucial role in AI-based control design, providing solid theoretical support and technical assurance for the intelligent transformation of ship navigation systems. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies that largely rely on physical models of ships to achieve autonomous navigation. However, the parameters of these physical models are difficult to obtain accurately, and the acquisition process is lengthy and costly, which greatly restricts further research into ship navigation systems. Although a few studies have achieved autonomous navigation solely through data-driven methods, these approaches suffer from high dependence on data quality, poor interpretability, and limited generalization capabilities.

[0006] To achieve the above objectives, this invention provides a data-driven reinforcement learning-based method for automated ship tracking and control, comprising: S1: Obtain a three-degree-of-freedom ship physical model with unknown parameters and uncertain disturbance terms affected by sea state, including: kinematic model and dynamic model; S2: Based on the three-degree-of-freedom physical model of the ship, a data-driven model of the ship is constructed by using the ship motion data collected by the Doppler log and combining it with the RNN method. S3: Based on the ship's data-driven model, the ship's automatic driving feedforward controller is designed using the backstepping method; S4: Based on the feedforward controller, define the cost function and Hamilton-Jacobi-Bellman HBJ equation, and design the optimal feedback controller for ship autopilot. S5: Based on the optimal feedback controller, the cost function is approximated using RBFNN to obtain the optimal estimated feedback controller; S6: Obtain the final controller based on the optimal estimation feedback controller combined with the feedforward controller.

[0007] In the preferred embodiment, step S1 includes: S11: Obtain a three-degree-of-freedom ship physical model with unknown parameters and uncertain disturbance terms affected by sea state. The formulas for the kinematic and dynamic models are as follows: In the formula, The position coordinates of the ship in an inertial coordinate system on a fixed Earth surface. and bow angle The position vector is composed of x and y. x represents the x-coordinate of the coordinate system, in meters (m), and y represents the y-coordinate of the coordinate system, in meters (m). The forward speed of the ship in the attached coordinate system Horizontal drift speed and bow roll rate The velocity vector is formed by collecting ship motion data. For controlling the propulsion of ships, input forward force. Lateral drift force and bow roll torque The control vector is composed of; The lateral disturbance forces on a ship caused by wind, waves, and currents in the attached coordinate system. Longitudinal interference force and bow disturbance torque The external environment disturbance vector is composed of, and satisfies ; express The upper bound of a positive constant; This is the coordinate system transformation matrix. and satisfy ,in Represents a real matrix with row i and column j; The matrix consists of the ship's weight inertia and hydrodynamic additional inertia. ; For Coriolis matrix, ; The linear hydrodynamic damping parameter matrix, ; matrix They are represented in the following forms respectively: In the formula: Unknown inertial parameters representing the ship's own weight, added mass, and hydrodynamic derivatives, including gravimetric inertia and hydrodynamic added inertia; This represents the unknown linear hydrodynamic damping coefficient. .

[0008] In the preferred embodiment, step S2 includes: S21: Based on a three-degree-of-freedom ship physical model, obtain the nominal form of the dynamic system, the formula is: Then, based on the Stone-Weilstrasse ship theorem, the nominal form of the ship's dynamics system can be rewritten using the RNN method as follows: In the formula, Represents the ideal RNN weights, And satisfy This represents the error between the ideal RNN weights and the estimated RNN weights. This represents the estimation of RNN weights. This represents the RNN reconstruction error. The activation function of a monotonically increasing RNN is defined as follows: In the formula, Represents the independent variable of the activation function. Represents positive numbers; S22: Based on the estimated RNN weights Given a defined data-driven model, the ship's approximate dynamics data-driven model is: In the formula, The state of the data-driven model is represented by data collected from ship motion data and satisfies the following conditions: This represents the error between the actual model state and the data-driven model state, obtained by subtracting the collected motion data. This represents the feedback term that compensates for the reconstruction error, and satisfies the formula: In the formula, Indicates the design parameters of the reconstructed model. , This represents an estimated value of the adjustable parameter. ; S23: Regarding state error Differentiating, we get: In the formula, This represents an ideal adjustable parameter that satisfies... , This represents the error between the ideal adjustable parameter and the estimated adjustable parameter. ; S24: Based on the derivative of the model state error Combining Lyapunov stability analysis strategy, an adaptive law for estimating RNN weights and variable parameters is constructed. The adaptive law is used to obtain the estimated weights and variable parameters of the RNN. The formula for the adaptive law of RNN weight estimation is: In the formula, This represents the learning rate of the RNN. Represents the update rate of the variable parameter, and the estimated value of the adjustable parameter. The formula for the adaptive law is: In the formula, Indicates the variable parameter update rate. ; S25: Combine the ship physical model from step S11 with the ship approximate dynamics data-driven model from step S22 to construct a three-degree-of-freedom ship data-driven model, as shown in the formula: In the formula, This represents the stable weights of an RNN, i.e. , This represents the error between the RNN reconstruction error, the external environmental perturbation vector, and the RNN's ideal and stable weights. The parameters of the composition ; express A positive constant with an upper bound, and satisfying .

[0009] In the preferred embodiment, step S3 includes: S31: Based on the ship's three-degree-of-freedom data-driven model, the controller formula is defined as follows: In the formula, It is a feedforward controller. For feedback controller, ; The tracking error of a ship is defined by the following formula: In the formula, Indicates the track reference signal, Indicates the virtual controller for autonomous driving. This indicates the track error between the designed track of the ship and the track reference signal. This indicates the speed error between the designed forward speed, drift speed, and bow roll rate of a ship and the desired forward speed, drift speed, and bow roll rate. S32: Differentiating the ship tracking track error and combining it with the ship kinematics data-driven model from step S25, we can obtain the following form: In the formula: express The first derivative; construct the first Lyapunov function, the formula is: Based on the derivative of the first Lyapunov function Combining the Lyapunov stability analysis strategy, a virtual controller for the ship is designed, using the following formula: In the formula: Indicates the control design parameters, ; S33: To facilitate the implementation of optimal feedback control design, based on the speed error in step S31, define: In the formula, ; Based on the controller definition in step S31 and the derivative of the ship speed error, combined with the ship dynamics data-driven model in step S25, the following form can be obtained: In the formula: express The first derivative; S34: Construct the second Lyapunov function based on the first Lyapunov function, using the following formula: Based on the derivative of the second Lyapunov function Combining the Lyapunov stability analysis strategy, a ship feedforward controller is designed, with the following formula: In the formula: Indicates the control design parameters, ; Substituting the derivative of the second Lyapunov function into the ship feedforward controller, we get: .

[0010] In the preferred embodiment, step S4 includes: S41: According to step S34 The last item is the formula for the ship's automated driving tracking optimization control system: S42: Define the cost function of the autonomous driving tracking optimization control system. The formula is: In the formula, Let represent a positive definite matrix, and ; Let represent a positive definite matrix, and t represents time; S43: Define the Hamiltonian function based on the cost function, the formula is: In the formula: express about The partial derivative; express Transpose of; S44: Minimize the cost function of the automated driving tracking optimization control system in step S42 to obtain the optimal cost function, as shown in the formula: S45: Based on the optimal cost function, obtain the HJB equation, the formula is: In the formula: express about The partial derivative; S46: Based on the HJB equation, the optimal feedback controller is obtained using the gradient descent method. The formula for the optimal feedback controller is: .

[0011] In the preferred embodiment, step S5 includes: S51: The optimal cost function is approximated using RBFNN. The formula for approximating the optimal cost function is as follows: In the formula, Let represent the desired RBFNN weights, and satisfy . This represents the error in the weights of the neural network. This represents the estimation of the weights in a neural network. Represents the activation function of a neural network. This indicates the number of neurons in the hidden layer. This represents the approximation error of the neural network, relating to the cost function used to approximate the optimal value. Find the partial derivative to obtain the partial derivative of the optimal cost function. The formula is: In the formula, express about The partial derivative, express about The partial derivative; S52: Based on the estimated neural network weights The evaluation network is determined, and the approximate optimal cost function is obtained. The formula is as follows: For the approximate optimal cost function with respect to Find the partial derivative to obtain the partial derivative of the approximately optimal cost function. The formula is: S53: Based on the partial derivative of the optimal cost function from step S51, substitute it into the optimal feedback controller and the HJB function, and the formula is: In the formula, Let represent the system reconstruction residual, and: Similarly, based on the partial derivative of the approximate optimal cost function obtained in step S52, the estimated optimal feedback controller and the estimated HJB equation are obtained, as shown in the following formulas: S54: Based on the estimated optimal feedback controller Substituting into step S41 of the ship's automated driving tracking optimization control system, the formula is: S55: To minimize the estimated HJB equation, the gradient descent method is used to design the RBFNN adaptive law, the formula of which is: In the formula, This represents the learning rate of the neural network. This indicates the adjustment parameters for the additional stabilization term. Let Lyapunov be a continuously differentiable function.

[0012] In the preferred embodiment, step S6 includes: S61: Combining the estimations of the feedforward controller from step S34 and the optimal feedback controller from step S53, the final controller of the ship is obtained, using the following formula: In the formula, The RNN weights represent ship speed data, including forward speed, collected using a Doppler log. Horizontal drift speed and bow roll rate The model error is obtained, and then substituted into the adaptive law of the estimated weights in step S24 to calculate the model error. This represents the derivative of the virtual controller. Indicates the design parameters, Let represent a positive definite matrix, and , and Represents the tangent function; S62: Based on the ship's final controller and the RBFNN adaptive law of step S55, realize automatic ship trajectory tracking.

[0013] The beneficial effects of this invention are as follows: By using a three-degree-of-freedom ship physical model with unknown parameters and uncertain disturbance terms affected by sea state, and a set of collected ship motion data, a data-driven model of the ship is constructed using the recurrent neural network (RNN) method. This not only preserves the interpretability and stability of the physical model, but also effectively avoids the problems of difficult, slow, and costly acquisition of ship model parameters in existing technologies, greatly promoting the upgrading of ship navigation systems. Based on this ship model-driven approach, the feedforward controller adopts a backstepping method design, and the optimal feedback controller adopts an adaptive dynamic programming design based on radial basis function neural networks (RBFNN), solving the problem of solving the nonlinear Milton-Jacobi-Bellman HJB equations. Finally, the combination of these two controllers not only ensures that the ship's autopilot can efficiently track the desired autopilot, but also improves the performance of the ship's navigation system. Attached Figure Description

[0014] Figure 1 This is a flowchart of the data-driven reinforcement learning-based ship autopilot tracking control method of the present invention; Figure 2 This is a schematic diagram illustrating the model error between the actual model state and the data-driven model state in this invention. Figure 3 The ship's position coordinates in this invention Simulation diagram of the tracking effect; Figure 4 For the ships in this invention, respectively Simulation results of tracking the reference signal above; Figure 5 For the ship in this invention Simulation diagram of tracking error; Figure 6 This is a simulation diagram showing the convergence result of the RBFNN weights in this invention; Figure 7 This is a simulation diagram of the effects of the ship's forward force, lateral drift force, and bow roll moment controller in this invention. Detailed Implementation

[0015] Example 1: This embodiment also includes step S7: using Lyapunov theory, proving the stability of the input to the state of the designed data-driven reinforcement learning-based ship autopilot tracking control system, and that all signals in the closed-loop system are eventually uniformly bounded.

[0016] Specifically, in order to more easily prove stability, the following assumptions are made: Assumption 1: RNN reconstruction error It is bounded, and the upper and lower bounds are the model error. The function, i.e., the expression, is: .

[0017] Assumption 2: Consider the ship's autopilot tracking optimization control system Cost function and optimal feedback controller ,definition It is a continuously differentiable Lyapunov function that satisfies the following requirements: Furthermore, there exists a positive definite matrix. Make: In the formula, Representation matrix The smallest eigenvalue; Assumption 3: In compact sets superior, and All are bounded, satisfying , All of them are positive numbers.

[0018] Theorem 1: For a strictly feedback-based ship autopilot tracking control system, based on Lyapunov theory, under the action of the designed RNN data-driven model, RNN adaptive law, feedforward controller, optimal feedback controller, and RBFNN adaptive law, by selecting appropriate parameters, it can be guaranteed that all signals of the closed-loop system are eventually uniformly bounded, and the tracking error converges in an optimal manner.

[0019] Lemma 1: (Young's inequality) For any The following inequalities hold: In the formula, and ; Proof of Theorem 1: S71: To prove that the RNN data-driven model and the RNN weights are eventually uniformly bounded, construct the Lyapunov function. And the Lyapunov function expression is: For Lyapunov functions Differentiating, we get: S72: Based on the derivative of the model state error in step S23 and the derivative of the RNN weight error in step S24 and adjustable parameter derivative Substituting the derivative of the Lyapunov function have to: Based on the RNN activation function inequality in step S21, Assumption 1, and Lemma 1 (Young's inequality), the following inequality can be obtained: Substituting the derivative of the Lyapunov function We can obtain: In the formula:

[0020] Therefore, when choosing the appropriate The value satisfies hour, This ensures that the RNN data-driven model and RNN weights are eventually uniformly bounded.

[0021] S73: Next, it is proved that the data-driven ship autopilot tracking control system and the RBFNN weights are eventually uniformly bounded, based on the Hamiltonian function obtained in step S53. Therefore, considering Then, substitute the above formula into the RBFNN adaptive law in step S55. We can obtain: S74: Based on step S34 Constructing Lyapunov functions The expression is: For Lyapunov functions Taking the derivative, we get: S75: Rewrite the automated driving tracking control optimization system to include an optimal controller. form Thus obtain ; Based on steps S11 and S25, and assumption 1, the parameter d satisfies: S76: Based on the above formula and assumptions 2 and 3, the derivative of the Lyapunov function is further obtained. for: In the formula, All represent positive numbers, where: Therefore, when the formula is satisfied: By selecting appropriate parameters, all signals of a data-driven ship autopilot tracking control system can eventually become uniformly bounded. The advantage lies in combining a three-degree-of-freedom ship physical model with unknown parameters and uncertain disturbances influenced by sea state with collected ship motion data, and employing a recurrent neural network (RNN) method to construct a data-driven ship model. This not only preserves the interpretability and stability of the physical model but also effectively avoids the problems of difficult, slow, and costly acquisition of ship model parameters in existing technologies, greatly promoting innovation in ship navigation systems. This invention provides a data-driven model-based ship autopilot tracking control method that divides the controller into a feedforward controller and an optimal feedback controller. Compared to general controller design methods, this method can design optimal feedback controllers for second-order and higher-order models, resulting in good tracking and control performance of the ship's autopilot system. Simulation experiments were conducted on the MATLAB platform, and an adaptive optimization controller was designed to achieve ship autopilot tracking. The system controller is more aligned with maritime practice, further verifying the effectiveness and rationality of the data-driven reinforcement learning-based ship autopilot tracking control method provided by this invention.

[0022] Example 2: To verify the effectiveness of the data-driven reinforcement learning-based ship autopilot tracking control method proposed in this invention, this embodiment conducts a computer simulation study on MATLAB, with the following parameter settings: The simulation object selected is the CyberShip II (CS2) at the Marine Control Laboratory of the Norwegian University of Science and Technology. This ship is a 1:70 scale replica of a supply ship, weighing 23.8 kg, with a length of 1.255 m and a beam of 0.29 m. When designing the CS2 data-driven model, the following parameters were set: , , , RNN initial weights For the corresponding dimension, the zero matrix, , Setting the ideal of unmanned autonomous driving as The final controller designed achieves control of CS2, and its design parameters... RBFNN initial weights The activation function is Center point selection The initial value of the ship control system is , .

[0023] Figure 2The model error is the difference between the actual model state and the data-driven model state. It can be seen that the model error has successfully converged to a bounded neighborhood centered at the origin. Figure 3 To demonstrate the trajectory tracking performance in the x0y plane, the results show that the output position signal can quickly track the reference signal. Figure 4 This visually demonstrates the tracking effect from three dimensions: position and heading. Figure 5 To track the error trajectory curve, the curve trend shows that the tracking steady-state error can be stably converged to below 0.01. Figure 6 The graph shows the convergence process of the RBFNN weights, where all weights converge to the ideal weights. Figure 7 To control the input variation curve, the control signal is relatively stable. Simulation results verify the effectiveness of the data-driven ship autopilot tracking control system constructed in this invention, and show that the controller designed based on this control system can effectively achieve ship autopilot tracking, with the tracking error quickly converging to a small residual set. This fully demonstrates that the output of the data-driven ship autopilot navigation system has good tracking performance, further verifying the effectiveness and rationality of the data-driven reinforcement learning-based ship autopilot tracking control method proposed in this invention.

[0024] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data-driven reinforcement learning-based method for tracking and controlling automated ship navigation, characterized in that, include: S1: Obtain a three-degree-of-freedom ship physical model with unknown parameters and uncertain disturbance terms affected by sea state, including: kinematic model and dynamic model; S2: Based on the three-degree-of-freedom physical model of the ship, a data-driven model of the ship is constructed by using the ship motion data collected by the Doppler log and combining it with the RNN method. S3: Based on the ship's data-driven model, the ship's automatic driving feedforward controller is designed using the backstepping method; S4: Based on the feedforward controller, define the cost function and Hamilton-Jacobi-Bellman HBJ equation, and design the optimal feedback controller for ship autopilot. S5: Based on the optimal feedback controller, the cost function is approximated using RBFNN to obtain the optimal estimated feedback controller; S6: Obtain the final controller based on the optimal estimation feedback controller combined with the feedforward controller.

2. The data-driven reinforcement learning-based ship autopilot tracking control method according to claim 1, characterized in that, Step S1 includes: S11: Obtain a three-degree-of-freedom ship physical model with unknown parameters and uncertain disturbance terms affected by sea state. The formulas for the kinematic and dynamic models are as follows: In the formula, The position coordinates of the ship in an inertial coordinate system on a fixed Earth surface. and bow angle The position vector is composed of x and y. x represents the x-coordinate of the coordinate system, in meters (m), and y represents the y-coordinate of the coordinate system, in meters (m). The forward speed of the ship in the attached coordinate system Horizontal drift speed and bow roll rate The velocity vector is formed by collecting ship motion data. For controlling the propulsion of ships, input forward force. Lateral drift force and bow roll torque The control vector is composed of; The lateral disturbance forces on a ship caused by wind, waves, and currents in the attached coordinate system. Longitudinal interference force and bow disturbance torque The external environment disturbance vector is composed of, and satisfies ; express The upper bound of a positive constant; This is the coordinate system transformation matrix. and satisfy ,in Represents a real matrix with row i and column j; The matrix consists of the ship's weight inertia and hydrodynamic additional inertia. ; For Coriolis matrix, ; The linear hydrodynamic damping parameter matrix, ; matrix They are represented in the following forms respectively: In the formula: Unknown inertial parameters representing the ship's own weight, added mass, and hydrodynamic derivatives, including gravimetric inertia and hydrodynamic added inertia; This represents the unknown linear hydrodynamic damping coefficient. .

3. The data-driven reinforcement learning-based ship autopilot tracking control method according to claim 1, characterized in that, Step S2 includes: S21: Based on a three-degree-of-freedom ship physical model, obtain the nominal form of the dynamic system, the formula is: Then, based on the Stone-Weilstrasse ship theorem, the nominal form of the ship's dynamics system can be rewritten using the RNN method as follows: In the formula, Represents the ideal RNN weights, And satisfy This represents the error between the ideal RNN weights and the estimated RNN weights. This represents the estimation of RNN weights. This represents the RNN reconstruction error. The activation function of a monotonically increasing RNN is defined as follows: In the formula, Represents the independent variable of the activation function. Represents positive numbers; S22: Based on the estimated RNN weights Given a defined data-driven model, the ship's approximate dynamics data-driven model is: In the formula, The state of the data-driven model is represented by data collected from ship motion data and satisfies the following conditions: This represents the error between the actual model state and the data-driven model state, obtained by subtracting the collected motion data. This represents the feedback term that compensates for the reconstruction error, and satisfies the formula: In the formula, Indicates the design parameters of the reconstructed model. , This represents an estimated value of the adjustable parameter. ; S23: Regarding state error Differentiating, we get: In the formula, This represents an ideal adjustable parameter that satisfies... , This represents the error between the ideal adjustable parameter and the estimated adjustable parameter. ; S24: Based on the derivative of the model state error Combining Lyapunov stability analysis strategy, an adaptive law for estimating RNN weights and variable parameters is constructed. The adaptive law is used to obtain the estimated weights and variable parameters of the RNN. The formula for the adaptive law of RNN weight estimation is: In the formula, This represents the learning rate of the RNN. Represents the update rate of the variable parameter, and the estimated value of the adjustable parameter. The formula for the adaptive law is: In the formula, Indicates the variable parameter update rate. ; S25: Combine the ship physical model from step S11 with the ship approximate dynamics data-driven model from step S22 to construct a three-degree-of-freedom ship data-driven model, as shown in the formula: In the formula, This represents the stable weights of an RNN, i.e. , This represents the error between the RNN reconstruction error, the external environmental perturbation vector, and the RNN's ideal and stable weights. The parameters of the composition ; express A positive constant with an upper bound, and satisfying .

4. The data-driven reinforcement learning-based ship autopilot tracking control method according to claim 1, characterized in that, Step S3 includes: S31: Based on the ship's three-degree-of-freedom data-driven model, the controller formula is defined as follows: In the formula, It is a feedforward controller. For feedback controller, ; The tracking error of a ship is defined by the following formula: In the formula, Indicates the track reference signal, Indicates the virtual controller for autonomous driving. This indicates the track error between the designed track of the ship and the track reference signal. This indicates the speed error between the designed forward speed, drift speed, and bow roll rate of a ship and the desired forward speed, drift speed, and bow roll rate. S32: Differentiating the ship tracking track error and combining it with the ship kinematics data-driven model from step S25, we can obtain the following form: In the formula: express The first derivative; construct the first Lyapunov function, the formula is: Based on the derivative of the first Lyapunov function Combining the Lyapunov stability analysis strategy, a virtual controller for the ship is designed, using the following formula: In the formula: Indicates the control design parameters, ; S33: To facilitate the implementation of optimal feedback control design, based on the speed error in step S31, define: In the formula, ; Based on the controller definition in step S31 and the derivative of the ship speed error, combined with the ship dynamics data-driven model in step S25, the following form can be obtained: In the formula: express The first derivative; S34: Construct the second Lyapunov function based on the first Lyapunov function, using the following formula: Based on the derivative of the second Lyapunov function Combining the Lyapunov stability analysis strategy, a ship feedforward controller is designed, with the following formula: In the formula: Indicates the control design parameters, ; Substituting the derivative of the second Lyapunov function into the ship feedforward controller, we get: 。 5. The data-driven reinforcement learning-based ship autopilot tracking control method according to claim 1, characterized in that, Step S4 includes: S41: According to step S34 The last item is the formula for the ship's automated driving tracking optimization control system: S42: Define the cost function of the autonomous driving tracking optimization control system. The formula is: In the formula, Let represent a positive definite matrix, and ; Let represent a positive definite matrix, and t represents time; S43: Define the Hamiltonian function based on the cost function, the formula is: In the formula: express about The partial derivative; express Transpose of; S44: Minimize the cost function of the automated driving tracking optimization control system in step S42 to obtain the optimal cost function, as shown in the formula: S45: Based on the optimal cost function, obtain the HJB equation, the formula is: In the formula: express about The partial derivative; S46: Based on the HJB equation, the optimal feedback controller is obtained using the gradient descent method. The formula for the optimal feedback controller is: 。 6. The data-driven reinforcement learning-based ship autopilot tracking control method according to claim 1, characterized in that, Step S5 includes: S51: The optimal cost function is approximated using RBFNN. The formula for approximating the optimal cost function is as follows: In the formula, Let represent the desired RBFNN weights, and satisfy . This represents the error in the weights of the neural network. This represents the estimation of the weights in a neural network. Represents the activation function of a neural network. This indicates the number of neurons in the hidden layer. This represents the approximation error of the neural network, relating to the cost function used to approximate the optimal value. Find the partial derivative to obtain the partial derivative of the optimal cost function. The formula is: In the formula, express about The partial derivative, express about The partial derivative; S52: Based on the estimated neural network weights The evaluation network is determined, and the approximate optimal cost function is obtained. The formula is as follows: For the approximate optimal cost function with respect to Find the partial derivative to obtain the partial derivative of the approximately optimal cost function. The formula is: S53: Based on the partial derivative of the optimal cost function from step S51, substitute it into the optimal feedback controller and the HJB function, and the formula is: In the formula, Let represent the system reconstruction residual, and: Similarly, based on the partial derivative of the approximate optimal cost function obtained in step S52, the estimated optimal feedback controller and the estimated HJB equation are obtained, as shown in the following formulas: S54: Based on the estimated optimal feedback controller Substituting into step S41 of the ship's automated driving tracking optimization control system, the formula is: S55: To minimize the estimated HJB equation, the gradient descent method is used to design the RBFNN adaptive law, the formula of which is: In the formula, This represents the learning rate of the neural network. This indicates the adjustment parameters for the additional stabilization term. Let Lyapunov be a continuously differentiable function.

7. The data-driven reinforcement learning-based ship autopilot tracking control method according to claim 1, characterized in that, Step S6 includes: S61: Combining the estimations of the feedforward controller from step S34 and the optimal feedback controller from step S53, the final controller of the ship is obtained, using the following formula: In the formula, The RNN weights represent ship speed data, including forward speed, collected using a Doppler log. Horizontal drift speed and bow roll rate The model error is obtained, and then substituted into the adaptive law of the estimated weights in step S24 to calculate the model error. This represents the derivative of the virtual controller. Indicates the design parameters, Let represent a positive definite matrix, and , and Represents the tangent function; S62: Based on the ship's final controller and the RBFNN adaptive law of step S55, realize automatic ship trajectory tracking.

Images