Underactuated ship parallel control method based on data physical driving reinforcement learning

Abstract

The invention discloses an under-actuated ship parallel control method based on data physical driving reinforcement learning, which relates to the technical field of ship motion control and comprises the following steps: establishing a mathematical model for a three-degree-of-freedom under-actuated ship with unknown dynamic characteristics and disturbance; real ship off-line data and the recurrent neural network are fused to construct a data physical drive ship model; establishing a virtual ship system for mapping a real ship based on the model and online real ship state information; designing a path tracking parallel feedforward controller on the virtual system by adopting a backstepping method and a dynamic surface control technology; designing an approximate parallel optimal controller in combination with a reinforcement learning optimal control method; the outputs of the two controllers are integrated, an actual control instruction is generated after constraint processing, and parallel tracking control over the ship path is achieved; the method can dynamically adapt to ship loading change and complex navigation working conditions, generalization ability and operation reliability are improved, excessive loss of equipment and energy waste are reduced, and the inherent limitation of single target optimization of a traditional control method is solved.

Description

Technical Field

[0001] This invention relates to the field of ship motion control technology, and in particular to an underactuated ship parallel control method based on data physics-driven reinforcement learning. Background Technology

[0002] As the mainstream vessel type in the shipping industry, underactuated vessels account for a very high proportion of merchant ships, engineering vessels, and special-purpose vessels. Breakthroughs in their remote control and autonomous navigation technologies are crucial for the intelligent upgrading of the shipping industry. With the deep integration of intelligent sensing, communication navigation, and automatic control technologies in the field of marine engineering, underactuated vessel control technology is ushering in a leapfrog development opportunity, laying a solid foundation for the high-quality development of the industry. Driven by the continuous iteration of satellite positioning, inertial navigation, and marine environmental perception technologies, underactuated vessels have acquired the ability to perceive, accurately collect, and reliably transmit multi-dimensional information in real time. Parallel control technology, with its core characteristics of virtual-real dynamic mapping and interactive collaborative optimization, provides a new paradigm for its motion control. After deep integration with multi-source heterogeneous data, it has become the core driving force for breaking through traditional control bottlenecks and achieving autonomous navigation.

[0003] Autonomous navigation of ships requires precise path tracking while ensuring stability, and simultaneously controlling the costs of propulsion power consumption and steering gear frequency to avoid equipment wear and energy waste. This necessitates a balance between minimizing tracking errors and efficiently utilizing control resources in the control system; achieving this equilibrium is a key challenge. Traditional control methods focus on single-objective optimization, making it difficult to synergistically improve both effectiveness and economy. Optimal control theory, with its advantage of "multi-objective trade-off optimization," combined with the virtual-real synergy of parallel control and the dynamic optimization capabilities of reinforcement learning, can construct a framework that balances path tracking accuracy and control economy. Through virtual simulation and real-time correction, it can achieve global optimization of the control strategy.

[0004] Existing control schemes mostly rely on accurate ship dynamics physical models, but acquiring core parameters is costly and time-consuming, and is affected by environmental changes, load adjustments, and strong coupling characteristics, making it difficult for the models to match actual working conditions and limiting the optimization of control accuracy and cost. Purely data-driven methods suffer from drawbacks such as strong data dependence, weak physical interpretability, and insufficient generalization ability, making it difficult to meet the comprehensive requirements of high safety, high reliability, and low cost. Summary of the Invention

[0005] This invention provides a data physics-driven reinforcement learning-based method for parallel control of underactuated ships to overcome the aforementioned technical problems.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows: A data-physics-driven reinforcement learning-based method for underactuated ship parallel control includes: S1: Construct a mathematical model of a three-degree-of-freedom underactuated ship with completely unknown dynamic characteristics and unknown external disturbances; S2: Collect offline data from actual ships, introduce recurrent neural networks (RNNs), and combine a three-degree-of-freedom underactuated ship mathematical model, offline data from actual ships, and recurrent neural networks to construct an underactuated ship data physical driving model. S3: Collect online status information of the actual ship, and construct a virtual underactuated ship system that maps to the actual ship based on the physical drive model of the underactuated ship data and the online status information; S4: Based on a virtual underactuated ship system, a path-following parallel feedforward controller is designed using backstepping and dynamic surface control techniques. S5: Combining reinforcement learning optimal control method and path tracking parallel feedforward controller, design an approximate parallel optimal controller; S6: Combine the path tracking parallel feedforward controller and the approximately parallel optimal controller to construct the output command, perform constraint processing on the output command to output the actual control command, and realize the parallel remote control of ship path tracking.

[0007] Furthermore, a mathematical model of a three-degree-of-freedom underactuated ship with completely unknown dynamic characteristics and unknown external disturbances is constructed, including: A mathematical model of a three-degree-of-freedom underactuated ship with completely unknown dynamic characteristics and unknown external disturbances is constructed. Its kinematic and dynamic model expressions are as follows:

[0008] in, The position vector of the underactuated ship in the inertial coordinate system is... coordinate, Coordinates and bow roll angle ; The velocity vector of the underactuated vessel in the attached coordinate system, i.e., its forward velocity. Horizontal drift speed and bow roll rate ; This is the transformation matrix between the inertial coordinate system and the attached coordinate system; Let be the inertial mass matrix of the underactuated ship, where , , , For ship quality, For rotational inertia, For the hydrodynamic derivatives in each direction of the attached coordinate system; For unknown nonlinear functions, covering Coriolis force and hydrodynamic damping parameters; The actual control vector, i.e., forward force. lateral drift force and bow roll torque ,in , , ,in , For an unknown gain function, This refers to the propeller speed; This refers to the rudder deflection. This represents the marine environmental disturbance vector in the attached coordinate system, i.e., the lateral disturbance force. Longitudinal interference force and bow disturbance torque ,satisfy , for The upper bound of a positive constant.

[0009] Furthermore, an underactuated ship data-physical driving model is constructed by combining a three-degree-of-freedom underactuated ship mathematical model, offline real-ship data, and recurrent neural networks, including: S21. The nominal form of the dynamic model in the mathematical model of a three-degree-of-freedom underactuated ship is:

[0010] By combining a recurrent neural network (RNN) and the ship's inertial mass matrix, the nominal dynamic model is reconstructed into the following form:

[0011] in, The ideal values ​​for the RNN weights are: , , , and The inverse of each element of the inertial mass matrix; For model reconstruction error; Let be a vector formed by the basis functions of an RNN, and It satisfies both monotonically increasing property and Lipschitz continuity, that is... , As the independent variable; A positive constant; S22. Construct a data physics-driven model for an underactuated ship based on the estimated values ​​of RNN weights. The corresponding approximate dynamic system expression is as follows:

[0012] in, This is the estimate of the velocity vector from the physical driving model of ship data; The modeling error is the difference between the actual ship velocity vector in the offline data and the model estimate. For the feedback term gain coefficient; ; For time; For time intervals; The RNN weight estimates are designed based on the Lyapunov stability analysis strategy. The adaptive law is expressed as follows:

[0013] In the formula: , It is a piecewise function; , The learning rate parameter for the RNN; The first derivative of the estimated values ​​of the RNN weight matrix elements. ; Modeling error Taking the derivative, we get:

[0014] in, Let be the RNN weight error, where ; ; S23. Combining the kinematic model in the mathematical model of a three-degree-of-freedom underactuated ship with the data-physics-driven model of an underactuated ship, a data-physics-driven model for an underactuated ship is constructed, expressed as:

[0015] in, Let be the steady-state value of the RNN weights, and ; The total error of the model is denoted as , where This represents the deviation between the ideal weights and the steady-state weights of the RNN; The upper bound satisfies ,and .

[0016] Furthermore, a virtual underactuated ship system based on the physical actuation model of underactuated ship data and its mapping to the actual ship includes: Based on the physical driving model of underactuated ship data, the state error between the real ship and the virtual ship is obtained to construct a virtual underactuated ship system. The expression of the virtual underactuated ship system is as follows:

[0017] in, The virtual ship's position state, i.e., coordinates ,coordinate and bow angle ; The speed status of the virtual ship, i.e., its forward speed. Horizontal drift speed and bow roll rate ; , It is a positive definite coefficient matrix; , These are the positional error and velocity error between the real ship and the virtual ship, respectively. Furthermore, based on a virtual underactuated ship system, a path-following parallel feedforward controller is designed using backstepping and dynamic surface control techniques, including: S41. Based on a virtual underactuated ship system, the vector representation of the virtual ship's dynamics is defined as follows:

[0018] in, These are the components of the dynamic characteristic vector; S42. Using the backstepping method, the position error and heading error are defined as follows:

[0019] in, , , and This represents the error in the horizontal axis and the error in the vertical axis;

[0020]

[0021] The track reference signal is generated from a preset reference path point; The derivatives of position error and heading error are given by the following formula:

[0022]

[0023] in, for The first derivative; Based on the Lyapunov stability analysis strategy, the parallel feedforward virtual controller is designed as follows:

[0024]

[0025] in, , A positive constant; S43. Using dynamic surface control technology, the parallel feedforward virtual controller Through a first-order filter After filtering, the following expression is obtained:

[0026] in, It is a time constant. ; for The first derivative; These are the initial values ​​for the first-order filter and the parallel feedforward virtual controller, respectively. S44. Define the speed error as:

[0027] Based on the vectors of the virtual ship's dynamic composition, the nonlinear function error of the underactuated ship data physical driving model is:

[0028] Based on the mathematical model of a three-degree-of-freedom underactuated ship, the rotational speed and rudder deflection of the underactuated ship are obtained as follows:

[0029] in, For parameters The estimated value; For parameter error; It is a symbolic function; Based on the Lyapunov stability analysis strategy, design a parameter adaptive law. and path tracking parallel feedforward controller Its expression is:

[0030] in, , , It is a positive constant.

[0031] Furthermore, by combining reinforcement learning optimal control methods and path-tracking parallel feedforward controllers, an approximate parallel optimal controller is designed, including: S51. Define the speed error system:

[0032] in, , As an intermediate variable, For the velocity error system, The intermediate variable is composed of the error of a nonlinear function. , is an intermediate variable. , It is a parallel optimal controller; Define the cost function of the system. The expression is:

[0033] in, , It is a positive definite matrix; For time; For parallel control input; The Hamiltonian function of the error system is defined as follows:

[0034] in, for about The partial derivative; for transpose; S52. Based on the Bellman optimality principle, the cost function... By minimizing the cost, the optimal cost function is obtained as follows:

[0035] Substituting the optimal cost function into the Hamiltonian function, we obtain the HJB equation:

[0036] in, for about The partial derivative; The gradient descent method is used to solve the HJB equations to obtain the parallel optimal controller. The expression is:

[0037] S53. Solve for the optimal cost function by evaluating the network. Its expression is:

[0038] in, The ideal value for evaluating network weights; To evaluate network weight error; To evaluate the estimated values ​​of network weights; are basis functions; To approximate the error; Regarding the optimal cost function Taking the partial derivative, we get:

[0039] in, for about The partial derivative; for about The partial derivative; Will Substituting into the HJB function and the parallel optimal controller, the expressions are as follows:

[0040]

[0041] in, To reconstruct the residuals, and ; S54, Based on weighted estimates The expression for the partial derivative of the approximate optimal cost function is as follows:

[0042] in, ; Weight estimates The adaptive law; the adaptive law for weight estimation is designed using the gradient descent method. To minimize the approximate HJB equation:

[0043] In the formula: The learning rate is the weight. Substituting the partial derivatives of the approximate optimal cost function into the HJB equation and the parallel optimal controller, we obtain the approximate HJB equation and the approximate parallel optimal controller as follows:

[0044]

[0045] S55. Substituting the approximate parallel optimal controller into the speed error system of S51, the expression for the approximate error system is obtained as follows: .

[0046] Furthermore, by combining the path-tracking parallel feedforward controller and the approximately parallel optimal controller, output commands are constructed, and constraint processing is applied to the output commands to produce actual control commands, including: S61. Combining the path-following parallel feedforward controller and the approximately parallel optimal controller, the overall controller is constructed as follows:

[0047] Substitute the main controller into the underactuated ship's speed and rudder deflection in S44 to obtain the converted speed and rudder deflection; S62. Considering the physical limitations of actual rotational speed and rudder angle, the input constraints for rotational speed and rudder angle are defined as follows:

[0048] in, This represents the upper limit of the rotational speed.

[0049] Beneficial Effects: This invention provides a data-physics-driven reinforcement learning-based method for underactuated ship parallel control, which has the following beneficial effects: I. By deeply integrating the core physical laws of ship dynamics with the adaptive advantages of data-driven methods, we can avoid the dependence of physical modeling on precise parameters and make up for the deficiency of physical interpretability of data-driven models. This enables the control system to dynamically adapt to changes in ship loading and complex navigation conditions, significantly improving generalization ability and operational reliability.

[0050] II. Leveraging the dynamic mapping and interactive optimization characteristics of the virtual-real system in parallel control, and combining the core characteristics of multi-objective trade-offs in optimal control theory, this invention effectively reduces propulsion system power consumption and steering gear frequency while minimizing path tracking errors in underactuated vessels. This reduces excessive equipment wear and energy waste, successfully balancing navigation accuracy and operational economy, overcoming the inherent limitations of single-objective optimization in traditional control methods. This invention deeply integrates data-physics-driven systems, reinforcement learning, and parallel control, eliminating the need for large-scale, high-cost tank tests and real-ship trials to obtain accurate model parameters, significantly lowering the technological implementation threshold. It is applicable to various mainstream underactuated vessels. Its widespread application can powerfully promote the upgrading of ship control towards unmanned, intelligent, and remote operation, providing core technological support for the intelligent transformation of the shipping industry, and possessing significant economic and practical value. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 A flowchart of a data physics-driven reinforcement learning-based underactuated ship parallel control method provided by the present invention; Figure 2 This is a schematic diagram illustrating the error between the actual model and the data physics-driven model in this embodiment; Figure 3 This is a simulation diagram of the tracking effect of the real and virtual ships on the position coordinates in this embodiment; Figure 4 This is a simulation diagram showing the velocity changes of each degree of freedom of the real and virtual ships in this embodiment; Figure 5This is a simulation diagram showing the convergence result of evaluating the network weight norm in this embodiment; Figure 6 This is a simulation diagram showing the convergence result of the RNN weight norm in this embodiment; Figure 7 This is a simulation diagram showing the changes in the rotational speed and rudder angle of the actual ship in this embodiment; Figure 8 This is a simulation diagram showing the convergence result of the underactuated ship gain function in this embodiment. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] This embodiment provides a data physics-driven reinforcement learning-based method for the parallel control of underactuated ships, such as... Figure 1 As shown, it includes: S1: Construct a mathematical model of a three-degree-of-freedom underactuated ship with completely unknown dynamic characteristics and unknown external disturbances; S2: Collect offline data from actual ships, introduce recurrent neural networks (RNNs), and combine a three-degree-of-freedom underactuated ship mathematical model, offline data from actual ships, and recurrent neural networks to construct an underactuated ship data physical driving model. S3: Collect online status information of the actual ship, and construct a virtual underactuated ship system that maps to the actual ship based on the physical drive model of the underactuated ship data and the online status information; S4: Based on a virtual underactuated ship system, a path-following parallel feedforward controller is designed using backstepping and dynamic surface control techniques. S5: Combining reinforcement learning optimal control method and path tracking parallel feedforward controller, design an approximate parallel optimal controller; S6: Combine the path tracking parallel feedforward controller and the approximately parallel optimal controller to construct the output command, perform constraint processing on the output command to output the actual control command, and realize the parallel remote control of ship path tracking.

[0055] Specifically, this invention first establishes an underactuated ship mathematical model encompassing completely unknown dynamic characteristics and unknown external disturbances. Completely unknown dynamic characteristics and unknown external disturbances refer to uncertain hydrodynamic parameters, unclear nonlinear coupling relationships, difficulty in real-time and accurate measurement or prediction of wind, wave, and current disturbances, and the presence of unstructured interference. Based on offline data from actual ships, and combining the fitting and feature learning capabilities of recurrent neural networks, the invention incorporates the ship's inertial mass matrix to construct a data physics-driven model that requires no precise parameter support and possesses a certain degree of physical interpretability. Using sensors to collect real-time state information such as the actual ship's position, heading, and speed, a virtual underactuated ship system mapped to the actual ship is constructed. Based on this virtual ship system, a parallel controller is designed by integrating backstepping and reinforcement learning optimal control methods, thereby achieving remote path tracking control of the actual ship under random disturbances and completely unknown dynamic characteristics in the marine environment.

[0056] This invention achieves deep fusion of offline data and the ship's inertial mass matrix, eliminating the need for precise ship body parameters and enhancing the physical interpretability of the data-driven model, thus avoiding the "black box" problem. Leveraging the parallel mapping interaction mechanism between virtual and real systems and combining the advantages of reinforcement learning, it maximizes the core effectiveness of parallel control in full-condition ship simulation, reduces reliance on human intervention, and promotes the unmanned, intelligent, and remote upgrading of the system. This provides a reliable parallel remote path tracking control solution for diverse scenarios such as near-shore remote operations.

[0057] In a specific embodiment, the scheme for constructing a mathematical model of a three-degree-of-freedom underactuated ship with completely unknown dynamic characteristics and unknown external disturbances is as follows: Its kinematic and dynamic model expressions are as follows:

[0058] in, The position vector of the underactuated ship in the inertial coordinate system is... coordinate, Coordinates and bow roll angle ; The velocity vector of the underactuated vessel in the attached coordinate system, i.e., its forward velocity. Horizontal drift speed and bow roll rate ; This is the transformation matrix between the inertial coordinate system and the attached coordinate system; Let be the inertial mass matrix of the underactuated ship, where , , , For ship quality, For rotational inertia, For the hydrodynamic derivatives in each direction of the attached coordinate system; For unknown nonlinear functions, covering Coriolis force and hydrodynamic damping parameters; The actual control vector, i.e., forward force. lateral drift force and bow roll torque ,in , , ,in , For an unknown gain function, This refers to the propeller speed; This refers to the rudder deflection. This represents the marine environmental disturbance vector in the attached coordinate system, i.e., the lateral disturbance force. Longitudinal interference force and bow disturbance torque ,satisfy , for The upper bound of a positive constant.

[0059] In a specific embodiment, the scheme of collecting offline data from actual ships, introducing a recurrent neural network (RNN), and combining a three-degree-of-freedom underactuated ship mathematical model, offline data from actual ships, and a recurrent neural network to construct an underactuated ship data physical driving model is as follows: S21. The nominal form of the dynamic model in the mathematical model of a three-degree-of-freedom underactuated ship is:

[0060] By combining a recurrent neural network (RNN) and the ship's inertial mass matrix, the nominal dynamic model is reconstructed into the following form:

[0061] in, The ideal values ​​for the RNN weights are: , , , and The inverse of each element of the inertial mass matrix; For model reconstruction error; Let be a vector formed by the basis functions of an RNN, and It satisfies both monotonically increasing property and Lipschitz continuity, that is... , As the independent variable; A positive constant; S22. Construct a data physics-driven model for an underactuated ship based on the estimated values ​​of RNN weights. The corresponding approximate dynamic system expression is as follows:

[0062] in, This is the estimate of the velocity vector from the physical driving model of ship data; The modeling error is the difference between the actual ship velocity vector in the offline data and the model estimate. For the feedback term gain coefficient; ; For time; For time intervals; The RNN weight estimates are designed based on the Lyapunov stability analysis strategy. The adaptive law is expressed as follows:

[0063] In the formula: , It is a piecewise function; , The learning rate parameter for the RNN; The first derivative of the estimated values ​​of the RNN weight matrix elements. ; Modeling error Taking the derivative, we get:

[0064] in, Let be the RNN weight error, where ; ; S23. Combining the kinematic model in the mathematical model of a three-degree-of-freedom underactuated ship with the data-physics-driven model of an underactuated ship, a data-physics-driven model for an underactuated ship is constructed, expressed as:

[0065] in, Let be the steady-state value of the RNN weights, and ; The total error of the model is denoted as , where This represents the deviation between the ideal weights and the steady-state weights of the RNN; The upper bound satisfies ,and .

[0066] In a specific embodiment, the scheme of collecting online status information of the actual ship and constructing a virtual underactuated ship system mapped to the actual ship based on the underactuated ship data physical drive model and online status information is as follows: Based on the physical driving model of underactuated ship data, the state error between the real ship and the virtual ship is obtained to construct a virtual underactuated ship system. The expression of the virtual underactuated ship system is as follows:

[0067] in, The virtual ship's position state, i.e., coordinates ,coordinate and bow angle ; The speed status of the virtual ship, i.e., its forward speed. Horizontal drift speed and bow roll rate ; , It is a positive definite coefficient matrix; , These are the positional error and velocity error between the real ship and the virtual ship, respectively. The derivative of the state error between the real ship and the virtual ship is used in the subsequent proof, and the expression is:

[0068] in, .

[0069] In a specific embodiment, based on a virtual underactuated ship system, the scheme for designing a path-tracking parallel feedforward controller using backstepping and dynamic surface control techniques is as follows: S41. Based on a virtual underactuated ship system, the vector representation of the virtual ship's dynamics is defined as follows:

[0070] in, These are the components of the dynamic characteristic vector; S42. Using the backstepping method, the position error and heading error are defined as follows:

[0071] in, , , and This represents the error in the horizontal axis and the error in the vertical axis; The desired heading angle;

[0072] The track reference signal is generated from a preset reference path point; The derivatives of position error and heading error are given by the following formula:

[0073]

[0074] in, for The first derivative; Based on the Lyapunov stability analysis strategy, the parallel feedforward virtual controller is designed as follows:

[0075]

[0076] in, , A positive constant; S43. Using dynamic surface control technology, the parallel feedforward virtual controller Through a first-order filter After filtering, the following expression is obtained:

[0077] in, It is a time constant. ; for The first derivative; These are the initial values ​​for the first-order filter and the parallel feedforward virtual controller, respectively. The DSC error is defined for the subsequent proof:

[0078] Taking the derivative of this error, we get:

[0079] in: And satisfy ( for (positive numbers of the upper bound); for right The partial derivatives; S44. Define the speed error as:

[0080] To support the design of the parallel optimal controller, based on the vector of the virtual ship's dynamic composition, the nonlinear function error of the underactuated ship data physical drive model is constructed as follows:

[0081] Taking the derivative of the speed error and combining it with the controller definition in step S41, we obtain the following equation for proof:

[0082] Based on the mathematical model of a three-degree-of-freedom underactuated ship, the rotational speed and rudder deflection of the underactuated ship are obtained as follows:

[0083] in, For parameters The estimated value; For parameter error; It is a symbolic function; Based on the Lyapunov stability analysis strategy, design a parameter adaptive law. and path tracking parallel feedforward controller Its expression is:

[0084] in, , , It is a positive constant.

[0085] Based on the stability analysis proposed in steps S42 and S44, the Lyapunov function is defined as follows:

[0086] right Taking the derivative and substituting the position, heading, speed, filter error, parameter adaptive law, and parallel virtual feedforward controller from the above steps, we get:

[0087] After further scaling using Young's inequality, we finally obtain:

[0088] In the formula: ; ; All are positive numbers; ; ; This is the position error vector; This is the velocity error vector; is the steady-state vector of the inertial mass matrix; .

[0089] In a specific embodiment, the scheme for designing an approximate parallel optimal controller by combining reinforcement learning optimal control method and path tracking parallel feedforward controller is as follows: S51. Define the speed error system:

[0090] in, , As an intermediate variable, For the velocity error system, The intermediate variable is composed of the error of a nonlinear function. , , It is a parallel optimal controller; Define the cost function of the system. The expression is:

[0091] in, , It is a positive definite matrix; For time; For parallel control input; The Hamiltonian function of the error system is defined as follows:

[0092] in, for about The partial derivative; for transpose; S52. Based on the Bellman optimality principle, the cost function... By minimizing the cost, the optimal cost function is obtained as follows:

[0093] Substituting the optimal cost function into the Hamiltonian function, we obtain the HJB equation:

[0094] in, for about The partial derivative; The gradient descent method is used to solve the HJB equations to obtain the parallel optimal controller. The expression is:

[0095] S53. Solve for the optimal cost function by evaluating the network. Its expression is:

[0096] in, The ideal value for evaluating network weights; To evaluate network weight error; To evaluate the estimated values ​​of network weights; are basis functions; To approximate the error; Regarding the optimal cost function Taking the partial derivative, we get:

[0097] in, for about The partial derivative; for about The partial derivative; Will Substituting into the HJB function and the parallel optimal controller, the expressions are as follows:

[0098]

[0099] in, To reconstruct the residuals, and ; S54, Based on weighted estimates The expression for the partial derivative of the approximate optimal cost function is as follows:

[0100] in, ; Weight estimates The adaptive law; the adaptive law for weight estimation is designed using the gradient descent method. To minimize the approximate HJB equation:

[0101] In the formula: The learning rate is the weight. Substituting the partial derivatives of the approximate optimal cost function into the HJB equation and the parallel optimal controller, we obtain the approximate HJB equation and the approximate parallel optimal controller as follows:

[0102]

[0103] S55. Substituting the approximate parallel optimal controller into the speed error system of S51, the expression for the approximate error system is obtained as follows: .

[0104] In a specific embodiment, the scheme for parallel remote control of ship path tracking is as follows: The output command is constructed by combining a path-tracking parallel feedforward controller and an approximately parallel optimal controller. The output command is then constrained to produce the actual control command. S61. Combining the path-following parallel feedforward controller and the approximately parallel optimal controller, the overall controller is constructed as follows:

[0105] Substitute the main controller into the underactuated ship's speed and rudder deflection in S44 to obtain the converted speed and rudder deflection; S62. Considering the physical limitations of actual rotational speed and rudder angle, the input constraints for rotational speed and rudder angle are defined as follows:

[0106] in, The upper limit of the rotational speed is set to 100 in this embodiment.

[0107] This embodiment also includes Lyapunov stability analysis to prove the stability of the input-to-state of the designed underactuated ship parallel control system based on data physics-driven reinforcement learning, and that all signals in the closed-loop system are eventually uniformly bounded: Specifically, to more easily prove stability, we make the following assumptions: Assumption 1: Model reconstruction error It is bounded, and the upper and lower bounds are the modeling errors. The function, i.e., the expression is ,in It is a positive constant.

[0108] Assumption 2: Ideal values ​​for RNN weights It is bounded, that is, it satisfies ,in It is a positive definite matrix.

[0109] Assumption 3: In compact sets superior, , , , and All are bounded, satisfying , , , , ,and ,in , , , and They are all positive constants.

[0110] Theorem 1: For an underactuated ship path tracking parallel control system with completely unknown dynamic characteristics and unknown external disturbances, based on Lyapunov stability theory, by constructing a physical driving model of real ship data, designing an RNN weight adaptive law, building a virtual underactuated ship system, and combining the synergistic effect of a parallel feedforward controller, approximately parallel optimal control, and evaluation network weight adaptive law, under the premise of reasonable selection of design parameters, it can not only ensure that all signals of the closed-loop system are eventually consistent and bounded, but also make the path tracking error approach zero in a bounded compact set with optimal convergence characteristics. Finally, it achieves a path tracking parallel control system with both high tracking accuracy and low control cost under complex uncertainty conditions.

[0111] Proof of Theorem 1: S71: To prove that the constructed physical-driven model of the real ship data and the RNN weights are eventually uniformly bounded, construct the Lyapunov function:

[0112] Differentiate it and use the derivative of the modeling error. and the derivative of RNN weight error Substituting, we get:

[0113] Combining the monotonically increasing property of basis functions, Assumption 1, Assumption 2, and Young's inequality, the following inequality can be derived:

[0114] In the formula: For matrix The smallest eigenvalue; ; for The identity matrix. Therefore, when choosing a suitable... Value, making That is, satisfying Therefore, it can be proven that the data physics-driven model of the actual ship and the RNN weights are eventually uniformly bounded.

[0115] S72: Prove that the constructed virtual underactuated ship dynamics system is stable when input to the state, and construct the Lyapunov function:

[0116] Differentiate it to find the error derivative between the real ship and the virtual ship. Substituting, we get:

[0117] In the formula: For matrix The smallest eigenvalue; .

[0118] Next, it is proven that the constructed virtual underactuated ship kinematics system is stable when input to the state, by constructing the Lyapunov function:

[0119] Differentiate it to find the error derivative between the real ship and the virtual ship. Substituting, we get:

[0120] In the formula: For matrix The smallest eigenvalue. Therefore, when choosing a suitable... , Value, making , When, that is, satisfied Therefore, it can be proven that the virtual underactuated ship system is stable when input to the state.

[0121] S73: Prove that the parallel control system and the evaluation network weights are eventually uniformly bounded, based on the Lyapunov function designed in step S34 for the parallel feedforward control. Construct Lyapunov functions :

[0122] For the above Lyapunov function Taking the derivative and substituting the relevant equations from steps S34, S41, and S73, we get:

[0123] Based on assumption 3, let's take one of them as an example to illustrate the scaling process:

[0124] The remaining items are processed in the same way, and we can obtain

[0125] In the formula: ; ; ; ; Therefore, when the condition is met... In this way, by selecting appropriate parameters, all signals of the data-driven ship path tracking parallel control system can eventually be made consistent and bounded.

[0126] Compared with existing technologies, the data-physics-driven reinforcement learning-based underactuated ship parallel control method disclosed in this embodiment has the following advantages: 1. This invention deeply integrates the physical laws inherent in the ship dynamics inertial mass matrix with the adaptive advantages of data-driven approaches. It avoids dependence on high-cost parameter testing, significantly reduces the cost of technology research and development and application, and ensures the physical interpretability of the control strategy. It can be widely adapted to various mainstream underactuated ships.

[0127] 2. This invention combines the multi-objective trade-off characteristics of optimal control with the virtual-real synergy advantages of parallel control. While ensuring the path tracking accuracy of underactuated ships, it effectively reduces propulsion power consumption and steering gear operation frequency, achieving synergistic optimization of navigation accuracy and operational economy, and solving the performance imbalance problem of traditional control schemes.

[0128] 3. The present invention conducted simulation experiments based on the MATLAB platform and designed a parallel controller to realize remote control of ship path following in unknown and complex marine environments. The system controller is more in line with navigation practice and further verifies the effectiveness and rationality of the data physics-driven reinforcement learning-based underactuated ship parallel control method provided by the present invention.

[0129] Example: To verify the effectiveness of the proposed data physics-driven reinforcement learning-based underactuated ship parallel control method, this embodiment employs computer simulation in MATLAB. The parameter settings are as follows: The simulation object is CyberShip II from the Norwegian University of Science and Technology, with a ship weight of [missing information]. The captain is The width of the boat is First, when constructing the physical-driven model of the actual ship data, parameters are set. , , , , Initial values ​​of RNN weights For the corresponding dimension, the zero matrix, RNN basis functions Then, when designing the virtual underactuated ship system, the design parameters... , Finally, in designing the parallel controller, the design parameters... , , , , , The initial value of the first-order filter is time constant , The initial value of the gain function is Design parameters ; The initial value of the network weights is evaluated. The evaluation network basis function is Center point selection The reference waypoint for the ship is set as follows. , , , , , The initial value of the actual ship condition is The initial value of the virtual ship state is .

[0130] The simulation results are shown in the figure: Figure 2 This is a schematic diagram of the error between the actual model and the data physics-driven model. It can be seen that the error has successfully converged to a bounded neighborhood centered on the origin. Figure 3 Displays the position coordinates of the real ship and the virtual ship. The path tracking effect on the plane shows that the output position signal can effectively track the reference signal. Figure 4 The image shows a waveform comparison of the velocities of the real ship and the virtual ship at each degree of freedom. The virtual ship can accurately track the speed changes of the real ship. Figure 5 To evaluate the convergence curve of the network weight norm, it is shown that the weight norm can stably converge to a fixed value. Figure 6 The convergence process of the RNN weight norm is shown, and it can also converge to a stable value. Figure 7 This is a curve showing the changes in ship speed and rudder angle. Figure 8 This is the convergence curve of the gain function.

[0131] Simulation results verify the effectiveness of the data-physics-driven virtual underactuated ship path tracking control system constructed in this invention: the parallel controller designed based on this control system can achieve good synchronous path tracking between the real and virtual ships. This fully demonstrates that the output of the data-physics-driven ship parallel control system has good tracking performance, further verifying the effectiveness and rationality of the data-physics-driven reinforcement learning-based underactuated ship parallel control method proposed in this invention.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data-physics-driven reinforcement learning-based method for underactuated ship parallel control, characterized in that, include: S1: Construct a mathematical model of a three-degree-of-freedom underactuated ship with completely unknown dynamic characteristics and unknown external disturbances; S2: Collect offline data from actual ships, introduce recurrent neural networks (RNNs), and combine a three-degree-of-freedom underactuated ship mathematical model, offline data from actual ships, and recurrent neural networks to construct an underactuated ship data physical driving model. S3: Collect online status information of the actual ship, and construct a virtual underactuated ship system that maps to the actual ship based on the physical drive model of the underactuated ship data and the online status information; S4: Based on a virtual underactuated ship system, a path-following parallel feedforward controller is designed using backstepping and dynamic surface control techniques. S5: Combining reinforcement learning optimal control method and path tracking parallel feedforward controller, design an approximate parallel optimal controller; S6: Combine the path tracking parallel feedforward controller and the approximately parallel optimal controller to construct the output command, perform constraint processing on the output command to output the actual control command, and realize the parallel remote control of ship path tracking.

2. The data-physics-driven reinforcement learning-based underactuated ship parallel control method according to claim 1, characterized in that, Constructing a mathematical model for a three-degree-of-freedom underactuated ship with completely unknown dynamic characteristics and unknown external disturbances, including: A mathematical model of a three-degree-of-freedom underactuated ship with completely unknown dynamic characteristics and unknown external disturbances is constructed. Its kinematic and dynamic model expressions are as follows: in, The position vector of the underactuated ship in the inertial coordinate system is... coordinate, Coordinates and bow roll angle ; The velocity vector of the underactuated vessel in the attached coordinate system, i.e., its forward velocity. Horizontal drift speed and bow roll rate ; This is the transformation matrix between the inertial coordinate system and the attached coordinate system; Let be the inertial mass matrix of the underactuated ship, where , , , For ship quality, For rotational inertia, For the hydrodynamic derivatives in each direction of the attached coordinate system; For unknown nonlinear functions, covering Coriolis force and hydrodynamic damping parameters; The actual control vector, i.e., forward force. lateral drift force and bow roll torque ,in , , ,in , For an unknown gain function, This refers to the propeller speed; This refers to the rudder deflection. This represents the marine environmental disturbance vector in the attached coordinate system, i.e., the lateral disturbance force. Longitudinal interference force and bow disturbance torque ,satisfy , for The upper bound of a positive constant.

3. The data-physics-driven reinforcement learning-based underactuated ship parallel control method according to claim 2, characterized in that, A data-physical driving model for underactuated ships is constructed by combining a three-degree-of-freedom underactuated ship mathematical model, offline data from actual ships, and recurrent neural networks, including: S21. The nominal form of the dynamic model in the mathematical model of a three-degree-of-freedom underactuated ship is: By combining a recurrent neural network (RNN) and the ship's inertial mass matrix, the nominal dynamic model is reconstructed into the following form: in, The ideal values ​​for the RNN weights are: , , , and The inverse of each element of the inertial mass matrix; For model reconstruction error; Let be a vector formed by the basis functions of an RNN, and It satisfies both monotonically increasing property and Lipschitz continuity, that is... , As the independent variable; A positive constant; S22. Construct a data physics-driven model for an underactuated ship based on the estimated values ​​of RNN weights. The corresponding approximate dynamic system expression is as follows: in, This is the estimate of the velocity vector from the physical driving model of ship data; The modeling error is the difference between the actual ship velocity vector in the offline data and the model estimate. For the feedback term gain coefficient; ; For time; For time intervals; The RNN weight estimates are designed based on the Lyapunov stability analysis strategy. The adaptive law is expressed as: In the formula: , It is a piecewise function; , The learning rate parameter for the RNN; The first derivative of the estimated values ​​of the RNN weight matrix elements. ; Modeling error Taking the derivative, we get: in, Let be the RNN weight error, where ; ; S23. Combining the kinematic model in the mathematical model of a three-degree-of-freedom underactuated ship with the data-physics-driven model of an underactuated ship, a data-physics-driven model for an underactuated ship is constructed, expressed as: in, Let be the steady-state value of the RNN weights, and ; The total error of the model is denoted as , where This represents the deviation between the ideal weights and the steady-state weights of the RNN; The upper bound satisfies ,and 。 4. The data-physics-driven reinforcement learning-based underactuated ship parallel control method according to claim 3, characterized in that, A virtual underactuated ship system based on the physical driving model of underactuated ship data and its mapping to the real ship includes: Based on the physical driving model of underactuated ship data, the state error between the real ship and the virtual ship is obtained to construct a virtual underactuated ship system. The expression of the virtual underactuated ship system is as follows: in, The virtual ship's position state, i.e., coordinates ,coordinate and bow angle ; The speed status of the virtual ship, i.e., its forward speed. Horizontal drift speed and bow roll rate ; , It is a positive definite coefficient matrix; , These represent the positional error and velocity error between the real ship and the virtual ship, respectively.

5. The data-physics-driven reinforcement learning-based underactuated ship parallel control method according to claim 4, characterized in that, Based on a virtual underactuated ship system, a path-following parallel feedforward controller is designed using backstepping and dynamic surface control techniques, including: S41. Based on a virtual underactuated ship system, the vector representation of the virtual ship's dynamics is defined as follows: in, These are the components of the dynamic characteristic vector; S42. Using the backstepping method, the position error and heading error are defined as follows: in, , , and This represents the error in the horizontal axis and the error in the vertical axis; The desired heading angle; The track reference signal is generated from a preset reference path point; The derivatives of position error and heading error are given by the following formula: in, for The first derivative; Based on the Lyapunov stability analysis strategy, the parallel feedforward virtual controller is designed as follows: in, , A positive constant; S43. Using dynamic surface control technology, the parallel feedforward virtual controller Through a first-order filter After filtering, the following expression is obtained: in, It is a time constant. ; for The first derivative; These are the initial values ​​for the first-order filter and the parallel feedforward virtual controller, respectively. S44. Define the speed error as: Based on the vectors of the virtual ship's dynamic composition, the nonlinear function error of the underactuated ship data physical driving model is: Based on the mathematical model of a three-degree-of-freedom underactuated ship, the rotational speed and rudder deflection of the underactuated ship are obtained as follows: in, For parameters The estimated value; For parameter error; It is a symbolic function; Based on the Lyapunov stability analysis strategy, design a parameter adaptive law. and path tracking parallel feedforward controller Its expression is: in, , , It is a positive constant.

6. The data-physics-driven reinforcement learning-based underactuated ship parallel control method according to claim 5, characterized in that, Combining reinforcement learning-based optimal control methods and path-following parallel feedforward controllers, an approximate parallel optimal controller is designed, including: S51. Define the speed error system: in, , As an intermediate variable, For the velocity error system, The intermediate variable is composed of the error of a nonlinear function. , is an intermediate variable. , It is a parallel optimal controller; Define the cost function of the system. The expression is: in, , It is a positive definite matrix; For time; For parallel control input; The Hamiltonian function of the error system is defined as follows: in, for about The partial derivative; for transpose; S52. Based on the Bellman optimality principle, the cost function... By minimizing the cost, the optimal cost function is obtained as follows: Substituting the optimal cost function into the Hamiltonian function, we obtain the HJB equation: in, for about The partial derivative; The gradient descent method is used to solve the HJB equations to obtain the parallel optimal controller. The expression is: S53. Solve for the optimal cost function by evaluating the network. Its expression is: in, The ideal value for evaluating network weights; To evaluate network weight error; To evaluate the estimated values ​​of network weights; are basis functions; To approximate the error; Regarding the optimal cost function Taking the partial derivative, we get: in, for about The partial derivative; for about The partial derivative; Will Substituting into the HJB function and the parallel optimal controller, the expressions are as follows: in, To reconstruct the residuals, and ; S54, Based on weighted estimates The expression for the partial derivative of the approximate optimal cost function is as follows: in, ; Weight estimates The adaptive law; the adaptive law for weight estimation is designed using the gradient descent method. To minimize the approximate HJB equation: In the formula: The learning rate is the weight. Substituting the partial derivatives of the approximate optimal cost function into the HJB equation and the parallel optimal controller, we obtain the approximate HJB equation and the approximate parallel optimal controller as follows: S55. Substituting the approximate parallel optimal controller into the speed error system of S51, the expression for the approximate error system is obtained as follows: 。 7. The data-physics-driven reinforcement learning-based underactuated ship parallel control method according to claim 6, characterized in that, The output command is constructed by combining the path-tracking parallel feedforward controller and the approximately parallel optimal controller. Constraints are applied to the output command to produce the actual control command, including: S61. Combining the path-following parallel feedforward controller and the approximately parallel optimal controller, the overall controller is constructed as follows: Substitute the main controller into the underactuated ship's speed and rudder deflection in S44 to obtain the converted speed and rudder deflection; S62. Considering the physical limitations of actual rotational speed and rudder angle, the input constraints for rotational speed and rudder angle are defined as follows: in, This represents the upper limit of the rotational speed.

Images