Ship zero-sum game tracking control method based on digital-object dual-drive reinforcement learning

Abstract

The invention discloses a ship zero-sum game tracking control method based on digital-object dual-drive reinforcement learning. The method comprises the following steps: constructing a data physical dual-drive model; adopting a Lyapunov backstepping method to hierarchically design a ship track tracking feedforward controller; defining a performance index function and a Hamiltonian-Jacobian-Ansaxophone equation based on an error system and a zero-sum game theory, and designing an optimal feedback controller and a worst interference strategy; a fuzzy logic system is utilized to approach an HJI equation solution, and Nash equilibrium of an output estimation value of an optimal feedback controller and a worst interference strategy is obtained; a feedforward controller, an optimal feedback controller and a worst interference strategy are fused, and a final ideal ship controller capable of directly executing an actual control instruction is obtained through constraint processing, so that ship track tracking is realized. The method solves the problems that the existing method generally has the inherent defects of high dependence on high-quality data, weak physical interpretability, insufficient generalization ability to complex working conditions and the like, and is difficult to meet the safety and reliability requirements of ship navigation.

Description

Technical Field

[0001] This invention relates to the field of ship motion control technology, and in particular to a zero-sum game tracking control method for ships based on dual-drive reinforcement learning of numbers and objects. Background Technology

[0002] With the deep penetration and integration of electronic and information technologies into ships, the field of ship motion control is ushering in unprecedented development opportunities and has laid a solid technological foundation. Especially driven by the continuous iteration and upgrading of sensor and Internet of Things technologies, ships now possess the ability to automatically sense, accurately collect, and transmit multi-dimensional information such as their own motion status in real time. Deeply integrating this real-time acquired multi-source heterogeneous information and data into the existing control technology system will undoubtedly become the core driving force for the innovation and upgrading of ship motion control technology and the realization of intelligent leaps, providing new possibilities for precise ship control in complex marine environments.

[0003] During navigation, ships continuously face the coupled effects of complex ocean disturbances such as wind, waves, and currents. The interaction between the controller and external disturbances essentially constitutes a typical "zero-sum game" relationship—the controller aims to minimize tracking error, while the disturbance aims to maximize error as its "adversarial objective." This poses a severe challenge to navigation safety and operational efficiency, placing higher demands on the anti-interference capability and dynamic adversarial performance of the ship's control system. The introduction of game theory provides a new perspective for solving this adversarial control problem. Its organic integration with optimal control theory and reinforcement learning can construct a dynamic adversarial optimization framework for the controller and disturbances, enabling the solution of the optimal control strategy under the worst-case disturbance strategy. This integrated technical approach provides solid theoretical support and reliable technical assurance for the intelligent transformation of ship motion control, helping ships achieve safe, efficient, and precise autonomous navigation in complex ocean environments.

[0004] Meanwhile, in existing ship motion control technologies, most solutions rely on physical models to construct the control framework. However, the core parameters of ship physical models need to be obtained through extensive, costly, and long-term tank tests and real-ship tests. Furthermore, due to the complexity of the marine environment and the coupling characteristics of ship dynamics, absolute accuracy is difficult to achieve. This practical bottleneck significantly restricts the research and development efficiency and performance breakthroughs of ship control technologies. Although a few studies have attempted to achieve control using data-driven methods, these methods generally suffer from inherent defects such as high dependence on high-quality data, weak physical interpretability, and insufficient generalization ability to complex operating conditions, making it difficult to meet the safety and reliability requirements of ship navigation. Therefore, organically integrating the physical laws inherent in ship physical models with the adaptive advantages of data-driven approaches to construct a control method that combines physical interpretability and dynamic adaptive compensation capabilities can not only avoid the limitations of single modeling methods but also improve control accuracy, possessing significant research value and engineering application prospects. Summary of the Invention

[0005] This invention provides a zero-sum game tracking control method for ships based on dual-drive reinforcement learning of numbers and objects, in order to overcome the above-mentioned technical problems.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows: A zero-sum game tracking and control method for ships based on dual-driven reinforcement learning of numbers and objects, specifically including the following steps: S1: Obtain a ship physical model that takes into account unknown wind, wave and current disturbances, input constraints and unknown dynamics; S2: Construct a data physics dual-drive model based on the ship physics model and a recurrent neural network; S3: Based on the Lyapunov backstepping method, define the ideal control input of the data-physical dual-drive model. Based on the preset desired ship trajectory, obtain the trajectory tracking error according to the data-physical dual-drive model, and construct a ship virtual controller based on the trajectory tracking error. Obtain the ship speed error from the ship virtual controller to define intermediate parameters for optimal feedback control design. After differentiating the ship speed error, obtain the rewritten ship speed error derivative based on the intermediate parameters and the ideal control input. Design a ship feedforward controller based on the rewritten ship speed error derivative. Define the Lyapunov function based on the trajectory tracking error and the ship speed error, and obtain the derivative of the Lyapunov function by differentiation. S4: Define the error system based on the derivative of the Lyapunov function; construct a performance index function based on the error system and zero-sum game theory; construct the Hamilton-Jacobi-Isax equation based on the performance index function to obtain the optimal feedback controller and the worst disturbance strategy. S5: Use the fuzzy logic system to approximate the optimal performance index function to obtain the output estimates of the optimal feedback controller and the worst interference strategy, and construct the neural network weight update rate of the fuzzy logic system based on the output estimates of the optimal feedback controller and the worst interference strategy. S6: Based on the neural network weight update rate, the ship's final ideal controller is constructed by combining the ship's feedforward controller with the optimal feedback controller and the worst-case disturbance strategy; the control input of the ship's final ideal controller is constrained by the input constraints in the ship's physical model to obtain the actual control input, and the ship's zero-sum game tracking control based on data physics dual-drive reinforcement learning is realized based on the actual control input.

[0007] Furthermore, the expression for the ship physical model described in S1 is: , In the formula: Indicates the ship in the inertial coordinate system coordinate, Coordinates and bow roll angle The position vector formed and ; Represents the forward speed of the ship in the attached coordinate system. Horizontal drift speed and bow roll rate The velocity vector formed and ; Denotes the coordinate system transformation matrix and ; This indicates the actual forward control force that the ship can directly execute. Lateral drift force and bow roll torque The control vector formed and ; This represents the lateral disturbance force on a ship caused by wind, waves, and currents in the attached coordinate system. Longitudinal interference force and bow disturbance torque The perturbation vector formed and ,satisfy ; express The upper bound of a positive constant; The matrix represents the ship's weight inertia and hydrodynamic additional inertia, and ; Describe the Coriolis matrix and ; Represents the hydrodynamic damping parameter matrix and ; Indicates transpose; Represent the space of real numbers; express The first derivative; The input constraints are defined as follows: , In the formula: Represents the control forces of each degree of freedom in the ship's physical model and ; Indicates the ideal control inputs for the ship's longitudinal, lateral, and bow directions; This indicates the maximum value of the control input for each degree of freedom of the ship; This represents the minimum control input value for each degree of freedom of the ship.

[0008] Furthermore, step S2 specifically includes the following steps: S21: The nominal dynamic model obtained from the ship's physical model, without considering input constraints, is as follows: , In the formula: Represents the ideal control input vectors for the ship's longitudinal, lateral, and bow directions, and ; S22: Based on the Stone-Weilstras theorem, the nominal dynamic model of the ship is rewritten using RNN basis functions as follows: , In the formula: Describes the ideal weights of the basis functions of an RNN that satisfy... ; This represents the estimated values ​​of the RNN network weights; This represents the error between the ideal weights and the estimated weights of the RNN basis functions; Indicates reconstruction error and ; Describes a vector composed of RNN basis functions and ; Represents the basis functions of a monotonically increasing RNN; S23: Based on the revised nominal dynamics model, construct the approximate dynamics system of the ship as follows: , , In the formula: Represents the estimated value of the ship data physical dual-drive model and satisfies ; Indicates modeling error; This represents the reconstruction error compensation feedback term; This represents the gain coefficient of the linear feedback term. ; The gain parameter estimate of the nonlinear feedback term and ; The gain parameter represents the ideal nonlinear feedback term; This represents the gain parameter error of the nonlinear feedback term; express The first derivative; S24: Based on the ship's approximate dynamics system and the aforementioned ship physical model, a data-physical dual-drive model is constructed as follows: .

[0009] Furthermore, step S3 specifically includes the following steps: S31: Based on Lyapunov's backstepping method, define the ideal control input for the data-physical dual-drive model. The expression for the ideal control input is: + , In the formula: Express the output of the feedforward controller; Express the output of the optimal feedback controller; This represents the output of the worst-case interference strategy; S32: Based on the defined data-physical dual-drive model, the ship's trajectory tracking error is: , In the formula: Indicates the tracking error; Indicates the track reference signal and ; S33: Obtain the derivative of the trajectory tracking error by taking the derivative of the trajectory tracking error, and rewrite the derivative of the trajectory tracking error according to the data-physics dual-drive model as follows: , In the formula: express The first derivative; S34: Based on the rewritten derivative of the trajectory tracking error, the ship virtual controller is designed as follows:

[0010] In the formula: Indicates the control design parameters and ; This represents the output of the ship's virtual controller; S35: Obtain the ship speed error based on the ship virtual controller and the data physics dual-drive model to define intermediate parameters for optimal feedback control design. The expression for these parameters is: , , , , In the formula: Indicates the error in ship speed; Indicates intermediate variables; S36: Obtain the derivative of the ship speed error by taking the derivative of the ship speed error, and obtain the rewritten derivative of the ship speed error based on the intermediate parameters and the ideal control input; , In the formula: express The first derivative; express The first derivative; S37: Based on the rewritten derivative of the ship speed error, the ship feedforward controller is designed as follows: , In the formula: Indicates the control design parameters and ; This indicates the output of the ship's feedforward controller; S38: Define the Lyapunov function based on track tracking error and ship speed error. for: , And for Lyapunov functions Find the derivative of the Lyapunov function. for: .

[0011] Furthermore, step S4 specifically includes the following steps: S41: The derivative of the Lyapunov function... The term is defined as an error system: , S42: Based on the error system and zero-sum game theory, the performance index function under zero-sum game is defined as follows: , In the formula: Indicates time parameters; Indicates the discount factor; Indicates the permissible control inputs; , , Denotes the positive definite gain matrix and , , ; S43: Based on the Nash-Pontryagin maximum-minimum principle, it is confirmed that the zero-sum game has a unique solution. The condition is that there exists a unique saddle point such that the optimal performance index function... for: , S44: Based on the performance index function, define the Hamiltonian function of the error system, whose expression is: , In the formula: express about The partial derivative; express Transpose of; Based on the Hamiltonian function and the optimal performance index function The Hamilton-Jacobi-Isax equations are obtained as follows: , In the formula: express about The partial derivative; S45: Construct the Hamilton-Jacobi-Isax equation based on the performance index function, and use the gradient descent method to obtain the optimal feedback controller and the worst-case disturbance strategy: , .

[0012] Furthermore, S5 specifically includes the following steps: S51: The optimal performance index function is approximated using a fuzzy logic system, and its expression is: , In the formula: Represents the ideal weights and satisfies ; This represents the error in the weights of the neural network; express The estimated value; Represents fuzzy basis functions; Indicates the approximation error; S52: Obtain the approximate optimal performance index function based on S51. partial derivatives for: , In the formula: express about The partial derivative; express about The partial derivative; S53: Based on weight The estimated value Determine the evaluation network and obtain the partial derivative estimate of the optimal performance index function based on S52. for: , S54: Partial derivative as described in S52 Substituting this into the optimal feedback controller and worst-case disturbance strategy in S45 and the Hamilton-Jacobi-Isax equation in S44, we obtain: , , , In the formula: Indicates the reconstructed residual and ; Indicates intermediate parameters and ; express The abbreviated form; Similarly, the partial derivative estimates obtained from S53 are... Substituting the optimal feedback controller and worst-case disturbance strategy in S45 and the Hamilton-Jacobi-Isaks equation in S54, we obtain the output estimate of the optimal feedback controller, the output estimate of the worst-case disturbance strategy, and the rewritten Hamilton-Jacobi-Isaks equation. , , , S55: Substitute the output estimates of the optimal feedback controller and the worst-case disturbance strategy into the error system in S41 to obtain the rewritten error system: , S56: Based on the rewritten error system, use gradient descent to construct the neural network weight update rate of the corresponding fuzzy logic system that minimizes the rewritten Hamiltonian-Jacobi-Isax equation. for; , In the formula: Represents the learning rate of the weights and .

[0013] Furthermore, step S6 specifically includes the following steps: S61: Based on the neural network weight update rate, and according to the output estimates of the ship's feedforward controller combined with the optimal feedback controller and the worst-case disturbance strategy, the final ideal controller of the ship is constructed as follows: , S62: Constrain the control input of the ship's final ideal controller by the input constraints in the ship's physical model to obtain the actual control input; realize zero-sum game tracking control of the ship based on data physics dual-drive reinforcement learning according to the actual control input.

[0014] This invention provides a zero-sum game tracking control method for ships based on dual-driven reinforcement learning of numbers and objects, with the following beneficial effects: 1. By deeply integrating the physical model with navigation data such as ship speed, a data-physical dual-drive model of "physical mechanism + data compensation" is constructed. This model not only fully preserves the interpretability and stability of the physical model, but also effectively avoids the dependence of data-driven methods on high-quality data, and can achieve efficient and stable control without relying on precise ship parameters.

[0015] 2. Based on the data-physics dual-drive model, the feedforward controller of the ship is designed hierarchically using the Lyapunov backstepping method to achieve accurate tracking of the ship's nominal dynamics, providing a solid foundation for the stability of the closed-loop system.

[0016] 3. By organically integrating zero-sum game differential theory with adaptive dynamic programming based on fuzzy logic systems, the optimal feedback controller and the worst-case disturbance strategy are simultaneously optimized, constructing a dynamic adversarial framework between the controller and the disturbance. This framework can accurately capture the "zero-sum game" essence of the controller and complex disturbances, achieving the optimal feedback control solution under the worst-case disturbance strategy. This significantly improves the ship's adaptability and anti-interference performance to random ocean disturbances, ensuring stable tracking of the target track even in complex scenarios where its physical constraints and dynamic characteristics are completely unknown.

[0017] 4. By integrating the tracking capability of feedforward control, the optimization capability of optimal feedback control, and the adversarial adaptation capability of worst-case disturbance strategies, an ideal control input is obtained, and physical constraint processing is used to generate directly executable actual control commands. This invention deeply integrates data-physical dual-drive, zero-sum game theory, and reinforcement learning, which not only solves the problems of accuracy and robustness in ship tracking control in complex marine environments, but also promotes the intelligent and adaptive upgrading of ship tracking control systems, providing reliable technical support for autonomous ship operations. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart of a zero-sum game tracking control method for ships based on dual-drive reinforcement learning of numbers and objects, according to the present invention. Figure 2 This is a schematic diagram illustrating the error between the actual model and the data physics dual-drive model in this embodiment; Figure 3 The position coordinates of the ship in this embodiment Simulation diagram of the tracking effect; Figure 4 In this embodiment, the ships are respectively in The simulation results of tracking the reference signal are shown above; Figure 5 This is a simulation diagram showing the convergence result of the weight norm of the fuzzy logic system in this embodiment; Figure 6 This is a simulation diagram showing the convergence results of the RNN weight norm and feedback term parameters in this embodiment; Figure 7 This is a simulation diagram showing the results of the ship's forward force, lateral drift force, and bow roll moment controller in this embodiment; Figure 8 This is a simulation diagram showing the results of the worst-case interference strategy for ships in this embodiment. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] This embodiment provides a zero-sum game tracking control method for ships based on dual-driven reinforcement learning of numbers and objects, such as... Figure 1 As shown, the specific steps include: S1: Obtain a three-degree-of-freedom ship physical model considering unknown wind, wave, and current disturbances, input constraints, and unknown dynamics; specifically, the ship physical model is the ship's kinematics and dynamics model, and its expression is: , In the formula: Indicates the ship in the inertial coordinate system coordinate, Coordinates and bow roll angle The position vector formed and ; Represents the forward speed of the ship in the attached coordinate system. Horizontal drift speed and bow roll rate The velocity vector formed and ; Denotes the coordinate system transformation matrix and ; This indicates the actual forward control force that the ship can directly execute. Lateral drift force and bow roll torque The control vector formed and ; This represents the lateral disturbance force on a ship caused by wind, waves, and currents in the attached coordinate system. Longitudinal interference force and bow disturbance torque The perturbation vector formed and ,satisfy ; express The upper bound of a positive constant; The matrix represents the ship's weight inertia and hydrodynamic additional inertia, and ; Describe the Coriolis matrix and ; Represents the hydrodynamic damping parameter matrix and ; Indicates transpose; Represent the space of real numbers; express The first derivative; And matrix , , , They are represented in the following forms respectively: , , , , In the formula: Ship mass; Indicates the moment of inertia; Represents the attached coordinate system Components along the axial direction; Represents the hydrodynamic derivatives in each direction of the attachment coordinates; The input constraints, which take into account the physical constraints of the actual controller, are defined as follows: , In the formula: Represents the control forces of each degree of freedom in the ship's physical model and ; Indicates the ideal control inputs for the ship's longitudinal, lateral, and bow directions; This indicates the maximum value of the control input for each degree of freedom of the ship; This represents the minimum control input value for each degree of freedom of the ship.

[0022] S2: Based on the ship's physical model and combined with a recurrent neural network, a data-physical dual-drive model is constructed. In this embodiment, based on the ship's physical model, speed and attitude data are collected through a Doppler log, and a data-physical dual-drive model is constructed by combining it with a recurrent neural network. The specific steps include: S21: The nominal dynamic model obtained from the ship's physical model, without considering input constraints, is as follows: , In the formula: Represents the ideal control input vectors for the ship's longitudinal, lateral, and bow directions, and ; S22: Based on the Stone-Weilstras theorem, the nominal dynamic model of the ship is rewritten using RNN basis functions as follows: , In the formula: This represents the ideal weights of the RNN basis functions. , And satisfy ; This represents the estimated values ​​of the RNN network weights; This represents the error between the ideal weights and the estimated weights of the RNN basis functions; Indicates reconstruction error and ; Describes a vector composed of RNN basis functions and ; The basis functions of a monotonically increasing RNN are those that satisfy: ; Describe the independent variable of the basis function; Represents positive integers; S23: Based on the estimated values ​​of the RNN weights The data-physical dual-drive model is determined, and based on the rewritten nominal dynamics model, the approximate dynamic system of the ship is constructed as follows: , , In the formula: Represents the estimated value of the ship data physical dual-drive model and satisfies ; Indicates modeling error; This represents the reconstruction error compensation feedback term; This represents the gain coefficient of the linear feedback term. ; The gain parameter estimate of the nonlinear feedback term and ; The gain parameter represents the ideal nonlinear feedback term; This represents the gain parameter error of the nonlinear feedback term; express The first derivative; This embodiment also includes the handling of modeling errors. Differentiating, we get: , In the formula: ; The derivative of the modeling error Perform stability analysis and design estimates for the RNN weights. Gain parameter estimates of the nonlinear feedback term The update law is used to obtain the gain parameters of the RNN weights and nonlinear feedback terms; The expression for the update law of the estimated weights of the RNN is: , In the formula: ; This represents the learning rate of the RNN. ; The gain parameter estimate of the nonlinear feedback term The expression for the update law is: , In the formula: This represents a predefined positive definite matrix; S24: Based on the approximate dynamics system of the ship and the aforementioned physical model of the ship, a three-degree-of-freedom data-physical dual-drive model is constructed, which does not consider input constraints but does consider unknown external environmental disturbances: ; S3: Based on the Lyapunov backstepping method, define the ideal control input of the data-physical dual-drive model. Based on the preset desired ship trajectory, obtain the trajectory tracking error according to the data-physical dual-drive model, and construct a ship virtual controller based on the trajectory tracking error. Obtain the ship speed error from the ship virtual controller to define intermediate parameters for optimal feedback control design. After differentiating the ship speed error, obtain the rewritten ship speed error derivative based on the intermediate parameters and the ideal control input. Design a ship feedforward controller based on the rewritten ship speed error derivative. Define the Lyapunov function based on the trajectory tracking error and the ship speed error, and obtain the derivative of the Lyapunov function by differentiation. The specific steps include: S31: Based on Lyapunov's backstepping method, define the ideal control input for the data-physical dual-drive model. The expression for the ideal control input is: + , In the formula: Express the output of the feedforward controller; Express the output of the optimal feedback controller; This represents the output of the worst-case interference strategy; S32: Based on the defined data-physical dual-drive model, the ship's trajectory tracking error is: , In the formula: Indicates the tracking error; Indicates the track reference signal and ; S33: Obtain the derivative of the trajectory tracking error by taking the derivative of the trajectory tracking error, and rewrite the derivative of the trajectory tracking error according to the data-physics dual-drive model as follows: , In the formula: express The first derivative; S34: Based on the rewritten derivative of the trajectory tracking error, the ship virtual controller is designed as follows:

[0023] In the formula: Indicates the control design parameters and ; This represents the output of the ship's virtual controller; S35: Obtain the ship speed error based on the ship virtual controller and the data physics dual-drive model to define intermediate parameters for optimal feedback control design. The expression for these parameters is: , , , , In the formula: Indicates the error in ship speed; Indicates intermediate variables; S36: Obtain the derivative of the ship speed error by taking the derivative of the ship speed error, and obtain the rewritten derivative of the ship speed error based on the intermediate parameters and the ideal control input; , In the formula: express The first derivative; express The first derivative; S37: Obtain the modified ship speed error derivative and perform stability analysis on the modified ship speed error derivative in order to design the ship feedforward controller: , In the formula: Indicates the control design parameters and ; This indicates the output of the ship's feedforward controller; S38: To prove the stability analysis proposed in step S36, a Lyapunov function is defined based on the track tracking error and the ship speed error. for: , And for Lyapunov functions Take the derivative, and substitute the aforementioned track tracking error, ship speed error, virtual controller, and feedforward controller into the equation to obtain the derivative of the Lyapunov function. for: ; S4: Define the error system based on the derivative of the Lyapunov function; construct a performance index function based on the error system and zero-sum game theory; construct the Hamilton-Jacobi-Isax equation based on the performance index function to obtain the optimal feedback controller and the worst disturbance strategy. The specific steps include: S41: According to differential game theory, and combining the definition of the error system based on the last term in the derivative of the Lyapunov function, the error system is as follows: , S42: Based on the error system and zero-sum game theory, the performance index function under zero-sum game is defined as follows: , In the formula: Indicates time parameters; Indicates the discount factor; Indicates the permissible control inputs; , , Denotes the positive definite gain matrix and , , ; S43: Based on the Nash-Pontryagin maximum-minimum principle, it is confirmed that the zero-sum game has a unique solution. The condition is that there exists a unique saddle point such that the optimal performance index function... for: , S44: Based on the performance index function, define the Hamiltonian function of the error system, whose expression is: , In the formula: express about The partial derivative; express Transpose of; Based on the Hamiltonian function and the optimal performance index function The Hamilton-Jacobi-Isax equation (HJI) is obtained as follows:

[0024] In the formula: express about The partial derivative; S45: Construct the Hamilton-Jacobi-Isax equation based on the performance index function, and use the gradient descent method to obtain the optimal feedback controller and the worst-case disturbance strategy: , , S5: Using a fuzzy logic system to approximate the optimal performance index function to obtain the output estimates of the optimal feedback controller and the worst interference strategy. In this embodiment, based on the above differential game framework, the fuzzy logic system is used to approximate the solution of the HJI equation to obtain the Nash equilibrium of the output estimates of the optimal feedback controller and the worst interference strategy, and the neural network weight update rate of the fuzzy logic system is constructed based on the output estimates of the optimal feedback controller and the worst interference strategy. The specific steps include: S51: The optimal performance index function is approximated using a fuzzy logic system, and its expression is: , In the formula: Represents the ideal weights and satisfies ; This represents the error in the weights of the neural network; express The estimated value; Represents fuzzy basis functions; Indicates the approximation error; S52: Obtain the approximate optimal performance index function based on S51. partial derivatives for: , In the formula: express about The partial derivative; express about The partial derivative; S53: Based on weight The estimated value Determine the evaluation network and obtain the partial derivative estimate of the optimal performance index function based on S52. for: , S54: Partial derivative as described in S52 Substituting this into the optimal feedback controller and worst-case disturbance strategy in S45 and the Hamilton-Jacobi-Isax equation in S44, we obtain: , , , In the formula: Indicates the reconstructed residual and ; Indicates intermediate parameters and ; express The abbreviated form; Similarly, the partial derivative estimates obtained from S53 are... Substituting the optimal feedback controller and worst-case disturbance strategy in S45 and the Hamilton-Jacobi-Isaks equation in S54, we obtain the output estimate of the optimal feedback controller, the output estimate of the worst-case disturbance strategy, and the rewritten Hamilton-Jacobi-Isaks equation. , , , S55: Substitute the output estimates of the optimal feedback controller and the worst-case disturbance strategy into the error system in S41 to obtain the rewritten error system: , S56: Based on the rewritten error system, use gradient descent to construct the neural network weight update rate of the corresponding fuzzy logic system that minimizes the rewritten Hamiltonian-Jacobi-Isax equation. for; , In the formula: Represents the learning rate of the weights and ; S6: Based on the neural network weight update rate, and according to the output estimates of the ship's feedforward controller combined with the optimal feedback controller and the worst-case disturbance strategy, a final ideal controller for the ship is constructed. The control input of the final ideal controller is constrained by the input constraints in the ship's physical model to obtain the actual control input. Based on the actual control input, zero-sum game tracking control of the ship based on data-physics dual-drive reinforcement learning is implemented. This embodiment achieves ship trajectory tracking by fusing the feedforward controller, the optimal feedback controller, and the worst-case disturbance strategy, and obtaining directly executable actual control commands through constraint processing. Specific steps include: S61: Based on the neural network weight update rate, and according to the output estimates of the ship's feedforward controller combined with the optimal feedback controller and the worst-case disturbance strategy, the final ideal controller of the ship is constructed as follows: , In the formula: The estimated weights of the RNN are obtained by processing ship speed data collected by a Doppler log. Indicates the tracking error and Indicates the velocity error for each degree of freedom; , , , Indicates the design parameters; Represents fuzzy basis functions; Represents the coordinate transformation matrix; S62: Constrain the control input of the ship's final ideal controller by the input constraints in the ship's physical model to obtain the actual control input; realize the ship's zero-sum game tracking control based on data physics dual-drive reinforcement learning according to the actual control input, thereby realizing the zero-sum game trajectory tracking control of the ship under random disturbances in the marine environment, its own physical constraints and dynamic characteristics are completely unknown.

[0025] This embodiment also includes step S7: using Lyapunov theory, proving the stability of the input to the state of the designed ship zero-sum game tracking control method based on data physics dual-driven reinforcement learning, and that all signals in the closed-loop system are eventually uniformly bounded.

[0026] Specifically, to more easily prove stability, we make the following assumptions: Assumption 1: RNN reconstruction error It is bounded, and the upper and lower bounds are the modeling errors. The function, i.e., the expression is . Assumption 2: In compact sets superior, , , and All are bounded, satisfying , , , ,and and ,in , , , , and All are positive parameters.

[0027] Theorem 1: For a ship trajectory tracking control system considering unknown wind, wave and current disturbances, input constraints and unknown dynamics, based on Lyapunov theory, under the action of the designed ship data-physical dual-drive model, RNN weight adaptive law, feedforward controller, output estimate of the optimal feedback controller, output estimate of the worst disturbance strategy and weight adaptive law of fuzzy logic system, by selecting appropriate parameters, it can be guaranteed that all signals of the closed-loop system are eventually uniformly bounded and the tracking error converges in an optimal manner.

[0028] Lemma 1: (Young's inequality) For any The following inequalities hold: , In the formula: , , and ; Proof of Theorem 1: S71: To prove that the ship data physics dual-drive model and the RNN weights are eventually uniformly bounded, construct the Lyapunov function. And the expression for the Lyapunov function is: , S72: For Lyapunov functions Find the derivative and convert it to the modeling error derivative. and the derivative of the RNN weight error and the gain parameter of the nonlinear feedback term Substituting, we can obtain: , Combining the monotonically increasing property of RNN basis functions in step S22, Assumption 1, and Lemma 1 (Young's inequality), we can obtain the following inequality: , Substituting the derivative of the Lyapunov function We can obtain: , In the formula: express The identity matrix; Therefore, when choosing the appropriate Value, making This ensures that the ship data physical dual-drive model and the RNN weights are eventually consistent and bounded; S73: Next, it is proved that the weights of the data-physical dual-drive ship error system and the fuzzy logic system are ultimately uniformly bounded, based on the Hamiltonian function in step S54. The following equation can be obtained; , Therefore, substituting the equation in S73 into the fuzzy logic system weight adaptive law in step S55... We can obtain: , S74: Lyapunov function based on feedforward control design in step S38 Construct new Lyapunov functions The expression is: , For the above Lyapunov function Find the derivative and substitute the relevant equations from steps S38, S41, and S73 into the differentiated Lyapunov function. We can obtain: , S75: Based on assumption 2, take one of them as an example to illustrate the scaling process. , The remaining items are processed in the same way, and we can obtain...

[0029] In the formula: ; ; ; ; , Therefore, when satisfied This allows for the selection of appropriate parameters to ensure that all signals of the data-physical dual-drive ship trajectory tracking control system are ultimately consistent and bounded.

[0030] Compared with existing technologies, the method described in this embodiment has the following advantages: 1. The method described in this embodiment deeply integrates the prior physical laws contained in the ship's physical model with the ship speed data collected by the sensor to construct a ship data-physical dual-drive model. This model not only retains the interpretability and stability of the physical model, but also effectively avoids the dependence of data-driven methods on high-quality data. It can achieve efficient and stable control without the need for precise ship parameters, which greatly reduces the engineering application threshold and implementation cost of the technology.

[0031] 2. The method described in this embodiment constructs a dynamic adversarial mechanism between the controller and the disturbance through a zero-sum game framework. Combined with reinforcement learning and fuzzy logic system, it achieves the optimal control solution under the worst disturbance strategy, significantly improving the ship's adaptive ability to complex disturbances such as wind, waves and currents.

[0032] 3. The method described in this embodiment was simulated using the MATLAB platform. A zero-sum game optimal controller was designed to achieve ship track tracking in unknown and complex marine environments. The system controller is more in line with navigation practice, further verifying the effectiveness and rationality of the method described in this embodiment.

[0033] To verify the effectiveness of the method described in this embodiment, a computer simulation study was conducted using MATLAB. The parameter settings are as follows: The simulation object selected is the CyberShipII from the Norwegian University of Science and Technology, with a weight of 23.8 kg, a length of 1.255 m, and a beam of 0.29 m. Parameters were set when designing the data-physics dual-drive model. , , , , , RNN weight initial values For the corresponding dimension, the zero matrix, basis functions Initial values ​​of feedback parameters When designing an ideal controller, the design parameters are... , , , , , The initial value of the weights in the fuzzy logic system is The basis functions are Center point selection The reference signal for the ship is set as follows: The initial state of the ship's control system is... .

[0034] The simulation results are shown in the figure: Figure 2 This is a schematic diagram of the error in the dual-driven model of the actual model and the data physics model. It can be seen that the error has successfully converged to the bounded neighborhood centered on the origin. Figure 3 For position coordinates The trajectory tracking effect on the plane shows that the output position signal can quickly track the reference signal; Figure 4 Then from position and heading The tracking effect is intuitively displayed in three dimensions; Figure 5 The figure shows the convergence curve of the weight norm of the fuzzy logic system. The results show that the weight norm can converge stably to a certain fixed value. Figure 6 The figure shows the convergence process of the RNN weight norm and feedback parameters. The weight norm and parameters can converge to stable values. Figure 7 The curve shows the actual change in the ship's control input, indicating that the control signal is relatively stable. Figure 8 The curve represents the variation of the worst-case interference strategy. Simulation results demonstrate that the effectiveness of the method described in this embodiment is verified, and the zero-sum game controller designed based on this control system can effectively track ship tracks, with the tracking error quickly converging to a small residual set. This fully illustrates that the output of the data-physical dual-drive ship tracking system has good tracking performance, further verifying the effectiveness and rationality of the method described in this embodiment.

[0035] In summary, the method described in this embodiment establishes a three-degree-of-freedom ship physical model considering input constraints, unknown wind, wave, and current disturbances, and unknown dynamics. It collects key navigation data such as ship speed and attitude using a Doppler log, and combines this with the data learning capabilities of recurrent neural networks to construct a data-physical dual-drive model that combines physical interpretability with data adaptive compensation capabilities. Based on this model, a Lyapunov backstepping method is used to hierarchically design the ship's feedforward controller, achieving accurate tracking of the ship's nominal dynamics. Then, through zero-sum game differential theory and adaptive dynamic programming based on fuzzy logic systems, the optimal feedback controller and worst-case disturbance strategy are simultaneously optimized, constructing a dynamic adversarial framework between the controller and disturbances. Finally, the feedforward control, optimal feedback control, and worst-case disturbance strategy are integrated to obtain the ideal control input, which is then processed through physical constraints to obtain the directly executable actual control input. This enables the ship to achieve trajectory tracking control under completely unknown conditions of random disturbances in the marine environment, its own physical constraints, and dynamic characteristics. The method described in this embodiment breaks through the limitations of traditional single modeling methods by deeply integrating data and physics, and can achieve efficient and stable control without relying on precise ship parameters. In addition, by organically combining zero-sum game theory and reinforcement learning, it significantly improves the ship's adaptability and anti-interference performance to complex disturbances, promotes the intelligent upgrade of the ship tracking and control system, and provides a reliable technical solution for the precise tracking and control of ships in diverse scenarios such as near-shore operations and ocean shipping.

[0036] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A ship zero-sum game tracking control method based on dual-driven reinforcement learning of numbers and objects, characterized in that, The specific steps include: S1: Obtain a ship physical model that takes into account unknown wind, wave and current disturbances, input constraints and unknown dynamics; S2: Construct a data physics dual-drive model based on the ship physics model and a recurrent neural network; S3: Based on the Lyapunov backstepping method, define the ideal control input of the data-physical dual-drive model. Based on the preset desired ship trajectory, obtain the trajectory tracking error according to the data-physical dual-drive model, and construct a ship virtual controller based on the trajectory tracking error. Obtain the ship speed error from the ship virtual controller to define intermediate parameters for optimal feedback control design. After differentiating the ship speed error, obtain the rewritten ship speed error derivative based on the intermediate parameters and the ideal control input. Design a ship feedforward controller based on the rewritten ship speed error derivative. Define the Lyapunov function based on the trajectory tracking error and the ship speed error, and obtain the derivative of the Lyapunov function by differentiation. S4: Define the error system based on the derivative of the Lyapunov function; construct a performance index function based on the error system and zero-sum game theory; construct the Hamilton-Jacobi-Isax equation based on the performance index function to obtain the optimal feedback controller and the worst disturbance strategy. S5: Use the fuzzy logic system to approximate the optimal performance index function to obtain the output estimates of the optimal feedback controller and the worst interference strategy, and construct the neural network weight update rate of the fuzzy logic system based on the output estimates of the optimal feedback controller and the worst interference strategy. S6: Based on the neural network weight update rate, construct the ship's final ideal controller according to the output estimate of the ship's feedforward controller combined with the optimal feedback controller and the worst disturbance strategy; By constraining the control input of the ship's final ideal controller through input constraints in the ship's physical model, the actual control input is obtained, and ship zero-sum game tracking control based on data physics dual-driven reinforcement learning is realized based on the actual control input.

2. The ship zero-sum game tracking control method based on dual-driven reinforcement learning according to claim 1, characterized in that, The expression for the ship's physical model described in S1 is: In the formula: Indicates the ship in the inertial coordinate system coordinate, Coordinates and bow roll angle The position vector formed and ; Represents the forward speed of the ship in the attached coordinate system. Horizontal drift speed and bow roll rate The velocity vector formed and ; Denotes the coordinate system transformation matrix and ; This indicates the actual forward control force that the ship can directly execute. lateral drift force and bow roll torque The control vector formed and ; This represents the lateral disturbance force on a ship caused by wind, waves, and currents in the attached coordinate system. Longitudinal interference force and bow disturbance torque The perturbation vector formed and ,satisfy ; express The upper bound of a positive constant; The matrix represents the ship's weight inertia and hydrodynamic additional inertia, and ; Describe the Coriolis matrix and ; Represents the hydrodynamic damping parameter matrix and ; Indicates transpose; Represent the space of real numbers; express The first derivative; The input constraints are defined as follows: In the formula: Represents the control forces of each degree of freedom in the ship's physical model and ; Indicates the ideal control inputs for the ship's longitudinal, lateral, and bow directions; This indicates the maximum value of the control input for each degree of freedom of the ship; This represents the minimum control input value for each degree of freedom of the ship.

3. The ship zero-sum game tracking control method based on dual-driven reinforcement learning according to claim 2, characterized in that, S2 specifically includes the following steps: S21: The nominal dynamic model obtained from the ship's physical model, without considering input constraints, is as follows: In the formula: Represents the ideal control input vectors for the ship's longitudinal, lateral, and bow directions, and ; S22: Based on the Stone-Weilstras theorem, the nominal dynamic model of the ship is rewritten using RNN basis functions as follows: In the formula: Describes the ideal weights of the RNN basis functions and satisfies ; This represents the estimated values ​​of the RNN network weights; This represents the error between the ideal weights and the estimated weights of the RNN basis functions; Indicates reconstruction error and ; Describes a vector composed of RNN basis functions and ; Represents the monotonically increasing basis functions of an RNN; S23: Based on the revised nominal dynamics model, construct the approximate dynamics system of the ship as follows: In the formula: Represents the estimated value of the ship data physical dual-drive model and satisfies ; Indicates modeling error; This represents the reconstruction error compensation feedback term; This represents the gain coefficient of the linear feedback term. ; The gain parameter estimate of the nonlinear feedback term and ; The gain parameter represents the ideal nonlinear feedback term; This represents the gain parameter error of the nonlinear feedback term; express The first derivative; S24: Based on the ship's approximate dynamics system and the aforementioned ship physical model, a data-physical dual-drive model is constructed as follows: 。 4. The ship zero-sum game tracking control method based on dual-driven reinforcement learning according to claim 3, characterized in that, S3 specifically includes the following steps: S31: Based on Lyapunov's backstepping method, define the ideal control input for the data-physical dual-drive model. The expression for the ideal control input is: + In the formula: Express the output of the feedforward controller; Express the output of the optimal feedback controller; This represents the output of the worst-case interference strategy; S32: Based on the defined data-physical dual-drive model, the ship's trajectory tracking error is: In the formula: Indicates the tracking error; Indicates the track reference signal and ; S33: Obtain the derivative of the trajectory tracking error by taking the derivative of the trajectory tracking error, and rewrite the derivative of the trajectory tracking error according to the data-physics dual-drive model as follows: In the formula: express The first derivative; S34: Based on the rewritten derivative of the trajectory tracking error, the ship virtual controller is designed as follows: In the formula: Indicates the control design parameters and ; This represents the output of the ship's virtual controller; S35: Obtain the ship speed error based on the ship virtual controller and the data physics dual-drive model to define intermediate parameters for optimal feedback control design. The expression for these parameters is: ， In the formula: Indicates the error in ship speed; Indicates intermediate variables; S36: Obtain the derivative of the ship speed error by taking the derivative of the ship speed error, and obtain the rewritten derivative of the ship speed error based on the intermediate parameters and the ideal control input; In the formula: express The first derivative; express The first derivative; S37: Based on the rewritten derivative of the ship speed error, the ship feedforward controller is designed as follows: In the formula: Indicates the control design parameters and ; This indicates the output of the ship's feedforward controller; S38: Define the Lyapunov function based on track tracking error and ship speed error. for: And for Lyapunov functions Find the derivative of the Lyapunov function. for: 。 5. The ship zero-sum game tracking control method based on dual-driven reinforcement learning according to claim 4, characterized in that, S4 specifically includes the following steps: S41: The derivative of the Lyapunov function... The term is defined as an error system: S42: Based on the error system and zero-sum game theory, the performance index function under zero-sum game is defined as follows: In the formula: Indicates time parameters; Indicates the discount factor; Indicates the permissible control inputs; , , Denotes the positive definite gain matrix and , , ; S43: Based on the Nash-Pontryagin maximum-minimum principle, it is confirmed that the zero-sum game has a unique solution. The condition is that there exists a unique saddle point such that the optimal performance index function... for: S44: Based on the performance index function, define the Hamiltonian function of the error system, whose expression is: In the formula: express about The partial derivative; express transpose; Based on the Hamiltonian function and the optimal performance index function The Hamilton-Jacobi-Isax equations are obtained as follows: In the formula: express about The partial derivative; S45: Construct the Hamilton-Jacobi-Isax equation based on the performance index function, and use the gradient descent method to obtain the optimal feedback controller and the worst-case disturbance strategy: ， 。 6. The ship zero-sum game tracking control method based on dual-drive reinforcement learning of numbers and objects according to claim 5, characterized in that, S5 specifically includes the following steps: S51: The optimal performance index function is approximated using a fuzzy logic system, and its expression is: In the formula: Represents the ideal weight and satisfies ; This represents the error in the weights of the neural network; express The estimated value; Representing fuzzy basis functions; Indicates the approximation error; S52: Obtain the approximate optimal performance index function based on S51. partial derivatives for: In the formula: express about The partial derivative; express about The partial derivative; S53: Based on weight The estimated value Determine the evaluation network and obtain the partial derivative estimate of the optimal performance index function based on S52. for: S54: Partial derivative as described in S52 Substituting this into the optimal feedback controller and worst-case disturbance strategy in S45 and the Hamilton-Jacobi-Isax equation in S44, we obtain: In the formula: Indicates the reconstructed residual and ; Indicates intermediate parameters and ; express The abbreviated form; Similarly, the partial derivative estimates obtained from S53 are... Substituting the optimal feedback controller and worst-case disturbance policy from S45 and the Hamilton-Jacobi-Isax equation from S54, we obtain the output estimate of the optimal feedback controller. The output estimate of the worst-case interference strategy And the rewritten Hamilton-Jacobi-Isaks equation is; S55: Substitute the output estimates of the optimal feedback controller and the worst-case disturbance strategy into the error system in S41 to obtain the rewritten error system: S56: Based on the rewritten error system, use gradient descent to construct the neural network weight update rate of the corresponding fuzzy logic system that minimizes the rewritten Hamiltonian-Jacobi-Isax equation. for; In the formula: Indicates the learning rate of the weights and .

7. The ship zero-sum game tracking control method based on dual-drive reinforcement learning of numbers and objects according to claim 6, characterized in that, S6 specifically includes the following steps: S61: Based on the neural network weight update rate, and according to the output estimates of the ship's feedforward controller combined with the optimal feedback controller and the worst-case disturbance strategy, the final ideal controller of the ship is constructed as follows: S62: Constrain the control input of the ship's final ideal controller by the input constraints in the ship's physical model to obtain the actual control input; realize zero-sum game tracking control of the ship based on data physics dual-drive reinforcement learning according to the actual control input.

Images