A ship lifelong learning control method and system considering state constraints
By constructing an Actor-Critic neural network architecture and a lifetime reinforcement learning mechanism, the problems of state constraints and catastrophic forgetting in ship control are solved, thereby improving safety and reliability and ensuring the multi-task performance of ships in complex environments.
Patent Information
- Application Number
- CN202610500695.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-16
- Publication Date
- 2026-06-19
AI Technical Summary
Existing ship control technologies suffer from deficiencies and catastrophic forgetting in state constraint processing and online learning, resulting in insufficient system safety and reliability.
A lifelong learning control method for ships that considers state constraints is constructed. By using an Actor-Critic neural network architecture and combining the optimal attitude and velocity logarithmic obstacle cost function, a lifelong reinforcement learning mechanism is introduced to realize online processing of state constraints and weight updates.
It effectively prevents the risk of collision and instability of ships in complex environments, ensures control performance and energy-saving optimization in multi-task operations, and improves the safety and robustness of the system.
Smart Images
Figure CN122239418A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the interdisciplinary field of ship motion control and artificial intelligence, and specifically relates to a lifelong learning control method and system for ships that considers state constraints. Background Technology
[0002] Maritime transport, a pillar of global trade, handles over 80% of bulk commodity transportation. In 2018, the International Maritime Organization (IMO) began studying the intelligence levels of autonomous surface vessels (ASMEs), classifying them into four categories: autonomous vessels with automated processes and crew decision support, remotely controlled vessels with crew, remotely controlled vessels without crew, and fully autonomous vessels. Significant progress has been made in ship assisted navigation, and the development of remotely controlled and fully autonomous vessels is a consensus both domestically and internationally, a cutting-edge technological field, and a hot topic in interdisciplinary research.
[0003] As the primary carrier of waterway transportation, the development of ships has dramatically changed the way we operate at sea. During navigation, ships are not only affected by time-varying external environmental disturbances such as wind, waves, and currents, but also face internal challenges such as uncertainty, sensor noise, and strict energy constraints. Existing ship control technologies have mainly gone through three stages: the first stage adopted traditional control methods such as PID and SMC; the second stage adopted adaptive methods such as MPC, fuzzy control, and adaptive backstepping control; and the third stage adopted intelligent control methods based on reinforcement learning and deep reinforcement learning. In particular, to reduce energy consumption while ensuring ship control accuracy, intelligent optimal control methods based on the Actor-Critic architecture are widely used in nonlinear systems because they can solve the Hamilton-Jacobi-Bellman equations online.
[0004] Despite the progress made in existing intelligent optimal control methods, the following two major technical bottlenecks still exist in practical engineering applications: First, there is a lack of attention to state constraint handling. In actual navigation, to ensure system safety, the ship's state (such as position error, speed, and heading angle) must be strictly limited within a preset safety range. However, most existing studies have neglected output constraints and full-state constraints. If the system cannot guarantee that its state is within the safety boundaries, it may lead to performance degradation or even catastrophic system failures (such as collisions or capsizing).
[0005] Second, there is the problem of "catastrophic forgetting" in online learning. Maritime missions are often complex, requiring ships to sequentially perform multiple different maneuvers (such as straight-line cruising, sharp turns, obstacle avoidance, etc.) or operate continuously under constantly changing hydrodynamic environments and disturbances. Traditional adaptive methods based on online neural networks often overwrite previously learned network weights when learning new tasks or adapting to new environments, causing the controller to "forget" experience from old tasks—a phenomenon known as "catastrophic forgetting." While some continuous learning techniques (such as regularization and memory replay) have been proposed, most are suitable for offline training and are difficult to directly transfer to online control systems with extremely high real-time and stability requirements. Existing few online attempts either lack proven stability or only apply protection mechanisms to parts of the network (such as only the evaluation network), leaving the action network still at risk of forgetting and failing to meet the reliability requirements of long-term, continuous, multi-task operations. Summary of the Invention
[0006] The technical problem to be solved by the present invention is to provide a lifelong learning control method and system for ships that considers state constraints, for optimal control of ship trajectory tracking under state constraints.
[0007] The technical solution adopted by this invention to solve the above-mentioned technical problems is as follows: a lifelong learning control method for ships considering state constraints, comprising the following steps: S1: Establish the kinematic and dynamic model of the ship, and calculate the attitude tracking error based on the ship's position, heading, and reference trajectory; S2: Construct the optimal attitude logarithmic obstacle cost function based on attitude tracking error; S3: Construct the first-layer Actor-Critic neural network architecture, derive the optimal virtual control law, and approximate the optimal attitude logarithmic obstacle cost function; S4: Based on the ship speed variable and the optimal virtual control law, calculate the speed tracking error and construct the optimal logarithmic barrier cost function for speed; S5: Construct the second-layer Actor-Critic neural network architecture, derive the optimal control law, and approximate the optimal logarithmic speed barrier cost function; S6: Based on the lifelong reinforcement learning mechanism, reconstruct the neural network weight update law in the first and second layer Actor-Critic neural network architecture.
[0008] According to the above scheme, in step S2, It is attitude tracking error. and For positive integers, It is the ideal optimal virtual control law. It is a state constraint boundary for position and heading angle, and an optimal attitude logarithmic obstacle-type cost function. Represented as: ; In step S4, It is a speed tracking error. and For positive integers, It is the ideal optimal virtual control law. It is a boundary constraint function for sway velocity, roll velocity, and bow angular velocity, with an optimal logarithmic barrier. Represented as: .
[0009] Furthermore, and As the initial boundary, and To shorten the convergence time, and User-defined convergence boundaries, position and heading angle state constraints And the boundary constraints for sway velocity, roll velocity, and pitch angular velocity. They are represented as follows: .
[0010] According to the above scheme, in step S3, and These are the positive design parameters for the control law. It is the action neural network estimate in the first layer of the Actor-Critic neural network architecture. These are the basis functions for the first layer of the Actor-Critic neural network architecture. basis functions The optimal virtual control law, expressed as the transpose of the partial derivative with respect to the attitude error, is: ; and These are the positive design parameters for the control law. It is the action neural network estimate in the second-layer Actor-Critic neural network architecture. These are the basis functions for the second-layer Actor-Critic neural network architecture. basis functions Regarding the transpose of the partial derivative with respect to the velocity error, in step S5, the optimal control law is expressed as: .
[0011] According to the above scheme, in step S6, This serves as a performance metric for evaluating the first-layer Actor-Critic neural network architecture. This refers to the performance metrics of the action network in the first layer of the Actor-Critic neural network architecture. This serves as a performance metric for evaluating the second-layer Actor-Critic neural network architecture. This refers to the performance metrics of the action network in the second-layer Actor-Critic neural network architecture. , , as well as ( () represents the design parameters. and For Fisher's information matrix, It is the first j Step the comment weight vector of the current task, It is the first j Step the action weight vector of the current task. It is the previous task number j The bounded evaluation weight vector of the step. It is the previous task number j The bounded action weight vector of the step. It is the first j The loss function of the evaluation network in a multi-layered Actor-Critic neural network architecture. It is the first j The performance metrics for the evaluation network and action network, which are constructed based on the error of the action network and the first and second layer Actor-Critic neural network structures, are as follows: .
[0012] Furthermore, in step S6, based on the performance metrics of the evaluation network and the performance metrics of the action network of the first-layer Actor-Critic neural network architecture, the normalized gradient descent method is used to design the weight update law of the first-layer Actor-Critic neural network. , and For design parameters, basis functions Taking the partial derivative of the attitude error , It is the Jacobian matrix of the action network in the first layer of the Actor-Critic neural network architecture, which is equal to about The derivative of The derivative of the desired trajectory; the action network update law of the first-layer Actor-Critic neural network architecture is: ; It is the Jacobian matrix of the evaluation network of the first-layer Actor-Critic neural network architecture, which is equal to about The derivative of ; the update law of the evaluation network in the first layer of the Actor-Critic neural network architecture is: .
[0013] Furthermore, in step S6, based on the performance metrics of the evaluation network and the performance metrics of the action network in the second-layer Actor-Critic neural network architecture, the normalized gradient descent method is used to design the weight update law of the second-layer Actor-Critic neural network. , and For design parameters, basis functions Taking the partial derivative with respect to the speed error , It is the Jacobian matrix of the action network in the second-layer Actor-Critic neural network architecture, which is equal to about The derivative; To approximate the model uncertainty of a ship, a radial basis function neural network is established to approximate the lumped uncertainty in the ship dynamics model online. The uncertainty includes unmodeled hydrodynamic damping and external wind and wave disturbance torque. The derivative of a first-order filter is given by: , For the error of the first-order filter, The filtering vector for the optimal virtual control law; the action network update law of the second-layer Actor-Critic neural network architecture is: ; It is the Jacobian matrix of the evaluation network in the second-layer Actor-Critic neural network architecture, which is equal to about The derivative of ; the update law of the evaluation network in the second-layer Actor-Critic neural network architecture is: .
[0014] Furthermore, in step S6, the first oneLoss function of the evaluation network in a layered Actor-Critic neural network architecture for: ; This represents the approximation error of the Hamilton-Jacobi-Bellman equation corresponding to the attitude tracking error. An approximation of the Hamilton-Jacobi-Bellman equations corresponding to the attitude tracking error: ; The Hamilton-Jacobi-Bellman equation corresponding to the attitude tracking error is as follows: ; The loss function of the evaluation network in the second-layer Actor-Critic neural network architecture. for:
[0015] This represents the approximation error of the Hamilton-Jacobi-Bellman equation corresponding to the velocity tracking error. The Hamilton-Jacobi-Bellman equation approximation for velocity tracking is as follows: ; The Hamilton-Jacobi-Bellman equation corresponding to the velocity tracking error: .
[0016] A lifelong learning control system for ships that considers state constraints includes a state perception and error calculation module, a logarithmic obstacle cost function construction module, an Actor-Critic neural network control module and a lifelong learning weight update module connected in sequence, as well as an uncertainty approximation and compensation module connected to the input of the Actor-Critic neural network control module, a virtual control law filtering module connected to the output of the Actor-Critic neural network control module and an execution control module. The state perception and error calculation module includes a perception unit and an error calculation unit. The perception unit is used to acquire the ship's position and heading in the inertial coordinate system, as well as its speed and desired trajectory command in the attached coordinate system, through shipborne sensors. The error calculation unit is used to calculate the attitude tracking error and speed tracking error, and the calculated error signal is transmitted to the logarithmic obstacle cost function construction module and the Actor-Critic neural network control module. The logarithmic obstacle cost function construction module includes a state constraint boundary function definition unit, an optimal attitude logarithmic obstacle cost function construction unit, and an optimal velocity logarithmic obstacle cost function construction unit; The uncertainty approximation and compensation module includes a radial basis function neural network for online approximation of lumped uncertainty terms in the ship dynamics model; The virtual control law filtering module is used to receive the optimal virtual control law and output the filtered signal, avoiding the computational complexity explosion problem caused by directly differentiating the optimal virtual control law; The Actor-Critic neural network control module includes a first-layer Actor-Critic neural network unit and a second-layer Actor-Critic neural network unit. The evaluation network of the first-layer Actor-Critic neural network unit is used to approximate the optimal attitude logarithmic obstacle cost function, and the action network is used to calculate the optimal virtual control law. The evaluation network of the second-layer Actor-Critic neural network unit is used to approximate the optimal velocity logarithmic obstacle cost function, and the action network is used to calculate the optimal control law. The lifelong learning weight update module includes a Fisher information matrix calculation unit and a regularization update unit; the Fisher information matrix calculation unit is used to calculate and store the importance of the neural network weights to historical tasks; The execution control module is used to convert the control quantity calculated by the lifelong reinforcement learning control method of the ship, which takes into account state constraints, into the thrust of the actual physical system.
[0017] A computer memory storing a computer program executable by a computer processor, the computer program executing a lifelong learning control method for ships that takes into account state constraints.
[0018] The beneficial effects of this invention are as follows: 1. This invention provides a lifelong learning control method and system for ships considering state constraints. By constructing optimal attitude logarithmic obstacle cost functions and optimal velocity logarithmic obstacle cost functions, the time-varying state constraint boundaries are internalized as performance indicators for the optimal control problem. These functions generate an infinitely large penalty value when the state approaches the safety boundary, thereby forcing the controller to automatically adjust its strategy. This fundamentally ensures that the position and velocity errors of the ship during complex maneuvers are strictly limited within a preset compact set, effectively preventing collision and instability risks. It achieves optimal control for ship trajectory tracking under state constraints and strictly constrained safety control across all states, significantly improving system safety.
[0019] 2. This invention solves the problem of catastrophic forgetting faced by traditional neural network-based control methods, endowing the controller with continuous multi-task adaptability. It introduces an online elastic weight consolidation mechanism into the Actor-Critic neural network architecture, calculates the evaluation and action network weight information matrix, accurately identifies the network weights that are crucial to historical tasks, and applies penalties to changes in these key weights through regularization terms during the update process. This enables ships to achieve dual optimization of control performance and energy saving when continuously executing multiple different trajectory tasks, while effectively retaining knowledge of old tasks, significantly improving the robustness and reliability of the system in long-cycle operations.
[0020] 3. This invention internalizes time-varying state constraint boundaries into performance indicators for optimal control problems, representing a novel lifetime reinforcement learning architecture. It restricts the updating of key task weights, enabling ships to achieve dual optimization of control performance and energy efficiency when continuously executing multiple different trajectory tracking tasks. It also effectively retains historical knowledge from various navigation tasks, significantly improving the robustness and reliability of the system in long-cycle operations.
[0021] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description
[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0023] Figure 1 This is a flowchart of an embodiment of the present invention.
[0024] Figure 2 This is the principle of the embodiments of the present invention.
[0025] Figure 3 This is a trajectory tracking effect diagram of an embodiment of the present invention.
[0026] Figure 4 This is a structural schematic diagram of an embodiment of the present invention.
[0027] In the diagram: 10. State perception and error calculation module; 20. Logarithmic barrier cost function construction module; 30. Uncertainty approximation and compensation module; 40. Virtual control law filtering module; 50. Actor-Critic neural network control module; 60. Lifetime learning weight update module; 70. Execution control module. Detailed Implementation
[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0029] Example 1 See Figure 1 and Figure 2 The specific steps of a lifelong learning control method for ships that considers state constraints are as follows: S1: Establish the kinematic and dynamic model of the ship and calculate the attitude tracking error; S11: Considering the ship's motion on the horizontal plane, define the attitude vector (position and heading) in the inertial coordinate system as follows: The velocity vectors in the attached coordinate system (representing sway velocity, roll velocity, and bow angular velocity, respectively) are: The kinematics and dynamics of a ship are expressed as follows:
[0030] in, Let be a rotation matrix. The inertia matrix, For the Coriolis and centripetal force matrices, For hydrodynamic damping matrix, For input force and torque, External environmental interference.
[0031] For ships The second derivative yields:
[0032] definition as well as Reconstruct the dynamic model:
[0033] in, For uncertain terms, This is the reconstructed instruction.
[0034] S12: Calculate attitude tracking error; See Figure 3 Set the desired reference trajectory as Define attitude tracking error for .
[0035] S2: Construct the optimal attitude logarithmic obstacle cost function, the specific steps of which include: S21: Define position and heading angle limits: Set position and heading angle state limit boundaries It monotonically decreases with time to achieve convergence from loose constraints to tight constraints:
[0036] in, As the initial boundary, To shorten the convergence time, The convergence boundary can be defined by the user.
[0037] S22: Constructing the optimal attitude logarithmic obstacle-type cost function :
[0038] in, It is attitude tracking error. and For positive integers, It is the ideal optimal virtual control law. These are the position and heading angle state limits.
[0039] S3: Construct the first-layer Actor-Critic neural network structure, derive the optimal virtual control law, and approximate the optimal logarithmic obstacle cost function. Specific steps include: S31: Design the Hamilton-Jacobi-Bellman equation corresponding to the attitude tracking error:
[0040] in, .
[0041] S32: Based on the Bellman optimality principle and the Hamilton-Jacobi-Bellman equation, derive the analytical form of the ideal optimal virtual control law:
[0042] S33: Constructing the first-layer Actor-Critic network architecture: Evaluation Networks: Approximate Performance Metrics The output is ; Action Networks: Approximate Ideal Optimal Virtual Control Laws Its output is the optimal virtual control law Specifically, it is expressed as:
[0043] in, and For positive design parameters, basis functions Transpose of the partial derivative of attitude tracking error.
[0044] The approximation of the Hamilton-Jacobi-Bellman equation corresponding to the attitude tracking error is obtained as follows: ; S34: Filter the optimal virtual control law by introducing a first-order filter:
[0045] in, This is the filtered virtual control signal. is the filter time constant.
[0046] S4: Construct the optimal velocity logarithmic barrier cost function; S41: Calculate the speed tracking error: .
[0047] S42: Define speed limit boundaries It monotonically decreases with time to achieve convergence from loose constraints to tight constraints:
[0048] in, As the initial boundary, To shorten the convergence time, The convergence boundary can be defined by the user.
[0049] S43: Constructing the optimal velocity logarithmic barrier cost function :
[0050] in, It is a speed tracking error. and For positive integers, It is the ideal optimal control law. These are the limits for sway velocity, roll velocity, and pitch angular velocity.
[0051] S5: Construct the second-layer Actor-Critic neural network architecture, derive the optimal control law, and the approximate optimal logarithmic velocity barrier cost function. Specific steps include: S51: Hamilton-Jacobi-Bellmann equations corresponding to design speed tracking error:
[0052] in, .
[0053] S52: Solve the Hamilton-Jacobi-Bellman equations corresponding to the speed tracking error, and derive the analytical form of the ideal optimal control law:
[0054] S53: Constructing the second-layer Actor-Critic network architecture: Evaluation Networks: Approximate Performance Metrics The output is ; Action Networks: Approximate Ideal Optimal Control Laws Its output optimal control law Specifically, it is expressed as:
[0055] in, and For positive design parameters, It is an estimate from an action neural network. As basis functions, basis functions Transpose of the partial derivative of the velocity tracking error.
[0056] The approximation of the Hamilton-Jacobi-Bellman equation corresponding to the speed tracking error is obtained as follows:
[0057] S54: Online approximation of lumped uncertainty terms in ship dynamics using radial basis function neural networks (Including unmodeled fluid dynamic damping) and external wind and wave interference wait):
[0058] in For weights, These are basis functions.
[0059] S6: Reconstruct the weight update law based on the lifelong reinforcement learning mechanism. The specific steps include: S61: Construct the loss function and action network error for the evaluation network. The specific loss function for the evaluation network is as follows:
[0060] in, This represents the approximation error of the Hamilton-Jacobi-Bellman equation corresponding to the attitude tracking error. This is an approximation of the Hamilton-Jacobi-Bellman equations corresponding to the attitude tracking error. This represents the approximation error of the Hamilton-Jacobi-Bellman equation corresponding to the velocity tracking error. The Hamilton-Jacobi-Bellman equation approximation is given for the velocity tracking error.
[0061] The specific errors of the action network are:
[0062]
[0063] S62: Construct evaluation metrics for network performance and action network performance metrics, specifically:
[0064] S63: The weight update law of the first layer of the Actor-Critic neural network is designed using the normalized gradient descent method; The action network update law of the first-layer Actor-Critic neural network architecture is as follows:
[0065] in, , and For design parameters, basis functions Taking the partial derivative of the attitude error , The derivative of the desired trajectory; The evaluation network update law of the first-layer Actor-Critic neural network architecture is as follows:
[0066] in, It is the Jacobian matrix of the evaluation network of the first-layer Actor-Critic neural network architecture, which is equal to about The derivative; S64: The weight update law of the second layer Actor-Critic neural network is designed using the normalized gradient descent method; The update law of action networks is as follows:
[0067] in, , and For design parameters, basis functions Taking the partial derivative with respect to the speed error , This is an approximate estimate of the ship's uncertainties. This is the derivative of the first-order filter.
[0068] The evaluation of network update laws is as follows:
[0069] in, It is the Jacobian matrix of the evaluation network in the second-layer Actor-Critic neural network architecture, which is equal to about The derivative of .
[0070] This embodiment internalizes the time-varying state constraint boundary into performance indicators for the optimal control problem by constructing optimal attitude logarithmic obstacle cost functions and optimal velocity logarithmic obstacle cost functions. These functions generate an infinite penalty value when the state approaches the safety boundary, forcing the controller to automatically adjust its strategy. This fundamentally ensures that the ship's attitude and velocity errors during complex maneuvers are strictly limited to a pre-set compact set, effectively preventing collision and instability risks. It achieves optimal control for ship trajectory tracking under state constraints and strictly constrained safety control across all states, significantly improving system safety.
[0071] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0072] Example 2 See Figure 4 This embodiment is used to implement the principle of the above method embodiments to construct a ship lifelong learning control system that considers state constraints, including... The state perception and error calculation module includes a perception unit and an error calculation unit. The state perception unit obtains the ship's position and heading in the inertial coordinate system, speed in the attached coordinate system, and desired trajectory command through shipborne sensors (such as GPS and IMU). The error calculation unit is responsible for calculating the attitude tracking error and speed tracking error. The calculated error signal is transmitted to the logarithmic obstacle cost function construction module and the Actor-Critic neural network control module. The logarithmic obstacle cost function construction module includes a state constraint boundary function definition unit, an optimal attitude logarithmic obstacle cost function construction unit, and an optimal velocity logarithmic obstacle cost function construction unit; The uncertainty approximation and compensation module, including a radial basis function neural network, is responsible for approximating the total uncertainty term in ship dynamics online. The virtual control command filtering module receives the optimal virtual control strategy and outputs the filtered signal, avoiding the explosion of computational complexity caused by directly differentiating the optimal virtual control strategy. The Actor-Critic neural network control module is the core intelligent decision-making center of the system. It includes a first-layer Actor-Critic neural network unit and a second-layer Actor-Critic neural network unit. In the first-layer Actor-Critic neural network unit, the evaluation network is responsible for approximating the optimal attitude logarithmic obstacle cost function, and the action network is responsible for calculating the optimal virtual control law. In the second-layer Actor-Critic neural network unit, the evaluation network is responsible for approximating the optimal velocity logarithmic obstacle cost function, and the action network is responsible for calculating the optimal control law. The lifelong learning weight update module includes a Fisher information matrix calculation unit and a regularization update unit. The Fisher information matrix calculation unit is responsible for calculating and storing the importance of the neural network weights to historical tasks.
[0073] The execution control module is responsible for converting the commands calculated by the lifelong learning control method for ships, which takes into account state constraints, into the thrust of the actual physical system.
[0074] Each submodule is mainly used to implement the various steps of the method implementation, which will not be elaborated here.
[0075] It should be noted that, depending on the implementation needs, the various steps / components described in this application can be broken down into more steps / components, or two or more steps / components or parts of the operation of steps / components can be combined into new steps / components to achieve the purpose of this invention.
[0076] This embodiment also includes a processor, a communication interface, a memory, and a communication bus; wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of a lifelong learning control method for ships that considers state constraints.
[0077] This embodiment also provides a computer-readable storage medium storing executable instructions that, when executed by a processor, enable the processor to implement a lifelong learning control method for ships that takes into account state constraints.
[0078] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects.
[0079] Furthermore, this application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0080] This application is described with reference to the flowchart of the method and computer program product according to Embodiment 1 and the block diagram of the device (system) according to Embodiment 3. It should be understood that each step or block in the flowchart or block diagram, as well as combinations of steps or blocks in the flowchart or block diagram, can be implemented by computer program instructions.
[0081] These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which are executable by the processor of the computer or other programmable data processing device, produce instructions for implementing the process. Figure 1 One or more processes or boxes Figure 1 A lifelong learning control system for ships that takes into account state constraints and specifies functions in one or more boxes.
[0082] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes or boxes Figure 1 The function specified in one or more boxes.
[0083] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes or boxes Figure 1 The steps of a lifelong learning control method for ships that takes into account state constraints are specified in one or more boxes.
[0084] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.
Claims
1. A ship lifelong learning control method considering state constraints, characterized in that: Includes the following steps: S1: Establish the kinematic and dynamic model of the ship, and calculate the attitude tracking error based on the ship's position, heading, and reference trajectory; S2: Construct the optimal attitude logarithmic obstacle cost function based on attitude tracking error; S3: Construct the first-layer Actor-Critic neural network architecture, derive the optimal virtual control law, and approximate the optimal attitude logarithmic obstacle cost function; S4: Based on the ship speed variable and the optimal virtual control law, calculate the speed tracking error and construct the optimal logarithmic barrier cost function for speed; S5: Construct the second-layer Actor-Critic neural network architecture, derive the optimal control law, and approximate the optimal logarithmic speed barrier cost function; S6: Based on the lifelong reinforcement learning mechanism, reconstruct the neural network weight update law in the first and second layer Actor-Critic neural network architecture.
2. The ship lifelong learning control method considering state constraints according to claim 1, characterized in that: The step S2 described above, is a pose tracking error, and is a normal number, is an ideal optimal virtual control law, is a position and heading angle state constraint boundary, and an optimal pose pair logarithmic barrier type cost function is expressed as: ； The step S4 is is a velocity tracking error, and is a normal number, is an ideal optimal virtual control law, is a surge velocity, sway velocity and yaw angle velocity constraint boundary, and an optimal velocity logarithmic barrier type cost function is expressed as: 。 3. The ship lifelong learning control method considering state constraints according to claim 2, characterized in that: and is the initial boundary, and is the convergence time, and is the user-defined setting convergence boundary, position and heading angle state constraint boundary and surge velocity, sway velocity and yaw angle velocity constraint boundary are respectively represented as: 。 4. The ship lifelong learning control method considering state constraints according to claim 1, characterized in that: The step S3 mentioned, and is a positive design parameter of the control law, is an action neural network estimate value in the first layer Actor-Critic neural network architecture, is a base function of the first layer Actor-Critic neural network architecture, is a base function The optimal virtual control law is expressed as the transpose of the partial derivative with respect to the attitude error. ； and These are the positive design parameters for the control law. It is the action neural network estimate in the second-layer Actor-Critic neural network architecture. These are the basis functions for the second-layer Actor-Critic neural network architecture. basis functions Regarding the transpose of the partial derivative with respect to the velocity error, in step S5, the optimal control law is expressed as: 。 5. The ship lifelong learning control method considering state constraints according to claim 1, characterized in that: In step S6, This serves as a performance metric for evaluating the first-layer Actor-Critic neural network architecture. This refers to the performance metrics of the action network in the first layer of the Actor-Critic neural network architecture. This serves as a performance metric for evaluating the second-layer Actor-Critic neural network architecture. This refers to the performance metrics of the action network in the second-layer Actor-Critic neural network architecture. , , as well as ( () represents the design parameters. and For Fisher's information matrix, It is the first j Step the comment weight vector of the current task, It is the first j Step the action weight vector of the current task. It is the previous task number j The bounded evaluation weight vector of the step. It is the previous task number j The bounded action weight vector of the step. It is the first j The loss function of the evaluation network in a multi-layered Actor-Critic neural network architecture. It is the first j The performance metrics for the evaluation network and action network, which are constructed based on the error of the action network and the first and second layer Actor-Critic neural network structures, are as follows: 。 6. The ship lifelong learning control method considering state constraints according to claim 5, characterized in that: In step S6, based on the performance metrics of the evaluation network and the performance metrics of the action network of the first-layer Actor-Critic neural network architecture, the normalized gradient descent method is used to design the weight update law of the first-layer Actor-Critic neural network. , and For design parameters, basis functions Taking the partial derivative of the attitude error , It is the Jacobian matrix of the action network in the first layer of the Actor-Critic neural network architecture, which is equal to about The derivative of The derivative of the desired trajectory; the action network update law of the first-layer Actor-Critic neural network architecture is: ； It is the Jacobian matrix of the evaluation network of the first-layer Actor-Critic neural network architecture, which is equal to about The derivative of ; the update law of the evaluation network in the first layer of the Actor-Critic neural network architecture is: 。 7. A lifelong learning control method for ships considering state constraints according to claim 5, characterized in that: In step S6, based on the performance metrics of the evaluation network and the performance metrics of the action network in the second-layer Actor-Critic neural network architecture, the normalized gradient descent method is used to design the weight update law of the second-layer Actor-Critic neural network. , and For design parameters, basis functions Taking the partial derivative with respect to the speed error , It is the Jacobian matrix of the action network in the second-layer Actor-Critic neural network architecture, which is equal to about The derivative; To approximate the model uncertainty of a ship, a radial basis function neural network is established to approximate the lumped uncertainty in the ship dynamics model online. The uncertainty includes unmodeled hydrodynamic damping and external wind and wave disturbance torque. The derivative of a first-order filter is given by: , For the error of the first-order filter, The filtering vector for the optimal virtual control law; the action network update law of the second-layer Actor-Critic neural network architecture is: ； It is the Jacobian matrix of the evaluation network in the second-layer Actor-Critic neural network architecture, which is equal to about The derivative of ; the update law of the evaluation network in the second-layer Actor-Critic neural network architecture is: 。 8. A lifelong learning control method for ships considering state constraints according to claim 5, characterized in that: In step S6, the first one Loss function of the evaluation network in a layered Actor-Critic neural network architecture for: ； This represents the approximation error of the Hamilton-Jacobi-Bellman equation corresponding to the attitude tracking error. An approximation of the Hamilton-Jacobi-Bellman equations corresponding to the attitude tracking error: ； The Hamilton-Jacobi-Bellman equation corresponding to the attitude tracking error is as follows: ； The loss function of the evaluation network in the second-layer Actor-Critic neural network architecture. for: This represents the approximation error of the Hamilton-Jacobi-Bellman equation corresponding to the velocity tracking error. The Hamilton-Jacobi-Bellman equation approximation for velocity tracking is as follows: ； The Hamilton-Jacobi-Bellman equation corresponding to the velocity tracking error: 。 9. A state-constrained ship lifelong learning control system for use in any one of claims 1 to 8, characterized in that: It includes a state perception and error calculation module, a logarithmic barrier cost function construction module, an Actor-Critic neural network control module and a lifelong learning weight update module connected in sequence, as well as an uncertainty approximation and compensation module connected to the input of the Actor-Critic neural network control module, a virtual control law filtering module connected to the output of the Actor-Critic neural network control module and an execution control module. The state perception and error calculation module includes a perception unit and an error calculation unit; the perception unit is used to acquire the ship's position and heading in the inertial coordinate system, as well as its speed and desired trajectory command in the appendage coordinate system, through shipborne sensors. The error calculation unit is used to calculate attitude tracking error and velocity tracking error. The calculated error signal is transmitted to the logarithmic obstacle cost function construction module and the Actor-Critic neural network control module. The logarithmic obstacle cost function construction module includes a state constraint boundary function definition unit, an optimal attitude logarithmic obstacle cost function construction unit, and an optimal velocity logarithmic obstacle cost function construction unit; The uncertainty approximation and compensation module includes a radial basis function neural network for online approximation of lumped uncertainty terms in the ship dynamics model; The virtual control law filtering module is used to receive the optimal virtual control law and output the filtered signal, avoiding the computational complexity explosion problem caused by directly differentiating the optimal virtual control law; The Actor-Critic neural network control module includes a first-layer Actor-Critic neural network unit and a second-layer Actor-Critic neural network unit. The evaluation network of the first-layer Actor-Critic neural network unit is used to approximate the optimal attitude logarithmic obstacle cost function, and the action network is used to calculate the optimal virtual control law. The evaluation network of the second-layer Actor-Critic neural network unit is used to approximate the optimal velocity logarithmic obstacle cost function, and the action network is used to calculate the optimal control law. The lifelong learning weight update module includes a Fisher information matrix calculation unit and a regularization update unit; the Fisher information matrix calculation unit is used to calculate and store the importance of the neural network weights to historical tasks; The execution control module is used to convert the control quantity calculated by the lifelong reinforcement learning control method of the ship, which takes into account state constraints, into the thrust of the actual physical system.
10. A computer memory, characterized in that: It contains a computer program that can be executed by a computer processor, which performs a lifelong learning control method for ships that takes into account state constraints as described in any one of claims 1 to 8.