A supercavitation underwater vehicle adaptive cascade control method and system based on reinforcement learning
By combining reinforcement learning and traditional PID controllers to form an adaptive cascade control method, the robustness and adaptability of supercavitating underwater vehicles in complex environments are solved, achieving precise control and efficient and stable operation of the vehicle.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUAZHONG UNIV OF SCI & TECH
- Filing Date
- 2026-04-02
- Publication Date
- 2026-07-21
AI Technical Summary
Existing control methods for supercavitating underwater vehicles lack robustness and adaptability when facing parameter changes and external disturbances. Traditional methods lack interpretability and safety, while reinforcement learning methods lack transparency.
An adaptive cascade control method based on reinforcement learning is adopted, combined with a traditional PID controller. The control gain is adjusted online by a reinforcement learning agent, and outer-loop depth control and inner-loop attitude control are designed. The control strategy is optimized using the Soft Actor-Critic algorithm.
Achieving precise control in complex dynamic environments improves the robustness and adaptability of the vehicle while maintaining the transparency and interpretability of traditional control structures.
Smart Images
Figure CN122431083A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of underwater vehicle control technology, and more specifically, relates to an adaptive cascade control method and system for supercavitating underwater vehicles based on reinforcement learning. Background Technology
[0002] Supercavitating underwater vehicles can achieve extremely high speeds by creating steam cavitation around their hull, significantly reducing viscous drag. This characteristic gives them a unique advantage in time-critical underwater missions such as rapid interception and long-range strikes. However, the presence of cavitation also makes the longitudinal motion control of the vehicle highly nonlinear. The vehicle's gliding force depends on the instantaneous changes in the cavitation geometry, and the hydrodynamic characteristics change drastically when the cavitation walls interact with the control surfaces or free surfaces, thus increasing the difficulty of control.
[0003] Current control methods for supercavitating underwater vehicles mostly rely on model-based control strategies, typically employing fixed gain or analytically derived gain scheduling rules. While effective under certain conditions, these methods have limitations in practical applications, particularly when faced with parameter variations, external disturbances, and the lack of accurate dynamic modeling, where their performance degrades significantly. Furthermore, traditional control methods such as PID controllers and linear feedback methods, although capable of deep tuning in some simple tasks, lack sufficient robustness and adaptability when dealing with complex dynamic systems that are nonlinear and strongly coupled.
[0004] Reinforcement learning, as a data-driven control method developed in recent years, is particularly suitable for systems that are difficult to model accurately or for which models are hard to obtain. For underwater vehicles, reinforcement learning performs exceptionally well in tasks such as depth tracking, enabling it to learn control strategies directly through interaction with the environment without relying on precise models. However, while the aforementioned reinforcement learning controllers can rival the performance of linear quadratic regulation or nonlinear model predictive control methods, existing reinforcement learning methods typically replace the entire control loop with a "black box" model. This approach lacks the interpretability and safety guarantees of traditional control systems. Summary of the Invention
[0005] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides an adaptive cascade control method and system for supercavitating underwater vehicles based on reinforcement learning. Its purpose is to retain the transparency and interpretability of traditional control structures while improving the robustness and adaptability of the vehicle in complex dynamic environments. This solves the technical problems of insufficient performance of traditional control methods in complex situations and the lack of interpretability and safety guarantees in reinforcement learning methods.
[0006] To achieve the above objectives, this invention provides an adaptive cascade control method for supercavitating underwater vehicles based on reinforcement learning, comprising the following steps: (1) Obtain the geometric shape and dynamic parameters of the supercavitating vehicle and establish a dynamic model of the supercavitating vehicle; The supercavitating vehicle's configuration consists of a slender, axisymmetric main body, a disk-shaped cavitation generator fixed at the front, and a tail fin fixed to the outer surface of the rear. Its state equation is expressed as follows: .in x For the state variables of a supercavitating vehicle, z is the depth. The pitch angle, Vertical velocity, q Angular velocity; To control the quantity, , and These are the control parameters for the cavitation system and the tail fin, respectively. The gliding force for supercavitating vehicles; A , B , C , D It is a constant matrix.
[0007] (2) Determine the desired depth position to be reached by the supercavitating underwater vehicle based on the preset target depth trajectory. z d ; (3) Design two submersible control modules to perform basic control. The two submersible control modules are combined with the dynamic model of the supercavitating vehicle and the supercavitating vehicle is controlled to navigate based on the desired depth and position.
[0008] Each submersible control module is a cascaded PID controller, which consists of nested inner and outer loop controllers. Specifically, the outer loop PI controller calculates the depth trajectory error based on a comparison between the desired depth position and the actual depth trajectory. e z and the set control parameters , Perform control adjustments to generate the desired angle. The inner-loop PID controller is based on the desired angle. The actual angle of the current vehicle The angle error is calculated based on the difference between them. and the set control parameters , , Further control is performed to generate control signals. Used to adjust the attitude of the aircraft; (4) Control signal Input the dynamic model of the supercavitating underwater vehicle and calculate the actual angle of the vehicle. and angular velocity q It also outputs the actual trajectory of the supercavitating underwater vehicle. z ; (5) The reinforcement learning agent uses the depth trajectory error fed back from step (3) e z and the state variables provided by the dynamic model The parameters of the outer loop PI controller and the inner loop PID controller are adjusted in real time.
[0009] The present invention also provides an adaptive cascade control system based on reinforcement learning, comprising: Two control modules are used to generate control parameters for the cavitation generator and tail fin, respectively. and , and For control signals The two components are used to control the cavitation device and tail fin of the supercavitating vehicle, respectively. The two modules are similar in basic structure, differing only in their controller parameters. , , , , The values vary, and the specific values are provided by the reinforcement learning module. Each deep-sea control module adopts a classic cascaded PID control architecture, decoupling the control task into two closed loops: the outer loop is the depth control loop, which is responsible for calculating the ideal attitude angle required to maintain the target depth, i.e., the desired angle, based on the set target depth trajectory using a PI controller. The inner loop is the attitude control loop, which uses PID control to quickly and accurately track the attitude commands given by the outer loop and outputs specific control signals. ; The supercavitating underwater vehicle dynamics module is based on input signals. The state variables at the current time step are updated using the state equation, based on the state variables from the previous time step. Where z is the depth, The pitch angle, Vertical velocity, q Let angular velocity be 'ω', and calculate the gliding force. The module is based on state equations. Perform a status update; The reinforcement learning module uses state variables. and gliding force and expected state variables For input, where x for , for The expected values are preprocessed and transformed into the state variables of the agent. The reward function and termination signal are transmitted to the agent, and the agent outputs action values back to the two latent control modules. The reward function is designed as follows: ,in J Indicates instantaneous cost. , The penalty term represents the trajectory error. The penalty for gliding force is indicated by the following: It is an "inertia" penalty mechanism added to prevent the agent from behaving too conservatively when the trajectory error is large but the gliding force is small. A constant survival reward. This is an additional stepwise reward when the state remains within a specified soft tolerance range, if and only if the state error is within the soft tolerance band. The value is 1. This is the final reward / penalty that is settled only at the end of the round. and The parameters are constants. The action value output by the agent is a ten-dimensional parameter vector. A split design is adopted to control two latent control modules in parallel: the first five dimensions are mapped to the first latent control module as a PID gain regulator, and the last five dimensions are used as the gain regulator of the second latent control module, so as to realize the coordinated online tuning of the inner and outer loop control parameters of the dual-module cascade PID structure.
[0010] The reinforcement learning agent employs the Soft Actor-Critic (SAC) reinforcement learning algorithm, whose objective is to maximize the expected cumulative reward. ,in For instant rewards, The entropy of the policy in the current state. It is the entropy coefficient; through interaction with the simulation environment, the agent optimizes the gain adjustment strategy in real time, and adjusts the gain of the outer loop depth control and inner loop attitude control online, thereby enhancing the system's adaptability to dynamic disturbances and model uncertainties.
[0011] Through the collaborative work of these modules, this invention effectively achieves precise control of the vehicle in nonlinear and dynamically changing environments, and significantly improves the robustness and adaptability of the system. The design of these modules not only retains the advantages of traditional cascade control systems, but also enhances the system's self-learning capability by introducing reinforcement learning agents. This allows the system to dynamically adjust control strategies under different operating conditions, ensuring high efficiency and stability in practical applications.
[0012] The present invention also provides a computer-readable storage medium storing computer instructions for causing a processor to perform the above-described method.
[0013] The present invention also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the above-described method.
[0014] Overall, compared with the prior art, the above-described technical solutions conceived in this invention provide an adaptive cascade control method and system based on reinforcement learning. To overcome the limitations of existing methods, this invention combines the traditional cascade control structure with reinforcement learning algorithms to adjust the control gain online, while integrating outer-loop depth control and inner-loop attitude control. This retains the transparency and interpretability of the traditional control structure and exhibits excellent robustness and adaptability in complex dynamic environments. Attached Figure Description
[0015] Figure 1 A flowchart of an adaptive cascade control method for supercavitating underwater vehicles based on reinforcement learning is provided in an embodiment of the present invention. Figure 2 A schematic diagram of an adaptive cascade control system for a supercavitating underwater vehicle based on reinforcement learning, provided for an embodiment of the present invention; Figure 3 This is a schematic diagram of the Simulink simulation platform structure for an adaptive cascade control system for supercavitating underwater vehicles based on reinforcement learning, provided in an embodiment of the present invention. Detailed Implementation
[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.
[0017] This invention provides an adaptive cascade control method for supercavitating underwater vehicles based on reinforcement learning, such as... Figure 1 As shown, it includes the following steps: (1) Obtain the geometric shape and dynamic parameters of the supercavitating vehicle and establish a dynamic model of the supercavitating vehicle; The supercavitating vehicle's configuration consists of a slender, axisymmetric main body, a disk-shaped cavitation generator fixed at the front, and a tail fin fixed to the outer surface of the rear. Its state equation is expressed as follows: .in x For the state variables of a supercavitating vehicle, z is the depth. The pitch angle, Vertical velocity, q Angular velocity; To control the quantity, , and These are the control parameters for the cavitation system and the tail fin, respectively. The gliding force for supercavitating vehicles; A , B , C , D It is a constant matrix.
[0018] in .
[0019] in , , , , , , , , , , .
[0020] The dynamic parameters of the supercavitating vehicle are shown in Table 1.
[0021] Table 1
[0022] (2) Determine the desired depth position to be reached by the supercavitating underwater vehicle based on the preset target depth trajectory. z d ; (3) Design two submersible control modules to perform basic control. The two submersible control modules are combined with the dynamic model of the supercavitating vehicle and the supercavitating vehicle is controlled to navigate based on the desired depth and position.
[0023] Each submersible control module is a cascaded PID controller, which consists of nested inner and outer loop controllers. Specifically, the outer loop PI controller calculates the depth trajectory error based on a comparison between the desired depth position and the actual depth trajectory. e z and the set control parameters , Perform control adjustments to generate the desired angle. The inner-loop PID controller is based on the desired angle. The actual angle of the current vehicle The angle error is calculated based on the difference between them. and the set control parameters , , Further control is performed to generate control signals. Used to adjust the attitude of the aircraft; Specifically, depth trajectory error , z For actual depth; for expected angle ; Angle error = , The function can limit the angle to Within range; control signals , and These are the control parameters for the cavitation system and the tail fin, respectively. , q ω is the angular velocity.
[0024] (4) Control signal Input the dynamic model of the supercavitating underwater vehicle and calculate the actual angle of the vehicle. and angular velocity q It also outputs the actual trajectory of the supercavitating underwater vehicle. z ; (5) The reinforcement learning agent uses the depth trajectory error fed back from step (3) e z and the state variables provided by the dynamic model S t The parameters of the outer loop PI controller and the inner loop PID controller are adjusted in real time.
[0025] in The controller parameters are , , , , .
[0026] This concludes the description of the specific implementation process of the reinforcement learning-based adaptive cascade control method for supercavitating underwater vehicles used in this patent. In practical implementation, the process provided by the technical solution of this invention can be automatically executed by those skilled in the art using computer software technology, or the corresponding system can be implemented in a modular manner.
[0027] This invention also provides an adaptive cascade control system based on reinforcement learning, such as... Figure 2 As shown, it includes: Two control modules are used to generate control parameters for the cavitation generator and tail fin, respectively. and The two modules described are similar in basic structure, differing only in their controller parameters. , , , , The values vary, with the specific values provided by the reinforcement learning module. Each underwater control module employs a classic cascaded PID control architecture, decoupling the control task into two closed loops: the outer loop is the depth control loop, responsible for calculating the ideal attitude angle required to maintain the target depth using a PI controller; the inner loop is the attitude control loop, which uses PID control to quickly and accurately track the attitude commands given by the outer loop and outputs specific control signals. The signals generated by the two underwater control modules are transmitted to a supercavitating underwater vehicle dynamics module. The supercavitating underwater vehicle dynamics module is based on input signals. The state variables at the current time step are updated using the state equation, based on the state variables from the previous time step. Where z is the depth, The pitch angle, Vertical velocity, q Let angular velocity be 'ω', and calculate the gliding force. The module is based on state equations. Perform state updates and connect to a reinforcement learning module; The reinforcement learning module uses state variables and gliding force and expected state variables For input, where x for , for The expected values are preprocessed and transformed into the state variables of the agent. The reward function and termination signal are transmitted to the agent, and the agent outputs an action value back to the control module. The reward function is designed as follows: ,in J Indicates instantaneous cost. , The penalty term represents the trajectory error. The penalty for gliding force is indicated by the following: It is an "inertia" penalty mechanism added to prevent the agent from behaving too conservatively when the trajectory error is large but the gliding force is small. A constant survival reward. This is an additional stepwise reward when the state remains within a specified soft tolerance range, if and only if the state error is within the soft tolerance band. The value is 1. This is the final reward / penalty that is settled only at the end of the round. and The parameters are constants. The action value output by the agent is a ten-dimensional parameter vector. A split design is adopted to control two latent control modules in parallel: the first five dimensions are mapped to the first latent control module as a PID gain regulator, and the last five dimensions are used as the gain regulator of the second latent control module, so as to realize the coordinated online tuning of the inner and outer loop control parameters of the dual-module cascade PID structure.
[0028] The reinforcement learning agent uses the SAC reinforcement learning algorithm, whose goal is to maximize the expected cumulative reward. ,in For instant rewards, The entropy of the policy in the current state. It is the entropy coefficient; the reinforcement learning agent optimizes the gain adjustment strategy in real time by interacting with the simulation environment, and adjusts the gain of the outer loop depth control and inner loop attitude control online, thereby enhancing the system's adaptability to dynamic disturbances and model uncertainties.
[0029] Figure 3 This is a schematic diagram of the Simulink simulation platform structure for an adaptive cascade control system for supercavitating underwater vehicles based on reinforcement learning, provided in an embodiment of the present invention.
[0030] Through the collaborative work of these modules, this invention effectively achieves precise control of the vehicle in nonlinear and dynamically changing environments, and significantly improves the robustness and adaptability of the system. The design of these modules not only retains the advantages of traditional cascade control systems, but also enhances the system's self-learning capability by introducing reinforcement learning agents. This allows the system to dynamically adjust control strategies under different operating conditions, ensuring high efficiency and stability in practical applications.
[0031] The present invention also relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.
[0032] Specifically, the memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital (SD) cards, flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.
[0033] This invention provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the method described in the above embodiments of this invention.
[0034] The technical features of the embodiments described above can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. It should be noted that the terms "in one embodiment," "for example," and "again" in this invention are intended to illustrate the invention and are not intended to limit the invention.
[0035] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.
Claims
1. An adaptive cascade control method for supercavitating underwater vehicles based on reinforcement learning, characterized in that, Includes the following steps: Based on the preset target depth trajectory, determine the desired depth position that the supercavitating underwater vehicle needs to reach; The outer-loop PI controller calculates the depth trajectory error based on a comparison between the desired depth position and the actual depth trajectory. e z and the set control parameters , Control and generate the desired angle ; The inner-loop PID controller is based on the desired angle. The actual angle of the current vehicle The angle error is calculated based on the difference between them. and the set control parameters , , Perform control and generate control signals. Used to adjust the attitude of the aircraft; control signal Input the dynamic model of the supercavitating underwater vehicle and calculate the actual angle of the vehicle. and angular velocity q It also outputs the actual trajectory of the supercavitating underwater vehicle. z ; The reinforcement learning agent uses depth trajectory error e z State variables provided by the supercavitating underwater vehicle dynamics model S t The parameters of the outer loop PI controller and the inner loop PID controller are adjusted in real time.
2. The adaptive cascade control method for supercavitating underwater vehicles as described in claim 1, characterized in that, The depth trajectory error e z Represented as: in z d For the desired depth location, z This represents the actual depth of the vehicle.
3. The adaptive cascade control method for supercavitating underwater vehicles as described in claim 1, characterized in that, The desired angle Represented as: Among them, control parameters and Controlled by the SAC algorithm; The angle error Represented as: = in The actual angle of the vehicle. The function is used to limit the angle to Within the range.
4. The adaptive cascade control method for supercavitating underwater vehicles as described in claim 1, characterized in that, The control signal Represented as: in and These are the control parameters for the cavitation system and the tail fin, respectively. and It is a constant coefficient; , q Angular velocity, control parameter , and It is regulated by the SAC algorithm.
5. The adaptive cascade control method for supercavitating underwater vehicles as described in claim 1, characterized in that, It also includes acquiring the geometric shape and dynamic parameters of the supercavitating vehicle and establishing a dynamic model of the supercavitating vehicle; The supercavitating vehicle is configured as an axisymmetric main body, a disk-shaped cavitator fixed at the front end, and a tail rudder fixed to the outer surface of the tail. Its state equation is expressed as follows: ,in x For the state variables of a supercavitating vehicle, z is the depth. The pitch angle, Vertical velocity, q Angular velocity; To control the quantity, , and These are the control parameters for the cavitation system and the tail fin, respectively. The gliding force for supercavitating vehicles; A , B , C , D It is a constant matrix.
6. An adaptive cascade control system for supercavitating underwater vehicles based on reinforcement learning, characterized in that, include: Two control modules are used to generate control parameters for the cavitation generator and tail fin, respectively. and , and For control signals The two components are used to control the cavitation device and tail fin of the supercavitating vehicle, respectively; the controller parameters of the two submarine control modules , , , , The values vary, and the specific values are provided by the reinforcement learning module. Each latent control module adopts a cascaded PID control architecture, decoupling the control task into two closed loops: the outer loop is the depth control loop, which is responsible for calculating the ideal attitude angle required to maintain the target depth, i.e., the desired angle, based on the set target depth trajectory using a PI controller. The inner loop is the attitude control loop, which uses PID control to quickly and accurately track the attitude commands given by the outer loop and outputs specific control signals. ; The supercavitating underwater vehicle dynamics module is used for input signal-based dynamics. The state variables at the current time step are updated using the state equation, based on the state variables from the previous time step. Where z is the depth, The pitch angle, Vertical velocity, q Let angular velocity be 'ω', and calculate the gliding force. The module is based on state equations. Perform a status update; The reinforcement learning module is used to learn about state variables. and gliding force and expected state variables For input, where x for , for The expected values are preprocessed and transformed into the state variables of the agent. The reward function and termination signal are transmitted to the agent, and the agent outputs action values back to the two latent control modules. The action value is a ten-dimensional parameter vector. It adopts a split design to control the two latent control modules in parallel: the first five dimensions are mapped to the first latent control module as a PID gain regulator, and the last five dimensions are used as the gain regulator of the second latent control module, so as to realize the coordinated online tuning of the inner and outer loop control parameters of the dual-module cascade PID structure.
7. The adaptive cascade control system for supercavitating underwater vehicles as described in claim 6, characterized in that, The agent employs the SAC reinforcement learning algorithm, the objective of which is to maximize the expected cumulative reward. ,in For instant rewards, The entropy of the policy in the current state. It is the entropy coefficient.
8. The adaptive cascade control system for supercavitating underwater vehicles as described in claim 6, characterized in that, In the SAC reinforcement learning algorithm used by the intelligent agent, the reward function is: ,in J Indicates instantaneous cost. , The penalty term represents the trajectory error. The penalty for gliding force is indicated. This indicates an additional penalty. For constant survival reward, This is an additional stepwise reward when the state remains within a specified soft tolerance range, if and only if the state error is within the soft tolerance band. =1, The final reward / penalty is settled only at the end of the round. and It is a constant.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.
10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method as described in any one of claims 1 to 6.