Multi-task rapid adaptive control method for underwater robot based on meta-reinforcement learning
By using a meta-reinforcement learning-based approach, a dynamic model and thrust allocation strategy for an underwater robot are constructed. A multi-task reinforcement learning controller is designed, which solves the problem of underwater robots quickly adapting to multiple tasks in complex environments and achieves precise position control and rapid training.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHENYANG INST OF AUTOMATION - CHINESE ACAD OF SCI
- Filing Date
- 2024-11-22
- Publication Date
- 2026-05-22
AI Technical Summary
When underwater robots face complex environments and new tasks, traditional controllers have long training cycles, making it difficult to quickly adapt to various position tracking tasks, and the dynamic models are difficult to build and are inaccurate.
A meta-reinforcement learning-based approach is adopted to construct an underwater robot dynamics model and thrust allocation strategy matrix, design a multi-task reinforcement learning controller, and optimize the initialization parameters through a meta-training platform to achieve fast adaptive control.
Achieving precise position control in complex environments enables rapid adaptation to different task requirements, reduces training cycles, and improves the control efficiency of underwater robots in multi-task scenarios.
Smart Images
Figure CN122072452A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the intersection of underwater robot control technology and machine learning, specifically to a fast adaptive control method for multi-task underwater robots based on meta-reinforcement learning. Background Technology
[0002] The advent of the ocean age has accelerated the innovation and intelligentization of underwater robot technology. In tasks such as seabed search and underwater facility inspection, underwater robots are required to perform different location tracking requirements according to the task, ensuring the effectiveness of underwater robot sampling and their own safety.
[0003] However, due to the high coupling inherent in underwater robots and the complexity of the underwater environment, building their dynamic models is often difficult and inaccurate. Furthermore, when facing new tasks, traditional underwater robot controllers require training from scratch, resulting in long training cycles that hinder practical applications. Designing and training controllers using meta-reinforcement learning can serve as a solution to these problems. Summary of the Invention
[0004] To address the shortcomings of existing underwater robot control technologies in terms of environmental adaptability, this invention provides a meta-reinforcement learning-based multi-task fast adaptive control method for underwater robots, based on the basic configuration of overdriven underwater robots. This method can meet the needs of underwater robots for various position tracking tasks during underwater operations, and utilizes a meta-reinforcement learning-based control method to better achieve rapid adaptability for various position tracking tasks.
[0005] The technical solution adopted in this invention is a fast adaptive control method for multi-task underwater robots based on meta-reinforcement learning, comprising:
[0006] Step 1: Construct the underwater robot's dynamic model and thrust allocation strategy matrix;
[0007] Step 2: Obtain position and velocity information through underwater robot sensors, and design three reinforcement learning controllers for different task requirements: position without overshoot, position with overshoot allowed, and thruster flexible control. The above controllers are based on neural network architecture, and their input parameters are the current relative position error and velocity, and the output is force and torque. The requirements of different tasks are achieved by setting the reward function.
[0008] Step 3: Introduce a meta-learning mechanism, build a meta-training platform, and use this platform to train the three reinforcement learning controllers mentioned above in the built dynamic model, and obtain a set of optimal initialization parameters to accelerate the subsequent specific task adaptation process.
[0009] Step 4: Deploy the obtained optimal initialization parameters and subtask reinforcement learning controller to the underwater robot and train it according to different tasks;
[0010] Step 5: Deploy the trained controller into engineering practice. Based on the real-time calculated relative distance and speed information to the target position, intelligently adjust the output of the thrusters to ensure accurate position control in various mission scenarios.
[0011] Step 1) includes the following steps:
[0012] Based on the six-degree-of-freedom dynamic characteristics of the underwater robot, the dynamic equations are established as follows:
[0013] Where M = M RB +M A M RB and M A Let v be the inertia matrix and added mass matrix of the AUV, where v = [uvwpqr]. T Includes the linear and angular velocity vectors of the AUV in a solid-state system; C(v) = C RB (v)+C A (v), where C RB (v) and C A (v) represents the Coriolis centripetal matrix and the additional Coriolis force and moment matrix of the AUV; D(v) = D L (v)+D Q (v), where D L (v) and D Q (v) represents the linear damping matrix and the quadratic drag matrix; g(η) is the restoring force vector; η = [x, y, z, φ, θ, ψ] contains the position vector and angle under the Earth-fixed system; τ = τ b +τ tet , τ b and τ tet Forces and torques generated for underwater robots, as well as external disturbance forces and torques.
[0014] For an overdriven underwater robot, the thrust of each thruster is: F = T + τ b ;
[0015] Where T is the thrust distribution matrix and F is the control variable of the thruster.
[0016] Step 2) specifically includes:
[0017] 2-1) Obtain position and velocity information through the sensors onboard the underwater robot and calculate the relative error with the target position;
[0018] 2-2) The three neural networks are uniformly initialized and used as reinforcement learning controllers for overshoot-free position tracking, overshoot-allowed position tracking, and thruster flexible position tracking; the inputs of the three controllers are forward position error and forward velocity, and the output is forward thrust.
[0019] 2-3) The reward function set for different controllers is used to calculate the cumulative reward and backpropagate to adjust the model parameters during multiple iterations, so that the cumulative reward is maximized and the model parameters at this time are taken as the optimal ones.
[0020] Step 3) specifically includes:
[0021] 3-1) During the meta-training phase, sub-task controllers are selected sequentially. A numerical simulation environment is constructed using the dynamic model defined in step 1. In each iteration of the simulation, the accumulated reward value is used to update the gradient of the neural network using the backpropagation algorithm, thereby driving the controller to gradually explore and achieve the goal of maximizing the accumulated reward, thus realizing the optimal control strategy. Furthermore, the fourth-order Runge-Kutta numerical integration method is used, based on the output τ of the sub-task reinforcement learning controller... b Iteratively solve for the current state vector v of the AUV;
[0022] 3-2) The reinforcement learning controller for the subtask is trained using a numerical simulation environment. At each time step during the training round, the current position of the AUV is calculated using integral operations, and then the position error relative to the target point is calculated. The controller receives the relative position error and the current velocity v as input information and dynamically calculates the control quantity τ. b ;
[0023] 3-3) During multiple iterations, the model parameters will be updated proportionally towards the weight parameters of the already trained subtask controller neural network. Where α is the update ratio. These are the initialization parameters for the neural network. The neural network parameters for subtask m are given, and the updated neural network parameters are used. The values are assigned to the model, allowing the model to continue iterative optimization under these parameters until the meta-training ends, at which point a set of optimal initialization parameters is obtained and saved.
[0024] The process involves deploying the optimal initialization parameters obtained through training in a simulation environment and the reinforcement learning controller selected based on different tasks to the underwater robot, and then training it in a real underwater environment; specifically, this includes:
[0025] 4-1) Load the optimized subtask reinforcement learning controller and optimal initialization parameters onto the AUV master control computer;
[0026] 4-2) Continue training the controller in the underwater environment and freeze the dimensional parameters of the remaining two controllers.
[0027] The trained subtask controller is applied in engineering practice to intelligently adjust the thruster output based on real-time calculated relative distance and velocity information to the target position. Specifically, this includes:
[0028] 5-1) Call step 2-1) to collect position and velocity information in real time, and calculate the relative position error Δx;
[0029] 5-2) Input Δx and velocity into the trained subtask reinforcement learning controller to obtain the X-axis thrust, which is then synthesized into a vector τ. b =[X,0,0,0,0,0] T ;
[0030] 5-3) Calculate the actuator motion control quantity F = Tτ using the thrust distribution matrix. b It is used to drive the actuators of underwater robots to generate corresponding motion mechanisms.
[0031] The present invention has the following beneficial effects and advantages:
[0032] 1. By adopting the controller technology based on meta-reinforcement learning in this invention, the underwater robot can be precisely controlled in uncertain and complex working environments by utilizing neural networks;
[0033] 2. Based on the meta-reinforcement learning controller technology, the optimal initialization parameters obtained in the simulation of this invention enable the sub-task reinforcement learning controller to perform a small amount of training in the real environment according to the set of parameters, complete the control transfer, and quickly adapt to different tasks. Attached Figure Description
[0034] Figure 1 This is a schematic diagram of the multi-task fast adaptive control method for underwater robots based on meta-reinforcement learning proposed in this invention. Detailed Implementation
[0035] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.
[0036] This invention provides a fast adaptive control method for multi-task underwater robots based on meta-reinforcement learning, comprising:
[0037] Step 1: Construct the underwater robot dynamics model framework and thrust allocation strategy matrix; including:
[0038] Based on the six-degree-of-freedom dynamic characteristics of the underwater robot, the dynamic equations are established as follows:
[0039] Where M = M RB +M A M RB and M A Let v be the inertia matrix and added mass matrix of the AUV, where v = [uvwpqr]. T Includes the linear and angular velocity vectors of the AUV in a solid-state system; C(v) = C RB (v)+C A (v), where C RB (v) and C A (v) represents the Coriolis centripetal matrix and the additional Coriolis force and moment matrix of the AUV; D(v) = D L (v)+D Q (v), where D L (v) and D Q (v) represents the linear damping matrix and the quadratic drag matrix; g(η) is the restoring force vector; η = [x, y, z, φ, θ, ψ] contains the position vector and angle under the Earth-fixed system; τ = τ b +τ tet , τ b and τ tet Forces and torques generated for underwater robots, as well as external disturbance forces and torques.
[0040] For an overdriven underwater robot, the thrust of each thruster is: F = T + τ b ;
[0041] Where T is the thrust distribution matrix and F is the control variable of the thruster.
[0042] Step 2: Obtain position and velocity information using the underwater robot's sensors. Three reinforcement learning controllers are designed to address diverse task requirements: one for position with no overshoot, one for position with allowable overshoot, and one for flexible thruster control. These controllers are based on a neural network architecture, with input parameters being the current relative position error and velocity, and outputs being force and torque. Different task requirements are achieved through the setting of a reward function; including:
[0043] 2-1) Obtain position and velocity information through the sensors onboard the underwater robot and calculate the relative error with the target position;
[0044] 2-2) The three neural networks are uniformly initialized and used as reinforcement learning controllers for overshoot-free position tracking, overshoot-allowed position tracking, and thruster flexible position tracking; the inputs of the three controllers are forward position error and forward velocity, and the output is forward thrust.
[0045] 2-3) Reward function settings for different controllers; The reward function is used as the basis for finding the optimal model parameters during training. That is, the cumulative reward is calculated and backpropagated to adjust the model parameters during multiple iterations so that the reward function is maximized, and the model parameters at this time are taken as the optimal ones.
[0046] In a position tracking controller without overshoot, the reward function r forbid =-|Δx t+1 |-e, where r forbid Let Δx be the reward of the underwater robot at time t. t+1 Let t be the relative position error of the underwater robot, and e be the position overshoot penalty term. When the current position of the underwater robot exceeds the target position, e = -10, and when the position does not exceed the target position, e = 0.
[0047] In an overshoot-allowed position tracking controller, the reward function r allow =-|Δx t+1 |;
[0048] In flexible position tracking of the thruster; the reward function r flexible =-|Δx t+1 |-λf t+1 Where λ is the flexibility coefficient, which is 0.001, and f t+1 This represents the force output by the underwater robot's thruster at time t+1.
[0049] Step 3 involves introducing a meta-learning mechanism, building a meta-training platform, and using this platform to train the three reinforcement learning controllers within the constructed dynamic model. This yields a set of optimal initialization parameters to accelerate the subsequent task adaptation process. Specifically, this includes:
[0050] 3-1) During the meta-training phase, sub-task controllers are selected sequentially, and a numerical simulation environment is constructed using the dynamic model defined in step 1. In each iteration of the simulation, the reward function guides the optimization direction of the neural network. By accumulating the obtained reward values, the backpropagation algorithm is used to update the gradient of the neural network, thereby driving the controller to gradually explore and achieve the goal of maximizing the accumulated reward, i.e., realizing the optimal control strategy. Furthermore, the fourth-order Runge-Kutta numerical integration method is used, based on the output τ of the sub-task reinforcement learning controller... b Iteratively solve for the current state vector v of the AUV;
[0051] 3-2) The reinforcement learning controller for the subtask is trained using a numerical simulation environment. At each time step during the training round, integral operations are used to determine the current position of the AUV, and then the position error relative to the target point is calculated. The controller receives the relative position error and the current velocity v as input information and dynamically calculates the control quantity τ. b ;
[0052] 3-3) During multiple iterations, the model parameters will be updated proportionally towards the weight parameters of the already trained subtask controller neural network, i.e. Where α is the update ratio. These are the initialization parameters for the neural network. The neural network parameters for subtask m are given, and the updated neural network parameters are used. The values are assigned to the model, allowing the model to continue iterative optimization under these parameters until the meta-training ends, at which point a set of optimal initialization parameters is obtained and saved.
[0053] 3-4) Save the parameters of the trained controller network.
[0054] Step 4 involves deploying the obtained optimal initialization parameters and subtask reinforcement learning controller to the underwater robot, and performing a small amount of training depending on the task; specifically including:
[0055] 4-1) Load the optimized and trained reinforcement learning controllers for the three forward, depth, and lateral positions onto the AUV master control computer;
[0056] 4-2) Continue training the three controllers in the underwater environment. When training a single controller, the AUV remains stationary in the dimensional parameters corresponding to the other two controllers.
[0057] Step 5: Deploy the trained controller into engineering practice. The control method schematic diagram is shown in the figure.
[0058] 5-1) Call step 2-1) to collect position and velocity information in real time, and calculate the relative position error Δx;
[0059] 5-2) Input Δx and velocity into the trained subtask reinforcement learning controller to obtain the X-axis thrust, and synthesize it into a vector τ. b =[X,0,0,0,0,0] T ;
[0060] 5-3) Calculate the actuator motion control quantity F = Tτ using the thrust distribution matrix. b It is used to drive the actuators of underwater robots to generate corresponding motion mechanisms.
[0061] The above description is merely an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, extensions, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.
Claims
1. A fast adaptive control method for multi-task underwater robots based on meta-reinforcement learning, characterized in that, include: Step 1: Construct the underwater robot's dynamic model and thrust allocation strategy matrix; Step 2: Obtain position and velocity information through underwater robot sensors, and design three reinforcement learning controllers for different task requirements: position without overshoot, position with overshoot allowed, and thruster flexible control. The above controllers are based on neural network architecture, and their input parameters are the current relative position error and velocity, and the output is force and torque. The requirements of different tasks are achieved by setting the reward function. Step 3: Introduce a meta-learning mechanism, build a meta-training platform, and use this platform to train the three reinforcement learning controllers mentioned above in the built dynamic model, and obtain a set of optimal initialization parameters to accelerate the subsequent specific task adaptation process. Step 4: Deploy the obtained optimal initialization parameters and subtask reinforcement learning controller to the underwater robot and train it according to different tasks; Step 5: Deploy the trained controller into engineering practice. Based on the real-time calculated relative distance and speed information to the target position, intelligently adjust the output of the thrusters to ensure accurate position control in various mission scenarios.
2. The multi-task fast adaptive control method for underwater robots based on meta-reinforcement learning according to claim 1, characterized in that, Step 1) includes the following steps: Based on the six-degree-of-freedom dynamic characteristics of the underwater robot, the dynamic equations are established as follows: Where M = M RB +M A M RB and M A Let v be the inertia matrix and added mass matrix of the AUV, where v = [uvwpqr]. T Includes the linear and angular velocity vectors of the AUV in a solid-state system; C(v) = C RB (v)+C A (v), where C RB (v) and C A (v) represents the Coriolis centripetal matrix and the additional Coriolis force and moment matrix of the AUV; D(v) = D L (v)+D Q (v), where D L (v) and D Q (v) represents the linear damping matrix and the quadratic drag matrix; g(η) is the restoring force vector; η = [x, y, z, φ, θ, ψ] contains the position vector and angle under the Earth-fixed system; τ = τ b +τ tet , τ b and τ tet Forces and torques generated for underwater robots, as well as external disturbance forces and torques.
3. The multi-task fast adaptive control method for underwater robots based on meta-reinforcement learning according to claim 1, characterized in that, For an overdriven underwater robot, the thrust of each thruster is: F = T + τ b ; Where T is the thrust distribution matrix and F is the control variable of the thruster.
4. The multi-task fast adaptive control method for underwater robots based on meta-reinforcement learning according to claim 1, characterized in that, Step 2) specifically includes: 2-1) Obtain position and velocity information through the sensors onboard the underwater robot and calculate the relative error with the target position; 2-2) The three neural networks are uniformly initialized and used as reinforcement learning controllers for overshoot-free position tracking, overshoot-allowed position tracking, and thruster flexible position tracking; the inputs of the three controllers are forward position error and forward velocity, and the output is forward thrust. 2-3) The reward function set for different controllers is used to calculate the cumulative reward and backpropagate to adjust the model parameters during multiple iterations, so that the cumulative reward is maximized and the model parameters at this time are taken as the optimal ones.
5. The multi-task fast adaptive control method for underwater robots based on meta-reinforcement learning according to claim 1, characterized in that, Step 3) specifically includes: 3-1) During the meta-training phase, sub-task controllers are selected sequentially. A numerical simulation environment is constructed using the dynamic model defined in step 1. In each iteration of the simulation, the accumulated reward value is used to update the gradient of the neural network using the backpropagation algorithm, thereby driving the controller to gradually explore and achieve the goal of maximizing the accumulated reward, thus realizing the optimal control strategy. Furthermore, the fourth-order Runge-Kutta numerical integration method is used, based on the output τ of the sub-task reinforcement learning controller... b Iteratively solve for the current state vector v of the AUV; 3-2) The reinforcement learning controller for the subtask is trained using a numerical simulation environment. At each time step during the training round, the current position of the AUV is calculated using integral operations, and then the position error relative to the target point is calculated. The controller receives the relative position error and the current velocity v as input information and dynamically calculates the control quantity τ. b ; 3-3) During multiple iterations, the model parameters will be updated proportionally towards the weight parameters of the already trained subtask controller neural network. Where α is the update ratio. These are the initialization parameters for the neural network. The neural network parameters for subtask m are given, and the updated neural network parameters are used. The values are assigned to the model, allowing the model to continue iterative optimization under these parameters until the meta-training ends, at which point a set of optimal initialization parameters is obtained and saved.
6. The multi-task fast adaptive control method for underwater robots based on meta-reinforcement learning according to claim 1, characterized in that, The process involves deploying the optimal initialization parameters obtained through training in a simulation environment and the reinforcement learning controller selected based on different tasks to the underwater robot, and then training it in a real underwater environment; specifically, this includes: 4-1) Load the optimized subtask reinforcement learning controller and optimal initialization parameters onto the AUV master control computer; 4-2) Continue training the controller in the underwater environment and freeze the dimensional parameters of the remaining two controllers.
7. The multi-task fast adaptive control method for underwater robots based on meta-reinforcement learning according to claim 1, characterized in that, The trained subtask controller is applied in engineering practice to intelligently adjust the thruster output based on real-time calculated relative distance and velocity information to the target position. Specifically, this includes: 5-1) Call step 2-1) to collect position and velocity information in real time, and calculate the relative position error Δx; 5-2) Input Δx and velocity into the trained subtask reinforcement learning controller to obtain the X-axis thrust, which is then synthesized into a vector τ. b =[X,0,0,0,0,0] T ; 5-3) Calculate the actuator motion control quantity F = Tτ using the thrust distribution matrix. b It is used to drive the actuators of underwater robots to generate corresponding motion mechanisms.